Updated 4.x design, features, and quickstart user manual pages

_docs-4/getting-started/design.md

@@ -32,8 +32,7 @@ machines.
 
 ## Components
 
-An instance of Accumulo includes many TabletServers, one Garbage Collector process,
-one Manager server and many Clients.
+An instance of Accumulo includes one or more Managers, TabletServers, Compactors, and GarbageCollectors and many clients.
 
 ### Tablet Server
 
@@ -45,7 +44,7 @@ values from all the files it has created and the sorted in-memory
 store.
 
 TabletServers also perform recovery of a tablet
-that was previously on a server that failed, reapplying any writes
+that was previously on a Tablet Server that failed, reapplying any writes
 found in the write-ahead log to the tablet.
 
 ### Garbage Collector
@@ -57,22 +56,23 @@ They will perform leader election among themselves to choose a single active ins
 
 ### Manager
 
-The Accumulo Manager is responsible for detecting and responding to TabletServer
-failure. It tries to balance the load across TabletServer by assigning tablets carefully
+The Accumulo Manager is responsible for managing Tablet, FaTE transaction, and external
+compaction state across the cluster. It assigns and balances the Tablets across TabletServers
 and instructing TabletServers to unload tablets when necessary. The Manager ensures all
 tablets are assigned to one TabletServer each, and handles table creation, alteration,
 and deletion requests from clients. The Manager also coordinates startup, graceful
 shutdown and recovery of changes in write-ahead logs when Tablet servers fail.
 
-Multiple managers may be run. The managers will choose among themselves a single manager,
-and the others will become backups if the manager should fail.
-
-### Tracer
+The Compaction Coordinator is a function that the primary Manager performs to coordinate
+the completion of major compactions using Compactor processes. The Coordinator is responsible
+for identifying what external compaction work needs to be done, and for communicating with the Compactors
+to assign work, get status updates, and cancel running external compactions.
 
-The Accumulo Tracer process supports the distributed timing API provided by Accumulo.
-One to many of these processes can be run on a cluster which will write the timing
-information to a given Accumulo table for future reference. See the
-[tracing documentation][tracing] for more information.
+Multiple managers may be run concurrently. The managers will choose among themselves a single manager
+to act as the primary manager and the other managers will act as assistant managers. Clients will
+connect only to the primary manager. The assistant managers participate in the execution and
+management of FaTE transactions. The primary manager also participates in the execution and management
+of FaTE transactions, and also performs tablet management and compaction coordinator functions.
 
 ### Monitor
 
@@ -88,27 +88,20 @@ Multiple Monitors can be run to provide hot-standby support in the face of failu
 forwarding of logs from remote hosts to the Monitor, only one Monitor process should be active
 at one time. Leader election will be performed internally to choose the active Monitor.
 
-### Compactor (experimental)
+### Compactor
 
-The Accumulo Compactor process is an optional application that can be used to run compactions
+The Accumulo Compactor process is an application that can be used to run compactions
 outside of the TabletServer. One to many Compactors can be run on a cluster and each Compactor
 process performs one compaction at a time. The Compactor registers its existence in ZooKeeper
-and communicates with the Compaction Coordinator to retrieve its work and to register the
-completion status of the compaction. The Compactor process will continue to perform compactions
-in situations where normal in-TabletServer compactions would fail, such as TabletServer restart
-and Tablet re-hosting.
+and communicates with the Compaction Coordinator in the primary Manager to retrieve its work and to register the
+completion status of the compaction.
 
-### Compaction Coordinator (experimental)
-
-The Accumulo Compaction Coordinator is an optional application that is required to run compactions
-outside of the TabletServer. The Coordinator is responsible for communicating with the
-TabletServers, to identify what external compaction work needs to be done, and the Compactors
-to assign work, get status updates, and cancel running external compactions.
+Compactors also perform the sorting phase of the write-ahead log recovery for tablets that were
+on a Tablet Server that failed. Once sorting completes, the tablet is assigned to a Tablet Server,
+which replays the sorted mutations. The Compactor looks for write-ahead log sorting work before
+requesting the next Major Compaction job.
 
-Multiple Coordinators may be run. The Coordinators will choose among themselves a single active Coordinator,
-and the others will become backups if the active Coordinator should fail.
-
-### Scan Server (experimental)
+### Scan Server
 
 The Accumulo Scan Server is an optional application that can be used to run scans on a tablet's data
 outside of the Tablet Server. Many Scan Servers can be run on a cluster and each Scan Server may run
@@ -120,6 +113,11 @@ Tablet Server. The Scan Server does not have any of the Tablet data that may res
 in-memory maps and the tablet may reference files that have been compacted as tablet metadata can
 be cached within the Scan Server (See Scan Server configuration properties).
 
+Scan Servers can also perform the sorting phase of the write-ahead log recovery for tablets that were
+on a Tablet Server that failed. Once sorting completes, the tablet is assigned to a Tablet Server,
+which replays the sorted mutations. The server property `sserver.wal.sort.concurrent.max` controls
+the number of threads in the Scan Server that will perform the write-ahead log sorting.
+
 ### Client
 
 Accumulo has a client library that can be used to write applications that write and read
@@ -129,11 +127,11 @@ data to/from Accumulo. See the [Accumulo clients documentation][clients] for mor
 
 Accumulo stores data in tables, which are partitioned into tablets. Tablets are
 partitioned on row boundaries so that all of the columns and values for a particular
-row are found together within the same tablet. The Manager assigns Tablets to one
-TabletServer at a time. This enables row-level transactions to take place without
+row are found together within the same tablet. When required, the Manager assigns a Tablet to one
+TabletServer. This enables row-level transactions to take place without
 using distributed locking or some other complicated synchronization mechanism. As
 clients insert and query data, and as machines are added and removed from the
-cluster, the Manager migrates tablets to ensure they remain available and that the
+cluster, the Manager may migrate tablets to ensure they remain available and that the
 ingest and query load is balanced across the cluster.
 
 ![data distribution]({{ site.baseurl }}/images/docs/data_distribution.png)
@@ -171,18 +169,19 @@ locality group. The diagram below shows the logical view and HDFS file view of a
 
 ## Compactions
 
-In order to manage the number of files per tablet, periodically the TabletServer
-performs Major Compactions of files within a tablet, in which some set of RFiles
-are combined into one file. The previous files will eventually be removed by the
-Garbage Collector. This also provides an opportunity to permanently remove
-deleted key-value pairs by omitting key-value pairs suppressed by a delete entry
-when the new file is created. See the [compaction documentation][compaction]
-for more information.
+In order to manage the number of files per tablet the Manager will periodically
+identify the tablets that need a Major Compaction and will give the highest
+priority job to a Compactor process when it requests the next job. The Major
+Compaction will merge data from an input set of RFiles into a single output file.
+The previous files will eventually be removed by the Garbage Collector.
+This also provides an opportunity to permanently remove deleted key-value pairs
+by omitting key-value pairs suppressed by a delete entry when the new file is
+created. See the [compaction documentation][compaction] for more information.
 
 ## Splitting
 
-When a table is created it has one tablet. As the table grows its initial
-tablet eventually splits into two tablets. It's likely that one of these
+When a table is created without split points it will have one tablet. As the table grows its initial
+tablet eventually splits into two tablets. If hosted, it's likely that one of these
 tablets will migrate to another tablet server. As the table continues to grow,
 its tablets will continue to split and be migrated. The decision to
 automatically split a tablet is based on the size of a tablets files. The
@@ -194,7 +193,8 @@ splitting.
 
 As data is deleted from a table, tablets may shrink. Over time this can lead
 to small or empty tablets. To deal with this, the [merging of tablets][merging]
-was introduced in Accumulo 1.4.
+was introduced in Accumulo 1.4. Accumulo 4.0 introduces [automatic merging][merging2]
+of tablets to optionally automate merging of tablets based on configurable thresholds.
 
 ## Fault-Tolerance
 
@@ -209,7 +209,7 @@ grouped by tablet.  TabletServers can quickly apply the mutations from the sorte
 that are destined for the tablets they have now been assigned.
 
 TabletServer failures are noted on the Manager's monitor page, accessible via
-`http://manager-address:9995/monitor`.
+`http://monitor-address:9995`.
 
 ![failure handling]({{ site.baseurl }}/images/docs/failure_handling.png)
 
@@ -218,5 +218,6 @@ TabletServer failures are noted on the Manager's monitor page, accessible via
 [tracing]: {% durl troubleshooting/tracing %}
 [clients]: {% durl getting-started/clients %}
 [merging]: {% durl getting-started/table_configuration#merging-tablets %}
+[merging2]: {% durl administration/merging %}
 [compaction]: {% durl getting-started/table_configuration#compaction %}
 [caching]: {% durl administration/caching %}

_docs-4/getting-started/features.md

@@ -64,9 +64,9 @@ can share a pool of datanodes.
 ### Manager fail over
 
 Multiple managers can be configured.  Zookeeper locks are used to determine
-which manager is active.  The remaining managers simply wait for the current
-manager to lose its lock.  Current manager state is held in the metadata table
-and Zookeeper.
+which manager is the primary manager.  The remaining managers will attempt to
+take on the primary manager role if the current primary manager loses its lock.
+Current manager state is held in the metadata table and Zookeeper.
 
 ### Logical time
 
@@ -305,14 +305,18 @@ Accumulo provides two BlockCacheManager implementations (LruBlockCacheManager an
 TinyLfuBlockCacheManager) that construct on-heap block caches. Users can provide
 alternate BlockCacheManager implementations using the property `tserver.cache.manager.class`.
 
+### On-Demand Tablet Unloader
+
+Tablets that have their availability set to on-demand can be unhosted. Accumulo has a default
+implementation that unloads idle tablets after a configurable amount of time.
+
 ## General Administration
 
 ### Monitor page
 
 The [Accumulo Monitor][monitor] provides basic information about the system health and
-performance.  It displays table sizes, ingest and query statistics, server
-load, and last-update information.  It also allows the user to view recent
-diagnostic logs and traces.
+performance.  It displays information about the server processes, table sizes, ingest,
+query and compaction statistics, FaTE and recovery information, and last-update information.
 
 <a class="p-3 border rounded d-block" href="/images/accumulo-monitor-1.png">
 <img src="/images/accumulo-monitor-1.png" class="img-fluid rounded" alt="monitor overview"/>
@@ -386,9 +390,8 @@ used to accommodate new data patterns in an existing table.
 
 ### Tablet Merging
 
-Tablet merging is a new feature. Merging of tablets can be requested in the
-shell; Accumulo does not merge tablets automatically. If desired, the METADATA
-tablets can be merged.
+Merging of tablets can be requested in the shell or tables can be configured to
+merge tablets automatically.
 
 ### Table Cloning
 
@@ -413,6 +416,13 @@ Added an operation to efficiently delete a range of rows from a table. Tablets
 that fall completely within a range are simply dropped. Tablets overlapping the
 beginning and end of the range are split, compacted, and then merged.
 
+### Tablet Hosting
+
+User table tablets are hosted optionally be default. Users can configure tablets
+for their tables to always or never be hosted. Optionally hosted tablets will be
+assigned to a Tablet Server if needed to support a user operation and then unhosted
+after a configurable period of time of inactivity.
+
 [FATE]: {% durl administration/fate %}
 [maven-accumulo-plugin]: {{ site.baseurl }}/release/accumulo-1.6.0/#maven-plugin
 [ConditionalMutation]: {% jurl org.apache.accumulo.core.data.ConditionalMutation %}

_docs-4/getting-started/quickstart.md

@@ -25,11 +25,7 @@ process name referenced in this documentation.
 ## Setup for testing or development
 
 If you are setting up Accumulo for **testing or development,** consider using
-the following tools:
-
-* [Uno] sets up Accumulo on a single machine for development
-* [Muchos] sets up Accumulo on a cluster (optionally launched in Amazon EC2 and
-  Microsoft Azure VM)
+[Uno] to set up a single instance Accumulo cluster.
 
 If you are setting up Accumulo for a **production** environment, follow the
 instructions below.
@@ -138,7 +134,7 @@ instance.
 The following command will perform the initialization.
 
 ```
-accumulo init
+accumulo inst init
 ```
 
 The initialization command will prompt for the following information.
@@ -176,7 +172,7 @@ command followed by the service name. For example, to start only the tserver,
 run:
 
 ```
-accumulo tserver
+accumulo proc tserver
 ```
 
 The process will run in the foreground. Use ctrl-c to quit.
@@ -226,11 +222,9 @@ file contains sections that should be configured with a list of node names in ya
  * [monitor] : Node where Accumulo monitoring web server is run.
  * [tserver] : Accumulo worker processes. List all of the nodes where tablet
    servers should run.
- * [sserver] : Optional. List of all nodes where scan servers should run.
- * [compaction.coordinator] : Optional. Must specify one node. Can specify a few
-   for fault tolerance.
- * [compaction.compactor] : Optional. Accumulo external compactor processes. List of
+ * [compactor] : Accumulo external compactor processes. List of
    all nodes where compactors should run.
+ * [sserver] : Optional. List of all nodes where scan servers should run.
 
 The Accumulo, Hadoop, and Zookeeper software should be present at the same
 location on every node. Also, the files in the `conf` directory must be copied to
@@ -290,10 +284,8 @@ When finished, use the following commands to stop Accumulo:
 [manager]: {% durl configuration/files#managers %}
 [tserver]: {% durl configuration/files#tservers %}
 [tracer]: {% durl configuration/files#tracers %}
-[compaction.coordinator]: {% durl configuration/files#compaction%20coordinator %}
-[compaction.compactor]: {% durl configuration/files#compaction%20compactor %}
+[compactor]: {% durl configuration/files#compaction%20compactor %}
 [sserver]: {% durl configuration/files#sserver %}
-[Uno]: https://github.com/apache/fluo-uno
-[Muchos]: https://github.com/apache/fluo-muchos
+[Uno]: https://github.com/apache/accumulo-uno
 [Erasure Coding]: https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html
 [ec-guide]: {% durl administration/erasure-coding %}