[martonp]

This PR introduces server/mesh, which provides the infrastructure for running DCRDEX as a two-node master/slave mesh.

This package lays the foundation for converting the server into a two-node replicated state machine. It handles all the communication between the two nodes and exposes an API for the rest of the server to integrate with. The application and database changes will follow separately. Those changes mainly involve refactoring the existing logic into command handlers, event appliers, master workers, and state loaders.

I've already completed the meshification of the rest of the server. Those changes still need some polish before review, but they can be viewed here:

https://github.com/martonp/dcrdex/tree/mesh

I also have a mainnet mesh running on these two nodes:

• 2.28.52.0:7232
• 204.168.178.234:7232

Let me know if you want to test it out, I can send prepaid bonds.

The goal was to preserve the existing client/server protocol. The only changes to the client are that the client can learn the second server's address and reconnect there if its current server becomes unavailable. Legacy clients can still connect to one server using the existing API.

Application API

Service is the entry point to the mesh package. The application registers four things:

• Events describe changes to the application state that both nodes apply in the same order. The application state may only be updated by a well defined event. The application registers an EventApplier for each event kind. It must commit the database changes in one transaction, then update the in-memory state after that transaction succeeds.
• Commands correspond to client or admin API requests that may change shared state. API handlers submit them to the mesh service, which executes them on the master. The application’s command handlers validate the requests and produce events when changes are needed.
• MasterWorkers run background tasks while the node is master, such as advancing markets and monitoring swaps. These produce server originated events, as opposed to events which are emitted due to a Command.
• StateLoaders load the in-memory representation of the active durable state (the state in the DB that is required for active orders and active swaps). These are run once at startup.

The full conversion currently has 10 command types and 16 event types. For example, the limit command validates an order and can produce an order_accepted event. A market worker produces events such as advance_epoch or epoch_processed. The application registers these types and implements their behavior. The mesh package is not aware of what they do.

Event application and replication

The master produces events containing the inputs needed for both nodes to apply the same state transition. Event application can only include deterministic computation based on the event itself and the previous state.

Each event’s database transaction records an event-log entry containing its sequence number and tip hash. Mesh applies events one at a time and streams committed entries from the master to the slave.

After successfully connecting, the slave subscribes to the master's event stream using its last committed event log position. The master reads later entries from its database and sends them in batches. The slave applies the events in order and checks their expected positions and hashes. The master waits for each batch to be acknowledged before sending the next.

Replication is asynchronous. The master can commit new events without waiting for the slave. Waiting for acknowledgement limits the event stream, not local event application.

The tip hash identifies event history by chaining each event to the previous tip. Starting from the same state, both nodes must produce the same state changes when applying the same events. A tip-hash mismatch while applying a replicated event halts the slave.

Commands

The mesh service executes commands on the master and delivers responses through the node that received the request. A command can produce an event or complete without changing shared state.

Response timing depends on where the request arrives:

• On the master: the response can be returned after local event application, before replication.
• On the slave: the slave forwards the command to the master. The master applies the event and sends it together with the command result. The slave applies the event before returning the result.

Commands that fail or complete without an event send their replies separately from the event stream.

A command may commit on the master even if the slave never receives its result. If the slave cannot determine the outcome, it returns ResultUnavailableError to the client. The application must make command handlers idempotent so the client can retry the same request without having a second event be emitted.

Startup, handshake, and readiness

A node configured with a peer must complete a handshake before serving clients. On each connection, the nodes perform a handshake and compare configurations, roles, and event-log positions to determine compatibility and select the master. Both nodes must share the same signing key and compatible market and asset configurations.

After the handshake is complete, the state loaders must run before event application / master workers can begin.

Then, before clients can connect:

• The master starts its master workers and waits for all of them to become ready.
• The slave subscribes to the master's event stream, and applies all events up to the master's tip.

Role selection and divergent histories

The handshake compares the nodes’ event histories:

• Equal: the same sequence number and tip hash.
• One extends the other: the longer log contains the shorter log’s last entry with the same tip hash.
• Divergent: the logs have different tip hashes at the same sequence number.

Histories are compatible if they are equal or one extends the other.

For compatible histories, an existing master keeps its role if its log is at least as long as the peer’s. If neither node is master, the longer log wins, with the lower node ID breaking a tie. The other node becomes slave and catches up if needed.

With divergent histories, an existing master rejects the joining peer and keeps serving. The joining peer halts. If neither or both nodes are master, both halt.

Two masters with equal histories also halt.

The exact role-selection rules are defined in reducer.go.

A node that has not established a role halts if the peer’s configuration is incompatible. An established master rejects the incompatible peer and keeps serving, so a misconfigured joining node cannot take it offline.

Snapshots

A fresh node can load a snapshot from a master instead of replaying the entire history. Replaying the full history is not possible if the node was created pre-mesh.

A snapshot includes all account data, active trade/swap data, and historical trade/swap data for the past 30 days. This logic is not part of this PR. The mesh package just handles the transfer.

A snapshot includes the event-log position corresponding to its database state. After loading it, the slave subscribes to events after that position.

Failover and shutdown

An established slave that loses its master waits 30 seconds before promoting itself, giving the master time to reconnect. A slave still performing its initial sync returns to pending if it loses the connection and does not promote from that state.

During graceful shutdown, the master stops its workers and closes event publication and waits for the slave to catch up. If the slave catches up, the master requests handoff. On receiving the request, the slave promotes immediately, avoiding the usual promotion delay.

Two failure scenarios are important:

• The master may have committed events that the slave never received. If the master crashes and the slave takes over, those events are absent from the promoted node. If the new master commits a conflicting event at the same sequence number, the histories fork. When the old process restarts and joins the new master, the returning node halts while the new master keeps serving.
• A lost connection does not prove the master stopped. During a network partition, the master can continue running while the slave eventually promotes. When they reconnect, equal or divergent histories cause both nodes to halt.

Mesh does not reconcile divergent histories or automatically discard committed events. The operator has to decide which history to keep. In the first failure scenario, which can happen quite easily if the master crashes, this is very simple. This is why we do not halt the serving master. In the split-brain scenario, which is rare, this would be a larger challenge.

Client message relaying / presence

The mesh package also provides the functionality to relay client messages between nodes and query which users are connected to the peer.

Messages can target an individual user or be broadcast to the peer's clients. This lets the master send swap requests and notifications to users connected to the slave.

The application combines the peer's response to a presence query with its local connection state to determine whether a user is still connected to the mesh. The master uses this when deciding whether to revoke booked orders after a prolonged disconnection.