Storage and observability
FoundationDB
Database nodes and persistent volume operations
This project operates nodes and persistent volumes for native FoundationDB. A controller retains desired configuration outside the managed database; nodes check local resources and maintain processes; FoundationDB itself owns transactions, replication, recovery and migration. Desired configuration, execution and data observations support startup, changes, draining and takeover.
Change runtime configuration while retaining data When adding or changing processes in an existing cluster, submit the complete desired configuration, observe node adoption and read known data through the database interface. Evacuate a data process before removing it; unrelated processes retain their directories and responsibilities.
Preserve continuity in configuration and data
An operator submits a complete cluster declaration describing nodes, volumes, processes and coordinators. The controller stores this desired configuration outside the database it manages. Configuration reads and management therefore have an independent persistent authority when the database is unavailable.
Every node uses the same read-only runtime image. Its actual local address on the controller connection selects an assignment; machine and volume identities are checked before native processes start. The image supplies programs and connection material, while assignments, data and writable state are established separately at runtime.
Initial setup and ordinary restart follow different paths
Initial setup requires a complete cluster declaration, explicit ownership of nodes and devices, and seed connection material for that cluster. The controller stores the declaration independently of the managed database. Each node obtains its runtime requirements from its current assignment.
Initializing a new volume is a separately authorized maintenance action: disable the target node, exclude previous writers, then act on the admitted device. An ordinary restart reuses the existing volumes and the connection file maintained by the database. A connection failure, missing file or blocked startup must not trigger automatic reseeding or a data reset.
Save the complete desired configuration
Declare the relationships among cluster, nodes, volumes, processes and coordinators, then confirm the current declaration is stored.
Establish or check volume ownership
Explicitly initialize a new volume; verify and reuse an existing one. Settle old writers before changing ownership.
Observe the node, then read the database
Check actual processes and listeners, then read independently expected data through the database interface. Configuration and data results are established separately.
Establish the operation before touching a disk
Volume initialization requires an explicit grant. The node must have disabled intent, the operation binds the exact device in its current declaration, and the operator authorizes loss of unmarked content. A control volume receives admitted cluster connection material before durable ownership information is published.
Ordinary startup reuses matching owned volumes and preserves process directories and the native writable connection file. Missing markers, conflicting ownership or damaged existing connection material block startup. The operator must also exclude aliases and writers on a previous host; a matching marker alone cannot establish those premises.
| Task | Premise | Completed result |
|---|---|---|
| Initialize a volume | Disabled node; authority over the exact device | Prepared filesystem and ownership information, without database startup |
| Reuse an owned volume | Matching identity, writer exclusion and a valid connection file | Preserved data, logs and native connection state |
| Create the first cluster | Independent evidence that participating volumes were never configured | Native database creation; query failure grants no such permission |
Observe commitment and application separately
A successful configuration write establishes durable intent. The node then checks devices, memory and runtime conditions, applies the change and reports both desired and effective identities. A whole configuration becomes running only after every declared process has been observed with its expected executable, arguments, data paths and listeners.
An unchanged configuration neither rewrites the monitor file nor restarts processes. Eligible parameter changes can replace only the affected process. Disablement or confirmed unassignment waits for processes, IO and mounts to settle before reporting stopped. A failed controller read preserves an admitted running configuration, while cold startup still needs a current assignment.
Scope maintenance to the selected processes
A coordinator endpoint change first adds a successor while retaining the old endpoint. The operation confirms reachability of the required old majority and every new member, performs the database handoff, then updates the seed file used by new clients for their first connection. Only then can the old endpoint retire. The handoff and seed update have separate results.
Before removing a data-bearing process, FoundationDB must evacuate data from its selected endpoint and completion must be established. This scoped maintenance preserves unrelated processes and retained directories. An ordinary restart that keeps its disks requires no evacuation; changing device ownership or data-directory mappings requires prior disablement and independent settlement of earlier writers.
Locate the problem in configuration, node or database state
An investigation asks three questions: what does the controller currently require, what has this node actually applied, and what can the database read or write? These facts have different sources and times. Saving desired configuration does not establish node adoption; a running monitor does not establish every required database process; a missing database sample does not establish a stopped service.
If the controller is temporarily unreadable, an active node retains its authorized configuration, while a cold node still needs its current assignment. When only one process changes, observe that process’s replacement and any required evacuation. Preserve unrelated processes and existing data directories instead of expanding local maintenance into cluster reconstruction.
Supported conclusion: the current report matches actual processes
The complete executable, arguments, data directory and listener set must match before the report establishes that the node applied the configuration.
Unsupported conclusion: a matching marker excludes every other writer
Old hosts, device aliases and surviving processes still need independent exclusion. An ownership marker cannot establish that premise.
Check next: database sampling is unavailable
Retain the source and time of the failed observation, then use the required database interface to check the business result. Do not turn absence into zero or downtime.
Keep the meaning of each observation intact
Database sampling runs independently of management requests. Availability, replication, filesystem headroom, migration backlog and transaction probes retain their sources and units. Collection failure is distinct from observed database unavailability, and missing fields remain missing. Runtime application reports and database health answer separate questions.
The design calls for verification of preserved data after reopening volumes, formatter dispatch only to authorized targets, selective process changes and retained responsibility after lost results or controller interruption. The evidence belongs at persistent readback, native process and resource boundaries; one successful acknowledgement cannot substitute for all of them.
| Observation | Supported interpretation | Separate evidence required |
|---|---|---|
| Node configuration applied | The corresponding native processes were observed | Database availability and full replication |
| Queued and in-flight bytes | Instantaneous migration backlog | Completion of a particular process evacuation |
| Transaction probe latency | Sample transaction time from the controller | Application request distributions and end-to-end percentiles |