Skip to article

Storage and observability

FoundationDB

Database nodes and persistent volume operations

This project operates nodes and persistent volumes for native FoundationDB. A controller retains desired configuration outside the managed database; nodes check local resources and maintain processes; FoundationDB itself owns transactions, replication, recovery and migration. Desired configuration, execution and data observations support startup, changes, draining and takeover.

Change runtime configuration while retaining data When adding or changing processes in an existing cluster, submit the complete desired configuration, observe node adoption and read known data through the database interface. Evacuate a data process before removing it; unrelated processes retain their directories and responsibilities.

Preserve continuity in configuration and data

An operator submits a complete cluster declaration describing nodes, volumes, processes and coordinators. The controller stores this desired configuration outside the database it manages. Configuration reads and management therefore have an independent persistent authority when the database is unavailable.

Every node uses the same read-only runtime image. Its actual local address on the controller connection selects an assignment; machine and volume identities are checked before native processes start. The image supplies programs and connection material, while assignments, data and writable state are established separately at runtime.

The layered architecture separates persisted intent and the shared image, node execution, and owned volumes. Configuration and image material reach nodes; native processes access XFS data.
Separate intent, running processes and data volumesRead by responsibility: the controller retains intent, nodes use the shared image and check local resources, and native database processes access their owned XFS volumes. Saved configuration, running processes and database availability are checked separately.

Initial setup and ordinary restart follow different paths

Initial setup requires a complete cluster declaration, explicit ownership of nodes and devices, and seed connection material for that cluster. The controller stores the declaration independently of the managed database. Each node obtains its runtime requirements from its current assignment.

Initializing a new volume is a separately authorized maintenance action: disable the target node, exclude previous writers, then act on the admitted device. An ordinary restart reuses the existing volumes and the connection file maintained by the database. A connection failure, missing file or blocked startup must not trigger automatic reseeding or a data reset.

  1. Save the complete desired configuration

    Declare the relationships among cluster, nodes, volumes, processes and coordinators, then confirm the current declaration is stored.

  2. Establish or check volume ownership

    Explicitly initialize a new volume; verify and reuse an existing one. Settle old writers before changing ownership.

  3. Observe the node, then read the database

    Check actual processes and listeners, then read independently expected data through the database interface. Configuration and data results are established separately.

Establish the operation before touching a disk

Volume initialization requires an explicit grant. The node must have disabled intent, the operation binds the exact device in its current declaration, and the operator authorizes loss of unmarked content. A control volume receives admitted cluster connection material before durable ownership information is published.

Ordinary startup reuses matching owned volumes and preserves process directories and the native writable connection file. Missing markers, conflicting ownership or damaged existing connection material block startup. The operator must also exclude aliases and writers on a previous host; a matching marker alone cannot establish those premises.

Device identity, ended prior writers and continuing exclusion jointly lead to reopening and readback. No missing premise can be replaced by either of the others.
Reopen existing data only after all three premises holdAll three inputs are required: identity and ownership match independently retained expectations, prior writers have ended, and the current owner maintains exclusion. They are not alternative paths. A separate readback checks the data after reopening.
Three distinct grants of authority
TaskPremiseCompleted result
Initialize a volumeDisabled node; authority over the exact devicePrepared filesystem and ownership information, without database startup
Reuse an owned volumeMatching identity, writer exclusion and a valid connection filePreserved data, logs and native connection state
Create the first clusterIndependent evidence that participating volumes were never configuredNative database creation; query failure grants no such permission

Observe commitment and application separately

A successful configuration write establishes durable intent. The node then checks devices, memory and runtime conditions, applies the change and reports both desired and effective identities. A whole configuration becomes running only after every declared process has been observed with its expected executable, arguments, data paths and listeners.

An unchanged configuration neither rewrites the monitor file nor restarts processes. Eligible parameter changes can replace only the affected process. Disablement or confirmed unassignment waits for processes, IO and mounts to settle before reporting stopped. A failed controller read preserves an admitted running configuration, while cold startup still needs a current assignment.

Scope maintenance to the selected processes

A coordinator endpoint change first adds a successor while retaining the old endpoint. The operation confirms reachability of the required old majority and every new member, performs the database handoff, then updates the seed file used by new clients for their first connection. Only then can the old endpoint retire. The handoff and seed update have separate results.

Before removing a data-bearing process, FoundationDB must evacuate data from its selected endpoint and completion must be established. This scoped maintenance preserves unrelated processes and retained directories. An ordinary restart that keeps its disks requires no evacuation; changing device ownership or data-directory mappings requires prior disablement and independent settlement of earlier writers.

Locate the problem in configuration, node or database state

An investigation asks three questions: what does the controller currently require, what has this node actually applied, and what can the database read or write? These facts have different sources and times. Saving desired configuration does not establish node adoption; a running monitor does not establish every required database process; a missing database sample does not establish a stopped service.

If the controller is temporarily unreadable, an active node retains its authorized configuration, while a cold node still needs its current assignment. When only one process changes, observe that process’s replacement and any required evacuation. Preserve unrelated processes and existing data directories instead of expanding local maintenance into cluster reconstruction.

  • Supported conclusion: the current report matches actual processes

    The complete executable, arguments, data directory and listener set must match before the report establishes that the node applied the configuration.

  • Unsupported conclusion: a matching marker excludes every other writer

    Old hosts, device aliases and surviving processes still need independent exclusion. An ownership marker cannot establish that premise.

  • Check next: database sampling is unavailable

    Retain the source and time of the failed observation, then use the required database interface to check the business result. Do not turn absence into zero or downtime.

Keep the meaning of each observation intact

Database sampling runs independently of management requests. Availability, replication, filesystem headroom, migration backlog and transaction probes retain their sources and units. Collection failure is distinct from observed database unavailability, and missing fields remain missing. Runtime application reports and database health answer separate questions.

The design calls for verification of preserved data after reopening volumes, formatter dispatch only to authorized targets, selective process changes and retained responsibility after lost results or controller interruption. The evidence belongs at persistent readback, native process and resource boundaries; one successful acknowledgement cannot substitute for all of them.

What an observation establishes
ObservationSupported interpretationSeparate evidence required
Node configuration appliedThe corresponding native processes were observedDatabase availability and full replication
Queued and in-flight bytesInstantaneous migration backlogCompletion of a particular process evacuation
Transaction probe latencySample transaction time from the controllerApplication request distributions and end-to-end percentiles