Backup & Recovery¶

FoundationDB provides robust backup, restore, and disaster recovery capabilities. This guide covers continuous backup to blob storage, point-in-time recovery, and cross-datacenter replication.

Overview¶

FoundationDB's backup system offers:

Backup Architecture¶

Backup agents run as separate processes that read mutation logs from the database and write them to a backup destination. Multiple agents can run for redundancy and performance.

graph LR
    subgraph "FoundationDB Cluster"
        TLog[Transaction Logs<br/>Mutations]
        SS[Storage Servers]
    end

    subgraph "Backup System"
        BA1[Backup Agent 1]
        BA2[Backup Agent 2]
    end

    subgraph "Backup Storage"
        S3[(S3 / Blob Store)]
        FS[(Filesystem)]
    end

    TLog --> BA1
    TLog --> BA2
    BA1 --> S3
    BA2 --> S3
    BA1 -.-> FS
    BA2 -.-> FS

    style S3 fill:#ff9800,color:#000
    style FS fill:#4caf50,color:#fff

Backup Destinations¶

URL Formats¶

Filesystem Backup¶

fdbbackup start -d file:///mnt/backup/fdb

Requirements: - Path must be accessible to all backup agents - Use network filesystem (NFS, EFS) for multi-machine setups - Ensure sufficient disk space for backup history

Blob Store Backup¶

fdbbackup start -d "blobstore://s3.amazonaws.com:443/my-bucket/fdb-backup?bucket=my-bucket&region=us-east-1"

Blob Store URL Parameters¶

Blob Store Credentials¶

Create a JSON credentials file:

{
  "accounts": {
    "s3.amazonaws.com": {
      "api_key": "AKIAIOSFODNN7EXAMPLE",
      "secret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    }
  }
}

Set via environment variable:

export FDB_BLOB_CREDENTIALS=/path/to/credentials.json

Or use AWS SDK authentication:

fdbbackup start -d "blobstore://s3.amazonaws.com/bucket?sdk_auth=true"

Starting a Backup¶

Basic Backup¶

fdbbackup start -C /etc/foundationdb/fdb.cluster -d file:///backup/fdb

Backup with Snapshot¶

Include a full snapshot for faster restores:

fdbbackup start -d file:///backup/fdb -z

The -z flag (or --no_stop_when_done) continues backing up after the initial snapshot.

Backup with Tags¶

Run multiple independent backups:

# Primary backup to S3
fdbbackup start -t primary -d "blobstore://s3.amazonaws.com/prod-backup"

# Secondary backup to local storage
fdbbackup start -t local -d file:///mnt/backup/fdb

Backup with Key Range¶

Back up only specific key ranges:

fdbbackup start -d file:///backup/fdb -k '\x00' '\xff'

Managing Backups¶

Check Backup Status¶

fdbbackup status -C /etc/foundationdb/fdb.cluster

Or for a specific tag:

fdbbackup status -t mytag

Example output:

Using cluster file `/etc/foundationdb/fdb.cluster'.

The backup on tag `default' is restorable but continuing to
 blobstore://s3.amazonaws.com:443/my-bucket (last completed log version is 12345678).

Backup is restorable to version 12345678 (approximately 5 seconds old).

List Backups¶

fdbbackup list -b file:///backup/fdb

Pause/Resume Backup¶

# Pause (backup continues but range log writing pauses)
fdbbackup modify -t default --active-snapshot-interval 0

# Resume
fdbbackup modify -t default --active-snapshot-interval 86400

Stop Backup¶

# Stop gracefully (keeps backup restorable)
fdbbackup discontinue -t default

# Abort immediately
fdbbackup abort -t default

Wait for Backup¶

Wait for backup to complete or reach a restorable state:

fdbbackup wait -t default

fdbbackup Command Reference¶

Common Options¶

Restoring from Backup¶

Basic Restore¶

fdbrestore start -r file:///backup/fdb -C /etc/foundationdb/fdb.cluster

Restore to Specific Version¶

fdbrestore start -r file:///backup/fdb --version 12345678

Restore to Specific Timestamp¶

fdbrestore start -r file:///backup/fdb --timestamp "2025-02-03 10:30:00"

Restore with Key Transformation¶

Restore to different key prefixes:

fdbrestore start -r file:///backup/fdb \
  --remove_prefix "prod/" \
  --add_prefix "staging/"

Check Restore Status¶

fdbrestore status -C /etc/foundationdb/fdb.cluster

Wait for Restore¶

fdbrestore wait

Abort Restore¶

fdbrestore abort

fdbrestore Command Reference¶

Restore Options¶

Running Backup Agents¶

Backup agents perform the actual backup work. Run them on your cluster machines.

Starting Backup Agent¶

backup_agent -C /etc/foundationdb/fdb.cluster

Backup Agent in foundationdb.conf¶

[backup_agent]
command = /usr/lib/foundationdb/backup_agent/backup_agent
logdir = /var/log/foundationdb

Multiple Agents¶

Run multiple backup agents for redundancy and performance. They coordinate automatically.

Agent with Blob Credentials¶

backup_agent -C /etc/foundationdb/fdb.cluster \
  --blob-credentials /path/to/credentials.json

Disaster Recovery (DR)¶

DR provides real-time replication to a standby cluster for immediate failover.

DR vs Backup¶

DR Architecture¶

graph LR
    subgraph "Primary Datacenter"
        P_TLog[Transaction Logs]
        P_SS[Storage Servers]
        P_Client[Clients<br/>Read/Write]
    end

    subgraph "DR Datacenter"
        DR_TLog[Transaction Logs]
        DR_SS[Storage Servers]
        DR_Client[Clients<br/>Read-Only]
    end

    DR_Agent[DR Agent]

    P_Client --> P_TLog
    P_TLog --> P_SS
    P_TLog --> DR_Agent
    DR_Agent --> DR_TLog
    DR_TLog --> DR_SS
    DR_Client -.-> DR_SS

    style P_TLog fill:#4caf50,color:#fff
    style DR_TLog fill:#ff9800,color:#000

Setting Up DR¶

1. Start DR on primary cluster:

   Bash

   fdbdr start -s /path/to/primary.cluster -d /path/to/dr.cluster
   

2. Run DR agents:

   Bash

   dr_agent -s /path/to/primary.cluster -d /path/to/dr.cluster
   

3. Check DR status:

   Bash

   fdbdr status -s /path/to/primary.cluster -d /path/to/dr.cluster
   

DR Switchover¶

Perform a planned switchover:

fdbdr switch -s /path/to/primary.cluster -d /path/to/dr.cluster

After switchover: - DR cluster becomes primary - Original primary becomes standby - Applications reconnect to new primary

DR Failover¶

For unplanned failover when the primary cluster is unavailable and fdbdr switch cannot be used (since it requires both clusters):

1. Abort the DR job to unlock the destination cluster and stop replication:

   Bash

   fdbdr abort -s /path/to/primary.cluster -d /path/to/dr.cluster
   

   After abort, the DR cluster retains a consistent snapshot of the source database from some point in the past. The cluster is unlocked and becomes writable.

2. Redirect applications to the DR cluster by updating their cluster files to point to the DR cluster.

3. (Optional) Set up reverse DR once the original primary is recovered, to replicate back from the now-active DR cluster:

   Bash

   fdbdr start -s /path/to/dr.cluster -d /path/to/primary.cluster
   dr_agent -s /path/to/dr.cluster -d /path/to/primary.cluster
   

# Connect to the DR cluster
fdbcli -C /path/to/dr.cluster
fdbcli> unlock <UID>

fdbdr Command Reference¶

TLS Support¶

All backup commands support TLS when connecting to TLS-enabled clusters:

fdbbackup start -d file:///backup/fdb \
  --tls-certificate-file /path/to/cert.pem \
  --tls-key-file /path/to/key.pem \
  --tls-ca-file /path/to/ca.pem

Or via environment variables:

export FDB_TLS_CERTIFICATE_FILE=/path/to/cert.pem
export FDB_TLS_KEY_FILE=/path/to/key.pem
fdbbackup start -d file:///backup/fdb

Backup Retention and Expiration¶

Expire Old Backups¶

Remove backup data older than specified version or timestamp:

# Expire by version
fdbbackup expire -d file:///backup/fdb --expire-before-version 12345678

# Expire by timestamp
fdbbackup expire -d file:///backup/fdb --expire-before-timestamp "2025-01-01 00:00:00"

# Force minimum restorability before expiring
fdbbackup expire -d file:///backup/fdb --restorable-after-version 12000000

Delete Backup Entirely¶

fdbbackup delete -d file:///backup/fdb

Disk Snapshot Backup¶

Disk snapshot backup is an alternative backup mechanism that captures a point-in-time, block-level image of every FoundationDB process's data directory by triggering block-level volume snapshots (AWS EBS, LVM, CSI VolumeSnapshot, etc.) coordinated across the cluster. Unlike fdbbackup, it does not stream a continuous mutation log to external storage — instead, it produces a single consistent disk image per role at a single FDB version. Operators choose this approach when restore throughput from fdbbackup is the bottleneck (a snapshot restore is bounded by the speed at which volumes can be attached or copied, not by log replay), and when continuous point-in-time recovery is not required. The mechanism has been part of FoundationDB since the 6.x line and is used in production by some large operators. The snapshot mechanism must come from the block layer underneath the filesystem (EBS, LVM, CSI VolumeSnapshot); see Prerequisites below.

When to Use¶

How It Works¶

When fdbcli> snapshot <binary> [args...] is invoked, the cluster controller orchestrates a synchronized snapshot across all stateful processes. Each fdbserver process then forks the operator-supplied snap_create binary, which is responsible for invoking the underlying volume-snapshot mechanism on that host's data directory.

graph TD
    Op[Operator]
    CLI["fdbcli> snapshot /bin/snap_create.sh"]
    CC[Cluster Controller<br/>Snapshot Orchestrator]

    subgraph "Storage Process"
        SS[fdbserver<br/>storage role]
        SS_Snap[snap_create<br/>--role=storage]
        SS_Disk[(Storage data dir)]
    end

    subgraph "TLog Process"
        TL[fdbserver<br/>tlog role]
        TL_Snap[snap_create<br/>--role=tlog]
        TL_Disk[(TLog data dir)]
    end

    subgraph "Coordinator Process"
        CO[fdbserver<br/>coordinator]
        CO_Snap[snap_create<br/>--role=coord]
        CO_Disk[(Coordinator data dir)]
    end

    Op --> CLI
    CLI --> CC
    CC --> SS
    CC --> TL
    CC --> CO
    SS --> SS_Snap --> SS_Disk
    TL --> TL_Snap --> TL_Disk
    CO --> CO_Snap --> CO_Disk

    style CC fill:#ff9800,color:#000
    style SS_Disk fill:#4caf50,color:#fff
    style TL_Disk fill:#4caf50,color:#fff
    style CO_Disk fill:#4caf50,color:#fff

The orchestrator quiesces the relevant subsystems and ensures that all per-role snapshots taken across the cluster reflect the same FDB version. The result is a set of disk images — one per role, per process — that together form a consistent backup of the cluster.

Setting Up the snap_create Binary¶

snap_create is an operator-supplied executable invoked by fdbserver once per role on each host when a snapshot is requested. It is responsible for actually triggering the underlying volume-snapshot operation (for example, an aws ec2 create-snapshot call, an lvcreate --snapshot, or a CSI VolumeSnapshot).

The simplest illustrative implementation copies the data directory to a separate location, similar to the upstream example:

#!/bin/bash
# /bin/snap_create.sh — illustrative example only.
# Real deployments should call EBS / LVM / CSI VolumeSnapshot APIs.
set -euo pipefail

UID=""
VERSION=""
PATH_ARG=""
ROLE=""

while [[ $# -gt 0 ]]; do
  case "$1" in
    --uid)     UID="$2";      shift 2 ;;
    --version) VERSION="$2";  shift 2 ;;
    --path)    PATH_ARG="$2"; shift 2 ;;
    --role)    ROLE="$2";     shift 2 ;;
    *)         shift ;;  # ignore extra operator-supplied args
  esac
done

DEST="/var/snapshots/${UID}/${ROLE}-${VERSION}"
mkdir -p "$DEST"
cp -a "$PATH_ARG"/. "$DEST"/

fdbserver injects four named arguments when invoking the binary, followed by any extra arguments the operator passed to fdbcli> snapshot:

Configuring the Cluster¶

To allow fdbserver to fork the snapshot binary, set whitelist_binpath in the [fdbserver] section of foundationdb.conf on every host:

[fdbserver]
command = /usr/sbin/fdbserver
whitelist_binpath = /bin/snap_create.sh

The path may be a colon-separated list if multiple binaries are permitted. Restart fdbserver (or trigger a rolling restart) for the change to take effect.

The SNAP_CREATE_MAX_TIMEOUT knob (default 5 minutes) bounds how long the orchestrator will wait for snap_create to complete on each process. Tune it via --knob-snap-create-max-timeout=<seconds> if your underlying snapshot mechanism is slow.

Recommended Metadata to Capture¶

Disk snapshot images by themselves are not sufficient to reconstitute a cluster — the operator must also record enough metadata to map images back to roles, processes, and FDB versions at restore time. Capture at least the following per snapshot:

Taking a Backup¶

From an fdbcli session attached to the cluster:

fdbcli> snapshot /bin/snap_create.sh --extra-arg value
Snapshot command succeeded with UID a1b2c3d4e5f60718293a4b5c6d7e8f90

Pass the absolute path to your snap_create binary (which must match whitelist_binpath) followed by any extra arguments your binary accepts. The UID printed in the response is the same UID injected as --uid to every invocation of snap_create across the cluster.

Restore Steps¶

A disk-snapshot restore reconstitutes a new FDB cluster from a previously captured set of per-role snapshot images. Roughly:

1. Locate snapshot images by UID. Identify all images that share the same snapshot UID — one per role per process across the original cluster.
2. Group by old IP / locality. Use the captured metadata to group images by the original process's IP, port, and locality. Each group corresponds to one process worth of state.
3. Provision new cluster nodes. Decide on the IP layout for the new cluster, build a mapping from old IP → new IP, and attach (or copy) each image to the corresponding new host into the same data directory layout per role.
4. Recompute fdb.cluster. Rewrite the cluster file with the new coordinator IPs (taken from the IP mapping). Distribute the new cluster file to every node.
5. Start fdbserver on the new nodes. With the data directories in place and the new cluster file pointing at the new coordinators, the cluster will recover automatically to the snapshot's FDB version.

Error Codes¶

snap_create failures and orchestration errors surface through standard FoundationDB error codes. The most relevant are:

Programmatic API¶

Disk snapshot backup can also be triggered from application code via the C API (apple/foundationdb#4241):

const char *uid = "a1b2c3d4e5f60718293a4b5c6d7e8f90";
FDBFuture *f = fdb_database_create_snapshot(db, uid, strlen(uid));

The caller supplies the UID (typically a freshly generated 32-character hex string), and fdbserver invokes the configured snap_create binary on each role exactly as it would for fdbcli> snapshot. The future resolves once the cluster-wide snapshot has either succeeded or failed.

Cleanup¶

FoundationDB does not garbage-collect old or failed disk-snapshot images. If a snapshot operation fails partway through, or if a successful snapshot ages out of the operator's retention policy, the on-disk (or on-EBS, or on-S3) artifacts must be expired by external tooling — for example, a cron job that lists snapshots older than N days and deletes them, or lifecycle policies on the underlying storage. Plan a cleanup strategy before enabling disk snapshot backup in production.

Monitoring Backups¶

Status in fdbcli¶

fdb> status
...
Backup and DR:
  Running backups        - 1
  Running DRs            - 0

Backup Metrics¶

Monitor these via machine-readable status:

Alerting Script¶

#!/usr/bin/env python3
import fdb
import json
import sys

fdb.api_version(730)
db = fdb.open()

@fdb.transactional
def get_status(tr):
    return json.loads(tr[b'\xff\xff/status/json'])

status = get_status(db)
backup = status.get('cluster', {}).get('layers', {}).get('backup', {})

if not backup.get('instances_running', 0):
    print("WARNING: No backup instances running")
    sys.exit(1)

for tag, info in backup.get('tags', {}).items():
    lag = info.get('last_restorable_seconds_behind_primary', float('inf'))
    if lag > 300:
        print(f"WARNING: Backup {tag} is {lag:.0f} seconds behind")
        sys.exit(1)

print("OK: Backups healthy")
sys.exit(0)

Best Practices¶

Backup Configuration¶

1. Run multiple backup agents - At least 2 for redundancy
2. Use blob storage for production - More durable than filesystem
3. Set appropriate snapshot intervals - Daily snapshots balance storage vs restore time
4. Tag your backups - Enables multiple independent backup streams

Testing and Validation¶

1. Test restores regularly - Monthly to a test cluster
2. Validate backup integrity - Use fdbbackup describe
3. Monitor backup lag - Alert if > 5 minutes behind
4. Document recovery procedures - RTO/RPO requirements

Retention Policy¶

1. Define retention requirements - Regulatory, operational
2. Automate expiration - Use fdbbackup expire in cron
3. Keep multiple generations - At least 7 days for point-in-time
4. Verify before expiring - Ensure newer backups are restorable

Security¶

1. Encrypt backup storage - S3 server-side encryption or client-side
2. Secure credentials - Use IAM roles, not long-term keys
3. Restrict backup access - Separate credentials for backup vs restore
4. Audit backup operations - Log all backup/restore commands

Backup Encryption NEW IN 7.3¶

Starting in FoundationDB 7.3, backups support native file-level encryption. When enabled, backup data is encrypted before being written to the backup destination, providing end-to-end encryption regardless of the storage backend.

Enabling Encryption¶

To enable encryption on an existing backup, use the fdbbackup modify command:

fdbbackup modify -t default --encryption

New backups can also be started with encryption enabled:

fdbbackup start -d "blobstore://s3.amazonaws.com/my-bucket/fdb-backup" --encryption

Checking Encryption Status¶

Backup status — Encryption key information is included in the backup status JSON output:

fdbbackup status -t default --json

The JSON output includes encryption key details when encryption is active.

Backup describe — The fdbbackup describe output includes a FileLevelEncryption field indicating whether the backup is encrypted:

fdbbackup describe -d "blobstore://s3.amazonaws.com/my-bucket/fdb-backup" --json

Look for the FileLevelEncryption field in the JSON response to confirm encryption is enabled.

Troubleshooting¶

Common Issues¶

Debug Commands¶

# Describe backup contents
fdbbackup describe -d file:///backup/fdb

# List backup tags
fdbbackup tags

# Cleanup incomplete operations
fdbbackup cleanup -d file:///backup/fdb

# Check backup agent logs
tail -f /var/log/foundationdb/backup_agent*.xml

Next Steps¶

• Learn Troubleshooting for common issues
• Review Monitoring for backup metrics
• See Configuration for backup agent settings