Backup & Recovery¶

FoundationDB provides robust backup, restore, and disaster recovery capabilities. This guide covers continuous backup to blob storage, point-in-time recovery, and cross-datacenter replication.

Overview¶

FoundationDB's backup system offers:

Backup Architecture¶

Backup agents run as separate processes that read mutation logs from the database and write them to a backup destination. Multiple agents can run for redundancy and performance.

graph LR
    subgraph "FoundationDB Cluster"
        TLog[Transaction Logs<br/>Mutations]
        SS[Storage Servers]
    end

    subgraph "Backup System"
        BA1[Backup Agent 1]
        BA2[Backup Agent 2]
    end

    subgraph "Backup Storage"
        S3[(S3 / Blob Store)]
        FS[(Filesystem)]
    end

    TLog --> BA1
    TLog --> BA2
    BA1 --> S3
    BA2 --> S3
    BA1 -.-> FS
    BA2 -.-> FS

    style S3 fill:#ff9800,color:#000
    style FS fill:#4caf50,color:#fff

Backup Destinations¶

URL Formats¶

Filesystem Backup¶

fdbbackup start -d file:///mnt/backup/fdb

Requirements: - Path must be accessible to all backup agents - Use network filesystem (NFS, EFS) for multi-machine setups - Ensure sufficient disk space for backup history

Blob Store Backup¶

fdbbackup start -d "blobstore://s3.amazonaws.com:443/my-bucket/fdb-backup?bucket=my-bucket&region=us-east-1"

Blob Store URL Parameters¶

Blob Store Credentials¶

Create a JSON credentials file:

{
  "accounts": {
    "s3.amazonaws.com": {
      "api_key": "AKIAIOSFODNN7EXAMPLE",
      "secret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    }
  }
}

Set via environment variable:

export FDB_BLOB_CREDENTIALS=/path/to/credentials.json

Or use AWS SDK authentication:

fdbbackup start -d "blobstore://s3.amazonaws.com/bucket?sdk_auth=true"

Starting a Backup¶

Basic Backup¶

fdbbackup start -C /etc/foundationdb/fdb.cluster -d file:///backup/fdb

Backup with Snapshot¶

Include a full snapshot for faster restores:

fdbbackup start -d file:///backup/fdb -z

The -z flag (or --no_stop_when_done) continues backing up after the initial snapshot.

Backup with Tags¶

Run multiple independent backups:

# Primary backup to S3
fdbbackup start -t primary -d "blobstore://s3.amazonaws.com/prod-backup"

# Secondary backup to local storage
fdbbackup start -t local -d file:///mnt/backup/fdb

Backup with Key Range¶

Back up only specific key ranges:

fdbbackup start -d file:///backup/fdb -k '\x00' '\xff'

Managing Backups¶

Check Backup Status¶

fdbbackup status -C /etc/foundationdb/fdb.cluster

Or for a specific tag:

fdbbackup status -t mytag

Example output:

Using cluster file `/etc/foundationdb/fdb.cluster'.

The backup on tag `default' is restorable but continuing to
 blobstore://s3.amazonaws.com:443/my-bucket (last completed log version is 12345678).

Backup is restorable to version 12345678 (approximately 5 seconds old).

List Backups¶

fdbbackup list -b file:///backup/fdb

Pause/Resume Backup¶

# Pause (backup continues but range log writing pauses)
fdbbackup modify -t default --active-snapshot-interval 0

# Resume
fdbbackup modify -t default --active-snapshot-interval 86400

Stop Backup¶

# Stop gracefully (keeps backup restorable)
fdbbackup discontinue -t default

# Abort immediately
fdbbackup abort -t default

Wait for Backup¶

Wait for backup to complete or reach a restorable state:

fdbbackup wait -t default

fdbbackup Command Reference¶

Common Options¶

Restoring from Backup¶

Basic Restore¶

fdbrestore start -r file:///backup/fdb -C /etc/foundationdb/fdb.cluster

Restore to Specific Version¶

fdbrestore start -r file:///backup/fdb --version 12345678

Restore to Specific Timestamp¶

fdbrestore start -r file:///backup/fdb --timestamp "2025-02-03 10:30:00"

Restore with Key Transformation¶

Restore to different key prefixes:

fdbrestore start -r file:///backup/fdb \
  --remove_prefix "prod/" \
  --add_prefix "staging/"

Check Restore Status¶

fdbrestore status -C /etc/foundationdb/fdb.cluster

Wait for Restore¶

fdbrestore wait

Abort Restore¶

fdbrestore abort

fdbrestore Command Reference¶

Restore Options¶

Running Backup Agents¶

Backup agents perform the actual backup work. Run them on your cluster machines.

Starting Backup Agent¶

backup_agent -C /etc/foundationdb/fdb.cluster

Backup Agent in foundationdb.conf¶

[backup_agent]
command = /usr/lib/foundationdb/backup_agent/backup_agent
logdir = /var/log/foundationdb

Multiple Agents¶

Run multiple backup agents for redundancy and performance. They coordinate automatically.

Agent with Blob Credentials¶

backup_agent -C /etc/foundationdb/fdb.cluster \
  --blob-credentials /path/to/credentials.json

Disaster Recovery (DR)¶

DR provides real-time replication to a standby cluster for immediate failover.

DR vs Backup¶

DR Architecture¶

graph LR
    subgraph "Primary Datacenter"
        P_TLog[Transaction Logs]
        P_SS[Storage Servers]
        P_Client[Clients<br/>Read/Write]
    end

    subgraph "DR Datacenter"
        DR_TLog[Transaction Logs]
        DR_SS[Storage Servers]
        DR_Client[Clients<br/>Read-Only]
    end

    DR_Agent[DR Agent]

    P_Client --> P_TLog
    P_TLog --> P_SS
    P_TLog --> DR_Agent
    DR_Agent --> DR_TLog
    DR_TLog --> DR_SS
    DR_Client -.-> DR_SS

    style P_TLog fill:#4caf50,color:#fff
    style DR_TLog fill:#ff9800,color:#000

Setting Up DR¶

1. Start DR on primary cluster:

   Bash

   fdbdr start -s /path/to/primary.cluster -d /path/to/dr.cluster
   

2. Run DR agents:

   Bash

   dr_agent -s /path/to/primary.cluster -d /path/to/dr.cluster
   

3. Check DR status:

   Bash

   fdbdr status -s /path/to/primary.cluster -d /path/to/dr.cluster
   

DR Switchover¶

Perform a planned switchover:

fdbdr switch -s /path/to/primary.cluster -d /path/to/dr.cluster

After switchover: - DR cluster becomes primary - Original primary becomes standby - Applications reconnect to new primary

DR Failover¶

For unplanned failover when primary is unavailable:

# On DR cluster
fdbcli> writemode on
fdbcli> clearrange "" "\xff"  # Only if needed to break replication

fdbdr Command Reference¶

TLS Support¶

All backup commands support TLS when connecting to TLS-enabled clusters:

fdbbackup start -d file:///backup/fdb \
  --tls-certificate-file /path/to/cert.pem \
  --tls-key-file /path/to/key.pem \
  --tls-ca-file /path/to/ca.pem

Or via environment variables:

export FDB_TLS_CERTIFICATE_FILE=/path/to/cert.pem
export FDB_TLS_KEY_FILE=/path/to/key.pem
fdbbackup start -d file:///backup/fdb

Backup Retention and Expiration¶

Expire Old Backups¶

Remove backup data older than specified version or timestamp:

# Expire by version
fdbbackup expire -d file:///backup/fdb --expire-before-version 12345678

# Expire by timestamp
fdbbackup expire -d file:///backup/fdb --expire-before-timestamp "2025-01-01 00:00:00"

# Force minimum restorability before expiring
fdbbackup expire -d file:///backup/fdb --restorable-after-version 12000000

Delete Backup Entirely¶

fdbbackup delete -d file:///backup/fdb

Monitoring Backups¶

Status in fdbcli¶

fdb> status
...
Backup and DR:
  Running backups        - 1
  Running DRs            - 0

Backup Metrics¶

Monitor these via machine-readable status:

Alerting Script¶

#!/usr/bin/env python3
import fdb
import json
import sys

fdb.api_version(730)
db = fdb.open()

@fdb.transactional
def get_status(tr):
    return json.loads(tr[b'\xff\xff/status/json'])

status = get_status(db)
backup = status.get('cluster', {}).get('layers', {}).get('backup', {})

if not backup.get('instances_running', 0):
    print("WARNING: No backup instances running")
    sys.exit(1)

for tag, info in backup.get('tags', {}).items():
    lag = info.get('last_restorable_seconds_behind_primary', float('inf'))
    if lag > 300:
        print(f"WARNING: Backup {tag} is {lag:.0f} seconds behind")
        sys.exit(1)

print("OK: Backups healthy")
sys.exit(0)

Best Practices¶

Backup Configuration¶

1. Run multiple backup agents - At least 2 for redundancy
2. Use blob storage for production - More durable than filesystem
3. Set appropriate snapshot intervals - Daily snapshots balance storage vs restore time
4. Tag your backups - Enables multiple independent backup streams

Testing and Validation¶

1. Test restores regularly - Monthly to a test cluster
2. Validate backup integrity - Use fdbbackup describe
3. Monitor backup lag - Alert if > 5 minutes behind
4. Document recovery procedures - RTO/RPO requirements

Retention Policy¶

1. Define retention requirements - Regulatory, operational
2. Automate expiration - Use fdbbackup expire in cron
3. Keep multiple generations - At least 7 days for point-in-time
4. Verify before expiring - Ensure newer backups are restorable

Security¶

1. Encrypt backup storage - S3 server-side encryption or client-side
2. Secure credentials - Use IAM roles, not long-term keys
3. Restrict backup access - Separate credentials for backup vs restore
4. Audit backup operations - Log all backup/restore commands

Troubleshooting¶

Common Issues¶

Debug Commands¶

# Describe backup contents
fdbbackup describe -d file:///backup/fdb

# List backup tags
fdbbackup tags

# Cleanup incomplete operations
fdbbackup cleanup -d file:///backup/fdb

# Check backup agent logs
tail -f /var/log/foundationdb/backup_agent*.xml

Next Steps¶

• Learn Troubleshooting for common issues
• Review Monitoring for backup metrics
• See Configuration for backup agent settings