Configuration¶

This guide covers configuring FoundationDB clusters for production use. You'll learn about cluster files, redundancy modes, storage engines, and process configuration.

System Requirements¶

Before deploying FoundationDB, ensure your systems meet these requirements:

Cluster File¶

The cluster file (fdb.cluster) defines how clients and servers connect to the cluster.

File Format¶

description:ID@IP:PORT,IP:PORT,...

Example:

my_production_db:abc12345@10.0.4.1:4500,10.0.4.2:4500,10.0.4.3:4500

Default Locations¶

Specifying the Cluster File¶

You can specify the cluster file in several ways (in order of precedence):

1. Command-line: fdbcli -C /path/to/fdb.cluster
2. Environment variable: FDB_CLUSTER_FILE
3. Current working directory: ./fdb.cluster
4. Default system location

IPv6 Support¶

FoundationDB supports IPv6 addresses. Use the format [IP]:PORT:

description:ID@127.0.0.1:4500,[::1]:4500

Coordination Servers¶

Coordinators are processes that help maintain cluster availability during network partitions. They store minimal shared state and don't impact transaction performance.

Choosing Coordinators¶

Follow these guidelines:

• Use an odd number of coordinators (3 or 5 recommended)
• Match coordinator count to your redundancy mode
• Place coordinators in different failure domains (racks, circuits, datacenters)
• Coordinator latency doesn't affect normal operations

Changing Coordinators¶

Use fdbcli to change coordinators without downtime:

fdb> coordinators 10.0.4.1:4500 10.0.4.2:4500 10.0.4.3:4500
Coordinators changed

Or let FoundationDB choose automatically:

fdb> coordinators auto
Coordinators changed

Verify the change:

fdb> status details

Changing Cluster Description¶

fdb> coordinators description=production_cluster
Coordination state changed

Redundancy Modes¶

Redundancy modes control data replication and fault tolerance.

graph TB
    subgraph "Triple Redundancy - Production Recommended"
        subgraph "Zone A"
            A1[Storage 1<br/>Replica 1]
        end
        subgraph "Zone B"
            B1[Storage 2<br/>Replica 2]
        end
        subgraph "Zone C"
            C1[Storage 3<br/>Replica 3]
        end
    end

    Data[Data Shard X] --> A1
    Data --> B1
    Data --> C1

    style A1 fill:#4caf50,color:#fff
    style B1 fill:#4caf50,color:#fff
    style C1 fill:#4caf50,color:#fff

Single Datacenter Modes¶

Datacenter-Aware Mode¶

graph TB
    subgraph "DC 1 - Primary"
        DC1_SS1[Storage]
        DC1_SS2[Storage]
    end
    subgraph "DC 2 - Satellite"
        DC2_SS1[Storage]
        DC2_SS2[Storage]
    end
    subgraph "DC 3 - Satellite"
        DC3_SS1[Storage]
        DC3_SS2[Storage]
    end

    Client --> DC1_SS1
    Client --> DC1_SS2

    DC1_SS1 -.->|Replicate| DC2_SS1
    DC1_SS1 -.->|Replicate| DC3_SS1
    DC1_SS2 -.->|Replicate| DC2_SS2
    DC1_SS2 -.->|Replicate| DC3_SS2

    style DC1_SS1 fill:#2196f3,color:#fff
    style DC1_SS2 fill:#2196f3,color:#fff
    style DC2_SS1 fill:#ff9800,color:#000
    style DC2_SS2 fill:#ff9800,color:#000
    style DC3_SS1 fill:#9c27b0,color:#fff
    style DC3_SS2 fill:#9c27b0,color:#fff

Configuring Regions¶

Regions enable automatic failover between two datacenters without adding WAN latency for commits, while maintaining all FoundationDB consistency properties.

Regions vs Datacenters¶

Understanding the difference between regions and datacenters is critical for multi-datacenter deployments:

graph TB
    subgraph "Region 1 (West Coast)"
        subgraph "Primary DC - WC1"
            R1_SS1[Storage Server]
            R1_SS2[Storage Server]
            R1_TL[Transaction Logs]
        end
        subgraph "Satellite DC - WC2"
            R1_SAT[Transaction Logs Only]
        end
    end

    subgraph "Region 2 (East Coast)"
        subgraph "Primary DC - EC1"
            R2_SS1[Storage Server]
            R2_SS2[Storage Server]
            R2_TL[Transaction Logs]
        end
        subgraph "Satellite DC - EC2"
            R2_SAT[Transaction Logs Only]
        end
    end

    R1_TL -.->|Async Replication| R2_TL

    style R1_SS1 fill:#2196f3,color:#fff
    style R1_SS2 fill:#2196f3,color:#fff
    style R1_TL fill:#4caf50,color:#fff
    style R1_SAT fill:#ff9800,color:#000
    style R2_SS1 fill:#9c27b0,color:#fff
    style R2_SS2 fill:#9c27b0,color:#fff
    style R2_TL fill:#4caf50,color:#fff
    style R2_SAT fill:#ff9800,color:#000

How Regions Work¶

Region configuration combines two features:

1. Asynchronous replication between regions — The remote region slightly lags behind the primary, avoiding WAN latency on commits
2. Synchronous satellite logs — Mutations are made durable in nearby satellite datacenters before commits succeed

When the primary datacenter fails, the satellite logs ensure no committed transactions are lost. The remote region catches up using these logs, then starts accepting new commits.

Region Configuration JSON¶

Regions are configured via a JSON document using fileconfigure in fdbcli:

fdb> fileconfigure regions.json
Configuration changed.

Example: Two-region configuration with satellites

{
  "regions": [{
    "datacenters": [{
      "id": "WC1",
      "priority": 1
    }, {
      "id": "WC2",
      "priority": 0,
      "satellite": 1,
      "satellite_logs": 2
    }],
    "satellite_redundancy_mode": "one_satellite_double"
  }, {
    "datacenters": [{
      "id": "EC1",
      "priority": 0
    }, {
      "id": "EC2",
      "priority": -1,
      "satellite": 1,
      "satellite_logs": 2
    }],
    "satellite_redundancy_mode": "one_satellite_double"
  }]
}

Configuration Parameters:

Satellite Redundancy Modes¶

Specifying Datacenter Locality¶

All processes must specify their datacenter using --locality-dcid or --datacenter-id:

## foundationdb.conf
[fdbserver.4500]
locality_dcid = WC1

[fdbserver.4501]
locality_dcid = WC1

Clients should also specify their datacenter using the datacenter-id database option to route requests to the nearest region.

Usable Regions¶

The usable_regions option controls how many regions store data replicas:

fdb> configure usable_regions=2
Configuration changed.

Incompatibility with three_datacenter Mode¶

If you need synchronous replication to three datacenters with low-latency reads from all locations, use three_datacenter mode. If you need automatic failover with lower commit latency, use region configuration.

Handling Datacenter Failures¶

When a primary datacenter fails:

1. The cluster recovers to the other region
2. Mutations accumulate on transaction logs while waiting for recovery
3. The cluster continues accepting commits (at reduced throughput—roughly ⅓ of normal)

To permanently drop a failed datacenter:

# Set failed DC priority to negative
fdb> fileconfigure regions_without_failed_dc.json

# Reduce to single region
fdb> configure usable_regions=1

Choosing Coordinators for Regions¶

For multi-region setups, coordinators should survive datacenter failures:

• Option 1: 5 coordinators in 5 different datacenters
• Option 2: 9 coordinators spread across 3 datacenters (3 per DC)

Configuring Redundancy¶

fdb> configure triple
Configuration changed.

Changing Redundancy Mode¶

You can change redundancy modes at any time. FoundationDB will replicate data accordingly while remaining available.

Storage Engines¶

FoundationDB supports multiple storage engines optimized for different workloads.

Available Engines¶

Configuring Storage Engine¶

fdb> configure ssd
Configuration changed.

Storage Engine Details¶

SSD Engine (ssd)

• Optimized for solid-state drives
• B-tree data structure
• Deferred space reclamation after deletes
• Poor performance on spinning disks

Redwood Engine (ssd-redwood-1)

• Improved B-tree implementation
• Higher throughput than standard SSD engine
• Lower write amplification

RocksDB Engine (ssd-rocksdb-v1)

• LSM-tree based (Log-Structured Merge Tree)
• Built-in compression
• Extensive tuning options
• Good for write-heavy workloads

Memory Engine (memory)

• All data resident in memory
• Writes logged to disk for durability
• Default 1GB limit per process (configurable via storage_memory)
• Good for spinning disks with small datasets

Process Configuration¶

FoundationDB processes are configured via foundationdb.conf.

Configuration File Location¶

Configuration File Structure¶

## foundationdb.conf

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
cluster-file = /etc/foundationdb/fdb.cluster
restart-delay = 60

[fdbserver]
command = /usr/sbin/fdbserver
public-address = auto:$ID
listen-address = public
datadir = /var/lib/foundationdb/data/$ID
logdir = /var/log/foundationdb
# logsize = 10MiB
# maxlogssize = 100MiB
# memory = 8GiB
# storage-memory = 1GiB

## Individual fdbserver processes
[fdbserver.4500]

[fdbserver.4501]

Key Configuration Options¶

Process Classes¶

Assign processes to roles for optimal resource utilization:

Configure in foundationdb.conf:

[fdbserver.4500]
class = storage

[fdbserver.4501]
class = transaction

Or via fdbcli:

fdb> setclass 10.0.4.1:4500 storage

Running Multiple Processes Per Machine¶

For multi-core machines, run one fdbserver per CPU core:

## 8-core machine example
[fdbserver.4500]
class = storage

[fdbserver.4501]
class = storage

[fdbserver.4502]
class = transaction

# ... etc

Locality Configuration¶

Configure locality for fault-tolerance aware data placement:

[fdbserver.4500]
locality_dcid = dc1
locality_zoneid = rack1
locality_machineid = server1

TLS Configuration¶

Enable encrypted communication between cluster components.

Enable TLS in Cluster File¶

Add :tls suffix to coordinator addresses:

my_cluster:abc12345@10.0.4.1:4500:tls,10.0.4.2:4500:tls

TLS Configuration Options¶

Default Certificate Locations¶

Quick Reference¶

Common fdbcli Configuration Commands¶

# Set redundancy mode
fdb> configure triple

# Set storage engine
fdb> configure ssd

# Combined configuration
fdb> configure triple ssd

# Set specific counts
fdb> configure grv_proxies=3
fdb> configure commit_proxies=5
fdb> configure logs=8

# Change coordinators
fdb> coordinators auto
fdb> coordinators 10.0.4.1:4500 10.0.4.2:4500 10.0.4.3:4500

# Set process class
fdb> setclass 10.0.4.1:4500 storage

Configuration Best Practices¶

1. Start with triple redundancy for production clusters of 5+ machines
2. Use ssd or ssd-redwood-1 storage engine for most workloads
3. Place coordinators in different failure domains
4. Assign process classes to optimize resource usage
5. Enable TLS for production deployments
6. Configure locality for multi-datacenter setups

Next Steps¶

• Set up Monitoring to track cluster health
• Configure Backup & Recovery for data protection
• Review Troubleshooting for common issues