fieldwitness/docs/federation.md

Federation Guide

Audience: System administrators and technical leads setting up cross-organization attestation sync between FieldWitness instances.

Prerequisites: A running FieldWitness instance (Tier 2 org server or Tier 3 relay), familiarity with the CLI, and trusted public keys from partner organizations.

---

Overview

FieldWitness federation synchronizes attestation records between organizations using a gossip protocol. Nodes periodically exchange Merkle roots, detect divergence, and fetch missing records. The system is eventually consistent with no central coordinator, no leader election, and no consensus protocol -- just append-only logs that converge.

Federation operates at two levels:

1. Offline bundles -- JSON export/import via sneakernet (USB drive). Works on all tiers including fully airgapped Tier 1 field devices.
2. Live gossip -- HTTP-based periodic sync between Tier 2 org servers and Tier 3 federation relays. Requires the federation extra (pip install fieldwitness[federation]).

Warning: Federation shares attestation records (image hashes, Ed25519 signatures, timestamps, and signer public keys). It never shares encryption keys, plaintext messages, original images, or steganographic payloads.

---

Architecture

Tier 1: Field Device          Tier 2: Org Server A         Tier 3: Relay (Iceland)
(Bootable USB)                (Docker / mini PC)           (VPS, zero key knowledge)
                                     |                            |
     USB sneakernet ------>   Port 5000 (Web UI)                  |
                              Port 8000 (Federation API)  <-----> Port 8000
                                     |                            |
                              Tier 2: Org Server B  <------------>
                              (Docker / mini PC)

Federation traffic flows:

• Tier 1 to Tier 2: USB sneakernet (offline bundles only)
• Tier 2 to Tier 3: gossip API over HTTPS (port 8000)
• Tier 2 to Tier 2: through a Tier 3 relay, or directly via sneakernet
• Tier 3 to Tier 3: gossip between relays in different jurisdictions

---

Gossip Protocol

How sync works

1. Node A sends its Merkle root and log size to Node B via GET /federation/status
2. If roots differ and B has more records, A requests a consistency proof via GET /federation/consistency-proof?old_size=N
3. If the proof verifies (B's log is a valid extension of A's), A fetches the missing records via GET /federation/records?start=N&count=50
4. A appends the new records to its local log
5. B performs the same process in reverse (bidirectional sync)

Records are capped at 100 per request to protect memory on resource-constrained devices.

Peer health tracking

Each peer tracks:

• last_seen -- timestamp of last successful contact
• last_root -- most recent Merkle root received from the peer
• last_size -- most recent log size
• healthy -- marked false after 3 consecutive failures
• consecutive_failures -- reset to 0 on success

Unhealthy peers are skipped during gossip rounds but remain registered. They are retried on the next full gossip round. Peer state persists in SQLite at ~/.fwmetadata/attestations/federation/peers.db.

Gossip interval

The default gossip interval is 60 seconds, configurable via the FIELDWITNESS_GOSSIP_INTERVAL environment variable. In Docker Compose, set it in the environment section:

environment:
  - FIELDWITNESS_GOSSIP_INTERVAL=60

Lower intervals mean faster convergence but more network traffic.

---

Setting Up Federation

Step 1: Exchange trust keys

Before two organizations can federate, they must trust each other's Ed25519 identity keys. Always verify fingerprints out-of-band (in person or over a known-secure voice channel).

On Organization A:

$ cp ~/.fwmetadata/identity/public.pem /media/usb/org-a-pubkey.pem

On Organization B:

$ fieldwitness keys trust --import /media/usb/org-a-pubkey.pem

Repeat in both directions so each organization trusts the other.

Warning: Do not skip fingerprint verification. If an adversary substitutes their own public key, they can forge attestation records that your instance will accept as trusted.

Step 2: Register peers (live gossip)

Through the web UI at /federation, or via the peer store directly:

# On Org A's server, register Org B's federation endpoint
$ fieldwitness federation peer add \
    --url https://orgb.example.org:8000 \
    --fingerprint a1b2c3d4e5f6...

Or through the web UI:

1. Navigate to /federation
2. Enter the peer's federation API URL and Ed25519 fingerprint
3. Click "Add Peer"

Step 3: Start the gossip loop

The gossip loop starts automatically when the server starts. On Docker deployments, the federation API runs on port 8000. Ensure this port is accessible between peers (firewall, security groups, etc.).

For manual one-time sync:

$ fieldwitness federation sync --peer https://orgb.example.org:8000

Step 4: Monitor sync status

The web UI at /federation shows:

• Local node status (Merkle root, log size, record count)
• Registered peers with health indicators
• Recent sync history (records received, errors)

---

Offline Federation (Sneakernet)

For Tier 1 field devices and airgapped environments, use offline bundles.

Exporting a bundle

$ fieldwitness chain export --output /media/usb/bundle.zip

To export only records from a specific investigation:

$ fieldwitness chain export --investigation "case-2026-001" --output /media/usb/bundle.zip

To export a specific index range:

$ fieldwitness chain export --start 100 --end 200 --output /media/usb/partial.zip

Importing a bundle

On the receiving instance:

$ fieldwitness chain import /media/usb/bundle.zip

During import:

• Records signed by untrusted fingerprints are rejected
• Duplicate records (matching SHA-256) are skipped
• Imported records are tagged with federated_from metadata
• A delivery acknowledgment record (fieldwitness/delivery-ack-v1) is automatically appended to the local chain

Delivery acknowledgments

When a bundle is imported, FieldWitness signs a fieldwitness/delivery-ack-v1 chain record that contains:

• The SHA-256 of the imported bundle file
• The sender's fingerprint
• The count of records received

This acknowledgment can be exported back to the sending organization as proof that the bundle was delivered and ingested. It creates a two-way federation handshake.

# On receiving org: export the acknowledgment back
$ fieldwitness chain export --start <ack_index> --end <ack_index> \
    --output /media/usb/delivery-ack.zip

---

Federation API Endpoints

The federation API is served by FastAPI/uvicorn on port 8000.

Method	Endpoint	Description
GET	/federation/status	Current Merkle root and log size
GET	/federation/records?start=N&count=M	Fetch attestation records (max 100)
GET	/federation/consistency-proof?old_size=N	Merkle consistency proof
POST	/federation/records	Push records to this node
GET	/health	Health check

Trust filtering on push

When records are pushed via POST /federation/records, the receiving node checks each record's attestor_fingerprint against its trust store. Records from unknown attestors are rejected. If no trust store is configured (empty trusted keys), all records are accepted (trust-on-first-use).

---

Federation Relay (Tier 3)

The federation relay is a minimal Docker container that runs only the federation API.

What the relay stores

• Attestation records: image SHA-256 hashes, perceptual hashes, Ed25519 signatures
• Chain linkage data: prev_hash, chain_index, claimed_ts
• Signer public keys

What the relay never sees

• AES-256-GCM channel keys or Ed25519 private keys
• Original images or media files
• Steganographic payloads or plaintext messages
• User credentials or session data
• Web UI content

Deploying a relay

$ cd deploy/docker
$ docker compose up relay -d

The relay listens on port 8001 (mapped to internal 8000). See docs/deployment.md Section 3 for full deployment details.

Jurisdiction considerations

Deploy relays in jurisdictions with strong press freedom protections:

• Iceland -- strong source protection laws, no mandatory data retention for this type of data
• Switzerland -- strict privacy laws, resistance to foreign legal requests
• Netherlands -- strong press freedom, EU GDPR protections

Consult with a press freedom lawyer for your specific situation.

---

Troubleshooting

Peer marked unhealthy

After 3 consecutive sync failures, a peer is marked unhealthy and skipped. Check:

1. Is the peer's federation API reachable? curl https://peer.example.org:8000/health
2. Is TLS configured correctly? The peer's API must be accessible over HTTPS in production.
3. Are firewall rules open for port 8000?
4. The peer will be retried on subsequent gossip rounds. Once a sync succeeds, the peer is marked healthy again.

Records rejected on import

Records are rejected if the signer's fingerprint is not in the local trust store. Import the sender's public key first:

$ fieldwitness keys trust --import /path/to/sender-pubkey.pem

Consistency proof failure

A consistency proof failure means the peer's log is not a valid extension of the local log. This indicates a potential fork -- the peer may have a different chain history. Investigate before proceeding:

1. Compare chain heads: fieldwitness chain status on both instances
2. If a fork is confirmed, one instance's records must be exported and re-imported into a fresh chain

Gossip not starting

The gossip loop requires the federation extra:

$ pip install "fieldwitness[federation]"

This installs aiohttp for async HTTP communication.