fieldwitness/docs/federation.md

# Federation Guide

**Audience**: System administrators and technical leads setting up cross-organization
attestation sync between FieldWitness instances.

**Prerequisites**: A running FieldWitness instance (Tier 2 org server or Tier 3 relay), familiarity
with the CLI, and trusted public keys from partner organizations.

---

## Overview

FieldWitness federation synchronizes attestation records between organizations using a gossip
protocol. Nodes periodically exchange Merkle roots, detect divergence, and fetch missing
records. The system is eventually consistent with no central coordinator, no leader
election, and no consensus protocol -- just append-only logs that converge.

Federation operates at two levels:

1. **Offline bundles** -- JSON export/import via sneakernet (USB drive). Works on all tiers
   including fully airgapped Tier 1 field devices.
2. **Live gossip** -- HTTP-based periodic sync between Tier 2 org servers and Tier 3
   federation relays. Requires the `federation` extra (`pip install fieldwitness[federation]`).

> **Warning:** Federation shares attestation records (image hashes, Ed25519 signatures,
> timestamps, and signer public keys). It never shares encryption keys, plaintext messages,
> original images, or steganographic payloads.

---

## Architecture

```
Tier 1: Field Device          Tier 2: Org Server A         Tier 3: Relay (Iceland)
(Bootable USB)                (Docker / mini PC)           (VPS, zero key knowledge)
                                     |                            |
     USB sneakernet ------>   Port 5000 (Web UI)                  |
                              Port 8000 (Federation API)  <-----> Port 8000
                                     |                            |
                              Tier 2: Org Server B  <------------>
                              (Docker / mini PC)
```

Federation traffic flows:

- **Tier 1 to Tier 2**: USB sneakernet (offline bundles only)
- **Tier 2 to Tier 3**: gossip API over HTTPS (port 8000)
- **Tier 2 to Tier 2**: through a Tier 3 relay, or directly via sneakernet
- **Tier 3 to Tier 3**: gossip between relays in different jurisdictions

---

## Gossip Protocol

### How sync works

1. Node A sends its Merkle root and log size to Node B via `GET /federation/status`
2. If roots differ and B has more records, A requests a consistency proof via
   `GET /federation/consistency-proof?old_size=N`
3. If the proof verifies (B's log is a valid extension of A's), A fetches the missing
   records via `GET /federation/records?start=N&count=50`
4. A appends the new records to its local log
5. B performs the same process in reverse (bidirectional sync)

Records are capped at 100 per request to protect memory on resource-constrained devices.

### Peer health tracking

Each peer tracks:

- `last_seen` -- timestamp of last successful contact
- `last_root` -- most recent Merkle root received from the peer
- `last_size` -- most recent log size
- `healthy` -- marked `false` after 3 consecutive failures
- `consecutive_failures` -- reset to 0 on success

Unhealthy peers are skipped during gossip rounds but remain registered. They are retried
on the next full gossip round. Peer state persists in SQLite at
`~/.fwmetadata/attestations/federation/peers.db`.

### Gossip interval

The default gossip interval is 60 seconds, configurable via the `FIELDWITNESS_GOSSIP_INTERVAL`
environment variable. In Docker Compose, set it in the environment section:

```yaml
environment:
  - FIELDWITNESS_GOSSIP_INTERVAL=60
```

Lower intervals mean faster convergence but more network traffic.

---

## Setting Up Federation

### Step 1: Exchange trust keys

Before two organizations can federate, they must trust each other's Ed25519 identity keys.
Always verify fingerprints out-of-band (in person or over a known-secure voice channel).

On Organization A:

```bash
$ cp ~/.fwmetadata/identity/public.pem /media/usb/org-a-pubkey.pem
```

On Organization B:

```bash
$ fieldwitness keys trust --import /media/usb/org-a-pubkey.pem
```

Repeat in both directions so each organization trusts the other.

> **Warning:** Do not skip fingerprint verification. If an adversary substitutes their
> own public key, they can forge attestation records that your instance will accept as
> trusted.

### Step 2: Register peers (live gossip)

Through the web UI at `/federation`, or via the peer store directly:

```bash
# On Org A's server, register Org B's federation endpoint
$ fieldwitness federation peer add \
    --url https://orgb.example.org:8000 \
    --fingerprint a1b2c3d4e5f6...
```

Or through the web UI:

1. Navigate to `/federation`
2. Enter the peer's federation API URL and Ed25519 fingerprint
3. Click "Add Peer"

### Step 3: Start the gossip loop

The gossip loop starts automatically when the server starts. On Docker deployments, the
federation API runs on port 8000. Ensure this port is accessible between peers (firewall,
security groups, etc.).

For manual one-time sync:

```bash
$ fieldwitness federation sync --peer https://orgb.example.org:8000
```

### Step 4: Monitor sync status

The web UI at `/federation` shows:

- Local node status (Merkle root, log size, record count)
- Registered peers with health indicators
- Recent sync history (records received, errors)

---

## Offline Federation (Sneakernet)

For Tier 1 field devices and airgapped environments, use offline bundles.

### Exporting a bundle

```bash
$ fieldwitness chain export --output /media/usb/bundle.zip
```

To export only records from a specific investigation:

```bash
$ fieldwitness chain export --investigation "case-2026-001" --output /media/usb/bundle.zip
```

To export a specific index range:

```bash
$ fieldwitness chain export --start 100 --end 200 --output /media/usb/partial.zip
```

### Importing a bundle

On the receiving instance:

```bash
$ fieldwitness chain import /media/usb/bundle.zip
```

During import:

- Records signed by untrusted fingerprints are rejected
- Duplicate records (matching SHA-256) are skipped
- Imported records are tagged with `federated_from` metadata
- A delivery acknowledgment record (`fieldwitness/delivery-ack-v1`) is automatically appended
  to the local chain

### Delivery acknowledgments

When a bundle is imported, FieldWitness signs a `fieldwitness/delivery-ack-v1` chain record that
contains:

- The SHA-256 of the imported bundle file
- The sender's fingerprint
- The count of records received

This acknowledgment can be exported back to the sending organization as proof that the
bundle was delivered and ingested. It creates a two-way federation handshake.

```bash
# On receiving org: export the acknowledgment back
$ fieldwitness chain export --start <ack_index> --end <ack_index> \
    --output /media/usb/delivery-ack.zip
```

---

## Federation API Endpoints

The federation API is served by FastAPI/uvicorn on port 8000.

| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/federation/status` | Current Merkle root and log size |
| `GET` | `/federation/records?start=N&count=M` | Fetch attestation records (max 100) |
| `GET` | `/federation/consistency-proof?old_size=N` | Merkle consistency proof |
| `POST` | `/federation/records` | Push records to this node |
| `GET` | `/health` | Health check |

### Trust filtering on push

When records are pushed via `POST /federation/records`, the receiving node checks each
record's `attestor_fingerprint` against its trust store. Records from unknown attestors
are rejected. If no trust store is configured (empty trusted keys), all records are
accepted (trust-on-first-use).

---

## Federation Relay (Tier 3)

The federation relay is a minimal Docker container that runs only the federation API.

### What the relay stores

- Attestation records: image SHA-256 hashes, perceptual hashes, Ed25519 signatures
- Chain linkage data: prev_hash, chain_index, claimed_ts
- Signer public keys

### What the relay never sees

- AES-256-GCM channel keys or Ed25519 private keys
- Original images or media files
- Steganographic payloads or plaintext messages
- User credentials or session data
- Web UI content

### Deploying a relay

```bash
$ cd deploy/docker
$ docker compose up relay -d
```

The relay listens on port 8001 (mapped to internal 8000). See `docs/deployment.md`
Section 3 for full deployment details.

### Jurisdiction considerations

Deploy relays in jurisdictions with strong press freedom protections:

- **Iceland** -- strong source protection laws, no mandatory data retention for this type of data
- **Switzerland** -- strict privacy laws, resistance to foreign legal requests
- **Netherlands** -- strong press freedom, EU GDPR protections

Consult with a press freedom lawyer for your specific situation.

---

## Troubleshooting

**Peer marked unhealthy**

After 3 consecutive sync failures, a peer is marked unhealthy and skipped. Check:

1. Is the peer's federation API reachable? `curl https://peer.example.org:8000/health`
2. Is TLS configured correctly? The peer's API must be accessible over HTTPS in production.
3. Are firewall rules open for port 8000?
4. The peer will be retried on subsequent gossip rounds. Once a sync succeeds, the peer
   is marked healthy again.

**Records rejected on import**

Records are rejected if the signer's fingerprint is not in the local trust store. Import
the sender's public key first:

```bash
$ fieldwitness keys trust --import /path/to/sender-pubkey.pem
```

**Consistency proof failure**

A consistency proof failure means the peer's log is not a valid extension of the local log.
This indicates a potential fork -- the peer may have a different chain history. Investigate
before proceeding:

1. Compare chain heads: `fieldwitness chain status` on both instances
2. If a fork is confirmed, one instance's records must be exported and re-imported into a
   fresh chain

**Gossip not starting**

The gossip loop requires the `federation` extra:

```bash
$ pip install "fieldwitness[federation]"
```

This installs `aiohttp` for async HTTP communication.