fieldwitness/docs/architecture/federation-protocol.md

# Federation Protocol Specification

**Status**: Design  
**Version**: 1 (protocol version)  
**Last updated**: 2026-04-01

## 1. Overview

The federation is a network of append-only log servers inspired by Certificate Transparency
(RFC 6962). Each server acts as a "blind notary" — it stores encrypted attestation bundles,
maintains a Merkle tree over them, and issues signed receipts proving when bundles were
received. Servers gossip with peers to ensure consistency and replicate data.

Federation servers never decrypt attestation content. They operate at the "federation
member" permission tier: they can verify chain summaries and signatures, but not read
the underlying attestation data.

## 2. Terminology

| Term | Definition |
|---|---|
| **Bundle** | An encrypted export bundle (FIELDWITNESSX1 format) containing chain records |
| **STH** | Signed Tree Head — a server's signed commitment to its current Merkle tree state |
| **Receipt** | A server-signed proof that a bundle was included in its log at a specific time |
| **Inclusion proof** | Merkle path from a leaf (bundle hash) to the tree root |
| **Consistency proof** | Proof that an older tree is a prefix of a newer tree (no entries removed) |
| **Gossip** | Peer-to-peer exchange of STHs and entries to maintain consistency |

## 3. Server Merkle Tree

### 3.1 Structure

The server maintains a single append-only Merkle tree. Each leaf is the SHA-256 hash
of a received bundle's raw bytes:

```
leaf[i] = SHA-256(bundle_bytes[i])
```

Internal nodes follow standard Merkle tree construction:
```
node = SHA-256(0x01 || left || right)    # internal node
leaf = SHA-256(0x00 || data)             # leaf node (domain separation)
```

Domain separation prefixes (`0x00` for leaves, `0x01` for internal nodes) prevent
second-preimage attacks, following CT convention (RFC 6962 §2.1).

### 3.2 Signed Tree Head (STH)

After each append (or periodically in batch mode), the server computes and signs a new
tree head:

```cbor
{
    0: tree_size,        # uint — number of leaves
    1: root_hash,        # bytes[32] — Merkle tree root
    2: timestamp,        # int — Unix µs, server's clock
    3: server_id,        # text — server identifier (domain or pubkey fingerprint)
    4: server_pubkey,    # bytes[32] — Ed25519 public key
    5: signature,        # bytes[64] — Ed25519(cbor(fields 0-4))
}
```

The STH is the server's signed commitment: "My tree has N entries with this root at this
time." Clients and peers can verify the signature and use consistency proofs to ensure
the tree only grows (never shrinks or forks).

### 3.3 Inclusion Proof

Proves a specific bundle is at index `i` in a tree of size `n`:

```
proof = [(sibling_hash, direction), ...]
```

Verification:
```
current = SHA-256(0x00 || bundle_bytes)
for (sibling, direction) in proof:
    if direction == "L":
        current = SHA-256(0x01 || sibling || current)
    else:
        current = SHA-256(0x01 || current || sibling)
assert current == sth.root_hash
```

### 3.4 Consistency Proof

Proves that tree of size `m` is a prefix of tree of size `n` (where `m < n`). This
guarantees the server hasn't removed or reordered entries.

The proof is a list of intermediate hashes that, combined with the old root, reconstruct
the new root. Verification follows RFC 6962 §2.1.2.

## 4. API Endpoints

All endpoints use CBOR for request/response bodies. Content-Type: `application/cbor`.

### 4.1 Submit Bundle

```
POST /v1/submit
```

**Request body**: Raw bundle bytes (application/octet-stream)

**Processing**:
1. Verify magic bytes `b"FIELDWITNESSX1"` and version
2. Parse chain summary
3. Verify `bundle_sig` against `signer_pubkey`
4. Compute `bundle_hash = SHA-256(0x00 || bundle_bytes)`
5. Check for duplicate (`bundle_hash` already in tree) — if duplicate, return existing receipt
6. Append `bundle_hash` to Merkle tree
7. Store bundle bytes (encrypted blob, as-is)
8. Generate and sign receipt

**Response** (CBOR):
```cbor
{
    0: bundle_id,          # bytes[16] — from chain summary
    1: bundle_hash,        # bytes[32] — leaf hash
    2: tree_size,          # uint — tree size after inclusion
    3: tree_index,         # uint — leaf index in tree
    4: timestamp,          # int — Unix µs, server's reception time
    5: inclusion_proof,    # array of bytes[32] — Merkle path
    6: sth,                # map — current STH (see §3.2)
    7: server_id,          # text — server identifier
    8: server_pubkey,      # bytes[32] — Ed25519 public key
    9: receipt_sig,        # bytes[64] — Ed25519(cbor(fields 0-8))
}
```

**Auth**: Federation member token required.

**Errors**:
- `400` — Invalid bundle format, bad signature
- `401` — Missing or invalid auth token
- `507` — Server storage full

### 4.2 Get Signed Tree Head

```
GET /v1/sth
```

**Response** (CBOR): STH map (see §3.2)

**Auth**: Public (no auth required).

### 4.3 Get Consistency Proof

```
GET /v1/consistency-proof?old={m}&new={n}
```

**Parameters**:
- `old` — previous tree size (must be > 0)
- `new` — current tree size (must be >= old)

**Response** (CBOR):
```cbor
{
    0: old_size,        # uint
    1: new_size,        # uint
    2: proof,           # array of bytes[32]
}
```

**Auth**: Public.

### 4.4 Get Inclusion Proof

```
GET /v1/inclusion-proof?hash={hex}&tree_size={n}
```

**Parameters**:
- `hash` — hex-encoded bundle hash (leaf hash)
- `tree_size` — tree size for the proof (use current STH tree_size)

**Response** (CBOR):
```cbor
{
    0: tree_index,      # uint — leaf index
    1: tree_size,       # uint
    2: proof,           # array of bytes[32]
}
```

**Auth**: Public.

### 4.5 Get Entries

```
GET /v1/entries?start={s}&end={e}
```

**Parameters**:
- `start` — first tree index (inclusive)
- `end` — last tree index (inclusive)
- Maximum range: 1000 entries per request

**Response** (CBOR):
```cbor
{
    0: entries,    # array of entry maps (see §4.5.1)
}
```

#### 4.5.1 Entry Map

```cbor
{
    0: tree_index,          # uint
    1: bundle_hash,         # bytes[32]
    2: chain_summary,       # CBOR map (from bundle, unencrypted)
    3: encrypted_blob,      # bytes — full FIELDWITNESSX1 bundle
    4: receipt_ts,          # int — Unix µs when received
}
```

**Auth**: Federation member token required.

### 4.6 Audit Summary

```
GET /v1/audit/summary?bundle_id={hex}
```

Returns the chain summary for a specific bundle without the encrypted payload.

**Response** (CBOR):
```cbor
{
    0: bundle_id,        # bytes[16]
    1: chain_summary,    # CBOR map (from bundle)
    2: tree_index,       # uint
    3: receipt_ts,       # int
    4: inclusion_proof,  # array of bytes[32] (against current STH)
}
```

**Auth**: Public.

## 5. Permission Tiers

### 5.1 Public Auditor

**Access**: Unauthenticated.

**Endpoints**: `/v1/sth`, `/v1/consistency-proof`, `/v1/inclusion-proof`, `/v1/audit/summary`

**Can verify**:
- The log exists and has a specific size at a specific time
- A specific bundle is included in the log at a specific position
- The log has not been forked (consistency proofs between STHs)
- Chain summary metadata (record count, hash range) for any bundle

**Cannot see**: Encrypted content, chain IDs, signer identities, raw bundles.

### 5.2 Federation Member

**Access**: Bearer token issued by server operator. Tokens are Ed25519-signed
credentials binding a public key to a set of permissions.

```cbor
{
    0: token_id,         # bytes[16] — UUID v7
    1: member_pubkey,    # bytes[32] — member's Ed25519 public key
    2: permissions,      # array of text — ["submit", "entries", "gossip"]
    3: issued_at,        # int — Unix µs
    4: expires_at,       # int — Unix µs (0 = no expiry)
    5: issuer_pubkey,    # bytes[32] — server's Ed25519 public key
    6: signature,        # bytes[64] — Ed25519(cbor(fields 0-5))
}
```

**Endpoints**: All public endpoints + `/v1/submit`, `/v1/entries`, gossip endpoints.

**Can see**: Everything a public auditor sees + chain IDs, signer public keys, full
encrypted bundles (but not decrypted content).

### 5.3 Authorized Recipient

Not enforced server-side. Recipients hold Ed25519 private keys whose corresponding
public keys appear in the bundle's recipients array. They can decrypt bundle content
locally after retrieving the encrypted blob via the entries endpoint.

The server has no knowledge of who can or cannot decrypt a given bundle.

## 6. Gossip Protocol

### 6.1 Overview

Federation servers maintain a list of known peers. Periodically (default: every 5 minutes),
each server initiates gossip with its peers to:

1. Exchange STHs — detect if any peer has entries the local server doesn't
2. Verify consistency — ensure no peer is presenting a forked log
3. Sync entries — pull missing entries from peers that have them

### 6.2 Gossip Flow

```
Server A                                Server B
   │                                       │
   │── POST /v1/gossip/sth ──────────────>│   (A sends its STH)
   │                                       │
   │<── response: B's STH ───────────────│    (B responds with its STH)
   │                                       │
   │  (A compares tree sizes)              │
   │  if B.tree_size > A.tree_size:        │
   │                                       │
   │── GET /v1/consistency-proof ────────>│   (verify B's tree extends A's)
   │<── proof ────────────────────────────│
   │                                       │
   │  (verify consistency proof)           │
   │                                       │
   │── GET /v1/entries?start=...&end=... >│   (pull missing entries)
   │<── entries ──────────────────────────│
   │                                       │
   │  (append entries to local tree)       │
   │  (recompute STH)                      │
   │                                       │
```

### 6.3 Gossip Endpoints

```
POST /v1/gossip/sth
```

**Request body** (CBOR): Sender's current STH.

**Response** (CBOR): Receiver's current STH.

**Auth**: Federation member token with `"gossip"` permission.

### 6.4 Fork Detection

If server A receives an STH from server B where:
- `B.tree_size <= A.tree_size` but `B.root_hash != A.root_hash` at the same size

Then B is presenting a different history. This is a **fork** — a critical security event.
The server should:

1. Log the fork with both STHs as evidence
2. Alert the operator
3. Continue serving its own tree (do not merge the forked tree)
4. Refuse to gossip further with the forked peer until operator resolution

### 6.5 Convergence

Under normal operation (no forks), servers converge to identical trees. The convergence
time depends on gossip interval and network topology. With a 5-minute interval and full
mesh topology among N servers, convergence after a new entry takes at most 5 minutes.

## 7. Receipts

### 7.1 Purpose

A receipt is the federation's proof that a bundle was received and included in the log
at a specific time. It is the critical artifact that closes the timestamp gap: the
offline device's claimed timestamp + the federation receipt = practical proof of timing.

### 7.2 Receipt Format

```cbor
{
    0: bundle_id,          # bytes[16] — from chain summary
    1: bundle_hash,        # bytes[32] — leaf hash in server's tree
    2: tree_size,          # uint — tree size at inclusion
    3: tree_index,         # uint — leaf position
    4: timestamp,          # int — Unix µs, server's clock
    5: inclusion_proof,    # array of bytes[32] — Merkle path
    6: sth,                # map — STH at time of inclusion
    7: server_id,          # text — server identifier
    8: server_pubkey,      # bytes[32] — Ed25519 public key
    9: receipt_sig,        # bytes[64] — Ed25519(cbor(fields 0-8))
}
```

### 7.3 Receipt Verification

To verify a receipt:

1. `Ed25519_Verify(server_pubkey, receipt_sig, cbor(fields 0-8))` — receipt is authentic
2. Verify `inclusion_proof` against `sth.root_hash` with `bundle_hash` at `tree_index`
3. Verify `sth.signature` — the STH itself is authentic
4. `sth.tree_size >= tree_size` — STH covers the inclusion
5. `sth.timestamp >= timestamp` — STH is at or after receipt time

### 7.4 Receipt Lifecycle

```
1. Loader submits bundle to federation server
2. Server issues receipt in submit response
3. Loader stores receipt locally (receipts/ directory)
4. Loader exports receipts to USB (CBOR file)
5. Offline device imports receipts
6. Receipt is stored alongside chain records as proof of federation timestamp
```

### 7.5 Multi-Server Receipts

A bundle submitted to N servers produces N independent receipts. Each receipt is from a
different server with a different timestamp and Merkle tree position. Multiple receipts
strengthen the timestamp claim — an adversary would need to compromise all N servers to
suppress evidence.

## 8. Storage Tiers

Federation servers manage bundle storage across three tiers based on age:

### 8.1 Hot Tier (0-30 days)

- **Format**: Individual files, one per bundle
- **Location**: `data/hot/{tree_index}.bundle`
- **Access**: Direct file read, O(1)
- **Purpose**: Fast access for recent entries, active gossip sync

### 8.2 Warm Tier (30-365 days)

- **Format**: Zstd-compressed segments, 1000 bundles per segment
- **Location**: `data/warm/segment-{start}-{end}.zst`
- **Access**: Decompress segment, extract entry
- **Compression**: Zstd level 3 (fast compression, moderate ratio)
- **Purpose**: Reduced storage for medium-term retention

### 8.3 Cold Tier (>1 year)

- **Format**: Zstd-compressed segments, maximum compression
- **Location**: `data/cold/segment-{start}-{end}.zst`
- **Access**: Decompress segment, extract entry
- **Compression**: Zstd level 19 (slow compression, best ratio)
- **Purpose**: Archival storage, rarely accessed

### 8.4 Tier Promotion

A background compaction process runs periodically (default: every 24 hours):

1. Identify hot entries older than 30 days
2. Group into segments of 1000
3. Compress and write to warm tier
4. Delete hot files
5. Repeat for warm → cold at 365 days

### 8.5 Merkle Tree Preservation

The Merkle tree is independent of storage tiers. Leaf hashes and the tree structure
are maintained in a separate data structure (compact tree format, stored in SQLite or
flat file). Moving bundles between storage tiers does not affect the tree.

Inclusion proofs and consistency proofs remain valid across tier promotions — they
reference the tree, not the storage location.

### 8.6 Metadata Database

SQLite database tracking all bundles:

```sql
CREATE TABLE bundles (
    tree_index     INTEGER PRIMARY KEY,
    bundle_id      BLOB NOT NULL,          -- UUID v7
    bundle_hash    BLOB NOT NULL,          -- leaf hash
    chain_id       BLOB NOT NULL,          -- source chain ID
    signer_pubkey  BLOB NOT NULL,          -- Ed25519 public key
    record_count   INTEGER NOT NULL,       -- records in bundle
    range_start    INTEGER NOT NULL,       -- first chain index
    range_end      INTEGER NOT NULL,       -- last chain index
    receipt_ts     INTEGER NOT NULL,       -- Unix µs reception time
    storage_tier   TEXT NOT NULL DEFAULT 'hot',  -- 'hot', 'warm', 'cold'
    storage_key    TEXT NOT NULL,          -- file path or segment reference
    created_at     TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
);

CREATE INDEX idx_bundles_bundle_id ON bundles(bundle_id);
CREATE INDEX idx_bundles_chain_id ON bundles(chain_id);
CREATE INDEX idx_bundles_bundle_hash ON bundles(bundle_hash);
CREATE INDEX idx_bundles_receipt_ts ON bundles(receipt_ts);
```

## 9. Server Configuration

```json
{
    "server_id": "my-server.example.org",
    "host": "0.0.0.0",
    "port": 8443,
    "data_dir": "/var/lib/fieldwitness-federation",
    "identity_key_path": "/etc/fieldwitness-federation/identity/private.pem",
    "peers": [
        {
            "url": "https://peer1.example.org:8443",
            "pubkey_hex": "abc123...",
            "name": "Peer One"
        }
    ],
    "gossip_interval_seconds": 300,
    "hot_retention_days": 30,
    "warm_retention_days": 365,
    "compaction_interval_hours": 24,
    "max_bundle_size_bytes": 10485760,
    "max_entries_per_request": 1000,
    "member_tokens": [
        {
            "name": "loader-1",
            "pubkey_hex": "def456...",
            "permissions": ["submit", "entries"]
        }
    ]
}
```

## 10. Error Codes

| HTTP Status | CBOR Error Code | Description |
|---|---|---|
| 400 | `"invalid_bundle"` | Bundle format invalid or signature verification failed |
| 400 | `"invalid_range"` | Requested entry range is invalid |
| 401 | `"unauthorized"` | Missing or invalid auth token |
| 403 | `"forbidden"` | Token lacks required permission |
| 404 | `"not_found"` | Bundle or entry not found |
| 409 | `"duplicate"` | Bundle already in log (returns existing receipt) |
| 413 | `"bundle_too_large"` | Bundle exceeds `max_bundle_size_bytes` |
| 507 | `"storage_full"` | Server cannot accept new entries |

Error response format:
```cbor
{
    0: error_code,     # text
    1: message,        # text — human-readable description
    2: details,        # map — optional additional context
}
```

## 11. Security Considerations

### 11.1 Server Compromise

A compromised server can:
- Read bundle metadata (chain IDs, signer pubkeys, timestamps) — **expected at member tier**
- Withhold entries from gossip — **detectable**: other servers will see inconsistent tree sizes
- Present a forked tree — **detectable**: consistency proofs will fail
- Issue false receipts — **detectable**: receipt's inclusion proof won't verify against other servers' STHs

A compromised server **cannot**:
- Read attestation content (encrypted with recipient keys)
- Forge attestation signatures (requires Ed25519 private key)
- Modify bundle contents (GCM authentication would fail)
- Reorder or remove entries from other servers' trees

### 11.2 Transport Security

All server-to-server and client-to-server communication should use TLS 1.3. The
federation protocol provides its own authentication (Ed25519 signatures on STHs and
receipts), but TLS prevents network-level attacks.

### 11.3 Clock Reliability

Federation server clocks should be synchronized via NTP. Receipt timestamps are only as
reliable as the server's clock. Deploying servers across multiple time zones and operators
provides cross-checks — wildly divergent receipt timestamps for the same bundle indicate
clock problems or compromise.