# FieldWitness Deployment Guide

Three-tier deployment model for field journalism, organizational evidence management,
and cross-organization federation.

This guide is for field deployers: IT staff at NGOs, technically competent journalists,
and anyone setting up FieldWitness for operational use. Read the tier descriptions first, then
jump to the section that matches your deployment.

---

## Deployment Overview

```
Tier 1: Field Device             Tier 2: Org Server            Tier 3: Federation Relay
(Bootable USB + laptop)          (Docker on mini PC / VPS)     (Docker on VPS)
                                                               
Reporter in the field            Newsroom / NGO office         Friendly jurisdiction
Amnesic, encrypted               Persistent storage            Attestation sync only
Pull USB = zero trace            Web UI + federation API       Zero knowledge of keys
                    \                      |                      /
                     \_____ sneakernet ____+____ gossip API ____/
```

**Tier 1 -- Field Device.** A bootable Debian Live USB stick that a reporter plugs into
any x86_64 laptop. Boots into a minimal desktop with Firefox pointed at the local web UI.
LUKS-encrypted persistent partition for keys, config, and attestations. Amnesic by default:
pull the USB and the host machine retains nothing.

**Tier 2 -- Org Server.** A persistent Docker deployment on a mini PC in the newsroom or a
trusted VPS. Runs the full FieldWitness web UI (port 5000) and the federation API (port 8000).
Stores attestations, manages keys, and synchronizes with federation relays.

**Tier 3 -- Federation Relay.** A lightweight Docker container on a VPS in a jurisdiction
with strong press protections (Iceland, Switzerland, Netherlands). Relays attestation
records between organizations. Stores only attestation hashes and signatures -- never keys,
plaintext, or original media. If seized, the attacker gets cryptographic hashes and nothing
actionable.

---

## 1. Tier 1: Field Device (Bootable USB)

### 1.1 What you need

- A USB drive, 16 GB minimum (32 GB recommended for persistent storage)
- Any x86_64 laptop to boot from (ideally a refurbished ThinkPad T440/T480 -- cheap,
  common, replaceable, good Linux support -- but any laptop that boots from USB works)
- A build machine with `live-build` installed (any Debian/Ubuntu system)

The resulting USB image is a Debian Live system with FieldWitness pre-installed. No pip, no
terminal, no manual setup. The reporter boots it and gets a working FieldWitness instance.

### 1.2 Building the USB image

On the build machine:

```bash
sudo apt install live-build

cd deploy/live-usb
sudo ./build.sh
```

This uses `lb` (live-build) to create a Debian Bookworm hybrid ISO. The build takes
10-20 minutes depending on your machine and network speed.

Output: `live-image-amd64.hybrid.iso`

Flash it to USB:

```bash
sudo dd if=live-image-amd64.hybrid.iso of=/dev/sdX bs=4M status=progress
```

Replace `/dev/sdX` with your USB device. Double-check the device path -- `dd` will
happily overwrite your system drive.

Alternatively, use Balena Etcher if you prefer a GUI.

### 1.3 Boot behavior

When a reporter boots from the USB:

1. GRUB loads the Debian Live system
2. A minimal desktop environment starts (no login prompt)
3. Firefox opens automatically, pointed at `https://127.0.0.1:5000`
4. The FieldWitness web UI prompts for first-user setup (on first boot) or login

No terminal interaction required for normal operation.

### 1.4 Persistent encrypted storage

The USB image includes a LUKS-encrypted persistent partition. On first boot, the reporter
sets a passphrase. All FieldWitness state lives on this partition:

```
/persistent/
  .fwmetadata/             Keys, config, attestations, chain data, auth
```

On subsequent boots, the system prompts for the LUKS passphrase to unlock the persistent
partition. If the passphrase is not entered (or wrong), FieldWitness starts in a fresh,
ephemeral state -- useful for crossing borders with a "clean" appearance.

### 1.5 Amnesic operation

The live system runs from RAM. When the USB is removed:

- The host laptop retains zero FieldWitness data (no files, no swap traces, no browser cache)
- The host's own storage is never written to
- RAM contents are gone on power-off

This is the primary security property. The laptop is disposable -- if seized, it contains
nothing. The USB stick is the only thing that matters, and it is LUKS-encrypted.

### 1.6 Killswitch on flash storage

On a live USB, the killswitch works by destroying the LUKS key rather than trying to
overwrite individual files. This is the only reliable secure deletion method on flash
storage, because:

- `shred` does not work reliably on flash (wear leveling preserves old blocks)
- LUKS key destruction makes the entire encrypted partition unrecoverable
- The LUKS header is small enough to fit in a single flash block, so destruction is
  near-instantaneous

When the killswitch fires on a Tier 1 device, it:

1. Overwrites the LUKS header (renders the partition permanently unreadable)
2. Zeroes the LUKS key slots
3. Triggers a reboot (system returns to amnesic state with no persistent data)

### 1.7 Recommended field hardware

Any x86_64 laptop works, but refurbished ThinkPads are ideal for field use:

- **ThinkPad T440 / T480**: common, cheap ($100-200 refurbished), excellent Linux
  support, replaceable parts, inconspicuous (every office has one)
- Avoid laptops with Secure Boot that cannot be disabled (some newer consumer models)
- Avoid laptops with soldered RAM below 4 GB

The laptop itself holds no sensitive data. If it is seized, lost, or broken, acquire
another and boot from the same USB.

---

## 2. Tier 2: Org Server (Docker)

### 2.1 What you need

- A mini PC (Intel NUC, Beelink, etc.) or trusted VPS with Docker installed
- 2 GB RAM minimum, 10 GB storage minimum
- Network connectivity (LAN for newsroom, or internet for VPS with federation)

### 2.2 Docker deployment

```bash
cd deploy/docker

# Start the org server
docker compose up server -d
```

This starts the full FieldWitness server with:

- **Port 5000**: Web UI (Flask/Waitress) -- stego, attestation, key management, admin
- **Port 8000**: Federation API (FastAPI/uvicorn) -- cross-org attestation sync

The Docker image is a multi-stage build (`deploy/docker/Dockerfile`, target: `server`).
It installs FieldWitness into a Python 3.12 virtualenv, copies frontend assets, and runs
`fieldwitness init` on first start followed by `fieldwitness serve`.

Data is persisted in a Docker volume (`server-data`) mounted at `/data` inside the
container. The `FIELDWITNESS_DATA_DIR` environment variable points FieldWitness at this volume.

### 2.3 Docker Compose reference

The full `deploy/docker/docker-compose.yml` defines both Tier 2 and Tier 3 services:

```yaml
services:
  server:                    # Tier 2: Full org server
    ports:
      - "5000:5000"          # Web UI
      - "8000:8000"          # Federation API
    volumes:
      - server-data:/data
    environment:
      - FIELDWITNESS_DATA_DIR=/data
      - FIELDWITNESS_GOSSIP_INTERVAL=60

  relay:                     # Tier 3: Federation relay
    ports:
      - "8001:8000"          # Federation API
    volumes:
      - relay-data:/data
    environment:
      - FIELDWITNESS_DATA_DIR=/data
```

Adjust port mappings and volume drivers as needed for your environment.

### 2.4 Kubernetes deployment

For organizations already running Kubernetes:

```bash
# Build images
docker build -t fieldwitness-server --target server -f deploy/docker/Dockerfile .
docker build -t fieldwitness-relay --target relay -f deploy/docker/Dockerfile .

# Deploy
kubectl apply -f deploy/kubernetes/namespace.yaml
kubectl apply -f deploy/kubernetes/server-deployment.yaml
```

Important constraints from `deploy/kubernetes/README.md`:

- **Single writer only.** Both deployments use `replicas: 1` with `Recreate` strategy.
  FieldWitness uses SQLite and append-only binary logs that require single-writer access. Do not
  scale horizontally.
- **Persistent volumes required.** The server needs 10Gi, the relay needs 5Gi. Adjust
  based on expected attestation volume.
- **Ingress not included.** Configure your own ingress controller with TLS termination.

### 2.5 TLS and reverse proxy

The Docker container runs without HTTPS internally (`--no-https` in the CMD). TLS should
be handled by:

- A reverse proxy (nginx, Caddy, Traefik) in front of the container
- Your Kubernetes ingress controller
- A cloud load balancer (if on a VPS)

For a simple mini-PC newsroom setup without a reverse proxy, override the CMD to remove
`--no-https` and let FieldWitness generate a self-signed certificate.

### 2.6 Backups

The Docker volume contains all FieldWitness state. Back it up:

```bash
# Stop the container, snapshot the volume, restart
docker compose stop server
docker run --rm -v server-data:/data -v /backup:/backup \
    busybox tar czf /backup/fieldwitness-$(date +%Y%m%d).tar.gz -C /data .
docker compose start server
```

Or use `fieldwitness keys export` from inside the container for key-only backups.

---

## 3. Tier 3: Federation Relay (Docker)

### 3.1 Purpose

The federation relay synchronizes attestation records between organizations. It is
deliberately limited:

- Stores only attestation records (image hashes, signatures, chain data)
- Never sees encryption keys, plaintext messages, or original images
- No web UI, no steganography features
- If seized, the attacker gets cryptographic hashes -- nothing actionable

### 3.2 Deployment

```bash
cd deploy/docker

# Start the relay
docker compose up relay -d
```

This starts the relay with:

- **Port 8001** (mapped to internal 8000): Federation API (FastAPI/uvicorn)
- Data volume: `relay-data` at `/data`

The Docker image uses the `relay` target from the same multi-stage Dockerfile. The relay
runs only `uvicorn fieldwitness.attest.api:app` -- the minimal federation endpoint.

### 3.3 Kubernetes deployment

```bash
kubectl apply -f deploy/kubernetes/namespace.yaml
kubectl apply -f deploy/kubernetes/relay-deployment.yaml
```

Same constraints as Tier 2: single replica, persistent volume (5Gi), bring your own
ingress with TLS.

### 3.4 Jurisdiction considerations

The relay should be hosted in a jurisdiction with:

- Strong press freedom protections
- No mandatory data retention laws that apply to this type of data
- Legal resistance to foreign seizure orders

Recommended: **Iceland**, **Switzerland**, **Netherlands**. These are suggestions, not
guarantees -- consult with a press freedom lawyer for your specific situation.

### 3.5 What happens if the relay is compromised

The relay stores attestation records: SHA-256 hashes of images, Ed25519 signatures, chain
linkage data, and signer public keys. It does not store:

- Encryption keys (AES-256-GCM channel keys or Ed25519 private keys)
- Original images or media
- Steganographic payloads or plaintext messages
- User credentials or session data

An attacker who seizes the relay learns that certain public keys signed certain image
hashes at certain times. They cannot recover the images, decrypt messages, or forge new
attestations.

---

## 4. Threat Level Configuration Presets

FieldWitness ships four configuration presets at `deploy/config-presets/`. Choose one based on
your operational threat environment and copy it to your config location.

```bash
cp deploy/config-presets/high-threat.json ~/.fwmetadata/config.json
```

### 4.1 Low Threat -- Press Freedom Country

File: `deploy/config-presets/low-threat.json`

For deployments in the Nordics, New Zealand, Canada, and similar environments where the
risk is accidental data loss rather than adversarial seizure.

| Setting | Value |
|---|---|
| Session timeout | 30 minutes |
| Killswitch | Disabled |
| Dead man's switch | Disabled |
| USB monitoring | Disabled |
| Tamper monitoring | Disabled |
| Backup reminders | Every 14 days |
| Cover name | None |

Chain and attestation are enabled for provenance integrity. Fieldkit security features
are off because the threat model does not warrant them.

### 4.2 Medium Threat -- Restricted Press

File: `deploy/config-presets/medium-threat.json`

For deployments in Turkey, Hungary, India, and similar environments with legal pressure
and risk of device seizure at borders.

| Setting | Value |
|---|---|
| Session timeout | 15 minutes |
| Killswitch | Enabled |
| Dead man's switch | 48 hours / 4 hours grace |
| USB monitoring | Enabled |
| Tamper monitoring | Disabled |
| Login lockout | 5 attempts / 15 min lockout |
| Backup reminders | Every 7 days |
| Cover name | "Office Document Manager" |

### 4.3 High Threat -- Active Conflict Zone

File: `deploy/config-presets/high-threat.json`

For deployments in Syria, Myanmar, Ethiopia, Iran, and similar environments with risk of
raids, equipment seizure, and physical coercion.

| Setting | Value |
|---|---|
| Session timeout | 5 minutes |
| Killswitch | Enabled |
| Dead man's switch | 12 hours / 1 hour grace |
| USB monitoring | Enabled |
| Tamper monitoring | Enabled |
| Login lockout | 3 attempts / 30 min lockout |
| Backup reminders | Daily |
| Cover name | "Local Inventory Tracker" |

### 4.4 Critical Threat -- Targeted Surveillance

File: `deploy/config-presets/critical-threat.json`

For a specific journalist or organization targeted by a state actor (Pegasus-level threat).

| Setting | Value |
|---|---|
| Host binding | 127.0.0.1 only (access via SSH tunnel) |
| Session timeout | 3 minutes |
| Killswitch | Enabled |
| Dead man's switch | 6 hours / 1 hour grace |
| USB monitoring | Enabled |
| Tamper monitoring | Enabled |
| Login lockout | 3 attempts / 60 min lockout |
| Backup reminders | Daily |
| Cover name | "System Statistics" |

At this threat level, also consider: full-disk encryption on all devices, removing SSH
after initial setup, running the web UI as a Tor hidden service, and physical security
measures that are outside the scope of this software.

### 4.5 Customizing presets

The presets are starting points. Override individual settings in `~/.fwmetadata/config.json`
after copying. The full configuration reference is in Section 8.

---

## 5. Initial Setup (All Tiers)

### 5.1 Initialize FieldWitness

On Tier 1 (USB), initialization happens automatically on first boot. On Tier 2/3 (Docker),
the container runs `fieldwitness init` on first start. For manual installs:

```bash
fieldwitness init
```

This creates the `~/.fwmetadata/` directory structure:

```
~/.fwmetadata/
  config.json           Unified configuration
  identity/             Ed25519 signing keypair (attest)
    private.pem
    public.pem
    identity.meta.json
  stego/             Stego state
    channel.key         AES-256-GCM channel key
  attestations/         Attest attestation log and index
  chain/                Hash chain data
    anchors/            External timestamp anchors
  auth/                 Web UI user database (SQLite)
  certs/                Self-signed TLS certificates
  fieldkit/             Killswitch, deadman, tamper, USB, geofence state
  temp/                 Ephemeral upload/processing files
    dropbox/            Source drop box submissions
  instance/             Flask session data
  audit.jsonl           Append-only audit trail
```

The `identity/` and `auth/` directories are created with mode 0700.

`fieldwitness init` generates:

- An Ed25519 identity keypair (for signing attestations)
- A channel key (for steganographic encoding)
- A default `config.json`

### 5.2 Apply a threat level preset

```bash
cp deploy/config-presets/<level>-threat.json ~/.fwmetadata/config.json
```

See Section 4 for preset descriptions.

### 5.3 First user setup

Start the server and create the first admin user through the web UI:

```bash
fieldwitness serve --host 0.0.0.0 --no-https
```

Navigate to `http://<host-ip>:5000` from a device on the same network. The web UI will
prompt you to create the first user account.

After creating the admin account, stop the server (Ctrl+C) and restart with HTTPS
(see Section 7).

---

## 6. Security Hardening

These hardening steps apply primarily to Tier 2 (org server on bare metal) and to the
build environment for Tier 1 USB images. Docker deployments handle most of this within
the container, but the host machine still needs attention.

### 6.1 Disable or encrypt swap

Python does not zero memory when objects are garbage-collected. Cryptographic keys,
passwords, and plaintext can persist in swap long after the process exits. On flash
storage (SSDs, USB drives), this data may survive even after the swap file is deleted
due to wear leveling.

**Option A: Disable swap entirely (recommended if you have 4 GB+ RAM)**

```bash
sudo swapoff -a
# Remove or comment out swap entries in /etc/fstab
```

**Option B: Encrypted swap**

```bash
# Use dm-crypt with a random key (regenerated each boot)
echo "swap /dev/sdXN /dev/urandom swap,cipher=aes-xts-plain64,size=256" | sudo tee -a /etc/crypttab
echo "/dev/mapper/swap none swap sw 0 0" | sudo tee -a /etc/fstab
```

Adjust the device path to match your partition layout.

### 6.2 Disable core dumps

A core dump from the FieldWitness process would contain key material in plaintext.

```bash
echo "* hard core 0" | sudo tee -a /etc/security/limits.conf
echo "kernel.core_pattern=/dev/null" | sudo tee -a /etc/sysctl.d/99-fieldwitness.conf
sudo sysctl -p /etc/sysctl.d/99-fieldwitness.conf
```

### 6.3 Firewall

```bash
sudo apt install -y ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 5000/tcp   # FieldWitness web UI
sudo ufw allow 8000/tcp   # Federation API (Tier 2 only)
sudo ufw enable
```

If running fully airgapped, also deny outgoing:

```bash
sudo ufw default deny outgoing
```

### 6.4 Disable unnecessary services

```bash
sudo systemctl disable bluetooth
sudo systemctl disable avahi-daemon
```

Adjust based on what your system has running. The goal is to minimize attack surface.

---

## 7. Running

### 7.1 Basic usage (bare metal / Tier 2 without Docker)

```bash
# LAN-only, no HTTPS (acceptable if the network is physically isolated)
fieldwitness serve --host 0.0.0.0 --no-https

# With self-signed HTTPS (recommended)
fieldwitness serve --host 0.0.0.0

# Custom port
fieldwitness serve --host 0.0.0.0 --port 8443
```

On first HTTPS start, FieldWitness auto-generates a self-signed certificate at
`~/.fwmetadata/certs/cert.pem`. Browsers will show a certificate warning -- this is expected
for self-signed certs. Instruct users to accept the warning or distribute the cert file
to client devices.

FieldWitness uses Waitress (pure Python, no C dependencies) as its production server with 4
worker threads by default. Adjust with `--workers`.

### 7.2 systemd service (bare metal Tier 2)

Create `/etc/systemd/system/fieldwitness.service`:

```ini
[Unit]
Description=FieldWitness Security Fieldkit
After=network.target

[Service]
Type=simple
User=fieldwitness
Group=fieldwitness
WorkingDirectory=/home/fieldwitness
Environment="PATH=/home/fieldwitness/fieldwitness-env/bin:/usr/bin"
ExecStart=/home/fieldwitness/fieldwitness-env/bin/fieldwitness serve --host 0.0.0.0 --workers 4
Restart=on-failure
RestartSec=5

# Hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/fieldwitness/.fieldwitness
PrivateTmp=yes

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now fieldwitness
sudo journalctl -u fieldwitness -f   # Watch logs
```

Add `--no-https` to `ExecStart` if running on a physically isolated LAN where TLS is
unnecessary.

---

## 8. Configuration Reference

Configuration lives at `~/.fwmetadata/config.json`. Edit it directly or use the web admin
panel. All fields have sensible defaults -- you only need to set what you want to change.

| Field | Default | Description |
|---|---|---|
| `host` | `127.0.0.1` | Bind address. Set to `0.0.0.0` for LAN access. |
| `port` | `5000` | TCP port for the web UI. |
| `https_enabled` | `true` | Enable self-signed HTTPS. |
| `auth_enabled` | `true` | Require login. Do not disable this. |
| `max_upload_mb` | `50` | Maximum file upload size in MB. |
| `session_timeout_minutes` | `15` | Idle session expiry. Lower is safer. |
| `login_lockout_attempts` | `5` | Failed logins before lockout. |
| `login_lockout_minutes` | `15` | Lockout duration after exceeding failed login attempts. |
| `default_embed_mode` | `auto` | Default steganographic embedding mode for Stego. |
| `killswitch_enabled` | `false` | Enable software killswitch. |
| `deadman_enabled` | `false` | Enable dead man's switch. |
| `deadman_interval_hours` | `24` | Hours between required check-ins. |
| `deadman_grace_hours` | `2` | Grace period after missed check-in before purge. |
| `deadman_warning_webhook` | `""` | URL to POST a JSON warning when check-in is overdue. Must be a public URL (SSRF protection blocks private IPs). |
| `usb_monitoring_enabled` | `false` | Monitor for unauthorized USB devices. |
| `tamper_monitoring_enabled` | `false` | File integrity monitoring. |
| `chain_enabled` | `true` | Wrap attestations in a hash chain. |
| `chain_auto_wrap` | `true` | Automatically chain attest attestations. |
| `backup_reminder_days` | `7` | Warn if no backup in this many days. `fieldwitness status` reports overdue backups. |
| `cover_name` | `""` | If set, used as the CN in the self-signed SSL certificate instead of "localhost". See Section 15 (Cover/Duress Mode). |

Example minimal config for a high-threat field deployment:

```json
{
  "host": "0.0.0.0",
  "port": 5000,
  "https_enabled": true,
  "session_timeout_minutes": 5,
  "deadman_enabled": true,
  "deadman_interval_hours": 12,
  "deadman_grace_hours": 1,
  "deadman_warning_webhook": "https://hooks.example.com/alert",
  "killswitch_enabled": true,
  "backup_reminder_days": 1,
  "cover_name": "Local Inventory Tracker"
}
```

---

## 9. Fieldkit Setup

Fieldkit features apply to all tiers, but their behavior differs slightly based on the
deployment type. On Tier 1 (USB), the killswitch destroys the LUKS key. On Tier 2/3
(Docker or bare metal), it shreds files and uninstalls packages.

### 9.1 Dead man's switch

The dead man's switch requires periodic check-ins. If you miss a check-in, FieldWitness sends
a warning during the grace period. If the grace period expires without a check-in, the
killswitch fires automatically and destroys all key material and data.

Arm it:

```bash
fieldwitness fieldkit deadman arm --interval 12 --grace 1
```

This requires a check-in every 12 hours, with a 1-hour grace period.

Check in:

```bash
fieldwitness fieldkit checkin
```

You can also check in through the web UI at `/fieldkit`.

Check status:

```bash
fieldwitness status
```

The dead man's switch enforcement loop runs as a background thread inside `fieldwitness serve`,
checking every 60 seconds. It will send a webhook warning (if configured via
`deadman_warning_webhook`) during the grace period, then execute a full purge if the grace
period expires. The webhook must be a public URL -- SSRF protection blocks private/internal
IP ranges.

For cron-based enforcement outside the web server (e.g., on a headless node), use:

```bash
fieldwitness fieldkit check-deadman
```

Exit codes: 0 = not armed or not overdue, 1 = unexpected error, 2 = killswitch fired.

Disarm:

```bash
fieldwitness fieldkit deadman disarm
```

### 9.2 Geofence

If you have a USB GPS module, you can set a geographic boundary. FieldWitness will trigger the
killswitch if the device moves outside the fence.

```bash
fieldwitness fieldkit geofence set --lat 50.4501 --lon 30.5234 --radius 5000
```

Coordinates are in decimal degrees, radius in meters. Most useful on Tier 1 field devices.

### 9.3 USB whitelist

Record currently connected USB devices as the trusted baseline:

```bash
fieldwitness fieldkit usb snapshot
```

When monitoring is enabled, FieldWitness will alert (or trigger killswitch, depending on config)
if an unknown USB device is connected.

### 9.4 Tamper baseline

Record file integrity baselines for critical files:

```bash
fieldwitness fieldkit tamper baseline
```

FieldWitness monitors for unexpected changes to tracked files when tamper monitoring is enabled.

### 9.5 Killswitch

The killswitch destroys all key material and data. The destruction sequence is designed to
maximize what is gone before any interruption:

1. **Ed25519 identity keys** (most critical -- without these, signed attestations cannot
   be forged and encrypted data is unrecoverable)
2. **AES-256-GCM channel key**
3. **Flask session secret**
4. **Auth database** (user accounts)
5. **Attestation log and chain data**
6. **Temp files and audit log**
7. **Configuration**
8. **System journal entries** for the fieldwitness unit
9. **Deep forensic scrub** (see below)
10. **Self-uninstall** of the fieldwitness pip package

On Tier 1 (USB), steps 1-10 are replaced by LUKS header destruction, which is faster and
more reliable on flash media (see Section 1.6).

Trigger manually:

```bash
fieldwitness fieldkit purge --confirm CONFIRM-PURGE
```

**Deep forensic scrub (Tier 2 bare metal only).** When the killswitch fires with `ALL`
scope on a non-USB deployment, it performs a deep forensic scrub that removes traces of
FieldWitness beyond the `~/.fwmetadata/` directory:

- **Python bytecache**: removes all `__pycache__` directories and `.pyc` files for
  fieldwitness, stego, and attest from site-packages
- **pip dist-info**: removes package metadata directories that would reveal what was
  installed
- **pip download cache**: removes cached wheels and source distributions under
  `~/.cache/pip/` matching fieldwitness/stego/attest
- **Shell history**: rewrites `~/.bash_history`, `~/.zsh_history`, and fish history to
  remove all lines containing "fieldwitness"
- **Self-uninstall**: runs `pip uninstall -y fieldwitness` to remove the package from the
  virtual environment

After a full purge, the system will show minimal evidence that FieldWitness was ever installed.
Note that this is best-effort -- filesystem journal entries, inode metadata, and flash
wear-leveling remnants may still exist. For complete deniability on Tier 2, use full-disk
encryption (LUKS) and physically destroy the storage media. On Tier 1, LUKS header
destruction handles this.

---

## 10. Key Management

FieldWitness manages two separate key domains:

- **Ed25519 identity key** (`~/.fwmetadata/identity/`) -- used for signing attestations.
  This is your provenance identity.
- **AES-256-GCM channel key** (`~/.fwmetadata/stego/channel.key`) -- used for
  steganographic encoding/decoding. Shared with anyone who needs to read your
  stego messages.

These are separate security concerns and are never merged.

### 10.1 Backup

Back up keys regularly. FieldWitness warns if no backup has been taken within the
`backup_reminder_days` window (default: 7 days).

```bash
fieldwitness keys export /media/usb/fieldwitness-backup.enc
```

This creates an encrypted bundle. You will be prompted for a passphrase. Store the USB
drive physically separate from the device.

On Tier 1 (USB), key backups are critical -- if the USB stick is lost or destroyed, the
keys are gone. Back up to a second USB drive and store it in a separate location.

### 10.2 Restore

```bash
fieldwitness keys import /media/usb/fieldwitness-backup.enc
```

### 10.3 Key rotation

Rotate the identity keypair (old key is archived, not destroyed):

```bash
fieldwitness keys rotate-identity
```

Rotate the channel key:

```bash
fieldwitness keys rotate-channel
```

After rotating keys, take a fresh backup immediately. Notify all collaborators of the
new identity fingerprint so they can update their trusted-key lists.

### 10.4 Trusting collaborator keys

Import a collaborator's public key so you can verify their attestations:

```bash
fieldwitness keys trust --import /media/usb/collaborator-pubkey.pem
```

Verify the fingerprint out-of-band (in person, over a secure channel) before trusting.

---

## 11. Source Drop Box

The source drop box provides a SecureDrop-like anonymous file intake that runs inside
FieldWitness. Sources do not need a FieldWitness account -- they receive a one-time upload URL and
submit files through their browser.

### 11.1 Creating tokens

An admin creates a time-limited upload token through the web UI at `/dropbox/admin` or
through the admin panel. Each token has:

- **Label**: a human-readable name for the source (stored server-side only)
- **Expiry**: how many hours the link is valid (default: 24)
- **Max files**: maximum number of uploads allowed on this link (default: 10)

Tokens are 32-byte cryptographically random URL-safe strings. Once created, the admin
receives a URL of the form `https://<host>:<port>/dropbox/upload/<token>`.

### 11.2 Sharing URLs securely

Share the upload URL with the source over an already-secure channel:

- In-person on paper (best for high-risk sources)
- Encrypted messaging (Signal, Wire)
- Verbal dictation over a secure voice call

Never send drop box URLs over unencrypted email or SMS.

### 11.3 What happens on upload

When a source uploads files:

1. The browser computes SHA-256 fingerprints client-side (via SubtleCrypto) before upload
   so the source has a verifiable record of what they submitted
2. EXIF metadata is extracted for evidentiary fields (GPS, timestamp) and then stripped
   from the stored copy to protect source device information
3. The original bytes are attested (signed) before stripping, so the attestation hash
   matches what the source actually submitted
4. The source receives a receipt code (HMAC-derived from the file hash and token) for
   each file
5. The token usage counter increments; once max files is reached, the link stops accepting

### 11.4 Receipt verification

Sources can verify their submission was received by posting their receipt code to
`/dropbox/verify-receipt`. This returns the filename, SHA-256, and reception timestamp if
the receipt is valid.

### 11.5 Operational security for the drop box

- **No FieldWitness branding**: the upload page is a minimal HTML form with no identifying
  marks, styled generically
- **No authentication required**: the source never creates an account or reveals
  identity information
- **Token self-destruction**: tokens are deleted from the SQLite database after expiry;
  expired tokens are cleaned up on every admin page load
- **Revocation**: admins can revoke tokens immediately from `/dropbox/admin`
- **Tor compatibility**: the upload page is a self-contained HTML page with inline
  JavaScript (SubtleCrypto only) and no external resources. It works over Tor Browser
  with JavaScript enabled. No CDN, no fonts, no analytics
- **No IP logging**: FieldWitness does not log source IP addresses. Ensure your reverse proxy
  (if any) also does not log access. If running behind Tor, the source's real IP is never
  visible to the server
- **Receipt codes are deterministic**: the receipt is an HMAC of the file's SHA-256 keyed
  by the token, so the source can independently verify it corresponds to their file

If operating in a high-risk environment, consider running FieldWitness as a Tor hidden service
(`.onion` address). Configure a torrc hidden service pointing to `127.0.0.1:5000` and
share the `.onion` URL instead of a LAN address.

### 11.6 Drop box file storage

Uploaded files are stored in `~/.fwmetadata/temp/dropbox/` with filenames derived from the
SHA-256 prefix. This directory has mode 0700. Token metadata and receipts are stored in a
SQLite database at `~/.fwmetadata/auth/dropbox.db`.

---

## 12. Chain Anchoring

Chain anchoring externally proves that your attestation chain existed before a given time.
A single anchor for the chain head implicitly timestamps every record that preceded it,
because the chain is append-only with hash linkage.

### 12.1 When to anchor

Anchor your chain:

- Before sharing evidence with a third party (proves the chain existed before disclosure)
- At regular intervals (daily or weekly) to establish a timeline
- Before and after major investigations or events
- Before key rotation (locks the existing chain state)

### 12.2 Automated anchoring (RFC 3161 TSA)

If the device has internet access (even temporarily), submit the chain head to a
Timestamping Authority:

```bash
fieldwitness chain anchor --tsa https://freetsa.org/tsr
```

This sends the chain head digest to the TSA, receives a signed timestamp token, and saves
both the anchor and the TSA response as a JSON file under `~/.fwmetadata/chain/anchors/`.

The TSA token is a cryptographically signed proof from a third party that the hash existed
at the timestamp. This is legally stronger than a self-asserted timestamp.

### 12.3 Manual anchoring

Without `--tsa`, the command exports the anchor hash for manual external submission:

```bash
fieldwitness chain anchor
```

This prints a compact text block containing the chain ID, head index, record count, and
digest hash. Publish this text to any external witness:

- Tweet or public social media post (timestamped by the platform)
- Email to a trusted third party (timestamped by the mail server)
- Newspaper classified advertisement
- Blockchain transaction (e.g., Bitcoin OP_RETURN)
- Notarized document

The anchor file is saved locally regardless of whether a TSA was used.

### 12.4 Airgapped anchoring procedure

For Tier 1 (USB) and other airgapped deployments:

1. Run `fieldwitness chain anchor` on the airgapped device
2. Copy the printed anchor text to a USB drive (text file, photograph of screen, or
   paper transcription)
3. On an internet-connected device, publish the anchor text to one or more external
   witnesses
4. Document the publication (URL, screenshot, transaction ID) and store it alongside
   the USB key backup

### 12.5 Verifying anchors

To verify that the current chain state matches a previously created anchor:

```bash
fieldwitness chain verify
```

This checks all hash linkage and signatures in the chain. If the chain has been tampered
with since the anchor was created, verification will fail.

---

## 13. Cross-Organization Federation

Federation allows multiple FieldWitness instances to exchange attestation records for
collaborative investigations. Bundles are self-authenticating: each record carries the
signer's public key, so the importer can verify signatures against their trust store.

In the three-tier model, federation traffic flows:

- **Tier 1 to Tier 2**: via sneakernet (USB drive) or LAN sync
- **Tier 2 to Tier 3**: via the federation API (port 8000) over the internet
- **Tier 2 to Tier 2**: via Tier 3 relay, or directly via sneakernet

### 13.1 Exchanging trust keys

Before two organizations can exchange attestation bundles, they must trust each other's
identity keys.

On Organization A:

```bash
# Export public key
cp ~/.fwmetadata/identity/public.pem /media/usb/org-a-pubkey.pem
```

On Organization B:

```bash
# Import Org A's key and verify fingerprint
fieldwitness keys trust --import /media/usb/org-a-pubkey.pem
```

Always verify fingerprints out-of-band (in person, over a known-secure voice channel).
Repeat in both directions so each organization trusts the other.

### 13.2 Exporting attestation bundles

Export a JSON bundle containing attestation records and chain data:

```bash
fieldwitness chain export --output /media/usb/investigation-bundle.zip
```

To export only records from a specific index range:

```bash
fieldwitness chain export --start 100 --end 200 --output /media/usb/partial-bundle.zip
```

The export includes:

- Attestation records with full signatures
- Chain records with hash linkage
- The signer's public key
- A standalone verification script (requires only Python + `cryptography`)
- A human-readable README

### 13.3 Importing attestation bundles

On the receiving organization's FieldWitness instance:

- Records are imported into the local attestation log with a `federated_from` metadata tag
- Records signed by untrusted fingerprints are rejected (unless trust-on-first-use is used)
- Duplicate records (matching SHA-256) are skipped automatically

### 13.4 Delivery acknowledgments

When a bundle is imported and the receiving instance has a chain store and private key,
FieldWitness automatically creates a delivery acknowledgment record in the local chain. This
records the bundle hash, sender fingerprint, and count of records received. The
acknowledgment provides a cryptographic receipt that the bundle was delivered and ingested.

### 13.5 Selective disclosure for legal proceedings

To produce evidence for a court order or legal discovery request without revealing the
entire chain:

```bash
fieldwitness chain disclose --indices 42,43,44 --output disclosure.json
```

This exports a proof bundle where the selected records are shown in full and all other
records appear only as hashes. A third party can verify that the selected records are part
of an unbroken hash chain without seeing the contents of other records.

### 13.6 Federation on airgapped networks

All federation is designed for offline/sneakernet operation:

1. Export the bundle to a USB drive on the sending instance
2. Physically carry the USB to the receiving instance
3. Import the bundle
4. Export the delivery acknowledgment back on USB if needed

No network connectivity is required at any point. This is the expected workflow for
Tier 1 field devices.

---

## 14. Evidence Packages and Cold Archives

FieldWitness provides two export formats for preserving evidence outside the running instance.

### 14.1 Evidence packages

An evidence package is a self-contained ZIP that bundles everything needed for independent
verification of specific attested images. Use evidence packages when you need to hand off
a subset of evidence to a third party (lawyer, court, partner organization).

Contents of an evidence package:

- `images/` -- original image files
- `manifest.json` -- attestation records and chain data for those images
- `public_key.pem` -- signer's Ed25519 public key
- `verify.py` -- standalone verification script (requires only Python 3.11+ and
  `cryptography`)
- `README.txt` -- human-readable instructions

The package is self-contained. No FieldWitness installation is required to verify the evidence.
The standalone `verify.py` script checks image SHA-256 hashes against attestation records
and verifies chain hash linkage.

**When to create evidence packages:**

- Before handing evidence to a legal team
- When sharing with a partner organization that does not run FieldWitness
- For court submission (the self-contained verifier is the key feature)
- Before any action that might destroy the running instance (travel through hostile
  checkpoints, anticipated raids)

### 14.2 Cold archives

A cold archive is a full snapshot of the entire FieldWitness evidence store, designed for
long-term preservation. It follows OAIS (ISO 14721) alignment: the archive is
self-describing, includes its own verification code, and documents the cryptographic
algorithms used.

Contents of a cold archive:

- `chain/` -- raw append-only hash chain binary, state checkpoint, and anchor files
- `attestations/` -- full attest attestation log and LMDB index
- `keys/public.pem` -- signer's public key
- `keys/bundle.enc` -- encrypted key bundle (optional, password-protected)
- `keys/trusted/` -- trusted collaborator public keys
- `manifest.json` -- archive metadata and integrity hashes
- `verify.py` -- standalone verification script
- `ALGORITHMS.txt` -- documents all cryptographic algorithms and formats used (Ed25519,
  SHA-256, AES-256-GCM, Argon2id, CBOR, etc.) so the archive remains verifiable even if
  FieldWitness no longer exists
- `README.txt` -- human-readable description

To restore a cold archive on a fresh FieldWitness instance:

```bash
fieldwitness archive import <archive.zip>
```

**When to create cold archives:**

- At regular intervals (weekly or monthly) as part of your backup strategy
- Before key rotation
- Before traveling with the device
- Before anticipated risk events
- When archiving a completed investigation

### 14.3 Legal discovery workflow

For legal discovery and court proceedings:

1. Use `fieldwitness chain disclose` for selective disclosure (Section 13.5) when you must
   respond to a specific request without revealing the full chain
2. Use evidence packages for handing specific images and their attestations to counsel
3. Use cold archives when full preservation is required

All three formats include standalone verification scripts so that the receiving party does
not need to install FieldWitness.

### 14.4 Long-term archival best practices

- Store cold archives on at least two separate physical media (USB drives, optical discs)
- Keep one copy offsite (safe deposit box, trusted third party)
- Include the encrypted key bundle in the archive (set a strong passphrase and store it
  separately from the archive media)
- Write the passphrase on paper and store it in a different physical location from the
  archive media
- Periodically verify archive integrity: unzip and run `python verify.py`
- The `ALGORITHMS.txt` file documents everything needed to write a verifier from scratch,
  even decades from now

---

## 15. Cover/Duress Mode

Cover mode disguises a FieldWitness installation so that casual inspection of the device does
not immediately reveal it as a security toolkit.

### 15.1 Renaming the data directory

By default, FieldWitness stores everything under `~/.fwmetadata/`. To use an inconspicuous name,
set the `FIELDWITNESS_DATA_DIR` environment variable:

```bash
export FIELDWITNESS_DATA_DIR=~/.local/share/inventory
fieldwitness init
```

All FieldWitness commands respect this variable. Add it to the fieldwitness user's shell profile or
the systemd service file:

```ini
Environment="FIELDWITNESS_DATA_DIR=/home/fieldwitness/.local/share/inventory"
```

In Docker deployments, set this in the environment section of the compose file. The
default Docker configuration already uses `/data`.

### 15.2 Cover name for SSL certificates

Set `cover_name` in config to change the Common Name (CN) in the self-signed SSL
certificate. Without this, the certificate CN defaults to "localhost". With a cover name,
a browser inspector sees a plausible-looking certificate:

```json
{
  "cover_name": "Local Inventory Manager"
}
```

Delete `~/.fwmetadata/certs/cert.pem` and restart the server to regenerate the certificate
with the new CN.

The threat level presets (Section 4) include appropriate cover names for each level.

### 15.3 Portable USB operation (Tier 1)

The Tier 1 bootable USB is the primary cover mechanism. When the USB is not inserted, the
host laptop shows no trace of FieldWitness. The USB itself is a LUKS-encrypted partition that
reveals nothing without the passphrase.

For additional cover, the USB can be labeled generically (e.g., "DEBIAN LIVE") and the
LUKS partition does not advertise its contents.

---

## 16. Operational Security Notes

FieldWitness is a tool, not a shield. Understand what it cannot do.

### What FieldWitness does not protect against

- **Physical coercion.** If someone forces you to unlock the device or reveal passwords,
  no software can help. The killswitch is for situations where you can act before
  interception, not during.
- **Social engineering.** FieldWitness cannot prevent users from being tricked into revealing
  credentials or disabling security features.
- **Leaving the browser open.** The session timeout helps, but if someone walks up to an
  unlocked browser session, they have access. Train users to close the browser or lock the
  screen.
- **Compromised client devices.** FieldWitness secures the server. If a user's laptop has
  malware, their browser session is compromised regardless of what the server does.
- **Tier 3 relay compromise.** If the relay is seized, attestation metadata (hashes,
  signatures, timestamps) is exposed. This reveals that certain public keys attested
  certain content at certain times. It does not reveal the content itself, but it may
  reveal patterns of activity.

### Shred limitations on flash storage

The killswitch uses `shred` on Linux (3-pass overwrite + zero) for Tier 2 bare metal
deployments. On spinning disks, this is effective. On SSDs and flash storage, **it is not
reliable** because:

- Flash translation layers remap physical blocks. Overwritten data may persist on
  remapped blocks.
- Wear leveling distributes writes across the flash, meaning the original block may be
  preserved.

FieldWitness's defense against this is **cryptographic erasure**: destroy the keys first, then
the data. Even if fragments of encrypted data survive on flash, they are useless without
the keys. The killswitch destroys keys before anything else, and keys are small enough to
fit in a single flash block.

On Tier 1 (USB), the killswitch destroys the LUKS header instead of shredding individual
files, which is the only reliable approach for flash media.

### Full-disk encryption (LUKS)

Tier 1 (USB) uses LUKS by default for the persistent partition. For Tier 2 bare metal
deployments, encrypt the data partition with LUKS. This way, even if the storage is
physically seized:

- All data at rest is encrypted
- Shred limitations are irrelevant because the underlying storage is encrypted
- Power-off = data inaccessible (assuming the LUKS passphrase is strong)

For Docker deployments, use encrypted volumes or an encrypted filesystem on the host.

### Memory considerations

Python does not securely zero memory. Key material, passwords, and plaintext may persist
in process memory and swap. The mitigations in Section 6 (disable swap, disable core
dumps) reduce the window, but a memory dump of the running process would expose secrets.
This is a fundamental limitation of Python-based security tools.

---

## 17. Troubleshooting

### Health check

FieldWitness exposes a `/health` endpoint on the web UI. Hit it to verify the server is running:

```bash
# Tier 1 or bare metal
curl -k https://127.0.0.1:5000/health

# Tier 2 Docker
curl http://localhost:5000/health

# Tier 3 relay
curl http://localhost:8001/health
```

The `-k` flag skips certificate verification for self-signed certs.

### System status

```bash
fieldwitness status
```

This checks identity key, channel key, trusted keys, dead man's switch state, geofence,
chain status, and backup status. Use `--json` for machine-readable output.

### Common issues

**Web UI unreachable from LAN**

1. Check that `host` is set to `0.0.0.0` in config, not `127.0.0.1`
2. Check firewall: `sudo ufw status` -- port 5000 must be allowed
3. Check the service is running: `sudo systemctl status fieldwitness` (bare metal) or
   `docker compose ps` (Docker)
4. Check the machine's IP: `ip addr show`

**Docker container exits immediately**

Check the logs:

```bash
docker compose logs server
```

Common causes: port conflict (5000 or 8000 already in use), volume permission issues,
or missing initialization. The container runs `fieldwitness init` on first start, which requires
write access to the `/data` volume.

**Certificate warnings in browser**

Expected with self-signed certificates. Users must click through the warning. To avoid
this, distribute `~/.fwmetadata/certs/cert.pem` to client devices and install it as a
trusted certificate.

**Dead man's switch fires unexpectedly**

The switch checks every 60 seconds. If the server loses power or the service crashes and
does not restart within `interval_hours + grace_hours`, the switch will fire on the next
start. Make sure the systemd service is set to `Restart=on-failure` (bare metal) or
`restart: unless-stopped` (Docker) and the system has reliable power.

If you need to perform maintenance, disarm the switch first:

```bash
fieldwitness fieldkit deadman disarm
```

Re-arm when maintenance is complete.

**Permission errors on ~/.fwmetadata/**

The `identity/`, `auth/`, and `certs/` directories are mode 0700. If running under a
different user than the one who ran `fieldwitness init`, you will get permission denied errors.
Always run FieldWitness as the same user. In Docker, the container runs as the `fieldwitness` user
created during image build.

**Drop box tokens expire immediately**

Token expiry is checked against UTC. If the system clock is wrong, tokens may appear
expired as soon as they are created. Verify the clock:

```bash
date -u
```

On airgapped systems without NTP, set the clock manually before creating tokens:

```bash
sudo date -s "2026-04-01 12:00:00"
```

**Chain anchor TSA submission fails**

TSA submission requires network access. On Tier 1 (USB) or other airgapped devices, use
manual anchoring instead (`fieldwitness chain anchor` without `--tsa`). If the TSA URL is
unreachable, the anchor is still saved locally -- only the external timestamp token is
missing.

**SSL certificate shows wrong name**

If you set `cover_name` after the certificate was already generated, delete the old
certificate and restart:

```bash
rm ~/.fwmetadata/certs/cert.pem ~/.fwmetadata/certs/key.pem
sudo systemctl restart fieldwitness
```

**Account lockout after repeated failed logins**

After `login_lockout_attempts` (default: 5) failed login attempts, the account is locked
for `login_lockout_minutes` (default: 15) minutes. Wait for the lockout to expire, or
restart the server to clear lockout state.

**Evidence package verify.py fails**

The standalone verification script requires Python 3.11+ and the `cryptography` package.
Install it in a fresh virtualenv on the verifying machine:

```bash
python3 -m venv verify-env
source verify-env/bin/activate
pip install cryptography
python verify.py
```

**Tier 1 USB fails to boot on a laptop**

- Ensure Secure Boot is disabled in BIOS/UEFI settings
- Set the USB drive as the first boot device (or use the one-time boot menu, usually F12)
- If the laptop has only UEFI boot (no legacy/CSM), the live image should still work --
  it is built as a hybrid ISO -- but some very old or very locked-down firmware may not
  cooperate

**Kubernetes pod stuck in CrashLoopBackOff**

Check logs with `kubectl logs -n fieldwitness <pod-name>`. Common cause: the PersistentVolumeClaim
is not bound. Verify with `kubectl get pvc -n fieldwitness`. The server needs 10Gi and the relay
needs 5Gi.

---

## Appendix A: Legacy Raspberry Pi Deployment

The Raspberry Pi was the original deployment target for FieldWitness. It remains a viable
option for fixed installations (e.g., a permanently installed newsroom server that does
not need to be portable). The three-tier model supersedes the RPi as the primary
deployment for most use cases because:

- Tier 1 (USB) is more portable and amnesic than a Pi with an SD card
- Tier 2 (Docker) is more maintainable than a bare-metal Pi install
- The Pi's ARM architecture complicates dependency builds (scipy, Pillow)

If you still want to use a Raspberry Pi for a fixed Tier 2 server:

- Use Raspberry Pi 4 Model B, 4 GB RAM minimum
- Use Raspberry Pi OS Lite (64-bit, Bookworm)
- Install system dependencies: `python3 python3-pip python3-venv python3-dev libjpeg-dev
  libjpeg62-turbo-dev zlib1g-dev libffi-dev libssl-dev gfortran libopenblas-dev`
- Follow the Tier 2 bare metal instructions (systemd service, hardening) in this guide
- For GPIO hardware killswitch support, install the `rpi` extra:
  `pip install "fieldwitness[rpi]"` (includes `gpiozero`)
- Wire a momentary push button between GPIO 17 and 3.3V with a 10k pull-down to GND.
  Default: 5-second hold to trigger. Configurable via `gpio_killswitch_hold_seconds`

Note: scipy may fail to build on low-memory Pis. Use `pip install --prefer-binary scipy`
or temporarily enable swap during the build.