# FieldWitness -- Claude Code Project Guide FieldWitness (FieldWitness) is an offline-first provenance attestation and gossip federation system for journalists, NGOs, and at-risk organizations. It establishes cryptographic chain-of-custody over evidence in airgapped and resource-constrained environments, syncs attestations across organizational boundaries via a gossip protocol with Merkle consistency proofs, and produces court-ready evidence packages with standalone verification. Steganography (Stego) and provenance attestation (Attest) are included as subpackages in this monorepo. Version 0.3.0 · Python >=3.11 · GPL-3.0 License ## Quick commands ```bash # Development install (single command -- stego and attest are inlined subpackages) pip install -e ".[dev]" pytest # Run tests black src/ tests/ frontends/ # Format code ruff check src/ tests/ frontends/ --fix # Lint mypy src/ # Type check ``` ### Pip extras `stego-dct`, `stego-audio`, `stego-compression`, `attest`, `cli`, `web`, `api`, `fieldkit`, `federation`, `rpi`, `all`, `dev` ## Architecture ``` src/fieldwitness/ Core library __init__.py Package init, __version__ (0.3.0) _availability.py Runtime checks for optional subpackages (has_stego, has_attest) api.py Optional unified FastAPI app (uvicorn fieldwitness.api:app) audit.py Append-only JSON-lines audit log (~/.fwmetadata/audit.jsonl) cli.py Click CLI entry point (fieldwitness command) paths.py All ~/.fwmetadata/* path constants (single source of truth, lazy resolution) config.py Unified config loader (FieldWitnessConfig dataclass + JSON) exceptions.py FieldWitnessError, ChainError, ChainIntegrityError, ChainAppendError, KeystoreError metadata.py Extract-then-strip EXIF pipeline with field classification evidence.py Self-contained evidence package export (ZIP with verify.py) archive.py Cold archive export for long-term preservation (OAIS-aligned) stego/ Steganography engine (inlined from fieldwitness.stego v4.3.0) encode.py / decode.py Core encode/decode API generate.py Cover image generation crypto.py AES-256-GCM encryption, channel fingerprints channel.py Channel key derivation + management steganography.py LSB steganography core dct_steganography.py DCT-domain JPEG steganography spread_steganography.py MDCT spread-spectrum audio steganography audio_steganography.py Audio stego pipeline video_steganography.py Video stego pipeline backends/ Pluggable stego backend registry (lsb, dct, protocol) batch.py Batch operations compression.py Payload compression (zstd, lz4) steganalysis.py Detection resistance analysis validation.py Input validation models.py Data models constants.py Magic bytes, version constants, AUDIO_ENABLED, VIDEO_ENABLED cli.py Stego-specific CLI commands api.py / api_auth.py Stego REST API + auth carrier_tracker.py Carrier image reuse tracking (warns on reuse) platform_presets.py Social-media-aware encoding presets image_utils.py / audio_utils.py / video_utils.py keygen.py / qr_utils.py / recovery.py / debug.py / utils.py attest/ Provenance attestation engine (inlined from fieldwitness.attest v0.2.0) attestation.py Core attestation creation + EXIF extraction verification.py Attestation verification crypto.py Ed25519 signing hashing.py Perceptual + cryptographic hashing (ImageHashes) embed.py Attestation embedding in images merkle.py Merkle tree + consistency/inclusion proofs binlog.py Binary attestation log lmdb_store.py LMDB-backed trust store storage.py Attestation storage abstraction (LocalStorage) federation.py GossipNode, HttpTransport, PeerInfo, SyncStatus peer_store.py SQLite-backed peer persistence for federation models.py Attestation, AttestationRecord, ImageHashes, Identity exceptions.py AttestError, AttestationError, VerificationError, FederationError cli.py Attest-specific CLI commands api.py Attest REST API + federation endpoints federation/ Federated attestation chain system chain.py ChainStore -- append-only hash chain with key rotation/recovery/delivery-ack entropy.py Entropy source for chain seeds (sys_uptime, fs_snapshot, proc_entropy, boot_id) models.py AttestationChainRecord, ChainState, EntropyWitnesses (frozen dataclasses) serialization.py CBOR canonical encoding + compute_record_hash + serialize/deserialize anchors.py RFC 3161 timestamps + manual chain anchors exchange.py Cross-org attestation bundle export/import (JSON bundles) keystore/ Unified key management manager.py KeystoreManager -- owns all key material (channel + identity + trust store + backup) models.py IdentityInfo, KeystoreStatus, RotationResult dataclasses export.py Encrypted key bundle export/import (SOOBNDL format) c2pa_bridge/ C2PA (Content Authenticity) bridge __init__.py Public API: export, has_c2pa() cert.py Self-signed X.509 cert generation from Ed25519 key export.py AttestationRecord -> C2PA manifest vendor_assertions.py org.fieldwitness.* assertion schemas fieldkit/ Field security features killswitch.py Emergency data destruction (PurgeScope.KEYS_ONLY | ALL, deep forensic scrub) deadman.py Dead man's switch (webhook warning + auto-purge) tamper.py File integrity monitoring (baseline snapshots) usb_monitor.py USB device whitelist (Linux/pyudev) geofence.py GPS boundary enforcement (gpsd integration) frontends/web/ Unified Flask web UI app.py App factory (create_app()), ~36k -- mounts all blueprints auth.py SQLite3 multi-user auth with lockout + rate limiting temp_storage.py File-based temp storage with expiry subprocess_stego.py Crash-safe subprocess isolation for stego stego_worker.py Background stego processing stego_routes.py Stego route helpers (~87k) ssl_utils.py Self-signed HTTPS cert generation (cover_name support) blueprints/ stego.py /encode, /decode, /generate attest.py /attest, /verify (~25k -- handles images + arbitrary files) fieldkit.py /fieldkit/* (killswitch, deadman, status) keys.py /keys/* (unified key management, trust store) admin.py /admin/* (user management) dropbox.py /dropbox/* (token-gated anonymous source upload, ~13k) federation.py /federation/* (peer status dashboard, peer add/remove) templates/ dropbox/admin.html Drop box admin panel federation/status.html Federation peer dashboard frontends/cli/ CLI package init (main entry point is src/fieldwitness/cli.py) deploy/ Deployment artifacts docker/ Dockerfile (multi-stage: builder, relay, server) + docker-compose.yml kubernetes/ namespace.yaml, server-deployment.yaml, relay-deployment.yaml live-usb/ build.sh + config/ for Debian Live USB (Tier 1) config-presets/ low-threat.json, medium-threat.json, high-threat.json, critical-threat.json docs/ Documentation deployment.md Three-tier deployment guide (~1500 lines) federation.md Gossip protocol, peer setup, offline bundles, CLI commands evidence-guide.md Evidence packages, cold archives, selective disclosure source-dropbox.md Source drop box setup: tokens, EXIF pipeline, receipts architecture/ federation.md System architecture overview (threat model, layers, key domains) chain-format.md Chain record spec (CBOR, entropy witnesses, serialization) export-bundle.md Export bundle spec (FIELDWITNESSX1 binary format, envelope encryption) federation-protocol.md Federation server protocol (CT-inspired, gossip, storage tiers) security/ threat-model.md Adversary model, guarantees, non-guarantees, known limitations planning/ why-fieldwitness.md Problem statement, positioning, scenarios c2pa-integration.md C2PA bridge architecture, concept mapping, implementation phases packaging-strategy.md Hosted demo, standalone binary, mobile app, onboarding flow gtm-feasibility.md Phased plan for credibility, funding, field testing training/ reporter-quickstart.md One-page reporter quick-start for Tier 1 USB (print + laminate) reporter-field-guide.md Comprehensive reporter guide: attest, stego, killswitch, evidence emergency-card.md Laminated wallet card: emergency destruction reference admin-reference.md Admin CLI cheat sheet, hardening checklist, troubleshooting admin-operations-guide.md Full admin operations: users, dropbox, federation, incidents ``` ## Three-tier deployment model ``` Tier 1: Field Device Tier 2: Org Server Tier 3: Federation Relay (Bootable USB + laptop) (Docker on mini PC / VPS) (Docker on VPS) Amnesic, LUKS-encrypted Persistent, full features Attestation sync only Reporter in the field Newsroom / NGO office Friendly jurisdiction ``` - Tier 1 <-> Tier 2: sneakernet (USB drive) or LAN - Tier 2 <-> Tier 3: federation API (port 8000) over internet - Tier 2 <-> Tier 2: via Tier 3 relay, or directly via sneakernet ## Dependency model Stego and Attest are inlined subpackages, not separate pip packages: - `from fieldwitness.stego import encode` for steganography - `from fieldwitness.attest import Attestation` for provenance attestation - Never `import fieldwitness.stego` or `import fieldwitness.attest` directly - `_availability.py` provides `has_stego()` / `has_attest()` for graceful degradation when optional extras are not installed ## Key design decisions - **Two key domains, never merged**: Stego AES-256-GCM (derived from factors) and Attest Ed25519 (signing identity) are separate security concerns - **Extract-then-strip model**: Stego strips all EXIF (carrier is vessel); attestation extracts evidentiary EXIF (GPS, timestamp) then strips dangerous fields (device serial) - **subprocess_stego.py copies verbatim** from fieldwitness.stego -- it's a crash-safety boundary - **All state under ~/.fwmetadata/** -- one directory to back up, one to destroy. `FIELDWITNESS_DATA_DIR` env var relocates everything (cover mode, USB mode) - **Offline-first**: All static assets vendored, no CDN. pip wheels bundled for airgap install - **Flask blueprints**: stego, attest, fieldkit, keys, admin, dropbox, federation - **Flask-WTF**: CSRF protection on all form endpoints; drop box is CSRF-exempt (sources don't have sessions) - **Client-side SHA-256**: Drop box upload page uses SubtleCrypto for pre-upload hashing - **Waitress**: Production WSGI server (replaces dev-only Flask server) - **FastAPI option**: `fieldwitness.api` provides a REST API alternative to the Flask web UI - **Pluggable backends**: Stego backends (LSB, DCT) registered via `backends/registry.py` - **ImageHashes generalized**: phash/dhash now optional, enabling non-image attestation - **Lazy path resolution**: All paths in paths.py resolve lazily via `__getattr__` from `BASE_DIR` so that runtime overrides (--data-dir, FIELDWITNESS_DATA_DIR) propagate correctly - **Two-way federation**: Delivery acknowledgment records (`fieldwitness/delivery-ack-v1`) enable handshake proof - **Chain record types** (in federation/chain.py): - `CONTENT_TYPE_KEY_ROTATION = "fieldwitness/key-rotation-v1"` -- signed by OLD key - `CONTENT_TYPE_KEY_RECOVERY = "fieldwitness/key-recovery-v1"` -- signed by NEW key - `CONTENT_TYPE_DELIVERY_ACK = "fieldwitness/delivery-ack-v1"` -- signed by receiver - **Gossip federation** (attest/federation.py): GossipNode with async peer sync, consistency proofs, HttpTransport over aiohttp. PeerStore for SQLite-backed persistence - **Threat level presets**: deploy/config-presets/ with low/medium/high/critical configs - **Selective disclosure**: Chain records can be exported with non-selected records redacted to hashes only (for legal discovery) - **Evidence packages**: Self-contained ZIPs with verify.py that needs only Python + cryptography - **Cold archives**: OAIS-aligned full-state export with ALGORITHMS.txt - **Transport-aware stego**: --transport whatsapp|signal|telegram auto-selects DCT/JPEG and pre-resizes carrier for platform survival ## Data directory layout (`~/.fwmetadata/`) ``` ~/.fwmetadata/ config.json Unified configuration (FieldWitnessConfig dataclass) audit.jsonl Append-only audit trail (JSON-lines) carrier_history.json Carrier reuse tracking database identity/ Ed25519 keypair (private.pem, public.pem, identity.meta.json) archived/ Timestamped old keypairs from rotations stego/ Channel key (channel.key) archived/ Timestamped old channel keys from rotations attestations/ Attest attestation store log.bin Binary attestation log index/ LMDB index peers.json Legacy peer file federation/ Federation state peers.db SQLite peer + sync history chain/ Hash chain (chain.bin, state.cbor) anchors/ External timestamp anchors (JSON files) auth/ Web UI auth databases fieldwitness.db User accounts dropbox.db Drop box tokens + receipts certs/ Self-signed TLS certificates (cert.pem, key.pem) fieldkit/ Fieldkit state deadman.json Dead man's switch timer tamper/baseline.json File integrity baselines usb/whitelist.json USB device whitelist geofence.json GPS boundary config temp/ Ephemeral file storage dropbox/ Source drop box submissions instance/ Flask instance (sessions, .secret_key) trusted_keys/ Collaborator Ed25519 public keys (trust store) / Per-key directory (public.pem + meta.json) last_backup.json Backup timestamp tracking ``` ## Code conventions Black (100-char), Ruff (E, F, I, N, W, UP), mypy (strict, ignore_missing_imports), imperative commit messages. ## Testing ```bash pytest # All tests with coverage pytest tests/test_chain.py # Chain-specific ``` Test files: `test_chain.py`, `test_chain_security.py`, `test_deadman_enforcement.py`, `test_key_rotation.py`, `test_killswitch.py`, `test_serialization.py`, `test_stego_audio.py`, `test_stego.py`, `test_attest_hashing.py`