fieldwitness/CLAUDE.md

FieldWitness -- Claude Code Project Guide

FieldWitness (FieldWitness) is an offline-first provenance attestation and gossip federation system for journalists, NGOs, and at-risk organizations. It establishes cryptographic chain-of-custody over evidence in airgapped and resource-constrained environments, syncs attestations across organizational boundaries via a gossip protocol with Merkle consistency proofs, and produces court-ready evidence packages with standalone verification. Steganography (Stego) and provenance attestation (Attest) are included as subpackages in this monorepo.

Version 0.3.0 · Python >=3.11 · GPL-3.0 License

Quick commands

# Development install (single command -- stego and attest are inlined subpackages)
pip install -e ".[dev]"

pytest                                       # Run tests
black src/ tests/ frontends/                 # Format code
ruff check src/ tests/ frontends/ --fix      # Lint
mypy src/                                    # Type check

Pip extras

stego-dct, stego-audio, stego-compression, attest, cli, web, api, fieldkit, federation, rpi, all, dev

Architecture

src/fieldwitness/                Core library
  __init__.py                Package init, __version__ (0.3.0)
  _availability.py           Runtime checks for optional subpackages (has_stego, has_attest)
  api.py                     Optional unified FastAPI app (uvicorn fieldwitness.api:app)
  audit.py                   Append-only JSON-lines audit log (~/.fwmetadata/audit.jsonl)
  cli.py                     Click CLI entry point (fieldwitness command)
  paths.py                   All ~/.fwmetadata/* path constants (single source of truth, lazy resolution)
  config.py                  Unified config loader (FieldWitnessConfig dataclass + JSON)
  exceptions.py              FieldWitnessError, ChainError, ChainIntegrityError, ChainAppendError, KeystoreError
  metadata.py                Extract-then-strip EXIF pipeline with field classification
  evidence.py                Self-contained evidence package export (ZIP with verify.py)
  archive.py                 Cold archive export for long-term preservation (OAIS-aligned)

  stego/                Steganography engine (inlined from fieldwitness.stego v4.3.0)
    encode.py / decode.py      Core encode/decode API
    generate.py                Cover image generation
    crypto.py                  AES-256-GCM encryption, channel fingerprints
    channel.py                 Channel key derivation + management
    steganography.py           LSB steganography core
    dct_steganography.py       DCT-domain JPEG steganography
    spread_steganography.py    MDCT spread-spectrum audio steganography
    audio_steganography.py     Audio stego pipeline
    video_steganography.py     Video stego pipeline
    backends/                  Pluggable stego backend registry (lsb, dct, protocol)
    batch.py                   Batch operations
    compression.py             Payload compression (zstd, lz4)
    steganalysis.py            Detection resistance analysis
    validation.py              Input validation
    models.py                  Data models
    constants.py               Magic bytes, version constants, AUDIO_ENABLED, VIDEO_ENABLED
    cli.py                     Stego-specific CLI commands
    api.py / api_auth.py       Stego REST API + auth
    carrier_tracker.py         Carrier image reuse tracking (warns on reuse)
    platform_presets.py        Social-media-aware encoding presets
    image_utils.py / audio_utils.py / video_utils.py
    keygen.py / qr_utils.py / recovery.py / debug.py / utils.py

  attest/                 Provenance attestation engine (inlined from fieldwitness.attest v0.1.0)
    attestation.py             Core attestation creation + EXIF extraction
    verification.py            Attestation verification
    crypto.py                  Ed25519 signing
    hashing.py                 Perceptual + cryptographic hashing (ImageHashes)
    embed.py                   Attestation embedding in images
    merkle.py                  Merkle tree + consistency/inclusion proofs
    binlog.py                  Binary attestation log
    lmdb_store.py              LMDB-backed trust store
    storage.py                 Attestation storage abstraction (LocalStorage)
    federation.py              GossipNode, HttpTransport, PeerInfo, SyncStatus
    peer_store.py              SQLite-backed peer persistence for federation
    models.py                  Attestation, AttestationRecord, ImageHashes, Identity
    exceptions.py              AttestError, AttestationError, VerificationError, FederationError
    cli.py                     Attest-specific CLI commands
    api.py                     Attest REST API + federation endpoints

  federation/              Federated attestation chain system
    chain.py                   ChainStore -- append-only hash chain with key rotation/recovery/delivery-ack
    entropy.py                 Entropy source for chain seeds (sys_uptime, fs_snapshot, proc_entropy, boot_id)
    models.py                  AttestationChainRecord, ChainState, EntropyWitnesses (frozen dataclasses)
    serialization.py           CBOR canonical encoding + compute_record_hash + serialize/deserialize
    anchors.py                 RFC 3161 timestamps + manual chain anchors
    exchange.py                Cross-org attestation bundle export/import (JSON bundles)

  keystore/                Unified key management
    manager.py                 KeystoreManager -- owns all key material (channel + identity + trust store + backup)
    models.py                  IdentityInfo, KeystoreStatus, RotationResult dataclasses
    export.py                  Encrypted key bundle export/import (SOOBNDL format)

  fieldkit/                Field security features
    killswitch.py              Emergency data destruction (PurgeScope.KEYS_ONLY | ALL, deep forensic scrub)
    deadman.py                 Dead man's switch (webhook warning + auto-purge)
    tamper.py                  File integrity monitoring (baseline snapshots)
    usb_monitor.py             USB device whitelist (Linux/pyudev)
    geofence.py                GPS boundary enforcement (gpsd integration)

frontends/web/             Unified Flask web UI
  app.py                     App factory (create_app()), ~36k -- mounts all blueprints
  auth.py                    SQLite3 multi-user auth with lockout + rate limiting
  temp_storage.py            File-based temp storage with expiry
  subprocess_stego.py        Crash-safe subprocess isolation for stego
  stego_worker.py            Background stego processing
  stego_routes.py            Stego route helpers (~87k)
  ssl_utils.py               Self-signed HTTPS cert generation (cover_name support)
  blueprints/
    stego.py                   /encode, /decode, /generate
    attest.py                  /attest, /verify (~25k -- handles images + arbitrary files)
    fieldkit.py                /fieldkit/* (killswitch, deadman, status)
    keys.py                    /keys/* (unified key management, trust store)
    admin.py                   /admin/* (user management)
    dropbox.py                 /dropbox/* (token-gated anonymous source upload, ~13k)
    federation.py              /federation/* (peer status dashboard, peer add/remove)
  templates/
    dropbox/admin.html         Drop box admin panel
    federation/status.html     Federation peer dashboard

frontends/cli/             CLI package init (main entry point is src/fieldwitness/cli.py)

deploy/                    Deployment artifacts
  docker/                    Dockerfile (multi-stage: builder, relay, server) + docker-compose.yml
  kubernetes/                namespace.yaml, server-deployment.yaml, relay-deployment.yaml
  live-usb/                  build.sh + config/ for Debian Live USB (Tier 1)
  config-presets/            low-threat.json, medium-threat.json, high-threat.json, critical-threat.json

docs/                      Documentation
  deployment.md              Three-tier deployment guide (~1500 lines)
  federation.md              Gossip protocol, peer setup, offline bundles, CLI commands
  evidence-guide.md          Evidence packages, cold archives, selective disclosure
  source-dropbox.md          Source drop box setup: tokens, EXIF pipeline, receipts
  architecture/
    federation.md              System architecture overview (threat model, layers, key domains)
    chain-format.md            Chain record spec (CBOR, entropy witnesses, serialization)
    export-bundle.md           Export bundle spec (FIELDWITNESSX1 binary format, envelope encryption)
    federation-protocol.md     Federation server protocol (CT-inspired, gossip, storage tiers)
  training/
    reporter-quickstart.md     One-page reporter quick-start for Tier 1 USB (print + laminate)
    reporter-field-guide.md    Comprehensive reporter guide: attest, stego, killswitch, evidence
    emergency-card.md          Laminated wallet card: emergency destruction reference
    admin-reference.md         Admin CLI cheat sheet, hardening checklist, troubleshooting
    admin-operations-guide.md  Full admin operations: users, dropbox, federation, incidents

Three-tier deployment model

Tier 1: Field Device        Tier 2: Org Server          Tier 3: Federation Relay
(Bootable USB + laptop)     (Docker on mini PC / VPS)   (Docker on VPS)
Amnesic, LUKS-encrypted     Persistent, full features   Attestation sync only
Reporter in the field       Newsroom / NGO office       Friendly jurisdiction

• Tier 1 <-> Tier 2: sneakernet (USB drive) or LAN
• Tier 2 <-> Tier 3: federation API (port 8000) over internet
• Tier 2 <-> Tier 2: via Tier 3 relay, or directly via sneakernet

Dependency model

Stego and Attest are inlined subpackages, not separate pip packages:

• from fieldwitness.stego import encode for steganography
• from fieldwitness.attest import Attestation for provenance attestation
• Never import fieldwitness.stego or import fieldwitness.attest directly
• _availability.py provides has_stego() / has_attest() for graceful degradation when optional extras are not installed

Key design decisions

• Two key domains, never merged: Stego AES-256-GCM (derived from factors) and Attest Ed25519 (signing identity) are separate security concerns
• Extract-then-strip model: Stego strips all EXIF (carrier is vessel); attestation extracts evidentiary EXIF (GPS, timestamp) then strips dangerous fields (device serial)
• subprocess_stego.py copies verbatim from fieldwitness.stego -- it's a crash-safety boundary
• All state under ~/.fwmetadata/ -- one directory to back up, one to destroy. FIELDWITNESS_DATA_DIR env var relocates everything (cover mode, USB mode)
• Offline-first: All static assets vendored, no CDN. pip wheels bundled for airgap install
• Flask blueprints: stego, attest, fieldkit, keys, admin, dropbox, federation
• Flask-WTF: CSRF protection on all form endpoints; drop box is CSRF-exempt (sources don't have sessions)
• Client-side SHA-256: Drop box upload page uses SubtleCrypto for pre-upload hashing
• Waitress: Production WSGI server (replaces dev-only Flask server)
• FastAPI option: fieldwitness.api provides a REST API alternative to the Flask web UI
• Pluggable backends: Stego backends (LSB, DCT) registered via backends/registry.py
• ImageHashes generalized: phash/dhash now optional, enabling non-image attestation
• Lazy path resolution: All paths in paths.py resolve lazily via __getattr__ from BASE_DIR so that runtime overrides (--data-dir, FIELDWITNESS_DATA_DIR) propagate correctly
• Two-way federation: Delivery acknowledgment records (fieldwitness/delivery-ack-v1) enable handshake proof
• Chain record types (in federation/chain.py):
  • CONTENT_TYPE_KEY_ROTATION = "fieldwitness/key-rotation-v1" -- signed by OLD key
  • CONTENT_TYPE_KEY_RECOVERY = "fieldwitness/key-recovery-v1" -- signed by NEW key
  • CONTENT_TYPE_DELIVERY_ACK = "fieldwitness/delivery-ack-v1" -- signed by receiver
• Gossip federation (attest/federation.py): GossipNode with async peer sync, consistency proofs, HttpTransport over aiohttp. PeerStore for SQLite-backed persistence
• Threat level presets: deploy/config-presets/ with low/medium/high/critical configs
• Selective disclosure: Chain records can be exported with non-selected records redacted to hashes only (for legal discovery)
• Evidence packages: Self-contained ZIPs with verify.py that needs only Python + cryptography
• Cold archives: OAIS-aligned full-state export with ALGORITHMS.txt
• Transport-aware stego: --transport whatsapp|signal|telegram auto-selects DCT/JPEG and pre-resizes carrier for platform survival

Data directory layout (~/.fwmetadata/)

~/.fwmetadata/
  config.json              Unified configuration (FieldWitnessConfig dataclass)
  audit.jsonl              Append-only audit trail (JSON-lines)
  carrier_history.json     Carrier reuse tracking database
  identity/                Ed25519 keypair (private.pem, public.pem, identity.meta.json)
    archived/              Timestamped old keypairs from rotations
  stego/                Channel key (channel.key)
    archived/              Timestamped old channel keys from rotations
  attestations/            Attest attestation store
    log.bin                  Binary attestation log
    index/                   LMDB index
    peers.json               Legacy peer file
    federation/              Federation state
      peers.db               SQLite peer + sync history
  chain/                   Hash chain (chain.bin, state.cbor)
    anchors/                 External timestamp anchors (JSON files)
  auth/                    Web UI auth databases
    fieldwitness.db                User accounts
    dropbox.db               Drop box tokens + receipts
  certs/                   Self-signed TLS certificates (cert.pem, key.pem)
  fieldkit/                Fieldkit state
    deadman.json             Dead man's switch timer
    tamper/baseline.json     File integrity baselines
    usb/whitelist.json       USB device whitelist
    geofence.json            GPS boundary config
  temp/                    Ephemeral file storage
    dropbox/                 Source drop box submissions
  instance/                Flask instance (sessions, .secret_key)
  trusted_keys/            Collaborator Ed25519 public keys (trust store)
    <fingerprint>/           Per-key directory (public.pem + meta.json)
  last_backup.json         Backup timestamp tracking

Code conventions

Black (100-char), Ruff (E, F, I, N, W, UP), mypy (strict, ignore_missing_imports), imperative commit messages.

Testing

pytest                    # All tests with coverage
pytest tests/test_chain.py  # Chain-specific

Test files: test_chain.py, test_chain_security.py, test_deadman_enforcement.py, test_key_rotation.py, test_killswitch.py, test_serialization.py, test_stego_audio.py, test_stego.py, test_attest_hashing.py