490f9d4a1d
Complete project rebrand for better positioning in the press freedom and digital security space. FieldWitness communicates both field deployment and evidence testimony — appropriate for the target audience of journalists, NGOs, and human rights organizations. Rename mapping: - soosef → fieldwitness (package, CLI, all imports) - soosef.stegasoo → fieldwitness.stego - soosef.verisoo → fieldwitness.attest - ~/.soosef/ → ~/.fwmetadata/ (innocuous data dir name) - SOOSEF_DATA_DIR → FIELDWITNESS_DATA_DIR - SoosefConfig → FieldWitnessConfig - SoosefError → FieldWitnessError Also includes: - License switch from MIT to GPL-3.0 - C2PA bridge module (Phase 0-2 MVP): cert.py, export.py, vendor_assertions.py - README repositioned to lead with provenance/federation, stego backgrounded - Threat model skeleton at docs/security/threat-model.md - Planning docs: docs/planning/c2pa-integration.md, docs/planning/gtm-feasibility.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
262 lines
15 KiB
Markdown
262 lines
15 KiB
Markdown
# FieldWitness -- Claude Code Project Guide
|
|
|
|
FieldWitness (FieldWitness) is an offline-first provenance attestation and gossip
|
|
federation system for journalists, NGOs, and at-risk organizations. It establishes
|
|
cryptographic chain-of-custody over evidence in airgapped and resource-constrained
|
|
environments, syncs attestations across organizational boundaries via a gossip protocol
|
|
with Merkle consistency proofs, and produces court-ready evidence packages with standalone
|
|
verification. Steganography (Stego) and provenance attestation (Attest) are included
|
|
as subpackages in this monorepo.
|
|
|
|
Version 0.2.0 · Python >=3.11 · GPL-3.0 License
|
|
|
|
## Quick commands
|
|
|
|
```bash
|
|
# Development install (single command -- stego and attest are inlined subpackages)
|
|
pip install -e ".[dev]"
|
|
|
|
pytest # Run tests
|
|
black src/ tests/ frontends/ # Format code
|
|
ruff check src/ tests/ frontends/ --fix # Lint
|
|
mypy src/ # Type check
|
|
```
|
|
|
|
### Pip extras
|
|
|
|
`stego-dct`, `stego-audio`, `stego-compression`, `attest`, `cli`, `web`, `api`,
|
|
`fieldkit`, `federation`, `rpi`, `all`, `dev`
|
|
|
|
## Architecture
|
|
|
|
```
|
|
src/fieldwitness/ Core library
|
|
__init__.py Package init, __version__ (0.2.0)
|
|
_availability.py Runtime checks for optional subpackages (has_stego, has_attest)
|
|
api.py Optional unified FastAPI app (uvicorn fieldwitness.api:app)
|
|
audit.py Append-only JSON-lines audit log (~/.fieldwitness/audit.jsonl)
|
|
cli.py Click CLI entry point (fieldwitness command)
|
|
paths.py All ~/.fieldwitness/* path constants (single source of truth, lazy resolution)
|
|
config.py Unified config loader (FieldWitnessConfig dataclass + JSON)
|
|
exceptions.py FieldWitnessError, ChainError, ChainIntegrityError, ChainAppendError, KeystoreError
|
|
metadata.py Extract-then-strip EXIF pipeline with field classification
|
|
evidence.py Self-contained evidence package export (ZIP with verify.py)
|
|
archive.py Cold archive export for long-term preservation (OAIS-aligned)
|
|
|
|
stego/ Steganography engine (inlined from fieldwitness.stego v4.3.0)
|
|
encode.py / decode.py Core encode/decode API
|
|
generate.py Cover image generation
|
|
crypto.py AES-256-GCM encryption, channel fingerprints
|
|
channel.py Channel key derivation + management
|
|
steganography.py LSB steganography core
|
|
dct_steganography.py DCT-domain JPEG steganography
|
|
spread_steganography.py MDCT spread-spectrum audio steganography
|
|
audio_steganography.py Audio stego pipeline
|
|
video_steganography.py Video stego pipeline
|
|
backends/ Pluggable stego backend registry (lsb, dct, protocol)
|
|
batch.py Batch operations
|
|
compression.py Payload compression (zstd, lz4)
|
|
steganalysis.py Detection resistance analysis
|
|
validation.py Input validation
|
|
models.py Data models
|
|
constants.py Magic bytes, version constants, AUDIO_ENABLED, VIDEO_ENABLED
|
|
cli.py Stego-specific CLI commands
|
|
api.py / api_auth.py Stego REST API + auth
|
|
carrier_tracker.py Carrier image reuse tracking (warns on reuse)
|
|
platform_presets.py Social-media-aware encoding presets
|
|
image_utils.py / audio_utils.py / video_utils.py
|
|
keygen.py / qr_utils.py / recovery.py / debug.py / utils.py
|
|
|
|
attest/ Provenance attestation engine (inlined from fieldwitness.attest v0.1.0)
|
|
attestation.py Core attestation creation + EXIF extraction
|
|
verification.py Attestation verification
|
|
crypto.py Ed25519 signing
|
|
hashing.py Perceptual + cryptographic hashing (ImageHashes)
|
|
embed.py Attestation embedding in images
|
|
merkle.py Merkle tree + consistency/inclusion proofs
|
|
binlog.py Binary attestation log
|
|
lmdb_store.py LMDB-backed trust store
|
|
storage.py Attestation storage abstraction (LocalStorage)
|
|
federation.py GossipNode, HttpTransport, PeerInfo, SyncStatus
|
|
peer_store.py SQLite-backed peer persistence for federation
|
|
models.py Attestation, AttestationRecord, ImageHashes, Identity
|
|
exceptions.py AttestError, AttestationError, VerificationError, FederationError
|
|
cli.py Attest-specific CLI commands
|
|
api.py Attest REST API + federation endpoints
|
|
|
|
federation/ Federated attestation chain system
|
|
chain.py ChainStore -- append-only hash chain with key rotation/recovery/delivery-ack
|
|
entropy.py Entropy source for chain seeds (sys_uptime, fs_snapshot, proc_entropy, boot_id)
|
|
models.py AttestationChainRecord, ChainState, EntropyWitnesses (frozen dataclasses)
|
|
serialization.py CBOR canonical encoding + compute_record_hash + serialize/deserialize
|
|
anchors.py RFC 3161 timestamps + manual chain anchors
|
|
exchange.py Cross-org attestation bundle export/import (JSON bundles)
|
|
|
|
keystore/ Unified key management
|
|
manager.py KeystoreManager -- owns all key material (channel + identity + trust store + backup)
|
|
models.py IdentityInfo, KeystoreStatus, RotationResult dataclasses
|
|
export.py Encrypted key bundle export/import (SOOBNDL format)
|
|
|
|
fieldkit/ Field security features
|
|
killswitch.py Emergency data destruction (PurgeScope.KEYS_ONLY | ALL, deep forensic scrub)
|
|
deadman.py Dead man's switch (webhook warning + auto-purge)
|
|
tamper.py File integrity monitoring (baseline snapshots)
|
|
usb_monitor.py USB device whitelist (Linux/pyudev)
|
|
geofence.py GPS boundary enforcement (gpsd integration)
|
|
|
|
frontends/web/ Unified Flask web UI
|
|
app.py App factory (create_app()), ~36k -- mounts all blueprints
|
|
auth.py SQLite3 multi-user auth with lockout + rate limiting
|
|
temp_storage.py File-based temp storage with expiry
|
|
subprocess_stego.py Crash-safe subprocess isolation for stego
|
|
stego_worker.py Background stego processing
|
|
stego_routes.py Stego route helpers (~87k)
|
|
ssl_utils.py Self-signed HTTPS cert generation (cover_name support)
|
|
blueprints/
|
|
stego.py /encode, /decode, /generate
|
|
attest.py /attest, /verify (~25k -- handles images + arbitrary files)
|
|
fieldkit.py /fieldkit/* (killswitch, deadman, status)
|
|
keys.py /keys/* (unified key management, trust store)
|
|
admin.py /admin/* (user management)
|
|
dropbox.py /dropbox/* (token-gated anonymous source upload, ~13k)
|
|
federation.py /federation/* (peer status dashboard, peer add/remove)
|
|
templates/
|
|
dropbox/admin.html Drop box admin panel
|
|
federation/status.html Federation peer dashboard
|
|
|
|
frontends/cli/ CLI package init (main entry point is src/fieldwitness/cli.py)
|
|
|
|
deploy/ Deployment artifacts
|
|
docker/ Dockerfile (multi-stage: builder, relay, server) + docker-compose.yml
|
|
kubernetes/ namespace.yaml, server-deployment.yaml, relay-deployment.yaml
|
|
live-usb/ build.sh + config/ for Debian Live USB (Tier 1)
|
|
config-presets/ low-threat.json, medium-threat.json, high-threat.json, critical-threat.json
|
|
|
|
docs/ Documentation
|
|
deployment.md Three-tier deployment guide (~1500 lines)
|
|
federation.md Gossip protocol, peer setup, offline bundles, CLI commands
|
|
evidence-guide.md Evidence packages, cold archives, selective disclosure
|
|
source-dropbox.md Source drop box setup: tokens, EXIF pipeline, receipts
|
|
architecture/
|
|
federation.md System architecture overview (threat model, layers, key domains)
|
|
chain-format.md Chain record spec (CBOR, entropy witnesses, serialization)
|
|
export-bundle.md Export bundle spec (FIELDWITNESSX1 binary format, envelope encryption)
|
|
federation-protocol.md Federation server protocol (CT-inspired, gossip, storage tiers)
|
|
training/
|
|
reporter-quickstart.md One-page reporter quick-start for Tier 1 USB (print + laminate)
|
|
reporter-field-guide.md Comprehensive reporter guide: attest, stego, killswitch, evidence
|
|
emergency-card.md Laminated wallet card: emergency destruction reference
|
|
admin-reference.md Admin CLI cheat sheet, hardening checklist, troubleshooting
|
|
admin-operations-guide.md Full admin operations: users, dropbox, federation, incidents
|
|
```
|
|
|
|
## Three-tier deployment model
|
|
|
|
```
|
|
Tier 1: Field Device Tier 2: Org Server Tier 3: Federation Relay
|
|
(Bootable USB + laptop) (Docker on mini PC / VPS) (Docker on VPS)
|
|
Amnesic, LUKS-encrypted Persistent, full features Attestation sync only
|
|
Reporter in the field Newsroom / NGO office Friendly jurisdiction
|
|
```
|
|
|
|
- Tier 1 <-> Tier 2: sneakernet (USB drive) or LAN
|
|
- Tier 2 <-> Tier 3: federation API (port 8000) over internet
|
|
- Tier 2 <-> Tier 2: via Tier 3 relay, or directly via sneakernet
|
|
|
|
## Dependency model
|
|
|
|
Stego and Attest are inlined subpackages, not separate pip packages:
|
|
- `from fieldwitness.stego import encode` for steganography
|
|
- `from fieldwitness.attest import Attestation` for provenance attestation
|
|
- Never `import fieldwitness.stego` or `import fieldwitness.attest` directly
|
|
- `_availability.py` provides `has_stego()` / `has_attest()` for graceful degradation
|
|
when optional extras are not installed
|
|
|
|
## Key design decisions
|
|
|
|
- **Two key domains, never merged**: Stego AES-256-GCM (derived from factors) and
|
|
Attest Ed25519 (signing identity) are separate security concerns
|
|
- **Extract-then-strip model**: Stego strips all EXIF (carrier is vessel); attestation
|
|
extracts evidentiary EXIF (GPS, timestamp) then strips dangerous fields (device serial)
|
|
- **subprocess_stego.py copies verbatim** from fieldwitness.stego -- it's a crash-safety boundary
|
|
- **All state under ~/.fieldwitness/** -- one directory to back up, one to destroy.
|
|
`FIELDWITNESS_DATA_DIR` env var relocates everything (cover mode, USB mode)
|
|
- **Offline-first**: All static assets vendored, no CDN. pip wheels bundled for airgap install
|
|
- **Flask blueprints**: stego, attest, fieldkit, keys, admin, dropbox, federation
|
|
- **Flask-WTF**: CSRF protection on all form endpoints; drop box is CSRF-exempt (sources
|
|
don't have sessions)
|
|
- **Client-side SHA-256**: Drop box upload page uses SubtleCrypto for pre-upload hashing
|
|
- **Waitress**: Production WSGI server (replaces dev-only Flask server)
|
|
- **FastAPI option**: `fieldwitness.api` provides a REST API alternative to the Flask web UI
|
|
- **Pluggable backends**: Stego backends (LSB, DCT) registered via `backends/registry.py`
|
|
- **ImageHashes generalized**: phash/dhash now optional, enabling non-image attestation
|
|
- **Lazy path resolution**: All paths in paths.py resolve lazily via `__getattr__` from
|
|
`BASE_DIR` so that runtime overrides (--data-dir, FIELDWITNESS_DATA_DIR) propagate correctly
|
|
- **Two-way federation**: Delivery acknowledgment records (`fieldwitness/delivery-ack-v1`)
|
|
enable handshake proof
|
|
- **Chain record types** (in federation/chain.py):
|
|
- `CONTENT_TYPE_KEY_ROTATION = "fieldwitness/key-rotation-v1"` -- signed by OLD key
|
|
- `CONTENT_TYPE_KEY_RECOVERY = "fieldwitness/key-recovery-v1"` -- signed by NEW key
|
|
- `CONTENT_TYPE_DELIVERY_ACK = "fieldwitness/delivery-ack-v1"` -- signed by receiver
|
|
- **Gossip federation** (attest/federation.py): GossipNode with async peer sync,
|
|
consistency proofs, HttpTransport over aiohttp. PeerStore for SQLite-backed persistence
|
|
- **Threat level presets**: deploy/config-presets/ with low/medium/high/critical configs
|
|
- **Selective disclosure**: Chain records can be exported with non-selected records
|
|
redacted to hashes only (for legal discovery)
|
|
- **Evidence packages**: Self-contained ZIPs with verify.py that needs only Python + cryptography
|
|
- **Cold archives**: OAIS-aligned full-state export with ALGORITHMS.txt
|
|
- **Transport-aware stego**: --transport whatsapp|signal|telegram auto-selects DCT/JPEG
|
|
and pre-resizes carrier for platform survival
|
|
|
|
## Data directory layout (`~/.fieldwitness/`)
|
|
|
|
```
|
|
~/.fieldwitness/
|
|
config.json Unified configuration (FieldWitnessConfig dataclass)
|
|
audit.jsonl Append-only audit trail (JSON-lines)
|
|
carrier_history.json Carrier reuse tracking database
|
|
identity/ Ed25519 keypair (private.pem, public.pem, identity.meta.json)
|
|
archived/ Timestamped old keypairs from rotations
|
|
stego/ Channel key (channel.key)
|
|
archived/ Timestamped old channel keys from rotations
|
|
attestations/ Attest attestation store
|
|
log.bin Binary attestation log
|
|
index/ LMDB index
|
|
peers.json Legacy peer file
|
|
federation/ Federation state
|
|
peers.db SQLite peer + sync history
|
|
chain/ Hash chain (chain.bin, state.cbor)
|
|
anchors/ External timestamp anchors (JSON files)
|
|
auth/ Web UI auth databases
|
|
fieldwitness.db User accounts
|
|
dropbox.db Drop box tokens + receipts
|
|
certs/ Self-signed TLS certificates (cert.pem, key.pem)
|
|
fieldkit/ Fieldkit state
|
|
deadman.json Dead man's switch timer
|
|
tamper/baseline.json File integrity baselines
|
|
usb/whitelist.json USB device whitelist
|
|
geofence.json GPS boundary config
|
|
temp/ Ephemeral file storage
|
|
dropbox/ Source drop box submissions
|
|
instance/ Flask instance (sessions, .secret_key)
|
|
trusted_keys/ Collaborator Ed25519 public keys (trust store)
|
|
<fingerprint>/ Per-key directory (public.pem + meta.json)
|
|
last_backup.json Backup timestamp tracking
|
|
```
|
|
|
|
## Code conventions
|
|
|
|
Black (100-char), Ruff (E, F, I, N, W, UP), mypy (strict, ignore_missing_imports),
|
|
imperative commit messages.
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
pytest # All tests with coverage
|
|
pytest tests/test_chain.py # Chain-specific
|
|
```
|
|
|
|
Test files: `test_chain.py`, `test_chain_security.py`, `test_deadman_enforcement.py`,
|
|
`test_key_rotation.py`, `test_killswitch.py`, `test_serialization.py`,
|
|
`test_stego_audio.py`, `test_stego.py`, `test_attest_hashing.py`
|