Update docs and comments for jpeglib migration (v4.2.0)
- Replace jpegio references with jpeglib in comments/docstrings - Update sanitize-for-image.sh to use system Python 3.11+ (no pyenv) - Update rpi/patches/README.md for jpeglib world - Add AUR build artifacts to .gitignore Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
5
.gitignore
vendored
5
.gitignore
vendored
@@ -97,3 +97,8 @@ rpi/*.tar.zst.zip
|
|||||||
rpi/*.img
|
rpi/*.img
|
||||||
rpi/*.img.zst
|
rpi/*.img.zst
|
||||||
rpi/*.img.zst.zip
|
rpi/*.img.zst.zip
|
||||||
|
|
||||||
|
# AUR build artifacts
|
||||||
|
aur-upload/
|
||||||
|
aur/.SRCINFO
|
||||||
|
aur/*.pkg.tar.zst
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ KEY PATTERNS
|
|||||||
============
|
============
|
||||||
|
|
||||||
1. SUBPROCESS ISOLATION
|
1. SUBPROCESS ISOLATION
|
||||||
Stegasoo's DCT mode uses scipy/jpegio which can crash on malformed input.
|
Stegasoo's DCT mode uses scipy/jpeglib which can crash on malformed input.
|
||||||
We run encode/decode in subprocesses so crashes don't take down the server:
|
We run encode/decode in subprocesses so crashes don't take down the server:
|
||||||
|
|
||||||
subprocess_stego = SubprocessStego(timeout=180)
|
subprocess_stego = SubprocessStego(timeout=180)
|
||||||
@@ -213,7 +213,7 @@ except ImportError:
|
|||||||
#
|
#
|
||||||
# This is a critical reliability pattern. Here's the problem:
|
# This is a critical reliability pattern. Here's the problem:
|
||||||
#
|
#
|
||||||
# scipy's DCT and jpegio can crash (segfault) on:
|
# scipy's DCT and jpeglib can crash (segfault) on:
|
||||||
# - Malformed JPEG files
|
# - Malformed JPEG files
|
||||||
# - Very large images that exhaust memory
|
# - Very large images that exhaust memory
|
||||||
# - Certain edge cases in coefficient manipulation
|
# - Certain edge cases in coefficient manipulation
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
Stegasoo Subprocess Worker (v4.0.0)
|
Stegasoo Subprocess Worker (v4.0.0)
|
||||||
|
|
||||||
This script runs in a subprocess and handles encode/decode operations.
|
This script runs in a subprocess and handles encode/decode operations.
|
||||||
If it crashes due to jpegio/scipy issues, the parent Flask process survives.
|
If it crashes due to jpeglib/scipy issues, the parent Flask process survives.
|
||||||
|
|
||||||
CHANGES in v4.0.0:
|
CHANGES in v4.0.0:
|
||||||
- Added channel_key support for encode/decode operations
|
- Added channel_key support for encode/decode operations
|
||||||
|
|||||||
@@ -132,7 +132,7 @@ class SubprocessStego:
|
|||||||
"""
|
"""
|
||||||
Subprocess-isolated steganography operations.
|
Subprocess-isolated steganography operations.
|
||||||
|
|
||||||
All operations run in a separate Python process. If jpegio or scipy
|
All operations run in a separate Python process. If jpeglib or scipy
|
||||||
crashes, only the subprocess dies - Flask keeps running.
|
crashes, only the subprocess dies - Flask keeps running.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|||||||
@@ -2,56 +2,39 @@
|
|||||||
|
|
||||||
This directory contains patches for dependencies that need modifications to build on ARM64.
|
This directory contains patches for dependencies that need modifications to build on ARM64.
|
||||||
|
|
||||||
|
## Current Status (v4.2+)
|
||||||
|
|
||||||
|
As of Stegasoo 4.2, we use **jpeglib** instead of jpegio. The jpeglib build process is handled inline in `setup.sh` and includes:
|
||||||
|
|
||||||
|
- Cloning from GitHub (PyPI tarball missing headers)
|
||||||
|
- Downloading libjpeg headers for each version (6b through 9f)
|
||||||
|
- Patching setup.py to skip turbo/mozjpeg (need cmake-generated headers)
|
||||||
|
|
||||||
|
See `setup.sh` for the full implementation.
|
||||||
|
|
||||||
|
## Legacy: jpegio Patches (v4.1 and earlier)
|
||||||
|
|
||||||
|
The `jpegio/` directory contains patches for the old jpegio dependency, which required removing x86-specific `-m64` compiler flags. These are no longer used but kept for reference.
|
||||||
|
|
||||||
|
## jpeglib Helper Script
|
||||||
|
|
||||||
|
The `jpeglib/install-jpeglib-arm64.sh` script is a standalone version of the jpeglib build process. It's not used by setup.sh (which has the logic inline) but can be useful for manual testing or debugging.
|
||||||
|
|
||||||
## Structure
|
## Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
patches/
|
patches/
|
||||||
<package>/
|
jpegio/ # Legacy (v4.1) - not used in v4.2+
|
||||||
arm64.patch # Standard unified diff patch file
|
arm64.patch
|
||||||
apply-patch.sh # Script with fallback strategies
|
apply-patch.sh
|
||||||
|
jpeglib/ # Reference script for manual builds
|
||||||
|
install-jpeglib-arm64.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
## How It Works
|
## Adding New Patches
|
||||||
|
|
||||||
The `apply-patch.sh` script tries multiple strategies in order:
|
If a new dependency needs ARM64 patches:
|
||||||
|
|
||||||
1. **Patch file** - Apply the `.patch` file using `patch -p1`
|
|
||||||
2. **Sed fallback** - Use sed for simple string replacements
|
|
||||||
3. **Python fallback** - Use regex for flexible pattern matching
|
|
||||||
|
|
||||||
This layered approach handles:
|
|
||||||
- Exact matches (patch file works)
|
|
||||||
- Minor upstream changes (sed catches variations)
|
|
||||||
- Significant changes (Python regex is most flexible)
|
|
||||||
- Already patched files (detected and skipped)
|
|
||||||
|
|
||||||
## Adding a New Patch
|
|
||||||
|
|
||||||
1. Create a directory: `patches/<package>/`
|
1. Create a directory: `patches/<package>/`
|
||||||
2. Create the patch file: `git diff > arm64.patch`
|
2. Add patch files or helper scripts
|
||||||
3. Create `apply-patch.sh` with appropriate fallback logic
|
3. Update `setup.sh` to apply the patch during installation
|
||||||
4. Update `setup.sh` to call the patch script
|
|
||||||
|
|
||||||
## jpegio Patch
|
|
||||||
|
|
||||||
The jpegio library includes x86-specific `-m64` compiler flags that fail on ARM64.
|
|
||||||
The patch removes these flags by replacing:
|
|
||||||
|
|
||||||
```python
|
|
||||||
cargs.append('-m64')
|
|
||||||
```
|
|
||||||
|
|
||||||
with:
|
|
||||||
|
|
||||||
```python
|
|
||||||
pass # ARM64: removed x86-specific -m64 flag
|
|
||||||
```
|
|
||||||
|
|
||||||
## Updating Patches
|
|
||||||
|
|
||||||
When upstream changes break a patch:
|
|
||||||
|
|
||||||
1. Clone the new version
|
|
||||||
2. Make the necessary modifications
|
|
||||||
3. Generate a new patch: `diff -u original modified > arm64.patch`
|
|
||||||
4. Test on a fresh Pi install
|
|
||||||
|
|||||||
@@ -264,49 +264,25 @@ if [ -n "$STEGASOO_DIR" ] && [ -d "$STEGASOO_DIR/venv" ]; then
|
|||||||
echo " Venv broken or stegasoo not installed, rebuilding..."
|
echo " Venv broken or stegasoo not installed, rebuilding..."
|
||||||
rm -rf "$STEGASOO_DIR/venv"
|
rm -rf "$STEGASOO_DIR/venv"
|
||||||
|
|
||||||
# Find Python 3.12 (prefer pyenv, fall back to system)
|
# Find system Python 3.11+ (no pyenv needed)
|
||||||
USER_HOME=$(eval echo "~$STEGASOO_USER")
|
PYTHON_BIN=""
|
||||||
PYENV_PYTHON="$USER_HOME/.pyenv/versions/3.12*/bin/python"
|
for py in python3.14 python3.13 python3.12 python3.11 python3; do
|
||||||
if compgen -G "$PYENV_PYTHON" > /dev/null 2>&1; then
|
if command -v "$py" &>/dev/null; then
|
||||||
PYTHON_BIN=$(ls $PYENV_PYTHON 2>/dev/null | head -1)
|
PYTHON_BIN=$(command -v "$py")
|
||||||
echo " Using pyenv Python: $PYTHON_BIN"
|
break
|
||||||
elif command -v python3.12 &>/dev/null; then
|
|
||||||
PYTHON_BIN="python3.12"
|
|
||||||
echo " Using system Python 3.12"
|
|
||||||
else
|
|
||||||
PYTHON_BIN="python3"
|
|
||||||
echo " Warning: Python 3.12 not found, using $($PYTHON_BIN --version)"
|
|
||||||
fi
|
|
||||||
|
|
||||||
sudo -u "$STEGASOO_USER" "$PYTHON_BIN" -m venv "$STEGASOO_DIR/venv"
|
|
||||||
sudo -u "$STEGASOO_USER" "$STEGASOO_DIR/venv/bin/pip" install --quiet --upgrade pip setuptools wheel
|
|
||||||
|
|
||||||
# On ARM64, jpegio needs patching before install
|
|
||||||
ARCH=$(uname -m)
|
|
||||||
if [[ "$ARCH" == "aarch64" || "$ARCH" == "arm64" ]]; then
|
|
||||||
echo " Building jpegio for ARM64 (this may take a minute)..."
|
|
||||||
# Install build deps
|
|
||||||
sudo -u "$STEGASOO_USER" "$STEGASOO_DIR/venv/bin/pip" install --quiet cython numpy
|
|
||||||
JPEGIO_DIR="/tmp/jpegio-build-$$"
|
|
||||||
rm -rf "$JPEGIO_DIR"
|
|
||||||
if git clone https://github.com/dwgoon/jpegio.git "$JPEGIO_DIR" 2>/dev/null; then
|
|
||||||
# Apply patch to remove -m64 flag
|
|
||||||
if [ -f "$STEGASOO_DIR/rpi/patches/jpegio/apply-patch.sh" ]; then
|
|
||||||
bash "$STEGASOO_DIR/rpi/patches/jpegio/apply-patch.sh" "$JPEGIO_DIR"
|
|
||||||
else
|
|
||||||
sed -i "s/cargs.append('-m64')/pass # ARM64 fix/g" "$JPEGIO_DIR/setup.py"
|
|
||||||
fi
|
|
||||||
# Change ownership so user can build
|
|
||||||
chown -R "$STEGASOO_USER:$STEGASOO_USER" "$JPEGIO_DIR"
|
|
||||||
sudo -u "$STEGASOO_USER" "$STEGASOO_DIR/venv/bin/pip" install "$JPEGIO_DIR"
|
|
||||||
rm -rf "$JPEGIO_DIR"
|
|
||||||
else
|
|
||||||
echo " Warning: Failed to clone jpegio, DCT mode may not work"
|
|
||||||
fi
|
fi
|
||||||
fi
|
done
|
||||||
|
|
||||||
sudo -u "$STEGASOO_USER" "$STEGASOO_DIR/venv/bin/pip" install --quiet -e "$STEGASOO_DIR[web]"
|
if [ -z "$PYTHON_BIN" ]; then
|
||||||
echo " Venv rebuilt and stegasoo installed"
|
echo " Error: Python 3.11+ not found"
|
||||||
|
VALIDATION_ERRORS=$((VALIDATION_ERRORS + 1))
|
||||||
|
else
|
||||||
|
echo " Using: $PYTHON_BIN ($($PYTHON_BIN --version 2>&1))"
|
||||||
|
sudo -u "$STEGASOO_USER" "$PYTHON_BIN" -m venv "$STEGASOO_DIR/venv"
|
||||||
|
sudo -u "$STEGASOO_USER" "$STEGASOO_DIR/venv/bin/pip" install --quiet --upgrade pip setuptools wheel
|
||||||
|
sudo -u "$STEGASOO_USER" "$STEGASOO_DIR/venv/bin/pip" install --quiet -e "$STEGASOO_DIR[web]"
|
||||||
|
echo " Venv rebuilt and stegasoo installed"
|
||||||
|
fi
|
||||||
else
|
else
|
||||||
echo " Venv OK"
|
echo " Venv OK"
|
||||||
fi
|
fi
|
||||||
|
|||||||
@@ -156,7 +156,7 @@ def has_dct_support() -> bool:
|
|||||||
dct_mod = _get_dct_module()
|
dct_mod = _get_dct_module()
|
||||||
return dct_mod.has_dct_support()
|
return dct_mod.has_dct_support()
|
||||||
except (ImportError, ValueError):
|
except (ImportError, ValueError):
|
||||||
# ValueError: numpy binary incompatibility (e.g., jpegio built against numpy 2.x)
|
# ValueError: numpy binary incompatibility (e.g., jpeglib built against numpy 2.x)
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
@@ -1097,7 +1097,7 @@ def peek_image(image_data: bytes) -> dict:
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
# Try DCT extraction (requires scipy/jpegio)
|
# Try DCT extraction (requires scipy/jpeglib)
|
||||||
try:
|
try:
|
||||||
from .dct_steganography import HAS_JPEGIO, HAS_SCIPY
|
from .dct_steganography import HAS_JPEGIO, HAS_SCIPY
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user