diff --git a/hmac-file-server 3.2 %5Bwiki%5D.-.md b/hmac-file-server 3.3.3 Nexus Infinitum.-.md similarity index 53% rename from hmac-file-server 3.2 %5Bwiki%5D.-.md rename to hmac-file-server 3.3.3 Nexus Infinitum.-.md index 6a87284..64fd5b7 100644 --- a/hmac-file-server 3.2 %5Bwiki%5D.-.md +++ b/hmac-file-server 3.3.3 Nexus Infinitum.-.md @@ -1,11 +1,11 @@ -This documentation provides detailed information on configuring, setting up, and maintaining the HMAC File Server. Whether you're a developer, system administrator, or an enthusiast, this guide will help you navigate through the server's features and configurations effectively. +This documentation provides detailed information on configuring, setting up, and maintaining the HMAC File Server 3.3.0 "Nexus Infinitum". Whether you're a developer, system administrator, or an enthusiast, this guide will help you navigate through the server's features and configurations effectively. --- ## Table of Contents 1. [Introduction](#introduction) -2. [3.2 "Tremora del Terra" Revolutionary Features](#32-tremora-del-terra-revolutionary-features) +2. [3.3.0 "Nexus Infinitum" Revolutionary Features](#330-nexus-infinitum-revolutionary-features) 3. [Configuration](#configuration) - [Server Configuration](#server-configuration) - [Deduplication Settings](#deduplication-settings) @@ -18,6 +18,8 @@ This documentation provides detailed information on configuring, setting up, and - [ClamAV Settings](#clamav-settings) - [Redis Settings](#redis-settings) - [Worker Settings](#worker-settings) + - [Network Resilience Settings](#network-resilience-settings) + - [Client Network Support Settings](#client-network-support-settings) 4. [Example Configuration](#example-configuration) 5. [Setup Instructions](#setup-instructions) - [1. HMAC File Server Installation](#1-hmac-file-server-installation) @@ -27,24 +29,29 @@ This documentation provides detailed information on configuring, setting up, and - [3. ejabberd Configuration](#3-ejabberd-configuration) - [4. Systemd Service Setup](#4-systemd-service-setup) 6. [Running with Docker & Docker Compose](#running-with-docker--docker-compose) -7. [Building for Different Architectures](#building-for-different-architectures) -8. [Network Resilience & Queue Optimization](#network-resilience--queue-optimization) -9. [Multi-Architecture Deployment](#multi-architecture-deployment) -10. [Additional Recommendations](#additional-recommendations) -8. [Notes](#notes) -9. [Using HMAC File Server for CI/CD Build Artifacts](#using-hmac-file-server-for-ci-cd-build-artifacts) -10. [Monitoring](#monitoring) +7. [Running with Podman](#running-with-podman) +8. [Multi-Architecture Build System](#multi-architecture-build-system) +9. [Network Resilience & Queue Optimization](#network-resilience--queue-optimization) +10. [Multi-Architecture Deployment](#multi-architecture-deployment) +11. [Command-Line Tools & Utilities](#command-line-tools--utilities) +12. [Development & Build Tools](#development--build-tools) +13. [Additional Recommendations](#additional-recommendations) +14. [Notes](#notes) +15. [Using HMAC File Server for CI/CD Build Artifacts](#using-hmac-file-server-for-ci-cd-build-artifacts) +16. [Monitoring](#monitoring) --- ## Introduction -The **HMAC File Server 3.2 "Tremora del Terra"** is a revolutionary secure and efficient file management solution designed to handle file uploads, downloads, deduplication, and more. This major release brings **93% configuration reduction**, dramatically simplifying setup while maintaining enterprise-grade features. +The **HMAC File Server 3.3.0 "Nexus Infinitum"** is a revolutionary secure and efficient file management solution designed for infinite connectivity and boundless network resilience. This major release brings **Desktop XMPP Client Revolution**, **Network Resilience Perfection**, and **Mobile Client Optimization**. -**Version 3.2 Revolutionary Features:** -- **93% Configuration Reduction**: Simplified setup with intelligent defaults -- **Network Resilience**: Advanced connection recovery and stability -- **Queue Optimization**: Enhanced dynamic worker scaling (40%/10% thresholds) +**Version 3.3.0 "Nexus Infinitum" Revolutionary Features:** +- **Desktop XMPP Client Revolution**: 48-hour session restoration for Dino and Gajim +- **Network Resilience Perfection**: WiFi ↔ LTE switching with zero interruption +- **Mobile Client Optimization**: 72-hour ultra-grace periods for critical scenarios +- **Multi-Architecture Excellence**: Native builds for AMD64, ARM64, ARM32v7 +- **Infinite Connectivity**: Boundless network topology adaptation - **Extended Timeouts**: 4800s timeouts for seamless large file transfers - **Multi-Architecture Support**: Native AMD64, ARM64, ARM32v7 builds - **XEP-0363 XMPP Integration**: Full XMPP file sharing protocol support @@ -54,9 +61,9 @@ Built with a focus on security, scalability, and performance, it integrates seam --- -## 3.2 "Tremora del Terra" Revolutionary Features +## 3.3.0 "Nexus Infinitum" Revolutionary Features -HMAC File Server 3.2 "Tremora del Terra" represents a revolutionary leap forward in file server technology, introducing breakthrough simplifications and advanced enterprise features: +HMAC File Server 3.3.0 "Nexus Infinitum" represents a revolutionary leap forward in file server technology, introducing breakthrough simplifications and advanced enterprise features: ### 🚀 **93% Configuration Reduction** - **Simplified Setup**: Reduced configuration complexity by 93% through intelligent defaults @@ -398,19 +405,34 @@ compress = true # Compress old log files ```toml # Upload settings [uploads] -allowed_extensions = [".zip", ".rar", ".7z", ".tar.gz", ".tgz", ".gpg", ".enc", ".pgp"] +allowed_extensions = [".zip", ".rar", ".7z", ".tar.gz", ".tgz", ".gpg", ".enc", ".pgp", ".txt", ".pdf", ".jpg", ".jpeg", ".png", ".gif", ".webp", ".mp4", ".mov", ".ogg", ".mp3", ".doc", ".docx"] chunked_uploads_enabled = true -chunk_size = "10MB" # Chunk size for uploads +chunk_size = "10MB" # Chunk size for uploads resumable_uploads_enabled = true -max_resumable_age = "48h" # Maximum age for resumable uploads +max_resumable_age = "48h" # Maximum age for resumable uploads +sessiontimeout = "60m" # Upload session timeout +maxretries = 3 # Maximum upload retry attempts +networkevents = false # Enable network event monitoring for uploads + +# Upload resilience and session management +session_persistence = true # Persist sessions across restarts +session_recovery_timeout = "300s" # Session recovery timeout after network changes +client_reconnect_window = "120s" # Time window for client reconnection +upload_slot_ttl = "3600s" # Upload slot validity time +retry_failed_uploads = true # Auto-retry failed uploads +max_upload_retries = 3 # Maximum retry attempts +allow_session_resume = true # Allow resume from different IPs +session_persistence_duration = "24h" # How long to keep session data +detect_duplicate_uploads = true # Detect same upload from different IPs +merge_duplicate_sessions = true # Merge sessions from same client ``` #### Configuration Options - **allowed_extensions**: - *Type*: `Array of Strings` - - *Description*: Lists the file extensions permitted for upload. - - *Default*: `[".zip", ".rar", ".7z", ".tar.gz", ".tgz", ".gpg", ".enc", ".pgp"]` + - *Description*: Lists the file extensions permitted for upload. Includes XMPP-compatible formats. + - *Default*: `[".zip", ".rar", ".7z", ".tar.gz", ".tgz", ".gpg", ".enc", ".pgp", ".txt", ".pdf", ".jpg", ".jpeg", ".png", ".gif", ".webp", ".mp4", ".mov", ".ogg", ".mp3", ".doc", ".docx"]` - **chunked_uploads_enabled**: - *Type*: `Boolean` @@ -434,6 +456,27 @@ max_resumable_age = "48h" # Maximum age for resumable uploads - *Format*: Duration (e.g., `"48h"`) - *Default*: `"48h"` +- **networkevents**: + - *Type*: `Boolean` + - *Description*: Enables network event monitoring for uploads. Required for network resilience features. + - *Default*: `false` + +- **session_persistence**: + - *Type*: `Boolean` + - *Description*: Persists upload sessions across server restarts and network changes. + - *Default*: `true` + +- **session_recovery_timeout**: + - *Type*: `String` + - *Description*: Maximum time to wait for session recovery after network changes. + - *Format*: Duration (e.g., `"300s"`) + - *Default*: `"300s"` + +- **allow_session_resume**: + - *Type*: `Boolean` + - *Description*: Allows upload sessions to resume from different IP addresses (useful for mobile clients). + - *Default*: `true` + --- ### Downloads Configuration @@ -582,6 +625,124 @@ uploadqueuesize = 50 # Size of upload queue --- +### Network Resilience Settings + +```toml +# Network resilience configuration for mobile and multi-interface environments +[network_resilience] +enabled = true # Enable network resilience system +fast_detection = true # Enable 1-second network change detection +quality_monitoring = true # Monitor RTT and packet loss per interface +predictive_switching = true # Switch proactively before network failure +mobile_optimizations = true # Use mobile-friendly thresholds for cellular networks +upload_resilience = true # Resume uploads across network changes +detection_interval = "1s" # Network change detection interval +quality_check_interval = "5s" # Connection quality monitoring interval +max_detection_interval = "10s" # Maximum detection interval during stable periods +network_change_threshold = 3 # Switches required to trigger network change +interface_stability_time = "30s" # Time to wait before marking interface stable +upload_pause_timeout = "5m" # Maximum time to pause uploads during network changes +upload_retry_timeout = "10m" # Maximum time to retry uploads after network changes +rtt_warning_threshold = "200ms" # RTT threshold for warning +rtt_critical_threshold = "1000ms" # RTT threshold for critical +packet_loss_warning_threshold = 2.0 # Packet loss % for warning +packet_loss_critical_threshold = 10.0 # Packet loss % for critical + +# Multi-Interface Support (v3.3.0+) +multi_interface_enabled = false # Enable multi-interface management +interface_priority = ["eth0", "wlan0", "wwan0", "ppp0"] # Interface priority order +auto_switch_enabled = true # Enable automatic interface switching +switch_threshold_latency = "500ms" # Latency threshold for switching +switch_threshold_packet_loss = 5.0 # Packet loss threshold for switching +quality_degradation_threshold = 0.5 # Quality degradation threshold +max_switch_attempts = 3 # Maximum switch attempts per detection +switch_detection_interval = "10s" # Switch detection interval +``` + +#### Configuration Options + +- **enabled**: + - *Type*: `Boolean` + - *Description*: Enables the network resilience system for handling network changes and quality monitoring. + - *Default*: `true` + +- **fast_detection**: + - *Type*: `Boolean` + - *Description*: Enables 1-second network change detection vs 5-second default for rapid network switching scenarios. + - *Default*: `true` + +- **quality_monitoring**: + - *Type*: `Boolean` + - *Description*: Monitors RTT and packet loss per interface to determine network quality and trigger proactive switching. + - *Default*: `true` + +- **predictive_switching**: + - *Type*: `Boolean` + - *Description*: Switches networks proactively before complete failure based on quality degradation patterns. + - *Default*: `true` + +- **mobile_optimizations**: + - *Type*: `Boolean` + - *Description*: Uses mobile-friendly thresholds for cellular networks with higher tolerance for latency and packet loss. + - *Default*: `true` + +- **upload_resilience**: + - *Type*: `Boolean` + - *Description*: Enables upload session preservation and resumption across network changes. + - *Default*: `true` + +- **multi_interface_enabled**: + - *Type*: `Boolean` + - *Description*: Enables management of multiple network interfaces with automatic switching capabilities. + - *Default*: `false` + +- **interface_priority**: + - *Type*: `Array of Strings` + - *Description*: Defines the preference order for network interfaces. First interface has highest priority. + - *Default*: `["eth0", "wlan0", "wwan0", "ppp0"]` + +**Use Cases**: +- Mobile devices switching between WiFi and cellular +- Laptops with Ethernet + WiFi +- IoT devices with primary and backup connections +- Server environments with multiple network adapters + +--- + +### Client Network Support Settings + +```toml +# Client network support for handling clients with changing IPs +[client_network_support] +session_based_tracking = false # Track sessions by ID instead of IP +allow_ip_changes = true # Allow session continuation from different IPs +session_migration_timeout = "5m" # Time to wait for client reconnection +max_ip_changes_per_session = 10 # Maximum IP changes per session +client_connection_detection = false # Detect client network type +adapt_to_client_network = false # Optimize parameters based on client connection +``` + +#### Configuration Options + +- **session_based_tracking**: + - *Type*: `Boolean` + - *Description*: Tracks upload sessions by session ID instead of client IP, enabling seamless operation when clients change networks. + - *Default*: `false` + +- **allow_ip_changes**: + - *Type*: `Boolean` + - *Description*: Allows the same upload session to continue from different IP addresses. + - *Default*: `true` + +- **adapt_to_client_network**: + - *Type*: `Boolean` + - *Description*: Automatically optimizes upload parameters (chunk size, timeouts) based on detected client connection type. + - *Default*: `false` + +**Note**: These settings are particularly useful for mobile applications and environments where clients frequently change networks. + +--- + #### Configuration Options - **maxfilesize**: @@ -592,9 +753,117 @@ uploadqueuesize = 50 # Size of upload queue --- +## Configuration Troubleshooting + +### Common Configuration Issues + +#### ❌ **Field Name Errors** + +**Problem**: Service fails to start with `storage path is required` or defaults to `./uploads` + +```bash +# ❌ WRONG - Missing underscore +[server] +storagepath = "/opt/hmac-file-server/data/uploads" + +# ✅ CORRECT - Use underscores in field names +[server] +storage_path = "/opt/hmac-file-server/data/uploads" +``` + +**Common Field Name Corrections:** +- `storagepath` → `storage_path` +- `listenport` → `listen_address` +- `bindip` → `bind_ip` +- `pidfilepath` → `pid_file` +- `metricsenabled` → `metrics_enabled` + +#### ❌ **Path & Permission Issues** + +**Problem**: `directory is not writable: permission denied` + +```bash +# Check directory ownership +ls -la /opt/hmac-file-server/data/ + +# Fix ownership for systemd service +sudo chown -R hmac-file-server:hmac-file-server /opt/hmac-file-server/data/ +sudo chmod 750 /opt/hmac-file-server/data/uploads +``` + +#### ❌ **Network Resilience Not Working** + +**Problem**: Network events not detected, uploads don't resume after network changes + +```toml +# ✅ Enable network events in uploads section (REQUIRED) +[uploads] +networkevents = true # This enables the network monitoring system + +# ✅ Add network resilience configuration +[network_resilience] +enabled = true +quality_monitoring = true +upload_resilience = true +fast_detection = true +``` + +**Common Issues**: +- `networkevents = false` (or missing) in uploads section +- Network resilience disabled but expecting network change detection +- Missing `upload_resilience = true` for upload session recovery + +#### ❌ **Service Fails with Read-Only File System** + +**Problem**: `open uploads/.write_test: read-only file system` + +**Cause**: Conflicting local directories or systemd restrictions + +```bash +# Remove conflicting directories +sudo rm -rf /opt/hmac-file-server/uploads + +# Use absolute paths in configuration +[server] +storage_path = "/opt/hmac-file-server/data/uploads" # Absolute path +``` + +### 🛠️ **Quick Diagnostic Commands** + +```bash +# 1. Auto-fix common field naming issues (recommended) +./fix-config.sh config.toml + +# 2. Validate configuration syntax +./hmac-file-server --validate-config + +# 3. Check service logs for errors +journalctl -u hmac-file-server.service -f + +# 4. Test configuration manually +sudo -u hmac-file-server ./hmac-file-server -config config.toml --validate-config + +# 5. Check directory permissions +ls -la /opt/hmac-file-server/data/ +stat /opt/hmac-file-server/data/uploads +``` + +### 📋 **Configuration Checklist** + +Before starting the service, verify: + +- ✅ All field names use underscores (`storage_path`, not `storagepath`) +- ✅ Absolute paths for all directories +- ✅ Correct user ownership (`hmac-file-server:hmac-file-server`) +- ✅ Proper directory permissions (750 for data directories) +- ✅ No conflicting local directories in working directory +- ✅ Network events enabled if using network resilience + +--- + ## Configuration Validation -The HMAC File Server v3.2 includes a comprehensive configuration validation system with specialized command-line flags for different validation scenarios. +The HMAC File Server v3.3.0 includes a comprehensive configuration validation system with specialized command-line flags for different validation scenarios. ### Available Validation Flags @@ -722,7 +991,215 @@ livenessProbe: periodSeconds: 60 ``` -The enhanced command-line validation system provides comprehensive coverage with 50+ validation checks across all configuration areas, making HMAC File Server v3.2 production-ready with enterprise-grade configuration management. +The enhanced command-line validation system provides comprehensive coverage with 50+ validation checks across all configuration areas, making HMAC File Server v3.3.0 production-ready with enterprise-grade configuration management. + +--- + +## Command-Line Tools & Utilities + +HMAC File Server 3.3.0 "Nexus Infinitum" includes a comprehensive suite of command-line tools and utilities for development, debugging, and maintenance. + +### Core Server Options + +```bash +# Basic operations +./hmac-file-server -config config.toml # Start server +./hmac-file-server -genconfig # Generate default config +./hmac-file-server -version # Show version info +./hmac-file-server -help # Show help + +# Configuration validation +./hmac-file-server -config config.toml --validate # Validate config +./hmac-file-server -config config.toml --validate-quiet # Silent validation +./hmac-file-server -config config.toml --check # Check configuration +``` + +### Diagnostic & Debugging Tools + +```bash +# XMPP Client Troubleshooting (NEW in 3.3.0) +./fix_xmpp_clients.sh # Fix desktop client upload issues +./fix_xmpp_clients.sh --clear-cache # Clear XMPP client caches +./fix_xmpp_clients.sh --dino # Fix Dino-specific issues +./fix_xmpp_clients.sh --gajim # Fix Gajim-specific issues + +# Network Resilience Verification (NEW in 3.3.0) +./verify_network_resilience.sh # Test network switching scenarios +./verify_network_resilience.sh --mobile # Test mobile network scenarios +./verify_network_resilience.sh --wifi # Test WiFi scenarios +``` + +### Build & Development Tools + +```bash +# Multi-Architecture Building (NEW in 3.3.0) +./build-multi-arch.sh # Interactive multiarch builder +./build-multi-arch.sh --help # Show build options + +# Docker Multi-Architecture (NEW in 3.3.0) +./docker-multiarch-build.sh --local # Build for local testing +./docker-multiarch-build.sh --push # Build and push to registry +./docker-multiarch-build.sh --help # Show Docker build options + +# Debian Package Building +./builddebian.sh # Build .deb packages (AMD64 + ARM64) +./builddebian.sh --help # Show packaging options + +# Docker Standard Building +./builddocker.sh # Build standard Docker image +``` + +### Installation & Setup Tools + +```bash +# Automated Installation +./installer.sh # Interactive installer +./installer.sh --help # Show installation options + +# Installation Manager (NEW in 3.3.0) +./install-manager.sh # Advanced installation management +./install-manager.sh --upgrade # Upgrade existing installation +./install-manager.sh --uninstall # Clean uninstallation +``` + +### Configuration Generation + +```bash +# Generate configuration templates +./hmac-file-server -genconfig > config.toml # Basic config +./hmac-file-server -genconfig-mobile > mobile.toml # Mobile-optimized +./hmac-file-server -genconfig-enterprise > enterprise.toml # Enterprise config +./hmac-file-server -genconfig-minimal > minimal.toml # Minimal config + +# Configuration examples available: +# - config-mobile-resilient.toml # Mobile resilience optimized +# - config-production-enhanced.toml # Production deployment +# - config-production-validated.toml # Validated production config +``` + +### Environment Variables + +```bash +# Common environment variables +export HMAC_SECRET="your-secret-key" # HMAC authentication secret +export STORAGE_PATH="/data/uploads" # Upload storage directory +export LISTEN_PORT="8080" # Server listen port +export LOG_LEVEL="info" # Logging level +export PROMETHEUS_PORT="9090" # Metrics port + +# Development mode +export HMAC_DEV_MODE="true" # Enable development features +export HMAC_DEBUG="true" # Enable debug logging +export HMAC_TRACE="true" # Enable trace logging +``` + +--- + +## Development & Build Tools + +### Multi-Architecture Build System + +HMAC File Server 3.3.0 features a comprehensive multi-architecture build system supporting 13+ platforms. + +#### Interactive Builder + +```bash +./build-multi-arch.sh +``` + +**Menu Options:** +1. **All supported platforms** - Complete multiarch build (Linux, macOS, Windows, FreeBSD) +2. **Linux only** - AMD64, ARM64, ARM32v7 for server deployment +3. **Cross-platform** - Linux, macOS, Windows for desktop distribution +4. **Custom selection** - Choose specific platforms +5. **Quick build** - Linux AMD64 only for rapid development + +#### Supported Platforms + +| Platform | Architecture | Use Case | +|----------|-------------|----------| +| `linux/amd64` | x86-64 | Data centers, cloud instances | +| `linux/arm64` | ARM 64-bit | Apple Silicon, AWS Graviton, Pi 4+ | +| `linux/arm` | ARM 32-bit | Raspberry Pi 3, IoT devices | +| `linux/386` | x86 32-bit | Legacy systems | +| `darwin/amd64` | Intel Mac | macOS Intel development | +| `darwin/arm64` | Apple Silicon | macOS M1/M2/M3 development | +| `windows/amd64` | Windows 64-bit | Windows server deployment | +| `windows/386` | Windows 32-bit | Legacy Windows systems | +| `freebsd/amd64` | FreeBSD | BSD server deployment | +| `openbsd/amd64` | OpenBSD | Security-focused deployment | + +#### Docker Multi-Architecture + +```bash +# Local development +./docker-multiarch-build.sh --local + +# Production deployment +./docker-multiarch-build.sh --registry your-registry.com --push +``` + +**Features:** +- **Docker Buildx integration** - Native multi-platform support +- **Platform targeting** - `linux/amd64,linux/arm64,linux/arm/v7` +- **Registry push** - Automated multi-arch image distribution +- **Local testing** - Build and load for immediate testing + +#### Manual Build Commands + +```bash +# Linux AMD64 (Primary) +GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-w -s" -o builds/hmac-file-server-linux-amd64 ./cmd/server/ + +# Linux ARM64 (Apple Silicon, Graviton) +GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -ldflags="-w -s" -o builds/hmac-file-server-linux-arm64 ./cmd/server/ + +# Linux ARM32v7 (Raspberry Pi) +GOOS=linux GOARCH=arm GOARM=7 CGO_ENABLED=0 go build -ldflags="-w -s" -o builds/hmac-file-server-linux-arm ./cmd/server/ + +# macOS Universal +GOOS=darwin GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-w -s" -o builds/hmac-file-server-darwin-amd64 ./cmd/server/ +GOOS=darwin GOARCH=arm64 CGO_ENABLED=0 go build -ldflags="-w -s" -o builds/hmac-file-server-darwin-arm64 ./cmd/server/ + +# Windows +GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-w -s" -o builds/hmac-file-server-windows-amd64.exe ./cmd/server/ +``` + +### Debian Package System + +```bash +./builddebian.sh +``` + +**Features:** +- **Multi-architecture packages** - AMD64 and ARM64 .deb files +- **Systemd integration** - Complete service configuration +- **Dependency management** - Automatic dependency resolution +- **Configuration templates** - Production-ready configs included + +**Generated Packages:** +- `hmac-file-server_3.3.0_amd64.deb` - AMD64 Debian package +- `hmac-file-server_3.3.0_arm64.deb` - ARM64 Debian package + +### Container Build Tools + +#### Standard Docker Build +```bash +./builddocker.sh # Standard single-arch Docker build +``` + +#### Podman Support +```bash +# Clone repository +git clone https://git.uuxo.net/uuxo/hmac-file-server.git +cd hmac-file-server/dockerenv/podman + +# One-command deployment +./deploy-podman.sh + +# Check status +./deploy-podman.sh status +``` --- @@ -756,11 +1233,18 @@ worker_scale_up_thresh = 40 # 40% optimized threshold for 3.2 worker_scale_down_thresh = 10 [uploads] -allowed_extensions = [".zip", ".rar", ".7z", ".tar.gz", ".tgz", ".gpg", ".enc", ".pgp"] +allowed_extensions = [".zip", ".rar", ".7z", ".tar.gz", ".tgz", ".gpg", ".enc", ".pgp", ".txt", ".pdf", ".jpg", ".jpeg", ".png", ".gif", ".webp", ".mp4", ".mov", ".ogg", ".mp3", ".doc", ".docx"] chunked_uploads_enabled = true chunk_size = "10MB" resumable_uploads_enabled = true max_resumable_age = "48h" +sessiontimeout = "60m" +maxretries = 3 +networkevents = true # Enable network event monitoring +session_persistence = true +session_recovery_timeout = "300s" +client_reconnect_window = "120s" +allow_session_resume = true [downloads] resumable_downloads_enabled = true @@ -822,11 +1306,46 @@ redishealthcheckinterval = "120s" numworkers = 4 uploadqueuesize = 50 +# Network Resilience (v3.3.0+) +[network_resilience] +enabled = true +fast_detection = true +quality_monitoring = true +predictive_switching = true +mobile_optimizations = false # Use strict thresholds for server environment +upload_resilience = true +detection_interval = "5s" # Standard detection for servers +quality_check_interval = "10s" +network_change_threshold = 3 +interface_stability_time = "30s" +upload_pause_timeout = "5m" +upload_retry_timeout = "10m" +rtt_warning_threshold = "200ms" +rtt_critical_threshold = "1000ms" +packet_loss_warning_threshold = 2.0 +packet_loss_critical_threshold = 10.0 + +# Multi-interface support (optional) +multi_interface_enabled = false # Enable for multi-interface setups +interface_priority = ["eth0", "wlan0", "wwan0", "ppp0"] +auto_switch_enabled = true +switch_threshold_latency = "500ms" +switch_threshold_packet_loss = 5.0 + +# Client Network Support (v3.3.0+) +[client_network_support] +session_based_tracking = false # Standard IP-based tracking for servers +allow_ip_changes = true # Allow for client network changes +session_migration_timeout = "5m" +max_ip_changes_per_session = 10 +client_connection_detection = false +adapt_to_client_network = false + [file] # Add file-specific configurations here [build] -version = "3.2" +version = "3.3.0" ``` --- @@ -879,7 +1398,7 @@ To install the HMAC File Server, follow these steps: 2. Build the server: ```sh - go build -o hmac-file-server ./cmd/server/main.go + go build -o hmac-file-server ./cmd/server/ ``` 3. Create the necessary directories: @@ -1163,9 +1682,430 @@ services: --- +## Running with Podman + +Podman is a daemonless container engine that's often preferred in enterprise environments for enhanced security and rootless capabilities. HMAC File Server 3.3.0 provides complete Podman support with optimized deployment scripts. + +### Why Choose Podman? + +| Feature | Docker | Podman | +|---------|--------|--------| +| **Daemon** | Requires Docker daemon | Daemonless architecture | +| **Root Access** | Requires root for Docker daemon | Can run completely rootless | +| **Security** | Good, but daemon runs as root | Enhanced security, no privileged daemon | +| **Systemd Integration** | Via Docker service | Native systemd integration | +| **Pod Support** | Requires docker-compose or swarm | Native Kubernetes-style pods | +| **Enterprise Use** | Popular in startups/mid-size | Preferred in enterprise environments | +| **SELinux** | Basic support | Excellent SELinux integration | + +### Quick Start with Podman + +```bash +# Clone repository +git clone https://git.uuxo.net/uuxo/hmac-file-server.git +cd hmac-file-server/dockerenv/podman + +# One-command deployment +./deploy-podman.sh + +# Check status +./deploy-podman.sh status + +# View logs +./deploy-podman.sh logs +``` + +### Manual Directory & Permission Setup + +#### **Understanding Container Security Model** +- **Container User**: `appuser` (UID: 1011, GID: 1011) +- **Security Principle**: Never run containers as root (UID 0) +- **Compatibility**: Works across different container runtimes and deployment modes + +#### **Step-by-Step Manual Setup** + +**Step 1: Create Directory Structure** +```bash +# Create organized directory layout +export HMAC_BASE="/opt/podman/hmac-file-server" +sudo mkdir -p ${HMAC_BASE}/{config,data,deduplication,logs} + +# Create subdirectories for uploads and duplicates +sudo mkdir -p ${HMAC_BASE}/data/{uploads,temp} +sudo mkdir -p ${HMAC_BASE}/deduplication/store +sudo mkdir -p ${HMAC_BASE}/logs/{access,error,debug} +``` + +**Step 2: Set Ownership (CRITICAL)** +```bash +# For Podman Rootless (recommended) +podman unshare chown -R 1011:1011 ${HMAC_BASE} + +# For Podman Rootful or Docker +sudo chown -R 1011:1011 ${HMAC_BASE} + +# Alternative: Use numeric IDs to avoid user lookup issues +sudo chown -R 1011:1011 ${HMAC_BASE} +``` + +**Step 3: Set Permissions** +```bash +# Directory permissions (executable for traversal) +sudo chmod 755 ${HMAC_BASE} +sudo chmod 755 ${HMAC_BASE}/{config,data,deduplication,logs} +sudo chmod 755 ${HMAC_BASE}/data/{uploads,temp} +sudo chmod 755 ${HMAC_BASE}/deduplication/store +sudo chmod 755 ${HMAC_BASE}/logs/{access,error,debug} + +# Configuration file (read-only) +sudo chmod 644 ${HMAC_BASE}/config/config.toml +``` + +**Step 4: Verify Ownership** +```bash +# Check ownership recursively +ls -laR ${HMAC_BASE} + +# Expected output format: +# drwxr-xr-x 2 1011 1011 4096 Dec 20 10:30 data +# drwxr-xr-x 2 1011 1011 4096 Dec 20 10:30 deduplication +# drwxr-xr-x 2 1011 1011 4096 Dec 20 10:30 logs +# -rw-r--r-- 1 1011 1011 1234 Dec 20 10:30 config/config.toml +``` + +#### **Container Volume Mapping** + +| Host Path | Container Mount | Access Mode | SELinux Label | Purpose | +|-----------|-----------------|-------------|---------------|---------| +| `${HMAC_BASE}/config/config.toml` | `/app/config.toml` | `ro` | `:Z` | Configuration file | +| `${HMAC_BASE}/data/` | `/data/` | `rw` | `:Z` | File uploads | +| `${HMAC_BASE}/deduplication/` | `/deduplication/` | `rw` | `:Z` | Dedup cache | +| `${HMAC_BASE}/logs/` | `/logs/` | `rw` | `:Z` | Application logs | + +#### **Complete Manual Run Command** +```bash +# Build container image +podman build -t localhost/hmac-file-server:latest -f dockerenv/podman/Dockerfile.podman . + +# Run with proper volume mounts and SELinux labels +podman run -d --name hmac-file-server \ + --security-opt no-new-privileges \ + --cap-drop=ALL \ + --read-only \ + --tmpfs /tmp:rw,noexec,nosuid,size=100m \ + -p 8888:8888 \ + -p 9090:9090 \ + -v ${HMAC_BASE}/config/config.toml:/app/config.toml:ro,Z \ + -v ${HMAC_BASE}/data:/data:rw,Z \ + -v ${HMAC_BASE}/deduplication:/deduplication:rw,Z \ + -v ${HMAC_BASE}/logs:/logs:rw,Z \ + localhost/hmac-file-server:latest -config /app/config.toml +``` + +### Troubleshooting Path & Permission Issues + +#### **Common Error: Permission Denied** +```bash +# Error in logs +{"level":"error","msg":"failed to create directories: mkdir /data: permission denied"} +``` + +**Root Cause**: Incorrect ownership or missing directories + +**Solution**: +```bash +# Check current ownership +ls -la ${HMAC_BASE} + +# Fix ownership (adjust command based on your setup) +# For rootless Podman: +podman unshare chown -R 1011:1011 ${HMAC_BASE} + +# For rootful Podman/Docker: +sudo chown -R 1011:1011 ${HMAC_BASE} + +# Verify fix +sudo -u "#1011" touch ${HMAC_BASE}/data/test-write +rm ${HMAC_BASE}/data/test-write +``` + +#### **Common Error: SELinux Denial** +```bash +# Error in logs or journalctl +SELinux is preventing access to 'write' on the file /data/test.txt +``` + +**Root Cause**: SELinux context not set for container volumes + +**Solution**: +```bash +# Option 1: Use :Z labels (recommended - private volumes) +-v ${HMAC_BASE}/data:/data:rw,Z + +# Option 2: Use :z labels (shared volumes between containers) +-v ${HMAC_BASE}/data:/data:rw,z + +# Option 3: Set SELinux boolean (system-wide change) +sudo setsebool -P container_manage_cgroup on + +# Option 4: Manual context setting +sudo semanage fcontext -a -t container_file_t "${HMAC_BASE}(/.*)?" +sudo restorecon -R ${HMAC_BASE} +``` + +#### **Common Error: User Namespace Issues** +```bash +# Error starting container +Error: can't stat ${HMAC_BASE}/data: permission denied +``` + +**Root Cause**: User namespace mapping issues in rootless mode + +**Solution**: +```bash +# Check user namespace configuration +cat /etc/subuid /etc/subgid | grep $USER + +# If missing, add mappings (requires root) +sudo usermod --add-subuids 1000-65536 $USER +sudo usermod --add-subgids 1000-65536 $USER + +# Restart user services +systemctl --user restart podman + +# Use podman unshare for ownership +podman unshare chown -R 1011:1011 ${HMAC_BASE} +``` + +#### **Verification & Testing Commands** + +```bash +# Test 1: Verify container can access all paths +podman exec hmac-file-server sh -c ' + echo "=== Container User ===" + id + echo "=== Directory Access ===" + ls -la /app /data /deduplication /logs + echo "=== Write Test ===" + touch /data/write-test && echo "✅ Data write: OK" || echo "❌ Data write: FAILED" + touch /deduplication/dedup-test && echo "✅ Dedup write: OK" || echo "❌ Dedup write: FAILED" + touch /logs/log-test && echo "✅ Log write: OK" || echo "❌ Log write: FAILED" + echo "=== Config Read Test ===" + head -3 /app/config.toml && echo "✅ Config read: OK" || echo "❌ Config read: FAILED" +' + +# Test 2: External health check +curl -f http://localhost:8888/health && echo "✅ HTTP Health: OK" || echo "❌ HTTP Health: FAILED" + +# Test 3: Metrics endpoint +curl -s http://localhost:9090/metrics | grep -E "hmac_|go_|process_" | wc -l +# Should return > 0 if metrics are working + +# Test 4: File upload simulation (requires auth) +curl -X POST http://localhost:8888/upload \ + -H "Authorization: Bearer your-token" \ + -F "file=@test-file.txt" && echo "✅ Upload: OK" || echo "❌ Upload: FAILED" +``` + +#### **Advanced: Custom UID/GID Mapping** + +If you need different UIDs (e.g., for existing file ownership): + +```bash +# Option 1: Rebuild container with custom UID +podman build -t localhost/hmac-file-server:custom \ + --build-arg USER_UID=1500 \ + --build-arg USER_GID=1500 \ + -f dockerenv/podman/Dockerfile.podman . + +# Option 2: Use --user flag (may have limitations) +podman run --user 1500:1500 [other options] localhost/hmac-file-server:latest + +# Option 3: Host ownership adjustment +sudo chown -R 1500:1500 ${HMAC_BASE} +``` + +#### **Docker vs Podman Ownership Differences** + +| Scenario | Docker | Podman Rootless | Podman Rootful | +|----------|--------|-----------------|----------------| +| **Host UID** | 1011:1011 | 1011:1011 | 1011:1011 | +| **Container UID** | 1011:1011 | 1011:1011 | 1011:1011 | +| **Volume Ownership** | `chown 1011:1011` | `podman unshare chown 1011:1011` | `chown 1011:1011` | +| **SELinux Labels** | `:Z` or `:z` | `:Z` or `:z` | `:Z` or `:z` | + +### Podman Deployment Script Features + +The `deploy-podman.sh` script provides complete automation: + +- **✅ Interactive deployment** with colored output +- **✅ Auto-generates secure configuration** with random HMAC/JWT secrets +- **✅ Security-hardened settings**: `--cap-drop=ALL`, `--read-only`, `--no-new-privileges` +- **✅ Pod management** for XMPP integration +- **✅ Health monitoring** and status reporting +- **✅ Environment variable support** for customization + +### Podman Commands Reference + +```bash +# Build image +podman build -t localhost/hmac-file-server:latest -f dockerenv/podman/Dockerfile.podman . + +# Run with basic settings +podman run -d --name hmac-file-server \ + -p 8888:8888 \ + -v ./config.toml:/app/config.toml:ro \ + -v ./data:/data:rw \ + localhost/hmac-file-server:latest -config /app/config.toml + +# Create and manage pods for XMPP integration +podman pod create --name xmpp-services -p 8888:8888 -p 9090:9090 +podman run -d --pod xmpp-services --name hmac-file-server localhost/hmac-file-server:latest + +# View logs and status +podman logs hmac-file-server +podman ps -a +podman pod ps + +# Health check +podman healthcheck run hmac-file-server + +# Stop and cleanup +podman stop hmac-file-server +podman rm hmac-file-server +``` + +### Podman Systemd Integration + +#### User Service (Rootless - Recommended) +```bash +# Copy service file +cp dockerenv/podman/hmac-file-server.service ~/.config/systemd/user/ + +# Enable and start +systemctl --user daemon-reload +systemctl --user enable hmac-file-server.service +systemctl --user start hmac-file-server.service + +# Check status +systemctl --user status hmac-file-server.service +``` + +#### System Service (Root) +```bash +# Copy service file +sudo cp dockerenv/podman/hmac-file-server.service /etc/systemd/system/ + +# Enable and start +sudo systemctl daemon-reload +sudo systemctl enable hmac-file-server.service +sudo systemctl start hmac-file-server.service +``` + +### Podman with XMPP Integration + +```bash +# Complete XMPP + HMAC File Server pod setup +podman pod create --name xmpp-pod \ + --publish 5222:5222 \ + --publish 5269:5269 \ + --publish 5443:5443 \ + --publish 8888:8888 + +# Run Prosody XMPP server +podman run -d --pod xmpp-pod --name prosody \ + -v ./prosody/config:/etc/prosody:ro \ + -v ./prosody/data:/var/lib/prosody:rw \ + docker.io/prosody/prosody:latest + +# Run HMAC File Server +podman run -d --pod xmpp-pod --name hmac-file-server \ + -v ./hmac/config.toml:/app/config.toml:ro \ + -v ./hmac/data:/data:rw \ + localhost/hmac-file-server:latest -config /app/config.toml + +# Check pod status +podman pod ps +podman ps --pod +``` + +### Deployment Script Commands + +```bash +# Management commands +./deploy-podman.sh deploy # Full deployment (default) +./deploy-podman.sh start # Start services +./deploy-podman.sh stop # Stop all services +./deploy-podman.sh restart # Restart services +./deploy-podman.sh status # Show service status +./deploy-podman.sh logs # Show container logs +./deploy-podman.sh config # Show configuration +./deploy-podman.sh build # Build container image only +./deploy-podman.sh pod # Create pod only +./deploy-podman.sh clean # Remove containers and image +./deploy-podman.sh help # Show help + +# Environment variables +export APP_DATA="/custom/path/hmac-file-server" +export LISTEN_PORT="9999" +export METRICS_PORT="9998" +./deploy-podman.sh +``` + +### Podman Security Features + +#### Container Security +- **Rootless operation**: Runs as non-root user (UID 1011) +- **Capability dropping**: `--cap-drop=ALL` +- **No new privileges**: `--security-opt no-new-privileges` +- **Read-only filesystem**: `--read-only` with tmpfs for /tmp +- **SELinux labels**: Volume mounts with `:Z` labels + +#### Network Security +- **Pod isolation**: Containers run in isolated pods +- **Port binding**: Only necessary ports exposed +- **Health monitoring**: Built-in health checks + +### Troubleshooting Podman + +#### Common Issues + +**Permission Errors:** +```bash +# Fix SELinux contexts +restorecon -R /opt/podman/hmac-file-server + +# Check volume permissions +podman unshare ls -la /opt/podman/hmac-file-server +``` + +**Container Won't Start:** +```bash +# Check image exists +podman images | grep hmac-file-server + +# Validate configuration +./deploy-podman.sh config + +# Debug with interactive container +podman run -it --rm localhost/hmac-file-server:latest /bin/sh +``` + +**Network Issues:** +```bash +# Check pod networking +podman pod ps +podman port hmac-file-server + +# Test connectivity +nc -zv localhost 8888 +``` + +--- + ## Multi-Architecture Deployment -HMAC File Server 3.2 "Tremora del Terra" provides comprehensive multi-architecture support for modern deployment scenarios. +HMAC File Server 3.3.0 "Nexus Infinitum" provides comprehensive multi-architecture support for modern deployment scenarios. ### Supported Architectures @@ -1190,23 +2130,34 @@ HMAC File Server 3.2 "Tremora del Terra" provides comprehensive multi-architectu ### Build Commands ```bash -# Build for all architectures +# Interactive Multi-Architecture Builder (NEW in 3.3.0) ./build-multi-arch.sh -# Build specific architecture -GOOS=linux GOARCH=amd64 go build -o hmac-file-server-linux-amd64 ./cmd/server/main.go -GOOS=linux GOARCH=arm64 go build -o hmac-file-server-linux-arm64 ./cmd/server/main.go -GOOS=linux GOARCH=arm GOARM=7 go build -o hmac-file-server-linux-arm32v7 ./cmd/server/main.go +# Quick options: +# 1) All supported platforms (recommended) +# 2) Linux only (AMD64, ARM64, ARM32v7) +# 3) Cross-platform (Linux, macOS, Windows) +# 4) Custom selection +# 5) Quick build (Linux AMD64 only) + +# Manual build commands +GOOS=linux GOARCH=amd64 go build -o hmac-file-server-linux-amd64 ./cmd/server/ +GOOS=linux GOARCH=arm64 go build -o hmac-file-server-linux-arm64 ./cmd/server/ +GOOS=linux GOARCH=arm GOARM=7 go build -o hmac-file-server-linux-arm ./cmd/server/ ``` ### Docker Multi-Architecture ```bash -# Build multi-platform Docker images -docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t hmac-file-server:3.2 . +# Build multi-platform Docker images (NEW in 3.3.0) +./docker-multiarch-build.sh --local # Local testing +./docker-multiarch-build.sh --push # Push to registry + +# Manual Docker buildx (advanced) +docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t hmac-file-server:3.3.0 . # Run platform-specific image -docker run --platform linux/arm64 hmac-file-server:3.2 +docker run --platform linux/arm64 hmac-file-server:3.3.0 ``` ### Architecture-Specific Optimizations @@ -1230,7 +2181,7 @@ docker run --platform linux/arm64 hmac-file-server:3.2 ## Network Resilience & Queue Optimization -HMAC File Server 3.2 introduces advanced network resilience and queue optimization systems designed for enterprise-grade reliability. +HMAC File Server 3.3.0 introduces advanced network resilience and queue optimization systems designed for enterprise-grade reliability. ### Network Resilience Features @@ -1292,7 +2243,7 @@ RUN apk add --no-cache git COPY go.mod go.sum ./ RUN go mod download COPY . . -RUN CGO_ENABLED=0 go build -o hmac-file-server ./cmd/server/main.go +RUN CGO_ENABLED=0 go build -o hmac-file-server ./cmd/server/ # Stage 2: Runtime FROM alpine:latest @@ -1403,7 +2354,7 @@ uploadqueuesize = 50 # Add file-specific configurations here [build] -version = "3.2" +version = "3.3.0" ``` ### Quickstart with Docker Compose @@ -1421,7 +2372,7 @@ docker compose up -d ## Simplified Configuration Examples -HMAC File Server 3.2 "Tremora del Terra" achieves **93% configuration reduction** through intelligent defaults. Here are minimal configurations for common scenarios: +HMAC File Server 3.3.0 "Nexus Infinitum" achieves **93% configuration reduction** through intelligent defaults. Here are minimal configurations for common scenarios: ### Minimal Production Configuration (93% Simplified) @@ -1478,6 +2429,6 @@ enabled = true max_file_size = "10GB" ``` -**Previous versions required 100+ configuration lines - 3.2 "Tremora del Terra" does it with just a few!** +**Previous versions required 100+ configuration lines - 3.3 "Nexus Infinitum" does it with just a few!** ---