diff --git a/CLAMAV_LARGE_FILE_FIX.md b/CLAMAV_LARGE_FILE_FIX.md deleted file mode 100644 index b9ecbde..0000000 --- a/CLAMAV_LARGE_FILE_FIX.md +++ /dev/null @@ -1,120 +0,0 @@ -# Large File Upload "Endless Encryption" Fix - -## ๐ŸŽฏ **ROOT CAUSE IDENTIFIED** - -The "endless encryption" issue was actually **ClamAV virus scanning** getting stuck on large files, not encryption itself. Here's what was happening: - -### ๐Ÿ” **Problem Analysis** -1. **ClamAV Size Limits**: ClamAV is configured to scan files up to 200MB only -2. **Missing Disabled Check**: Server ignored `clamavenabled = false` setting -3. **Timeout Issues**: 30-second ClamAV timeout insufficient for large files -4. **Stuck in Queue**: Large files queued for scanning but never completed - -## โœ… **COMPREHENSIVE FIX IMPLEMENTED** - -### 1. **ClamAV Enabled Check** -```go -func processScan(task ScanTask) error { - // Check if ClamAV is enabled before processing - if !conf.ClamAV.ClamAVEnabled { - log.Infof("ClamAV disabled, skipping scan for file: %s", task.AbsFilename) - return nil - } - // ... rest of scanning logic -} -``` - -### 2. **Smart Size-Based Scanning** -```go -func scanFileWithClamAV(filename string) error { - // Check file size and skip scanning if too large (ClamAV limit is ~200MB) - maxScanSize := int64(200 * 1024 * 1024) // 200MB limit - if fileInfo.Size() > maxScanSize { - log.Infof("File %s (%d bytes) exceeds ClamAV scan limit, skipping scan") - return nil - } - // ... scanning logic with reduced timeouts -} -``` - -### 3. **Intelligent Timeout Scaling** -- **Small files** (< 50MB): 30-second timeout -- **Large files** (50MB+): 10-second timeout -- **Huge files** (200MB+): Skip scanning entirely - -## ๐Ÿ“Š **Current Configuration Status** - -### Production Config (`/etc/hmac-file-server/config.toml`) -```toml -[clamav] -clamavenabled = false # โœ… ClamAV is disabled -clamavsocket = "/var/run/clamav/clamd.ctl" -numscanworkers = 2 -scanfileextensions = [".txt", ".pdf", ".jpg", ".png"] -``` - -### Enhanced Logic -- **ClamAV Disabled**: All files skip scanning entirely -- **ClamAV Enabled**: Smart size-based scanning with timeouts -- **No Blocking**: Large uploads proceed immediately without scanning delays - -## ๐Ÿš€ **Expected Results** - -### Before Fix -- Small files: โœ… Work perfectly -- Large files: โŒ "Endless encryption" (stuck in ClamAV scan) -- Upload status: Frozen at encryption stage - -### After Fix -- Small files: โœ… Work perfectly (unchanged) -- Large files: โœ… **Fast upload completion** -- Upload status: **Normal progression through all stages** - -## ๐Ÿ” **Monitoring Commands** - -### Check Upload Processing -```bash -# Monitor upload activity (should see immediate completion) -sudo journalctl -u hmac-file-server -f | grep -E "upload|scan|clam" - -# Watch for ClamAV skip messages -sudo journalctl -u hmac-file-server -f | grep -i "skipping scan" - -# Monitor file processing stages -sudo tail -f /var/log/hmac-file-server/hmac-file-server.log -``` - -### Test Large Upload -```bash -# Should complete quickly without scanning delays -curl -X PUT "https://share.uuxo.net/test/large-file.mp4" \ - -H "User-Agent: Gajim 2.3.3" \ - --data-binary @largefile.mp4 -``` - -## ๐Ÿ“ˆ **Performance Improvements** - -| File Size | Before Fix | After Fix | Improvement | -|-----------|------------|-----------|-------------| -| < 50MB | Fast | Fast | No change | -| 50-200MB | Stuck/Slow | Fast | 90%+ faster | -| 200MB+ | Endless | Fast | โˆž faster | -| 970MB | Never | **Works** | **Fixed!** | - -## โœ… **Deployment Status** - -- **โœ… ClamAV Logic**: Fixed to respect disabled setting -- **โœ… Size Limits**: Intelligent 200MB scan threshold -- **โœ… Timeout Handling**: Reduced timeouts for large files -- **โœ… Server Deployed**: Updated server running with fixes -- **โœ… Zero Impact**: Small files unaffected - -## ๐ŸŽฏ **Ready for Testing** - -Your 970MB file upload should now: -1. **Start immediately** (no "endless encryption") -2. **Skip ClamAV scanning** (disabled in config) -3. **Complete normally** (all timeouts fixed) -4. **Show proper progress** (no freezing at encryption stage) - -The "endless encryption" problem is **permanently solved** for all file sizes! diff --git a/CLAMAV_SCANNING_FIX.md b/CLAMAV_SCANNING_FIX.md deleted file mode 100644 index 3813773..0000000 --- a/CLAMAV_SCANNING_FIX.md +++ /dev/null @@ -1,102 +0,0 @@ -# Large File "Encrypting" Issue - RESOLVED - -## ๐Ÿ” **Root Cause Identified** - -The "encrypting" status that lasted endlessly was actually **ClamAV virus scanning** getting stuck on large files. The misleading UI message made it appear as an encryption issue, but it was actually: - -1. **ClamAV Enabled**: `clamavenabled = true` in config -2. **Large File Scanning**: Files >200MB were hitting scan limits/timeouts -3. **Configuration Gap**: `maxscansize = "200MB"` wasn't being read by the code -4. **Extension Mismatch**: Video files (`.mp4`) weren't in the scan extension whitelist - -## โœ… **Comprehensive Fix Implemented** - -### 1. **Smart File Size Filtering** -```go -// Now reads maxscansize from config.toml -maxScanSize := parseSize(conf.ClamAV.MaxScanSize) // "200MB" from config -if fileInfo.Size() > maxScanSize { - log.Infof("File %s (%d bytes) exceeds scan limit, skipping scan") - return nil // Skip scanning, allow upload to proceed -} -``` - -### 2. **Extension-Based Scanning** -```toml -# Your config only scans these dangerous types: -scanfileextensions = [".txt", ".pdf", ".doc", ".docx", ".xls", ".xlsx", ".exe", ".zip", ".rar", ".7z", ".tar", ".gz"] -``` - -**Video files (`.mp4`, `.mov`, `.avi`) are now automatically skipped!** - -### 3. **Progressive Timeout Handling** -- **Small files (< 10MB)**: 10 second timeout -- **Medium files (10-50MB)**: 30 second timeout -- **Large files (50-200MB)**: 60 second timeout -- **Files > 200MB**: **Automatic skip** (no scanning) - -### 4. **Enhanced Logging** -```bash -# Now you'll see clear log messages: -"File video.mp4 with extension .mp4 not in scan list, skipping ClamAV scan" -"File large.zip (500MB) exceeds ClamAV scan limit (200MB), skipping scan" -``` - -## ๐Ÿš€ **Expected Results** - -### Large Video Files (970MB+) -- โœ… **No more endless "encrypting"** -- โœ… **Automatic scan bypass** (files > 200MB) -- โœ… **Extension whitelist skip** (`.mp4` not in scan list) -- โœ… **Upload proceeds immediately** after signature validation - -### Small Dangerous Files -- โœ… **Quick scanning** for executables, documents, archives -- โœ… **10-60 second timeouts** based on file size -- โœ… **Virus protection** maintained for risky file types - -## ๐Ÿ“Š **Performance Improvements** - -| File Type | Size | Previous Behavior | New Behavior | -|-----------|------|------------------|--------------| -| `.mp4` video | 970MB | โŒ Stuck "encrypting" | โœ… Skip scan, upload immediately | -| `.zip` archive | 50MB | โŒ 30s timeout risk | โœ… 60s timeout, reliable scan | -| `.exe` binary | 10MB | โŒ Potential timeout | โœ… 30s timeout, secure scan | -| `.pdf` document | 5MB | โŒ Unnecessary delay | โœ… 10s timeout, fast scan | - -## ๐Ÿ” **Monitoring Commands** - -### Watch Upload Progress -```bash -# Monitor ClamAV decisions in real-time -sudo journalctl -u hmac-file-server -f | grep -i "scan\|clam\|skip" - -# Example output you should see: -# "File video.mp4 with extension .mp4 not in scan list, skipping ClamAV scan" -# "File large.zip (500MB) exceeds scan limit (200MB), skipping scan" -``` - -### Test Large Upload -```bash -# Your 970MB uploads should now show: -sudo tail -f /var/log/hmac-file-server/hmac-file-server.log | grep "skip\|scan\|upload" -``` - -## โœ… **Deployment Status** - -- **โœ… Configuration**: `maxscansize` now properly parsed from config -- **โœ… Extension Filter**: Video files automatically skipped -- **โœ… Size Limits**: Files >200MB bypass scanning entirely -- **โœ… Timeout Handling**: Progressive timeouts prevent hangs -- **โœ… Server**: Restarted with all fixes applied - -## ๐ŸŽฏ **Ready for Testing** - -Try uploading your large video file in Gajim again. You should see: - -1. **No "encrypting" delay** - upload starts immediately -2. **Logs show scan skip** - extension or size based -3. **Fast completion** - no virus scanning bottleneck -4. **Success message** - file uploaded and accessible - -The fix is **universal** and works for all file types and sizes while maintaining security for genuinely risky files! diff --git a/CLAMAV_SECURITY_CONFIG.md b/CLAMAV_SECURITY_CONFIG.md deleted file mode 100644 index b98aee1..0000000 --- a/CLAMAV_SECURITY_CONFIG.md +++ /dev/null @@ -1,163 +0,0 @@ -# ClamAV Security Configuration Template - -## ๐Ÿ”’ **Critical Security File Extensions** - -These are the file types that should ALWAYS be scanned by ClamAV as they can contain malicious code: - -### **Executable Files (HIGH RISK)** -```toml -# Windows executables -".exe", ".com", ".bat", ".cmd", ".scr", ".pif", ".dll", ".sys" - -# Unix/Linux executables -".sh", ".bash", ".csh", ".ksh", ".zsh", ".bin", ".run", ".deb", ".rpm" - -# Cross-platform -".jar", ".app", ".dmg", ".pkg" -``` - -### **Script Files (HIGH RISK)** -```toml -# Web scripts -".php", ".asp", ".aspx", ".jsp", ".cgi", ".pl", ".py", ".rb" - -# Office macros -".docm", ".xlsm", ".pptm", ".dotm", ".xltm", ".potm" - -# JavaScript/VBScript -".js", ".vbs", ".vbe", ".wsf", ".wsh" -``` - -### **Archive Files (MEDIUM RISK)** -```toml -# Compressed archives (can contain executables) -".zip", ".rar", ".7z", ".tar", ".gz", ".bz2", ".xz", ".tgz", ".tar.gz" - -# Installer packages -".msi", ".cab", ".iso" -``` - -### **Document Files (LOW-MEDIUM RISK)** -```toml -# Only if they support macros or embedding -".doc", ".docx", ".xls", ".xlsx", ".ppt", ".pptx", ".pdf" -``` - -## ๐ŸŽฏ **Recommended ClamAV Configuration** - -### **High Security (Paranoid)** -```toml -[clamav] -clamavenabled = true -maxscansize = "100MB" # Smaller limit for faster processing -scanfileextensions = [ - # Executables - ".exe", ".com", ".bat", ".cmd", ".scr", ".pif", ".dll", ".sys", - ".sh", ".bash", ".bin", ".run", ".deb", ".rpm", ".jar", ".app", - - # Scripts - ".php", ".asp", ".aspx", ".jsp", ".cgi", ".pl", ".py", ".rb", - ".js", ".vbs", ".vbe", ".wsf", ".wsh", - - # Macro documents - ".docm", ".xlsm", ".pptm", ".dotm", ".xltm", ".potm", - - # Archives - ".zip", ".rar", ".7z", ".tar", ".gz", ".bz2", ".tgz", ".msi", ".iso" -] -``` - -### **Balanced Security (Recommended)** -```toml -[clamav] -clamavenabled = true -maxscansize = "200MB" # Current setting -scanfileextensions = [ - # Critical executables only - ".exe", ".com", ".bat", ".cmd", ".scr", ".dll", - ".sh", ".bash", ".bin", ".jar", - - # High-risk scripts - ".php", ".asp", ".jsp", ".js", ".vbs", - - # Macro documents - ".docm", ".xlsm", ".pptm", - - # Compressed files - ".zip", ".rar", ".7z", ".tar.gz", ".msi" -] -``` - -### **Performance Optimized (Fast)** -```toml -[clamav] -clamavenabled = true -maxscansize = "50MB" # Smaller files only -scanfileextensions = [ - # Only the most dangerous - ".exe", ".com", ".bat", ".scr", ".dll", - ".sh", ".bin", ".jar", ".php", ".js", ".zip" -] -``` - -## ๐Ÿšซ **Files That Should NEVER Be Scanned** - -These file types are safe and scanning them wastes resources: - -```toml -# Media files (completely safe) -".mp4", ".avi", ".mov", ".mkv", ".wmv", ".flv", ".webm", -".mp3", ".wav", ".flac", ".aac", ".ogg", ".m4a", -".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".svg", ".webp", - -# Text/Data files (safe) -".txt", ".log", ".csv", ".json", ".xml", ".yaml", ".yml", - -# Large data files (safe, would be slow to scan) -".sql", ".dump", ".backup", ".tar.xz", ".img", ".vmdk" -``` - -## โšก **Performance Impact Analysis** - -| File Type | Size | Scan Time | Security Risk | Recommendation | -|-----------|------|-----------|---------------|----------------| -| `.exe` | 10MB | 2-5s | โš ๏ธ HIGH | Always scan | -| `.zip` | 50MB | 10-30s | โš ๏ธ MEDIUM | Scan if <200MB | -| `.mp4` | 1GB | 5+ minutes | โœ… NONE | Never scan | -| `.pdf` | 5MB | 1-3s | โš ๏ธ LOW | Optional | - -## ๐Ÿ”ง **Implementation for Production** - -Update `/etc/hmac-file-server/config.toml`: - -```toml -[clamav] -clamavenabled = true # Enable for security -maxscansize = "200MB" # Skip very large files -numscanworkers = 2 -clamavsocket = "/var/run/clamav/clamd.ctl" - -# CRITICAL SECURITY FILES ONLY -scanfileextensions = [ - # Windows executables - ".exe", ".com", ".bat", ".cmd", ".scr", ".dll", - - # Unix executables - ".sh", ".bash", ".bin", ".jar", - - # Dangerous scripts - ".php", ".asp", ".jsp", ".js", ".vbs", - - # Macro-enabled documents - ".docm", ".xlsm", ".pptm", - - # Compressed archives (can hide malware) - ".zip", ".rar", ".7z", ".tar.gz", ".msi" -] -``` - -This configuration: -- โœ… **Protects against malware** in dangerous file types -- โœ… **Skips harmless media files** entirely -- โœ… **Fast processing** for large uploads -- โœ… **Configurable** via standard config file diff --git a/DEDUPLICATION_1GB_OPTIMIZATION.md b/DEDUPLICATION_1GB_OPTIMIZATION.md deleted file mode 100644 index 98ac591..0000000 --- a/DEDUPLICATION_1GB_OPTIMIZATION.md +++ /dev/null @@ -1,81 +0,0 @@ -# Deduplication Optimization - 1GB Threshold - -## Updated Configuration - -### Production Setting Applied -```toml -[deduplication] -maxsize = "1GB" # Files larger than 1GB skip deduplication -enabled = true -directory = "/opt/hmac-file-server/data/dedup" -``` - -## Performance Impact - -### File Processing Logic -- **Small files (< 1GB)**: Get deduplication with SHA256 hash computation -- **Large files (โ‰ฅ 1GB)**: Skip deduplication entirely, upload at network speed -- **Most video files**: Will now bypass the hash computation that caused "endless encryption" - -### Why 1GB is Optimal -1. **Covers most media files**: Most video files, even high-quality ones, are under 1GB -2. **Still enables deduplication**: Smaller files (documents, images, small videos) still benefit -3. **Eliminates bottlenecks**: Very large files upload without processing delays -4. **Storage efficiency**: Deduplication remains active for the majority of files - -## Expected Results - -### Before (with no size limit) -``` -970MB video file upload: -1. Network transfer: ~30 seconds (depends on connection) -2. SHA256 computation: 2-5 minutes (CPU intensive) -3. Total time: 2.5-5.5 minutes + "endless encryption" appearance -``` - -### After (with 1GB limit) -``` -970MB video file upload: -1. Network transfer: ~30 seconds -2. SHA256 computation: SKIPPED -3. Total time: ~30 seconds + immediate completion -``` - -### Larger files (>1GB) -``` -Any file >1GB: -- Bypasses deduplication completely -- Uploads at pure network speed -- No processing delays -- Immediate completion -``` - -## Client Behavior Improvement - -### XMPP Clients (Gajim, Dino, Conversations) -- **Before**: Progress bar stuck on "encryption" for minutes -- **After**: Smooth progress at actual upload speed -- **User Experience**: Upload completes as expected without delays - -### File Types Affected -- **Videos (.mp4, .mkv, .avi, .mov)**: Major improvement for files approaching 1GB -- **Large archives (.zip, .tar, .7z)**: Faster uploads for big archives -- **ISO files**: No more delays on large disc images -- **High-res media**: Large photo/video collections upload quickly - -## System Status - -โœ… **Production Server**: Updated and running with 1GB threshold -โœ… **Docker Config**: Updated to match production settings -โœ… **nginx Timeouts**: Already configured for large file support (4800s) -โœ… **ClamAV**: Disabled to avoid scanning delays -โœ… **Upload Limits**: 10GB maximum file size supported - -## Monitoring - -The system is ready for immediate testing. Your large video files should now upload without the "endless encryption" delays you experienced before. - ---- -*Configuration Applied: $(date)* -*Deduplication Threshold: 1GB* -*Status: Optimized for large file uploads* diff --git a/DOCUMENTATION_UPDATE_SUMMARY.md b/DOCUMENTATION_UPDATE_SUMMARY.md deleted file mode 100644 index feda728..0000000 --- a/DOCUMENTATION_UPDATE_SUMMARY.md +++ /dev/null @@ -1,122 +0,0 @@ -# Documentation and Test Suite Update Summary - -## ๐Ÿ“ **Test Suite Organization** - -### New Structure -``` -tests/ -โ”œโ”€โ”€ README.md # Comprehensive testing documentation -โ”œโ”€โ”€ test_final_xmpp.sh # XEP-0363 protocol testing (v1,v2,v3,token) -โ”œโ”€โ”€ test_deduplication.sh # File deduplication testing -โ”œโ”€โ”€ test_upload_queue.sh # Queue performance testing -โ”œโ”€โ”€ comprehensive_upload_test.sh # Complete upload testing -โ”œโ”€โ”€ debug_upload.sh # Debugging utilities -โ”œโ”€โ”€ monitor_*.sh # Server monitoring scripts -โ”œโ”€โ”€ test_*.bin # Test data files (1MB, 50MB, 215MB, 4GB) -โ”œโ”€โ”€ test_*.txt # Text test files -โ””โ”€โ”€ xep0363_analysis.ipynb # Protocol analysis notebook -``` - -### Benefits -- โœ… **Clean project root**: Main directory focused on core files -- โœ… **Organized testing**: All test scripts and data centralized -- โœ… **Easy discovery**: Clear test documentation and examples -- โœ… **Comprehensive coverage**: Protocol, performance, and feature testing - -## ๐Ÿ“š **Documentation Updates** - -### README.md โœ… **UPDATED** -- โœ… Configuration examples updated to current field names -- โœ… Extended timeout values (4800s) for large files -- โœ… Deduplication settings with 1GB maxsize -- โœ… Dynamic worker scaling configuration -- โœ… New Testing section with quick examples -- โœ… Updated reverse proxy timeout recommendations - -### WIKI.md โœ… **UPDATED** -- โœ… Complete configuration section overhaul -- โœ… All field names updated to current structure -- โœ… Extended timeout documentation (4800s) -- โœ… Deduplication configuration with maxsize -- โœ… ClamAV selective scanning configuration -- โœ… Dynamic worker scaling documentation -- โœ… Configuration best practices section -- โœ… Example configurations updated - -### INSTALL.MD โœ… **UPDATED** -- โœ… Production configuration example updated -- โœ… Field names modernized -- โœ… Extended timeout recommendations - -### BUILD_GUIDE.md โœ… **UPDATED** -- โœ… Configuration examples updated -- โœ… Extended timeout values -- โœ… Dynamic worker scaling settings -- โœ… Deduplication configuration - -### NETWORK_RESILIENCE_GUIDE.md โœ… **UPDATED** -- โœ… Configuration syntax updated -- โœ… Extended timeout values -- โœ… Dynamic worker settings - -### PROTOCOL_SPECIFICATIONS.MD โœ… **CURRENT** -- โœ… Already up-to-date with current protocol implementations - -### Technical Fix Documents โœ… **CURRENT** -- โœ… LARGE_FILE_UPLOAD_FIX.md - Already references 4800s timeouts -- โœ… DEDUPLICATION_1GB_OPTIMIZATION.md - Current with 1GB maxsize -- โœ… FINAL_STATUS_REPORT.md - Comprehensive and current - -## ๐Ÿ”ง **Configuration Updates Applied** - -### Key Changes -1. **Field Name Modernization**: - - `listenport` โ†’ `listen_address` - - `storagepath` โ†’ `storage_path` - - `metricsenabled` โ†’ `metrics_enabled` - - `deduplicationenabled` โ†’ `deduplication_enabled` - -2. **Timeout Extensions**: - - All timeout values updated from 300s/3600s to 4800s - - Reverse proxy configurations updated to match - -3. **New Features Documented**: - - Dynamic worker scaling (`enable_dynamic_workers`) - - Deduplication size limits (`maxsize = "1GB"`) - - Selective ClamAV scanning (`scanfileextensions`, `maxscansize`) - - Extended resumable uploads (`max_resumable_age`) - -4. **Best Practices Added**: - - Performance optimization guidelines - - Large file handling recommendations - - Security considerations - - Monitoring guidance - -## ๐ŸŽฏ **Project Benefits** - -### Developer Experience -- โœ… **Clean workspace**: Easy navigation of core files -- โœ… **Comprehensive testing**: Complete test suite with documentation -- โœ… **Current documentation**: All examples work with latest configuration -- โœ… **Clear guidance**: Step-by-step setup and configuration instructions - -### User Experience -- โœ… **Accurate documentation**: Configuration examples that actually work -- โœ… **Extended timeout support**: Large file uploads properly documented -- โœ… **Performance optimization**: Best practices for production deployment -- โœ… **Testing tools**: Easy verification of functionality - -### Production Ready -- โœ… **4800s timeout configuration**: Handles GB-sized file uploads -- โœ… **Deduplication optimization**: 1GB limit prevents performance issues -- โœ… **Dynamic scaling**: Automatic worker adjustment for varying loads -- โœ… **Monitoring support**: Comprehensive testing and monitoring tools - -## ๐Ÿ“‹ **Next Steps** - -1. **Test the organized structure**: Run tests from new `tests/` directory -2. **Validate documentation**: Use updated configuration examples -3. **Monitor performance**: Utilize new monitoring scripts -4. **Scale as needed**: Leverage dynamic worker scaling for production loads - -The HMAC File Server 3.2 is now fully documented and tested with a clean, organized project structure! ๐Ÿš€ diff --git a/ENDLESS_ENCRYPTION_FIX.md b/ENDLESS_ENCRYPTION_FIX.md deleted file mode 100644 index 6a20ed0..0000000 --- a/ENDLESS_ENCRYPTION_FIX.md +++ /dev/null @@ -1,164 +0,0 @@ -# Large File Upload "Endless Encryption" Issue - Root Cause Analysis & Fix - -## Problem Identification - -### User Report -- **Symptom**: "small files yes works perfect but large are now taking from feeling endless - cause crypting takes endless" -- **Client Behavior**: Gajim, Dino, and Conversations all show "endless encryption" progress for large files -- **Evidence**: Screenshot shows "Completing my D...p.17 [WIEN].mp4" with progress bar stuck - -### Root Cause Analysis - -The "endless encryption" delay was **NOT** actually encryption, but **SHA256 hash computation** for deduplication: - -#### What Was Happening -1. **Deduplication Process**: Every uploaded file was being processed for SHA256 hash computation -2. **No Size Limit**: The production config had `deduplication_enabled = true` but no `maxsize` limit -3. **Hash Computation**: Large video files (like your .mp4) require reading the entire file to compute SHA256 -4. **Blocking Operation**: This hash computation was happening synchronously, blocking the upload completion - -#### Technical Details -```bash -# Before Fix - Production Config -[deduplication] -enabled = true -directory = "/opt/hmac-file-server/data/dedup" -# Missing: maxsize parameter - -# Result: ALL files processed for SHA256, including large videos -``` - -## The Fix Applied - -### 1. Added Deduplication Size Limit -```bash -# After Fix - Production Config -[deduplication] -maxsize = "100MB" # NEW: Skip deduplication for files > 100MB -enabled = true -directory = "/opt/hmac-file-server/data/dedup" -``` - -### 2. How The Fix Works -- **Small Files (< 100MB)**: Still get deduplication benefits with SHA256 processing -- **Large Files (> 100MB)**: Skip deduplication entirely, upload directly without hash computation -- **Performance**: Large video files now upload at network speed without processing delays - -### 3. Code Enhancement Verification -The server code already had the smart deduplication logic we implemented: - -```go -// From helpers.go - Enhanced deduplication function -func handleDeduplication(ctx context.Context, absFilename string) error { - // Parse maxsize from config, default to 500MB if not set - maxDedupSizeStr := conf.Deduplication.MaxSize - if maxDedupSizeStr != "" { - if parsedSize, parseErr := parseSize(maxDedupSizeStr); parseErr == nil { - maxDedupSize = parsedSize - } - } - - // Skip deduplication if file is too large - if info.Size() > maxDedupSize { - log.Debugf("File %s (%d bytes) exceeds deduplication size limit (%d bytes), skipping", - absFilename, info.Size(), maxDedupSize) - return nil - } - - // Only compute SHA256 for smaller files - // ... hash computation logic ... -} -``` - -## Why This Explains XMPP Client Behavior - -### Gajim, Dino, Conversations - Universal Issue -- **XEP-0363 Protocol**: All XMPP clients use the same HTTP File Upload protocol -- **Single PUT Request**: Must upload entire file in one HTTP request -- **Progress Indication**: Clients show "encryption" progress while waiting for server response -- **Server Processing**: Server was computing SHA256 hash before responding with success - -### The "Encryption" Confusion -- **Not Encryption**: No actual encryption was happening on large files -- **Hash Computation**: SHA256 deduplication hash was being computed -- **Client Perspective**: HTTP request in progress, showing as "encryption/processing" -- **Time Correlation**: Larger files = longer SHA256 computation = longer "encryption" display - -## Performance Impact - -### Before Fix -``` -Large File Upload Timeline: -1. Client starts PUT request -2. Server receives file -3. Server computes SHA256 hash (SLOW - minutes for large files) -4. Server stores file -5. Server responds with success -6. Client shows completion - -Total Time: Network transfer + SHA256 computation time -``` - -### After Fix -``` -Large File Upload Timeline: -1. Client starts PUT request -2. Server receives file -3. Server skips SHA256 (file > 100MB) -4. Server stores file directly -5. Server responds with success immediately -6. Client shows completion - -Total Time: Network transfer time only -``` - -## Configuration Verification - -### Current Production Settings -```toml -[server] -max_upload_size = "10GB" # Large file support -deduplication_enabled = true # Smart deduplication enabled - -[deduplication] -maxsize = "100MB" # NEW: Size limit for hash computation -enabled = true -directory = "/opt/hmac-file-server/data/dedup" - -[clamav] -clamavenabled = false # Disabled to avoid scanning delays -``` - -### nginx Timeout Configuration -```nginx -# HTTP Proxy - /etc/nginx/conf.d/share.conf -client_max_body_size 10240M; # 10GB support -proxy_read_timeout 4800s; # 80 minute timeout - -# Stream Proxy - /etc/nginx/nginx-stream.conf -proxy_timeout 4800s; # 80 minute stream timeout -``` - -## Testing Recommendation - -### Immediate Test -1. Try uploading the same large .mp4 file again -2. It should now complete quickly without "endless encryption" -3. Upload time should be roughly: file size รท internet upload speed - -### Monitoring -```bash -# Use the monitoring script to watch upload activity -/root/hmac-file-server/monitor_uploads.sh -``` - -## Summary - -The "endless encryption" issue was deduplication SHA256 hash computation running on all files regardless of size. By adding `maxsize = "100MB"` to the deduplication config, large files now bypass this processing and upload at full network speed while smaller files still benefit from deduplication. - -**Result**: Large file uploads should now complete in seconds/minutes instead of appearing to encrypt endlessly. - ---- -*Fix Applied: $(date)* -*Server Status: Running with optimizations* -*Issue: Resolved - Deduplication size limit implemented* diff --git a/FINAL_STATUS_REPORT.md b/FINAL_STATUS_REPORT.md deleted file mode 100644 index bce7748..0000000 --- a/FINAL_STATUS_REPORT.md +++ /dev/null @@ -1,129 +0,0 @@ -# XMPP Upload Issue Resolution Status Report -## Date: July 18, 2025 - -### ๐ŸŽฏ **PROBLEM SOLVED: HTTPRequestError:UNKNOWN: 0** - -## **Root Cause Analysis** -The `` error in Gajim, Dino, and Conversations was caused by: - -1. **HMAC Authentication Failures**: XMPP clients were receiving HTTP 401 responses -2. **Protocol Mismatch**: ejabberd was using v1/v2/token protocols with incorrect HMAC calculations -3. **Server Configuration Issues**: Initial `force_protocol = ""` caused startup failures - -## **Issues Resolved** โœ… - -### 1. **Server Startup Issue** -- **Problem**: `FATA[0000] Failed to initialize network protocol: invalid forceprotocol value:` -- **Solution**: Fixed `/etc/hmac-file-server/config.toml` by commenting out empty `force_protocol = ""` -- **Status**: โœ… RESOLVED - Server now uses `force_protocol = "auto"` - -### 2. **Performance Issues** -- **Problem**: "Endless encryption" delays from SHA256 deduplication computation -- **Solution**: Disabled deduplication (`deduplication_enabled = false`) -- **Status**: โœ… RESOLVED - No more computation delays - -### 3. **File Extension Blocking** -- **Problem**: Video files (.mp4, .mkv, etc.) were blocked by `global_extensions` -- **Solution**: Added video formats to allowed extensions list -- **Status**: โœ… RESOLVED - All file types now supported - -### 4. **ClamAV Scanning Delays** -- **Problem**: Large file scanning causing upload timeouts -- **Solution**: Disabled ClamAV (`clamavenabled = false`) -- **Status**: โœ… RESOLVED - No more scanning delays - -## **Protocol Testing Results** ๐Ÿงช - -### Working Protocols: -- **โœ… XEP-0363 v3**: HTTP 200 SUCCESS - - HMAC Format: `HMAC-SHA256(PUT\n{expires}\n{path})` - - URL Format: `https://share.uuxo.net/{path}?v3={hmac}&expires={timestamp}` - - **TEST CONFIRMED**: Multiple successful uploads - -### Failing Protocols: -- **โŒ v1 Protocol**: HTTP 401 (HMAC calculation mismatch) -- **โŒ v2 Protocol**: HTTP 401 (HMAC calculation mismatch) -- **โŒ token Protocol**: HTTP 401 (HMAC calculation mismatch) - -## **Current Infrastructure Status** ๐Ÿ”ง - -### Services Status: -- **โœ… HMAC File Server 3.2**: Active and running (PID: 2945780) -- **โœ… nginx Proxy**: Active with extended timeouts (4800s) -- **โœ… Redis**: Connected and operational -- **โœ… SSL/TLS**: Valid certificate for *.uuxo.net - -### Network Chain: -``` -XMPP Clients โ†’ ejabberd โ†’ Internet โ†’ -nginx:443 โ†’ nginx:4443 โ†’ HMAC:8080 -``` -- **โœ… All components verified working** - -### Configuration Highlights: -- **Max Upload**: 10GB -- **Timeouts**: 4800s (1.33 hours) -- **Extensions**: All video/document formats allowed -- **Deduplication**: Disabled for performance -- **ClamAV**: Disabled to avoid delays -- **Secret**: Configured and verified working - -## **Test Results Summary** ๐Ÿ“Š - -### Infrastructure Tests: -- **โœ… nginx proxy chain**: Requests properly routed -- **โœ… SSL certificate**: Valid and trusted -- **โœ… DNS resolution**: Working correctly -- **โœ… Backend connectivity**: HMAC server reachable - -### Upload Tests: -- **โœ… v3 Protocol**: Successfully uploaded multiple test files -- **โœ… File download**: Uploaded files accessible via HTTPS -- **โœ… Large files**: No timeout issues with extended configuration -- **โœ… Video files**: .mp4, .mkv, .avi all allowed - -## **Solution for XMPP Clients** ๐ŸŽฏ - -### **Immediate Fix**: -Configure ejabberd to use **XEP-0363 v3 protocol** which is confirmed working. - -### **ejabberd Configuration**: -Update your ejabberd configuration to use v3 HMAC generation: -```yaml -modules: - mod_http_upload_external: - protocol: v3 - secret: "f6g4ldPvQM7O2UTFeBEUUj33VrXypDAcsDt0yqKrLiOr5oQW" - hmac_calculation: "PUT\n{expires}\n{path}" -``` - -### **Alternative Solutions**: -1. **Option A**: Fix v1/v2/token HMAC calculations in ejabberd -2. **Option B**: Update XMPP clients to use v3 protocol URLs -3. **Option C**: Debug specific protocol ejabberd currently uses - -## **Files Successfully Tested** ๐Ÿ“ -- `/opt/hmac-file-server/data/uploads/xmpp_test_v3.txt` (35 bytes) -- `/opt/hmac-file-server/data/uploads/recheck_test.txt` (working) -- Multiple protocol variant tests completed - -## **Monitoring Tools Created** ๐Ÿ” -- `comprehensive_upload_test.sh`: Complete upload testing framework -- `monitor_nginx.sh`: nginx access log monitoring -- `monitor_server.sh`: HMAC server log monitoring -- `test_final_xmpp.sh`: Protocol-specific testing - -## **Next Steps** ๐Ÿš€ - -1. **Configure ejabberd** to use v3 protocol (confirmed working) -2. **Test with real XMPP clients** using v3 URLs -3. **Monitor upload success** with existing monitoring tools -4. **Optional**: Fix v1/v2 protocols if needed for backward compatibility - -## **Status**: ๐ŸŸข **RESOLVED** -**The HTTPRequestError:UNKNOWN: 0 issue is solved. v3 protocol works perfectly with proper HMAC authentication.** - ---- -*Report generated: $(date)* -*Server Version: HMAC File Server 3.2* -*Test Status: All critical tests passing* diff --git a/LARGE_FILE_UPLOAD_FIX.md b/LARGE_FILE_UPLOAD_FIX.md deleted file mode 100644 index d24d975..0000000 --- a/LARGE_FILE_UPLOAD_FIX.md +++ /dev/null @@ -1,106 +0,0 @@ -# Large File Upload Fix for XMPP Clients - -## Problem Analysis - -XMPP clients (Gajim, Dino, Conversations) were failing to upload files larger than ~100MB due to HMAC signature expiry. The XEP-0363 protocol uses signed URLs with limited validity periods that were too short for large file uploads. - -## Root Cause - -1. **HMAC Signature Expiry**: Prosody XMPP server generates signed URLs with 300s (5 minute) expiry -2. **Large Upload Duration**: 970MB file on 10 Mbps connection takes ~13 minutes -3. **Protocol Limitation**: XEP-0363 requires single HTTP PUT (no chunking) -4. **Timeline Conflict**: Upload duration exceeds signature validity - -## Solution Implemented - -### Server-Side Grace Period Extension - -Modified `validateV3HMAC()` function in `cmd/server/main.go` to: - -1. **Base Grace Period**: 1 hour (3600s) extension beyond expiry -2. **Large File Scaling**: Additional 1 minute per 50MB for files > 100MB -3. **Intelligent Detection**: Uses Content-Length header to determine file size -4. **Logging**: Detailed logging of grace period calculations - -### Algorithm Details - -``` -grace_period = 3600s (base) -if file_size > 100MB: - additional_time = (file_size / 50MB) * 60s - grace_period += additional_time - -effective_expiry = original_expiry + grace_period -``` - -### Example Calculations - -- **100MB file**: 3600s grace period (1 hour total) -- **500MB file**: 4200s grace period (70 minutes total) -- **970MB file**: 4560s grace period (76 minutes total) -- **2GB file**: 6000s grace period (100 minutes total) - -## Testing Recommendations - -1. **Monitor Logs**: Check for grace period messages during large uploads -2. **Test Progressive Sizes**: 100MB, 500MB, 1GB, 2GB files -3. **Multiple Clients**: Test with Gajim, Dino, and Conversations -4. **Network Variations**: Test on different connection speeds - -## Configuration Notes - -- **Server Timeouts**: Already set to 4800s (80 minutes) โœ… -- **nginx Timeouts**: Already set to 4800s (80 minutes) โœ… -- **Max Upload Size**: Already set to 10GB โœ… -- **Grace Period**: Now dynamically calculated โœ… - -## Client-Specific Considerations - -### Gajim -- Uses Python's requests library -- Default timeout ~300s (may need client-side config) -- Check `~/.config/gajim/config` for timeout settings - -### Dino -- Uses libsoup HTTP library -- Timeout behavior varies by version -- May need source modification for very large files - -### Conversations -- Android HTTP client with system timeouts -- Generally more tolerant of longer uploads -- Network changes can interrupt uploads - -## Future Improvements - -1. **Chunked Upload Extension**: Implement non-standard chunking for ultra-large files -2. **Progressive Upload**: Add resumable upload capability -3. **Client Timeout Detection**: Detect and handle client disconnections -4. **Bandwidth Estimation**: Dynamically adjust grace periods based on upload speed - -## Monitoring Commands - -```bash -# Check server logs for grace period usage -sudo journalctl -u hmac-file-server -f | grep -i grace - -# Monitor upload progress -sudo tail -f /var/log/hmac-file-server/hmac-file-server.log | grep -i upload - -# Check current connections -sudo netstat -tuln | grep :8080 -``` - -## Deployment Status - -โœ… **Fixed**: Large file upload signature expiry -โœ… **Deployed**: Updated server running with grace period extension -โœ… **Tested**: Server restart and basic functionality confirmed -๐Ÿ”„ **Next**: Test 970MB upload with XMPP client - -## Version Information - -- **Server Version**: HMAC File Server 3.2 -- **Fix Applied**: 2025-07-18 04:48:04 UTC -- **Grace Period**: Dynamic (1-100+ minutes based on file size) -- **Backward Compatibility**: Maintained for all existing uploads diff --git a/NETWORK_RESILIENCE_GUIDE.md b/NETWORK_RESILIENCE_GUIDE.md deleted file mode 100644 index 49c54b0..0000000 --- a/NETWORK_RESILIENCE_GUIDE.md +++ /dev/null @@ -1,251 +0,0 @@ -# Network Resilience Implementation Guide - -## Overview - -This implementation adds network switching resilience to the HMAC File Server **without modifying any core functions**. The solution uses a wrapper/middleware approach that enhances existing functionality while maintaining backward compatibility. - -## New Features Added - -### 1. Chunked/Resumable Uploads -- **Endpoint**: `POST/PUT /upload/chunked` -- **Status Check**: `GET /upload/status?session_id=` -- Breaks large uploads into smaller chunks (5MB default) -- Resumes interrupted uploads automatically -- Survives network switching scenarios - -### 2. Network Change Detection -- Monitors network interface status every 5 seconds -- Automatically pauses uploads during network changes -- Resumes uploads after network stabilization (2-second delay) -- Logs network events for monitoring - -### 3. Upload Session Persistence -- Stores upload state in Redis (if available) or local disk -- Sessions persist across server restarts -- Automatic cleanup of expired sessions (24-hour default) -- Tracks chunk completion status - -### 4. Enhanced Connection Management -- Configurable timeouts optimized for mobile networks -- Retry logic with exponential backoff -- Better keep-alive settings for mobile scenarios - -## Configuration - -Add these settings to your `config.toml`: - -```toml -[uploads] -chunked_uploads_enabled = true # Enable chunked uploads -resumable_uploads_enabled = true # Enable resumable functionality -chunk_size = "10MB" # Chunk size (optimized for performance) -max_resumable_age = "48h" # Session persistence time - -[timeouts] -readtimeout = "4800s" # 80 minutes (extended for large files) -writetimeout = "4800s" # 80 minutes (extended for large files) -idletimeout = "4800s" # 80 minutes (extended for large files) - -[server] -enable_dynamic_workers = true # Enable dynamic worker scaling -worker_scale_up_thresh = 50 # Scale up when queue reaches 50 -worker_scale_down_thresh = 10 # Scale down when queue drops to 10 - -[workers] -numworkers = 4 # Base number of workers -uploadqueuesize = 50 # Upload queue size - -[deduplication] -enabled = true # Enable file deduplication -maxsize = "1GB" # Deduplicate files under 1GB only - -[server] -networkevents = true # Enable network monitoring -``` - -## API Usage - -### Traditional Upload (existing, unchanged) -```bash -curl -X POST -F 'file=@large_file.zip' \ - -H "X-Signature: HMAC_SIGNATURE" \ - http://localhost:8080/upload -``` - -### New Chunked Upload - -#### 1. Start Upload Session -```bash -curl -X POST \ - -H "X-Filename: large_file.zip" \ - -H "X-Total-Size: 104857600" \ - -H "X-Signature: HMAC_SIGNATURE" \ - http://localhost:8080/upload/chunked - -# Response: -{ - "session_id": "1234567890_abc123", - "chunk_size": 5242880, - "total_chunks": 20 -} -``` - -#### 2. Upload Chunks -```bash -# Upload chunk 0 -curl -X PUT \ - -H "X-Upload-Session-ID: 1234567890_abc123" \ - -H "X-Chunk-Number: 0" \ - -H "Content-Type: application/octet-stream" \ - --data-binary @chunk_0.bin \ - http://localhost:8080/upload/chunked - -# Response: -{ - "success": true, - "chunk": 0, - "uploaded_bytes": 5242880, - "total_size": 104857600, - "progress": 0.05, - "completed": false -} -``` - -#### 3. Check Status -```bash -curl "http://localhost:8080/upload/status?session_id=1234567890_abc123" - -# Response: -{ - "session_id": "1234567890_abc123", - "filename": "large_file.zip", - "total_size": 104857600, - "uploaded_bytes": 52428800, - "progress": 0.5, - "completed": false, - "chunks": 10 -} -``` - -## Client-Side JavaScript Example - -```javascript -class NetworkResilientUploader { - constructor(file, endpoint, options = {}) { - this.file = file; - this.endpoint = endpoint; - this.chunkSize = options.chunkSize || 5 * 1024 * 1024; // 5MB - this.maxRetries = options.maxRetries || 5; - } - - async upload() { - // Start session - const session = await this.startSession(); - console.log('Upload session started:', session.session_id); - - // Upload chunks - const totalChunks = Math.ceil(this.file.size / this.chunkSize); - for (let i = 0; i < totalChunks; i++) { - await this.uploadChunk(session.session_id, i); - console.log(`Chunk ${i + 1}/${totalChunks} uploaded`); - } - - console.log('Upload completed!'); - } - - async startSession() { - const response = await fetch(this.endpoint, { - method: 'POST', - headers: { - 'X-Filename': this.file.name, - 'X-Total-Size': this.file.size.toString(), - 'X-Signature': 'YOUR_HMAC_SIGNATURE' - } - }); - return response.json(); - } - - async uploadChunk(sessionId, chunkNumber, retryCount = 0) { - try { - const start = chunkNumber * this.chunkSize; - const end = Math.min(start + this.chunkSize, this.file.size); - const chunk = this.file.slice(start, end); - - const response = await fetch(this.endpoint, { - method: 'PUT', - headers: { - 'X-Upload-Session-ID': sessionId, - 'X-Chunk-Number': chunkNumber.toString() - }, - body: chunk - }); - - if (!response.ok) throw new Error(`HTTP ${response.status}`); - return response.json(); - - } catch (error) { - if (retryCount < this.maxRetries) { - const delay = Math.pow(2, retryCount) * 1000 + Math.random() * 1000; - await new Promise(resolve => setTimeout(resolve, delay)); - return this.uploadChunk(sessionId, chunkNumber, retryCount + 1); - } - throw error; - } - } -} - -// Usage -const uploader = new NetworkResilientUploader(fileInput.files[0], '/upload/chunked'); -uploader.upload().catch(console.error); -``` - -## Benefits for Network Switching - -### Before (Problems) -- โŒ Upload fails completely on network interruption -- โŒ User loses all progress when switching WiFi/WLAN -- โŒ Large files problematic on mobile networks -- โŒ No recovery from temporary connection issues - -### After (Solutions) -- โœ… Upload resumes automatically after network switch -- โœ… Progress preserved during network interruptions -- โœ… Optimal chunk sizes for mobile networks -- โœ… Automatic retry with exponential backoff -- โœ… Network change detection and graceful handling -- โœ… Session persistence across server restarts - -## Monitoring - -Check logs for network resilience status: -```bash -# Network change detection -2025-07-17 12:34:56 INFO Network change detected -2025-07-17 12:34:56 INFO Pausing all active uploads due to network change -2025-07-17 12:34:58 INFO Resuming all paused uploads - -# Upload session activity -2025-07-17 12:35:00 INFO Active upload sessions: 3 -2025-07-17 12:35:00 INFO Network resilience: 5 uploads active -``` - -## Implementation Notes - -### Non-Intrusive Design -- **No core function modifications**: All existing upload handlers remain unchanged -- **Backward compatibility**: Traditional uploads continue to work exactly as before -- **Optional features**: Network resilience only activates when `chunkeduploadsenabled = true` -- **Gradual adoption**: Users can test chunked uploads alongside existing uploads - -### File Structure -``` -cmd/server/ -โ”œโ”€โ”€ main.go # Core server (minimal changes) -โ”œโ”€โ”€ helpers.go # Router setup (minimal changes) -โ”œโ”€โ”€ upload_session.go # NEW: Session management -โ”œโ”€โ”€ network_resilience.go # NEW: Network monitoring -โ”œโ”€โ”€ chunked_upload_handler.go # NEW: Chunked upload API -โ””โ”€โ”€ integration.go # NEW: Non-intrusive integration -``` - -This approach ensures that your existing upload functionality remains stable while adding robust network switching capabilities for users who need them. diff --git a/NETWORK_SWITCHING_IMPROVEMENTS.md b/NETWORK_SWITCHING_IMPROVEMENTS.md deleted file mode 100644 index 15b4978..0000000 --- a/NETWORK_SWITCHING_IMPROVEMENTS.md +++ /dev/null @@ -1,162 +0,0 @@ -# Network Switching Improvements for HMAC File Server - -## โœ… Implementation Complete - -The network resilience features have been successfully implemented and are ready to use! - -### ๐Ÿš€ Quick Start - -1. **Build the enhanced server:** - ```bash - ./buildgo.sh - ``` - -2. **Use the network-resilient configuration:** - ```bash - cp config-network-resilience.toml config.toml - # Edit config.toml with your settings - ``` - -3. **Start the server:** - ```bash - ./hmac-file-server --config config.toml - ``` - -4. **Test the features:** - ```bash - ./test_network_resilience.sh - ``` - -## ๐Ÿ”ง What's Been Fixed - -### โœ… Implementation Status - -### โœ… Implementation Status - -| Feature | Status | Description | -|---------|--------|-------------| -| **Chunked/Resumable Uploads** | โœ… **IMPLEMENTED** | 5MB chunks, survives network interruptions | -| **Network Change Detection** | โœ… **IMPLEMENTED** | Monitors interfaces, pauses/resumes uploads | -| **Session Persistence** | โœ… **IMPLEMENTED** | Redis or disk storage, 24h default timeout | -| **Enhanced Timeouts** | โœ… **IMPLEMENTED** | 5-minute read/write, 10-minute idle | -| **Retry Logic** | โœ… **IMPLEMENTED** | Exponential backoff with jitter | -| **Backward Compatibility** | โœ… **GUARANTEED** | Zero changes to existing upload handlers | - -### ๐Ÿ“‚ New Files Added - -- `cmd/server/upload_session.go` - Session management and persistence -- `cmd/server/network_resilience.go` - Network monitoring and pause/resume -- `cmd/server/chunked_upload_handler.go` - New chunked upload endpoint -- `cmd/server/integration.go` - Non-intrusive integration layer -- `config-network-resilience.toml` - Ready-to-use configuration -- `test_network_resilience.sh` - Automated testing script - -### ๐ŸŒ New API Endpoints - -| Endpoint | Method | Purpose | -|----------|--------|---------| -| `/chunked-upload` | POST | Start new chunked upload session | -| `/chunked-upload` | PUT | Upload individual chunks | -| `/upload-status` | GET | Check upload progress | -| `/upload` | POST | Traditional uploads (unchanged) | - -## ๐Ÿ“ฑ Network Switching Benefits - -### Before (Problems Fixed) -- โŒ Upload fails completely on network interruption -- โŒ Progress lost when switching WiFi/WLAN -- โŒ Large files problematic on mobile networks -- โŒ No recovery from connection drops - -### After (Solutions Implemented) -- โœ… **Seamless network switching** - uploads pause and resume automatically -- โœ… **Progress preservation** - no lost data during interruptions -- โœ… **Mobile optimized** - 5MB chunks perfect for cellular/WiFi -- โœ… **Intelligent retry** - exponential backoff handles temporary failures -- โœ… **Session persistence** - survives server restarts - -## ๐Ÿ“‹ Usage Examples - -### Traditional Upload (Unchanged) -```bash -curl -X POST -H "X-Signature: HMAC" -F 'file=@document.pdf' http://localhost:8080/upload -``` - -### New Chunked Upload -```bash -# 1. Start session -curl -X POST \ - -H "X-Filename: large_video.mp4" \ - -H "X-Total-Size: 104857600" \ - -H "X-Signature: HMAC" \ - http://localhost:8080/chunked-upload - -# 2. Upload chunks (automatically handles network switches) -curl -X PUT \ - -H "X-Upload-Session-ID: session_123" \ - -H "X-Chunk-Number: 0" \ - --data-binary @chunk_0.bin \ - http://localhost:8080/chunked-upload - -# 3. Check progress -curl "http://localhost:8080/upload-status?session_id=session_123" -``` - -## โš™๏ธ Configuration - -Essential settings for network resilience: - -```toml -[server] -networkevents = true # Enable network monitoring - -[uploads] -chunkeduploadsenabled = true # Enable chunked uploads -chunksize = "5MB" # Optimal for mobile -sessiontimeout = "24h" # Session persistence - -[timeouts] -readtimeout = "300s" # 5 minutes for mobile -writetimeout = "300s" # 5 minutes for slow networks -idletimeout = "600s" # 10 minutes keep-alive -``` - -## ๐Ÿ”จ Build & Deploy - -### Build Process -```bash -# Automatic build with network resilience detection -./buildgo.sh - -# Output confirms features are included: -# [INFO] Found network resilience: upload_session.go -# [INFO] Found network resilience: network_resilience.go -# [INFO] Found network resilience: chunked_upload_handler.go -# [INFO] Found network resilience: integration.go -# [BUILD] Build successful! Binary created: ./hmac-file-server -``` - -### Testing -```bash -# Comprehensive feature testing -./test_network_resilience.sh - -# Expected output: -# โœ… Traditional upload works -# โœ… Chunked upload session creation works -# โœ… Upload status endpoint works -# โœ… Health endpoint works -# โœ… Metrics endpoint works -``` - -## ๐ŸŽฏ Perfect for Your Use Case - -This implementation specifically solves your **notebook network switching** problem: - -1. **WLAN โ†” WiFi Switching**: Uploads automatically pause during network changes and resume when stable -2. **Mobile-Friendly**: 5MB chunks work well over cellular and WiFi connections -3. **Android/iOS Compatible**: HTTP-based solution works with any mobile platform -4. **Zero Disruption**: Existing users see no changes, new users get enhanced reliability -5. **Production Ready**: Full session persistence, monitoring, and error handling - -Your users can now upload large files from notebooks/mobile devices without worrying about network switching interrupting their transfers! ๐Ÿš€ diff --git a/PERFORMANCE_OPTIMIZATION.md b/PERFORMANCE_OPTIMIZATION.md deleted file mode 100644 index 024d414..0000000 --- a/PERFORMANCE_OPTIMIZATION.md +++ /dev/null @@ -1,157 +0,0 @@ -# Optimized Configuration for Large File Performance - -## ๐ŸŽฏ **Root Cause of "Feeling the Same" Issue** - -The problem was **deduplication post-processing** - after uploads complete, the server was: -1. Computing SHA256 hash of entire file (970MB = ~30-60 seconds) -2. Moving files and creating hard links -3. This happened **after** upload but **before** client success response - -## โœ… **Performance Optimizations Applied** - -### 1. **Smart Deduplication Size Limits** -```toml -[deduplication] -enabled = true -directory = "/opt/hmac-file-server/data/dedup" -maxsize = "500MB" # NEW: Skip deduplication for files >500MB -``` - -### 2. **Enhanced ClamAV Security Configuration** -```toml -[clamav] -clamavenabled = true -maxscansize = "200MB" -numscanworkers = 2 -clamavsocket = "/var/run/clamav/clamd.ctl" - -# ONLY scan genuinely dangerous file types -scanfileextensions = [ - # Critical executables - ".exe", ".com", ".bat", ".cmd", ".scr", ".dll", ".sys", - ".sh", ".bash", ".bin", ".run", ".jar", ".app", - - # Dangerous scripts - ".php", ".asp", ".aspx", ".jsp", ".js", ".vbs", ".py", - - # Macro-enabled documents - ".docm", ".xlsm", ".pptm", ".dotm", ".xltm", ".potm", - - # Compressed archives (can hide malware) - ".zip", ".rar", ".7z", ".tar", ".gz", ".msi", ".iso" -] -``` - -### 3. **Files That Should NEVER Be Scanned/Deduplicated** -```bash -# Media files (safe, large, unique) -.mp4, .avi, .mov, .mkv, .wmv, .flv, .webm -.mp3, .wav, .flac, .aac, .ogg, .m4a -.jpg, .jpeg, .png, .gif, .bmp, .tiff, .svg - -# Large data files (safe, often unique) -.sql, .dump, .backup, .img, .vmdk, .vdi -``` - -## ๐Ÿš€ **Expected Performance Improvements** - -| File Type | Size | Before Fix | After Fix | Improvement | -|-----------|------|------------|-----------|-------------| -| `.mp4` video | 970MB | โŒ 60s dedup delay | โœ… Instant | **60x faster** | -| `.exe` binary | 50MB | โš ๏ธ Slow scan + dedup | โœ… Fast scan only | **10x faster** | -| `.zip` archive | 200MB | โš ๏ธ Slow scan + dedup | โœ… Skip both | **20x faster** | -| `.txt` document | 1MB | โœ… Fast | โœ… Fast | No change | - -## ๐Ÿ”ง **Recommended Production Configuration** - -### **High Performance Setup** -```toml -[server] -max_upload_size = "10GB" -deduplication_enabled = true - -[deduplication] -enabled = true -directory = "/opt/hmac-file-server/data/dedup" -maxsize = "100MB" # Only deduplicate small files - -[clamav] -clamavenabled = true -maxscansize = "50MB" # Only scan small potentially dangerous files -scanfileextensions = [".exe", ".com", ".bat", ".scr", ".dll", ".sh", ".jar", ".zip", ".rar"] -``` - -### **Balanced Security/Performance Setup** -```toml -[deduplication] -enabled = true -maxsize = "500MB" # Medium-sized files get deduplicated - -[clamav] -clamavenabled = true -maxscansize = "200MB" # Current setting -scanfileextensions = [ - ".exe", ".com", ".bat", ".cmd", ".scr", ".dll", - ".sh", ".bash", ".bin", ".jar", ".php", ".js", - ".zip", ".rar", ".7z", ".tar.gz" -] -``` - -### **Maximum Security Setup** -```toml -[deduplication] -enabled = false # Disable for maximum speed - -[clamav] -clamavenabled = true -maxscansize = "1GB" # Scan larger files -scanfileextensions = [ - # All potentially dangerous types - ".exe", ".com", ".bat", ".cmd", ".scr", ".dll", ".sys", - ".sh", ".bash", ".bin", ".jar", ".php", ".asp", ".js", - ".doc", ".docx", ".xls", ".xlsx", ".pdf", - ".zip", ".rar", ".7z", ".tar", ".gz", ".iso" -] -``` - -## ๐Ÿ“Š **File Type Classification** - -### **Critical Security Risk (Always Scan)** -- Executables: `.exe`, `.com`, `.bat`, `.scr`, `.dll`, `.sys` -- Scripts: `.sh`, `.bash`, `.php`, `.js`, `.py`, `.vbs` -- System files: `.jar`, `.app`, `.deb`, `.rpm`, `.msi` - -### **Medium Risk (Scan if Small)** -- Documents: `.doc`, `.docx`, `.xls`, `.xlsx`, `.pdf` -- Archives: `.zip`, `.rar`, `.7z`, `.tar.gz` - -### **No Security Risk (Never Scan)** -- Media: `.mp4`, `.avi`, `.mp3`, `.jpg`, `.png` -- Data: `.txt`, `.csv`, `.json`, `.log`, `.sql` - -## ๐Ÿ” **Monitoring Commands** - -### Check Deduplication Skips -```bash -sudo journalctl -u hmac-file-server -f | grep -i "exceeds deduplication size limit" -``` - -### Check ClamAV Skips -```bash -sudo journalctl -u hmac-file-server -f | grep -i "exceeds.*scan limit\|not in scan list" -``` - -### Monitor Upload Performance -```bash -sudo tail -f /var/log/hmac-file-server/hmac-file-server.log | grep -E "(upload|dedup|scan)" -``` - -## โœ… **Current Status** - -- **โœ… ClamAV**: Smart size and extension filtering -- **โœ… Deduplication**: Size-based skipping (default 500MB limit) -- **โœ… Performance**: Large files bypass both bottlenecks -- **โœ… Security**: Maintained for genuinely risky file types -- **โœ… Configurable**: All limits adjustable via config.toml - -Large uploads should now complete **immediately** without post-processing delays! diff --git a/PROTOCOL_SPECIFICATIONS.MD b/PROTOCOL_SPECIFICATIONS.MD deleted file mode 100644 index 7bb9ddf..0000000 --- a/PROTOCOL_SPECIFICATIONS.MD +++ /dev/null @@ -1,295 +0,0 @@ -# HMAC File Server Authentication Protocol Specifications - -This document outlines the different authentication protocols supported by the HMAC File Server for secure file uploads and downloads. The server supports multiple authentication methods to ensure backward compatibility while providing enhanced security features. - -## Overview - -The HMAC File Server supports two primary authentication mechanisms: -1. **HMAC-based Authentication** (Multiple versions: v1, v2, token, v3) -2. **JWT Authentication** (Bearer tokens) - -All protocols use SHA256 hashing and require a shared secret key configured on the server. - ---- - -## HMAC Authentication Protocols - -### Common Elements -- **Algorithm**: HMAC-SHA256 -- **Secret**: Shared secret key configured in `[security]` section -- **Transport**: URL query parameters for HMAC, headers for signatures -- **Encoding**: Hexadecimal encoding for HMAC values - ---- - -### Legacy v1 Protocol (`v` parameter) - -**Overview**: The original HMAC authentication protocol. - -**URL Format**: -``` -PUT /filename.ext?v=HMAC_SIGNATURE -``` - -**Message Construction**: -``` -fileStorePath + "\x20" + contentLength -``` - -**Example**: -```bash -# For file "test.txt" with 1024 bytes -# Message: "test.txt\x201024" -curl -X PUT "http://localhost:8080/test.txt?v=a1b2c3d4..." --data-binary @test.txt -``` - -**Implementation Notes**: -- Uses space character (`\x20`) as separator -- Content-Length header must be accurate -- Simplest protocol, minimal metadata validation - ---- - -### Enhanced v2 Protocol (`v2` parameter) - -**Overview**: Enhanced version including content type validation. - -**URL Format**: -``` -PUT /filename.ext?v2=HMAC_SIGNATURE -``` - -**Message Construction**: -``` -fileStorePath + "\x00" + contentLength + "\x00" + contentType -``` - -**Example**: -```bash -# For file "document.pdf" with 2048 bytes -# Message: "document.pdf\x002048\x00application/pdf" -curl -X PUT "http://localhost:8080/document.pdf?v2=e5f6g7h8..." --data-binary @document.pdf -``` - -**Implementation Notes**: -- Uses null characters (`\x00`) as separators -- Content-Type automatically detected from file extension -- Fallback to "application/octet-stream" for unknown extensions - ---- - -### Token Protocol (`token` parameter) - -**Overview**: Alternative parameter name for v2-style authentication. - -**URL Format**: -``` -PUT /filename.ext?token=HMAC_SIGNATURE -``` - -**Message Construction**: Same as v2 protocol -``` -fileStorePath + "\x00" + contentLength + "\x00" + contentType -``` - -**Example**: -```bash -curl -X PUT "http://localhost:8080/image.jpg?token=i9j0k1l2..." --data-binary @image.jpg -``` - -**Implementation Notes**: -- Identical to v2 protocol but uses `token` parameter -- Useful for clients that prefer different parameter naming - ---- - -### v3 Protocol - mod_http_upload_external Compatible (`v3` parameter) - -**Overview**: Specifically designed for Prosody's `mod_http_upload_external` compatibility with expiration support. - -**URL Format**: -``` -PUT /path/to/file.ext?v3=HMAC_SIGNATURE&expires=UNIX_TIMESTAMP -``` - -**Message Construction**: -``` -METHOD + "\n" + expires + "\n" + requestPath -``` - -**Example**: -```bash -# Current timestamp: 1717804800 -# Message: "PUT\n1717804800\n/upload/myfile.txt" -curl -X PUT "http://localhost:8080/upload/myfile.txt?v3=m3n4o5p6...&expires=1717804800" --data-binary @myfile.txt -``` - -**Verification Process**: -1. Extract `v3` signature and `expires` timestamp -2. Validate `expires` is in the future -3. Construct message: `"{METHOD}\n{expires}\n{path}"` -4. Calculate HMAC-SHA256 of message -5. Compare with provided signature - -**Implementation Notes**: -- Includes expiration timestamp validation -- Prevents replay attacks through time-based validation -- Path-only signing (no query parameters in signed message) -- HTTP method is part of the signed message - ---- - -## JWT Authentication - -**Overview**: Token-based authentication using JSON Web Tokens. - -**Configuration**: -```toml -[security] -enablejwt = true -jwtsecret = "your-256-bit-secret" -jwtalgorithm = "HS256" -jwtexpiration = "24h" -``` - -**Header Format**: -``` -Authorization: Bearer JWT_TOKEN -``` - -**Fallback Query Parameter**: -``` -GET /file.txt?token=JWT_TOKEN -``` - -**Example Usage**: -```bash -# Header-based JWT -curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIs..." http://localhost:8080/file.txt - -# Query parameter fallback -curl "http://localhost:8080/file.txt?token=eyJhbGciOiJIUzI1NiIs..." -``` - -**JWT Claims**: Standard JWT claims (exp, iat, iss, etc.) as configured. - ---- - -## POST Upload Authentication - -### X-Signature Header Method - -**Overview**: For multipart form uploads via POST requests. - -**Header Format**: -``` -X-Signature: HMAC_OF_REQUEST_PATH -``` - -**Message Construction**: -``` -requestPath (e.g., "/upload") -``` - -**Example**: -```bash -# HMAC of "/upload" -curl -X POST \ - -H "X-Signature: CALCULATED_HMAC" \ - -F 'file=@myfile.txt' \ - http://localhost:8080/upload -``` - ---- - -## Authentication Priority and Fallbacks - -The server checks authentication in the following order: - -1. **JWT Authentication** (if `enablejwt = true`) - - Authorization header (Bearer token) - - Query parameter `token` - -2. **HMAC Authentication** (if JWT disabled or not found) - - X-Signature header (for POST uploads) - - v3 protocol (with expires validation) - - v2 protocol - - token protocol - - v1 protocol (legacy) - ---- - -## Security Considerations - -### HMAC Protocols -- **Secret Management**: Use strong, randomly generated secrets -- **Time Validation**: v3 protocol includes expiration to prevent replay attacks -- **Content Validation**: v2/token protocols include content-type validation -- **Path Sanitization**: All protocols validate and sanitize file paths - -### JWT Authentication -- **Token Expiration**: Configure appropriate expiration times -- **Secret Rotation**: Regularly rotate JWT signing keys -- **Algorithm Security**: Default HS256 is secure for most use cases -- **Transport Security**: Always use HTTPS in production - -### General Security -- **HTTPS Only**: Use TLS encryption for all production deployments -- **Rate Limiting**: Implement reverse proxy rate limiting -- **File Validation**: Configure allowed file extensions -- **Virus Scanning**: Enable ClamAV integration for malware detection -- **Access Logs**: Monitor authentication failures and suspicious activity - ---- - -## Migration Guide - -### From v1 to v2 -- Update HMAC calculation to include content type -- Change separator from `\x20` to `\x00` -- No breaking changes in URL structure - -### From HMAC to JWT -- Set `enablejwt = true` in configuration -- Generate JWT tokens server-side or use external auth provider -- HMAC authentication remains available as fallback - -### Adding v3 Support -- Implement expiration timestamp generation -- Update HMAC calculation to include HTTP method and expiration -- Useful for mod_http_upload_external compatibility - ---- - -## Example Implementations - -### Client-Side HMAC Generation (Python) -```python -import hmac -import hashlib -import time - -def generate_v3_signature(secret, method, expires, path): - message = f"{method}\n{expires}\n{path}" - signature = hmac.new( - secret.encode(), - message.encode(), - hashlib.sha256 - ).hexdigest() - return signature - -# Example usage -secret = "your-hmac-secret" -expires = int(time.time()) + 3600 # 1 hour from now -signature = generate_v3_signature(secret, "PUT", expires, "/upload/file.txt") -``` - -### Server-Side Validation (Reference) -See the main.go file for complete implementation details of all validation functions: -- `validateHMAC()`: Legacy v1, v2, and token protocols -- `validateV3HMAC()`: v3 protocol with expiration -- `validateJWTFromRequest()`: JWT validation - ---- - -This specification ensures consistent implementation across clients and provides multiple authentication options for different use cases and security requirements. diff --git a/README.MD b/README.MD deleted file mode 100644 index 41c6837..0000000 --- a/README.MD +++ /dev/null @@ -1,570 +0,0 @@ -# HMAC File Server 3.2 - -## Overview -The **HMAC File Server** ensures secure file uploads and downloads using HMAC authentication and JWT tokens. It incorporates comprehensive security features, file versioning, deduplication, ISO container support, virus scanning, and Unix socket support for enhanced flexibility. Redis integration provides efficient caching and session management. Prometheus metrics and graceful shutdown mechanisms ensure reliable and efficient file handling. - -Special thanks to **Thomas Leister** for inspiration drawn from [prosody-filer](https://git.uuxo.net/uuxo/prosody-filer). - -## Features -- **Multiple Authentication Methods**: HMAC-based authentication and JWT token support -- **Multiple Protocol Support**: v1 (legacy), v2, v3 (mod_http_upload_external), and token-based uploads -- **File Management**: Deduplication, configurable TTL for automatic file cleanup -- **Upload Methods**: POST multipart uploads, PUT uploads for legacy protocols, v3 protocol support -- **Security**: Virus scanning via ClamAV, configurable file extensions validation -- **Performance**: Chunked uploads and downloads, worker pool management with auto-scaling -- **Storage Options**: Local storage, ISO container mounting for specialized needs -- **Monitoring**: Prometheus metrics integration with detailed system and operation metrics -- **Network Support**: IPv4/IPv6 dual-stack support with protocol forcing options -- **Configuration**: Hot-reloading of logging settings via SIGHUP signal - -## Table of Contents -1. [Installation](#installation) -2. [Configuration](#configuration) -3. [Authentication](#authentication) -4. [API Endpoints](#api-endpoints) -5. [Usage Examples](#usage-examples) -6. [Testing](#testing) -7. [Setup](#setup) - - [Reverse Proxy](#reverse-proxy) - - [Systemd Service](#systemd-service) -8. [Building](#building) -9. [Docker Support](#docker-support) -10. [Changelog](#changelog) -11. [License](#license) - ---- - -## Installation - -### Quick Installation for XMPP Operators - -The easiest way to install HMAC File Server is using the automated installer: - -```bash -git clone https://git.uuxo.net/uuxo/hmac-file-server.git -cd hmac-file-server -sudo ./installer.sh -``` - -The installer supports two deployment options: -- **Native Installation**: Traditional systemd service with Go build -- **Docker Deployment**: Containerized deployment with docker-compose - -**Native installation** features: -- Install Go 1.24 (if needed) -- Create system user and directories -- Build and configure the server -- Set up systemd service -- Optionally install Redis and ClamAV - -**Docker deployment** features: -- Complete docker-compose setup with Redis and ClamAV -- Multi-stage Dockerfile for optimized builds -- Automated container orchestration -- Easy start/stop scripts - -For detailed installation instructions, see [INSTALL.MD](INSTALL.MD). - -### Manual Installation - -> **Tip:** The automated installer now supports both native systemd installation and Docker deployment. For automated Docker setup, run `sudo ./installer.sh` and select option 2. For manual Docker setup, see the Wiki for detailed instructions. The official image is available at `ghcr.io/plusone/hmac-file-server:latest`. - -#### Prerequisites -- Go **1.24** or higher -- Redis server (optional, for caching) -- ClamAV (optional, for virus scanning) - -#### Steps -1. Clone the repository: - ```bash - git clone https://git.uuxo.net/uuxo/hmac-file-server.git - cd hmac-file-server - ``` - -2. Build the server: - ```bash - go build -o hmac-file-server ./cmd/server/main.go - ``` - -3. Generate example configuration: - ```bash - ./hmac-file-server -genconfig - # or write to file: - ./hmac-file-server -genconfig-path config.toml - ``` - -4. Create necessary directories: - ```bash - mkdir -p /path/to/hmac-file-server/data/ - mkdir -p /path/to/hmac-file-server/deduplication/ - ``` - -5. Edit your `config.toml` file with appropriate settings. - -6. Start the server: - ```bash - ./hmac-file-server -config config.toml - ``` - ---- - -## Uninstallation - -The installer script provides comprehensive uninstallation options with data preservation: - -```bash -sudo ./installer.sh --uninstall -``` - -### Uninstall Options - -The uninstaller offers five data handling options: - -1. **๐Ÿ—‘๏ธ Delete all data** - Complete removal (requires typing "DELETE" to confirm) -2. **๐Ÿ’พ Preserve uploads and deduplication data** - Keeps important files, removes logs -3. **๐Ÿ“‹ Preserve all data** - Keeps uploads, deduplication data, and logs -4. **๐ŸŽฏ Custom selection** - Choose exactly what to preserve -5. **โŒ Cancel** - Exit without making changes - -### Data Preservation - -When preserving data, the uninstaller: -- Creates timestamped backups in `/var/backups/hmac-file-server-YYYYMMDD-HHMMSS/` -- Shows file counts and sizes before deletion decisions -- Safely moves data to backup locations -- Provides clear feedback on what was preserved or removed - -### What Gets Removed - -The uninstaller always removes: -- โœ“ Systemd service and service file -- โœ“ Installation directory (`/opt/hmac-file-server`) -- โœ“ Configuration directory (`/etc/hmac-file-server`) -- โœ“ System user (`hmac-server`) -- โœ“ Binary files in common locations - -Data directories are handled according to your selection. - ---- - -## Configuration - -The server uses a comprehensive `config.toml` file with the following main sections: - -### Key Configuration Sections - -- **[server]**: Basic server settings (port, storage, metrics, dynamic workers) -- **[security]**: HMAC secrets, JWT configuration -- **[uploads/downloads]**: File handling settings, allowed extensions, chunked transfers -- **[logging]**: Log levels, file rotation settings -- **[deduplication]**: File deduplication settings and storage efficiency -- **[clamav]**: Virus scanning configuration with selective scanning -- **[redis]**: Cache and session management -- **[workers]**: Thread pool and performance tuning with auto-scaling -- **[iso]**: ISO container mounting (specialized storage) -- **[timeouts]**: HTTP timeout configurations for large file handling -- **[versioning]**: File versioning and history management - -### Example Configuration - -```toml -[server] -listen_address = ":8080" -storage_path = "/srv/hmac-file-server/uploads" -metrics_enabled = true -metrics_path = "/metrics" -max_upload_size = "10GB" -max_header_bytes = 1048576 -cleanup_interval = "24h" -max_file_age = "720h" -deduplication_enabled = true -min_free_bytes = "1GB" -file_naming = "original" # Options: "original", "HMAC" -force_protocol = "" # Options: "http", "https" - if set, redirects -enable_dynamic_workers = true -worker_scale_up_thresh = 50 -worker_scale_down_thresh = 10 - -[security] -secret = "your-secure-hmac-secret-64-chars-long" -enablejwt = false -jwtsecret = "your-jwt-secret" -jwtalgorithm = "HS256" -jwtexpiration = "24h" - -[uploads] -allowed_extensions = [".zip", ".rar", ".7z", ".tar.gz", ".tgz", ".gpg", ".enc", ".pgp"] -chunked_uploads_enabled = true -chunk_size = "10MB" -resumable_uploads_enabled = true -max_resumable_age = "48h" - -[downloads] -resumable_downloads_enabled = true -chunked_downloads_enabled = true -chunk_size = "8192" -allowed_extensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif"] - -[deduplication] -enabled = true -directory = "/opt/hmac-file-server/data/dedup" -maxsize = "1GB" - -[timeouts] -readtimeout = "4800s" # Extended for large file uploads -writetimeout = "4800s" # Extended for large file uploads -idletimeout = "4800s" - -[clamav] -clamavenabled = true -clamavsocket = "/var/run/clamav/clamd.ctl" -numscanworkers = 2 -# Only scan potentially dangerous file types -scanfileextensions = [".txt", ".pdf", ".doc", ".exe", ".zip", ".rar"] -maxscansize = "200MB" # ClamAV scanning limit - -[redis] -redisenabled = true -redisdbindex = 0 -redisaddr = "localhost:6379" -redispassword = "" - -[workers] -numworkers = 4 -uploadqueuesize = 50 - -[logging] -level = "info" -file = "/var/log/hmac-file-server.log" -max_size = 100 -max_backups = 7 -max_age = 30 -compress = true -``` - -### Important Configuration Notes - -**Large File Support**: The extended timeout values (`readtimeout`/`writetimeout` = 4800s) are crucial for handling large file uploads (GB-sized files). These must be matched in your reverse proxy configuration. - -**Deduplication**: When enabled, identical files are stored only once using hard links, significantly saving storage space. The `maxsize` setting limits which files are deduplicated. - -**Dynamic Workers**: Auto-scaling workers (`enable_dynamic_workers = true`) automatically adjust server capacity based on upload queue length, improving performance under varying loads. - -**Security**: The `scanfileextensions` setting in ClamAV limits virus scanning to potentially dangerous file types, improving performance for large media files. - -For complete configuration details, see the [Wiki](./WIKI.MD). - ---- - -## Authentication - -The server supports multiple authentication methods: - -### 1. HMAC Authentication (Default) -- **Legacy v1**: Basic HMAC with path + content length -- **v2**: Enhanced HMAC with content type validation -- **Token**: Alternative HMAC parameter name -- **v3**: mod_http_upload_external compatible with expiration - -### 2. JWT Authentication -When `enablejwt = true`: -- Bearer tokens in Authorization header -- Query parameter `token` as fallback -- Configurable expiration and algorithm - -### Authentication Examples - -```bash -# HMAC v2 upload -curl -X PUT "http://localhost:8080/myfile.txt?v2=HMAC_SIGNATURE" -d @file.txt - -# JWT upload -curl -X POST -H "Authorization: Bearer JWT_TOKEN" -F 'file=@myfile.txt' http://localhost:8080/upload - -# v3 protocol (mod_http_upload_external) -curl -X PUT "http://localhost:8080/upload/file.txt?v3=SIGNATURE&expires=TIMESTAMP" -d @file.txt -``` - ---- - -## API Endpoints - -### Upload Endpoints -- **POST /upload**: Multipart form uploads (modern clients) -- **PUT /{filename}**: Direct uploads with HMAC or JWT authentication -- **PUT with v3 protocol**: mod_http_upload_external compatible uploads - -### Download Endpoints -- **GET /{filename}**: Direct file downloads -- **HEAD /{filename}**: File metadata (size, type) -- **GET /download/{filename}**: Alternative download endpoint - -### Management Endpoints -- **GET /health**: Health check endpoint for monitoring -- **GET /metrics**: Prometheus metrics (if enabled) -- **Various helper endpoints**: Defined in router setup - ---- - -## Usage Examples - -### Upload Examples - -#### Multipart POST Upload -```bash -curl -X POST -F 'file=@example.jpg' \ - -H "X-Signature: HMAC_OF_PATH" \ - http://localhost:8080/upload -``` - -#### Legacy PUT Upload (v2) -```bash -# Calculate HMAC of: filename + "\x00" + content_length + "\x00" + content_type -curl -X PUT "http://localhost:8080/example.jpg?v2=CALCULATED_HMAC" \ - --data-binary @example.jpg -``` - -#### v3 Protocol Upload (mod_http_upload_external) -```bash -# HMAC of: "PUT\n{timestamp}\n/path/to/file" -curl -X PUT "http://localhost:8080/upload/file.txt?v3=SIGNATURE&expires=1234567890" \ - --data-binary @file.txt -``` - -#### JWT Upload -```bash -curl -X POST \ - -H "Authorization: Bearer YOUR_JWT_TOKEN" \ - -F 'file=@example.jpg' \ - http://localhost:8080/upload -``` - -### Download Examples - -#### Direct Download -```bash -curl http://localhost:8080/example.jpg -o downloaded_file.jpg -``` - -#### Get File Info -```bash -curl -I http://localhost:8080/example.jpg -``` - -#### Health Check -```bash -curl http://localhost:8080/health -``` - ---- - -## Testing - -The HMAC File Server includes a comprehensive test suite located in the `tests/` directory. - -### Quick Testing - -```bash -cd tests - -# Test XEP-0363 protocol compatibility -./test_final_xmpp.sh - -# Test deduplication functionality -./test_deduplication.sh - -# Test upload queue performance -./test_upload_queue.sh -``` - -### Test Categories - -**Protocol Testing**: Validate XEP-0363 v1, v2, v3, and token protocols -**Performance Testing**: Upload queue, concurrent uploads, large file handling -**Feature Testing**: Deduplication, resumable uploads, chunked transfers -**Monitoring**: Real-time server status and upload activity - -### Test Data - -- Small files (1MB) for basic functionality -- Medium files (50MB) for performance testing -- Large files (4GB) for stress testing extended timeouts -- Chunked data for upload segmentation testing - -For detailed testing documentation, see [`tests/README.md`](tests/README.md). - ---- - -## Setup - -### Reverse Proxy - -#### Nginx Configuration -```nginx -server { - listen 80; - server_name your-domain.com; - client_max_body_size 10G; # Important for large uploads - - location / { - proxy_pass http://localhost:8080; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - - # Timeout settings for large uploads - proxy_read_timeout 4800; - proxy_connect_timeout 60; - proxy_send_timeout 4800; - } -} -``` - -#### Apache2 Configuration -```apache - - ServerName your-domain.com - - ProxyPreserveHost On - ProxyPass / http://localhost:8080/ - ProxyPassReverse / http://localhost:8080/ - - # Large upload support - LimitRequestBody 10737418240 # 10GB - ProxyTimeout 4800 - -``` - -### Systemd Service - -```ini -[Unit] -Description=HMAC File Server -After=network.target redis.service - -[Service] -Type=simple -ExecStart=/path/to/hmac-file-server -config /path/to/config.toml -ExecReload=/bin/kill -SIGHUP $MAINPID -WorkingDirectory=/path/to/hmac-file-server -Restart=always -RestartSec=10 -User=hmac-server -Group=hmac-server - -# Security settings -NoNewPrivileges=true -PrivateTmp=true -ProtectSystem=strict -ReadWritePaths=/path/to/uploads /path/to/logs - -[Install] -WantedBy=multi-user.target -``` - -Enable and start: -```bash -sudo systemctl daemon-reload -sudo systemctl enable hmac-file-server -sudo systemctl start hmac-file-server - -# Reload configuration (logging settings) -sudo systemctl reload hmac-file-server -``` - ---- - -## Building - -### Local Build -```bash -go build -o hmac-file-server ./cmd/server/main.go -``` - -### Cross-Platform Builds -```bash -# Linux amd64 -GOOS=linux GOARCH=amd64 go build -o hmac-file-server-linux-amd64 ./cmd/server/main.go - -# Linux arm64 -GOOS=linux GOARCH=arm64 go build -o hmac-file-server-linux-arm64 ./cmd/server/main.go - -# Windows -GOOS=windows GOARCH=amd64 go build -o hmac-file-server-windows-amd64.exe ./cmd/server/main.go -``` - ---- - -## Docker Support - -### Quick Start with Docker Compose -```yaml -version: '3.8' -services: - hmac-file-server: - image: ghcr.io/plusone/hmac-file-server:latest - ports: - - "8080:8080" - - "9090:9090" # Metrics - volumes: - - ./config:/etc/hmac-file-server - - ./uploads:/opt/hmac-file-server/data/uploads - environment: - - CONFIG_PATH=/etc/hmac-file-server/config.toml - restart: unless-stopped -``` - -### Docker Build -```bash -docker build -t hmac-file-server . -docker run -p 8080:8080 -v $(pwd)/config.toml:/etc/hmac-file-server/config.toml hmac-file-server -``` - -See the Wiki for detailed Docker setup instructions. - ---- - -## Changelog - -### Version 3.2 (Stable) -- **Development Version**: Active development branch with latest features -- **Enhanced Documentation**: Updated comprehensive documentation and protocol specifications -- **Configuration Improvements**: Better configuration validation and structure -- **Authentication System**: Full JWT and multi-protocol HMAC support - -### Version 3.1-Stable (2025-06-08) -- **v3 Protocol Support**: Added mod_http_upload_external compatibility -- **Enhanced Authentication**: Improved HMAC validation with multiple protocol support -- **JWT Integration**: Complete JWT authentication system -- **File Naming Options**: HMAC-based or original filename preservation -- **Network Improvements**: IPv4/IPv6 dual-stack with protocol forcing - -### Version 3.0-Stable (2025-06-07) -- **Docker Support**: Official Docker images and compose files -- **Go 1.24 Requirement**: Updated minimum Go version -- **Configuration Improvements**: Better validation and hot-reloading -- **Performance Enhancements**: Worker auto-scaling and memory optimization - -### Previous Versions -See [CHANGELOG.MD](./CHANGELOG.MD) for complete version history. - ---- - -## License - -Apache License 2.0 - -Copyright (c) 2025 Alexander Renz - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. diff --git a/RELEASE_3.2_ULTIMATE_FIXED.md b/RELEASE_3.2_ULTIMATE_FIXED.md new file mode 100644 index 0000000..7aad72d --- /dev/null +++ b/RELEASE_3.2_ULTIMATE_FIXED.md @@ -0,0 +1,360 @@ +# HMAC File Server 3.2 Ultimate Fixed - Release Notes + +## ๐Ÿš€ Major Release: Complete Configuration Modernization & Enhanced Multi-Platform Support + +**Release Date:** July 18, 2025 +**Version:** 3.2 Ultimate Fixed +**Codename:** "Architecture Revolution" + +--- + +## ๐ŸŽฏ What's New in 3.2 Ultimate Fixed + +This release represents a **complete modernization** of HMAC File Server with comprehensive configuration updates, enhanced multi-architecture support, and improved project organization. Every aspect of the server has been refined for better performance, reliability, and ease of deployment. + +### ๐Ÿ”ง Configuration System Overhaul + +#### **Modernized Field Names** +All configuration fields have been updated to use consistent, modern naming conventions: + +| **Old Field Name** | **New Field Name** | **Purpose** | +|-------------------|-------------------|-------------| +| `listenport` | `listen_address` | Server binding address and port | +| `storagepath` | `storage_path` | File storage directory | +| `metricsenabled` | `metrics_enabled` | Prometheus metrics toggle | +| `readtimeout` | `read_timeout` | HTTP read timeout | +| `writetimeout` | `write_timeout` | HTTP write timeout | +| `idletimeout` | `idle_timeout` | HTTP idle timeout | + +#### **Extended Timeout Support** +- **4800-second timeouts** for large file handling (up from 30s) +- Perfect for multi-gigabyte file transfers +- Eliminates timeout errors during long uploads/downloads +- Configurable per operation type + +#### **Enhanced Deduplication System** +```toml +[deduplication] +enabled = true +directory = "./deduplication" +maxsize = "1GB" # Files larger than 1GB bypass deduplication for performance +``` + +#### **Dynamic Worker Scaling** +```toml +[server] +enable_dynamic_workers = true +worker_scale_up_thresh = 50 # Scale up when queue exceeds 50 +worker_scale_down_thresh = 10 # Scale down when queue drops below 10 +``` + +### ๐Ÿ—๏ธ Multi-Architecture Build System + +#### **New Build Script Features** +The `buildgo.sh` script now supports: + +- **Interactive Architecture Selection Menu** +- **Cross-compilation Support** for: + - **AMD64** (x86_64) - Standard servers and desktops + - **ARM64** (AArch64) - Modern ARM processors, Raspberry Pi 4+ + - **ARM32v7** - Older ARM devices, Raspberry Pi 3 and earlier +- **Build-All Option** - Creates all architectures in one command +- **Smart Binary Naming** - `hmac-file-server_amd64`, `hmac-file-server_arm64`, etc. +- **Color-coded Output** for better user experience + +#### **Usage Examples** +```bash +# Interactive mode with menu +./buildgo.sh + +# Menu options: +# 1) AMD64 (x86_64) +# 2) ARM64 (AArch64) +# 3) ARM32v7 +# 4) Build All Architectures +# 5) Native Build +``` + +### ๐Ÿ“ Project Organization Improvements + +#### **Test Suite Reorganization** +- All test scripts moved to dedicated `tests/` directory +- Comprehensive test documentation in `tests/README.md` +- Organized test categories: + - **Upload Tests** - Various file sizes and types + - **Network Tests** - Connection resilience and recovery + - **Performance Tests** - Load testing and benchmarks + - **Integration Tests** - Full system validation + +#### **Test Files Available** +- `test_1mb.bin` / `test_1mb.txt` - Small file testing +- `test_50mb.bin` - Medium file testing +- `test_215mb.bin` - Large file testing +- `test_4gb.bin` / `test_4gb.txt` - Massive file testing +- `chunk_0.bin` - Chunked upload testing + +### ๐Ÿ›ก๏ธ Security & Performance Enhancements + +#### **ClamAV Selective Scanning** +```toml +[clamav] +# Only scan potentially dangerous file types +scanfileextensions = [".txt", ".pdf", ".doc", ".docx", ".exe", ".zip", ".rar"] +# Skip files larger than 200MB (ClamAV performance limit) +maxscansize = "200MB" +``` + +#### **Smart File Handling** +- **Deduplication** with hard-link optimization +- **Pre-caching** for frequently accessed files +- **Resumable uploads/downloads** for network resilience +- **Chunked transfer** support for large files + +### ๐Ÿณ Docker & Deployment Improvements + +#### **Enhanced Docker Configuration** +- Updated `dockerenv/config/config.toml` with all modern settings +- Optimized container resource usage +- Better volume mapping for persistent storage +- Improved health check configurations + +#### **Production-Ready Defaults** +```toml +[server] +max_upload_size = "10GB" +cleanup_interval = "24h" +max_file_age = "720h" # 30 days +min_free_bytes = "1GB" +``` + +### ๐Ÿ“– Documentation Overhaul + +#### **Completely Updated Documentation** +- **README.md** - Modern configuration examples and usage +- **WIKI.md** - Comprehensive configuration reference +- **INSTALL.md** - Production deployment guide +- **BUILD_GUIDE.md** - Multi-architecture build instructions +- **NETWORK_RESILIENCE_GUIDE.md** - Network handling best practices + +#### **Configuration Best Practices** +All documentation now includes: +- **Timeout configuration** for different use cases +- **Performance tuning** recommendations +- **Security hardening** guidelines +- **Troubleshooting** common issues + +--- + +## ๐Ÿ”„ Migration Guide + +### From 3.1.x to 3.2 Ultimate Fixed + +#### **Configuration Updates Required** +1. **Update field names** in your `config.toml`: + ```bash + # Old format + listenport = ":8080" + storagepath = "/uploads" + metricsenabled = true + + # New format + listen_address = ":8080" + storage_path = "/uploads" + metrics_enabled = true + ``` + +2. **Update timeout values** for better large file support: + ```toml + [timeouts] + readtimeout = "4800s" + writetimeout = "4800s" + idletimeout = "4800s" + ``` + +3. **Enable new features**: + ```toml + [server] + enable_dynamic_workers = true + + [deduplication] + enabled = true + maxsize = "1GB" + ``` + +#### **No Breaking Changes** +- Backward compatibility maintained for core functionality +- Old configuration files will work with warnings +- Gradual migration supported + +--- + +## ๐Ÿš€ Quick Start + +### **1. Download & Build** +```bash +# Clone repository +git clone https://github.com/your-org/hmac-file-server.git +cd hmac-file-server + +# Build for your architecture +./buildgo.sh +# Select option from interactive menu + +# Or build all architectures +./buildgo.sh +# Select option 4 "Build All Architectures" +``` + +### **2. Configure** +```bash +# Copy example configuration +cp config-example.toml config.toml + +# Edit for your environment +nano config.toml +``` + +### **3. Run** +```bash +# Start server +./hmac-file-server -config config.toml + +# Or with Docker +cd dockerenv +docker-compose up -d +``` + +--- + +## ๐Ÿงช Testing + +### **Run Test Suite** +```bash +# Run all tests +cd tests +./run_all_tests.sh + +# Run specific test category +./test_upload_performance.sh +./test_network_resilience.sh +``` + +### **Available Tests** +- **Upload/Download** functionality +- **Network resilience** and recovery +- **Multi-architecture** binary validation +- **Configuration** validation +- **Performance** benchmarking + +--- + +## ๐Ÿ“Š Performance Improvements + +| **Feature** | **3.1.x** | **3.2 Ultimate** | **Improvement** | +|-------------|-----------|------------------|-----------------| +| Upload Timeout | 30s | 4800s | **160x longer** | +| Large File Support | Limited | 10GB+ | **Unlimited** | +| Worker Scaling | Static | Dynamic | **Auto-scaling** | +| Deduplication | Basic | Smart (1GB limit) | **Performance optimized** | +| Architecture Support | AMD64 only | AMD64/ARM64/ARM32 | **Multi-platform** | +| Build Time | Manual | Automated menu | **User-friendly** | + +--- + +## ๐Ÿ› ๏ธ Technical Specifications + +### **System Requirements** +- **Minimum RAM:** 512MB +- **Recommended RAM:** 2GB+ for large files +- **Disk Space:** 100MB + storage for files +- **Go Version:** 1.19+ for building + +### **Supported Platforms** +- **Linux AMD64** (x86_64) +- **Linux ARM64** (AArch64) +- **Linux ARM32** (ARMv7) +- **Docker** containers +- **Kubernetes** deployments + +### **Network Protocols** +- **HTTP/HTTPS** with configurable redirect +- **XEP-0363** compliant file upload +- **Chunked transfer** encoding +- **Resumable** uploads/downloads + +--- + +## ๐Ÿค Contributing + +### **Development Setup** +1. Fork the repository +2. Create feature branch +3. Use `./buildgo.sh` for testing builds +4. Run test suite: `cd tests && ./run_all_tests.sh` +5. Submit pull request + +### **Documentation Updates** +- Update relevant `.md` files +- Test configuration examples +- Validate cross-references + +--- + +## ๐Ÿ“ Changelog Summary + +### **Added** +- โœ… Multi-architecture build system (AMD64/ARM64/ARM32) +- โœ… Interactive build script with menu selection +- โœ… Dynamic worker scaling with configurable thresholds +- โœ… Extended timeout support (4800s) for large files +- โœ… Smart deduplication with size limits +- โœ… Comprehensive test suite organization +- โœ… Modern configuration field naming +- โœ… Enhanced ClamAV selective scanning + +### **Changed** +- ๐Ÿ”„ Configuration field names modernized +- ๐Ÿ”„ Timeout defaults increased for large file support +- ๐Ÿ”„ Documentation completely updated +- ๐Ÿ”„ Project structure reorganized with tests/ folder +- ๐Ÿ”„ Docker configuration optimized + +### **Fixed** +- ๐Ÿ› Large file upload timeout issues +- ๐Ÿ› Configuration inconsistencies across documentation +- ๐Ÿ› Build script platform limitations +- ๐Ÿ› Test script organization and discoverability + +### **Deprecated** +- โš ๏ธ Old configuration field names (still supported with warnings) + +--- + +## ๐Ÿ† Credits + +**Development Team:** +- Core server enhancements +- Multi-architecture build system +- Configuration modernization +- Documentation overhaul +- Test suite organization + +**Special Thanks:** +- Community feedback on timeout issues +- Multi-platform deployment requests +- Configuration consistency improvements + +--- + +## ๐Ÿ“ž Support + +- **Documentation:** [WIKI.md](WIKI.md) +- **Installation:** [INSTALL.md](INSTALL.md) +- **Build Guide:** [BUILD_GUIDE.md](BUILD_GUIDE.md) +- **Network Setup:** [NETWORK_RESILIENCE_GUIDE.md](NETWORK_RESILIENCE_GUIDE.md) +- **Issues:** GitHub Issues +- **Discussions:** GitHub Discussions + +--- + +**HMAC File Server 3.2 Ultimate Fixed** - *Powering reliable file transfers across all architectures* ๐Ÿš€ diff --git a/UNIVERSAL_LARGE_UPLOAD_FIX.md b/UNIVERSAL_LARGE_UPLOAD_FIX.md deleted file mode 100644 index 73baa75..0000000 --- a/UNIVERSAL_LARGE_UPLOAD_FIX.md +++ /dev/null @@ -1,108 +0,0 @@ -# Universal Large File Upload Solution - -## โœ… **COMPREHENSIVE FIX IMPLEMENTED** - -This global solution addresses "Bad Gateway" errors for **ALL XMPP clients** (Gajim, Dino, Conversations) without client-specific workarounds. - -## ๐Ÿ”ง **Multi-Layer Solution Applied** - -### 1. **nginx Stream Proxy** (Port 443 โ†’ 4443) -```bash -proxy_timeout 4800s # 80 minutes -proxy_connect_timeout 4800s # 80 minutes (was 300s) -``` - -### 2. **nginx HTTP Proxy** (Port 4443 โ†’ 8080) -```bash -client_max_body_size 10G # Maximum file size (was 1G) -client_body_timeout 4800s # 80 minutes -proxy_connect_timeout 4800s # 80 minutes -proxy_send_timeout 4800s # 80 minutes -proxy_read_timeout 4800s # 80 minutes -proxy_socket_keepalive on # Connection persistence -proxy_max_temp_file_size 0 # No temp file limits -``` - -### 3. **HMAC File Server** (Port 8080) -```bash -readtimeout = "4800s" # 80 minutes -writetimeout = "4800s" # 80 minutes -max_upload_size = "10GB" # Maximum file size -``` - -### 4. **Enhanced Signature Validation** -- **Base Grace Period**: 1 hour for all uploads -- **XMPP Client Detection**: 2 hours for Gajim/Dino/Conversations -- **Large File Scaling**: +2 minutes per 100MB for files >100MB -- **Maximum Grace**: 4 hours total (prevents abuse) - -## ๐Ÿ“Š **Grace Period Examples** - -| File Size | Client Type | Grace Period | Total Validity | -|-----------|-------------|--------------|----------------| -| 100MB | Standard | 1 hour | ~65 minutes | -| 100MB | XMPP | 2 hours | ~125 minutes | -| 970MB | XMPP | 2h 20m | ~145 minutes | -| 2GB | XMPP | 2h 40m | ~165 minutes | - -## ๐ŸŽฏ **Why This Fixes "Bad Gateway"** - -1. **Timeout Chain Aligned**: All layers now use 4800s (80 minutes) -2. **Body Size Limits**: Increased from 1GB to 10GB across the stack -3. **Client Detection**: XMPP clients get extended grace periods automatically -4. **Connection Persistence**: Keeps connections alive during long uploads -5. **Error Resilience**: Automatic retry on timeout/gateway errors - -## ๐Ÿ” **Monitoring Commands** - -### Real-time Upload Monitoring -```bash -# Watch XMPP client uploads with grace period info -sudo journalctl -u hmac-file-server -f | grep -E "grace|XMPP|Gajim|Dino|Conversations" - -# Monitor nginx proxy errors -sudo tail -f /var/log/nginx/upload_errors.log - -# Check current upload connections -sudo netstat -tuln | grep -E ":8080|:4443" -``` - -### Test Large Upload -```bash -# Test 970MB upload to verify fix -curl -X PUT "https://share.uuxo.net/path/to/large/file.mp4?v3=signature&expires=timestamp" \ - -H "Content-Type: video/mp4" \ - -H "User-Agent: Gajim 2.3.3" \ - --data-binary @largefile.mp4 -``` - -## โœ… **Deployment Status** - -- **โœ… nginx Stream**: Updated with 4800s timeouts -- **โœ… nginx HTTP**: Enhanced with 10G limits and connection persistence -- **โœ… HMAC Server**: XMPP client detection and dynamic grace periods -- **โœ… Services**: Both nginx and hmac-file-server restarted and running -- **โœ… Testing**: Ready for 970MB+ uploads via XMPP clients - -## ๐Ÿš€ **Expected Results** - -1. **Gajim**: No more "Bad Gateway" errors on large uploads -2. **Dino**: Improved timeout handling for large files -3. **Conversations**: Better upload reliability on mobile networks -4. **All Clients**: Universal support up to 10GB files - -## ๐Ÿ“ˆ **Performance Improvements** - -- **Upload Reliability**: 95%+ success rate for files up to 2GB -- **Timeout Buffer**: 4x safety margin (vs previous 5-minute limit) -- **Client Compatibility**: Universal solution for all XMPP clients -- **Network Resilience**: Handles slow connections and network interruptions - -## ๐Ÿ”„ **Next Steps** - -1. **Test with Gajim**: Upload your 970MB file again -2. **Monitor Logs**: Check for grace period messages and client detection -3. **Verify Success**: Upload should complete without "Bad Gateway" -4. **Scale Test**: Try progressively larger files (1GB, 2GB) if needed - -The fix is **universally applicable** and doesn't require any client-specific configurations or modifications. diff --git a/UPLOAD_COMPLETION_FIX.md b/UPLOAD_COMPLETION_FIX.md deleted file mode 100644 index e936e1a..0000000 --- a/UPLOAD_COMPLETION_FIX.md +++ /dev/null @@ -1,117 +0,0 @@ -# Upload Completion Issue - Diagnostic & Fix - -## Problem Analysis - -### User Report -- โœ… **Upload starts correctly**: HMAC validation working -- โœ… **Transfer completes**: File uploads without "endless encryption" delay -- โŒ **Final step fails**: "Not found" error after upload completion - -### Root Cause Identified -The issue occurs in the **final storage step**, specifically in the deduplication process: - -#### Deduplication Process Steps -1. **File Upload**: โœ… Completes successfully -2. **SHA256 Computation**: โœ… Working (now skipped for files >1GB) -3. **File Movement**: โŒ `os.Rename()` and `os.Link()` operations failing -4. **Hard Link Creation**: โŒ Causing "not found" response - -#### Technical Details -```go -// From helpers.go - Deduplication process -if err := os.Rename(absFilename, existingPath); err != nil { - log.Warnf("Failed to move file for deduplication: %v", err) - return nil // Don't fail upload - BUT THIS MIGHT STILL CAUSE ISSUES -} - -if err := os.Link(existingPath, absFilename); err != nil { - log.Warnf("Failed to create link after deduplication: %v", err) - // File restoration attempt may fail - return nil -} -``` - -## Fix Applied - -### Immediate Solution -```bash -# Temporarily disabled deduplication to isolate the issue -deduplication_enabled = false -``` - -### Testing Strategy -1. **Upload with deduplication disabled**: Should complete successfully -2. **Monitor file storage**: Verify files appear in upload directory -3. **Check for "not found" errors**: Should be eliminated -4. **Confirm client success**: XMPP clients should show successful upload - -## Expected Results - -### Before Fix (with deduplication) -``` -Upload Process: -1. HMAC validation: โœ… Success -2. File transfer: โœ… Success -3. File created: โœ… Success -4. Deduplication: โŒ Hard link failure -5. Client response: โŒ "Not found" -``` - -### After Fix (deduplication disabled) -``` -Upload Process: -1. HMAC validation: โœ… Success -2. File transfer: โœ… Success -3. File stored directly: โœ… Success -4. Deduplication: โญ๏ธ Skipped -5. Client response: โœ… Success -``` - -## Long-term Solution Options - -### Option 1: Fix Deduplication Hard Links -- Investigate NFS hard link limitations -- Implement fallback to file copying instead of linking -- Add better error handling for link failures - -### Option 2: Disable Deduplication for Large Files Only -- Keep deduplication for small files (where it works) -- Disable only for large files that were causing issues -- Maintains storage efficiency for smaller files - -### Option 3: Alternative Deduplication Strategy -- Use symbolic links instead of hard links -- Implement reference counting system -- Store deduplicated files in separate location - -## Monitoring & Verification - -### Test Script Created -```bash -/root/hmac-file-server/test_upload_completion.sh -``` - -### Real-time Monitoring -- nginx access logs -- HMAC server logs -- Upload directory file creation -- Client response verification - -## Current Status - -โœ… **Deduplication disabled** to eliminate the storage failure -โœ… **Upload speed optimized** (1GB limit prevents SHA256 delays) -โœ… **Server running** with simplified storage process -๐Ÿ”„ **Testing phase** to confirm fix resolves "not found" issue - -## Next Steps - -1. **Test upload completion** with current configuration -2. **Verify client success** (no more "not found" errors) -3. **Decide on long-term deduplication strategy** based on test results -4. **Re-enable optimized deduplication** if hard link issues can be resolved - ---- -*Issue: Final storage step failing in deduplication process* -*Fix: Deduplication temporarily disabled* -*Status: Testing upload completion* diff --git a/UPLOAD_PERFORMANCE_VERIFICATION.md b/UPLOAD_PERFORMANCE_VERIFICATION.md deleted file mode 100644 index 56f5692..0000000 --- a/UPLOAD_PERFORMANCE_VERIFICATION.md +++ /dev/null @@ -1,106 +0,0 @@ -# Upload Performance Verification Report - -## Current System Status - -โœ… **HMAC File Server 3.2**: Running successfully with optimized configuration -โœ… **nginx Proxy Chain**: Configured with 10GB limits and 4800s timeouts -โœ… **XEP-0363 Compatibility**: Universal support for all XMPP clients -โœ… **Performance Optimizations**: ClamAV and deduplication enhancements deployed - -## Configuration Verification - -### Server Configuration (/etc/hmac-file-server/config.toml) -``` -max_upload_size = "10GB" # Large file support enabled -clamavenabled = false # ClamAV disabled to avoid scanning delays -deduplication_enabled = true # Smart deduplication with size limits -file_naming = "original" # Proper MIME type handling -force_protocol = "auto" # Fixed protocol initialization -``` - -### nginx Configuration -- **HTTP Proxy** (/etc/nginx/conf.d/share.conf): 10GB client_max_body_size -- **Stream Proxy** (/etc/nginx/nginx-stream.conf): 4800s timeout -- **Extended Timeouts**: All layers configured for large file transfers - -## Performance Optimization Summary - -### 1. ClamAV Smart Scanning -- **Implementation**: Size and extension-based filtering -- **Logic**: Only scan potentially dangerous files (exe, bin, com, sh) -- **Result**: Media files bypass scanning for instant upload completion - -### 2. Deduplication Enhancement -- **Implementation**: Configurable size limits with graceful error handling -- **Logic**: Skip deduplication for files above configured threshold -- **Result**: Large files avoid SHA256 computation bottlenecks - -### 3. Timeout Optimization -- **nginx Stream**: proxy_timeout 4800s (80 minutes) -- **nginx HTTP**: Multiple timeout directives for large transfers -- **HMAC Server**: Extended grace periods for XMPP client compatibility - -## Log Analysis Results - -### Historical Upload Activity (October 2024) -From `/var/log/nginx/share_access.log.1`: -- **Gajim Client**: Multiple successful GET operations for media files -- **File Types**: webp, mp4, webm (large video files) -- **Issue Found**: One PUT request returned 405 (Method Not Allowed) -- **Indication**: Upload functionality was partially broken before our fixes - -### Current Status -- **Share Logs**: Empty since last configuration (indicates no recent test uploads) -- **Server Status**: Active and responding (health endpoint returns 200) -- **Configuration**: All optimizations properly applied and active - -## Performance Enhancement Impact - -### Before Optimizations -- โŒ ClamAV scanning caused "endless encryption" delays -- โŒ Deduplication SHA256 computation for all files -- โŒ 100MB artificial limits in upload handlers -- โŒ Short timeouts causing "Bad Gateway" errors - -### After Optimizations -- โœ… Smart ClamAV scanning only for dangerous file types -- โœ… Deduplication bypassed for large files to avoid bottlenecks -- โœ… 10GB upload support with proper size validation -- โœ… Extended timeouts preventing gateway errors - -## Verification Tools Created - -### Real-time Monitoring -```bash -/root/hmac-file-server/monitor_uploads.sh -``` -- Monitors HMAC server and nginx logs simultaneously -- Highlights upload activity, errors, and performance events -- Ready for live upload testing verification - -### Performance Documentation -- `PERFORMANCE_OPTIMIZATION.md`: Complete optimization guide -- `CLAMAV_SECURITY_CONFIG.md`: Security scanning configuration -- `UNIVERSAL_LARGE_UPLOAD_FIX.md`: Comprehensive fix documentation - -## Next Steps for Verification - -1. **Live Upload Test**: Use the monitoring script during a large file upload -2. **Performance Measurement**: Monitor transfer speeds and completion times -3. **Client Compatibility**: Test with Gajim, Dino, and Conversations -4. **Edge Case Testing**: Verify behavior with various file types and sizes - -## Conclusion - -All performance optimizations have been successfully implemented and deployed: -- Large file uploads now bypass unnecessary ClamAV scanning -- Deduplication is intelligently applied based on file size -- nginx timeout chain supports multi-GB file transfers -- XMPP clients receive proper protocol compliance - -The system is ready for large file uploads with optimized performance while maintaining security for genuinely dangerous file types. - ---- -*Report generated: $(date)* -*HMAC File Server Version: 3.2* -*Optimization Status: Complete* diff --git a/VIDEO_EXTENSION_FIX.md b/VIDEO_EXTENSION_FIX.md deleted file mode 100644 index fed4331..0000000 --- a/VIDEO_EXTENSION_FIX.md +++ /dev/null @@ -1,97 +0,0 @@ -# CRITICAL FIX: Video File Upload Block Resolved - -## Root Cause Found! - -### The Real Issue -The `` error was caused by **file extension blocking**, not deduplication issues. - -### Configuration Problem -```toml -# BEFORE (blocking video files) -global_extensions = [".txt", ".pdf", ".jpg", ".png", ".docx", ".xlsx", ".zip"] - -# AFTER (allowing video files) -global_extensions = [".txt", ".pdf", ".jpg", ".png", ".docx", ".xlsx", ".zip", ".mp4", ".mkv", ".avi", ".mov", ".wmv", ".flv", ".webm", ".mpeg"] -``` - -### Why This Caused the Error -1. **global_extensions overrides** upload/download extension settings -2. **Video files (.mp4, .avi, etc.)** were not in the global list -3. **Server rejected uploads** before they even started processing -4. **XMPP client** received rejection as `` - -## Fix Applied - -### Production Configuration Updated -โœ… **Video extensions added** to global_extensions list -โœ… **Server restarted** with new configuration -โœ… **Docker config updated** to match production - -### Current Status -- **Deduplication**: Disabled (to avoid hard link issues) -- **File extensions**: Video files now allowed -- **Upload speed**: Optimized (no SHA256 delays) -- **Storage**: Direct file storage (no complex operations) - -## Expected Results - -### Before Fix -``` -Upload Process: -1. XMPP client requests upload URL -2. Client attempts PUT request with .mp4 file -3. Server checks global_extensions -4. Server rejects: .mp4 not in allowed list -5. Client receives: -``` - -### After Fix -``` -Upload Process: -1. XMPP client requests upload URL -2. Client attempts PUT request with .mp4 file -3. Server checks global_extensions -4. Server accepts: .mp4 is in allowed list -5. File uploads successfully -6. Client receives: Success response -``` - -## Test Recommendation - -Try uploading your .mp4 file again. It should now: -- โœ… **Start immediately** (no extension rejection) -- โœ… **Upload quickly** (no deduplication delays) -- โœ… **Complete successfully** (no storage issues) -- โœ… **Show success** in XMPP client (no error messages) - -## Configuration Summary - -### Current Production Settings -```toml -[server] -global_extensions = [".txt", ".pdf", ".jpg", ".png", ".docx", ".xlsx", ".zip", ".mp4", ".mkv", ".avi", ".mov", ".wmv", ".flv", ".webm", ".mpeg"] -deduplication_enabled = false -max_upload_size = "10GB" -file_naming = "original" - -[clamav] -clamavenabled = false -``` - -### nginx Timeouts -- **HTTP proxy**: 10GB body size, 4800s timeout -- **Stream proxy**: 4800s timeout -- **Chain**: 443 โ†’ 4443 โ†’ 8080 - -## Issue Resolution Timeline - -1. โŒ **"Endless encryption"**: Fixed by 1GB deduplication limit -2. โŒ **"Not found" after upload**: Fixed by disabling deduplication -3. โŒ **``**: Fixed by allowing video extensions - -**All issues now resolved!** ๐ŸŽ‰ - ---- -*Critical Fix: Video file extensions added to global_extensions* -*Status: Ready for successful video file uploads* -*Date: $(date)* diff --git a/hmac-file-server.pid b/hmac-file-server.pid deleted file mode 100644 index 3d47cf2..0000000 --- a/hmac-file-server.pid +++ /dev/null @@ -1 +0,0 @@ -2896344 \ No newline at end of file diff --git a/monitor_nginx.sh b/monitor_nginx.sh deleted file mode 100644 index e69de29..0000000 diff --git a/monitor_server.sh b/monitor_server.sh deleted file mode 100644 index e69de29..0000000 diff --git a/monitor_uploads.sh b/monitor_uploads.sh deleted file mode 100644 index e69de29..0000000 diff --git a/verify_xmpp_upload.sh b/verify_xmpp_upload.sh deleted file mode 100644 index e69de29..0000000