diff --git a/EJABBERD_MODULE_PROPOSAL.md b/EJABBERD_MODULE_PROPOSAL.md deleted file mode 100644 index e69de29..0000000 diff --git a/GAJIM_BAD_GATEWAY_FIX.md b/GAJIM_BAD_GATEWAY_FIX.md deleted file mode 100644 index e35b894..0000000 --- a/GAJIM_BAD_GATEWAY_FIX.md +++ /dev/null @@ -1,169 +0,0 @@ -# Gajim "Bad Gateway" Fix - Enhanced Multi-Upload CORS Implementation -*HMAC File Server 3.3.0 "Nexus Infinitum" - XMPP Client Compatibility Enhancement* - -## Issue Resolution - -**Problem**: Gajim reports "bad gateway" errors intermittently during file uploads, specifically on **multi-upload scenarios** (second, third uploads fail). - -**Root Cause**: -1. Server didn't handle CORS preflight (OPTIONS) requests properly -2. Missing extended CORS headers for multi-upload session management -3. No session state tracking for persistent connections used by Gajim - -**Solution**: Implemented comprehensive CORS support with multi-upload session management. - -## Technical Implementation - -### 1. Enhanced CORS Middleware -```go -corsWrapper := func(handler http.HandlerFunc) http.HandlerFunc { - return func(w http.ResponseWriter, r *http.Request) { - // Enhanced CORS headers for Gajim multi-upload support - w.Header().Set("Access-Control-Allow-Origin", "*") - w.Header().Set("Access-Control-Allow-Methods", "GET, PUT, POST, DELETE, OPTIONS, HEAD") - w.Header().Set("Access-Control-Allow-Headers", "Authorization, Content-Type, Content-Length, X-Requested-With, X-Upload-ID, X-Session-Token, X-File-Name, X-File-Size, Range, Content-Range") - w.Header().Set("Access-Control-Expose-Headers", "Content-Length, Content-Range, X-Upload-Status, X-Session-ID, Location, ETag") - w.Header().Set("Access-Control-Max-Age", "86400") - w.Header().Set("Access-Control-Allow-Credentials", "false") - - // Handle OPTIONS preflight for all endpoints - if r.Method == http.MethodOptions { - w.WriteHeader(http.StatusOK) - return - } - - handler(w, r) - } -} -``` - -### 2. Multi-Upload Session Management -```go -// Enhanced session handling for multi-upload scenarios (Gajim fix) -sessionID := r.Header.Get("X-Session-ID") -if sessionID == "" { - // Generate session ID for multi-upload tracking - sessionID = generateUploadSessionID("upload", r.Header.Get("User-Agent"), getClientIP(r)) -} - -// Set session headers for client continuation -w.Header().Set("X-Session-ID", sessionID) -w.Header().Set("X-Upload-Session-Timeout", "3600") // 1 hour -``` - -### 3. XMPP Protocol Session Support -```go -// Enhanced session handling for multi-upload scenarios (Gajim XMPP fix) -sessionID := r.Header.Get("X-Session-ID") -if sessionID == "" { - // Generate session ID for XMPP multi-upload tracking - sessionID = generateUploadSessionID("legacy", r.Header.Get("User-Agent"), getClientIP(r)) -} - -// Set session headers for XMPP client continuation -w.Header().Set("X-Session-ID", sessionID) -w.Header().Set("X-Upload-Session-Timeout", "3600") // 1 hour -w.Header().Set("X-Upload-Type", "legacy-xmpp") -``` - -## Enhanced CORS Headers for Multi-Upload - -### Basic CORS Headers -| Header | Value | Purpose | -|--------|--------|---------| -| `Access-Control-Allow-Origin` | `*` | Allow requests from any origin | -| `Access-Control-Allow-Methods` | `GET, PUT, POST, DELETE, OPTIONS, HEAD` | Permitted HTTP methods | -| `Access-Control-Max-Age` | `86400` | Cache preflight for 24 hours | -| `Access-Control-Allow-Credentials` | `false` | Public file server mode | - -### Multi-Upload Support Headers -| Header | Value | Purpose | -|--------|--------|---------| -| `Access-Control-Allow-Headers` | `Authorization, Content-Type, Content-Length, X-Requested-With, X-Upload-ID, X-Session-Token, X-File-Name, X-File-Size, Range, Content-Range` | Extended upload metadata support | -| `Access-Control-Expose-Headers` | `Content-Length, Content-Range, X-Upload-Status, X-Session-ID, Location, ETag` | Upload state management | - -### Session Management Headers -| Header | Purpose | Example Value | -|--------|---------|---------------| -| `X-Session-ID` | Track multi-upload sessions | `upload_c03d9835ed0efcbb` | -| `X-Upload-Session-Timeout` | Session validity period | `3600` (1 hour) | -| `X-Upload-Type` | Upload protocol type | `legacy-xmpp` | - -## Client Compatibility - -### โœ… Fixed Issues -- **Gajim**: No more "bad gateway" errors during uploads -- **Web XMPP clients**: Full CORS support for browser-based clients -- **Converse.js**: Enhanced compatibility for web deployment -- **Future XMPP clients**: Standards-compliant CORS implementation - -### ๐Ÿ”ง Technical Flow -1. **First Upload**: Client sends OPTIONS preflight โ†’ Server responds with CORS headers + session ID -2. **Subsequent Uploads**: Client reuses session ID โ†’ Server recognizes multi-upload context -3. **Session Tracking**: Server maintains upload state across requests -4. **No more 502/404 errors**: Seamless multi-file upload experience - -### ๐Ÿ“Š Multi-Upload Scenario -``` -Gajim Upload Sequence: - Upload 1: OPTIONS โ†’ 200 OK (session created) โ†’ PUT โ†’ 201 Created โœ… - Upload 2: OPTIONS โ†’ 200 OK (session reused) โ†’ PUT โ†’ 201 Created โœ… - Upload 3: OPTIONS โ†’ 200 OK (session reused) โ†’ PUT โ†’ 201 Created โœ… -``` - -**Before Fix**: Second upload would get 404/502 "bad gateway" -**After Fix**: All uploads in sequence work seamlessly - -## Testing Results - -```bash -$ ./test-gajim-cors-fix.sh -๐Ÿงช Testing CORS Functionality for Gajim Compatibility -======================================================== - -โœ… OPTIONS request successful (HTTP 200) -โœ… Access-Control-Allow-Headers: Authorization, Content-Type, Content-Length, X-Requested-With -โœ… Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS, HEAD -โœ… Access-Control-Allow-Origin: * -โœ… Access-Control-Max-Age: 86400 -โœ… GET request with CORS successful (HTTP 200) -โœ… XMPP client preflight successful - -๐ŸŽฏ SUMMARY: ALL TESTS PASSED -โœ… Gajim's 'bad gateway' error should be FIXED! -``` - -## Impact - -### Before Fix -``` -Gajim โ†’ OPTIONS /upload โ†’ 404 Not Found โ†’ "bad gateway" error -``` - -### After Fix -``` -Gajim โ†’ OPTIONS /upload โ†’ 200 OK (with CORS headers) โ†’ Proceeds with upload โ†’ Success -``` - -## Backward Compatibility - -- โœ… **100% backward compatible** - existing XMPP clients continue working -- โœ… **Standards compliant** - follows W3C CORS specification -- โœ… **XEP-0363 compatible** - maintains XMPP HTTP File Upload compliance -- โœ… **Performance optimized** - 24-hour preflight caching - -## Deployment - -The fix is automatically included in HMAC File Server 3.3.0 and later. No configuration changes required. - -### Verification -```bash -# Test CORS functionality -curl -X OPTIONS http://your-server:8080/ -v - -# Should return HTTP 200 with CORS headers -``` - ---- -*Fixed: August 26, 2025* -*HMAC File Server 3.3.0 "Nexus Infinitum" - Enhanced XMPP Client Ecosystem* diff --git a/fix_xmpp_clients.sh b/fix_xmpp_clients.sh deleted file mode 100755 index c66018e..0000000 --- a/fix_xmpp_clients.sh +++ /dev/null @@ -1,175 +0,0 @@ -#!/bin/bash -# ๐Ÿงน XMPP Client Cache Cleaner for Upload Issues -# Fixes Dino and Gajim upload problems after restart -# Date: August 26, 2025 - -set -euo pipefail - -# Colors -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -RED='\033[0;31m' -NC='\033[0m' - -echo -e "${BLUE}๐Ÿงน XMPP CLIENT CACHE CLEANER${NC}" -echo "==============================" -echo "Fixing Dino and Gajim upload issues after restart" -echo "" - -# Function to safely stop processes -stop_process() { - local process_name="$1" - echo -e "${YELLOW}๐Ÿ›‘ Stopping $process_name...${NC}" - - if pgrep -f "$process_name" >/dev/null; then - pkill -f "$process_name" - sleep 2 - - # Force kill if still running - if pgrep -f "$process_name" >/dev/null; then - pkill -9 -f "$process_name" 2>/dev/null || true - sleep 1 - fi - - if ! pgrep -f "$process_name" >/dev/null; then - echo -e "${GREEN}โœ… $process_name stopped${NC}" - else - echo -e "${RED}โš ๏ธ $process_name may still be running${NC}" - fi - else - echo -e "${GREEN}โœ… $process_name not running${NC}" - fi -} - -# Function to clear cache directory -clear_cache() { - local app_name="$1" - local cache_dir="$2" - - if [ -d "$cache_dir" ]; then - echo -e "${YELLOW}๐Ÿ—‘๏ธ Clearing $app_name cache: $cache_dir${NC}" - rm -rf "$cache_dir" 2>/dev/null || true - echo -e "${GREEN}โœ… $app_name cache cleared${NC}" - else - echo -e "${BLUE}โ„น๏ธ $app_name cache not found: $cache_dir${NC}" - fi -} - -# Function to clear upload-related files -clear_upload_files() { - local app_name="$1" - local data_dir="$2" - - if [ -d "$data_dir" ]; then - echo -e "${YELLOW}๐Ÿ” Clearing $app_name upload-related files...${NC}" - - # Find and remove upload/token related files - local files_removed=0 - for pattern in "*upload*" "*token*" "*session*" "*cache*"; do - while IFS= read -r -d '' file; do - rm -f "$file" 2>/dev/null && ((files_removed++)) || true - done < <(find "$data_dir" -name "$pattern" -type f -print0 2>/dev/null || true) - done - - if [ $files_removed -gt 0 ]; then - echo -e "${GREEN}โœ… Removed $files_removed upload-related files from $app_name${NC}" - else - echo -e "${BLUE}โ„น๏ธ No upload-related files found in $app_name${NC}" - fi - else - echo -e "${BLUE}โ„น๏ธ $app_name data directory not found: $data_dir${NC}" - fi -} - -# Function to backup data (optional) -backup_data() { - local app_name="$1" - local data_dir="$2" - local backup_dir="${data_dir}.backup.$(date +%Y%m%d_%H%M%S)" - - if [ -d "$data_dir" ]; then - echo -e "${YELLOW}๐Ÿ’พ Creating backup of $app_name data...${NC}" - if cp -r "$data_dir" "$backup_dir" 2>/dev/null; then - echo -e "${GREEN}โœ… Backup created: $backup_dir${NC}" - else - echo -e "${RED}โš ๏ธ Failed to create backup for $app_name${NC}" - fi - fi -} - -# Main execution -echo -e "${BLUE}Step 1: Stopping XMPP clients${NC}" -echo "-----------------------------" -stop_process "dino" -stop_process "gajim" - -echo "" -echo -e "${BLUE}Step 2: Creating backups (optional)${NC}" -echo "-----------------------------------" -if [ "${1:-}" = "--backup" ]; then - backup_data "Dino" "$HOME/.local/share/dino" - backup_data "Gajim" "$HOME/.local/share/gajim" -else - echo -e "${YELLOW}โ„น๏ธ Skipping backups (use --backup flag to create backups)${NC}" -fi - -echo "" -echo -e "${BLUE}Step 3: Clearing caches${NC}" -echo "---------------------" -clear_cache "Dino" "$HOME/.cache/dino" -clear_cache "Gajim" "$HOME/.cache/gajim" - -echo "" -echo -e "${BLUE}Step 4: Clearing upload-related files${NC}" -echo "------------------------------------" -clear_upload_files "Dino" "$HOME/.local/share/dino" -clear_upload_files "Gajim" "$HOME/.local/share/gajim" - -echo "" -echo -e "${BLUE}Step 5: Restarting XMPP clients${NC}" -echo "------------------------------" - -# Check if display is available -if [ -z "${DISPLAY:-}" ]; then - echo -e "${RED}โš ๏ธ No DISPLAY environment variable - cannot start GUI clients${NC}" - echo "Please manually start Dino and Gajim after setting DISPLAY" -else - echo -e "${YELLOW}๐Ÿš€ Starting Dino...${NC}" - if command -v dino >/dev/null 2>&1; then - dino & - echo -e "${GREEN}โœ… Dino started${NC}" - else - echo -e "${RED}โŒ Dino not found in PATH${NC}" - fi - - echo -e "${YELLOW}๐Ÿš€ Starting Gajim...${NC}" - if command -v gajim >/dev/null 2>&1; then - gajim & - echo -e "${GREEN}โœ… Gajim started${NC}" - else - echo -e "${RED}โŒ Gajim not found in PATH${NC}" - fi -fi - -echo "" -echo -e "${GREEN}๐ŸŽ‰ CLEANUP COMPLETE!${NC}" -echo "===================" -echo "" -echo -e "${GREEN}โœ… What was done:${NC}" -echo " โ€ข Stopped Dino and Gajim processes" -echo " โ€ข Cleared application caches" -echo " โ€ข Removed upload/token related files" -echo " โ€ข Restarted XMPP clients" -echo "" -echo -e "${BLUE}๐Ÿงช Next steps:${NC}" -echo " 1. Wait for clients to fully load" -echo " 2. Try uploading a small file in both clients" -echo " 3. Upload should work with fresh authentication" -echo "" -echo -e "${YELLOW}๐Ÿ“‹ If upload still fails:${NC}" -echo " โ€ข Check server logs: tail -f /var/log/hmac-file-server-mobile.log" -echo " โ€ข Use enhanced server: ./hmac-file-server-desktop-fixed -config config-mobile-resilient.toml" -echo " โ€ข Check network configuration with: ip addr show" -echo "" -echo "Cache cleanup completed at $(date)" diff --git a/large-file-performance-fix-summary.sh b/large-file-performance-fix-summary.sh deleted file mode 100755 index 79d4920..0000000 --- a/large-file-performance-fix-summary.sh +++ /dev/null @@ -1,167 +0,0 @@ -#!/bin/bash -# Large File Upload Performance Fix Summary & Verification - -echo "๐ŸŽ‰ LARGE FILE UPLOAD PERFORMANCE FIX - COMPLETE SOLUTION" -echo "=========================================================" - -echo "" -echo "๐Ÿ“‹ PROBLEM ANALYSIS:" -echo " Original Issue: 'on large files the finishing on server side takes long'" -echo " Specific Impact: 'if too long error in client (ONLY LARGE FILES ABOVE 1GB)'" -echo " Root Cause: Synchronous post-processing (deduplication + virus scanning)" -echo " Client Impact: Timeout errors waiting for server ACK after 100% transfer" - -echo "" -echo "๐Ÿ’ก SOLUTION IMPLEMENTED:" -echo " Strategy: Immediate 200 OK response + asynchronous post-processing" -echo " Threshold: Files >1GB trigger async mode" -echo " Components: Deduplication + virus scanning moved to background" -echo " Benefit: Client gets instant success confirmation" - -echo "" -echo "๐Ÿ”ง TECHNICAL IMPLEMENTATION:" -echo "==========================" - -echo "" -echo "1. Code Changes Applied:" -echo " โœ… cmd/server/main.go: Modified handleUpload() function" -echo " โœ… cmd/server/main.go: Modified handleV3Upload() function" -echo " โœ… cmd/server/main.go: Modified handleLegacyUpload() function" -echo " โœ… All upload endpoints now support async large file processing" - -echo "" -echo "2. Processing Logic:" -echo " ๐Ÿ“ File size check: if written > 1GB (1024*1024*1024 bytes)" -echo " โšก Immediate response: HTTP 200/201 with upload metadata" -echo " ๐Ÿ”„ Background goroutine: handles deduplication + virus scanning" -echo " ๐Ÿ“Š Metrics: Updated immediately for client response" - -echo "" -echo "3. Response Headers for Large Files:" -echo " X-Large-File-Processing: async" -echo " X-Post-Processing: background" -echo " X-Upload-Success: true" -echo " X-Upload-Duration: [time until response sent]" - -echo "" -echo "๐Ÿงช VERIFICATION RESULTS:" -echo "=======================" - -# Check server status -SERVER_STATUS=$(systemctl is-active hmac-file-server) -if [ "$SERVER_STATUS" = "active" ]; then - echo "โœ… Server Status: Running with async processing enabled" -else - echo "โŒ Server Status: Not running - need to start server" -fi - -# Check CORS functionality -CORS_TEST=$(curl -s -X OPTIONS "http://localhost:8080/" \ - -H "Origin: https://gajim.org" \ - -H "User-Agent: Gajim/1.8.4" \ - -w "HTTP_CODE:%{http_code}") - -CORS_CODE=$(echo "$CORS_TEST" | grep -o "HTTP_CODE:[0-9]*" | cut -d: -f2) -if [ "$CORS_CODE" = "200" ]; then - echo "โœ… CORS Functionality: Working (HTTP $CORS_CODE)" -else - echo "โŒ CORS Functionality: Issues detected (HTTP $CORS_CODE)" -fi - -# Check configuration -DEDUP_STATUS=$(grep -E "deduplication.*enabled.*true|DeduplicationEnabled.*true" /opt/hmac-file-server/config.toml 2>/dev/null && echo "enabled" || echo "disabled") -echo "โœ… Deduplication: $DEDUP_STATUS (async for large files)" - -TIMEOUT_STATUS=$(grep -E "readtimeout.*7200s|writetimeout.*7200s" /opt/hmac-file-server/config.toml 2>/dev/null && echo "extended" || echo "standard") -echo "โœ… Timeouts: $TIMEOUT_STATUS (supports large file uploads)" - -echo "" -echo "๐Ÿš€ PERFORMANCE IMPROVEMENTS:" -echo "============================" - -echo "" -echo "BEFORE (Synchronous Processing):" -echo " ๐Ÿ“ค Client uploads 1GB file โ†’ 100% transfer complete" -echo " โณ Client waits for deduplication (30-60 seconds)" -echo " โณ Client waits for virus scanning (10-30 seconds)" -echo " โณ Total wait time: 40-90 seconds after upload" -echo " โŒ Client timeout: Upload appears to fail" - -echo "" -echo "AFTER (Asynchronous Processing):" -echo " ๐Ÿ“ค Client uploads 1GB file โ†’ 100% transfer complete" -echo " โœ… Immediate HTTP 200 OK response (~1 second)" -echo " ๐Ÿ”„ Server continues processing in background" -echo " โœ… Client success: Upload completes immediately" - -echo "" -echo "๐Ÿ“Š EXPECTED PERFORMANCE GAINS:" -echo " โšก Response time: ~95% faster for large files" -echo " ๐Ÿ“ˆ Client success rate: ~100% (no more timeouts)" -echo " ๐Ÿ”„ Server throughput: Improved (no blocking)" -echo " ๐Ÿ’พ Storage efficiency: Maintained (async deduplication)" -echo " ๐Ÿ”’ Security: Maintained (async virus scanning)" - -echo "" -echo "๐ŸŽฏ FINAL VERIFICATION:" -echo "=====================" - -echo "" -echo "โœ… IMPLEMENTATION STATUS:" -echo " โœ… Code deployed and server restarted" -echo " โœ… All upload handlers modified (main, v3, legacy)" -echo " โœ… 1GB threshold implemented for async processing" -echo " โœ… Background goroutines handle post-processing" -echo " โœ… Immediate response headers configured" - -echo "" -echo "โœ… COMPATIBILITY MAINTAINED:" -echo " โœ… Small files (<1GB): Synchronous processing (unchanged)" -echo " โœ… Large files (>1GB): Asynchronous processing (new)" -echo " โœ… XMPP clients: Enhanced session management" -echo " โœ… Gajim multi-upload: CORS + timeout fixes active" - -echo "" -echo "๐Ÿ” MONITORING RECOMMENDATIONS:" -echo "=============================" - -echo "" -echo "Server Logs to Watch:" -echo " ๐Ÿ” 'Large file detected' - Confirms async mode activation" -echo " ๐Ÿ”„ 'Background deduplication' - Shows async dedup progress" -echo " ๐Ÿ”„ 'Background virus scan' - Shows async scanning progress" -echo " โœ… 'Background...completed' - Confirms post-processing success" - -echo "" -echo "Performance Metrics:" -echo " ๐Ÿ“Š Upload response times (should be ~1s for large files)" -echo " ๐Ÿ“ˆ Client success rates (should approach 100%)" -echo " ๐Ÿ’พ Server CPU/Memory during large uploads" -echo " ๐Ÿ”„ Background processing completion rates" - -echo "" -echo "๐ŸŽ‰ SOLUTION COMPLETE!" -echo "====================" - -echo "" -echo "โœ… PROBLEM SOLVED:" -echo " โŒ BEFORE: Large file uploads caused client timeouts" -echo " โœ… AFTER: Large file uploads complete immediately" - -echo "" -echo "โœ… CLIENT EXPERIENCE:" -echo " ๐Ÿ“ค Upload large file โ†’ Immediate success" -echo " โšก No more waiting for server post-processing" -echo " ๐ŸŽฏ 100% success rate for uploads" - -echo "" -echo "โœ… SERVER EFFICIENCY:" -echo " ๐Ÿ”„ Post-processing continues in background" -echo " ๐Ÿ“ˆ Higher throughput (no blocking uploads)" -echo " ๐Ÿ’พ Maintained deduplication benefits" -echo " ๐Ÿ”’ Maintained security scanning" - -echo "" -echo "๐Ÿš€ READY FOR PRODUCTION!" -echo "Your server now handles large file uploads optimally." -echo "Clients will no longer experience timeouts on files >1GB."