uuxo/hmac-file-server
1
1
Fork 0
Code Pull Requests Actions Packages Projects Releases 1 Wiki Activity

🔥 Tremora del Terra: ultimate hmac-file-server fix – final push before the drop 💾🔐

Browse Source
This commit is contained in:
Alexander Renz
2025-07-18 11:18:34 +00:00
parent 3c87abcc80
commit de8fc326df
23 changed files with 360 additions and 2851 deletions
Download Patch File Download Diff File Expand all files Collapse all files

251
NETWORK_RESILIENCE_GUIDE.md
View File

@ -1,251 +0,0 @@
# Network Resilience Implementation Guide
## Overview
This implementation adds network switching resilience to the HMAC File Server **without modifying any core functions**. The solution uses a wrapper/middleware approach that enhances existing functionality while maintaining backward compatibility.
## New Features Added
### 1. Chunked/Resumable Uploads
- **Endpoint**: `POST/PUT /upload/chunked`
- **Status Check**: `GET /upload/status?session_id=<id>`
- Breaks large uploads into smaller chunks (5MB default)
- Resumes interrupted uploads automatically
- Survives network switching scenarios
### 2. Network Change Detection
- Monitors network interface status every 5 seconds
- Automatically pauses uploads during network changes
- Resumes uploads after network stabilization (2-second delay)
- Logs network events for monitoring
### 3. Upload Session Persistence
- Stores upload state in Redis (if available) or local disk
- Sessions persist across server restarts
- Automatic cleanup of expired sessions (24-hour default)
- Tracks chunk completion status
### 4. Enhanced Connection Management
- Configurable timeouts optimized for mobile networks
- Retry logic with exponential backoff
- Better keep-alive settings for mobile scenarios
## Configuration
Add these settings to your `config.toml`:
```toml
[uploads]
chunked_uploads_enabled = true # Enable chunked uploads
resumable_uploads_enabled = true # Enable resumable functionality
chunk_size = "10MB" # Chunk size (optimized for performance)
max_resumable_age = "48h" # Session persistence time
[timeouts]
readtimeout = "4800s" # 80 minutes (extended for large files)
writetimeout = "4800s" # 80 minutes (extended for large files)
idletimeout = "4800s" # 80 minutes (extended for large files)
[server]
enable_dynamic_workers = true # Enable dynamic worker scaling
worker_scale_up_thresh = 50 # Scale up when queue reaches 50
worker_scale_down_thresh = 10 # Scale down when queue drops to 10
[workers]
numworkers = 4 # Base number of workers
uploadqueuesize = 50 # Upload queue size
[deduplication]
enabled = true # Enable file deduplication
maxsize = "1GB" # Deduplicate files under 1GB only
[server]
networkevents = true # Enable network monitoring
```
## API Usage
### Traditional Upload (existing, unchanged)
```bash
curl -X POST -F 'file=@large_file.zip' \
-H "X-Signature: HMAC_SIGNATURE" \
http://localhost:8080/upload
```
### New Chunked Upload
#### 1. Start Upload Session
```bash
curl -X POST \
-H "X-Filename: large_file.zip" \
-H "X-Total-Size: 104857600" \
-H "X-Signature: HMAC_SIGNATURE" \
http://localhost:8080/upload/chunked
# Response:
{
"session_id": "1234567890_abc123",
"chunk_size": 5242880,
"total_chunks": 20
}
```
#### 2. Upload Chunks
```bash
# Upload chunk 0
curl -X PUT \
-H "X-Upload-Session-ID: 1234567890_abc123" \
-H "X-Chunk-Number: 0" \
-H "Content-Type: application/octet-stream" \
--data-binary @chunk_0.bin \
http://localhost:8080/upload/chunked
# Response:
{
"success": true,
"chunk": 0,
"uploaded_bytes": 5242880,
"total_size": 104857600,
"progress": 0.05,
"completed": false
}
```
#### 3. Check Status
```bash
curl "http://localhost:8080/upload/status?session_id=1234567890_abc123"
# Response:
{
"session_id": "1234567890_abc123",
"filename": "large_file.zip",
"total_size": 104857600,
"uploaded_bytes": 52428800,
"progress": 0.5,
"completed": false,
"chunks": 10
}
```
## Client-Side JavaScript Example
```javascript
class NetworkResilientUploader {
constructor(file, endpoint, options = {}) {
this.file = file;
this.endpoint = endpoint;
this.chunkSize = options.chunkSize || 5 * 1024 * 1024; // 5MB
this.maxRetries = options.maxRetries || 5;
}
async upload() {
// Start session
const session = await this.startSession();
console.log('Upload session started:', session.session_id);
// Upload chunks
const totalChunks = Math.ceil(this.file.size / this.chunkSize);
for (let i = 0; i < totalChunks; i++) {
await this.uploadChunk(session.session_id, i);
console.log(`Chunk ${i + 1}/${totalChunks} uploaded`);
}
console.log('Upload completed!');
}
async startSession() {
const response = await fetch(this.endpoint, {
method: 'POST',
headers: {
'X-Filename': this.file.name,
'X-Total-Size': this.file.size.toString(),
'X-Signature': 'YOUR_HMAC_SIGNATURE'
}
});
return response.json();
}
async uploadChunk(sessionId, chunkNumber, retryCount = 0) {
try {
const start = chunkNumber * this.chunkSize;
const end = Math.min(start + this.chunkSize, this.file.size);
const chunk = this.file.slice(start, end);
const response = await fetch(this.endpoint, {
method: 'PUT',
headers: {
'X-Upload-Session-ID': sessionId,
'X-Chunk-Number': chunkNumber.toString()
},
body: chunk
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
return response.json();
} catch (error) {
if (retryCount < this.maxRetries) {
const delay = Math.pow(2, retryCount) * 1000 + Math.random() * 1000;
await new Promise(resolve => setTimeout(resolve, delay));
return this.uploadChunk(sessionId, chunkNumber, retryCount + 1);
}
throw error;
}
}
}
// Usage
const uploader = new NetworkResilientUploader(fileInput.files[0], '/upload/chunked');
uploader.upload().catch(console.error);
```
## Benefits for Network Switching
### Before (Problems)
- ❌ Upload fails completely on network interruption
- ❌ User loses all progress when switching WiFi/WLAN
- ❌ Large files problematic on mobile networks
- ❌ No recovery from temporary connection issues
### After (Solutions)
- ✅ Upload resumes automatically after network switch
- ✅ Progress preserved during network interruptions
- ✅ Optimal chunk sizes for mobile networks
- ✅ Automatic retry with exponential backoff
- ✅ Network change detection and graceful handling
- ✅ Session persistence across server restarts
## Monitoring
Check logs for network resilience status:
```bash
# Network change detection
2025-07-17 12:34:56 INFO Network change detected
2025-07-17 12:34:56 INFO Pausing all active uploads due to network change
2025-07-17 12:34:58 INFO Resuming all paused uploads
# Upload session activity
2025-07-17 12:35:00 INFO Active upload sessions: 3
2025-07-17 12:35:00 INFO Network resilience: 5 uploads active
```
## Implementation Notes
### Non-Intrusive Design
- **No core function modifications**: All existing upload handlers remain unchanged
- **Backward compatibility**: Traditional uploads continue to work exactly as before
- **Optional features**: Network resilience only activates when `chunkeduploadsenabled = true`
- **Gradual adoption**: Users can test chunked uploads alongside existing uploads
### File Structure
```
cmd/server/
├── main.go # Core server (minimal changes)
├── helpers.go # Router setup (minimal changes)
├── upload_session.go # NEW: Session management
├── network_resilience.go # NEW: Network monitoring
├── chunked_upload_handler.go # NEW: Chunked upload API
└── integration.go # NEW: Non-intrusive integration
```
This approach ensures that your existing upload functionality remains stable while adding robust network switching capabilities for users who need them.