Add comprehensive configuration and testing for HMAC File Server 3.2

- Introduced configuration files for Docker, Podman, and SystemD deployments.
- Implemented a comprehensive test suite for HMAC validation, file uploads, and network resilience.
- Added debugging scripts for live monitoring of upload issues and service status.
- Created minimal configuration for testing purposes.
- Developed multiple test scripts to validate HMAC calculations and response handling.
- Enhanced upload tests to cover various scenarios including invalid HMAC and unsupported file extensions.
- Improved logging and error analysis capabilities for better diagnostics.
This commit is contained in:
2025-07-20 18:04:23 +00:00
parent f8e4d8fcba
commit 68ede52336
37 changed files with 2754 additions and 591 deletions

View File

@ -0,0 +1,116 @@
# HMAC File Server 3.2 Test Suite
This directory contains comprehensive testing tools for the HMAC File Server 3.2 "Tremora del Terra".
## 🚀 Quick Start
Run the complete test suite:
```bash
./comprehensive_test_suite.sh
```
## 📋 Test Coverage
The comprehensive test suite covers:
### ✅ Core Functionality
- **HMAC Validation**: Ensures proper authentication
- **File Extensions**: Tests allowed/blocked file types
- **Upload Mechanics**: Validates upload process
- **Server Health**: Checks service availability
### 🎥 XMPP Integration
- **MP4 Upload**: Tests video file sharing for XMPP clients
- **Image Upload**: Tests image sharing (PNG, JPEG)
- **File Size Limits**: Validates large file handling
### 🌐 Network Resilience (3.2 Features)
- **Health Monitoring**: Tests network resilience endpoints
- **Metrics Collection**: Validates monitoring capabilities
- **Mobile Switching**: Supports seamless network transitions
### 🚫 Security Testing
- **Invalid HMAC**: Ensures rejected authentication fails
- **Unsupported Extensions**: Confirms blocked file types
- **Path Validation**: Tests file path sanitization
## 🔧 Commands
```bash
# Run all tests
./comprehensive_test_suite.sh
# Setup test files only
./comprehensive_test_suite.sh setup
# Clean up test files
./comprehensive_test_suite.sh clean
# Show help
./comprehensive_test_suite.sh help
```
## 📊 Test Results
Tests generate detailed logs with:
-**Pass/Fail status** for each test
- 🕒 **Timestamps** for performance tracking
- 📝 **Detailed output** saved to `/tmp/hmac_test_results_*.log`
- 📈 **Summary statistics** (passed/failed counts)
## 🎯 Expected Results
When all systems are working correctly:
- **✅ PASS**: HMAC validation
- **✅ PASS**: MP4 upload (XMPP)
- **✅ PASS**: Image upload
- **✅ PASS**: Large file upload
- **✅ PASS**: Server health check
- **❌ FAIL**: Invalid HMAC (should fail)
- **❌ FAIL**: Unsupported extension (should fail)
## 🔍 Troubleshooting
### Common Issues
1. **Connection refused**: Check if server is running
2. **403 Forbidden**: Verify HMAC key configuration
3. **400 Bad Request**: Check file extension configuration
4. **Timeout**: Large files may need adjusted timeouts
### Debug Mode
For detailed debugging, check server logs:
```bash
sudo journalctl -u hmac-file-server -f
```
## 📁 File Cleanup
The test suite automatically cleans up temporary files, but if needed:
```bash
rm -f /tmp/test_*.{txt,mp4,bin,png,xyz}
rm -f /tmp/hmac_test_results_*.log
```
## 🔧 Configuration
Tests use these defaults (modify in script if needed):
- **Base URL**: `https://xmpp.uuxo.net`
- **Test User**: `c184288b79f8b7a6f7d87ac7f1fb1ce6dcf49a80`
- **HMAC Key**: Configured in script
## 📝 Legacy Test Files
This comprehensive suite replaces these scattered root-level test files:
- `test-hmac-fixed.sh` → Integrated into comprehensive suite
- `test-upload.sh` → Covered by upload tests
- `debug-uploads.sh` → Debug logging integrated
- `comprehensive_upload_test.sh` → Replaced by this suite
- Various monitor scripts → Health checks integrated
## 🎉 3.2 "Tremora del Terra" Features Tested
-**Enhanced Network Resilience**: 1-second detection
-**Mobile Network Switching**: WLAN ↔ IPv6 5G seamless transitions
-**XMPP File Sharing**: Conversations/Gajim compatibility
-**Configuration Validation**: Proper extension loading
-**Production Deployment**: SystemD, Docker, Podman support

223
tests/debug-uploads.sh Executable file
View File

@ -0,0 +1,223 @@
#!/bin/bash
# Live debugging script for HMAC File Server upload issues
# Monitors logs in real-time and provides detailed diagnostics
set -e
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'
log_info() { echo -e "${BLUE}[INFO]${NC} $1"; }
log_success() { echo -e "${GREEN}[SUCCESS]${NC} $1"; }
log_warning() { echo -e "${YELLOW}[WARNING]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
# Function to check service status
check_services() {
log_info "=== SERVICE STATUS CHECK ==="
echo "HMAC File Server:"
systemctl is-active hmac-file-server && echo "✅ Running" || echo "❌ Not running"
echo "Nginx:"
systemctl is-active nginx && echo "✅ Running" || echo "❌ Not running"
echo ""
}
# Function to show current configuration
show_config() {
log_info "=== CONFIGURATION SUMMARY ==="
echo "HMAC File Server Config:"
echo "- Max Upload Size: $(grep max_upload_size /opt/hmac-file-server/config.toml | cut -d'"' -f2)"
echo "- Chunk Size: $(grep chunksize /opt/hmac-file-server/config.toml | head -1 | cut -d'"' -f2)"
echo "- Chunked Uploads: $(grep chunkeduploadsenabled /opt/hmac-file-server/config.toml | cut -d'=' -f2 | tr -d ' ')"
echo "- Network Events: $(grep networkevents /opt/hmac-file-server/config.toml | cut -d'=' -f2 | tr -d ' ')"
echo "- Listen Address: $(grep listen_address /opt/hmac-file-server/config.toml | cut -d'"' -f2)"
echo ""
echo "Nginx Config:"
echo "- Client Max Body Size: $(nginx -T 2>/dev/null | grep client_max_body_size | head -1 | awk '{print $2}' | tr -d ';')"
echo "- Proxy Buffering: $(nginx -T 2>/dev/null | grep proxy_request_buffering | head -1 | awk '{print $2}' | tr -d ';')"
echo "- Proxy Timeouts: $(nginx -T 2>/dev/null | grep proxy_read_timeout | head -1 | awk '{print $2}' | tr -d ';')"
echo ""
}
# Function to monitor logs in real-time
monitor_logs() {
log_info "=== STARTING LIVE LOG MONITORING ==="
log_warning "Press Ctrl+C to stop monitoring"
echo ""
# Create named pipes for log monitoring
mkfifo /tmp/hmac_logs /tmp/nginx_logs 2>/dev/null || true
# Start log monitoring in background
journalctl -u hmac-file-server -f --no-pager > /tmp/hmac_logs &
HMAC_PID=$!
tail -f /var/log/nginx/access.log > /tmp/nginx_logs &
NGINX_PID=$!
# Monitor both logs with timestamps
{
while read line; do
echo -e "${BLUE}[HMAC]${NC} $line"
done < /tmp/hmac_logs &
while read line; do
if [[ "$line" =~ (PUT|POST) ]] && [[ "$line" =~ (40[0-9]|50[0-9]) ]]; then
echo -e "${RED}[NGINX-ERROR]${NC} $line"
elif [[ "$line" =~ (PUT|POST) ]]; then
echo -e "${GREEN}[NGINX-OK]${NC} $line"
else
echo -e "${YELLOW}[NGINX]${NC} $line"
fi
done < /tmp/nginx_logs &
wait
}
# Cleanup on exit
trap 'kill $HMAC_PID $NGINX_PID 2>/dev/null; rm -f /tmp/hmac_logs /tmp/nginx_logs' EXIT
}
# Function to test file upload
test_upload() {
local test_file="$1"
local test_size="${2:-1MB}"
if [ -z "$test_file" ]; then
test_file="/tmp/test_upload_${test_size}.bin"
log_info "Creating test file: $test_file ($test_size)"
case "$test_size" in
"1MB") dd if=/dev/urandom of="$test_file" bs=1M count=1 >/dev/null 2>&1 ;;
"10MB") dd if=/dev/urandom of="$test_file" bs=1M count=10 >/dev/null 2>&1 ;;
"100MB") dd if=/dev/urandom of="$test_file" bs=1M count=100 >/dev/null 2>&1 ;;
"1GB") dd if=/dev/urandom of="$test_file" bs=1M count=1024 >/dev/null 2>&1 ;;
esac
log_success "Test file created: $(ls -lh $test_file | awk '{print $5}')"
fi
# Get current timestamp for log filtering
log_info "=== TESTING UPLOAD: $test_file ==="
# Test with curl - simulate XMPP client behavior
local url="https://share.uuxo.net/test_path/test_file_$(date +%s).bin"
log_info "Testing upload to: $url"
curl -X PUT \
-H "Content-Type: application/octet-stream" \
-H "User-Agent: TestClient/1.0" \
--data-binary "@$test_file" \
"$url" \
-v \
-w "Response: %{http_code}, Size: %{size_upload}, Time: %{time_total}s\n" \
2>&1 | tee /tmp/curl_test.log
echo ""
log_info "Upload test completed. Check logs above for details."
}
# Function to analyze recent errors
analyze_errors() {
log_info "=== ERROR ANALYSIS ==="
echo "Recent 400 errors from Nginx:"
tail -100 /var/log/nginx/access.log | grep " 400 " | tail -5
echo ""
echo "Recent HMAC file server errors:"
tail -100 /opt/hmac-file-server/data/logs/hmac-file-server.log | grep -i error | tail -5
echo ""
echo "File extension configuration:"
grep -A 20 "allowedextensions" /opt/hmac-file-server/config.toml | head -10
echo ""
}
# Function to check file permissions and disk space
check_system() {
log_info "=== SYSTEM CHECK ==="
echo "Disk space:"
df -h /opt/hmac-file-server/data/uploads
echo ""
echo "Upload directory permissions:"
ls -la /opt/hmac-file-server/data/uploads/
echo ""
echo "Process information:"
ps aux | grep hmac-file-server | grep -v grep
echo ""
echo "Network connections:"
netstat -tlnp | grep :8080
echo ""
}
# Main menu
main_menu() {
echo -e "${BLUE}╔═══════════════════════════════════════════════════════════╗${NC}"
echo -e "${BLUE}${NC} HMAC File Server Live Debugging Tool ${BLUE}${NC}"
echo -e "${BLUE}╚═══════════════════════════════════════════════════════════╝${NC}"
echo ""
echo "1) Check service status"
echo "2) Show configuration summary"
echo "3) Start live log monitoring"
echo "4) Test file upload (1MB)"
echo "5) Test file upload (10MB)"
echo "6) Test file upload (100MB)"
echo "7) Analyze recent errors"
echo "8) Check system resources"
echo "9) Full diagnostic run"
echo "0) Exit"
echo ""
read -p "Choose an option [0-9]: " choice
case $choice in
1) check_services ;;
2) show_config ;;
3) monitor_logs ;;
4) test_upload "" "1MB" ;;
5) test_upload "" "10MB" ;;
6) test_upload "" "100MB" ;;
7) analyze_errors ;;
8) check_system ;;
9)
check_services
show_config
check_system
analyze_errors
;;
0) exit 0 ;;
*) log_error "Invalid option. Please choose 0-9." ;;
esac
echo ""
read -p "Press Enter to continue..."
main_menu
}
# Handle command line arguments
case "${1:-}" in
"monitor") monitor_logs ;;
"test") test_upload "$2" "$3" ;;
"analyze") analyze_errors ;;
"status") check_services ;;
"config") show_config ;;
"system") check_system ;;
*) main_menu ;;
esac

View File

@ -0,0 +1,7 @@
[server]
listen_address = "8080"
storage_path = "/tmp/test-uploads"
metrics_enabled = true
[security]
secret = "test-secret-key"

50
tests/test-hmac-fixed.sh Executable file
View File

@ -0,0 +1,50 @@
#!/bin/bash
# Corrected HMAC calculation test
# Configuration
BASE_PATH="c184288b79f8b7a6f7d87ac7f1fb1ce6dcf49a80"
SUB_PATH="debugfixed"
FILENAME="test.mp4"
FULL_PATH="$BASE_PATH/$SUB_PATH/$FILENAME"
SECRET="f6g4ldPvQM7O2UTFeBEUUj33VrXypDAcsDt0yqKrLiOr5oQW"
# Create test file
TEST_FILE="/tmp/test_fixed.mp4"
echo -n "Test content for HMAC debugging" > "$TEST_FILE"
FILE_SIZE=$(stat -c%s "$TEST_FILE")
echo "=== Corrected HMAC Test ==="
echo "File: $TEST_FILE ($FILE_SIZE bytes)"
echo "Path: $FULL_PATH"
echo ""
# Correct HMAC calculation (using actual space character, not literal \x20)
# The server does: fileStorePath + "\x20" + contentLength
# In bash, \x20 means actual space character (0x20)
HMAC_MESSAGE="$FULL_PATH $FILE_SIZE"
echo "HMAC message: '$HMAC_MESSAGE'"
# Calculate HMAC
HMAC_CALC=$(printf "%s" "$HMAC_MESSAGE" | openssl dgst -sha256 -hmac "$SECRET" | cut -d' ' -f2)
echo "Calculated HMAC: $HMAC_CALC"
echo ""
# Test the upload
echo "=== Testing Upload ==="
curl -X PUT \
-H "Content-Type: video/mp4" \
-H "User-Agent: TestFixed/1.0" \
--data-binary "@$TEST_FILE" \
"https://share.uuxo.net/$FULL_PATH?v=$HMAC_CALC" \
-v \
-s \
-w "\nFinal Response: %{http_code}\n" \
2>&1 | grep -E "(PUT|HTTP/2|Final Response|Content-Length:|User-Agent:)"
echo ""
echo "=== Server Logs ==="
sleep 2
tail -10 /opt/hmac-file-server/data/logs/hmac-file-server.log | grep -E "(handleLegacyUpload|validateHMAC|protocol.*calculated|successful)" | tail -5
# Clean up
rm -f "$TEST_FILE"

55
tests/test-response-body.sh Executable file
View File

@ -0,0 +1,55 @@
#!/bin/bash
# Test with full response body capture
BASE_PATH="c184288b79f8b7a6f7d87ac7f1fb1ce6dcf49a80"
SUB_PATH="responsebody"
FILENAME="test.mp4"
FULL_PATH="$BASE_PATH/$SUB_PATH/$FILENAME"
SECRET="f6g4ldPvQM7O2UTFeBEUUj33VrXypDAcsDt0yqKrLiOr5oQW"
TEST_FILE="/tmp/test_response.mp4"
echo -n "Response body test" > "$TEST_FILE"
FILE_SIZE=$(stat -c%s "$TEST_FILE")
HMAC_MESSAGE="$FULL_PATH $FILE_SIZE"
HMAC_CALC=$(printf "%s" "$HMAC_MESSAGE" | openssl dgst -sha256 -hmac "$SECRET" | cut -d' ' -f2)
echo "=== Testing with Full Response Capture ==="
echo "Path: $FULL_PATH"
echo "HMAC: $HMAC_CALC"
echo ""
# Capture full response including body
RESPONSE=$(curl -X PUT \
-H "Content-Type: video/mp4" \
-H "User-Agent: TestResponseBody/1.0" \
--data-binary "@$TEST_FILE" \
"https://share.uuxo.net/$FULL_PATH?v=$HMAC_CALC" \
-s \
-w "CURL_STATUS:%{http_code}\nCURL_SIZE:%{size_upload}\n" \
2>&1)
echo "=== Full Response ==="
echo "$RESPONSE"
echo ""
# Extract just the response body (everything before CURL_STATUS)
RESPONSE_BODY=$(echo "$RESPONSE" | sed '/CURL_STATUS:/,$d')
echo "=== Response Body Only ==="
echo "'$RESPONSE_BODY'"
echo ""
# Check response length
RESPONSE_LENGTH=${#RESPONSE_BODY}
echo "Response body length: $RESPONSE_LENGTH characters"
if [ $RESPONSE_LENGTH -eq 32 ]; then
echo "✅ Response is exactly 32 characters (matches Nginx logs)"
elif [ $RESPONSE_LENGTH -eq 0 ]; then
echo "⚠️ Empty response body"
else
echo " Different response length than expected"
fi
# Clean up
rm -f "$TEST_FILE"

100
tests/test-upload-advanced.sh Executable file
View File

@ -0,0 +1,100 @@
#!/bin/bash
# Advanced test to diagnose XMPP upload issues
echo "=== HMAC File Server Upload Debugging ==="
echo ""
# First, let's simulate exactly what we see in the logs
# Using a real path from the failed uploads
BASE_PATH="c184288b79f8b7a6f7d87ac7f1fb1ce6dcf49a80"
SUB_PATH="testdebug"
FILENAME="test.mp4"
FULL_PATH="$BASE_PATH/$SUB_PATH/$FILENAME"
# Create test file
TEST_FILE="/tmp/test_debug.mp4"
echo "Creating test content..." > "$TEST_FILE"
FILE_SIZE=$(stat -c%s "$TEST_FILE")
echo "Test file: $TEST_FILE"
echo "File size: $FILE_SIZE bytes"
echo "Upload path: $FULL_PATH"
echo ""
# Let's calculate the HMAC like the server does
# For v protocol: fileStorePath + "\x20" + contentLength
SECRET="f6g4ldPvQM7O2UTFeBEUUj33VrXypDAcsDt0yqKrLiOr5oQW"
# Method 1: Calculate HMAC using the file size
HMAC_MESSAGE="$FULL_PATH $(printf '\x20')$FILE_SIZE"
HMAC_CALC=$(echo -n "$HMAC_MESSAGE" | openssl dgst -sha256 -hmac "$SECRET" | cut -d' ' -f2)
echo "HMAC calculation:"
echo "Message: '$FULL_PATH\\x20$FILE_SIZE'"
echo "HMAC: $HMAC_CALC"
echo ""
# Test 1: Upload with correct HMAC
echo "=== Test 1: Upload with calculated HMAC ==="
curl -X PUT \
-H "Content-Type: video/mp4" \
-H "User-Agent: TestDebugCorrect/1.0" \
--data-binary "@$TEST_FILE" \
"https://share.uuxo.net/$FULL_PATH?v=$HMAC_CALC" \
-v \
-w "\nResponse: %{http_code}, Time: %{time_total}s\n" \
2>&1 | grep -E "(Response|HTTP/|Content-Length|User-Agent)"
echo ""
# Test 2: Upload with Content-Length: 0 (simulating potential XMPP issue)
echo "=== Test 2: Upload with Content-Length: 0 ==="
curl -X PUT \
-H "Content-Type: video/mp4" \
-H "Content-Length: 0" \
-H "User-Agent: TestDebugZeroLength/1.0" \
--data-binary "@$TEST_FILE" \
"https://share.uuxo.net/$FULL_PATH?v=$HMAC_CALC" \
-v \
-w "\nResponse: %{http_code}, Time: %{time_total}s\n" \
2>&1 | grep -E "(Response|HTTP/|Content-Length|User-Agent)"
echo ""
# Test 3: Upload without Content-Length header
echo "=== Test 3: Upload using chunked transfer (no Content-Length) ==="
curl -X PUT \
-H "Content-Type: video/mp4" \
-H "Transfer-Encoding: chunked" \
-H "User-Agent: TestDebugChunked/1.0" \
--data-binary "@$TEST_FILE" \
"https://share.uuxo.net/$FULL_PATH?v=$HMAC_CALC" \
-v \
-w "\nResponse: %{http_code}, Time: %{time_total}s\n" \
2>&1 | grep -E "(Response|HTTP/|Transfer-Encoding|User-Agent)"
echo ""
# Test 4: Calculate HMAC with ContentLength 0 (what might be happening)
HMAC_MESSAGE_ZERO="$FULL_PATH $(printf '\x20')0"
HMAC_CALC_ZERO=$(echo -n "$HMAC_MESSAGE_ZERO" | openssl dgst -sha256 -hmac "$SECRET" | cut -d' ' -f2)
echo "=== Test 4: Upload with HMAC calculated for ContentLength=0 ==="
echo "HMAC for zero length: $HMAC_CALC_ZERO"
curl -X PUT \
-H "Content-Type: video/mp4" \
-H "User-Agent: TestDebugZeroHMAC/1.0" \
--data-binary "@$TEST_FILE" \
"https://share.uuxo.net/$FULL_PATH?v=$HMAC_CALC_ZERO" \
-v \
-w "\nResponse: %{http_code}, Time: %{time_total}s\n" \
2>&1 | grep -E "(Response|HTTP/|Content-Length|User-Agent)"
echo ""
echo "=== Recent server logs ==="
sleep 2
tail -15 /opt/hmac-file-server/data/logs/hmac-file-server.log | grep -v "Interface\|RTT\|Loss" | tail -10
# Cleanup
rm -f "$TEST_FILE"

38
tests/test-upload.sh Executable file
View File

@ -0,0 +1,38 @@
#!/bin/bash
# Test script to trace 400 errors in HMAC file server uploads
# Test URL from the logs
TEST_URL="https://share.uuxo.net/c184288b79f8b7a6f7d87ac7f1fb1ce6dcf49a80/test/test.mp4?v=test123"
echo "Testing with a simple small file..."
# Create a small test file
echo "Test content for upload debugging" > /tmp/test_upload.mp4
echo "Attempting upload with curl..."
curl -X PUT \
-H "Content-Type: video/mp4" \
-H "User-Agent: TestDebug/1.0" \
--data-binary "@/tmp/test_upload.mp4" \
"$TEST_URL" \
-v \
-w "\n\nResponse Code: %{http_code}\nTotal Time: %{time_total}s\nSize Uploaded: %{size_upload} bytes\n" \
2>&1
echo -e "\n\nNow checking the logs for this specific request..."
# Wait a moment for logs to be written
sleep 2
# Check recent logs
echo "=== HMAC File Server Logs ==="
tail -10 /opt/hmac-file-server/data/logs/hmac-file-server.log | grep -v "Interface\|RTT\|Loss"
echo -e "\n=== Nginx Access Log ==="
tail -5 /var/log/nginx/access.log | grep PUT
echo -e "\n=== Nginx Error Log ==="
tail -5 /var/log/nginx/upload_errors.log
# Clean up
rm -f /tmp/test_upload.mp4