uuxo/hmac-file-server
1
1
Fork 0
Code Pull Requests Actions Packages Projects Releases 1 Wiki Activity

release: hmac-file-server 3.2

Browse Source
This commit is contained in:
Alexander Renz
2025-06-13 04:24:11 +02:00
parent cc3a4f4dd7
commit 16f50940d0
34 changed files with 10354 additions and 2255 deletions
Download Patch File Download Diff File Expand all files Collapse all files

592
README.MD
View File

@ -1,216 +1,454 @@
# HMAC File Server Release Notes
# HMAC File Server 3.2
**HMAC File Server** is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, and optional ClamAV scanning for file integrity and security. This server is built with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration.
## Overview
The **HMAC File Server** ensures secure file uploads and downloads using HMAC authentication and JWT tokens. It incorporates comprehensive security features, file versioning, deduplication, ISO container support, virus scanning, and Unix socket support for enhanced flexibility. Redis integration provides efficient caching and session management. Prometheus metrics and graceful shutdown mechanisms ensure reliable and efficient file handling.
Special thanks to **Thomas Leister** for inspiration drawn from [prosody-filer](https://github.com/ThomasLeister/prosody-filer).
## Features
- **Multiple Authentication Methods**: HMAC-based authentication and JWT token support
- **Multiple Protocol Support**: v1 (legacy), v2, v3 (mod_http_upload_external), and token-based uploads
- **File Management**: Deduplication, configurable TTL for automatic file cleanup
- **Upload Methods**: POST multipart uploads, PUT uploads for legacy protocols, v3 protocol support
- **Security**: Virus scanning via ClamAV, configurable file extensions validation
- **Performance**: Chunked uploads and downloads, worker pool management with auto-scaling
- **Storage Options**: Local storage, ISO container mounting for specialized needs
- **Monitoring**: Prometheus metrics integration with detailed system and operation metrics
- **Network Support**: IPv4/IPv6 dual-stack support with protocol forcing options
- **Configuration**: Hot-reloading of logging settings via SIGHUP signal
- **HMAC Authentication:** Secure file uploads and downloads with HMAC tokens.
- **File Versioning:** Enable versioning for uploaded files with configurable retention.
- **Chunked and Resumable Uploads:** Handle large files efficiently with support for resumable and chunked uploads.
- **ClamAV Scanning:** Optional virus scanning for uploaded files.
- **Prometheus Metrics:** Monitor system and application-level metrics.
- **Redis Integration:** Use Redis for caching or storing application states.
- **File Expiration:** Automatically delete files after a specified TTL.
- **Graceful Shutdown:** Handles signals and ensures proper cleanup.
- **Deduplication:** Remove duplicate files based on hashing for storage efficiency.
## Table of Contents
1. [Installation](#installation)
2. [Configuration](#configuration)
3. [Authentication](#authentication)
4. [API Endpoints](#api-endpoints)
5. [Usage Examples](#usage-examples)
6. [Setup](#setup)
- [Reverse Proxy](#reverse-proxy)
- [Systemd Service](#systemd-service)
7. [Building](#building)
8. [Docker Support](#docker-support)
9. [Changelog](#changelog)
10. [License](#license)
---
## Installation
### Prerequisites
### Quick Installation for XMPP Operators
- Go 1.20+
- Redis (optional, if Redis integration is enabled)
- ClamAV (optional, if file scanning is enabled)
### Clone and Build
The easiest way to install HMAC File Server is using the automated installer:
```bash
git clone https://github.com/your-repo/hmac-file-server.git
git clone https://github.com/PlusOne/hmac-file-server.git
cd hmac-file-server
go build -o hmac-file-server main.go
sudo ./installer.sh
```
The installer will:
- Install Go 1.24 (if needed)
- Create system user and directories
- Build and configure the server
- Set up systemd service
- Optionally install Redis and ClamAV
For detailed installation instructions, see [INSTALL.MD](INSTALL.MD).
### Manual Installation
> **Tip:** You can also run HMAC File Server using Docker Compose for easy deployment. See the Wiki for Docker setup instructions. The official image is available at `ghcr.io/plusone/hmac-file-server:latest`.
#### Prerequisites
- Go **1.24** or higher
- Redis server (optional, for caching)
- ClamAV (optional, for virus scanning)
#### Steps
1. Clone the repository:
```bash
git clone https://github.com/PlusOne/hmac-file-server.git
cd hmac-file-server
```
2. Build the server:
```bash
go build -o hmac-file-server ./cmd/server/main.go
```
3. Generate example configuration:
```bash
./hmac-file-server -genconfig
# or write to file:
./hmac-file-server -genconfig-path config.toml
```
4. Create necessary directories:
```bash
mkdir -p /path/to/hmac-file-server/data/
mkdir -p /path/to/hmac-file-server/deduplication/
```
5. Edit your `config.toml` file with appropriate settings.
6. Start the server:
```bash
./hmac-file-server -config config.toml
```
---
## Uninstallation
The installer script provides comprehensive uninstallation options with data preservation:
```bash
sudo ./installer.sh --uninstall
```
### Uninstall Options
The uninstaller offers five data handling options:
1. **🗑️ Delete all data** - Complete removal (requires typing "DELETE" to confirm)
2. **💾 Preserve uploads and deduplication data** - Keeps important files, removes logs
3. **📋 Preserve all data** - Keeps uploads, deduplication data, and logs
4. **🎯 Custom selection** - Choose exactly what to preserve
5. **❌ Cancel** - Exit without making changes
### Data Preservation
When preserving data, the uninstaller:
- Creates timestamped backups in `/var/backups/hmac-file-server-YYYYMMDD-HHMMSS/`
- Shows file counts and sizes before deletion decisions
- Safely moves data to backup locations
- Provides clear feedback on what was preserved or removed
### What Gets Removed
The uninstaller always removes:
- ✓ Systemd service and service file
- ✓ Installation directory (`/opt/hmac-file-server`)
- ✓ Configuration directory (`/etc/hmac-file-server`)
- ✓ System user (`hmac-server`)
- ✓ Binary files in common locations
Data directories are handled according to your selection.
---
## Configuration
The server configuration is managed through a `config.toml` file. Below are the supported configuration options:
The server uses a comprehensive `config.toml` file with the following main sections:
### **Server Configuration**
### Key Configuration Sections
| Key | Description | Example |
|------------------------|-----------------------------------------------------|---------------------------------|
| `ListenPort` | Port or Unix socket to listen on | `":8080"` |
| `UnixSocket` | Use a Unix socket (`true`/`false`) | `false` |
| `Secret` | Secret key for HMAC authentication | `"your-secret-key"` |
| `StoragePath` | Directory to store uploaded files | `"/mnt/storage/hmac-file-server"` |
| `LogLevel` | Logging level (`info`, `debug`, etc.) | `"info"` |
| `LogFile` | Log file path (optional) | `"/var/log/hmac-file-server.log"` |
| `MetricsEnabled` | Enable Prometheus metrics (`true`/`false`) | `true` |
| `MetricsPort` | Prometheus metrics server port | `"9090"` |
| `FileTTL` | File Time-to-Live duration | `"168h0m0s"` |
| `DeduplicationEnabled` | Enable file deduplication based on hashing | `true` |
| `MinFreeBytes` | Minimum free space required on storage path (in bytes) | `104857600` |
- **[server]**: Basic server settings (port, storage, metrics)
- **[security]**: HMAC secrets, JWT configuration
- **[uploads/downloads]**: File handling settings, allowed extensions
- **[logging]**: Log levels, file rotation settings
- **[clamav]**: Virus scanning configuration
- **[redis]**: Cache and session management
- **[workers]**: Thread pool and performance tuning
- **[iso]**: ISO container mounting (specialized storage)
- **[timeouts]**: HTTP timeout configurations
### **Uploads**
| Key | Description | Example |
|----------------------------|-----------------------------------------------|-------------|
| `ResumableUploadsEnabled` | Enable resumable uploads | `true` |
| `ChunkedUploadsEnabled` | Enable chunked uploads | `true` |
| `ChunkSize` | Chunk size for chunked uploads (bytes) | `1048576` |
| `AllowedExtensions` | Allowed file extensions for uploads | `[".png", ".jpg"]` |
### **Time Settings**
| Key | Description | Example |
|------------------|--------------------------------|----------|
| `ReadTimeout` | HTTP server read timeout | `"2h"` |
| `WriteTimeout` | HTTP server write timeout | `"2h"` |
| `IdleTimeout` | HTTP server idle timeout | `"2h"` |
### **ClamAV Configuration**
| Key | Description | Example |
|--------------------|-------------------------------------------|----------------------------------|
| `ClamAVEnabled` | Enable ClamAV virus scanning (`true`) | `true` |
| `ClamAVSocket` | Path to ClamAV Unix socket | `"/var/run/clamav/clamd.ctl"` |
| `NumScanWorkers` | Number of workers for file scanning | `2` |
### **Redis Configuration**
| Key | Description | Example |
|----------------------------|----------------------------------|-------------------|
| `RedisEnabled` | Enable Redis integration | `true` |
| `RedisDBIndex` | Redis database index | `0` |
| `RedisAddr` | Redis server address | `"localhost:6379"`|
| `RedisPassword` | Password for Redis authentication| `""` |
| `RedisHealthCheckInterval` | Health check interval for Redis | `"30s"` |
### **Workers and Connections**
| Key | Description | Example |
|------------------------|------------------------------------|-------------------|
| `NumWorkers` | Number of upload workers | `2` |
| `UploadQueueSize` | Size of the upload queue | `50` |
---
## Running the Server
### Basic Usage
Run the server with a configuration file:
```bash
./hmac-file-server -config ./config.toml
```
### Metrics Server
If `MetricsEnabled` is `true`, the Prometheus metrics server will run on the port specified in `MetricsPort` (default: `9090`).
---
## Development Notes
- **Versioning:** Enabled via `EnableVersioning`. Ensure `MaxVersions` is set appropriately to prevent storage issues.
- **File Cleaner:** The file cleaner runs hourly and deletes files older than the configured `FileTTL`.
- **Redis Health Check:** Automatically monitors Redis connectivity and logs warnings on failure.
---
## Testing
To run the server locally for development:
```bash
go run main.go -config ./config.toml
```
Use tools like **cURL** or **Postman** to test file uploads and downloads.
### Example File Upload with HMAC Token
```bash
curl -X PUT -H "Authorization: Bearer <HMAC-TOKEN>" -F "file=@example.txt" http://localhost:8080/uploads/example.txt
```
Replace `<HMAC-TOKEN>` with a valid HMAC signature generated using the configured `Secret`.
---
## Monitoring
Prometheus metrics include:
- File upload/download durations
- Memory usage
- CPU usage
- Active connections
- HTTP requests metrics (total, method, path)
---
## Example `config.toml`
### Example Configuration
```toml
[server]
bind_ip = "0.0.0.0"
listenport = "8080"
unixsocket = false
storagepath = "/mnt/storage/"
loglevel = "info"
logfile = "/var/log/file-server.log"
storagepath = "./uploads"
metricsenabled = true
metricsport = "9090"
DeduplicationEnabled = true
filettl = "336h" # 14 days
minfreebytes = 104857600 # 100 MB in bytes
[timeouts]
readtimeout = "4800s"
writetimeout = "4800s"
idletimeout = "24h"
deduplicationenabled = true
filenaming = "HMAC" # Options: "HMAC", "original", "None"
forceprotocol = "auto" # Options: "ipv4", "ipv6", "auto"
[security]
secret = "example-secret-key"
[versioning]
enableversioning = false
maxversions = 1
secret = "your-secure-hmac-secret"
enablejwt = false
jwtsecret = "your-jwt-secret"
jwtalgorithm = "HS256"
jwtexpiration = "24h"
[uploads]
resumableuploadsenabled = true
allowedextensions = [".txt", ".pdf", ".jpg", ".png", ".zip"]
chunkeduploadsenabled = true
chunksize = 8192
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm", ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"]
[clamav]
clamavenabled = true
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 2
[redis]
redisenabled = true
redisdbindex = 0
redisaddr = "localhost:6379"
redispassword = ""
redishealthcheckinterval = "120s"
[workers]
numworkers = 2
uploadqueuesize = 50
chunksize = "10MB"
```
This configuration file is set up with essential features like Prometheus integration, ClamAV scanning, and file handling with deduplication and versioning options. Adjust the settings according to your infrastructure needs.
For complete configuration details, see the [Wiki](./WIKI.MD).
### Additional Features
---
- **Deduplication**: Automatically remove duplicate files based on hashing.
- **Versioning**: Store multiple versions of files and keep a maximum of `MaxVersions` versions.
- **ClamAV Integration**: Scan uploaded files for viruses using ClamAV.
- **Redis Caching**: Utilize Redis for caching file metadata for faster access.
## Authentication
This release ensures an efficient and secure file management system, suited for environments requiring high levels of data security and availability.
```
The server supports multiple authentication methods:
### 1. HMAC Authentication (Default)
- **Legacy v1**: Basic HMAC with path + content length
- **v2**: Enhanced HMAC with content type validation
- **Token**: Alternative HMAC parameter name
- **v3**: mod_http_upload_external compatible with expiration
### 2. JWT Authentication
When `enablejwt = true`:
- Bearer tokens in Authorization header
- Query parameter `token` as fallback
- Configurable expiration and algorithm
### Authentication Examples
```bash
# HMAC v2 upload
curl -X PUT "http://localhost:8080/myfile.txt?v2=HMAC_SIGNATURE" -d @file.txt
# JWT upload
curl -X POST -H "Authorization: Bearer JWT_TOKEN" -F 'file=@myfile.txt' http://localhost:8080/upload
# v3 protocol (mod_http_upload_external)
curl -X PUT "http://localhost:8080/upload/file.txt?v3=SIGNATURE&expires=TIMESTAMP" -d @file.txt
```
---
## API Endpoints
### Upload Endpoints
- **POST /upload**: Multipart form uploads (modern clients)
- **PUT /{filename}**: Direct uploads with HMAC or JWT authentication
- **PUT with v3 protocol**: mod_http_upload_external compatible uploads
### Download Endpoints
- **GET /{filename}**: Direct file downloads
- **HEAD /{filename}**: File metadata (size, type)
- **GET /download/{filename}**: Alternative download endpoint
### Management Endpoints
- **GET /health**: Health check endpoint for monitoring
- **GET /metrics**: Prometheus metrics (if enabled)
- **Various helper endpoints**: Defined in router setup
---
## Usage Examples
### Upload Examples
#### Multipart POST Upload
```bash
curl -X POST -F 'file=@example.jpg' \
-H "X-Signature: HMAC_OF_PATH" \
http://localhost:8080/upload
```
#### Legacy PUT Upload (v2)
```bash
# Calculate HMAC of: filename + "\x00" + content_length + "\x00" + content_type
curl -X PUT "http://localhost:8080/example.jpg?v2=CALCULATED_HMAC" \
--data-binary @example.jpg
```
#### v3 Protocol Upload (mod_http_upload_external)
```bash
# HMAC of: "PUT\n{timestamp}\n/path/to/file"
curl -X PUT "http://localhost:8080/upload/file.txt?v3=SIGNATURE&expires=1234567890" \
--data-binary @file.txt
```
#### JWT Upload
```bash
curl -X POST \
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
-F 'file=@example.jpg' \
http://localhost:8080/upload
```
### Download Examples
#### Direct Download
```bash
curl http://localhost:8080/example.jpg -o downloaded_file.jpg
```
#### Get File Info
```bash
curl -I http://localhost:8080/example.jpg
```
#### Health Check
```bash
curl http://localhost:8080/health
```
---
## Setup
### Reverse Proxy
#### Nginx Configuration
```nginx
server {
listen 80;
server_name your-domain.com;
client_max_body_size 10G; # Important for large uploads
location / {
proxy_pass http://localhost:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeout settings for large uploads
proxy_read_timeout 300;
proxy_connect_timeout 60;
proxy_send_timeout 300;
}
}
```
#### Apache2 Configuration
```apache
<VirtualHost *:80>
ServerName your-domain.com
ProxyPreserveHost On
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
# Large upload support
LimitRequestBody 10737418240 # 10GB
ProxyTimeout 300
</VirtualHost>
```
### Systemd Service
```ini
[Unit]
Description=HMAC File Server
After=network.target redis.service
[Service]
Type=simple
ExecStart=/path/to/hmac-file-server -config /path/to/config.toml
ExecReload=/bin/kill -SIGHUP $MAINPID
WorkingDirectory=/path/to/hmac-file-server
Restart=always
RestartSec=10
User=hmac-server
Group=hmac-server
# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/path/to/uploads /path/to/logs
[Install]
WantedBy=multi-user.target
```
Enable and start:
```bash
sudo systemctl daemon-reload
sudo systemctl enable hmac-file-server
sudo systemctl start hmac-file-server
# Reload configuration (logging settings)
sudo systemctl reload hmac-file-server
```
---
## Building
### Local Build
```bash
go build -o hmac-file-server ./cmd/server/main.go
```
### Cross-Platform Builds
```bash
# Linux amd64
GOOS=linux GOARCH=amd64 go build -o hmac-file-server-linux-amd64 ./cmd/server/main.go
# Linux arm64
GOOS=linux GOARCH=arm64 go build -o hmac-file-server-linux-arm64 ./cmd/server/main.go
# Windows
GOOS=windows GOARCH=amd64 go build -o hmac-file-server-windows-amd64.exe ./cmd/server/main.go
```
---
## Docker Support
### Quick Start with Docker Compose
```yaml
version: '3.8'
services:
hmac-file-server:
image: ghcr.io/plusone/hmac-file-server:latest
ports:
- "8080:8080"
- "9090:9090" # Metrics
volumes:
- ./config:/etc/hmac-file-server
- ./uploads:/opt/hmac-file-server/data/uploads
environment:
- CONFIG_PATH=/etc/hmac-file-server/config.toml
restart: unless-stopped
```
### Docker Build
```bash
docker build -t hmac-file-server .
docker run -p 8080:8080 -v $(pwd)/config.toml:/etc/hmac-file-server/config.toml hmac-file-server
```
See the Wiki for detailed Docker setup instructions.
---
## Changelog
### Version 3.2 (Stable)
- **Development Version**: Active development branch with latest features
- **Enhanced Documentation**: Updated comprehensive documentation and protocol specifications
- **Configuration Improvements**: Better configuration validation and structure
- **Authentication System**: Full JWT and multi-protocol HMAC support
### Version 3.1-Stable (2025-06-08)
- **v3 Protocol Support**: Added mod_http_upload_external compatibility
- **Enhanced Authentication**: Improved HMAC validation with multiple protocol support
- **JWT Integration**: Complete JWT authentication system
- **File Naming Options**: HMAC-based or original filename preservation
- **Network Improvements**: IPv4/IPv6 dual-stack with protocol forcing
### Version 3.0-Stable (2025-06-07)
- **Docker Support**: Official Docker images and compose files
- **Go 1.24 Requirement**: Updated minimum Go version
- **Configuration Improvements**: Better validation and hot-reloading
- **Performance Enhancements**: Worker auto-scaling and memory optimization
### Previous Versions
See [CHANGELOG.MD](./CHANGELOG.MD) for complete version history.
---
## License
MIT License
Copyright (c) 2025 Alexander Renz
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.