# HMAC File Server A secure file server with HMAC authentication and configurable features. **HMAC File Server** is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, deduplication, optional ClamAV scanning for file integrity and security, and image thumbnail generation. This server is built with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration. > **Credits:** The **HMAC File Server** is based on the source code of [Thomas Leister's prosody-filer](https://github.com/ThomasLeister/prosody-filer). Many features and design elements have been inspired or derived from this project. --- ## Features - **HMAC Authentication:** Secure file uploads and downloads with HMAC tokens. - **File Versioning:** Enable versioning for uploaded files with configurable retention. - **Chunked and Resumable Uploads:** Handle large files efficiently with support for resumable and chunked uploads. - **Deduplication:** Automatically remove duplicate files based on hashing for storage efficiency. - **ClamAV Scanning:** Optional virus scanning for uploaded files. - **Prometheus Metrics:** Monitor system and application-level metrics. - **Redis Integration:** Use Redis for caching or storing application states. - **File Expiration:** Automatically delete files after a specified TTL. - **Graceful Shutdown:** Handles signals and ensures proper cleanup. - **Auto-Adjust Worker Scaling:** Dynamically optimize worker threads based on system resources. - **Precaching:** Pre-cache file structures on startup for faster access. - **Thumbnail Creation:** Generate image thumbnails for uploaded files. - **ISO Container Management:** Optional mounting and handling of ISO-based filesystems. --- ## Repository - **Primary Repository**: [GitHub Repository](https://github.com/PlusOne/hmac-file-server) - **Alternative Repository**: [uuxo.net Git Repository](https://git.uuxo.net/uuxo/hmac-file-server) --- ## Installation ### Prerequisites - **Go 1.20+** - **Redis** (optional, if Redis integration is enabled) - **ClamAV** (optional, if file scanning is enabled) - **genisoimage** (optional, if ISO container management is enabled) ### Clone and Build ```bash # Clone from the primary repository git clone https://github.com/PlusOne/hmac-file-server.git # OR clone from the alternative repository git clone https://git.uuxo.net/uuxo/hmac-file-server.git cd hmac-file-server go build -o hmac-file-server main.go ``` #### Build for `arm64` ```bash cd /path/to/hmac-file-server GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.3-stable_arm64 main.go ``` #### Build for `amd64` ```bash cd /path/to/hmac-file-server GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.3-stable_amd64 main.go ``` --- ## Configuration The server configuration is managed through a `config.toml` file. Below are the supported configuration options: ### Auto-Adjust Feature When `AutoAdjustWorkers` is enabled, the number of workers for HMAC operations and ClamAV scans is dynamically determined based on system resources. This ensures efficient resource utilization. If `AutoAdjustWorkers = true`, the values for `NumWorkers` and `NumScanWorkers` in the configuration file will be ignored, and the server will automatically adjust these values. ### Network Events Monitoring Setting `NetworkEvents = false` in the server configuration disables the logging and tracking of network-related events within the application. This means that functionalities such as monitoring IP changes or recording network activity will be turned off. ### Precaching The `precaching` feature allows the server to pre-cache storage paths for faster access. This can improve performance by reducing the time needed to access frequently used storage paths. ### Thumbnail Creation Set `enabled = true` in the `[thumbnails]` section of `config.toml` to enable image thumbnail generation. --- ## Example `config.toml` Below is an example configuration file (`config.toml`) you can use as a reference (with sensitive data replaced by placeholder/example data): ```toml # Server configuration listenport = "8080" # TCP port for incoming requests unixsocket = false # Use Unix domain socket instead of TCP storagepath = "/path/to/hmac-file-server/data/" # Directory to store uploaded files loglevel = "debug" # Logging level: "debug", "info", "warn", "error" logfile = "/path/to/hmac-file-server.log" # Path to log file; leave empty to use stdout metricsenabled = true # Enable Prometheus metrics metricsport = "9090" # Port for Prometheus metrics deduplicationenabled = true minfreebytes = "5GB" # Minimum free disk space required filettl = "2Y" # Time-to-live for files (2 years) filettlenabled = false # Enable TTL checks and cleanup autoadjustworkers = true # Automatically adjust worker threads based on load networkevents = false # Enable detailed network event logging pidfilepath = "./hmac-file-server.pid" # Path to PID file precaching = true # Pre-cache file structures on startup tempPath = "/tmp/hmac-file-server" # Path for temporary file uploads # Deduplication settings [deduplication] enabled = true directory = "/path/to/hmac-file-server/deduplication/" # Path to deduplication metadata store # Thumbnails settings [thumbnails] enabled = true directory = "/path/to/hmac-file-server/thumbnails/" # Directory for storing thumbnails size = "200x200" # Thumbnail dimensions thumbnailintervalscan = "1h" # Interval for scheduled thumbnail generation concurrency = 5 # Number of concurrent thumbnail generation tasks # ISO settings [iso] enabled = false size = "1TB" # Maximum ISO size mountpoint = "/path/to/hmac-file-server/iso/" # ISO mount point charset = "utf-8" # Filesystem character set encoding # Timeout settings [timeouts] readtimeout = "3600s" # Maximum time to read a request (1 hour) writetimeout = "3600s" # Maximum time to write a response (1 hour) idletimeout = "3600s" # Maximum keep-alive time for idle connections (1 hour) # Security settings [security] secret = "your-secure-secret-key" # HMAC shared secret key (change to a secure value) # Versioning settings [versioning] enableversioning = false maxversions = 1 # Number of file versions to retain # Upload settings [uploads] resumableuploadsenabled = false chunkeduploadsenabled = true chunksize = "32MB" # Chunk size for uploads allowedextensions = ["*"] # All file types are allowed for uploads # Downloads settings [downloads] resumabledownloadsenabled = false chunkeddownloadsenabled = true chunksize = "32MB" # ClamAV settings [clamav] clamavenabled = true clamavsocket = "/path/to/clamav/clamd.ctl" # Path to ClamAV socket numscanworkers = 4 # Number of concurrent scan workers scanfileextensions = [ ".exe", ".dll", ".bin", ".com", ".bat", ".sh", ".php", ".js" ] # Redis settings [redis] redisenabled = true redisdbindex = 0 redisaddr = "localhost:6379" # Redis server address redispassword = "" # Redis password if required redishealthcheckinterval = "120s" # Interval for Redis health checks # Worker settings [workers] numworkers = 4 uploadqueuesize = 5000 # File settings [file] filerevision = 1 # Internal revision number for file handling logic # Logging settings [logging] level = "info" file = "/var/log/hmac-file-server.log" max_size = 100 max_backups = 7 max_age = 30 compress = true ``` --- ## Configuration Verification The application ensures that all configuration parameters defined in `config.toml` are correctly implemented and operational. During startup, the server performs the following verification steps: 1. **Configuration Parsing:** - Parses the `config.toml` file into the application's `Config` struct. 2. **Validation:** - Checks that all required configuration parameters are set. - Validates the correctness of configuration values (e.g., proper duration formats, non-empty directories). 3. **Service Initialization:** - Initializes services like Redis and ClamAV based on the configuration. - Ensures that conditional services (e.g., Thumbnails, ISO) are only started if enabled in the configuration. If any configuration parameter is missing or invalid, the server will log an error and terminate to prevent running with incorrect settings. --- ## Running the Server ### Basic Usage Run the server with a configuration file: ```bash ./hmac-file-server -config ./config.toml ``` Ensure that the `config.toml` is correctly set up before running the server. --- ### Metrics Server If `metricsenabled` is set to `true`, the Prometheus metrics server will be available on the port specified in `metricsport` (e.g., `http://localhost:9090/metrics`). Additional metrics for deduplication and ISO container operations are now available. --- ## Health Checks The server includes built-in health checks to verify that all configured services are operational. Logs will provide detailed information about the status of each service and any issues encountered during startup. --- ## Testing To run the server locally for development: ```bash go run main.go -config ./config.toml ``` Use tools like **cURL** or **Postman** to test file uploads and downloads. ### Example File Upload with HMAC Token ```bash curl -X PUT -H "Authorization: Bearer " -F "file=@example.txt" http://localhost:8080/uploads/example.txt ``` Replace `` with a valid HMAC signature generated using the configured `secret`. --- ## Monitoring Prometheus metrics include: - **File Operations:** - File upload/download durations - Uploaded/downloaded file sizes - Total number of uploads/downloads - Total number of upload/download errors - **System Metrics:** - Memory usage - CPU usage - Number of goroutines - Active connections - **HTTP Requests:** - Total HTTP requests broken down by method and path Access the metrics at `http://localhost:9090/metrics` (assuming default `metricsport`). --- ## Additional Features - **Deduplication:** Automatically remove duplicate files based on SHA256 hashing to save storage space. - **Versioning:** Store multiple versions of files and retain a configurable number of versions. - **ClamAV Integration:** Scan uploaded files for viruses using ClamAV to ensure file integrity and security. - **Redis Caching:** Utilize Redis for caching file metadata to improve access times and performance. - **Auto-Adjust Worker Scaling:** Optimize the number of workers dynamically based on system resources to maintain optimal performance. - **Precaching:** Pre-cache file structures on startup to reduce access times for frequently accessed files. - **Thumbnail Creation:** Generate image thumbnails for uploaded files to provide quick previews. - **ISO Container Management:** Optional mounting and handling of ISO-based filesystems for specialized storage needs. --- ### Overview of Other Projects (xep0363) | Feature/Project | HMAC FS | mod_http_upload_ext | xmpp-http-upload (horazont) | Prosody Filer | ngx_http_upload | xmpp-http-upload (nyovaya) | |-----------------------------|---------|----------------------|-----------------------------|---------------|------------------|----------------------------| | **Language** | Go | PHP | Python | Go | C (Nginx) | Python | | **Environment** | Standalone | Prosody module | Standalone | Standalone | Nginx | Standalone | | **XMPP** | No | Yes | Yes | Yes | No | Yes | | **External Storage** | Yes | No | Possible via plugins | No | No | Yes | | **Authentication / Security** | HMAC | Token-based | Token-based | None | Basic / None | Token-based | --- ## Build & Run 1. **Clone the Repository:** ```bash git clone https://github.com/PlusOne/hmac-file-server.git cd hmac-file-server ``` 2. **Build the Server:** ```bash go build -o hmac-file-server main.go ``` 3. **Run the Server:** ```bash ./hmac-file-server -config ./config.toml ``` --- ## Best Practices and Recommendations 1. **Configuration Consistency:** - Ensure all configuration keys in `config.toml` match those expected in the code. For example, in the `[thumbnails]` section, use `thumbnailintervalscan` instead of `scanInterval` to align with the code. 2. **Security:** - **HMAC Secret:** - **Change Immediately:** The `secret` under `[security]` should be a strong, unique string. Replace `"your-secure-secret-key"` with a securely generated key. - **ClamAV:** - Ensure ClamAV is installed and the `clamavsocket` path is correct if `[clamav].clamavenabled` is `true`. - **Redis:** - Secure your Redis instance, especially if it's exposed to external networks. Use strong passwords and restrict access as needed. 3. **Resource Allocation:** - Adjust `numworkers` and `uploadqueuesize` in the `[workers]` section based on your server's hardware capabilities and expected traffic. - Monitor system performance to ensure that auto-adjust features are working optimally. 4. **Monitoring and Metrics:** - Regularly monitor Prometheus metrics to keep an eye on server performance, resource usage, and potential issues. - Set up alerts based on critical metrics to proactively handle problems. 5. **Logging:** - Ensure that log files are rotated and managed properly to prevent disk space issues. - Consider enabling JSON logging (`loggingjson = true`) for better integration with log management systems. 6. **Testing:** - Perform thorough testing of file uploads/downloads, especially with large files and under high load. - Test ClamAV scanning with both clean and malicious files to ensure security features are functioning correctly. 7. **Maintenance:** - Regularly update dependencies to benefit from security patches and performance improvements. - Clean up old file versions and ensure deduplication is functioning to optimize storage usage. 8. **Backup:** - Implement a backup strategy for critical data, especially if file versioning is enabled. 9. **Documentation:** - Keep the `README.md` and other documentation up-to-date with any code changes to assist in maintenance and onboarding new contributors. --- ## Troubleshooting - **Cannot Connect to Redis:** - Ensure Redis is running and accessible at the address specified in `redisaddr`. - Verify that the `redispassword` is correct if authentication is enabled. - **ClamAV Scanning Fails:** - Check if ClamAV is installed and the `clamavsocket` path is correct. - Ensure that the ClamAV daemon (`clamd`) is running. - **Insufficient Disk Space:** - Monitor the disk space and adjust `minfreebytes` in the configuration as needed. - Enable file TTL and cleanup to automatically remove old files. - **Metrics Not Available:** - Ensure that `metricsenabled` is set to `true` and the server is running without errors. - Check if the specified `metricsport` is not blocked by a firewall. - **Thumbnail Generation Not Working:** - Verify that the `[thumbnails].enabled` is set to `true` and the `directory` exists with appropriate permissions. - Check the logs for any errors related to image processing. --- ## Contributing Contributions are welcome! Please follow these steps to contribute: 1. **Fork the Repository** 2. **Create a Feature Branch** ```bash git checkout -b feature/YourFeatureName ``` 3. **Commit Your Changes** ```bash git commit -m "Add your message here" ``` 4. **Push to the Branch** ```bash git push origin feature/YourFeatureName ``` 5. **Open a Pull Request** Provide a clear description of your changes and the problem they address. --- ## License This project is licensed under the [MIT License](LICENSE). --- By following this updated `README.md`, you can ensure that users and contributors have a clear understanding of the **HMAC File Server**'s capabilities, configuration options, and best practices for deployment and maintenance.