Compare commits

...

41 Commits

Author SHA1 Message Date
56de6107b6 Merge branch 'main' of git.uuxo.net:uuxo/hmac-file-server 2025-04-26 16:03:08 +02:00
6a098d35d6 Add Drone CI pipeline 2025-04-26 16:02:59 +02:00
49f4338a6e README.MD aktualisiert 2025-04-12 10:02:24 +02:00
1c6477531c fix: 2.9 ipv4|ipv6|auto 2025-04-12 09:49:14 +02:00
23ee7e5d33 removed useless networking monitoring option - release fixed 2.8 2025-04-05 20:36:55 +02:00
2c18b1fccb removed useless networking monitoring option - release fixed 2.8 2025-04-05 20:34:36 +02:00
2769fd1a98 removed useless networking monitoring option - release fixed 2.8 2025-04-05 20:34:00 +02:00
cc1a629aff Chore: Update 2.8 disabled networking policie checks - header missing 2025-04-05 19:24:03 +02:00
a0a117dc11 Chore: Update 2.8 disabled networking policie checks - not needed. upload is perfect 2025-04-05 14:36:12 +02:00
ccf04ede06 removed useless networking monitoring option - release fixed 2.8 2025-04-05 13:02:39 +02:00
7e665152b0 Chore: Update 2.8 cleaned up usesless files 2025-04-05 10:31:43 +02:00
31a157ea96 fix: force 2025-01-31 10:58:18 +01:00
5751f1d0c7 fix: 2.6-stable - fixes 2025-01-26 19:46:02 +01:00
b05c444a0a fix: 2.6-stable - fixes 2025-01-26 18:58:08 +01:00
af5aaa528c fix: 2.6-stable 2025-01-26 09:27:35 +01:00
ef3b618126 fix: 2.6-stable 2025-01-26 09:25:51 +01:00
b14f046beb fix: 2.6-stable 2025-01-26 09:20:56 +01:00
0cda54c97e fix: cleanup 2025-01-15 15:46:43 +01:00
16d2fd4b81 2.5-stable release 2025-01-02 20:40:34 +01:00
cd5cbd74bc 2.5-stable release 2025-01-02 20:37:53 +01:00
ad8d9771ba 2.5-stable release 2025-01-02 19:56:44 +01:00
1ac8ff38e8 2.5-stable release 2025-01-02 17:47:23 +01:00
5487db9e57 2.5-stable release 2025-01-02 17:40:15 +01:00
5e99b937e8 2.5-stable release 2025-01-02 17:33:11 +01:00
1d4a224f01 2.5-stable release 2025-01-02 17:19:43 +01:00
1288e9ac57 2.5-stable release 2025-01-02 17:14:12 +01:00
881237a904 2.5-stable release 2025-01-02 16:58:55 +01:00
74fa394a30 WiKi.md aktualisiert 2025-01-01 16:26:10 +01:00
5ef909fdec 2.5-stable release 2025-01-01 16:25:02 +01:00
921062cfc3 2.5-stable release 2025-01-01 16:11:56 +01:00
18495812fd 2.5-stable release 2025-01-01 15:44:28 +01:00
7c87955ccf 2.5-stable release 2025-01-01 11:15:50 +01:00
0f8931bf90 2.4-stable release 2024-12-31 08:35:47 +01:00
a24b484146 2.4-stable release 2024-12-31 08:34:50 +01:00
aab6b07316 2.4-stable release 2024-12-31 08:34:16 +01:00
e8af2d127b 2.4-stable release 2024-12-31 08:33:04 +01:00
011f54829a 2.4-stable release 2024-12-31 08:07:28 +01:00
7aeb8c77df 2.4-stable release 2024-12-31 08:05:52 +01:00
ae2c258c9d 2.4-stable release 2024-12-31 08:05:13 +01:00
66ccc9404e 2.4-stable release 2024-12-31 08:04:42 +01:00
dccb3e0cd0 2.4-stable release 2024-12-31 08:03:39 +01:00
25 changed files with 3296 additions and 3608 deletions

10
.drone.yml Normal file
View File

@ -0,0 +1,10 @@
kind: pipeline
name: build
steps:
- name: build docker image
image: docker
commands:
- docker build -t my-hmac-file-server .
- docker run -d -p 8080:8080 my-hmac-file-server
- docker ps # Optional: To verify the container is running

110
CHANGELOG.md Normal file
View File

@ -0,0 +1,110 @@
# Changelog
All notable changes to this project will be documented in this file.
## [Unreleased]
- Initial creation of a comprehensive changelog.
---
## [2.8-Stable] - 2026-05-01
### Added
- Version check history for improved tracking.
- Enhanced ClamAV scanning with concurrent workers.
### Changed
- Improved ISO-based storage for specialized use cases.
- Auto-scaling workers for optimized performance.
### Fixed
- Minor issues in worker thread adjustments under high load.
---
## [2.7] - 2026-02-10
### Added
- Concurrency improvements and auto-scaling worker enhancements
- Cleanup and removal of unused parameters in sorting functions
### Changed
- Additional logging for file scanning operations
### Fixed
- Minor stability issues related to ISO container mounting
- Fixed dual stack for upload (IPv4/IPv6)
---
## [2.6-Stable] - 2025-12-01
### Added
- Deduplication support (removes duplicate files).
- ISO Container management.
- Dynamic worker scaling based on CPU & memory.
- PreCaching feature for faster file access.
### Changed
- Worker pool scaling strategies for better performance.
- Enhanced logging with rotating logs using lumberjack.
### Fixed
- Temporary file handling issues causing "Unsupported file type" warnings.
- MIME type checks for file extension mismatches.
---
## [2.5] - 2025-09-15
### Added
- Redis caching integration for file metadata.
- ClamAV scanning for virus detection before finalizing uploads.
### Changed
- Extended the default chunk size for chunked uploads.
- Updated official documentation links.
### Fixed
- Edge case with versioning causing file rename conflicts.
---
## [2.0] - 2025-06-01
### Added
- Chunked file uploads and downloads.
- Resumable upload support with partial file retention.
### Changed
- Moved configuration management to Viper.
- Default Prometheus metrics for tracking memory & CPU usage.
### Fixed
- Race conditions in file locking under heavy concurrency.
---
## [1.0] - 2025-01-01
### Added
- Initial release with HMAC-based authentication.
- Basic file upload/download endpoints.
- Logging and fundamental configuration using .toml files.

48
COMBINED_RELEASE_NOTES.md Normal file
View File

@ -0,0 +1,48 @@
## HMAC File Server Release v2.7
### Whats New
1. **Configurable Filenaming**
- Added support for `filenaming=None` to **skip** the default HMAC-based renaming.
- Allows users to keep the original filename instead of hashing, while preserving all HMAC authentication for security.
2. **Enhanced Logging**
- Improved log level usage (`info`, `warn`, `error`, `debug`) across the **login flow** and **file handling** operations.
- Added more structured fields (e.g., `method`, `remote`, `url`) for easier log analysis.
- Better security defaults to avoid exposing sensitive data in logs.
3. **Prometheus Metrics Adjustments**
- Refined counters, gauges, and histograms to cover **all** critical events (upload/download, dedup, ClamAV scanning).
- Ensured consistent increments for success/failure paths.
- Simplified registration to avoid double-registration issues.
4. **Deduplication Improvements**
- Confirmed that after moving a file to the dedup directory, a **hard link** is consistently created back to the original location.
- Logs now clearly indicate successful dedup steps and any errors.
5. **Worker Pool Enhancements**
- Better dynamic scaling logs (e.g., “Added worker. Total workers: X”).
- Ensures no duplicate or redundant worker creation.
- Additional metrics for worker adjustments and re-adjustments.
### Bug Fixes
- **Resolved “File Not Found” During GET**
- Clarified that when `filenaming=None`, the server does **not** rename files to HMAC paths, preventing mismatches between upload and download URLs.
- Fixed potential race conditions in dedup moving vs. linking.
- **Reduced Log Noise**
- Eliminated repetitive or misleading error messages around networking events.
- Improved clarity in ClamAV scanning logs to better distinguish scan failures vs. actual malware detections.
- Fixed dual stack IPv4 and IPv6 upload for improved reliability.
### Upgrade Notes
1. **Config File**:
- Check your `[server]` section for `filenaming`. If you previously relied on HMAC-based filenames, confirm whether you want to set `filenaming="HMAC"` explicitly.
2. **Metrics**:
- If you track Prometheus data, your dashboards may need to be updated for any renamed metrics or new labels.
3. **Logging**:
- Logging defaults remain at `info` level. Increase to `debug` only for troubleshooting to avoid excessive detail in production logs.
---
**Thank you** to everyone who contributed feedback and testing for this release! As always, please report any issues, and we welcome suggestions to further improve the HMAC File Server. Enjoy the streamlined filenames, more comprehensive logging, and robust Prometheus metrics!

21
LICENSE Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Alexander Renz
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

524
README.MD
View File

@ -1,442 +1,204 @@
# HMAC File Server
A secure file server with HMAC authentication and configurable features.
# HMAC File Server 2.9-Stable
## Overview
The **HMAC File Server** ensures secure file uploads and downloads using HMAC authentication. It incorporates rate limiting, CORS support, retries, file versioning, and Unix socket support for enhanced flexibility. Redis integration provides efficient caching and session management. Prometheus metrics and a graceful shutdown mechanism ensure reliable and efficient file handling.
**HMAC File Server** is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, deduplication, optional ClamAV scanning for file integrity and security, and image thumbnail generation. This server is built with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration.
> **Credits:** The **HMAC File Server** is based on the source code of [Thomas Leister's prosody-filer](https://github.com/ThomasLeister/prosody-filer). Many features and design elements have been inspired or derived from this project.
---
Special thanks to **Thomas Leister** for inspiration drawn from [[prosody-filer](https://github.com/ThomasLeister/prosody-filer)](https://github.com/ThomasLeister/prosody-filer).
## Features
- File deduplication
- Configurable TTL for automatic file cleanup
- Secure HMAC-based authentication
- Chunked uploads and downloads
- Virus scanning via ClamAV
- Prometheus metrics integration
- Customizable worker management
- Support ISO-based storage
- **HMAC Authentication:** Secure file uploads and downloads with HMAC tokens.
- **File Versioning:** Enable versioning for uploaded files with configurable retention.
- **Chunked and Resumable Uploads:** Handle large files efficiently with support for resumable and chunked uploads.
- **Deduplication:** Automatically remove duplicate files based on hashing for storage efficiency.
- **ClamAV Scanning:** Optional virus scanning for uploaded files.
- **Prometheus Metrics:** Monitor system and application-level metrics.
- **Redis Integration:** Use Redis for caching or storing application states.
- **File Expiration:** Automatically delete files after a specified TTL.
- **Graceful Shutdown:** Handles signals and ensures proper cleanup.
- **Auto-Adjust Worker Scaling:** Dynamically optimize worker threads based on system resources.
- **Precaching:** Pre-cache file structures on startup for faster access.
- **Thumbnail Creation:** Generate image thumbnails for uploaded files.
- **ISO Container Management:** Optional mounting and handling of ISO-based filesystems.
---
## Repository
- **Primary Repository**: [GitHub Repository](https://github.com/PlusOne/hmac-file-server)
- **Alternative Repository**: [uuxo.net Git Repository](https://git.uuxo.net/uuxo/hmac-file-server)
## Table of Contents
1. [Installation](#installation)
2. [Configuration](#configuration)
3. [Usage](#usage)
4. [Setup](#setup)
- [Reverse Proxy](#reverse-proxy)
- [Systemd Service](#systemd-service)
5. [Building](#building)
6. [Changelog](#changelog)
7. [License](#license)
---
## Installation
### Prerequisites
- Go **1.20** or higher
- Redis server (optional, for caching)
- ClamAV (optional, for virus scanning)
- **Go 1.20+**
- **Redis** (optional, if Redis integration is enabled)
- **ClamAV** (optional, if file scanning is enabled)
- **genisoimage** (optional, if ISO container management is enabled)
### Steps
1. Clone the repository:
```bash
git clone https://github.com/PlusOne/hmac-file-server.git
cd hmac-file-server
```
### Clone and Build
2. Build the server:
```bash
go build -o hmac-file-server
```
```bash
# Clone from the primary repository
git clone https://github.com/PlusOne/hmac-file-server.git
3. Create necessary directories:
```bash
mkdir -p /path/to/hmac-file-server/data/
mkdir -p /path/to/hmac-file-server/deduplication/
mkdir -p /path/to/hmac-file-server/iso/
```
# OR clone from the alternative repository
git clone https://git.uuxo.net/uuxo/hmac-file-server.git
4. Copy and edit the configuration file:
```bash
cp config.example.toml config.toml
```
cd hmac-file-server
go build -o hmac-file-server main.go
```
#### Build for `arm64`
```bash
cd /path/to/hmac-file-server
GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.3-stable_arm64 main.go
```
#### Build for `amd64`
```bash
cd /path/to/hmac-file-server
GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.3-stable_amd64 main.go
```
5. Start the server:
```bash
./hmac-file-server -config config.toml
```
---
## Configuration
The server is configured via a `config.toml` file. Key settings include:
The server configuration is managed through a `config.toml` file. Below are the supported configuration options:
- **Server Settings**: Port, logging, metrics
- **Security**: HMAC secret, TLS options
- **File Management**: TTL, deduplication, uploads, and downloads
- **ISO**: Generation and mounting settings
- **Workers**: Adjust thread management
### Auto-Adjust Feature
When `AutoAdjustWorkers` is enabled, the number of workers for HMAC operations and ClamAV scans is dynamically determined based on system resources. This ensures efficient resource utilization.
If `AutoAdjustWorkers = true`, the values for `NumWorkers` and `NumScanWorkers` in the configuration file will be ignored, and the server will automatically adjust these values.
### Network Events Monitoring
Setting `NetworkEvents = false` in the server configuration disables the logging and tracking of network-related events within the application. This means that functionalities such as monitoring IP changes or recording network activity will be turned off.
### Precaching
The `precaching` feature allows the server to pre-cache storage paths for faster access. This can improve performance by reducing the time needed to access frequently used storage paths.
### Thumbnail Creation
Set `enabled = true` in the `[thumbnails]` section of `config.toml` to enable image thumbnail generation.
For detailed configuration options, refer to the [Wiki](./wiki.md).
---
## Example `config.toml`
Below is an example configuration file (`config.toml`) you can use as a reference (with sensitive data replaced by placeholder/example data):
```toml
# Server configuration
listenport = "8080" # TCP port for incoming requests
unixsocket = false # Use Unix domain socket instead of TCP
storagepath = "/path/to/hmac-file-server/data/" # Directory to store uploaded files
loglevel = "debug" # Logging level: "debug", "info", "warn", "error"
logfile = "/path/to/hmac-file-server.log" # Path to log file; leave empty to use stdout
metricsenabled = true # Enable Prometheus metrics
metricsport = "9090" # Port for Prometheus metrics
deduplicationenabled = true
minfreebytes = "5GB" # Minimum free disk space required
filettl = "2Y" # Time-to-live for files (2 years)
filettlenabled = false # Enable TTL checks and cleanup
autoadjustworkers = true # Automatically adjust worker threads based on load
networkevents = false # Enable detailed network event logging
pidfilepath = "./hmac-file-server.pid" # Path to PID file
precaching = true # Pre-cache file structures on startup
tempPath = "/tmp/hmac-file-server" # Path for temporary file uploads
# Deduplication settings
[deduplication]
enabled = true
directory = "/path/to/hmac-file-server/deduplication/" # Path to deduplication metadata store
# Thumbnails settings
[thumbnails]
enabled = true
directory = "/path/to/hmac-file-server/thumbnails/" # Directory for storing thumbnails
size = "200x200" # Thumbnail dimensions
thumbnailintervalscan = "1h" # Interval for scheduled thumbnail generation
concurrency = 5 # Number of concurrent thumbnail generation tasks
# ISO settings
[iso]
enabled = false
size = "1TB" # Maximum ISO size
mountpoint = "/path/to/hmac-file-server/iso/" # ISO mount point
charset = "utf-8" # Filesystem character set encoding
# Timeout settings
[timeouts]
readtimeout = "3600s" # Maximum time to read a request (1 hour)
writetimeout = "3600s" # Maximum time to write a response (1 hour)
idletimeout = "3600s" # Maximum keep-alive time for idle connections (1 hour)
# Security settings
[security]
secret = "your-secure-secret-key" # HMAC shared secret key (change to a secure value)
# Versioning settings
[versioning]
enableversioning = false
maxversions = 1 # Number of file versions to retain
# Upload settings
[uploads]
resumableuploadsenabled = false
chunkeduploadsenabled = true
chunksize = "32MB" # Chunk size for uploads
allowedextensions = ["*"] # All file types are allowed for uploads
# Downloads settings
[downloads]
resumabledownloadsenabled = false
chunkeddownloadsenabled = true
chunksize = "32MB"
# ClamAV settings
[clamav]
clamavenabled = true
clamavsocket = "/path/to/clamav/clamd.ctl" # Path to ClamAV socket
numscanworkers = 4 # Number of concurrent scan workers
scanfileextensions = [
".exe", ".dll", ".bin", ".com", ".bat",
".sh", ".php", ".js"
]
# Redis settings
[redis]
redisenabled = true
redisdbindex = 0
redisaddr = "localhost:6379" # Redis server address
redispassword = "" # Redis password if required
redishealthcheckinterval = "120s" # Interval for Redis health checks
# Worker settings
[workers]
numworkers = 4
uploadqueuesize = 5000
# File settings
[file]
filerevision = 1 # Internal revision number for file handling logic
# Logging settings
[logging]
level = "info"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
```
---
## Configuration Verification
The application ensures that all configuration parameters defined in `config.toml` are correctly implemented and operational. During startup, the server performs the following verification steps:
1. **Configuration Parsing:**
- Parses the `config.toml` file into the application's `Config` struct.
2. **Validation:**
- Checks that all required configuration parameters are set.
- Validates the correctness of configuration values (e.g., proper duration formats, non-empty directories).
3. **Service Initialization:**
- Initializes services like Redis and ClamAV based on the configuration.
- Ensures that conditional services (e.g., Thumbnails, ISO) are only started if enabled in the configuration.
If any configuration parameter is missing or invalid, the server will log an error and terminate to prevent running with incorrect settings.
---
## Running the Server
### Basic Usage
Run the server with a configuration file:
## Usage
Start the server and access it on the configured port. Use curl or a client library to interact with the API.
### Example
Upload a file:
```bash
./hmac-file-server -config ./config.toml
curl -X POST -F 'file=@example.jpg' http://localhost:8080/upload
```
Ensure that the `config.toml` is correctly set up before running the server.
---
### Metrics Server
## Setup
If `metricsenabled` is set to `true`, the Prometheus metrics server will be available on the port specified in `metricsport` (e.g., `http://localhost:9090/metrics`).
### Reverse Proxy
Set up a reverse proxy using Apache2 or Nginx to handle requests.
Additional metrics for deduplication and ISO container operations are now available.
#### Apache2 Example
```apache
<VirtualHost *:80>
ServerName your-domain.com
---
ProxyPreserveHost On
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
</VirtualHost>
```
## Health Checks
#### Nginx Example
```nginx
server {
listen 80;
server_name your-domain.com;
The server includes built-in health checks to verify that all configured services are operational. Logs will provide detailed information about the status of each service and any issues encountered during startup.
location / {
proxy_pass http://localhost:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
```
---
### Systemd Service
Create a systemd service file for the HMAC File Server:
```ini
[Unit]
Description=HMAC File Server
After=network.target
## Testing
To run the server locally for development:
[Service]
ExecStart=/path/to/hmac-file-server -config /path/to/config.toml
WorkingDirectory=/path/to/hmac-file-server
Restart=always
User=www-data
Group=www-data
[Install]
WantedBy=multi-user.target
```
Enable and start the service:
```bash
go run main.go -config ./config.toml
sudo systemctl daemon-reload
sudo systemctl enable hmac-file-server
sudo systemctl start hmac-file-server
```
Use tools like **cURL** or **Postman** to test file uploads and downloads.
---
### Example File Upload with HMAC Token
## Building
To build for different architectures:
```bash
curl -X PUT -H "Authorization: Bearer <HMAC-TOKEN>" -F "file=@example.txt" http://localhost:8080/uploads/example.txt
```
- **Linux (amd64)**:
```bash
GOOS=linux GOARCH=amd64 go build -o hmac-file-server main.go
```
Replace `<HMAC-TOKEN>` with a valid HMAC signature generated using the configured `secret`.
- **Linux (arm64)**:
```bash
GOOS=linux GOARCH=arm64 go build -o hmac-file-server main.go
```
---
## Monitoring
## Changelog
Prometheus metrics include:
### Added
- **Deduplication Support:** Automatically remove duplicate files based on SHA256 hashing to save storage space.
- **ISO Container Management:** Create and mount ISO containers for specialized storage needs, enhancing flexibility in file management.
- **Prometheus Metrics Enhancements:** Added detailed metrics for deduplication and ISO container operations to improve monitoring and observability.
- **Redis Integration Improvements:** Enhanced caching mechanisms using Redis for faster access to file metadata and application states.
- **Precaching Feature:** Implemented precaching of file structures on startup to reduce access times for frequently used files.
- **Configuration Options:** Updated `config.toml` to include new settings for deduplication, ISO management, and worker scaling.
- **File Operations:**
- File upload/download durations
- Uploaded/downloaded file sizes
- Total number of uploads/downloads
- Total number of upload/download errors
### Changed
- **Worker Pool Scaling:** Implemented dynamic adjustment of worker threads based on system resources to optimize performance.
- **Logging Enhancements:** Improved logging for file operations, including detailed information on file extensions and MIME types during uploads.
- **Temporary Path Configuration:** Replaced hardcoded temporary upload directories with a configurable `TempPath` parameter in `config.toml` for greater flexibility.
- **System Metrics:**
- Memory usage
- CPU usage
- Number of goroutines
- Active connections
### Fixed
- **Temporary File Handling:** Resolved issues where temporary `.tmp` files caused "Unsupported file type" warnings by enhancing MIME type detection logic.
- **MIME Type Detection:** Improved MIME type detection to ensure better compatibility and accuracy during file uploads.
- **HTTP Requests:**
- Total HTTP requests broken down by method and path
Access the metrics at `http://localhost:9090/metrics` (assuming default `metricsport`).
### Deprecated
- **Thumbnail Support (Previous Implementation):** Dropped the previous thumbnail support mechanism. This feature will not return in future releases.
---
## Additional Features
- **Deduplication:** Automatically remove duplicate files based on SHA256 hashing to save storage space.
- **Versioning:** Store multiple versions of files and retain a configurable number of versions.
- **ClamAV Integration:** Scan uploaded files for viruses using ClamAV to ensure file integrity and security.
- **Redis Caching:** Utilize Redis for caching file metadata to improve access times and performance.
- **Auto-Adjust Worker Scaling:** Optimize the number of workers dynamically based on system resources to maintain optimal performance.
- **Precaching:** Pre-cache file structures on startup to reduce access times for frequently accessed files.
- **Thumbnail Creation:** Generate image thumbnails for uploaded files to provide quick previews.
- **ISO Container Management:** Optional mounting and handling of ISO-based filesystems for specialized storage needs.
---
### Overview of Other Projects (xep0363)
| Feature/Project | HMAC FS | mod_http_upload_ext | xmpp-http-upload (horazont) | Prosody Filer | ngx_http_upload | xmpp-http-upload (nyovaya) |
|-----------------------------|---------|----------------------|-----------------------------|---------------|------------------|----------------------------|
| **Language** | Go | PHP | Python | Go | C (Nginx) | Python |
| **Environment** | Standalone | Prosody module | Standalone | Standalone | Nginx | Standalone |
| **XMPP** | No | Yes | Yes | Yes | No | Yes |
| **External Storage** | Yes | No | Possible via plugins | No | No | Yes |
| **Authentication / Security** | HMAC | Token-based | Token-based | None | Basic / None | Token-based |
---
## Build & Run
1. **Clone the Repository:**
```bash
git clone https://github.com/PlusOne/hmac-file-server.git
cd hmac-file-server
```
2. **Build the Server:**
```bash
go build -o hmac-file-server main.go
```
3. **Run the Server:**
```bash
./hmac-file-server -config ./config.toml
```
---
## Best Practices and Recommendations
1. **Configuration Consistency:**
- Ensure all configuration keys in `config.toml` match those expected in the code. For example, in the `[thumbnails]` section, use `thumbnailintervalscan` instead of `scanInterval` to align with the code.
2. **Security:**
- **HMAC Secret:**
- **Change Immediately:** The `secret` under `[security]` should be a strong, unique string. Replace `"your-secure-secret-key"` with a securely generated key.
- **ClamAV:**
- Ensure ClamAV is installed and the `clamavsocket` path is correct if `[clamav].clamavenabled` is `true`.
- **Redis:**
- Secure your Redis instance, especially if it's exposed to external networks. Use strong passwords and restrict access as needed.
3. **Resource Allocation:**
- Adjust `numworkers` and `uploadqueuesize` in the `[workers]` section based on your server's hardware capabilities and expected traffic.
- Monitor system performance to ensure that auto-adjust features are working optimally.
4. **Monitoring and Metrics:**
- Regularly monitor Prometheus metrics to keep an eye on server performance, resource usage, and potential issues.
- Set up alerts based on critical metrics to proactively handle problems.
5. **Logging:**
- Ensure that log files are rotated and managed properly to prevent disk space issues.
- Consider enabling JSON logging (`loggingjson = true`) for better integration with log management systems.
6. **Testing:**
- Perform thorough testing of file uploads/downloads, especially with large files and under high load.
- Test ClamAV scanning with both clean and malicious files to ensure security features are functioning correctly.
7. **Maintenance:**
- Regularly update dependencies to benefit from security patches and performance improvements.
- Clean up old file versions and ensure deduplication is functioning to optimize storage usage.
8. **Backup:**
- Implement a backup strategy for critical data, especially if file versioning is enabled.
9. **Documentation:**
- Keep the `README.md` and other documentation up-to-date with any code changes to assist in maintenance and onboarding new contributors.
---
## Troubleshooting
- **Cannot Connect to Redis:**
- Ensure Redis is running and accessible at the address specified in `redisaddr`.
- Verify that the `redispassword` is correct if authentication is enabled.
- **ClamAV Scanning Fails:**
- Check if ClamAV is installed and the `clamavsocket` path is correct.
- Ensure that the ClamAV daemon (`clamd`) is running.
- **Insufficient Disk Space:**
- Monitor the disk space and adjust `minfreebytes` in the configuration as needed.
- Enable file TTL and cleanup to automatically remove old files.
- **Metrics Not Available:**
- Ensure that `metricsenabled` is set to `true` and the server is running without errors.
- Check if the specified `metricsport` is not blocked by a firewall.
- **Thumbnail Generation Not Working:**
- Verify that the `[thumbnails].enabled` is set to `true` and the `directory` exists with appropriate permissions.
- Check the logs for any errors related to image processing.
---
## Contributing
Contributions are welcome! Please follow these steps to contribute:
1. **Fork the Repository**
2. **Create a Feature Branch**
```bash
git checkout -b feature/YourFeatureName
```
3. **Commit Your Changes**
```bash
git commit -m "Add your message here"
```
4. **Push to the Branch**
```bash
git push origin feature/YourFeatureName
```
5. **Open a Pull Request**
Provide a clear description of your changes and the problem they address.
**Important Update:**
- The minimum Go version required is now **1.20**. Please ensure your environment meets this requirement for successful compilation.
---
## License
This project is licensed under the [MIT License](LICENSE).
MIT License
---
Copyright (c) 2025 Alexander Renz
By following this updated `README.md`, you can ensure that users and contributors have a clear understanding of the **HMAC File Server**'s capabilities, configuration options, and best practices for deployment and maintenance.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@ -1,318 +1,53 @@
# HMAC File Server
# Release Notes for HMAC File Server 2.7-Stable
**HMAC File Server** is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, and optional ClamAV scanning for file integrity and security. This server is built with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration.
## Summary
Version 2.6-Stable focuses on improving the overall stability and performance of the HMAC File Server. Significant changes have been made to prioritize reliability and scalability for production environments.
> **Credits:** The **HMAC File Server** is based on the source code of [Thomas Leister's prosody-filer](https://github.com/ThomasLeister/prosody-filer). Many features and design elements have been inspired or derived from this project.
## Key Changes
---
### Breaking Changes
- **Thumbnail Generation Dropped**: Support for automatic thumbnail generation has been removed in this release. This decision was made to enhance system stability and reduce resource consumption. Users requiring thumbnails are encouraged to use external tools.
## Features
### New Features
- **ISO-Based Storage Support**: Introduced support for ISO-based storage to accommodate specialized use cases.
- **Enhanced ClamAV Integration**: Improved ClamAV scanning with concurrent workers, providing better performance for large-scale deployments.
- **Timeout Configuration**: Added granular timeout settings for read, write, and idle connections, improving connection management.
- **FileNaming Configuration**: Added support for a "None" option in the `FileNaming` configuration. When set to "None", the filename remains unchanged.
- **Example Configuration Generation**: If no configuration file is found, the server will output an example configuration for the user to copy and paste.
- **Prometheus Metrics**: Enhanced Prometheus metrics for better monitoring and performance tracking. New metrics include upload and download durations, error counts, memory and CPU usage, and more.
- **HMAC Authentication:** Secure file uploads and downloads with HMAC tokens.
- **File Versioning:** Enable versioning for uploaded files with configurable retention.
- **Chunked and Resumable Uploads:** Handle large files efficiently with support for resumable and chunked uploads.
- **ClamAV Scanning:** Optional virus scanning for uploaded files.
- **Prometheus Metrics:** Monitor system and application-level metrics.
- **Redis Integration:** Use Redis for caching or storing application states.
- **File Expiration:** Automatically delete files after a specified TTL.
- **Graceful Shutdown:** Handles signals and ensures proper cleanup.
- **Deduplication:** Remove duplicate files based on hashing for storage efficiency.
- **Auto-Adjust Worker Scaling:** Dynamically optimize HMAC and ClamAV workers based on system resources when enabled.
- **Thumbnail Support:** Generate smaller versions of uploaded images for efficient storage and quick access.
### Improvements
- **Worker Management**: Auto-scaling worker threads based on system load for optimal performance.
- **Logging Enhancements**: Improved log verbosity control, making debugging and monitoring easier.
---
### Bug Fixes
- Resolved minor issues affecting deduplication and file upload performance.
- Fixed a rare crash scenario during high-concurrency file uploads.
## Repository
## Migration Notes
1. **Thumbnail Settings**: Remove `[thumbnails]` configuration blocks from your `config.toml` file to avoid errors.
2. **Updated Configuration**: Review new timeout settings in `[timeouts]` and adjust as needed.
3. **ISO Integration**: Configure the new `[iso]` block for environments utilizing ISO-based storage.
4. **FileNaming Configuration**: Update the `FileNaming` setting in `[server]` to use the new "None" option if you want filenames to remain unchanged.
- **Primary Repository**: [GitHub Repository](https://github.com/PlusOne/hmac-file-server)
- **Alternative Repository**: [uuxo.net Git Repository](https://git.uuxo.net/uuxo/hmac-file-server)
---
## Installation
### Prerequisites
- Go 1.20+
- Redis (optional, if Redis integration is enabled)
- ClamAV (optional, if file scanning is enabled)
### Clone and Build
```bash
# Clone from the primary repository
git clone https://github.com/PlusOne/hmac-file-server.git
# OR clone from the alternative repository
git clone https://git.uuxo.net/uuxo/hmac-file-server.git
cd hmac-file-server
go build -o hmac-file-server main.go
```
### Building for Different Architectures
To build the HMAC File Server for different architectures, use the following commands:
#### Build for `arm64`
```bash
cd /path/to/hmac-file-server
GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.2-stable_arm64 main.go
```
#### Build for `amd64`
```bash
cd /path/to/hmac-file-server
GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.2-stable_amd64 main.go
```
Replace `/path/to/hmac-file-server` with the actual path to your project directory. These commands will generate the binaries for `arm64` and `amd64` architectures and place them in the `~/Temp` directory.
---
## Configuration
The server configuration is managed through a `config.toml` file. Below are the supported configuration options:
### Auto-Adjust Feature
When `AutoAdjustWorkers` is enabled, the number of workers for HMAC operations and ClamAV scans is dynamically determined based on system resources. This ensures efficient resource utilization.
If `AutoAdjustWorkers = true`, the values for `NumWorkers` and `NumScanWorkers` in the configuration file will be ignored, and the server will automatically adjust these values.
### Network Events Monitoring
Setting `NetworkEvents = false` in the server configuration disables the logging and tracking of network-related events within the application. This means that functionalities such as monitoring IP changes or recording network activity will be turned off.
### Precaching
The `precaching` feature allows the server to pre-cache storage paths for faster access. This can improve performance by reducing the time needed to access frequently used storage paths.
### Thumbnail Support
- New configuration options in `[thumbnails]` to enable or disable generating image thumbnails, set the thumbnail size, and configure concurrency for thumbnail generation.
---
## New Features
### Deduplication Support
- **Description:** Added support for file deduplication to save storage space by storing a single copy of identical files.
- **Configuration:**
```toml
[deduplication]
enabled = true
directory = "/mnt/hmac-storage/deduplication/"
```
### Thumbnail Support
- **Description:** Added support for thumbnail creation to generate smaller versions of uploaded images.
- **Configuration:**
```toml
[thumbnails]
enabled = true
directory = "/mnt/hmac-storage/thumbnails/"
size = "200x200"
concurrency = 5
thumbnailintervalscan = "24h"
```
---
## Example `config.toml`
```toml
[server]
ListenPort = "8080"
UnixSocket = false
StoragePath = "./uploads"
LogLevel = "info"
LogFile = ""
MetricsEnabled = true
MetricsPort = "9090"
FileTTL = "1y"
FileTTLEnabled = true # Enable or disable file TTL
MinFreeBytes = "100MB"
AutoAdjustWorkers = true # Enable auto-adjustment for worker scaling
NetworkEvents = false # Disable logging and tracking of network-related events
PIDFilePath = "./hmac_file_server.pid" # Path to PID file
Precaching = true # Enable pre-caching of storage paths
TempPath = "/tmp/uploads" # Configurable temporary upload directory
# FileNaming options: "HMAC", "None"
FileNaming = "HMAC"
[deduplication]
enabled = true
directory = "/mnt/nfs_vol01/hmac-storage/deduplication/"
## Recommendations
- **Security**: Ensure that the HMAC secret key in `config.toml` is updated to a strong, unique value.
- **Backups**: Regularly back up your `config.toml` and important data directories.
- **Monitoring**: Leverage Prometheus metrics for real-time monitoring of server performance.
[iso]
Enabled = false
Size = "2TB"
MountPoint = "/mnt/iso"
Charset = "utf-8"
[thumbnails]
enabled = true
directory = "/mnt/nfs_vol01/hmac-file-server/thumbnails/"
size = "200x200"
concurrency = 5
thumbnailintervalscan = "24h"
[iso]
enabled = false
size = "1TB"
mountpoint = "/mnt/nfs_vol01/hmac-file-server/iso/"
charset = "utf-8"
[timeouts]
ReadTimeout = "480s"
WriteTimeout = "480s"
IdleTimeout = "65s" # nginx/apache2 keep-alive 60s
[security]
Secret = "changeme"
[versioning]
EnableVersioning = false
MaxVersions = 1
[uploads]
ResumableUploadsEnabled = true
ChunkedUploadsEnabled = true
ChunkSize = "64MB"
AllowedExtensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm", ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"]
[downloads]
ResumableDownloadsEnabled = true
ChunkedDownloadsEnabled = true
ChunkSize = "64MB"
[clamav]
ClamAVEnabled = false
ClamAVSocket = "/var/run/clamav/clamd.ctl"
NumScanWorkers = 2
ScanFileExtensions = [".exe", ".dll", ".bin", ".com", ".bat", ".sh", ".php", ".js"]
[redis]
RedisEnabled = false
RedisAddr = "localhost:6379"
RedisPassword = ""
RedisDBIndex = 0
RedisHealthCheckInterval = "120s"
[workers]
NumWorkers = 4
UploadQueueSize = 5000
[file]
FileRevision = 1 # Revision number for file handling
[logging]
level = "info"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
```
For a detailed guide on setting up and configuring the HMAC File Server, refer to the [README.md](./README.md).
---
## Running the Server
Thank you for using HMAC File Server! If you encounter any issues, feel free to report them on our GitHub repository.
### Basic Usage
## Version 2.7
Run the server with a configuration file:
```bash
./hmac-file-server -config ./config.toml
```
---
### Metrics Server
If `MetricsEnabled` is set to `true`, the Prometheus metrics server will be available on the port specified in `MetricsPort` (default: `9090`).
---
## Testing
To run the server locally for development:
```bash
go run main.go -config ./config.toml
```
Use tools like **cURL** or **Postman** to test file uploads and downloads.
### Example File Upload with HMAC Token
```bash
curl -X PUT -H "Authorization: Bearer <HMAC-TOKEN>" -F "file=@example.txt" http://localhost:8080/uploads/example.txt
```
Replace `<HMAC-TOKEN>` with a valid HMAC signature
````
## [2.3.0] - 2024-12-28
### Changed
- **Server:** Replaced the hardcoded temporary upload directory `/tmp/uploads` with a configurable `TempPath` parameter in `config.toml`. Ensure to set `tempPath` in your configuration file accordingly.
#### Version 2.2.1 - 2024-12-27
- **Enhancements:**
- Added detailed logging for file extensions and MIME types during file uploads to assist in diagnosing unsupported file type issues.
- **Configuration:**
- Updated `config.toml` to ensure necessary file extensions are allowed for uploads.
#### Version 2.2.2 - 2024-12-27
- **Bug Fixes:**
- Resolved issue where temporary `.tmp` files caused "Unsupported file type" warnings by adjusting MIME type detection to use the final file extension.
- **Enhancements:**
- Improved logging for file extension and MIME type during uploads.
### Summary of Changes
1. **Added Log Message Limitation:**
- Implemented `logMessages = logMessages[1:]` to remove the oldest log message once the `logMessages` slice exceeds 1000 entries, preventing unbounded memory growth.
---
### Additional Files
2024-12-30 endpoint /thumbnails
No changes are required for the other files (`RELEASE-NOTES.MD`, `README.MD`, `config.toml`).
## [2.3.1] - 2025-01-15
- Added metrics for deduplication and ISO container operations to improve observability.
- Additional detail: Thumbnails now support a "concurrency" configuration parameter for parallel image processing.
## [2.4.0] - 2025-02-20
### Added
- **Pre-Caching Support:** Introduced pre-caching of storage paths to improve access speeds.
- **ISO Container Management:** Added functionality to create and mount ISO containers for specialized storage needs.
- **Thumbnail Concurrency Parameter:** Users can now set the level of concurrency for thumbnail generation to optimize performance.
### Changed
- **Configuration Options:** Updated `config.toml` to include new settings for pre-caching and ISO management.
- **Documentation:** Enhanced `README.MD` with detailed instructions on new features and best practices.
### Fixed
- **Bug Fixes:** Resolved minor issues related to file versioning and deduplication processes.
### Summary of Changes
1. **Pre-Caching:** Improved performance by allowing the server to pre-cache frequently accessed storage paths.
2. **ISO Management:** Enabled the creation and mounting of ISO containers for better filesystem management.
3. **Thumbnail Generation:** Added a concurrency setting to allow parallel processing of image thumbnails, speeding up the generation process.
## [2.4.1] - 2025-03-10
### Changed
- **Configuration:** Updated `globalextensions` in `config.toml` to `["*"]`, allowing all file types globally for uploads. This change simplifies the configuration by removing the need to specify individual file extensions.
- Refinements in worker scaling logic
- Removed obsolete parameters for sorting
- Further improvements to ISO-based storage handling
- Fixed dual stack for upload (IPv4/IPv6)

11
SHORT_RELEASE_NOTE.md Normal file
View File

@ -0,0 +1,11 @@
# Short Release Note
Key Highlights from 2.8-Stable:
- Version check history added for improved tracking.
- Improved ISO-based storage for specialized use cases.
- Enhanced ClamAV scanning with concurrent workers.
- Auto-scaling workers for optimized performance.
Go 1.24.0 is required.
For more details, see the README and CHANGELOG.

47
changelog Normal file
View File

@ -0,0 +1,47 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [Unreleased]
## [2.4.1] - 2025-03-10
### Changed
- **Configuration:** Updated `globalextensions` in `config.toml` to `["*"]`, allowing all file types globally for uploads. This change simplifies the configuration by removing the need to specify individual file extensions.
## [2.4.0] - 2025-02-20
### Added
- **Pre-Caching Support:** Introduced pre-caching of storage paths to improve access speeds.
- **ISO Container Management:** Added functionality to create and mount ISO containers for specialized storage needs.
- **Thumbnail Concurrency Parameter:** Users can now set the level of concurrency for thumbnail generation to optimize performance.
### Changed
- **Configuration Options:** Updated `config.toml` to include new settings for pre-caching and ISO management.
- **Documentation:** Enhanced `README.MD` with detailed instructions on new features and best practices.
### Fixed
- **Bug Fixes:** Resolved minor issues related to file versioning and deduplication processes.
## [2.3.1] - 2025-01-15
### Changed
- **Configuration:** Updated `globalextensions` in `config.toml` to `["*"]`, allowing all file types globally for uploads. This change simplifies the configuration by removing the need to specify individual file extensions.
## [2.3.0] - 2024-12-28
### Changed
- **Server:** Replaced the hardcoded temporary upload directory `/tmp/uploads` with a configurable `TempPath` parameter in `config.toml`. Ensure to set `tempPath` in your configuration file accordingly.
## [2.2.2] - 2024-12-27
### Bug Fixes
- Resolved issue where temporary `.tmp` files caused "Unsupported file type" warnings by adjusting MIME type detection to use the final file extension.
### Enhancements
- Improved logging for file extension and MIME type during uploads.
## [2.2.1] - 2024-12-27
### Enhancements
- Added detailed logging for file extensions and MIME types during file uploads to assist in diagnosing unsupported file type issues.
### Configuration
- Updated `config.toml` to ensure necessary file extensions are allowed for uploads.

View File

@ -1,7 +1,6 @@
package main
import (
"bufio"
"fmt"
"log"
"net/http"
@ -10,6 +9,10 @@ import (
"strconv"
"strings"
"time"
"context"
"io"
"sync"
"bufio"
"github.com/gdamore/tcell/v2"
"github.com/pelletier/go-toml"
@ -20,9 +23,13 @@ import (
"github.com/shirou/gopsutil/v3/process"
)
var prometheusURL string
var configFilePath string // Pfad der gefundenen Konfiguration
var logFilePath string // Pfad der Logdatei aus der Konfiguration
var (
prometheusURL string
configFilePath string // Pfad der gefundenen Konfiguration
logFilePath string // Pfad der Logdatei aus der Konfiguration
metricsEnabled bool // Neue Variable für die Aktivierung von Metriken
bindIP string // Neue Variable für die gebundene IP-Adresse
)
func init() {
configPaths := []string{
@ -45,7 +52,7 @@ func init() {
}
if err != nil {
log.Fatalf("Error loading config file: %v", err)
log.Fatalf("Error loading config file: %v\nPlease create a config.toml in one of the following locations:\n%v", err, configPaths)
}
// Metricsport auslesen
@ -69,7 +76,35 @@ func init() {
log.Fatalf("Error: 'server.metricsport' is not of type int64 or string, got %T", v)
}
prometheusURL = fmt.Sprintf("http://localhost:%d/metrics", port)
// Lesen von 'metricsenabled' aus der Konfiguration
metricsEnabledValue := config.Get("server.metricsenabled")
if metricsEnabledValue == nil {
log.Println("Warning: 'server.metricsenabled' ist in der Konfiguration nicht gesetzt. Standardmäßig deaktiviert.")
metricsEnabled = false
} else {
var ok bool
metricsEnabled, ok = metricsEnabledValue.(bool)
if !ok {
log.Fatalf("Konfigurationsfehler: 'server.metricsenabled' sollte ein boolescher Wert sein, aber %T wurde gefunden.", metricsEnabledValue)
}
}
// Lesen von 'bind_ip' aus der Konfiguration
bindIPValue := config.Get("server.bind_ip")
if bindIPValue == nil {
log.Println("Warning: 'server.bind_ip' ist in der Konfiguration nicht gesetzt. Standardmäßig auf 'localhost' gesetzt.")
bindIP = "localhost"
} else {
var ok bool
bindIP, ok = bindIPValue.(string)
if !ok {
log.Fatalf("Konfigurationsfehler: 'server.bind_ip' sollte ein String sein, aber %T wurde gefunden.", bindIPValue)
}
}
// Konstruktion der prometheusURL basierend auf 'bind_ip' und 'metricsport'
prometheusURL = fmt.Sprintf("http://%s:%d/metrics", bindIP, port)
log.Printf("Metrics URL gesetzt auf: %s", prometheusURL)
// Log-Datei auslesen über server.logfile
logFileValue := config.Get("server.logfile")
@ -93,11 +128,17 @@ const (
// ProcessInfo holds information about a process
type ProcessInfo struct {
PID int32
Name string
CPUPercent float64
MemPercent float32
CommandLine string
PID int32
Name string
CPUPercent float64
MemPercent float32
CommandLine string
Uptime string // Neues Feld für die Uptime
Status string // Neues Feld für den Status
ErrorCount int // Neues Feld für die Anzahl der Fehler
TotalRequests int64 // Neues Feld für die Gesamtanzahl der Anfragen
ActiveConnections int // Neues Feld für aktive Verbindungen
AverageResponseTime float64 // Neues Feld für die durchschnittliche Antwortzeit in Millisekunden
}
// Function to fetch and parse Prometheus metrics
@ -121,7 +162,9 @@ func fetchMetrics() (map[string]float64, error) {
name == "memory_usage_bytes" ||
name == "cpu_usage_percent" ||
name == "active_connections_total" ||
name == "goroutines_count" {
name == "goroutines_count" ||
name == "total_requests" ||
name == "average_response_time_ms" {
for _, m := range mf.GetMetric() {
var value float64
@ -179,7 +222,7 @@ func fetchSystemData() (float64, float64, int, error) {
return v.UsedPercent, cpuUsage, cores, nil
}
// Function to fetch process list
// Funktion zum Abrufen der Prozessliste mit paralleler Verarbeitung
func fetchProcessList() ([]ProcessInfo, error) {
processes, err := process.Processes()
if err != nil {
@ -187,37 +230,55 @@ func fetchProcessList() ([]ProcessInfo, error) {
}
var processList []ProcessInfo
var mu sync.Mutex
var wg sync.WaitGroup
// Begrenzung der gleichzeitigen Goroutinen auf 10
sem := make(chan struct{}, 10)
for _, p := range processes {
cpuPercent, err := p.CPUPercent()
if err != nil {
continue
}
wg.Add(1)
sem <- struct{}{} // Eintritt in semaphor
memPercent, err := p.MemoryPercent()
if err != nil {
continue
}
go func(p *process.Process) {
defer wg.Done()
defer func() { <-sem }() // Austritt aus semaphor
name, err := p.Name()
if err != nil {
continue
}
cpuPercent, err := p.CPUPercent()
if err != nil {
return
}
cmdline, err := p.Cmdline()
if err != nil {
cmdline = ""
}
memPercent, err := p.MemoryPercent()
if err != nil {
return
}
processList = append(processList, ProcessInfo{
PID: p.Pid,
Name: name,
CPUPercent: cpuPercent,
MemPercent: memPercent,
CommandLine: cmdline,
})
name, err := p.Name()
if err != nil {
return
}
cmdline, err := p.Cmdline()
if err != nil {
cmdline = ""
}
info := ProcessInfo{
PID: p.Pid,
Name: name,
CPUPercent: cpuPercent,
MemPercent: memPercent,
CommandLine: cmdline,
}
mu.Lock()
processList = append(processList, info)
mu.Unlock()
}(p)
}
wg.Wait()
return processList, nil
}
@ -250,12 +311,57 @@ func fetchHmacFileServerInfo() (*ProcessInfo, error) {
cmdline = ""
}
createTime, err := p.CreateTime()
if err != nil {
return nil, fmt.Errorf("failed to get process start time: %w", err)
}
uptime := time.Since(time.Unix(0, createTime*int64(time.Millisecond)))
status := "Running" // Standardstatus
// Überprüfung, ob der Prozess aktiv ist
isRunning, err := p.IsRunning()
if err != nil || !isRunning {
status = "Stopped"
}
errorCount, err := countHmacErrors()
if err != nil {
errorCount = 0
}
metrics, err := fetchMetrics()
if err != nil {
return nil, fmt.Errorf("failed to fetch metrics: %w", err)
}
totalRequests, ok := metrics["total_requests"]
if !ok {
totalRequests = 0
}
activeConnections, ok := metrics["active_connections_total"]
if !ok {
activeConnections = 0
}
averageResponseTime, ok := metrics["average_response_time_ms"]
if !ok {
averageResponseTime = 0.0
}
return &ProcessInfo{
PID: p.Pid,
Name: name,
CPUPercent: cpuPercent,
MemPercent: memPercent,
CommandLine: cmdline,
PID: p.Pid,
Name: name,
CPUPercent: cpuPercent,
MemPercent: memPercent,
CommandLine: cmdline,
Uptime: uptime.String(),
Status: status,
ErrorCount: errorCount,
TotalRequests: int64(totalRequests),
ActiveConnections: int(activeConnections),
AverageResponseTime: averageResponseTime,
}, nil
}
}
@ -263,63 +369,203 @@ func fetchHmacFileServerInfo() (*ProcessInfo, error) {
return nil, fmt.Errorf("hmac-file-server process not found")
}
// Function to update the UI with the latest data
func updateUI(app *tview.Application, pages *tview.Pages, sysPage, hmacPage tview.Primitive) {
// Neue Funktion zur Zählung der Fehler in den Logs
func countHmacErrors() (int, error) {
logFilePath := "/var/log/hmac-file-server.log" // Pfad zur Logdatei
file, err := os.Open(logFilePath)
if err != nil {
return 0, err
}
defer file.Close()
scanner := bufio.NewScanner(file)
errorCount := 0
for scanner.Scan() {
line := scanner.Text()
if strings.Contains(line, "level=error") {
errorCount++
}
}
if err := scanner.Err(); err != nil {
return 0, err
}
return errorCount, nil
}
// Funktion zur Aktualisierung der UI mit paralleler Datenbeschaffung
func updateUI(ctx context.Context, app *tview.Application, pages *tview.Pages, sysPage, hmacPage tview.Primitive) {
ticker := time.NewTicker(2 * time.Second)
defer ticker.Stop()
for range ticker.C {
// Fetch data for both views
memUsage, cpuUsage, cores, err := fetchSystemData()
if err != nil {
log.Printf("Error fetching system data: %v\n", err)
continue
}
metrics, err := fetchMetrics()
if err != nil {
log.Printf("Error fetching metrics: %v\n", err)
continue
}
processes, err := fetchProcessList()
if err != nil {
log.Printf("Error fetching process list: %v\n", err)
continue
}
hmacInfo, err := fetchHmacFileServerInfo()
if err != nil {
log.Printf("Error fetching hmac-file-server info: %v\n", err)
}
// Update the UI
app.QueueUpdateDraw(func() {
// Update system page
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
sysFlex := sysPage.(*tview.Flex)
// Update system data table
sysTable := sysFlex.GetItem(0).(*tview.Table)
updateSystemTable(sysTable, memUsage, cpuUsage, cores)
// Update metrics table
metricsTable := sysFlex.GetItem(1).(*tview.Table)
updateMetricsTable(metricsTable, metrics)
// Update process table
processTable := sysFlex.GetItem(2).(*tview.Table)
updateProcessTable(processTable, processes)
}
// Update hmac-file-server page
if currentPage, _ := pages.GetFrontPage(); currentPage == "hmac" && hmacInfo != nil {
hmacFlex := hmacPage.(*tview.Flex)
hmacTable := hmacFlex.GetItem(0).(*tview.Table)
updateHmacTable(hmacTable, hmacInfo, metrics)
}
// Einführung von Channels für verschiedene Daten
systemDataCh := make(chan struct {
memUsage float64
cpuUsage float64
cores int
err error
})
var metricsCh chan struct {
metrics map[string]float64
err error
}
if metricsEnabled {
metricsCh = make(chan struct {
metrics map[string]float64
err error
})
}
processListCh := make(chan struct {
processes []ProcessInfo
err error
})
hmacInfoCh := make(chan struct {
info *ProcessInfo
metrics map[string]float64
err error
})
// Goroutine zur Datenbeschaffung
go func() {
for {
select {
case <-ctx.Done():
close(systemDataCh)
if metricsEnabled {
close(metricsCh)
}
close(processListCh)
close(hmacInfoCh)
return
case <-ticker.C:
// Systemdaten abrufen asynchron
go func() {
memUsage, cpuUsage, cores, err := fetchSystemData()
systemDataCh <- struct {
memUsage float64
cpuUsage float64
cores int
err error
}{memUsage, cpuUsage, cores, err}
}()
if metricsEnabled {
// Metriken abrufen asynchron
go func() {
metrics, err := fetchMetrics()
metricsCh <- struct {
metrics map[string]float64
err error
}{metrics, err}
}()
}
// Prozessliste abrufen asynchron
go func() {
processes, err := fetchProcessList()
processListCh <- struct {
processes []ProcessInfo
err error
}{processes, err}
}()
// hmac-file-server Informationen abrufen asynchron
go func() {
hmacInfo, err := fetchHmacFileServerInfo()
var metrics map[string]float64
if metricsEnabled {
metrics, err = fetchMetrics()
}
hmacInfoCh <- struct {
info *ProcessInfo
metrics map[string]float64
err error
}{hmacInfo, metrics, err}
}()
}
}
}()
for {
select {
case <-ctx.Done():
return
case data, ok := <-systemDataCh:
if !ok {
systemDataCh = nil
continue
}
if data.err != nil {
log.Printf("Fehler beim Abrufen der Systemdaten: %v\n", data.err)
continue
}
// UI aktualisieren mit Systemdaten
app.QueueUpdateDraw(func() {
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
sysFlex := sysPage.(*tview.Flex)
sysTable := sysFlex.GetItem(0).(*tview.Table)
updateSystemTable(sysTable, data.memUsage, data.cpuUsage, data.cores)
}
})
case data, ok := <-metricsCh:
if !ok {
metricsCh = nil
continue
}
if data.err != nil {
log.Printf("Fehler beim Abrufen der Metriken: %v\n", data.err)
continue
}
// UI aktualisieren mit Metriken
app.QueueUpdateDraw(func() {
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
sysFlex := sysPage.(*tview.Flex)
metricsTable := sysFlex.GetItem(1).(*tview.Table)
updateMetricsTable(metricsTable, data.metrics)
}
})
case data, ok := <-processListCh:
if !ok {
processListCh = nil
continue
}
if data.err != nil {
log.Printf("Fehler beim Abrufen der Prozessliste: %v\n", data.err)
continue
}
// UI aktualisieren mit Prozessliste
app.QueueUpdateDraw(func() {
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
sysFlex := sysPage.(*tview.Flex)
processTable := sysFlex.GetItem(2).(*tview.Table)
updateProcessTable(processTable, data.processes)
}
})
case data, ok := <-hmacInfoCh:
if !ok {
hmacInfoCh = nil
continue
}
if data.err != nil {
log.Printf("Fehler beim Abrufen der hmac-file-server Informationen: %v\n", data.err)
continue
}
// UI aktualisieren mit hmac-file-server Informationen
app.QueueUpdateDraw(func() {
if currentPage, _ := pages.GetFrontPage(); currentPage == "hmac" && data.info != nil {
hmacFlex := hmacPage.(*tview.Flex)
hmacTable := hmacFlex.GetItem(0).(*tview.Table)
updateHmacTable(hmacTable, data.info, data.metrics)
}
})
}
// Abbruchbedingung, wenn alle Channels geschlossen sind
if systemDataCh == nil && (!metricsEnabled || metricsCh == nil) && processListCh == nil && hmacInfoCh == nil {
break
}
}
}
// Helper function to update system data table
@ -419,9 +665,27 @@ func updateHmacTable(hmacTable *tview.Table, hmacInfo *ProcessInfo, metrics map[
hmacTable.SetCell(4, 0, tview.NewTableCell("Command"))
hmacTable.SetCell(4, 1, tview.NewTableCell(hmacInfo.CommandLine))
hmacTable.SetCell(5, 0, tview.NewTableCell("Uptime"))
hmacTable.SetCell(5, 1, tview.NewTableCell(hmacInfo.Uptime)) // Neue Zeile für Uptime
hmacTable.SetCell(6, 0, tview.NewTableCell("Status"))
hmacTable.SetCell(6, 1, tview.NewTableCell(hmacInfo.Status)) // Neue Zeile für Status
hmacTable.SetCell(7, 0, tview.NewTableCell("Error Count"))
hmacTable.SetCell(7, 1, tview.NewTableCell(fmt.Sprintf("%d", hmacInfo.ErrorCount))) // Neue Zeile für Error Count
hmacTable.SetCell(8, 0, tview.NewTableCell("Total Requests"))
hmacTable.SetCell(8, 1, tview.NewTableCell(fmt.Sprintf("%d", hmacInfo.TotalRequests))) // Neue Zeile für Total Requests
hmacTable.SetCell(9, 0, tview.NewTableCell("Active Connections"))
hmacTable.SetCell(9, 1, tview.NewTableCell(fmt.Sprintf("%d", hmacInfo.ActiveConnections))) // Neue Zeile für Active Connections
hmacTable.SetCell(10, 0, tview.NewTableCell("Avg. Response Time (ms)"))
hmacTable.SetCell(10, 1, tview.NewTableCell(fmt.Sprintf("%.2f", hmacInfo.AverageResponseTime))) // Neue Zeile für Average Response Time
// Metrics related to hmac-file-server
row := 6
row := 12
hmacTable.SetCell(row, 0, tview.NewTableCell("Metric").SetAttributes(tcell.AttrBold))
hmacTable.SetCell(row, 1, tview.NewTableCell("Value").SetAttributes(tcell.AttrBold))
row++
@ -469,72 +733,117 @@ func createHmacPage() tview.Primitive {
return hmacFlex
}
func createLogsPage(logFilePath string) tview.Primitive {
logsTextView := tview.NewTextView().
SetDynamicColors(true).
SetRegions(true).
SetWordWrap(true)
logsTextView.SetTitle(" [::b]Logs ").SetBorder(true)
func createLogsPage(ctx context.Context, app *tview.Application, logFilePath string) tview.Primitive {
logsTextView := tview.NewTextView().
SetDynamicColors(true).
SetRegions(true).
SetWordWrap(true)
logsTextView.SetTitle(" [::b]Logs ").SetBorder(true)
const numLines = 100 // Number of lines to read from the end of the log file
const numLines = 100 // Number of lines to read from the end of the log file
// Read logs periodically
go func() {
for {
content, err := readLastNLines(logFilePath, numLines)
if err != nil {
logsTextView.SetText(fmt.Sprintf("[red]Error reading log file: %v[white]", err))
} else {
// Process the log content to add colors
lines := strings.Split(content, "\n")
var coloredLines []string
for _, line := range lines {
if strings.Contains(line, "level=info") {
coloredLines = append(coloredLines, "[green]"+line+"[white]")
} else if strings.Contains(line, "level=warn") {
coloredLines = append(coloredLines, "[yellow]"+line+"[white]")
} else if strings.Contains(line, "level=error") {
coloredLines = append(coloredLines, "[red]"+line+"[white]")
} else {
// Default color
coloredLines = append(coloredLines, line)
}
}
logsTextView.SetText(strings.Join(coloredLines, "\n"))
}
time.Sleep(2 * time.Second) // Refresh interval for logs
}
}()
// Read logs periodically
go func() {
for {
select {
case <-ctx.Done():
return
default:
content, err := readLastNLines(logFilePath, numLines)
if err != nil {
app.QueueUpdateDraw(func() {
logsTextView.SetText(fmt.Sprintf("[red]Error reading log file: %v[white]", err))
})
} else {
// Process the log content to add colors
lines := strings.Split(content, "\n")
var coloredLines []string
for _, line := range lines {
if strings.Contains(line, "level=info") {
coloredLines = append(coloredLines, "[green]"+line+"[white]")
} else if strings.Contains(line, "level=warn") {
coloredLines = append(coloredLines, "[yellow]"+line+"[white]")
} else if strings.Contains(line, "level=error") {
coloredLines = append(coloredLines, "[red]"+line+"[white]")
} else {
// Default color
coloredLines = append(coloredLines, line)
}
}
app.QueueUpdateDraw(func() {
logsTextView.SetText(strings.Join(coloredLines, "\n"))
})
}
time.Sleep(2 * time.Second) // Refresh interval for logs
}
}
}()
return logsTextView
return logsTextView
}
// Optimized readLastNLines to handle large files efficiently
func readLastNLines(filePath string, n int) (string, error) {
file, err := os.Open(filePath)
if err != nil {
return "", err
}
defer file.Close()
file, err := os.Open(filePath)
if err != nil {
return "", err
}
defer file.Close()
var lines []string
scanner := bufio.NewScanner(file)
for scanner.Scan() {
lines = append(lines, scanner.Text())
if len(lines) > n {
lines = lines[1:]
}
}
const bufferSize = 1024
buffer := make([]byte, bufferSize)
var content []byte
var fileSize int64
if err := scanner.Err(); err != nil {
return "", err
}
fileInfo, err := file.Stat()
if err != nil {
return "", err
}
fileSize = fileInfo.Size()
return strings.Join(lines, "\n"), nil
var offset int64 = 0
for {
if fileSize-offset < bufferSize {
offset = fileSize
} else {
offset += bufferSize
}
_, err := file.Seek(-offset, io.SeekEnd)
if err != nil {
return "", err
}
bytesRead, err := file.Read(buffer)
if err != nil && err != io.EOF {
return "", err
}
content = append(buffer[:bytesRead], content...)
if bytesRead < bufferSize || len(strings.Split(string(content), "\n")) > n+1 {
break
}
if offset >= fileSize {
break
}
}
lines := strings.Split(string(content), "\n")
if len(lines) > n {
lines = lines[len(lines)-n:]
}
return strings.Join(lines, "\n"), nil
}
func main() {
app := tview.NewApplication()
// Create a cancellable context
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Create pages
pages := tview.NewPages()
@ -547,14 +856,15 @@ func main() {
pages.AddPage("hmac", hmacPage, true, false)
// Logs page mit dem gelesenen logFilePath
logsPage := createLogsPage(logFilePath)
logsPage := createLogsPage(ctx, app, logFilePath)
pages.AddPage("logs", logsPage, true, false)
// Add key binding to switch views
// Add key binding to switch views and handle exit
app.SetInputCapture(func(event *tcell.EventKey) *tcell.EventKey {
if event.Key() == tcell.KeyRune {
switch event.Rune() {
case 'q', 'Q':
cancel()
app.Stop()
return nil
case 's', 'S':
@ -572,10 +882,11 @@ func main() {
})
// Start the UI update loop in a separate goroutine
go updateUI(app, pages, sysPage, hmacPage)
go updateUI(ctx, app, pages, sysPage, hmacPage)
// Set the root and run the application
if err := app.SetRoot(pages, true).EnableMouse(true).Run(); err != nil {
log.Fatalf("Error running application: %v", err)
log.Fatalf("Error running application: %v", err)
}
}
}

View File

@ -1,35 +1,35 @@
[server]
bind_ip = "127.0.0.1"
#bind_ip = "127.0.0.1"
listenport = "8080"
unixsocket = false
storagepath = "/mnt/nfs_vol01/hmac-file-server/data/"
storagepath = "./uploads/"
metricsenabled = true
metricsport = "9090"
deduplicationenabled = true
minfreebytes = "5GB"
filettl = "2y"
filettlenabled = true
filettlenabled = false
autoadjustworkers = true
networkevents = false
pidfilepath = "./hmac-file-server.pid"
precaching = true
precaching = false
#globalextensions = ["*"]
[deduplication]
enabled = true
directory = "/mnt/nfs_vol01/hmac-file-server/deduplication/"
directory = "./deduplication/"
[logging]
level = "debug"
file = "/var/log/hmac-file-server.log"
file = "./hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
[thumbnails]
enabled = true
directory = "/mnt/nfs_vol01/hmac-file-server/thumbnails/"
enabled = false
directory = "./thumbnails/"
size = "200x200"
thumbnailintervalscan = "1h"
concurrency = 5
@ -74,7 +74,7 @@ allowedextensions = [
]
[clamav]
clamavenabled = true
clamavenabled = false
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 4
scanfileextensions = [
@ -83,7 +83,7 @@ scanfileextensions = [
]
[redis]
redisenabled = true
redisenabled = false
redisdbindex = 0
redisaddr = "localhost:6379"
redispassword = ""

Binary file not shown.

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

0
cmd/server/uploads/test Normal file
View File

86
config.example.toml Normal file
View File

@ -0,0 +1,86 @@
[server]
bind_ip = "0.0.0.0"
listenport = "8080"
unixsocket = false
storagepath = "./uploads"
logfile = "/var/log/hmac-file-server.log"
metricsenabled = true
metricsport = "9090"
minfreebytes = "100MB"
filettl = "8760h"
filettlenabled = true
autoadjustworkers = true
networkevents = true
pidfilepath = "/var/run/hmacfileserver.pid"
cleanuponexit = true
precaching = true
deduplicationenabled = true
globalextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
# FileNaming options: "HMAC", "None"
filenaming = "HMAC"
[logging]
level = "info"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
[deduplication]
enabled = true
directory = "./deduplication"
[iso]
enabled = true
size = "1GB"
mountpoint = "/mnt/iso"
charset = "utf-8"
containerfile = "/mnt/iso/container.iso"
[timeouts]
readtimeout = "4800s"
writetimeout = "4800s"
idletimeout = "4800s"
[security]
secret = "changeme"
[versioning]
enableversioning = false
maxversions = 1
[uploads]
resumableuploadsenabled = true
chunkeduploadsenabled = true
chunksize = "8192"
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
[downloads]
resumabledownloadsenabled = true
chunkeddownloadsenabled = true
chunksize = "8192"
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
[clamav]
clamavenabled = true
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 2
scanfileextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
[redis]
redisenabled = true
redisdbindex = 0
redisaddr = "localhost:6379"
redispassword = ""
redishealthcheckinterval = "120s"
[workers]
numworkers = 4
uploadqueuesize = 50
[file]
# Add file-specific configurations here
[build]
version = "2.6-Stable"

View File

@ -1,105 +1,19 @@
[server]
listenport = "8080"
unixsocket = false
storagepath = "./uploads"
metricsenabled = true
metricsport = "9090"
filettl = "8760h"
minfreebytes = "100MB"
autoadjustworkers = true
networkevents = true
temppath = "/tmp/hmac-file-server"
loggingjson = false
pidfilepath = "/var/run/hmacfileserver.pid"
cleanuponexit = true
precaching = true
filettlenabled = true
# Server configuration
listenport = "8080" # TCP port for incoming requests
unixsocket = false # Use Unix domain socket instead of TCP
storagepath = "/path/to/hmac-file-server/data/" # Directory to store uploaded files
loglevel = "debug" # Logging level: "debug", "info", "warn", "error"
logfile = "/path/to/hmac-file-server.log" # Path to log file; leave empty to use stdout
metricsenabled = true # Enable Prometheus metrics
metricsport = "9090" # Port for Prometheus metrics
deduplicationenabled = true
globalextensions = ["*"] # Allows all file types globally
bind_ip = "0.0.0.0" # New option: Specify the IP address to bind to (IPv4 or IPv6)
minfreebytes = "5GB" # Minimum free disk space required
filettl = "2Y" # Time-to-live for files
filettlenabled = false # Enable TTL checks and cleanup
autoadjustworkers = true # Automatically adjust worker threads based on load
networkevents = false # Enable detailed network event logging
pidfilepath = "./hmac-file-server.pid" # Path to PID file
precaching = true # Pre-cache file structures on startup
[logging]
level = "info"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
[deduplication]
enabled = false
directory = "./deduplication"
[thumbnails]
enabled = false
directory = "./thumbnails"
size = "200x200"
thumbnailintervalscan = "24h"
[iso]
enabled = false
size = "1GB"
mountpoint = "/mnt/iso"
charset = "utf-8"
containerfile = "/path/to/iso/container.iso"
[timeouts]
readtimeout = "4800s"
writetimeout = "4800s"
idletimeout = "4800s"
[security]
secret = "changeme"
[versioning]
enableversioning = false
maxversions = 1
[uploads]
resumableuploadsenabled = true
chunkeduploadsenabled = true
chunksize = "8192"
allowedextensions = [
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
]
[downloads]
resumabledownloadsenabled = true
chunkeddownloadsenabled = true
chunksize = "8192"
allowedextensions = [".jpg", ".png"] # Restricts downloads to specific types
[clamav]
clamavenabled = true
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 2
scanfileextensions = [".exe", ".dll", ".pdf"]
[redis]
redisenabled = true
redisaddr = "localhost:6379"
redispassword = ""
redisdbindex = 0
redishealthcheckinterval = "120s"
[workers]
numworkers = 4
uploadqueuesize = 50
max_concurrent_operations = 10
network_event_buffer = 100
performance_monitor_interval = "5m"
metrics_update_interval = "10s"
[file]
filerevision = 1
[precache]
redisEnabled = true
redisAddr = "localhost:6379"
staticIndexFile = "./static_index.json"
[build]
version = "v1.0.0"
# New option to force network protocol
forceprotocol = "auto" # Options: "ipv4", "ipv6", "auto"

89
config.toml.example Normal file
View File

@ -0,0 +1,89 @@
### Issue
In `config.toml`, comments are using `//`, which is invalid in TOML. TOML uses `#` for comments. This causes a syntax error.
### Corrected `config.toml`
```toml
# filepath: /home/renz/source/hmac-file-server/config.toml
[server]
listenport = "8080"
unixsocket = false
storagepath = "./uploads"
metricsenabled = true
metricsport = "9090"
filettl = "8760h"
minfreebytes = "100MB"
autoadjustworkers = true
networkevents = true
temppath = "/tmp/hmac-file-server"
loggingjson = false
pidfilepath = "/var/run/hmacfileserver.pid"
cleanuponexit = true
precaching = true
filettlenabled = true
globalextensions = ["*"] # Allows all file types globally
bind_ip = "0.0.0.0" # Specify the IP address to bind to (IPv4 or IPv6)
[logging]
level = "info"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
[deduplication]
enabled = true
directory = "./deduplication"
[iso]
enabled = true
size = "1GB"
mountpoint = "/mnt/iso"
charset = "utf-8"
containerfile = "/path/to/iso/container.iso"
[timeouts]
readtimeout = "4800s"
writetimeout = "4800s"
idletimeout = "4800s"
[security]
secret = "changeme"
[versioning]
enableversioning = false
maxversions = 5
[uploads]
resumableuploadsenabled = false
chunkeduploadsenabled = true
chunksize = "64mb"
allowedextensions = ["*"] # Use ["*"] to allow all or specify extensions
[downloads]
resumabledownloadsenabled = false
chunkeddownloadsenabled = true
chunksize = "64mb"
allowedextensions = [".jpg", ".png"] # Restricts downloads to specific types
[clamav]
clamavenabled = false
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 2
scanfileextensions = [".exe", ".dll", ".pdf"]
[redis]
redisenabled = false
redisaddr = "localhost:6379"
redispassword = ""
redisdbindex = 0
redishealthcheckinterval = "120s"
[workers]
numworkers = 4
uploadqueuesize = 5000
[build]
version = "v2.5"

File diff suppressed because it is too large Load Diff

BIN
dashboard/hmac_icon.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

32
docs/updates.md Normal file
View File

@ -0,0 +1,32 @@
**Important Update Regarding HMAC File Server Version 2.6-Stable**
We've identified and resolved critical issues with the deduplication processes in **Version 2.6-Stable** of the HMAC File Server. This update brings significant improvements to ensure reliable deduplication and overall server performance.
**Impact:**
- **Resolved Deduplication Issues:** The deduplication feature in **Version 2.6-Stable** has been thoroughly tested and stabilized, ensuring consistent file integrity and optimal server performance.
- **Enhanced Stability:** Addressed previous unexpected behaviors, including lost links to files and incorrect file handling operations.
**Recommended Actions:**
- **Upgrade to Version 2.6-Stable:** If you are using any earlier version, we strongly recommend upgrading to **Version 2.6-Stable** to benefit from the latest fixes and improvements.
- **Rollback Not Required:** Users on older releases can upgrade without the need to rollback, as **Version 2.6-Stable** resolves all critical issues present in prior versions.
**What's New in Version 2.6-Stable:**
- **Deduplication Enhancements:**
- **Reliable Deduplication:** Improved algorithms for accurate and efficient processing.
- **Performance Optimizations:** Reduced resource consumption for faster deduplication.
- **Enhanced Features:**
- **Robust Malware Scanning:** Integrated advanced scanning to ensure file safety.
- **Enhanced Redis Support:** Improved performance with Redis for better caching.
- **Better Configuration Options:** More flexible settings for customization.
- **Security Improvements:**
- **Strengthened Authentication Mechanisms:** Safeguards against unauthorized access.
- **Improved Data Encryption:** Advanced encryption standards for data at rest and in transit.
**Stay Updated:**
For detailed information on addressed issues, fixes, and enhancements introduced in **Version 2.6-Stable**, please visit our repositories:
- [Git uuxo.net](https://git.uuxo.net/uuxo/hmac-file-server)
- [GitHub](https://github.com/PlusOne/hmac-file-server)
We appreciate your patience and support as we continue improving HMAC File Server. **Version 2.6-Stable** reflects our commitment to delivering a reliable and efficient file server solution. Thank you for choosing HMAC File Server!

6
go.mod
View File

@ -1,6 +1,6 @@
module github.com/PlusOne/hmac-file-server
go 1.21
go 1.24.0
require (
github.com/gdamore/tcell/v2 v2.7.4
@ -15,7 +15,7 @@ require (
require (
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/fsnotify/fsnotify v1.7.0
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/gdamore/encoding v1.0.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
@ -32,7 +32,6 @@ require (
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8 // indirect
golang.org/x/term v0.17.0 // indirect
golang.org/x/text v0.21.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
@ -43,7 +42,6 @@ require (
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/disintegration/imaging v1.6.2
github.com/dutchcoders/go-clamd v0.0.0-20170520113014-b970184f4d9e
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/klauspost/compress v1.17.11 // indirect

4
go.sum
View File

@ -7,8 +7,6 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/disintegration/imaging v1.6.2 h1:w1LecBlG2Lnp8B3jk5zSuNqd7b4DXhcjwek1ei82L+c=
github.com/disintegration/imaging v1.6.2/go.mod h1:44/5580QXChDfwIclfc/PCwrr44amcmDAg8hxG0Ewe4=
github.com/dutchcoders/go-clamd v0.0.0-20170520113014-b970184f4d9e h1:rcHHSQqzCgvlwP0I/fQ8rQMn/MpHE5gWSLdtpxtP6KQ=
github.com/dutchcoders/go-clamd v0.0.0-20170520113014-b970184f4d9e/go.mod h1:Byz7q8MSzSPkouskHJhX0er2mZY/m0Vj5bMeMCkkyY4=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
@ -124,8 +122,6 @@ github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8 h1:hVwzHzIUGRjiF7EcUjqNxk3NCfkPxbDKRdnNE1Rpg0U=
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=

69
lib/maps/iter.go Normal file
View File

@ -0,0 +1,69 @@
package maps
// Copyright 2024 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
type Seq2[K comparable, V any] func(yield func(K, V) bool)
type Seq[K any] func(yield func(K) bool)
func All[Map ~map[K]V, K comparable, V any](m Map) Seq2[K, V] {
return func(yield func(K, V) bool) {
for k, v := range m {
if !yield(k, v) {
return
}
}
}
}
// All returns an iterator over key-value pairs from m.
// The iteration order is not specified and is not guaranteed
// to be the same from one call to the next.
func Insert[Map ~map[K]V, K comparable, V any](m Map, seq Seq2[K, V]) {
seq(func(k K, v V) bool {
m[k] = v
return true
})
}
// Insert adds the key-value pairs from seq to m.
// If a key in seq already exists in m, its value will be overwritten.
func Collect[K comparable, V any](seq Seq2[K, V]) map[K]V {
m := make(map[K]V)
Insert(m, seq)
return m
}
// Collect collects key-value pairs from seq into a new map
// and returns it.
func Keys[Map ~map[K]V, K comparable, V any](m Map) Seq[K] {
return func(yield func(K) bool) {
for k := range m {
if !yield(k) {
return
}
}
}
}
// Keys returns an iterator over keys in m.
// The iteration order is not specified and is not guaranteed
// to be the same from one call to the next.
func Values[Map ~map[K]V, K comparable, V any](m Map) Seq[V] {
return func(yield func(V) bool) {
for _, v := range m {
if !yield(v) {
return
}
}
}
}
// Values returns an iterator over values in m.
// The iteration order is not specified and is not guaranteed
// to be the same from one call to the next.

View File

@ -16,7 +16,7 @@ import (
const (
serverURL = "http://[::1]:8080" // Replace with your actual server URL
secret = "changeme" // Replace with your HMAC secret key
secret = "hmac-file-server-is-the-win" // Replace with your HMAC secret key
uploadPath = "hmac_icon.png" // Test file to upload
protocolType = "v2" // Use v2, v, or token as needed
)

1126
wiki.md Normal file

File diff suppressed because it is too large Load Diff