uuxo/hmac-file-server
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

2.4-stable release

Browse Source
This commit is contained in:
Alexander Renz 2024-12-30 11:03:21 +01:00
parent fa4fe8a932
commit 7c5432fbe7
10 changed files with 2941 additions and 923 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

311
README.MD
View File

@ -1,4 +1,5 @@
# HMAC File Server
A secure file server with HMAC authentication and configurable features.
**HMAC File Server** is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, deduplication, optional ClamAV scanning for file integrity and security, and image thumbnail generation. This server is built with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration.
@ -53,6 +54,17 @@ cd hmac-file-server
go build -o hmac-file-server main.go
```
#### Build for `arm64`
```bash
cd /path/to/hmac-file-server
GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.3-stable_arm64 main.go
```
#### Build for `amd64`
```bash
cd /path/to/hmac-file-server
GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.3-stable_amd64 main.go
```
---
## Configuration
@ -100,6 +112,7 @@ autoadjustworkers = true # Automatically adjust worker threads based on load
networkevents = false # Enable detailed network event logging
pidfilepath = "./hmac-file-server.pid" # Path to PID file
precaching = true # Pre-cache file structures on startup
tempPath = "/tmp/hmac-file-server" # Path for temporary file uploads
# Deduplication settings
[deduplication]
@ -112,6 +125,7 @@ enabled = true
directory = "/path/to/hmac-file-server/thumbnails/" # Directory for storing thumbnails
size = "200x200" # Thumbnail dimensions
thumbnailintervalscan = "1h" # Interval for scheduled thumbnail generation
concurrency = 5 # Number of concurrent thumbnail generation tasks
# ISO settings
[iso]
@ -140,12 +154,7 @@ maxversions = 1 # Number of file versions to retain
resumableuploadsenabled = false
chunkeduploadsenabled = true
chunksize = "32MB" # Chunk size for uploads
allowedextensions = [
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
]
allowedextensions = ["*"] # All file types are allowed for uploads
# Downloads settings
[downloads]
@ -179,10 +188,38 @@ uploadqueuesize = 5000
# File settings
[file]
filerevision = 1 # Internal revision number for file handling logic
# Logging settings
[logging]
level = "info"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
```
---
## Configuration Verification
The application ensures that all configuration parameters defined in `config.toml` are correctly implemented and operational. During startup, the server performs the following verification steps:
1. **Configuration Parsing:**
- Parses the `config.toml` file into the application's `Config` struct.
2. **Validation:**
- Checks that all required configuration parameters are set.
- Validates the correctness of configuration values (e.g., proper duration formats, non-empty directories).
3. **Service Initialization:**
- Initializes services like Redis and ClamAV based on the configuration.
- Ensures that conditional services (e.g., Thumbnails, ISO) are only started if enabled in the configuration.
If any configuration parameter is missing or invalid, the server will log an error and terminate to prevent running with incorrect settings.
---
## Running the Server
### Basic Usage
@ -193,273 +230,21 @@ Run the server with a configuration file:
./hmac-file-server -config ./config.toml
```
Ensure that the `config.toml` is correctly set up before running the server.
---
### Metrics Server
If `metricsenabled` is set to `true`, the Prometheus metrics server will be available on the port specified in `metricsport` (e.g., `http://localhost:9090/metrics`).
---
## Testing
To run the server locally for development:
```bash
go run main.go -config ./config.toml
```
Use tools like **cURL** or **Postman** to test file uploads and downloads.
### Example File Upload with HMAC Token
```bash
curl -X PUT -H "Authorization: Bearer <HMAC-TOKEN>" -F "file=@example.txt" http://localhost:8080/uploads/example.txt
```
Replace `<HMAC-TOKEN>` with a valid HMAC signature generated using the configured `secret`.
Additional metrics for deduplication and ISO container operations are now available.
---
## Monitoring
## Health Checks
Prometheus metrics include:
- **File Operations:**
- File upload/download durations
- Uploaded/downloaded file sizes
- Total number of uploads/downloads
- Total number of upload/download errors
- **System Metrics:**
- Memory usage
- CPU usage
- Number of goroutines
- Active connections
- **HTTP Requests:**
- Total HTTP requests broken down by method and path
Access the metrics at `http://localhost:9090/metrics` (assuming default `metricsport`).
---
## Additional Features
- **Deduplication:** Automatically remove duplicate files based on SHA256 hashing to save storage space.
- **Versioning:** Store multiple versions of files and retain a configurable number of versions.
- **ClamAV Integration:** Scan uploaded files for viruses using ClamAV to ensure file integrity and security.
- **Redis Caching:** Utilize Redis for caching file metadata to improve access times and performance.
- **Auto-Adjust Worker Scaling:** Optimize the number of workers dynamically based on system resources to maintain optimal performance.
- **Precaching:** Pre-cache file structures on startup to reduce access times for frequently accessed files.
- **Thumbnail Creation:** Generate image thumbnails for uploaded files to provide quick previews.
---
### Overview of Other Projects (xep0363)
| Feature/Project | HMAC FS | mod_http_upload_ext | xmpp-http-upload (horazont) | Prosody Filer | ngx_http_upload | xmpp-http-upload (nyovaya) |
|-----------------------------|---------|----------------------|-----------------------------|---------------|------------------|----------------------------|
| **Language** | Go | PHP | Python | Go | C (Nginx) | Python |
| **Environment** | Standalone | Prosody module | Standalone | Standalone | Nginx | Standalone |
| **XMPP** | No | Yes | Yes | Yes | No | Yes |
| **External Storage** | Yes | No | Possible via plugins | No | No | Yes |
| **Authentication / Security** | HMAC | Token-based | Token-based | None | Basic / None | Token-based |
---
## Build & Run
1. **Clone the Repository:**
```bash
git clone https://github.com/PlusOne/hmac-file-server.git
cd hmac-file-server
```
2. **Build the Server:**
```bash
go build -o hmac-file-server main.go
```
3. **Run the Server:**
```bash
./hmac-file-server -config ./config.toml
```
---
## Configuration Details
### [server]
- **listenport:** TCP port for incoming requests (e.g., `"8080"`).
- **unixsocket:** Use a Unix domain socket (`true`/`false`) instead of TCP.
- **storagepath:** Local directory to store uploaded files.
- **loglevel:** Logging verbosity (`"debug"`, `"info"`, `"warn"`, `"error"`).
- **logfile:** Path to a file for logging; empty uses stdout.
- **metricsenabled:** Enable or disable Prometheus metrics endpoint.
- **metricsport:** Port for metrics (e.g., `"9090"`).
- **deduplicationenabled:** Enable or disable file deduplication.
- **minfreebytes:** Minimum free space required on disk (e.g., `"5GB"`).
- **filettl:** Default time-to-live for files (e.g., `"2Y"` for 2 years).
- **filettlenabled:** Enable or disable TTL checks and cleanup.
- **autoadjustworkers:** Dynamically adjust worker threads based on load.
- **networkevents:** Enable detailed network event logging (may be verbose).
- **pidfilepath:** Where the server writes its PID file.
- **precaching:** Pre-cache file structures on startup for faster access.
### [deduplication]
- **enabled:** Turn file deduplication on or off.
- **directory:** Path to a deduplication metadata store.
### [thumbnails]
- **enabled:** Generate thumbnails for uploaded images.
- **directory:** Where thumbnail files are stored.
- **size:** Dimensions for generated thumbnails (e.g., `"200x200"`).
- **thumbnailintervalscan:** Interval for scheduled thumbnail generation (e.g., `"1h"`).
> **Note:** The configuration key `thumbnailintervalscan` is used in the code. Ensure consistency in `config.toml` by using this key instead of `scanInterval`.
### [iso]
*(Only one `[iso]` block should be active if needed)*
- **enabled:** Mount an ISO-based filesystem (for specialized use).
- **size:** Maximum allowed size (e.g., `"1TB"`).
- **mountpoint:** Path to mount the ISO.
- **charset:** Filesystem character set encoding (e.g., `"utf-8"`).
### [timeouts]
- **readtimeout:** Maximum time to read a request (e.g., `"3600s"` for 1 hour).
- **writetimeout:** Maximum time to write a response (e.g., `"3600s"` for 1 hour).
- **idletimeout:** Maximum keep-alive time for idle connections (e.g., `"3600s"` for 1 hour).
### [security]
- **secret:** HMAC shared secret key for signing requests and operations.
> **Warning:** Change this to a secure, random string in production environments to ensure the security of HMAC operations.
### [versioning]
- **enableversioning:** Keep multiple versions of the same file (`true`/`false`).
- **maxversions:** How many versions of a file to retain (e.g., `1`).
### [uploads]
- **resumableuploadsenabled:** Support for chunked/resumable file uploads (`true`/`false`).
- **chunkeduploadsenabled:** Enable chunked uploads specifically (`true`/`false`).
- **chunksize:** Default chunk size for chunked uploads (e.g., `"32MB"`).
- **allowedextensions:** Restrict which file types are allowed.
```toml
allowedextensions = [
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
]
```
### [downloads]
- **resumabledownloadsenabled:** Support for chunked/resumable downloads (`true`/`false`).
- **chunkeddownloadsenabled:** Enable chunk-by-chunk downloading (`true`/`false`).
- **chunksize:** Default chunk size for chunked downloads (e.g., `"32MB"`).
> **Note:** The `allowedextensions` key is **not** part of the `[downloads]` configuration based on the provided code. Ensure that this key is omitted to prevent configuration errors.
### [clamav]
- **clamavenabled:** Integrate ClamAV for virus scanning (`true`/`false`).
- **clamavsocket:** Path to ClamAV socket (`.ctl` file, e.g., `"/path/to/clamav/clamd.ctl"`).
- **numscanworkers:** Number of concurrent scan workers (e.g., `4`).
- **scanfileextensions:** Extensions to scan for viruses.
```toml
scanfileextensions = [
".exe", ".dll", ".bin", ".com", ".bat",
".sh", ".php", ".js"
]
```
### [redis]
- **redisenabled:** Use Redis for caching or session tracking (`true`/`false`).
- **redisaddr:** Address of the Redis server (e.g., `"localhost:6379"`).
- **redispassword:** Redis authentication password, if needed.
- **redisdbindex:** Redis database index (integer, e.g., `0`).
- **redishealthcheckinterval:** Interval to check Redis connectivity (e.g., `"120s"`).
### [workers]
- **numworkers:** Number of worker threads for handling requests (e.g., `4`).
- **uploadqueuesize:** Maximum queue length for incoming uploads (e.g., `5000`).
### [file]
- **filerevision:** Internal revision number for file handling logic (e.g., `1`).
---
### **Important Notes**
- **Secret Key:**
- **Change the `secret` immediately** to a unique, strong string for production environments to ensure the security of HMAC operations.
- **ISO Configuration:**
- **Only use one `[iso]` block** in your `config.toml` to avoid conflicts. If ISO mounting is not required, ensure that `[iso].enabled` is set to `false`.
- **Configuration Consistency:**
- **Ensure all configuration keys in `config.toml` match those expected in the code.** For example, in the `[thumbnails]` section, use `thumbnailintervalscan` instead of `scanInterval` to align with the code.
- **Directory Paths:**
- Replace all placeholder paths (e.g., `"/path/to/hmac-file-server/data/"`) with actual paths relevant to your server environment.
- Ensure that directories specified in the configuration exist and have appropriate permissions.
- **Allowed Extensions:**
- The `[uploads].allowedextensions` restricts the types of files that can be uploaded. Adjust this list based on your security requirements.
- **ClamAV Integration:**
- Ensure that ClamAV is installed and properly configured on your system if `[clamav].clamavenabled` is set to `true`.
- The `clamavsocket` should point to the correct ClamAV socket file.
- **Redis Usage:**
- If Redis is enabled (`[redis].redisenabled = true`), ensure that the Redis server is running and accessible at the specified `redisaddr`.
- **Resource Allocation:**
- Adjust `numworkers` and `uploadqueuesize` in the `[workers]` section based on your server's hardware capabilities and expected traffic.
- The `autoadjustworkers` option can help dynamically manage worker threads based on system resources, but monitor its behavior to ensure optimal performance.
- **Metrics Monitoring:**
- Prometheus metrics are enabled by default. Access them at the specified `metricsport` (e.g., `http://localhost:9090/metrics`).
- Utilize these metrics to monitor server performance and identify potential bottlenecks.
- **File TTL and Cleanup:**
- The `[server].filettl` and `[server].filettlenabled` settings manage the automatic deletion of old files. Ensure these are configured according to your data retention policies.
---
## Running the Server
### Basic Usage
Run the server with a configuration file:
```bash
./hmac-file-server -config ./config.toml
```
---
### Metrics Server
If `metricsenabled` is set to `true`, the Prometheus metrics server will be available on the port specified in `metricsport` (e.g., `http://localhost:9090/metrics`).
The server includes built-in health checks to verify that all configured services are operational. Logs will provide detailed information about the status of each service and any issues encountered during startup.
---

101
RELEASE-NOTES.MD
View File

@ -18,6 +18,7 @@
- **Graceful Shutdown:** Handles signals and ensures proper cleanup.
- **Deduplication:** Remove duplicate files based on hashing for storage efficiency.
- **Auto-Adjust Worker Scaling:** Dynamically optimize HMAC and ClamAV workers based on system resources when enabled.
- **Thumbnail Support:** Generate smaller versions of uploaded images for efficient storage and quick access.
---
@ -49,6 +50,24 @@ cd hmac-file-server
go build -o hmac-file-server main.go
```
### Building for Different Architectures
To build the HMAC File Server for different architectures, use the following commands:
#### Build for `arm64`
```bash
cd /path/to/hmac-file-server
GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.2-stable_arm64 main.go
```
#### Build for `amd64`
```bash
cd /path/to/hmac-file-server
GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.2-stable_amd64 main.go
```
Replace `/path/to/hmac-file-server` with the actual path to your project directory. These commands will generate the binaries for `arm64` and `amd64` architectures and place them in the `~/Temp` directory.
---
## Configuration
@ -69,9 +88,9 @@ Setting `NetworkEvents = false` in the server configuration disables the logging
The `precaching` feature allows the server to pre-cache storage paths for faster access. This can improve performance by reducing the time needed to access frequently used storage paths.
### Added thumbnail support
### Thumbnail Support
- New configuration option `thumbnail` in `[server]` to enable or disable generating image thumbnails
- New configuration options in `[thumbnails]` to enable or disable generating image thumbnails, set the thumbnail size, and configure concurrency for thumbnail generation.
---
@ -96,6 +115,8 @@ The `precaching` feature allows the server to pre-cache storage paths for faster
enabled = true
directory = "/mnt/hmac-storage/thumbnails/"
size = "200x200"
concurrency = 5
thumbnailintervalscan = "24h"
```
---
@ -118,6 +139,7 @@ AutoAdjustWorkers = true # Enable auto-adjustment for worker scaling
NetworkEvents = false # Disable logging and tracking of network-related events
PIDFilePath = "./hmac_file_server.pid" # Path to PID file
Precaching = true # Enable pre-caching of storage paths
TempPath = "/tmp/uploads" # Configurable temporary upload directory
[deduplication]
enabled = true
@ -133,6 +155,8 @@ Charset = "utf-8"
enabled = true
directory = "/mnt/nfs_vol01/hmac-file-server/thumbnails/"
size = "200x200"
concurrency = 5
thumbnailintervalscan = "24h"
[iso]
enabled = false
@ -182,6 +206,14 @@ UploadQueueSize = 5000
[file]
FileRevision = 1 # Revision number for file handling
[logging]
level = "info"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
```
---
@ -220,4 +252,67 @@ Use tools like **cURL** or **Postman** to test file uploads and downloads.
curl -X PUT -H "Authorization: Bearer <HMAC-TOKEN>" -F "file=@example.txt" http://localhost:8080/uploads/example.txt
```
Replace `<HMAC-TOKEN>` with a valid HMAC signature
Replace `<HMAC-TOKEN>` with a valid HMAC signature
````
## [2.3.0] - 2024-12-28
### Changed
- **Server:** Replaced the hardcoded temporary upload directory `/tmp/uploads` with a configurable `TempPath` parameter in `config.toml`. Ensure to set `tempPath` in your configuration file accordingly.
#### Version 2.2.1 - 2024-12-27
- **Enhancements:**
- Added detailed logging for file extensions and MIME types during file uploads to assist in diagnosing unsupported file type issues.
- **Configuration:**
- Updated `config.toml` to ensure necessary file extensions are allowed for uploads.
#### Version 2.2.2 - 2024-12-27
- **Bug Fixes:**
- Resolved issue where temporary `.tmp` files caused "Unsupported file type" warnings by adjusting MIME type detection to use the final file extension.
- **Enhancements:**
- Improved logging for file extension and MIME type during uploads.
### Summary of Changes
1. **Added Log Message Limitation:**
- Implemented `logMessages = logMessages[1:]` to remove the oldest log message once the `logMessages` slice exceeds 1000 entries, preventing unbounded memory growth.
---
### Additional Files
2024-12-30 endpoint /thumbnails
No changes are required for the other files (`RELEASE-NOTES.MD`, `README.MD`, `config.toml`).
## [2.3.1] - 2025-01-15
- Added metrics for deduplication and ISO container operations to improve observability.
- Additional detail: Thumbnails now support a "concurrency" configuration parameter for parallel image processing.
## [2.4.0] - 2025-02-20
### Added
- **Pre-Caching Support:** Introduced pre-caching of storage paths to improve access speeds.
- **ISO Container Management:** Added functionality to create and mount ISO containers for specialized storage needs.
- **Thumbnail Concurrency Parameter:** Users can now set the level of concurrency for thumbnail generation to optimize performance.
### Changed
- **Configuration Options:** Updated `config.toml` to include new settings for pre-caching and ISO management.
- **Documentation:** Enhanced `README.MD` with detailed instructions on new features and best practices.
### Fixed
- **Bug Fixes:** Resolved minor issues related to file versioning and deduplication processes.
### Summary of Changes
1. **Pre-Caching:** Improved performance by allowing the server to pre-cache frequently accessed storage paths.
2. **ISO Management:** Enabled the creation and mounting of ISO containers for better filesystem management.
3. **Thumbnail Generation:** Added a concurrency setting to allow parallel processing of image thumbnails, speeding up the generation process.
## [2.4.1] - 2025-03-10
### Changed
- **Configuration:** Updated `globalextensions` in `config.toml` to `["*"]`, allowing all file types globally for uploads. This change simplifies the configuration by removing the need to specify individual file extensions.

86
cmd/server/config.toml
View File

@ -1,43 +1,52 @@
[server]
bind_ip = "127.0.0.1"
listenport = "8080"
unixsocket = false
storagepath = "/"
loglevel = "debug"
logfile = "./tmp/hmac-file-server.log"
storagepath = "/mnt/nfs_vol01/hmac-file-server/data/"
metricsenabled = true
metricsport = "8081"
filettl = "180d"
minfreebytes = "2GB"
metricsport = "9090"
deduplicationenabled = true
minfreebytes = "5GB"
filettl = "2y"
filettlenabled = true
autoadjustworkers = true
networkevents = false
temppath = "./tmp"
loggingjson = false
pidfilepath = "hmac_file_server.pid"
cleanuponexit = true
precaching = false
pidfilepath = "./hmac-file-server.pid"
precaching = true
#globalextensions = ["*"]
[deduplication]
enabled = false
directory = "./deduplication"
enabled = true
directory = "/mnt/nfs_vol01/hmac-file-server/deduplication/"
[logging]
level = "debug"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
[thumbnails]
enabled = false
directory = "./thumbnails"
enabled = true
directory = "/mnt/nfs_vol01/hmac-file-server/thumbnails/"
size = "200x200"
thumbnailintervalscan = "1h"
concurrency = 5
[iso]
enabled = false
size = "1TB"
mountpoint = "./iso"
mountpoint = "/mnt/nfs_vol01/hmac-file-server/iso/"
charset = "utf-8"
[timeouts]
readtimeout = "1800s"
writetimeout = "1800s"
idletimeout = "1800s"
readtimeout = "3600s"
writetimeout = "3600s"
idletimeout = "3600s"
[security]
secret = "a-orc-and-a-humans-is-drinking-ale"
secret = "hmac-file-server-is-the-win"
[versioning]
enableversioning = false
@ -45,31 +54,44 @@ maxversions = 1
[uploads]
resumableuploadsenabled = false
chunkeduploadsenabled = false
chunksize = "64MB"
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm", ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg", ".zip", ".rar"]
chunkeduploadsenabled = true
chunksize = "32MB"
allowedextensions = [
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
]
[downloads]
resumabledownloadsenabled = false
chunkeddownloadsenabled = false
chunksize = "64MB"
chunksize = "32MB"
allowedextensions = [
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
]
[clamav]
clamavenabled = false
clamavenabled = true
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 4
scanfileextensions = [".exe", ".dll", ".bin", ".com", ".bat", ".sh", ".php", ".js"]
scanfileextensions = [
".exe", ".dll", ".bin", ".com", ".bat",
".sh", ".php", ".js"
]
[redis]
redisenabled = false
redisdbindex = 1
redisenabled = true
redisdbindex = 0
redisaddr = "localhost:6379"
redispassword = ""
redishealthcheckinterval = "60s"
redishealthcheckinterval = "120s"
[workers]
numworkers = 4
uploadqueuesize = 1000
uploadqueuesize = 5000
[file]
filerevision = 1
filerevision = 1

1512
cmd/server/hmac-file-server.log
View File

File diff suppressed because it is too large Load Diff

1118
cmd/server/main.go
View File

File diff suppressed because it is too large Load Diff

105
config.toml Normal file
View File

@ -0,0 +1,105 @@
[server]
listenport = "8080"
unixsocket = false
storagepath = "./uploads"
metricsenabled = true
metricsport = "9090"
filettl = "8760h"
minfreebytes = "100MB"
autoadjustworkers = true
networkevents = true
temppath = "/tmp/hmac-file-server"
loggingjson = false
pidfilepath = "/var/run/hmacfileserver.pid"
cleanuponexit = true
precaching = true
filettlenabled = true
deduplicationenabled = true
globalextensions = ["*"] # Allows all file types globally
bind_ip = "0.0.0.0" # New option: Specify the IP address to bind to (IPv4 or IPv6)
[logging]
level = "info"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
[deduplication]
enabled = false
directory = "./deduplication"
[thumbnails]
enabled = false
directory = "./thumbnails"
size = "200x200"
thumbnailintervalscan = "24h"
[iso]
enabled = false
size = "1GB"
mountpoint = "/mnt/iso"
charset = "utf-8"
containerfile = "/path/to/iso/container.iso"
[timeouts]
readtimeout = "4800s"
writetimeout = "4800s"
idletimeout = "4800s"
[security]
secret = "changeme"
[versioning]
enableversioning = false
maxversions = 1
[uploads]
resumableuploadsenabled = true
chunkeduploadsenabled = true
chunksize = "8192"
allowedextensions = [
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
]
[downloads]
resumabledownloadsenabled = true
chunkeddownloadsenabled = true
chunksize = "8192"
allowedextensions = [".jpg", ".png"] # Restricts downloads to specific types
[clamav]
clamavenabled = true
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 2
scanfileextensions = [".exe", ".dll", ".pdf"]
[redis]
redisenabled = true
redisaddr = "localhost:6379"
redispassword = ""
redisdbindex = 0
redishealthcheckinterval = "120s"
[workers]
numworkers = 4
uploadqueuesize = 50
max_concurrent_operations = 10
network_event_buffer = 100
performance_monitor_interval = "5m"
metrics_update_interval = "10s"
[file]
filerevision = 1
[precache]
redisEnabled = true
redisAddr = "localhost:6379"
staticIndexFile = "./static_index.json"
[build]
version = "v1.0.0"

624
dashboard/dashboard.json
View File

@ -39,12 +39,11 @@
"showLineNumbers": false,
"showMiniMap": false
},
"content": "<div style=\"text-align: center; background-color: #111217; padding: 20px;\">\n <h3 style=\"color: white; font-family: 'Arial', sans-serif; font-weight: bold;\">HMAC Dashboard</h3>\n <img src=\"https://block.uuxo.net/hmac_icon.png\" alt=\"HMAC Icon\" style=\"width: 50px; height: 50px; display: block; margin: 10px auto;\">\n <p style=\"font-family: 'Verdana', sans-serif; color: white;\">\n This dashboard monitors <strong style=\"color: #FF5733;\">key metrics</strong> for the \n <span style=\"font-style: italic; color: #007BFF;\">HMAC File Server</span>.\n </p>\n</div>\n",
"content": "<div style=\"text-align: center; background-color: transparent; padding: 20px;\">\n <h3 style=\"color: white; font-family: 'Arial', sans-serif; font-weight: bold;\">HMAC Dashboard</h3>\n <img src=\"https://block.uuxo.net/hmac_icon.png\" alt=\"HMAC Icon\" style=\"width: 50px; height: 50px; display: block; margin: 10px auto;\">\n <p style=\"font-family: 'Verdana', sans-serif; color: white;\">\n This dashboard monitors <strong style=\"color: #FF5733;\">key metrics</strong> for the \n <span style=\"font-style: italic; color: #007BFF;\">HMAC File Server</span>.\n </p>\n</div>\n",
"mode": "html"
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"title": "HMAC Dashboard",
"transparent": true,
"type": "text"
},
{
@ -106,7 +105,7 @@
"sizing": "auto",
"valueMode": "color"
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
@ -165,7 +164,7 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
@ -222,7 +221,7 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
@ -285,7 +284,7 @@
"textMode": "value",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
@ -348,7 +347,7 @@
"textMode": "value",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
@ -361,77 +360,6 @@
"title": "HMAC Downloads",
"type": "stat"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 3,
"x": 0,
"y": 11
},
"id": 15,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "bduehd5vqv1moa"
},
"editorMode": "code",
"expr": "hmac_file_server_upload_duration_seconds_sum + hmac_file_server_download_duration_seconds_sum",
"hide": false,
"instant": false,
"legendFormat": "__auto",
"range": true,
"refId": "B"
}
],
"title": "HMAC Up/Down Duration",
"type": "stat"
},
{
"datasource": {
"default": true,
@ -458,8 +386,8 @@
},
"gridPos": {
"h": 5,
"w": 3,
"x": 3,
"w": 6,
"x": 0,
"y": 11
},
"id": 10,
@ -480,7 +408,7 @@
"textMode": "value",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
@ -494,130 +422,6 @@
"title": "HMAC GO Threads",
"type": "stat"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 3,
"x": 6,
"y": 11
},
"id": 21,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "value",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"targets": [
{
"editorMode": "code",
"expr": "hmac_file_deletions_total",
"format": "table",
"legendFormat": "{{hmac-file-server}}",
"range": true,
"refId": "A"
}
],
"title": "HMAC FileTTL Deletion(s)",
"type": "stat"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 3,
"x": 9,
"y": 11
},
"id": 20,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "value",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"targets": [
{
"editorMode": "code",
"expr": "hmac_cache_misses_total",
"format": "table",
"legendFormat": "{{hmac-file-server}}",
"range": true,
"refId": "A"
}
],
"title": "HMAC Cache Misses",
"type": "stat"
},
{
"datasource": {
"default": true,
@ -647,8 +451,8 @@
},
"gridPos": {
"h": 5,
"w": 3,
"x": 12,
"w": 6,
"x": 6,
"y": 11
},
"id": 16,
@ -669,16 +473,17 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
"exemplar": false,
"expr": "hmac_active_connections_total",
"format": "table",
"instant": false,
"format": "time_series",
"instant": true,
"interval": "",
"legendFormat": "__auto",
"range": true,
"range": false,
"refId": "A"
}
],
@ -710,8 +515,8 @@
},
"gridPos": {
"h": 5,
"w": 3,
"x": 15,
"w": 6,
"x": 12,
"y": 11
},
"id": 19,
@ -732,7 +537,7 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
@ -794,7 +599,7 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
@ -855,7 +660,7 @@
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.3.1",
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
@ -867,6 +672,385 @@
],
"title": "HMAC Download Errors",
"type": "stat"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 5,
"x": 0,
"y": 16
},
"id": 22,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
"expr": "hmac_thumbnails_processed_total",
"legendFormat": "Upload Errors",
"range": true,
"refId": "A"
}
],
"title": "Thumbnails Processed",
"type": "stat"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 5,
"x": 5,
"y": 16
},
"id": 23,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
"expr": "hmac_thumbnail_processing_errors_total",
"legendFormat": "Upload Errors",
"range": true,
"refId": "A"
}
],
"title": "Thumbnails Processed Errors",
"type": "stat"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 5,
"x": 10,
"y": 16
},
"id": 24,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
"expr": "hmac_thumbnail_processing_time_seconds_count",
"legendFormat": "Upload Errors",
"range": true,
"refId": "A"
}
],
"title": "Thumbnails Process Time",
"type": "stat"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 3,
"x": 15,
"y": 16
},
"id": 15,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.4.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "bduehd5vqv1moa"
},
"editorMode": "code",
"expr": "hmac_file_server_upload_duration_seconds_sum + hmac_file_server_download_duration_seconds_sum",
"hide": false,
"instant": false,
"legendFormat": "__auto",
"range": true,
"refId": "B"
}
],
"title": "HMAC Up/Down Duration",
"type": "stat"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 3,
"x": 18,
"y": 16
},
"id": 20,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "value",
"wideLayout": true
},
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
"expr": "hmac_cache_misses_total",
"format": "table",
"legendFormat": "{{hmac-file-server}}",
"range": true,
"refId": "A"
}
],
"title": "HMAC Cache Misses",
"type": "stat"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 3,
"x": 21,
"y": 16
},
"id": 21,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "value",
"wideLayout": true
},
"pluginVersion": "11.4.0",
"targets": [
{
"editorMode": "code",
"expr": "hm",
"format": "table",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "HMAC FileTTL Deletion(s)",
"type": "stat"
}
],
"preload": false,
@ -876,13 +1060,13 @@
"list": []
},
"time": {
"from": "now-5m",
"from": "now-24h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "HMAC File Server Metrics",
"uid": "de0ye5t0hzq4ge",
"version": 129,
"version": 137,
"weekStart": ""
}
}

3
go.mod
View File

@ -15,7 +15,7 @@ require (
require (
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/fsnotify/fsnotify v1.7.0
github.com/gdamore/encoding v1.0.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
@ -53,7 +53,6 @@ require (
github.com/prometheus/common v0.61.0
github.com/prometheus/procfs v0.15.1 // indirect
github.com/rivo/tview v0.0.0-20241103174730-c76f7879f592
github.com/robfig/cron/v3 v3.0.1
github.com/tklauser/go-sysconf v0.3.14 // indirect
github.com/tklauser/numcpus v0.9.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect

2
go.sum
View File

@ -79,8 +79,6 @@ github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJ
github.com/rivo/uniseg v0.4.3/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/shirou/gopsutil v3.21.11+incompatible h1:+1+c1VGhc88SSonWP6foOcLhvnKlUeu/erjjvaPEYiI=

2
test/hmac_test.go
View File

@ -16,7 +16,7 @@ import (
const (
serverURL = "http://[::1]:8080" // Replace with your actual server URL
secret = "a-orc-and-a-humans-is-drinking-ale" // Replace with your HMAC secret key
secret = "changeme" // Replace with your HMAC secret key
uploadPath = "hmac_icon.png" // Test file to upload
protocolType = "v2" // Use v2, v, or token as needed
)