Compare commits
10 Commits
2.8-stable
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 56de6107b6 | |||
| 6a098d35d6 | |||
| 49f4338a6e | |||
| 1c6477531c | |||
| 23ee7e5d33 | |||
| 2c18b1fccb | |||
| 2769fd1a98 | |||
| cc1a629aff | |||
| a0a117dc11 | |||
| ccf04ede06 |
10
.drone.yml
Normal file
10
.drone.yml
Normal file
@ -0,0 +1,10 @@
|
||||
kind: pipeline
|
||||
name: build
|
||||
|
||||
steps:
|
||||
- name: build docker image
|
||||
image: docker
|
||||
commands:
|
||||
- docker build -t my-hmac-file-server .
|
||||
- docker run -d -p 8080:8080 my-hmac-file-server
|
||||
- docker ps # Optional: To verify the container is running
|
||||
18
CHANGELOG.md
18
CHANGELOG.md
@ -8,6 +8,24 @@ All notable changes to this project will be documented in this file.
|
||||
|
||||
---
|
||||
|
||||
## [2.8-Stable] - 2026-05-01
|
||||
|
||||
### Added
|
||||
|
||||
- Version check history for improved tracking.
|
||||
- Enhanced ClamAV scanning with concurrent workers.
|
||||
|
||||
### Changed
|
||||
|
||||
- Improved ISO-based storage for specialized use cases.
|
||||
- Auto-scaling workers for optimized performance.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Minor issues in worker thread adjustments under high load.
|
||||
|
||||
---
|
||||
|
||||
## [2.7] - 2026-02-10
|
||||
|
||||
### Added
|
||||
|
||||
@ -1,5 +1,4 @@
|
||||
# HMAC File Server 2.6-Stable
|
||||
|
||||
# HMAC File Server 2.9-Stable
|
||||
## Overview
|
||||
The **HMAC File Server** ensures secure file uploads and downloads using HMAC authentication. It incorporates rate limiting, CORS support, retries, file versioning, and Unix socket support for enhanced flexibility. Redis integration provides efficient caching and session management. Prometheus metrics and a graceful shutdown mechanism ensure reliable and efficient file handling.
|
||||
|
||||
|
||||
@ -1,6 +1,7 @@
|
||||
# Short Release Note
|
||||
|
||||
Key Highlights from 2.7-Stable:
|
||||
Key Highlights from 2.8-Stable:
|
||||
- Version check history added for improved tracking.
|
||||
- Improved ISO-based storage for specialized use cases.
|
||||
- Enhanced ClamAV scanning with concurrent workers.
|
||||
- Auto-scaling workers for optimized performance.
|
||||
|
||||
892
WiKi.md
892
WiKi.md
@ -1,892 +0,0 @@
|
||||
This documentation provides detailed information on configuring, setting up, and maintaining the HMAC File Server. Whether you're a developer, system administrator, or an enthusiast, this guide will help you navigate through the server's features and configurations effectively.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Introduction](#introduction)
|
||||
2. [Configuration](#configuration)
|
||||
- [Server Configuration](#server-configuration)
|
||||
- [Deduplication Settings](#deduplication-settings)
|
||||
- [Thumbnails Settings](#thumbnails-settings)
|
||||
- [ISO Settings](#iso-settings)
|
||||
- [Timeout Settings](#timeout-settings)
|
||||
- [Security Settings](#security-settings)
|
||||
- [Versioning Settings](#versioning-settings)
|
||||
- [Uploads Settings](#uploads-settings)
|
||||
- [Downloads Settings](#downloads-settings)
|
||||
- [ClamAV Settings](#clamav-settings)
|
||||
- [Redis Settings](#redis-settings)
|
||||
- [Worker Settings](#worker-settings)
|
||||
- [File Settings](#file-settings)
|
||||
3. [Example Configuration](#example-configuration)
|
||||
4. [Setup Instructions](#setup-instructions)
|
||||
- [1. HMAC File Server Installation](#1-hmac-file-server-installation)
|
||||
- [2. Reverse Proxy Configuration](#2-reverse-proxy-configuration)
|
||||
- [Apache2 Reverse Proxy](#apache2-reverse-proxy)
|
||||
- [Nginx Reverse Proxy](#nginx-reverse-proxy)
|
||||
- [3. ejabberd Configuration](#3-ejabberd-configuration)
|
||||
- [4. Systemd Service Setup](#4-systemd-service-setup)
|
||||
5. [Building for Different Architectures](#building-for-different-architectures)
|
||||
6. [Additional Recommendations](#additional-recommendations)
|
||||
7. [Notes](#notes)
|
||||
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
The **HMAC File Server** is a secure and efficient file management solution designed to handle file uploads, downloads, deduplication, and more. Built with a focus on security, scalability, and performance, it integrates seamlessly with various tools and services to provide a comprehensive file handling experience.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
The HMAC File Server is configured using a `config.toml` file. Below are the detailed explanations of each configuration section and their respective options.
|
||||
|
||||
### Server Configuration
|
||||
|
||||
```toml
|
||||
# Server configuration
|
||||
listenport = "8080" # TCP port for incoming requests
|
||||
unixsocket = false # Use Unix domain socket instead of TCP
|
||||
storagepath = "/path/to/hmac-file-server/data/" # Directory to store uploaded files
|
||||
loglevel = "debug" # Logging level: "debug", "info", "warn", "error"
|
||||
logfile = "/path/to/hmac-file-server.log" # Path to log file; leave empty to use stdout
|
||||
metricsenabled = true # Enable Prometheus metrics
|
||||
metricsport = "9090" # Port for Prometheus metrics
|
||||
deduplicationenabled = true
|
||||
minfreebytes = "5GB" # Minimum free disk space required
|
||||
filettl = "2Y" # Time-to-live for files
|
||||
filettlenabled = false # Enable TTL checks and cleanup
|
||||
autoadjustworkers = true # Automatically adjust worker threads based on load
|
||||
networkevents = false # Enable detailed network event logging
|
||||
pidfilepath = "./hmac-file-server.pid" # Path to PID file
|
||||
precaching = true # Pre-cache file structures on startup
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **listenport**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the TCP port on which the server listens for incoming requests.
|
||||
- *Default*: `"8080"`
|
||||
|
||||
- **unixsocket**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Determines whether to use a Unix domain socket instead of a TCP port for communication.
|
||||
- *Default*: `false`
|
||||
|
||||
- **storagepath**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the directory path where uploaded files are stored. Ensure this path exists and has appropriate permissions.
|
||||
- *Default*: `"/path/to/hmac-file-server/data/"`
|
||||
|
||||
- **loglevel**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the verbosity level of logs.
|
||||
- *Options*: `"debug"`, `"info"`, `"warn"`, `"error"`
|
||||
- *Default*: `"debug"`
|
||||
|
||||
- **logfile**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the file path for logging. If left empty, logs are output to `stdout`.
|
||||
- *Default*: `"/path/to/hmac-file-server.log"`
|
||||
|
||||
- **metricsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables the Prometheus metrics endpoint.
|
||||
- *Default*: `true`
|
||||
|
||||
- **metricsport**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the port on which Prometheus metrics are exposed.
|
||||
- *Default*: `"9090"`
|
||||
|
||||
- **deduplicationenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables file deduplication to optimize storage usage.
|
||||
- *Default*: `true`
|
||||
|
||||
- **minfreebytes**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the minimum free disk space required for the server to operate effectively.
|
||||
- *Default*: `"5GB"`
|
||||
|
||||
- **filettl**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the default Time-to-Live (TTL) for files, determining how long files are retained before deletion.
|
||||
- *Format*: Duration (e.g., `"2Y"` for two years)
|
||||
- *Default*: `"2Y"`
|
||||
|
||||
- **filettlenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables TTL checks and automatic file cleanup based on the `filettl` value.
|
||||
- *Default*: `false`
|
||||
|
||||
- **autoadjustworkers**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Automatically adjusts the number of worker threads based on server load and system resources.
|
||||
- *Default*: `true`
|
||||
|
||||
- **networkevents**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables detailed logging of network events, which can be useful for debugging but may increase log verbosity.
|
||||
- *Default*: `false`
|
||||
|
||||
- **pidfilepath**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the file path where the server writes its Process ID (PID) file. This is useful for managing the server process.
|
||||
- *Default*: `"./hmac-file-server.pid"`
|
||||
|
||||
- **precaching**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables pre-caching of file structures on startup to improve access speed and performance.
|
||||
- *Default*: `true`
|
||||
|
||||
---
|
||||
|
||||
### Deduplication Settings
|
||||
|
||||
```toml
|
||||
# Deduplication settings
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/deduplication/" # Path to deduplication metadata store
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **enabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables the deduplication feature, which helps in eliminating duplicate files to save storage space.
|
||||
- *Default*: `true`
|
||||
|
||||
- **directory**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the directory path where deduplication metadata is stored. Ensure this directory exists and has appropriate permissions.
|
||||
- *Default*: `"/path/to/hmac-file-server/deduplication/"`
|
||||
|
||||
---
|
||||
|
||||
### Thumbnails Settings
|
||||
|
||||
```toml
|
||||
# Thumbnails settings
|
||||
[thumbnails]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/thumbnails/" # Directory for storing thumbnails
|
||||
size = "200x200" # Thumbnail dimensions
|
||||
thumbnailintervalscan = "1h" # Interval for scheduled thumbnail generation
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **enabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables the automatic generation of thumbnails for uploaded images.
|
||||
- *Default*: `true`
|
||||
|
||||
- **directory**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the directory where generated thumbnails are stored. Ensure this directory exists and has appropriate permissions.
|
||||
- *Default*: `"/path/to/hmac-file-server/thumbnails/"`
|
||||
|
||||
- **size**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the dimensions to which thumbnails are resized. The format should be `"widthxheight"` (e.g., `"200x200"`).
|
||||
- *Default*: `"200x200"`
|
||||
|
||||
- **thumbnailintervalscan**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the frequency at which the server scans for new or updated images to generate thumbnails.
|
||||
- *Format*: Duration (e.g., `"1h"` for one hour)
|
||||
- *Default*: `"1h"`
|
||||
|
||||
---
|
||||
|
||||
### ISO Settings
|
||||
|
||||
```toml
|
||||
# ISO settings
|
||||
[iso]
|
||||
enabled = false
|
||||
size = "1TB" # Maximum ISO size
|
||||
mountpoint = "/path/to/hmac-file-server/iso/" # ISO mount point
|
||||
charset = "utf-8" # Filesystem character set encoding
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **enabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables the mounting of an ISO-based filesystem for specialized storage needs.
|
||||
- *Default*: `false`
|
||||
|
||||
- **size**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the maximum allowed size for the ISO container.
|
||||
- *Default*: `"1TB"`
|
||||
|
||||
- **mountpoint**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the directory path where the ISO is mounted. Ensure this path exists and has appropriate permissions.
|
||||
- *Default*: `"/path/to/hmac-file-server/iso/"`
|
||||
|
||||
- **charset**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the filesystem character set encoding for the ISO.
|
||||
- *Default*: `"utf-8"`
|
||||
|
||||
> **Note**: Ensure only one `[iso]` block is active in your `config.toml` to avoid configuration conflicts.
|
||||
|
||||
---
|
||||
|
||||
### Timeout Settings
|
||||
|
||||
```toml
|
||||
# Timeout settings
|
||||
[timeouts]
|
||||
readtimeout = "3600s" # Maximum time to read a request
|
||||
writetimeout = "3600s" # Maximum time to write a response
|
||||
idletimeout = "3600s" # Maximum keep-alive time for idle connections
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **readtimeout**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the maximum duration for reading the entire request, including the body.
|
||||
- *Format*: Duration (e.g., `"3600s"` for one hour)
|
||||
- *Default*: `"3600s"`
|
||||
|
||||
- **writetimeout**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the maximum duration before timing out writes of the response.
|
||||
- *Format*: Duration (e.g., `"3600s"` for one hour)
|
||||
- *Default*: `"3600s"`
|
||||
|
||||
- **idletimeout**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the maximum amount of time to wait for the next request when keep-alives are enabled.
|
||||
- *Format*: Duration (e.g., `"3600s"` for one hour)
|
||||
- *Default*: `"3600s"`
|
||||
|
||||
---
|
||||
|
||||
### Security Settings
|
||||
|
||||
```toml
|
||||
# Security settings
|
||||
[security]
|
||||
secret = "your-secure-secret-key" # HMAC shared secret key (change to a secure value)
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **secret**:
|
||||
- *Type*: `String`
|
||||
- *Description*: The HMAC shared secret key used for signing requests and operations.
|
||||
- *Default*: `"your-secure-secret-key"`
|
||||
- *Warning*: **Change this immediately** to a unique, strong string in production environments to ensure the security of HMAC operations.
|
||||
|
||||
---
|
||||
|
||||
### Versioning Settings
|
||||
|
||||
```toml
|
||||
# Versioning settings
|
||||
[versioning]
|
||||
enableversioning = false
|
||||
maxversions = 1 # Number of file versions to retain
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **enableversioning**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables the versioning feature, which maintains multiple versions of the same file.
|
||||
- *Default*: `false`
|
||||
|
||||
- **maxversions**:
|
||||
- *Type*: `Integer`
|
||||
- *Description*: Specifies the maximum number of versions to retain for each file.
|
||||
- *Default*: `1`
|
||||
|
||||
---
|
||||
|
||||
### Uploads Settings
|
||||
|
||||
```toml
|
||||
# Upload settings
|
||||
[uploads]
|
||||
resumableuploadsenabled = false
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "32MB" # Chunk size for uploads
|
||||
allowedextensions = [
|
||||
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
|
||||
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
|
||||
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
|
||||
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
|
||||
]
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **resumableuploadsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables support for resumable (chunked) file uploads.
|
||||
- *Default*: `false`
|
||||
|
||||
- **chunkeduploadsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Specifically enables or disables chunked uploads.
|
||||
- *Default*: `true`
|
||||
|
||||
- **chunksize**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the size of each chunk in chunked uploads.
|
||||
- *Format*: Size (e.g., `"32MB"`)
|
||||
- *Default*: `"32MB"`
|
||||
|
||||
- **allowedextensions**:
|
||||
- *Type*: `Array of Strings`
|
||||
- *Description*: Lists the file extensions permitted for upload.
|
||||
- *Default*:
|
||||
```toml
|
||||
allowedextensions = [
|
||||
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
|
||||
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
|
||||
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
|
||||
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Downloads Settings
|
||||
|
||||
```toml
|
||||
# Downloads settings
|
||||
[downloads]
|
||||
resumabledownloadsenabled = false
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "32MB"
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **resumabledownloadsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables support for resumable (chunked) downloads.
|
||||
- *Default*: `false`
|
||||
|
||||
- **chunkeddownloadsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Specifically enables or disables chunked downloads.
|
||||
- *Default*: `true`
|
||||
|
||||
- **chunksize**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the size of each chunk in chunked downloads.
|
||||
- *Format*: Size (e.g., `"32MB"`)
|
||||
- *Default*: `"32MB"`
|
||||
|
||||
> **Note**: The `allowedextensions` key is **not** part of the `[downloads]` configuration based on the provided code. Ensure that it is omitted to prevent configuration errors.
|
||||
|
||||
---
|
||||
|
||||
### ClamAV Settings
|
||||
|
||||
```toml
|
||||
# ClamAV settings
|
||||
[clamav]
|
||||
clamavenabled = true
|
||||
clamavsocket = "/path/to/clamav/clamd.ctl" # Path to ClamAV socket
|
||||
numscanworkers = 4 # Number of concurrent scan workers
|
||||
scanfileextensions = [
|
||||
".exe", ".dll", ".bin", ".com", ".bat",
|
||||
".sh", ".php", ".js"
|
||||
]
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **clamavenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables ClamAV integration for virus scanning of uploaded files.
|
||||
- *Default*: `true`
|
||||
|
||||
- **clamavsocket**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the file path to the ClamAV socket (`.ctl` file). Ensure ClamAV is installed and the socket path is correct.
|
||||
- *Default*: `"/path/to/clamav/clamd.ctl"`
|
||||
|
||||
- **numscanworkers**:
|
||||
- *Type*: `Integer`
|
||||
- *Description*: Sets the number of concurrent workers dedicated to scanning files with ClamAV.
|
||||
- *Default*: `4`
|
||||
|
||||
- **scanfileextensions**:
|
||||
- *Type*: `Array of Strings`
|
||||
- *Description*: Lists the file extensions that should be scanned for viruses.
|
||||
- *Default*:
|
||||
```toml
|
||||
scanfileextensions = [
|
||||
".exe", ".dll", ".bin", ".com", ".bat",
|
||||
".sh", ".php", ".js"
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Redis Settings
|
||||
|
||||
```toml
|
||||
# Redis settings
|
||||
[redis]
|
||||
redisenabled = true
|
||||
redisdbindex = 0
|
||||
redisaddr = "localhost:6379" # Redis server address
|
||||
redispassword = "" # Redis password if required
|
||||
redishealthcheckinterval = "120s" # Interval for Redis health checks
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **redisenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables Redis integration for caching or session tracking.
|
||||
- *Default*: `true`
|
||||
|
||||
- **redisaddr**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the address of the Redis server (e.g., `"localhost:6379"`).
|
||||
- *Default*: `"localhost:6379"`
|
||||
|
||||
- **redispassword**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the Redis authentication password, if required.
|
||||
- *Default*: `""`
|
||||
|
||||
- **redisdbindex**:
|
||||
- *Type*: `Integer`
|
||||
- *Description*: Specifies the Redis database index to use.
|
||||
- *Default*: `0`
|
||||
|
||||
- **redishealthcheckinterval**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the interval for performing health checks on the Redis connection.
|
||||
- *Format*: Duration (e.g., `"120s"` for two minutes)
|
||||
- *Default*: `"120s"`
|
||||
|
||||
---
|
||||
|
||||
### Worker Settings
|
||||
|
||||
```toml
|
||||
# Worker settings
|
||||
[worker]
|
||||
numworkers = 10 # Number of worker threads
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **numworkers**:
|
||||
- *Type*: `Integer`
|
||||
- *Description*: Specifies the number of worker threads to handle file operations.
|
||||
- *Default*: `10`
|
||||
|
||||
---
|
||||
|
||||
### File Settings
|
||||
|
||||
```toml
|
||||
# File settings
|
||||
[file]
|
||||
maxfilesize = "10GB" # Maximum file size for uploads
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **maxfilesize**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the maximum allowed file size for uploads.
|
||||
- *Format*: Size (e.g., `"10GB"`)
|
||||
- *Default*: `"10GB"`
|
||||
|
||||
---
|
||||
|
||||
## Example Configuration
|
||||
|
||||
Below is an example `config.toml` file with default settings:
|
||||
|
||||
```toml
|
||||
# Example HMAC File Server configuration
|
||||
|
||||
# Server configuration
|
||||
listenport = "8080"
|
||||
unixsocket = false
|
||||
storagepath = "/path/to/hmac-file-server/data/"
|
||||
loglevel = "debug"
|
||||
logfile = "/path/to/hmac-file-server.log"
|
||||
metricsenabled = true
|
||||
metricsport = "9090"
|
||||
deduplicationenabled = true
|
||||
minfreebytes = "5GB"
|
||||
filettl = "2Y"
|
||||
filettlenabled = false
|
||||
autoadjustworkers = true
|
||||
networkevents = false
|
||||
pidfilepath = "./hmac-file-server.pid"
|
||||
precaching = true
|
||||
|
||||
# Deduplication settings
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/deduplication/"
|
||||
|
||||
# Thumbnails settings
|
||||
[thumbnails]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/thumbnails/"
|
||||
size = "200x200"
|
||||
thumbnailintervalscan = "1h"
|
||||
|
||||
# ISO settings
|
||||
[iso]
|
||||
enabled = false
|
||||
size = "1TB"
|
||||
mountpoint = "/path/to/hmac-file-server/iso/"
|
||||
charset = "utf-8"
|
||||
|
||||
# Timeout settings
|
||||
[timeouts]
|
||||
readtimeout = "3600s"
|
||||
writetimeout = "3600s"
|
||||
idletimeout = "3600s"
|
||||
|
||||
# Security settings
|
||||
[security]
|
||||
secret = "your-secure-secret-key"
|
||||
|
||||
# Versioning settings
|
||||
[versioning]
|
||||
enableversioning = false
|
||||
maxversions = 1
|
||||
|
||||
# Upload settings
|
||||
[uploads]
|
||||
resumableuploadsenabled = false
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "32MB"
|
||||
allowedextensions = [
|
||||
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
|
||||
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
|
||||
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
|
||||
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
|
||||
]
|
||||
|
||||
# Download settings
|
||||
[downloads]
|
||||
resumabledownloadsenabled = false
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "32MB"
|
||||
|
||||
# ClamAV settings
|
||||
[clamav]
|
||||
clamavenabled = true
|
||||
clamavsocket = "/path/to/clamav/clamd.ctl"
|
||||
numscanworkers = 4
|
||||
scanfileextensions = [
|
||||
".exe", ".dll", ".bin", ".com", ".bat",
|
||||
".sh", ".php", ".js"
|
||||
]
|
||||
|
||||
# Redis settings
|
||||
[redis]
|
||||
redisenabled = true
|
||||
redisdbindex = 0
|
||||
redisaddr = "localhost:6379"
|
||||
redispassword = ""
|
||||
redishealthcheckinterval = "120s"
|
||||
|
||||
# Worker settings
|
||||
[worker]
|
||||
numworkers = 10
|
||||
|
||||
# File settings
|
||||
[file]
|
||||
maxfilesize = "10GB"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
### 1. HMAC File Server Installation
|
||||
|
||||
To install the HMAC File Server, follow these steps:
|
||||
|
||||
1. Clone the repository:
|
||||
```sh
|
||||
git clone https://github.com/your-repo/hmac-file-server.git
|
||||
cd hmac-file-server
|
||||
```
|
||||
|
||||
2. Build the server:
|
||||
```sh
|
||||
go build -o hmac-file-server
|
||||
```
|
||||
|
||||
3. Create the necessary directories:
|
||||
```sh
|
||||
mkdir -p /path/to/hmac-file-server/data/
|
||||
mkdir -p /path/to/hmac-file-server/deduplication/
|
||||
mkdir -p /path/to/hmac-file-server/thumbnails/
|
||||
mkdir -p /path/to/hmac-file-server/iso/
|
||||
```
|
||||
|
||||
4. Copy the example configuration file:
|
||||
```sh
|
||||
cp config.example.toml config.toml
|
||||
```
|
||||
|
||||
5. Edit the `config.toml` file to match your environment and preferences.
|
||||
|
||||
6. Start the server:
|
||||
```sh
|
||||
./hmac-file-server -config config.toml
|
||||
```
|
||||
|
||||
### 2. Reverse Proxy Configuration
|
||||
|
||||
To set up a reverse proxy for the HMAC File Server, you can use either Apache2 or Nginx. Below are the configuration examples for both.
|
||||
|
||||
#### Apache2 Reverse Proxy
|
||||
|
||||
1. Enable the necessary Apache2 modules:
|
||||
```sh
|
||||
sudo a2enmod proxy
|
||||
sudo a2enmod proxy_http
|
||||
sudo a2enmod headers
|
||||
sudo a2enmod rewrite
|
||||
```
|
||||
|
||||
2. Create a new virtual host configuration file:
|
||||
```sh
|
||||
sudo nano /etc/apache2/sites-available/hmac-file-server.conf
|
||||
```
|
||||
|
||||
3. Add the following configuration to the file:
|
||||
```apache
|
||||
<VirtualHost *:80>
|
||||
ServerName your-domain.com
|
||||
|
||||
ProxyPreserveHost On
|
||||
ProxyPass / http://localhost:8080/
|
||||
ProxyPassReverse / http://localhost:8080/
|
||||
|
||||
<Location />
|
||||
Require all granted
|
||||
Header always set X-Content-Type-Options "nosniff"
|
||||
Header always set X-Frame-Options "DENY"
|
||||
Header always set X-XSS-Protection "1; mode=block"
|
||||
</Location>
|
||||
</VirtualHost>
|
||||
```
|
||||
|
||||
4. Enable the new site and restart Apache2:
|
||||
```sh
|
||||
sudo a2ensite hmac-file-server.conf
|
||||
sudo systemctl restart apache2
|
||||
```
|
||||
|
||||
#### Nginx Reverse Proxy
|
||||
|
||||
1. Install Nginx if not already installed:
|
||||
```sh
|
||||
sudo apt-get update
|
||||
sudo apt-get install nginx
|
||||
```
|
||||
|
||||
2. Create a new server block configuration file:
|
||||
```sh
|
||||
sudo nano /etc/nginx/sites-available/hmac-file-server
|
||||
```
|
||||
|
||||
3. Add the following configuration to the file:
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name your-domain.com;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:8080;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Content-Type-Options "nosniff";
|
||||
proxy_set_header X-Frame-Options "DENY";
|
||||
proxy_set_header X-XSS-Protection "1; mode=block";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. Enable the new site and restart Nginx:
|
||||
```sh
|
||||
sudo ln -s /etc/nginx/sites-available/hmac-file-server /etc/nginx/sites-enabled/
|
||||
sudo systemctl restart nginx
|
||||
```
|
||||
|
||||
### 3. ejabberd Configuration
|
||||
|
||||
To configure ejabberd for use with the HMAC File Server, follow these steps:
|
||||
|
||||
1. Install ejabberd:
|
||||
```sh
|
||||
sudo apt-get update
|
||||
sudo apt-get install ejabberd
|
||||
```
|
||||
|
||||
2. Edit the ejabberd configuration file:
|
||||
```sh
|
||||
sudo nano /etc/ejabberd/ejabberd.yml
|
||||
```
|
||||
|
||||
3. Add the following configuration to the file:
|
||||
```yaml
|
||||
hosts:
|
||||
- "your-domain.com"
|
||||
|
||||
listen:
|
||||
-
|
||||
port: 5222
|
||||
module: ejabberd_c2s
|
||||
certfile: "/etc/ejabberd/ejabberd.pem"
|
||||
starttls: true
|
||||
starttls_required: true
|
||||
protocol_options:
|
||||
- "no_sslv3"
|
||||
- "no_tlsv1"
|
||||
- "no_tlsv1_1"
|
||||
ciphers: "HIGH:!aNULL:!eNULL:!3DES:@STRENGTH"
|
||||
dhfile: "/etc/ejabberd/dhparams.pem"
|
||||
max_stanza_size: 65536
|
||||
shaper: c2s_shaper
|
||||
access: c2s
|
||||
|
||||
-
|
||||
port: 5269
|
||||
module: ejabberd_s2s_in
|
||||
certfile: "/etc/ejabberd/ejabberd.pem"
|
||||
starttls: true
|
||||
starttls_required: true
|
||||
protocol_options:
|
||||
- "no_sslv3"
|
||||
- "no_tlsv1"
|
||||
- "no_tlsv1_1"
|
||||
ciphers: "HIGH:!aNULL:!eNULL:!3DES:@STRENGTH"
|
||||
dhfile: "/etc/ejabberd/dhparams.pem"
|
||||
max_stanza_size: 131072
|
||||
shaper: s2s_shaper
|
||||
access: s2s
|
||||
|
||||
acl:
|
||||
local:
|
||||
user_regexp: ""
|
||||
|
||||
access_rules:
|
||||
local:
|
||||
allow: local
|
||||
```
|
||||
|
||||
4. Restart ejabberd:
|
||||
```sh
|
||||
sudo systemctl restart ejabberd
|
||||
```
|
||||
|
||||
### 4. Systemd Service Setup
|
||||
|
||||
To set up the HMAC File Server as a systemd service, follow these steps:
|
||||
|
||||
1. Create a new systemd service file:
|
||||
```sh
|
||||
sudo nano /etc/systemd/system/hmac-file-server.service
|
||||
```
|
||||
|
||||
2. Add the following configuration to the file:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=HMAC File Server
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
ExecStart=/path/to/hmac-file-server -config /path/to/config.toml
|
||||
WorkingDirectory=/path/to/hmac-file-server
|
||||
Restart=always
|
||||
User=www-data
|
||||
Group=www-data
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
3. Reload systemd and enable the service:
|
||||
```sh
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable hmac-file-server
|
||||
sudo systemctl start hmac-file-server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Building for Different Architectures
|
||||
|
||||
To build the HMAC File Server for different architectures, you can use the following commands:
|
||||
|
||||
### Building for Linux (x86_64)
|
||||
|
||||
```sh
|
||||
GOOS=linux GOARCH=amd64 go build -o hmac-file-server-linux-amd64
|
||||
```
|
||||
|
||||
### Building for macOS (x86_64)
|
||||
|
||||
```sh
|
||||
GOOS=darwin GOARCH=amd64 go build -o hmac-file-server-darwin-amd64
|
||||
```
|
||||
|
||||
### Building for Windows (x86_64)
|
||||
|
||||
```sh
|
||||
GOOS=windows GOARCH=amd64 go build -o hmac-file-server-windows-amd64.exe
|
||||
```
|
||||
|
||||
### Building for ARM (32-bit)
|
||||
|
||||
```sh
|
||||
GOOS=linux GOARCH=arm GOARM=7 go build -o hmac-file-server-linux-arm
|
||||
```
|
||||
|
||||
### Building for ARM (64-bit)
|
||||
|
||||
```sh
|
||||
GOOS=linux GOARCH=arm64 go build -o hmac-file-server-linux-arm64
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Additional Recommendations
|
||||
|
||||
- **Security**: Ensure that the `secret` key in the `config.toml` file is changed to a unique, strong value to secure HMAC operations.
|
||||
- **Backups**: Regularly back up the `config.toml` file and any important data stored by the HMAC File Server.
|
||||
- **Monitoring**: Use monitoring tools like Prometheus and Grafana to keep track of server performance and metrics.
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- The HMAC File Server is designed to be flexible and configurable. Adjust the settings in the `config.toml` file to match your specific requirements and environment.
|
||||
- For any issues or questions, refer to the project's GitHub repository and documentation.
|
||||
````
|
||||
@ -136,6 +136,7 @@ type ServerConfig struct {
|
||||
Logging LoggingConfig `mapstructure:"logging"`
|
||||
GlobalExtensions []string `mapstructure:"globalextensions"`
|
||||
FileNaming string `mapstructure:"filenaming"`
|
||||
ForceProtocol string `mapstructure:"forceprotocol"` // NEW field
|
||||
// Removed TempPath, LoggingJSON
|
||||
}
|
||||
|
||||
@ -381,23 +382,41 @@ func createAndMountISO(size, mountpoint, charset string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
var dialer = &net.Dialer{
|
||||
DualStack: true,
|
||||
Timeout: 5 * time.Second,
|
||||
KeepAlive: 30 * time.Second, // Added keep-alive for better network change handling
|
||||
func initializeNetworkProtocol(forceProtocol string) (*net.Dialer, error) {
|
||||
switch forceProtocol {
|
||||
case "ipv4":
|
||||
return &net.Dialer{
|
||||
Timeout: 5 * time.Second,
|
||||
DualStack: false,
|
||||
Control: func(network, address string, c syscall.RawConn) error {
|
||||
if network == "tcp6" {
|
||||
return fmt.Errorf("IPv6 is disabled by forceprotocol setting")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}, nil
|
||||
case "ipv6":
|
||||
return &net.Dialer{
|
||||
Timeout: 5 * time.Second,
|
||||
DualStack: false,
|
||||
Control: func(network, address string, c syscall.RawConn) error {
|
||||
if network == "tcp4" {
|
||||
return fmt.Errorf("IPv4 is disabled by forceprotocol setting")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}, nil
|
||||
case "auto":
|
||||
return &net.Dialer{
|
||||
Timeout: 5 * time.Second,
|
||||
DualStack: true,
|
||||
}, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("invalid forceprotocol value: %s", forceProtocol)
|
||||
}
|
||||
}
|
||||
|
||||
var dualStackClient = &http.Client{
|
||||
Transport: &http.Transport{
|
||||
DialContext: dialer.DialContext,
|
||||
ForceAttemptHTTP2: true, // Enforce HTTP/2
|
||||
IdleConnTimeout: 90 * time.Second, // Longer idle connections
|
||||
DisableKeepAlives: false, // Ensure keep-alives are enabled
|
||||
MaxIdleConns: 100,
|
||||
MaxIdleConnsPerHost: 10,
|
||||
// ...existing code...
|
||||
},
|
||||
}
|
||||
var dualStackClient *http.Client
|
||||
|
||||
func main() {
|
||||
setDefaults()
|
||||
@ -476,8 +495,9 @@ func main() {
|
||||
log.Infof("Server PreCaching: %v", conf.Server.PreCaching)
|
||||
log.Infof("Server FileTTLEnabled: %v", conf.Server.FileTTLEnabled)
|
||||
log.Infof("Server DeduplicationEnabled: %v", conf.Server.DeduplicationEnabled)
|
||||
log.Infof("Server BindIP: %s", conf.Server.BindIP) // Hinzugefügt: Logging für BindIP
|
||||
log.Infof("Server FileNaming: %s", conf.Server.FileNaming) // Added: Logging for FileNaming
|
||||
log.Infof("Server BindIP: %s", conf.Server.BindIP) // Hinzugefügt: Logging für BindIP
|
||||
log.Infof("Server FileNaming: %s", conf.Server.FileNaming) // Added: Logging for FileNaming
|
||||
log.Infof("Server ForceProtocol: %s", conf.Server.ForceProtocol) // Added: Logging for ForceProtocol
|
||||
|
||||
err = writePIDFile(conf.Server.PIDFilePath) // Write PID file after config is loaded
|
||||
if err != nil {
|
||||
@ -597,6 +617,17 @@ func main() {
|
||||
log.Fatalf("Invalid IdleTimeout: %v", err)
|
||||
}
|
||||
|
||||
// Initialize network protocol based on forceprotocol setting
|
||||
dialer, err := initializeNetworkProtocol(conf.Server.ForceProtocol)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to initialize network protocol: %v", err)
|
||||
}
|
||||
dualStackClient = &http.Client{
|
||||
Transport: &http.Transport{
|
||||
DialContext: dialer.DialContext,
|
||||
},
|
||||
}
|
||||
|
||||
server := &http.Server{
|
||||
Addr: conf.Server.BindIP + ":" + conf.Server.ListenPort, // Geändert: Nutzung von BindIP
|
||||
Handler: router,
|
||||
@ -641,6 +672,9 @@ func main() {
|
||||
log.Fatalf("Server failed: %v", err)
|
||||
}
|
||||
} else {
|
||||
if conf.Server.ListenPort == "0.0.0.0" {
|
||||
log.Info("Binding to 0.0.0.0. Any net/http logs you see are normal for this universal address.")
|
||||
}
|
||||
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||
log.Fatalf("Server failed: %v", err)
|
||||
}
|
||||
@ -672,6 +706,7 @@ deduplicationenabled = true
|
||||
globalextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
|
||||
# FileNaming options: "HMAC", "None"
|
||||
filenaming = "HMAC"
|
||||
forceprotocol = "auto"
|
||||
|
||||
[logging]
|
||||
level = "info"
|
||||
@ -737,7 +772,7 @@ uploadqueuesize = 50
|
||||
# Add file-specific configurations here
|
||||
|
||||
[build]
|
||||
version = "2.8-Stable"
|
||||
version = "2.9-Stable"
|
||||
`)
|
||||
}
|
||||
|
||||
@ -831,6 +866,7 @@ func setDefaults() {
|
||||
viper.SetDefault("server.loggingjson", false)
|
||||
viper.SetDefault("server.filettlenabled", true)
|
||||
viper.SetDefault("server.deduplicationenabled", true)
|
||||
viper.SetDefault("server.forceprotocol", "auto")
|
||||
|
||||
viper.SetDefault("timeouts.readtimeout", "4800s")
|
||||
viper.SetDefault("timeouts.writetimeout", "4800s")
|
||||
@ -1185,6 +1221,9 @@ func logSystemInfo() {
|
||||
log.Infof("Platform Family: %s", hInfo.PlatformFamily)
|
||||
log.Infof("Platform Version: %s", hInfo.PlatformVersion)
|
||||
log.Infof("Kernel Version: %s", hInfo.KernelVersion)
|
||||
|
||||
// Log the forceprotocol configuration
|
||||
log.Infof("Force Protocol: %s", conf.Server.ForceProtocol)
|
||||
}
|
||||
|
||||
func initMetrics() {
|
||||
@ -1704,7 +1743,26 @@ func handleRequest(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
log.WithFields(logrus.Fields{"method": r.Method, "url": r.URL.String(), "remote": clientIP}).Info("Incoming request")
|
||||
|
||||
// Log the requested URL for debugging
|
||||
log.Infof("handleRequest: Received URL path: %s", r.URL.String())
|
||||
|
||||
p := r.URL.Path
|
||||
fileStorePath := strings.TrimPrefix(p, "/")
|
||||
if fileStorePath == "" || fileStorePath == "/" {
|
||||
log.WithField("path", fileStorePath).Warn("No file specified in URL")
|
||||
// Updated to return 404 with a clear message instead of forbidden.
|
||||
http.Error(w, "File not specified in URL. Please include the file path after the host.", http.StatusNotFound)
|
||||
flushLogMessages()
|
||||
return
|
||||
}
|
||||
// NEW: Compute absolute file path from storage path and fileStorePath.
|
||||
absFilename, err := sanitizeFilePath(conf.Server.StoragePath, fileStorePath)
|
||||
if err != nil {
|
||||
log.WithError(err).Warn("Invalid file path")
|
||||
http.Error(w, "Invalid file path", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
a, err := url.ParseQuery(r.URL.RawQuery)
|
||||
if err != nil {
|
||||
log.Warn("Failed to parse query parameters")
|
||||
@ -1712,26 +1770,6 @@ func handleRequest(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
fileStorePath := strings.TrimPrefix(p, "/")
|
||||
if fileStorePath == "" || fileStorePath == "/" {
|
||||
log.WithFields(logrus.Fields{
|
||||
"event": "AccessAttempt",
|
||||
"severity": "warning",
|
||||
}).Warn("Access to root directory is forbidden")
|
||||
http.Error(w, "Forbidden", http.StatusForbidden)
|
||||
flushLogMessages()
|
||||
return
|
||||
} else if fileStorePath[0] == '/' {
|
||||
fileStorePath = fileStorePath[1:]
|
||||
}
|
||||
|
||||
absFilename, err := sanitizeFilePath(conf.Server.StoragePath, fileStorePath)
|
||||
if err != nil {
|
||||
log.WithFields(logrus.Fields{"file": fileStorePath, "error": err}).Warn("Invalid file path")
|
||||
http.Error(w, "Invalid file path", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
switch r.Method {
|
||||
case http.MethodPut:
|
||||
handleUpload(w, r, absFilename, fileStorePath, a)
|
||||
@ -1911,6 +1949,21 @@ func handleUpload(w http.ResponseWriter, r *http.Request, absFilename, fileStore
|
||||
logMessages = append(logMessages, fmt.Sprintf("Processing completed successfully for %s", finalFilename))
|
||||
uploadsTotal.Inc()
|
||||
|
||||
// Instead of ignoring Callback-URL, send an asynchronous HTTP POST.
|
||||
callbackURL := r.Header.Get("Callback-URL")
|
||||
if callbackURL != "" {
|
||||
go func(url, filename string) {
|
||||
payload := fmt.Sprintf(`{"file": "%s"}`, filename)
|
||||
resp, err := http.Post(url, "application/json", strings.NewReader(payload))
|
||||
if err != nil {
|
||||
log.Warnf("Failed callback to %s: %v", url, err)
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
log.Infof("Callback to %s succeeded with status %s", url, resp.Status)
|
||||
}(callbackURL, finalFilename)
|
||||
}
|
||||
|
||||
// Log all messages at once
|
||||
for _, msg := range logMessages {
|
||||
log.Info(msg)
|
||||
@ -2396,7 +2449,7 @@ func DeduplicateFiles(storeDir string) error {
|
||||
err := filepath.Walk(storeDir, func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
logrus.WithError(err).Errorf("Error accessing path %s", path)
|
||||
return nil
|
||||
return nil // Continue walking
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return nil
|
||||
|
||||
24
config.toml
24
config.toml
@ -1,5 +1,19 @@
|
||||
[server]
|
||||
// ...existing code...
|
||||
# FileNaming options: "HMAC", "None"
|
||||
FileNaming = "HMAC"
|
||||
// ...existing code...
|
||||
# Server configuration
|
||||
listenport = "8080" # TCP port for incoming requests
|
||||
unixsocket = false # Use Unix domain socket instead of TCP
|
||||
storagepath = "/path/to/hmac-file-server/data/" # Directory to store uploaded files
|
||||
loglevel = "debug" # Logging level: "debug", "info", "warn", "error"
|
||||
logfile = "/path/to/hmac-file-server.log" # Path to log file; leave empty to use stdout
|
||||
metricsenabled = true # Enable Prometheus metrics
|
||||
metricsport = "9090" # Port for Prometheus metrics
|
||||
deduplicationenabled = true
|
||||
minfreebytes = "5GB" # Minimum free disk space required
|
||||
filettl = "2Y" # Time-to-live for files
|
||||
filettlenabled = false # Enable TTL checks and cleanup
|
||||
autoadjustworkers = true # Automatically adjust worker threads based on load
|
||||
networkevents = false # Enable detailed network event logging
|
||||
pidfilepath = "./hmac-file-server.pid" # Path to PID file
|
||||
precaching = true # Pre-cache file structures on startup
|
||||
|
||||
# New option to force network protocol
|
||||
forceprotocol = "auto" # Options: "ipv4", "ipv6", "auto"
|
||||
|
||||
@ -27,8 +27,8 @@
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 6,
|
||||
"w": 24,
|
||||
"h": 7,
|
||||
"w": 3,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
@ -42,7 +42,7 @@
|
||||
"content": "<div style=\"text-align: center; background-color: transparent; padding: 20px;\">\n <h3 style=\"color: white; font-family: 'Arial', sans-serif; font-weight: bold;\">HMAC Dashboard</h3>\n <img src=\"https://git.uuxo.net/uuxo/hmac-file-server/raw/branch/main/dashboard/hmac_icon.png\" alt=\"HMAC Icon\" style=\"width: 50px; height: 50px; display: block; margin: 10px auto;\">\n <p style=\"font-family: 'Verdana', sans-serif; color: white;\">\n This dashboard monitors <strong style=\"color: #FF5733;\">key metrics</strong> for the \n <span style=\"font-style: italic; color: #007BFF;\">HMAC File Server</span>.\n </p>\n</div>\n",
|
||||
"mode": "html"
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"title": "HMAC Dashboard",
|
||||
"type": "text"
|
||||
},
|
||||
@ -77,8 +77,8 @@
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 6
|
||||
"x": 3,
|
||||
"y": 0
|
||||
},
|
||||
"id": 14,
|
||||
"options": {
|
||||
@ -105,7 +105,7 @@
|
||||
"sizing": "auto",
|
||||
"valueMode": "color"
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -143,8 +143,8 @@
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 6
|
||||
"x": 9,
|
||||
"y": 0
|
||||
},
|
||||
"id": 18,
|
||||
"options": {
|
||||
@ -164,7 +164,7 @@
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -203,9 +203,9 @@
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 4,
|
||||
"x": 12,
|
||||
"y": 6
|
||||
"w": 5,
|
||||
"x": 15,
|
||||
"y": 0
|
||||
},
|
||||
"id": 10,
|
||||
"options": {
|
||||
@ -225,7 +225,7 @@
|
||||
"textMode": "value",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -262,8 +262,8 @@
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 4,
|
||||
"x": 16,
|
||||
"y": 6
|
||||
"x": 20,
|
||||
"y": 0
|
||||
},
|
||||
"id": 17,
|
||||
"options": {
|
||||
@ -283,7 +283,7 @@
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -297,77 +297,6 @@
|
||||
"title": "HMAC GoRoutines",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"default": true,
|
||||
"type": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 4,
|
||||
"x": 20,
|
||||
"y": 6
|
||||
},
|
||||
"id": 15,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "bduehd5vqv1moa"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "hmac_file_server_upload_duration_seconds_sum + hmac_file_server_download_duration_seconds_sum",
|
||||
"hide": false,
|
||||
"instant": false,
|
||||
"legendFormat": "__auto",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "HMAC Up/Down Duration",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"default": true,
|
||||
@ -397,7 +326,7 @@
|
||||
"h": 7,
|
||||
"w": 5,
|
||||
"x": 0,
|
||||
"y": 13
|
||||
"y": 7
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
@ -417,7 +346,7 @@
|
||||
"textMode": "value",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -460,7 +389,7 @@
|
||||
"h": 7,
|
||||
"w": 5,
|
||||
"x": 5,
|
||||
"y": 13
|
||||
"y": 7
|
||||
},
|
||||
"id": 12,
|
||||
"options": {
|
||||
@ -480,7 +409,7 @@
|
||||
"textMode": "value",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -493,6 +422,77 @@
|
||||
"title": "HMAC Downloads",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"default": true,
|
||||
"type": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 3,
|
||||
"x": 10,
|
||||
"y": 7
|
||||
},
|
||||
"id": 15,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "bduehd5vqv1moa"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "hmac_file_server_upload_duration_seconds_sum + hmac_file_server_download_duration_seconds_sum",
|
||||
"hide": false,
|
||||
"instant": false,
|
||||
"legendFormat": "__auto",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "HMAC Up/Down Duration",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"default": true,
|
||||
@ -519,9 +519,9 @@
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 5,
|
||||
"x": 10,
|
||||
"y": 13
|
||||
"w": 3,
|
||||
"x": 13,
|
||||
"y": 7
|
||||
},
|
||||
"id": 13,
|
||||
"options": {
|
||||
@ -541,7 +541,7 @@
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -580,9 +580,9 @@
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 5,
|
||||
"x": 15,
|
||||
"y": 13
|
||||
"w": 3,
|
||||
"x": 16,
|
||||
"y": 7
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
@ -602,7 +602,7 @@
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -641,9 +641,9 @@
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 4,
|
||||
"x": 20,
|
||||
"y": 13
|
||||
"w": 5,
|
||||
"x": 19,
|
||||
"y": 7
|
||||
},
|
||||
"id": 21,
|
||||
"options": {
|
||||
@ -663,7 +663,7 @@
|
||||
"textMode": "value",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -704,7 +704,7 @@
|
||||
"h": 7,
|
||||
"w": 3,
|
||||
"x": 0,
|
||||
"y": 20
|
||||
"y": 14
|
||||
},
|
||||
"id": 19,
|
||||
"options": {
|
||||
@ -724,7 +724,7 @@
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -746,7 +746,21 @@
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"fillOpacity": 80,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"fieldMinMax": false,
|
||||
"mappings": [],
|
||||
@ -762,7 +776,8 @@
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"unit": "files"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
@ -770,42 +785,37 @@
|
||||
"h": 7,
|
||||
"w": 7,
|
||||
"x": 3,
|
||||
"y": 20
|
||||
"y": 14
|
||||
},
|
||||
"id": 22,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
"tooltip": {
|
||||
"mode": "single",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"exemplar": false,
|
||||
"expr": "hmac_active_connections_total",
|
||||
"expr": "increase(hmac_file_server_clamav_scans_total[24h])",
|
||||
"format": "time_series",
|
||||
"instant": false,
|
||||
"instant": true,
|
||||
"interval": "",
|
||||
"legendFormat": "__auto",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "HMAC Active Connection(s)",
|
||||
"type": "stat"
|
||||
"title": "HMAC ClamAV San (24h)",
|
||||
"type": "histogram"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
@ -818,36 +828,17 @@
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"barWidthFactor": 0.6,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 0,
|
||||
"fillOpacity": 80,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"fieldMinMax": false,
|
||||
@ -873,7 +864,7 @@
|
||||
"h": 7,
|
||||
"w": 7,
|
||||
"x": 10,
|
||||
"y": 20
|
||||
"y": 14
|
||||
},
|
||||
"id": 23,
|
||||
"options": {
|
||||
@ -884,17 +875,16 @@
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"hideZeros": false,
|
||||
"mode": "single",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"exemplar": false,
|
||||
"expr": "hmac_deduplication_errors_total",
|
||||
"expr": "increase(hmac_file_server_clamav_errors_total[24h])",
|
||||
"format": "time_series",
|
||||
"instant": true,
|
||||
"interval": "",
|
||||
@ -903,8 +893,8 @@
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "HMAC Duplication Error",
|
||||
"type": "timeseries"
|
||||
"title": "HMAC ClamAV SanError(s) (24h)",
|
||||
"type": "histogram"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
@ -951,7 +941,7 @@
|
||||
"h": 7,
|
||||
"w": 7,
|
||||
"x": 17,
|
||||
"y": 20
|
||||
"y": 14
|
||||
},
|
||||
"id": 16,
|
||||
"options": {
|
||||
@ -962,12 +952,11 @@
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"hideZeros": false,
|
||||
"mode": "single",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "11.5.2",
|
||||
"pluginVersion": "11.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
@ -986,20 +975,19 @@
|
||||
}
|
||||
],
|
||||
"preload": false,
|
||||
"refresh": "10s",
|
||||
"schemaVersion": 40,
|
||||
"tags": [],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-5m",
|
||||
"from": "now-24h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "HMAC File Server Metrics",
|
||||
"uid": "de0ye5t0hzq4ge",
|
||||
"version": 158,
|
||||
"version": 153,
|
||||
"weekStart": ""
|
||||
}
|
||||
69
lib/maps/iter.go
Normal file
69
lib/maps/iter.go
Normal file
@ -0,0 +1,69 @@
|
||||
package maps
|
||||
|
||||
// Copyright 2024 The Go Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
type Seq2[K comparable, V any] func(yield func(K, V) bool)
|
||||
type Seq[K any] func(yield func(K) bool)
|
||||
|
||||
func All[Map ~map[K]V, K comparable, V any](m Map) Seq2[K, V] {
|
||||
return func(yield func(K, V) bool) {
|
||||
for k, v := range m {
|
||||
if !yield(k, v) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// All returns an iterator over key-value pairs from m.
|
||||
// The iteration order is not specified and is not guaranteed
|
||||
// to be the same from one call to the next.
|
||||
|
||||
func Insert[Map ~map[K]V, K comparable, V any](m Map, seq Seq2[K, V]) {
|
||||
seq(func(k K, v V) bool {
|
||||
m[k] = v
|
||||
return true
|
||||
})
|
||||
}
|
||||
|
||||
// Insert adds the key-value pairs from seq to m.
|
||||
// If a key in seq already exists in m, its value will be overwritten.
|
||||
|
||||
func Collect[K comparable, V any](seq Seq2[K, V]) map[K]V {
|
||||
m := make(map[K]V)
|
||||
Insert(m, seq)
|
||||
return m
|
||||
}
|
||||
|
||||
// Collect collects key-value pairs from seq into a new map
|
||||
// and returns it.
|
||||
|
||||
func Keys[Map ~map[K]V, K comparable, V any](m Map) Seq[K] {
|
||||
return func(yield func(K) bool) {
|
||||
for k := range m {
|
||||
if !yield(k) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Keys returns an iterator over keys in m.
|
||||
// The iteration order is not specified and is not guaranteed
|
||||
// to be the same from one call to the next.
|
||||
|
||||
func Values[Map ~map[K]V, K comparable, V any](m Map) Seq[V] {
|
||||
return func(yield func(V) bool) {
|
||||
for _, v := range m {
|
||||
if !yield(v) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Values returns an iterator over values in m.
|
||||
// The iteration order is not specified and is not guaranteed
|
||||
// to be the same from one call to the next.
|
||||
320
wiki.md
320
wiki.md
@ -28,6 +28,8 @@ This documentation provides detailed information on configuring, setting up, and
|
||||
5. [Building for Different Architectures](#building-for-different-architectures)
|
||||
6. [Additional Recommendations](#additional-recommendations)
|
||||
7. [Notes](#notes)
|
||||
8. [Using HMAC File Server for CI/CD Build Artifacts](#using-hmac-file-server-for-ci-cd-build-artifacts)
|
||||
9. [Monitoring](#monitoring)
|
||||
|
||||
---
|
||||
|
||||
@ -60,6 +62,9 @@ autoadjustworkers = true # Automatically adjust worker threads based on load
|
||||
networkevents = false # Enable detailed network event logging
|
||||
pidfilepath = "./hmac-file-server.pid" # Path to PID file
|
||||
precaching = true # Pre-cache file structures on startup
|
||||
|
||||
# New option to force network protocol
|
||||
forceprotocol = "auto" # Options: "ipv4", "ipv6", "auto"
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
@ -141,6 +146,14 @@ precaching = true # Pre-cache file structures on startup
|
||||
- *Description*: Enables pre-caching of file structures on startup to improve access speed and performance.
|
||||
- *Default*: `true`
|
||||
|
||||
- **forceprotocol**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the network protocol to use for server communication.
|
||||
- `"ipv4"`: Forces the server to use IPv4.
|
||||
- `"ipv6"`: Forces the server to use IPv6.
|
||||
- `"auto"`: Uses the system's default behavior (dual-stack).
|
||||
- *Default*: `"auto"`
|
||||
|
||||
---
|
||||
|
||||
### Deduplication Settings
|
||||
@ -493,6 +506,7 @@ autoadjustworkers = true
|
||||
networkevents = false
|
||||
pidfilepath = "./hmac-file-server.pid"
|
||||
precaching = true
|
||||
forceprotocol = "auto"
|
||||
|
||||
# Deduplication settings
|
||||
[deduplication]
|
||||
@ -681,8 +695,6 @@ To set up a reverse proxy for the HMAC File Server, you can use either Apache2 o
|
||||
sudo systemctl restart nginx
|
||||
```
|
||||
|
||||
You're correct—my statement included unnecessary comments about the configuration. Here's the fully revised configuration without comments or meta-discussion:
|
||||
|
||||
---
|
||||
|
||||
#### 3. ejabberd Configuration
|
||||
@ -794,18 +806,6 @@ To build the HMAC File Server for different architectures, you can use the follo
|
||||
GOOS=linux GOARCH=amd64 go build -o hmac-file-server-linux-amd64
|
||||
```
|
||||
|
||||
### Building for macOS (x86_64)
|
||||
|
||||
```sh
|
||||
GOOS=darwin GOARCH=amd64 go build -o hmac-file-server-darwin-amd64
|
||||
```
|
||||
|
||||
### Building for Windows (x86_64)
|
||||
|
||||
```sh
|
||||
GOOS=windows GOARCH=amd64 go build -o hmac-file-server-windows-amd64.exe
|
||||
```
|
||||
|
||||
### Building for ARM (32-bit)
|
||||
|
||||
```sh
|
||||
@ -818,6 +818,30 @@ GOOS=linux GOARCH=arm GOARM=7 go build -o hmac-file-server-linux-arm
|
||||
GOOS=linux GOARCH=arm64 go build -o hmac-file-server-linux-arm64
|
||||
```
|
||||
|
||||
### Building the Monitoring Tool
|
||||
|
||||
The monitoring tool (`monitor.go`) is located in the `server/cmd/monitor/` directory and is compiled separately from the main HMAC File Server. Below are the instructions for building the monitoring tool:
|
||||
|
||||
#### Building for Linux (x86_64)
|
||||
|
||||
```sh
|
||||
GOOS=linux GOARCH=amd64 go build -o monitor-linux-amd64 ./server/cmd/monitor/monitor.go
|
||||
```
|
||||
|
||||
#### Building for ARM (32-bit)
|
||||
|
||||
```sh
|
||||
GOOS=linux GOARCH=arm GOARM=7 go build -o monitor-linux-arm ./server/cmd/monitor/monitor.go
|
||||
```
|
||||
|
||||
#### Building for ARM (64-bit)
|
||||
|
||||
```sh
|
||||
GOOS=linux GOARCH=arm64 go build -o monitor-linux-arm64 ./server/cmd/monitor/monitor.go
|
||||
```
|
||||
|
||||
Once built, the monitoring tool can be executed independently to track system performance, Prometheus metrics, and active processes.
|
||||
|
||||
---
|
||||
|
||||
## Additional Recommendations
|
||||
@ -832,3 +856,271 @@ GOOS=linux GOARCH=arm64 go build -o hmac-file-server-linux-arm64
|
||||
|
||||
- The HMAC File Server is designed to be flexible and configurable. Adjust the settings in the `config.toml` file to match your specific requirements and environment.
|
||||
- For any issues or questions, refer to the project's GitHub repository and documentation.
|
||||
|
||||
## Using HMAC File Server for CI/CD Build Artifacts
|
||||
|
||||
This guide explains how to use [HMAC File Server](https://github.com/PlusOne/hmac-file-server) to securely upload and download build artifacts in CI/CD pipelines.
|
||||
|
||||
---
|
||||
|
||||
## Why Use HMAC File Server?
|
||||
|
||||
- Secure, HMAC-authenticated access
|
||||
- Self-hosted, no third-party storage needed
|
||||
- Configurable TTL, versioning, and deduplication
|
||||
- Prometheus metrics for monitoring
|
||||
- Easily integrated into GitHub Actions, GitLab CI, Jenkins, etc.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Set Up HMAC File Server
|
||||
|
||||
Clone and build the server:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
cd hmac-file-server
|
||||
go build -o hmac-file-server
|
||||
cp config.example.toml config.toml
|
||||
mkdir -p /data/artifacts
|
||||
./hmac-file-server -config config.toml
|
||||
```
|
||||
|
||||
Update `config.toml` with:
|
||||
|
||||
```toml
|
||||
[hmac]
|
||||
secret = "your-secret-key"
|
||||
|
||||
[upload]
|
||||
enabled = true
|
||||
path = "/data/artifacts"
|
||||
|
||||
[download]
|
||||
enabled = true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Generate Signed URLs
|
||||
|
||||
Use HMAC to generate signed URLs for secure upload/download.
|
||||
|
||||
### Upload Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
FILE_PATH="./build/output.tar.gz"
|
||||
FILENAME="output.tar.gz"
|
||||
SECRET="your-secret-key"
|
||||
BASE_URL="https://your-hmac-server.com"
|
||||
|
||||
TIMESTAMP=$(date +%s)
|
||||
SIGNATURE=$(echo -n "$FILENAME$TIMESTAMP" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/^.* //')
|
||||
|
||||
curl -X PUT "$BASE_URL/upload/$FILENAME?ts=$TIMESTAMP&sig=$SIGNATURE" --data-binary "@$FILE_PATH"
|
||||
```
|
||||
|
||||
### Download Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
FILENAME="output.tar.gz"
|
||||
SECRET="your-secret-key"
|
||||
BASE_URL="https://your-hmac-server.com"
|
||||
|
||||
TIMESTAMP=$(date +%s)
|
||||
SIGNATURE=$(echo -n "$FILENAME$TIMESTAMP" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/^.* //')
|
||||
|
||||
curl -O "$BASE_URL/download/$FILENAME?ts=$TIMESTAMP&sig=$SIGNATURE"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Integrate into CI/CD
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v3
|
||||
|
||||
- name: Build
|
||||
run: |
|
||||
mkdir -p build
|
||||
echo "example artifact content" > build/output.tar.gz
|
||||
|
||||
- name: Upload Artifact to HMAC Server
|
||||
run: bash scripts/upload-artifact.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Optional Features
|
||||
|
||||
- **TTL**: Auto-delete artifacts after a set time
|
||||
- **Deduplication**: Only store unique files
|
||||
- **Versioning**: Track changes to files over time
|
||||
- **Virus Scanning**: Integrate with ClamAV
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
The HMAC File Server provides a built-in monitoring interface to track system performance, Prometheus metrics, and active processes. Below is an overview of the monitoring features:
|
||||
|
||||
### System Data
|
||||
|
||||
The monitoring interface displays key system metrics, including:
|
||||
- **CPU Usage**: Current CPU usage percentage.
|
||||
- **Memory Usage**: Current memory usage percentage.
|
||||
- **CPU Cores**: Number of CPU cores available.
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
The server exposes Prometheus metrics for tracking upload and download statistics:
|
||||
- **hmac_file_server_upload_errors_total**: Total number of upload errors.
|
||||
- **hmac_file_server_uploads_total**: Total number of successful uploads.
|
||||
- **hmac_file_server_downloads_total**: Total number of successful downloads.
|
||||
|
||||
These metrics can be integrated with Prometheus and visualized using tools like Grafana.
|
||||
|
||||
### Process List
|
||||
|
||||
The monitoring interface also provides a list of active processes, including:
|
||||
- Process ID (PID)
|
||||
- CPU usage percentage
|
||||
- Memory usage percentage
|
||||
- Command or service name
|
||||
|
||||
This information helps in identifying resource-intensive processes and debugging performance issues.
|
||||
|
||||
### Example Monitoring Output
|
||||
|
||||
Below is an example of the monitoring interface output:
|
||||
|
||||
```
|
||||
System Data
|
||||
Metric Value
|
||||
CPU Usage 2.78%
|
||||
Memory Usage 26.49%
|
||||
CPU Cores 4
|
||||
|
||||
Prometheus Metrics
|
||||
hmac_file_server_upload_errors_total 1.00
|
||||
hmac_file_server_uploads_total 4.00
|
||||
hmac_file_server_downloads_total 15.00
|
||||
|
||||
Process List
|
||||
PID CPU MEM COMMAND
|
||||
907752 0.12 2.69 /lib/systemd/systemd-journald
|
||||
4055132 0.12 0.03 /usr/sbin/qemu-ga
|
||||
2370782 0.11 0.00 kworker/0:2-wg-crypt-wg1
|
||||
2371119 0.10 0.08 bash
|
||||
2371096 0.10 0.14 sshd: root@pts/0
|
||||
2369170 0.09 0.00 kworker/0:0-mm_percpu_wq
|
||||
2371240 0.07 0.00 kworker/0:1-wg-crypt-wg1
|
||||
2371099 0.06 0.13 systemd --user
|
||||
868714 0.05 0.59 php-fpm: pool www
|
||||
```
|
||||
|
||||
For more details on integrating Prometheus metrics, refer to the [Prometheus documentation](https://prometheus.io/docs/).
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
HMAC File Server is open-source and MIT licensed.
|
||||
|
||||
---
|
||||
|
||||
## Resources
|
||||
|
||||
- [HMAC File Server GitHub Repo](https://github.com/PlusOne/hmac-file-server)
|
||||
- [Configuration Docs](https://github.com/PlusOne/hmac-file-server/wiki)
|
||||
|
||||
## Version 3.0 Release Note
|
||||
|
||||
Version 2.8 is the last release before we begin integrating additional features and focusing on further stability patches.
|
||||
|
||||
## CI/CD with HMAC File Server – Summary
|
||||
|
||||
Sure! Here is a brief guide on how to use the HMAC File Server in your CI/CD pipeline:
|
||||
|
||||
---
|
||||
|
||||
### 1. Server Setup
|
||||
|
||||
```bash
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
cd hmac-file-server
|
||||
go build -o hmac-file-server
|
||||
cp config.example.toml config.toml
|
||||
mkdir -p /data/artifacts
|
||||
./hmac-file-server -config config.toml
|
||||
```
|
||||
|
||||
Update config.toml:
|
||||
```toml
|
||||
[hmac]
|
||||
secret = "your-secret-key"
|
||||
|
||||
[upload]
|
||||
enabled = true
|
||||
path = "/data/artifacts"
|
||||
|
||||
[download]
|
||||
enabled = true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Upload & Download with HMAC
|
||||
|
||||
#### Upload Script
|
||||
|
||||
```bash
|
||||
FILE="output.tar.gz"
|
||||
TS=$(date +%s)
|
||||
SIG=$(echo -n "$FILE$TS" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/^.* //')
|
||||
curl -X PUT "$URL/upload/$FILE?ts=$TS&sig=$SIG" --data-binary "@build/$FILE"
|
||||
```
|
||||
|
||||
#### Download Script
|
||||
|
||||
```bash
|
||||
TS=$(date +%s)
|
||||
SIG=$(echo -n "$FILE$TS" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/^.* //')
|
||||
curl -O "$URL/download/$FILE?ts=$TS&sig=$SIG"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Using in CI/CD (GitHub Actions)
|
||||
|
||||
```yaml
|
||||
- name: Build
|
||||
run: |
|
||||
mkdir -p build
|
||||
echo "artifact" > build/output.tar.gz
|
||||
|
||||
- name: Upload
|
||||
env:
|
||||
SECRET: ${{ secrets.HMAC_SECRET }}
|
||||
run: bash scripts/upload.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Advantages
|
||||
|
||||
- Secure (HMAC)
|
||||
- Self-hosted
|
||||
- Easy to integrate
|
||||
- No dependencies on third-party providers
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user