uuxo/hmac-file-server
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

fix: 3.0 ipv4|ipv6|auto -genconfig

Browse Source
This commit is contained in:
Alexander Renz
2025-06-05 14:58:14 +02:00
parent 56de6107b6
commit c975004eb7
4 changed files with 211 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
CHANGELOG.md
View File

@ -3,42 +3,35 @@
All notable changes to this project will be documented in this file.
## [Unreleased]
- Initial creation of a comprehensive changelog.
---
## [2.8-Stable] - 2026-05-01
### Added
### Added (2.8)
- Version check history for improved tracking.
- Enhanced ClamAV scanning with concurrent workers.
### Changed
### Changed (2.8)
- Improved ISO-based storage for specialized use cases.
- Auto-scaling workers for optimized performance.
### Fixed
### Fixed (2.8)
- Minor issues in worker thread adjustments under high load.
---
## [2.7] - 2026-02-10
### Added
### Added (2.7)
- Concurrency improvements and auto-scaling worker enhancements
- Cleanup and removal of unused parameters in sorting functions
### Changed
### Changed (2.7)
- Additional logging for file scanning operations
### Fixed
### Fixed (2.7)
- Minor stability issues related to ISO container mounting
- Fixed dual stack for upload (IPv4/IPv6)
@ -46,20 +39,17 @@ All notable changes to this project will be documented in this file.
## [2.6-Stable] - 2025-12-01
### Added
### Added (2.6)
- Deduplication support (removes duplicate files).
- ISO Container management.
- Dynamic worker scaling based on CPU & memory.
- PreCaching feature for faster file access.
### Changed
### Changed (2.6)
- Worker pool scaling strategies for better performance.
- Enhanced logging with rotating logs using lumberjack.
### Fixed
### Fixed (2.6)
- Temporary file handling issues causing "Unsupported file type" warnings.
- MIME type checks for file extension mismatches.
@ -67,44 +57,37 @@ All notable changes to this project will be documented in this file.
## [2.5] - 2025-09-15
### Added
### Added (2.5)
- Redis caching integration for file metadata.
- ClamAV scanning for virus detection before finalizing uploads.
### Changed
### Changed (2.5)
- Extended the default chunk size for chunked uploads.
- Updated official documentation links.
### Fixed
### Fixed (2.5)
- Edge case with versioning causing file rename conflicts.
---
## [2.0] - 2025-06-01
### Added
### Added (2.0)
- Chunked file uploads and downloads.
- Resumable upload support with partial file retention.
### Changed
### Changed (2.0)
- Moved configuration management to Viper.
- Default Prometheus metrics for tracking memory & CPU usage.
### Fixed
### Fixed (2.0)
- Race conditions in file locking under heavy concurrency.
---
## [1.0] - 2025-01-01
### Added
### Added (1.0)
- Initial release with HMAC-based authentication.
- Basic file upload/download endpoints.
- Logging and fundamental configuration using .toml files.

125
cmd/server/main.go
View File

@ -423,8 +423,30 @@ func main() {
var configFile string
flag.StringVar(&configFile, "config", "./config.toml", "Path to configuration file \"config.toml\".")
var genConfig bool
var genConfigPath string
flag.BoolVar(&genConfig, "genconfig", false, "Print example configuration and exit.")
flag.StringVar(&genConfigPath, "genconfig-path", "", "Write example configuration to the given file and exit.")
flag.Parse()
if genConfig {
printExampleConfig()
os.Exit(0)
}
if genConfigPath != "" {
f, err := os.Create(genConfigPath)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to create file: %v\n", err)
os.Exit(1)
}
defer f.Close()
w := bufio.NewWriter(f)
fmt.Fprint(w, getExampleConfigString())
w.Flush()
fmt.Printf("Example config written to %s\n", genConfigPath)
os.Exit(0)
}
// Initialize Viper
viper.SetConfigType("toml")
@ -622,9 +644,16 @@ func main() {
if err != nil {
log.Fatalf("Failed to initialize network protocol: %v", err)
}
// Enhanced dual-stack HTTP client for robust IPv4/IPv6 and resource management
// See: https://pkg.go.dev/net/http#Transport for details on these settings
dualStackClient = &http.Client{
Transport: &http.Transport{
DialContext: dialer.DialContext,
DialContext: dialer.DialContext,
IdleConnTimeout: 90 * time.Second, // Close idle connections after 90s
MaxIdleConns: 100, // Max idle connections across all hosts
MaxIdleConnsPerHost: 10, // Max idle connections per host
TLSHandshakeTimeout: 10 * time.Second, // Timeout for TLS handshake
ResponseHeaderTimeout: 15 * time.Second, // Timeout for reading response headers
},
}
@ -776,6 +805,98 @@ version = "2.9-Stable"
`)
}
func getExampleConfigString() string {
return `
[server]
bind_ip = "0.0.0.0"
listenport = "8080"
unixsocket = false
storagepath = "./uploads"
logfile = "/var/log/hmac-file-server.log"
metricsenabled = true
metricsport = "9090"
minfreebytes = "100MB"
filettl = "8760h"
filettlenabled = true
autoadjustworkers = true
networkevents = true
pidfilepath = "/var/run/hmacfileserver.pid"
cleanuponexit = true
precaching = true
deduplicationenabled = true
globalextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
# FileNaming options: "HMAC", "None"
filenaming = "HMAC"
forceprotocol = "auto"
[logging]
level = "info"
file = "/var/log/hmac-file-server.log"
max_size = 100
max_backups = 7
max_age = 30
compress = true
[deduplication]
enabled = true
directory = "./deduplication"
[iso]
enabled = true
size = "1GB"
mountpoint = "/mnt/iso"
charset = "utf-8"
containerfile = "/mnt/iso/container.iso"
[timeouts]
readtimeout = "4800s"
writetimeout = "4800s"
idletimeout = "4800s"
[security]
secret = "changeme"
[versioning]
enableversioning = false
maxversions = 1
[uploads]
resumableuploadsenabled = true
chunkeduploadsenabled = true
chunksize = "8192"
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
[downloads]
resumabledownloadsenabled = true
chunkeddownloadsenabled = true
chunksize = "8192"
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
[clamav]
clamavenabled = true
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 2
scanfileextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
[redis]
redisenabled = true
redisdbindex = 0
redisaddr = "localhost:6379"
redispassword = ""
redishealthcheckinterval = "120s"
[workers]
numworkers = 4
uploadqueuesize = 50
[file]
# Add file-specific configurations here
[build]
version = "2.9-Stable"
`
}
func max(a, b int) int {
if a > b {
return a
@ -1876,6 +1997,8 @@ func handleUpload(w http.ResponseWriter, r *http.Request, absFilename, fileStore
err = createFile(tempFilename, r)
if err != nil {
log.WithFields(logrus.Fields{
"filename": finalFilename,
}).WithError(err).Error("Error creating temp file")
http.Error(w, "Error writing temp file", http.StatusInternalServerError)

39
test/server_flags_test.go Normal file
View File

@ -0,0 +1,39 @@
package main
import (
"os"
"os/exec"
"strings"
"testing"
)
// TestGenConfigFlag runs the server with --genconfig and checks output for expected config keys
func TestGenConfigFlag(t *testing.T) {
cmd := exec.Command("go", "run", "../cmd/server/main.go", "--genconfig")
output, err := cmd.CombinedOutput()
if err != nil && !strings.Contains(string(output), "[server]") {
t.Fatalf("Failed to run with --genconfig: %v\nOutput: %s", err, output)
}
if !strings.Contains(string(output), "[server]") || !strings.Contains(string(output), "bind_ip") {
t.Errorf("Example config missing expected keys. Output: %s", output)
}
}
// TestIPv4IPv6Flag runs the server with forceprotocol=ipv4 and ipv6 and checks for startup errors
func TestIPv4IPv6Flag(t *testing.T) {
for _, proto := range []string{"ipv4", "ipv6", "auto"} {
cmd := exec.Command("go", "run", "../cmd/server/main.go", "--config", "../cmd/server/config.toml")
cmd.Env = append(os.Environ(), "FORCEPROTOCOL="+proto)
// Set Go module cache environment variables if not already set
if os.Getenv("GOMODCACHE") == "" {
cmd.Env = append(cmd.Env, "GOMODCACHE="+os.Getenv("HOME")+"/go/pkg/mod")
}
if os.Getenv("GOPATH") == "" {
cmd.Env = append(cmd.Env, "GOPATH="+os.Getenv("HOME")+"/go")
}
output, err := cmd.CombinedOutput()
if err != nil && !strings.Contains(string(output), "Configuration loaded successfully") {
t.Errorf("Server failed to start with forceprotocol=%s: %v\nOutput: %s", proto, err, output)
}
}
}

36
wiki.md
View File

@ -697,6 +697,23 @@ To set up a reverse proxy for the HMAC File Server, you can use either Apache2 o
---
### Proxy Best Practices & Recommendations
For production deployments, consider the following reverse proxy best practices:
- **Timeouts**: Set reasonable timeouts (e.g., `proxy_read_timeout 300;` in Nginx) to avoid hanging connections.
- **Buffer Sizes**: Increase buffer sizes for large file uploads/downloads if needed (e.g., `client_max_body_size 2G;` in Nginx).
- **Headers**: Always set security headers (`X-Content-Type-Options`, `X-Frame-Options`, `X-XSS-Protection`).
- **Forwarded Headers**: Ensure `X-Forwarded-For` and `X-Forwarded-Proto` are set for correct client IP and protocol logging.
- **HTTP/2**: Enable HTTP/2 for better performance if supported by your proxy and clients.
- **SSL/TLS**: Terminate SSL at the proxy and use strong ciphers. Redirect HTTP to HTTPS.
- **Health Checks**: Configure health checks for the backend server to enable automatic failover or alerting.
- **Access Controls**: Restrict access to the management endpoints (e.g., `/metrics`) to trusted IPs only.
See the official Nginx and Apache documentation for more advanced tuning options.
---
#### 3. ejabberd Configuration
```yaml
@ -944,6 +961,12 @@ curl -O "$BASE_URL/download/$FILENAME?ts=$TIMESTAMP&sig=$SIGNATURE"
### GitHub Actions Example
```yaml
name: Build and Upload to HMAC
on:
push:
branches: [ "main" ]
jobs:
build:
runs-on: ubuntu-latest
@ -964,10 +987,14 @@ jobs:
## Optional Features
- **TTL**: Auto-delete artifacts after a set time
- **Deduplication**: Only store unique files
- **Versioning**: Track changes to files over time
- **Virus Scanning**: Integrate with ClamAV
- **TTL**
Auto-delete artifacts after a set time.
- **Deduplication**
Only store unique files.
- **Versioning**
Track changes to files over time.
- **Virus Scanning**
Integrate with ClamAV to scan uploaded files.
---
@ -978,6 +1005,7 @@ The HMAC File Server provides a built-in monitoring interface to track system pe
### System Data
The monitoring interface displays key system metrics, including:
- **CPU Usage**: Current CPU usage percentage.
- **Memory Usage**: Current memory usage percentage.
- **CPU Cores**: Number of CPU cores available.