uuxo/hmac-file-server
1
1
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 4 Wiki Activity

4 Releases 8 Tags

RSS Feed
  • 2.4-stable 7c5432fbe7

    2.4-stable

    renz released this 2024-12-30 11:04:03 +01:00 | 41 commits to main since this release

    #HMAC File Server 2.4-Stable

    HMAC File Server is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, and optional ClamAV scanning for file integrity and security. It is designed with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration.

    Credits: The HMAC File Server is based on the source code of Thomas Leister's prosody-filer. Many features and design elements have been inspired or derived from this project.

    Features

    • HMAC Authentication: Secure file uploads and downloads using HMAC tokens.
    • File Versioning: Enable versioning for uploaded files with configurable retention.
    • Chunked and Resumable Uploads: Efficient handling of large files through resumable and chunked uploads.
    • ClamAV Scanning: Optional virus scanning for uploaded files.
    • Prometheus Metrics: Monitor system and application-level metrics.
    • Redis Integration: Use Redis for caching or application state management.
    • File Expiration: Automatically delete files after a specified TTL.
    • Graceful Shutdown: Handles signals and ensures proper cleanup.
    • Deduplication: Save storage space by removing duplicate files based on hashing.
    • Auto-Adjust Worker Scaling: Dynamically optimize worker scaling for HMAC and ClamAV based on system resources.
    • Thumbnail Support: Generate smaller versions of uploaded images for efficient storage and quick access.

    Repository

    Installation

    Prerequisites

    • Go 1.20+
    • Redis (optional, for Redis integration)
    • ClamAV (optional, for file scanning)

    Clone and Build

    # Clone from the primary repository
git clone https://github.com/PlusOne/hmac-file-server.git

# OR clone from the alternative repository
git clone https://git.uuxo.net/uuxo/hmac-file-server.git

cd hmac-file-server
go build -o hmac-file-server main.go

    Building for Different Architectures

    Build for arm64

    cd /path/to/hmac-file-server
GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.2-stable_arm64 main.go

    Build for amd64

    cd /path/to/hmac-file-server
GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.2-stable_amd64 main.go

    Replace /path/to/hmac-file-server with the actual project directory path. These commands will generate binaries for the specified architectures in the ~/Temp directory.

    Configuration

    The server configuration is managed through a config.toml file. Below are the supported configuration options:

    Key Features

    • Auto-Adjust Workers: Automatically optimize worker counts for HMAC operations and ClamAV scans based on system resources.
    • Network Events Monitoring: Disable network event logging by setting NetworkEvents = false.
    • Precaching: Enable pre-caching of storage paths for faster access.
    • Thumbnail Support: Configure thumbnail generation options, including size and concurrency.

    Example config.toml

    [server]
ListenPort = "8080"
UnixSocket = false
StoragePath = "./uploads"
LogLevel = "info"
LogFile = ""
MetricsEnabled = true
MetricsPort = "9090"
FileTTL = "1y"
FileTTLEnabled = true
MinFreeBytes = "100MB"
AutoAdjustWorkers = true
NetworkEvents = false
PIDFilePath = "./hmac_file_server.pid"
Precaching = true
TempPath = "/tmp/uploads"

[deduplication]
enabled = true
directory = "/mnt/nfs_vol01/hmac-storage/deduplication/"

[thumbnails]
enabled = true
directory = "/mnt/nfs_vol01/hmac-file-server/thumbnails/"
size = "200x200"
concurrency = 5
thumbnailintervalscan = "24h"

[timeouts]
ReadTimeout = "480s"
WriteTimeout = "480s"
IdleTimeout = "65s"

[security]
Secret = "changeme"

[uploads]
ResumableUploadsEnabled = true
ChunkedUploadsEnabled = true
ChunkSize = "64MB"
AllowedExtensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg"]

[clamav]
ClamAVEnabled = false
ClamAVSocket = "/var/run/clamav/clamd.ctl"
NumScanWorkers = 2
ScanFileExtensions = [".exe", ".dll"]

[redis]
RedisEnabled = false
RedisAddr = "localhost:6379"
RedisPassword = ""
RedisDBIndex = 0
RedisHealthCheckInterval = "120s"

    Running the Server

    Basic Usage

    Run the server with a configuration file:

    ./hmac-file-server -config ./config.toml

    Metrics Server

    If MetricsEnabled is true, the Prometheus metrics server will be available on the port specified in MetricsPort (default: 9090).

    New Features and Changes

    [2.3.0] - 2024-12-28

    Added

    • Configurable temporary upload directory with TempPath.
    • Improved logging for file uploads and MIME type detection.

    Fixed

    • Resolved issues with .tmp files causing unsupported file type warnings.

    Upgrade Instructions

    1. Backup Current Configuration:

      cp ./config.toml ./config.toml.backup

    2. Pull the Latest Version:

      git pull origin v2.4-stable

    3. Update Dependencies:

      go mod tidy

    4. Merge Configuration Changes:
      Compare and update your config.toml file with the new example provided.

    5. Build the Server:

      go build -o hmac-file-server main.go

    6. Restart the Server:

      ./hmac-file-server -config ./config.toml

    Support

    For issues or feature requests, visit our GitHub Issues.

    Downloads