uuxo/hmac-file-server
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
hmac-file-server/WiKi.md

24 KiB
Raw Blame History

This documentation provides detailed information on configuring, setting up, and maintaining the HMAC File Server. Whether you're a developer, system administrator, or an enthusiast, this guide will help you navigate through the server's features and configurations effectively.

Table of Contents

  1. Introduction
  2. Configuration
  3. Example Configuration
  4. Setup Instructions
  5. Building for Different Architectures
  6. Additional Recommendations
  7. Notes

Introduction

The HMAC File Server is a secure and efficient file management solution designed to handle file uploads, downloads, deduplication, and more. Built with a focus on security, scalability, and performance, it integrates seamlessly with various tools and services to provide a comprehensive file handling experience.

Configuration

The HMAC File Server is configured using a config.toml file. Below are the detailed explanations of each configuration section and their respective options.

Server Configuration

# Server configuration
listenport = "8080"  # TCP port for incoming requests
unixsocket = false   # Use Unix domain socket instead of TCP
storagepath = "/path/to/hmac-file-server/data/"  # Directory to store uploaded files
loglevel = "debug"   # Logging level: "debug", "info", "warn", "error"
logfile = "/path/to/hmac-file-server.log"  # Path to log file; leave empty to use stdout
metricsenabled = true   # Enable Prometheus metrics
metricsport = "9090"    # Port for Prometheus metrics
deduplicationenabled = true
minfreebytes = "5GB"     # Minimum free disk space required
filettl = "2Y"           # Time-to-live for files
filettlenabled = false   # Enable TTL checks and cleanup
autoadjustworkers = true # Automatically adjust worker threads based on load
networkevents = false    # Enable detailed network event logging
pidfilepath = "./hmac-file-server.pid" # Path to PID file
precaching = true        # Pre-cache file structures on startup

Configuration Options

  • listenport:

    • Type: String
    • Description: Specifies the TCP port on which the server listens for incoming requests.
    • Default: "8080"

  • unixsocket:

    • Type: Boolean
    • Description: Determines whether to use a Unix domain socket instead of a TCP port for communication.
    • Default: false

  • storagepath:

    • Type: String
    • Description: Defines the directory path where uploaded files are stored. Ensure this path exists and has appropriate permissions.
    • Default: "/path/to/hmac-file-server/data/"

  • loglevel:

    • Type: String
    • Description: Sets the verbosity level of logs.
    • Options: "debug", "info", "warn", "error"
    • Default: "debug"

  • logfile:

    • Type: String
    • Description: Specifies the file path for logging. If left empty, logs are output to stdout.
    • Default: "/path/to/hmac-file-server.log"

  • metricsenabled:

    • Type: Boolean
    • Description: Enables or disables the Prometheus metrics endpoint.
    • Default: true

  • metricsport:

    • Type: String
    • Description: Defines the port on which Prometheus metrics are exposed.
    • Default: "9090"

  • deduplicationenabled:

    • Type: Boolean
    • Description: Enables or disables file deduplication to optimize storage usage.
    • Default: true

  • minfreebytes:

    • Type: String
    • Description: Specifies the minimum free disk space required for the server to operate effectively.
    • Default: "5GB"

  • filettl:

    • Type: String
    • Description: Sets the default Time-to-Live (TTL) for files, determining how long files are retained before deletion.
    • Format: Duration (e.g., "2Y" for two years)
    • Default: "2Y"

  • filettlenabled:

    • Type: Boolean
    • Description: Enables or disables TTL checks and automatic file cleanup based on the filettl value.
    • Default: false

  • autoadjustworkers:

    • Type: Boolean
    • Description: Automatically adjusts the number of worker threads based on server load and system resources.
    • Default: true

  • networkevents:

    • Type: Boolean
    • Description: Enables detailed logging of network events, which can be useful for debugging but may increase log verbosity.
    • Default: false

  • pidfilepath:

    • Type: String
    • Description: Specifies the file path where the server writes its Process ID (PID) file. This is useful for managing the server process.
    • Default: "./hmac-file-server.pid"

  • precaching:

    • Type: Boolean
    • Description: Enables pre-caching of file structures on startup to improve access speed and performance.
    • Default: true

Deduplication Settings

# Deduplication settings
[deduplication]
enabled = true
directory = "/path/to/hmac-file-server/deduplication/"  # Path to deduplication metadata store

Configuration Options

  • enabled:

    • Type: Boolean
    • Description: Enables or disables the deduplication feature, which helps in eliminating duplicate files to save storage space.
    • Default: true

  • directory:

    • Type: String
    • Description: Specifies the directory path where deduplication metadata is stored. Ensure this directory exists and has appropriate permissions.
    • Default: "/path/to/hmac-file-server/deduplication/"

Thumbnails Settings

# Thumbnails settings
[thumbnails]
enabled = true
directory = "/path/to/hmac-file-server/thumbnails/"  # Directory for storing thumbnails
size = "200x200"  # Thumbnail dimensions
thumbnailintervalscan = "1h"  # Interval for scheduled thumbnail generation

Configuration Options

  • enabled:

    • Type: Boolean
    • Description: Enables or disables the automatic generation of thumbnails for uploaded images.
    • Default: true

  • directory:

    • Type: String
    • Description: Defines the directory where generated thumbnails are stored. Ensure this directory exists and has appropriate permissions.
    • Default: "/path/to/hmac-file-server/thumbnails/"

  • size:

    • Type: String
    • Description: Specifies the dimensions to which thumbnails are resized. The format should be "widthxheight" (e.g., "200x200").
    • Default: "200x200"

  • thumbnailintervalscan:

    • Type: String
    • Description: Sets the frequency at which the server scans for new or updated images to generate thumbnails.
    • Format: Duration (e.g., "1h" for one hour)
    • Default: "1h"

ISO Settings

# ISO settings
[iso]
enabled = false
size = "1TB"  # Maximum ISO size
mountpoint = "/path/to/hmac-file-server/iso/"  # ISO mount point
charset = "utf-8"  # Filesystem character set encoding

Configuration Options

  • enabled:

    • Type: Boolean
    • Description: Enables or disables the mounting of an ISO-based filesystem for specialized storage needs.
    • Default: false

  • size:

    • Type: String
    • Description: Defines the maximum allowed size for the ISO container.
    • Default: "1TB"

  • mountpoint:

    • Type: String
    • Description: Specifies the directory path where the ISO is mounted. Ensure this path exists and has appropriate permissions.
    • Default: "/path/to/hmac-file-server/iso/"

  • charset:

    • Type: String
    • Description: Sets the filesystem character set encoding for the ISO.
    • Default: "utf-8"

Note

: Ensure only one [iso] block is active in your config.toml to avoid configuration conflicts.

Timeout Settings

# Timeout settings
[timeouts]
readtimeout = "3600s"    # Maximum time to read a request
writetimeout = "3600s"   # Maximum time to write a response
idletimeout = "3600s"    # Maximum keep-alive time for idle connections

Configuration Options

  • readtimeout:

    • Type: String
    • Description: Sets the maximum duration for reading the entire request, including the body.
    • Format: Duration (e.g., "3600s" for one hour)
    • Default: "3600s"

  • writetimeout:

    • Type: String
    • Description: Defines the maximum duration before timing out writes of the response.
    • Format: Duration (e.g., "3600s" for one hour)
    • Default: "3600s"

  • idletimeout:

    • Type: String
    • Description: Specifies the maximum amount of time to wait for the next request when keep-alives are enabled.
    • Format: Duration (e.g., "3600s" for one hour)
    • Default: "3600s"

Security Settings

# Security settings
[security]
secret = "your-secure-secret-key"  # HMAC shared secret key (change to a secure value)

Configuration Options

  • secret:
    • Type: String
    • Description: The HMAC shared secret key used for signing requests and operations.
    • Default: "your-secure-secret-key"
    • Warning: Change this immediately to a unique, strong string in production environments to ensure the security of HMAC operations.

Versioning Settings

# Versioning settings
[versioning]
enableversioning = false
maxversions = 1  # Number of file versions to retain

Configuration Options

  • enableversioning:

    • Type: Boolean
    • Description: Enables or disables the versioning feature, which maintains multiple versions of the same file.
    • Default: false

  • maxversions:

    • Type: Integer
    • Description: Specifies the maximum number of versions to retain for each file.
    • Default: 1

Uploads Settings

# Upload settings
[uploads]
resumableuploadsenabled = false
chunkeduploadsenabled = true
chunksize = "32MB"  # Chunk size for uploads
allowedextensions = [
    ".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
    ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
    ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
    ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
]

Configuration Options

  • resumableuploadsenabled:

    • Type: Boolean
    • Description: Enables or disables support for resumable (chunked) file uploads.
    • Default: false

  • chunkeduploadsenabled:

    • Type: Boolean
    • Description: Specifically enables or disables chunked uploads.
    • Default: true

  • chunksize:

    • Type: String
    • Description: Defines the size of each chunk in chunked uploads.
    • Format: Size (e.g., "32MB")
    • Default: "32MB"

  • allowedextensions:

    • Type: Array of Strings
    • Description: Lists the file extensions permitted for upload.
    • Default: 
      allowedextensions = [
    ".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
    ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
    ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
    ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
]

Downloads Settings

# Downloads settings
[downloads]
resumabledownloadsenabled = false
chunkeddownloadsenabled = true
chunksize = "32MB"

Configuration Options

  • resumabledownloadsenabled:

    • Type: Boolean
    • Description: Enables or disables support for resumable (chunked) downloads.
    • Default: false

  • chunkeddownloadsenabled:

    • Type: Boolean
    • Description: Specifically enables or disables chunked downloads.
    • Default: true

  • chunksize:

    • Type: String
    • Description: Defines the size of each chunk in chunked downloads.
    • Format: Size (e.g., "32MB")
    • Default: "32MB"

Note

: The allowedextensions key is not part of the [downloads] configuration based on the provided code. Ensure that it is omitted to prevent configuration errors.

ClamAV Settings

# ClamAV settings
[clamav]
clamavenabled = true
clamavsocket = "/path/to/clamav/clamd.ctl"  # Path to ClamAV socket
numscanworkers = 4  # Number of concurrent scan workers
scanfileextensions = [
    ".exe", ".dll", ".bin", ".com", ".bat",
    ".sh", ".php", ".js"
]

Configuration Options

  • clamavenabled:

    • Type: Boolean
    • Description: Enables or disables ClamAV integration for virus scanning of uploaded files.
    • Default: true

  • clamavsocket:

    • Type: String
    • Description: Specifies the file path to the ClamAV socket (.ctl file). Ensure ClamAV is installed and the socket path is correct.
    • Default: "/path/to/clamav/clamd.ctl"

  • numscanworkers:

    • Type: Integer
    • Description: Sets the number of concurrent workers dedicated to scanning files with ClamAV.
    • Default: 4

  • scanfileextensions:

    • Type: Array of Strings
    • Description: Lists the file extensions that should be scanned for viruses.
    • Default: 
      scanfileextensions = [
    ".exe", ".dll", ".bin", ".com", ".bat",
    ".sh", ".php", ".js"
]

Redis Settings

# Redis settings
[redis]
redisenabled = true
redisdbindex = 0
redisaddr = "localhost:6379"  # Redis server address
redispassword = ""            # Redis password if required
redishealthcheckinterval = "120s"  # Interval for Redis health checks

Configuration Options

  • redisenabled:

    • Type: Boolean
    • Description: Enables or disables Redis integration for caching or session tracking.
    • Default: true

  • redisaddr:

    • Type: String
    • Description: Specifies the address of the Redis server (e.g., "localhost:6379").
    • Default: "localhost:6379"

  • redispassword:

    • Type: String
    • Description: Sets the Redis authentication password, if required.
    • Default: ""

  • redisdbindex:

    • Type: Integer
    • Description: Specifies the Redis database index to use.
    • Default: 0

  • redishealthcheckinterval:

    • Type: String
    • Description: Defines the interval for performing health checks on the Redis connection.
    • Format: Duration (e.g., "120s" for two minutes)
    • Default: "120s"

Worker Settings

# Worker settings
[worker]
numworkers = 10  # Number of worker threads

Configuration Options

  • numworkers:
    • Type: Integer
    • Description: Specifies the number of worker threads to handle file operations.
    • Default: 10

File Settings

# File settings
[file]
maxfilesize = "10GB"  # Maximum file size for uploads

Configuration Options

  • maxfilesize:
    • Type: String
    • Description: Defines the maximum allowed file size for uploads.
    • Format: Size (e.g., "10GB")
    • Default: "10GB"

Example Configuration

Below is an example config.toml file with default settings:

# Example HMAC File Server configuration

# Server configuration
listenport = "8080"
unixsocket = false
storagepath = "/path/to/hmac-file-server/data/"
loglevel = "debug"
logfile = "/path/to/hmac-file-server.log"
metricsenabled = true
metricsport = "9090"
deduplicationenabled = true
minfreebytes = "5GB"
filettl = "2Y"
filettlenabled = false
autoadjustworkers = true
networkevents = false
pidfilepath = "./hmac-file-server.pid"
precaching = true

# Deduplication settings
[deduplication]
enabled = true
directory = "/path/to/hmac-file-server/deduplication/"

# Thumbnails settings
[thumbnails]
enabled = true
directory = "/path/to/hmac-file-server/thumbnails/"
size = "200x200"
thumbnailintervalscan = "1h"

# ISO settings
[iso]
enabled = false
size = "1TB"
mountpoint = "/path/to/hmac-file-server/iso/"
charset = "utf-8"

# Timeout settings
[timeouts]
readtimeout = "3600s"
writetimeout = "3600s"
idletimeout = "3600s"

# Security settings
[security]
secret = "your-secure-secret-key"

# Versioning settings
[versioning]
enableversioning = false
maxversions = 1

# Upload settings
[uploads]
resumableuploadsenabled = false
chunkeduploadsenabled = true
chunksize = "32MB"
allowedextensions = [
    ".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
    ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
    ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
    ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
]

# Download settings
[downloads]
resumabledownloadsenabled = false
chunkeddownloadsenabled = true
chunksize = "32MB"

# ClamAV settings
[clamav]
clamavenabled = true
clamavsocket = "/path/to/clamav/clamd.ctl"
numscanworkers = 4
scanfileextensions = [
    ".exe", ".dll", ".bin", ".com", ".bat",
    ".sh", ".php", ".js"
]

# Redis settings
[redis]
redisenabled = true
redisdbindex = 0
redisaddr = "localhost:6379"
redispassword = ""
redishealthcheckinterval = "120s"

# Worker settings
[worker]
numworkers = 10

# File settings
[file]
maxfilesize = "10GB"

Setup Instructions

1. HMAC File Server Installation

To install the HMAC File Server, follow these steps:

  1. Clone the repository:

    git clone https://github.com/your-repo/hmac-file-server.git
cd hmac-file-server

  2. Build the server:

    go build -o hmac-file-server

  3. Create the necessary directories:

    mkdir -p /path/to/hmac-file-server/data/
mkdir -p /path/to/hmac-file-server/deduplication/
mkdir -p /path/to/hmac-file-server/thumbnails/
mkdir -p /path/to/hmac-file-server/iso/

  4. Copy the example configuration file:

    cp config.example.toml config.toml

  5. Edit the config.toml file to match your environment and preferences.

  6. Start the server:

    ./hmac-file-server -config config.toml

2. Reverse Proxy Configuration

To set up a reverse proxy for the HMAC File Server, you can use either Apache2 or Nginx. Below are the configuration examples for both.

Apache2 Reverse Proxy

  1. Enable the necessary Apache2 modules:

    sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod headers
sudo a2enmod rewrite

  2. Create a new virtual host configuration file:

    sudo nano /etc/apache2/sites-available/hmac-file-server.conf

  3. Add the following configuration to the file:

    <VirtualHost *:80>
    ServerName your-domain.com

    ProxyPreserveHost On
    ProxyPass / http://localhost:8080/
    ProxyPassReverse / http://localhost:8080/

    <Location />
        Require all granted
        Header always set X-Content-Type-Options "nosniff"
        Header always set X-Frame-Options "DENY"
        Header always set X-XSS-Protection "1; mode=block"
    </Location>
</VirtualHost>

  4. Enable the new site and restart Apache2:

    sudo a2ensite hmac-file-server.conf
sudo systemctl restart apache2

Nginx Reverse Proxy

  1. Install Nginx if not already installed:

    sudo apt-get update
sudo apt-get install nginx

  2. Create a new server block configuration file:

    sudo nano /etc/nginx/sites-available/hmac-file-server

  3. Add the following configuration to the file:

    server {
    listen 80;
    server_name your-domain.com;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Content-Type-Options "nosniff";
        proxy_set_header X-Frame-Options "DENY";
        proxy_set_header X-XSS-Protection "1; mode=block";
    }
}

  4. Enable the new site and restart Nginx:

    sudo ln -s /etc/nginx/sites-available/hmac-file-server /etc/nginx/sites-enabled/
sudo systemctl restart nginx

3. ejabberd Configuration

To configure ejabberd for use with the HMAC File Server, follow these steps:

  1. Install ejabberd:

    sudo apt-get update
sudo apt-get install ejabberd

  2. Edit the ejabberd configuration file:

    sudo nano /etc/ejabberd/ejabberd.yml

  3. Add the following configuration to the file:

    hosts:
  - "your-domain.com"

listen:
  -
    port: 5222
    module: ejabberd_c2s
    certfile: "/etc/ejabberd/ejabberd.pem"
    starttls: true
    starttls_required: true
    protocol_options:
      - "no_sslv3"
      - "no_tlsv1"
      - "no_tlsv1_1"
    ciphers: "HIGH:!aNULL:!eNULL:!3DES:@STRENGTH"
    dhfile: "/etc/ejabberd/dhparams.pem"
    max_stanza_size: 65536
    shaper: c2s_shaper
    access: c2s

  -
    port: 5269
    module: ejabberd_s2s_in
    certfile: "/etc/ejabberd/ejabberd.pem"
    starttls: true
    starttls_required: true
    protocol_options:
      - "no_sslv3"
      - "no_tlsv1"
      - "no_tlsv1_1"
    ciphers: "HIGH:!aNULL:!eNULL:!3DES:@STRENGTH"
    dhfile: "/etc/ejabberd/dhparams.pem"
    max_stanza_size: 131072
    shaper: s2s_shaper
    access: s2s

acl:
  local:
    user_regexp: ""

access_rules:
  local:
    allow: local

  4. Restart ejabberd:

    sudo systemctl restart ejabberd

4. Systemd Service Setup

To set up the HMAC File Server as a systemd service, follow these steps:

  1. Create a new systemd service file:

    sudo nano /etc/systemd/system/hmac-file-server.service

  2. Add the following configuration to the file:

    [Unit]
Description=HMAC File Server
After=network.target

[Service]
ExecStart=/path/to/hmac-file-server -config /path/to/config.toml
WorkingDirectory=/path/to/hmac-file-server
Restart=always
User=www-data
Group=www-data

[Install]
WantedBy=multi-user.target

  3. Reload systemd and enable the service:

    sudo systemctl daemon-reload
sudo systemctl enable hmac-file-server
sudo systemctl start hmac-file-server

Building for Different Architectures

To build the HMAC File Server for different architectures, you can use the following commands:

Building for Linux (x86_64)

GOOS=linux GOARCH=amd64 go build -o hmac-file-server-linux-amd64

Building for macOS (x86_64)

GOOS=darwin GOARCH=amd64 go build -o hmac-file-server-darwin-amd64

Building for Windows (x86_64)

GOOS=windows GOARCH=amd64 go build -o hmac-file-server-windows-amd64.exe

Building for ARM (32-bit)

GOOS=linux GOARCH=arm GOARM=7 go build -o hmac-file-server-linux-arm

Building for ARM (64-bit)

GOOS=linux GOARCH=arm64 go build -o hmac-file-server-linux-arm64

Additional Recommendations

  • Security: Ensure that the secret key in the config.toml file is changed to a unique, strong value to secure HMAC operations.
  • Backups: Regularly back up the config.toml file and any important data stored by the HMAC File Server.
  • Monitoring: Use monitoring tools like Prometheus and Grafana to keep track of server performance and metrics.

Notes

  • The HMAC File Server is designed to be flexible and configurable. Adjust the settings in the config.toml file to match your specific requirements and environment.
  • For any issues or questions, refer to the project's GitHub repository and documentation.