Compare commits
19 Commits
2.5-stable
...
main
Author | SHA1 | Date | |
---|---|---|---|
5751f1d0c7 | |||
b05c444a0a | |||
af5aaa528c | |||
ef3b618126 | |||
b14f046beb | |||
0cda54c97e | |||
16d2fd4b81 | |||
cd5cbd74bc | |||
ad8d9771ba | |||
1ac8ff38e8 | |||
5487db9e57 | |||
5e99b937e8 | |||
1d4a224f01 | |||
1288e9ac57 | |||
881237a904 | |||
74fa394a30 | |||
5ef909fdec | |||
921062cfc3 | |||
18495812fd |
21
LICENSE
Normal file
21
LICENSE
Normal file
@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Alexander Renz
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
547
README.MD
547
README.MD
@ -1,446 +1,205 @@
|
||||
# HMAC File Server
|
||||
A secure file server with HMAC authentication and configurable features.
|
||||
# HMAC File Server 2.5-Stable
|
||||
|
||||
**HMAC File Server** is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, deduplication, optional ClamAV scanning for file integrity and security, and image thumbnail generation. This server is built with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration.
|
||||
## Overview
|
||||
The **HMAC File Server** ensures secure file uploads and downloads using HMAC authentication. It incorporates rate limiting, CORS support, retries, file versioning, and Unix socket support for enhanced flexibility. Redis integration provides efficient caching and session management. Prometheus metrics and a graceful shutdown mechanism ensure reliable and efficient file handling.
|
||||
|
||||
> **Credits:** The **HMAC File Server** is based on the source code of [Thomas Leister's prosody-filer](https://github.com/ThomasLeister/prosody-filer). Many features and design elements have been inspired or derived from this project.
|
||||
|
||||
---
|
||||
Special thanks to **Thomas Leister** for inspiration drawn from [[prosody-filer](https://github.com/ThomasLeister/prosody-filer)](https://github.com/ThomasLeister/prosody-filer).
|
||||
|
||||
## Features
|
||||
- File deduplication
|
||||
- Configurable TTL for automatic file cleanup
|
||||
- Secure HMAC-based authentication
|
||||
- Chunked uploads and downloads
|
||||
- Virus scanning via ClamAV
|
||||
- Prometheus metrics integration
|
||||
- Customizable worker management
|
||||
- Support ISO-based storage
|
||||
|
||||
- **HMAC Authentication:** Secure file uploads and downloads with HMAC tokens.
|
||||
- **File Versioning:** Enable versioning for uploaded files with configurable retention.
|
||||
- **Chunked and Resumable Uploads:** Handle large files efficiently with support for resumable and chunked uploads.
|
||||
- **Deduplication:** Automatically remove duplicate files based on hashing for storage efficiency.
|
||||
- **ClamAV Scanning:** Optional virus scanning for uploaded files.
|
||||
- **Prometheus Metrics:** Monitor system and application-level metrics.
|
||||
- **Redis Integration:** Use Redis for caching or storing application states.
|
||||
- **File Expiration:** Automatically delete files after a specified TTL.
|
||||
- **Graceful Shutdown:** Handles signals and ensures proper cleanup.
|
||||
- **Auto-Adjust Worker Scaling:** Dynamically optimize worker threads based on system resources.
|
||||
- **Precaching:** Pre-cache file structures on startup for faster access.
|
||||
- **Thumbnail Creation:** Generate image thumbnails for uploaded files.
|
||||
- **ISO Container Management:** Optional mounting and handling of ISO-based filesystems.
|
||||
|
||||
---
|
||||
|
||||
## Repository
|
||||
|
||||
- **Primary Repository**: [GitHub Repository](https://github.com/PlusOne/hmac-file-server)
|
||||
- **Alternative Repository**: [uuxo.net Git Repository](https://git.uuxo.net/uuxo/hmac-file-server)
|
||||
## Table of Contents
|
||||
1. [Installation](#installation)
|
||||
2. [Configuration](#configuration)
|
||||
3. [Usage](#usage)
|
||||
4. [Setup](#setup)
|
||||
- [Reverse Proxy](#reverse-proxy)
|
||||
- [Systemd Service](#systemd-service)
|
||||
5. [Building](#building)
|
||||
6. [Changelog](#changelog)
|
||||
7. [License](#license)
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
- Go **1.20** or higher
|
||||
- Redis server (optional, for caching)
|
||||
- ClamAV (optional, for virus scanning)
|
||||
|
||||
- **Go 1.20+**
|
||||
- **Redis** (optional, if Redis integration is enabled)
|
||||
- **ClamAV** (optional, if file scanning is enabled)
|
||||
- **genisoimage** (optional, if ISO container management is enabled)
|
||||
### Steps
|
||||
1. Clone the repository:
|
||||
```bash
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
cd hmac-file-server
|
||||
```
|
||||
|
||||
### Clone and Build
|
||||
2. Build the server:
|
||||
```bash
|
||||
go build -o hmac-file-server
|
||||
```
|
||||
|
||||
```bash
|
||||
# Clone from the primary repository
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
3. Create necessary directories:
|
||||
```bash
|
||||
mkdir -p /path/to/hmac-file-server/data/
|
||||
mkdir -p /path/to/hmac-file-server/deduplication/
|
||||
mkdir -p /path/to/hmac-file-server/iso/
|
||||
```
|
||||
|
||||
# OR clone from the alternative repository
|
||||
git clone https://git.uuxo.net/uuxo/hmac-file-server.git
|
||||
4. Copy and edit the configuration file:
|
||||
```bash
|
||||
cp config.example.toml config.toml
|
||||
```
|
||||
|
||||
cd hmac-file-server
|
||||
go build -o hmac-file-server main.go
|
||||
```
|
||||
|
||||
#### Build for `arm64`
|
||||
```bash
|
||||
cd /path/to/hmac-file-server
|
||||
GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.3-stable_arm64 main.go
|
||||
```
|
||||
#### Build for `amd64`
|
||||
```bash
|
||||
cd /path/to/hmac-file-server
|
||||
GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.3-stable_amd64 main.go
|
||||
```
|
||||
5. Start the server:
|
||||
```bash
|
||||
./hmac-file-server -config config.toml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
The server is configured via a `config.toml` file. Key settings include:
|
||||
|
||||
The server configuration is managed through a `config.toml` file. Below are the supported configuration options:
|
||||
- **Server Settings**: Port, logging, metrics
|
||||
- **Security**: HMAC secret, TLS options
|
||||
- **File Management**: TTL, deduplication, uploads, and downloads
|
||||
- **ISO**: Generation and mounting settings
|
||||
- **Workers**: Adjust thread management
|
||||
|
||||
### Auto-Adjust Feature
|
||||
|
||||
When `AutoAdjustWorkers` is enabled, the number of workers for HMAC operations and ClamAV scans is dynamically determined based on system resources. This ensures efficient resource utilization.
|
||||
|
||||
If `AutoAdjustWorkers = true`, the values for `NumWorkers` and `NumScanWorkers` in the configuration file will be ignored, and the server will automatically adjust these values.
|
||||
|
||||
### Network Events Monitoring
|
||||
|
||||
Setting `NetworkEvents = false` in the server configuration disables the logging and tracking of network-related events within the application. This means that functionalities such as monitoring IP changes or recording network activity will be turned off.
|
||||
|
||||
### Precaching
|
||||
|
||||
The `precaching` feature allows the server to pre-cache storage paths for faster access. This can improve performance by reducing the time needed to access frequently used storage paths.
|
||||
|
||||
### Thumbnail Creation
|
||||
|
||||
Set `enabled = true` in the `[thumbnails]` section of `config.toml` to enable image thumbnail generation.
|
||||
For detailed configuration options, refer to the [Wiki](./wiki.md).
|
||||
|
||||
---
|
||||
|
||||
## Example `config.toml`
|
||||
|
||||
Below is an example configuration file (`config.toml`) you can use as a reference (with sensitive data replaced by placeholder/example data):
|
||||
|
||||
```toml
|
||||
# Server configuration
|
||||
listenport = "8080" # TCP port for incoming requests
|
||||
unixsocket = false # Use Unix domain socket instead of TCP
|
||||
storagepath = "/path/to/hmac-file-server/data/" # Directory to store uploaded files
|
||||
loglevel = "debug" # Logging level: "debug", "info", "warn", "error"
|
||||
logfile = "/path/to/hmac-file-server.log" # Path to log file; leave empty to use stdout
|
||||
metricsenabled = true # Enable Prometheus metrics
|
||||
metricsport = "9090" # Port for Prometheus metrics
|
||||
deduplicationenabled = true
|
||||
minfreebytes = "5GB" # Minimum free disk space required
|
||||
filettl = "2Y" # Time-to-live for files (2 years)
|
||||
filettlenabled = false # Enable TTL checks and cleanup
|
||||
autoadjustworkers = true # Automatically adjust worker threads based on load
|
||||
networkevents = false # Enable detailed network event logging
|
||||
pidfilepath = "./hmac-file-server.pid" # Path to PID file
|
||||
precaching = true # Pre-cache file structures on startup
|
||||
tempPath = "/tmp/hmac-file-server" # Path for temporary file uploads
|
||||
|
||||
# Deduplication settings
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/deduplication/" # Path to deduplication metadata store
|
||||
|
||||
# Thumbnails settings
|
||||
[thumbnails]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/thumbnails/" # Directory for storing thumbnails
|
||||
size = "200x200" # Thumbnail dimensions
|
||||
thumbnailintervalscan = "1h" # Interval for scheduled thumbnail generation
|
||||
concurrency = 5 # Number of concurrent thumbnail generation tasks
|
||||
|
||||
# ISO settings
|
||||
[iso]
|
||||
enabled = false
|
||||
size = "1TB" # Maximum ISO size
|
||||
mountpoint = "/path/to/hmac-file-server/iso/" # ISO mount point
|
||||
charset = "utf-8" # Filesystem character set encoding
|
||||
|
||||
# Timeout settings
|
||||
[timeouts]
|
||||
readtimeout = "3600s" # Maximum time to read a request (1 hour)
|
||||
writetimeout = "3600s" # Maximum time to write a response (1 hour)
|
||||
idletimeout = "3600s" # Maximum keep-alive time for idle connections (1 hour)
|
||||
|
||||
# Security settings
|
||||
[security]
|
||||
secret = "your-secure-secret-key" # HMAC shared secret key (change to a secure value)
|
||||
|
||||
# Versioning settings
|
||||
[versioning]
|
||||
enableversioning = false
|
||||
maxversions = 1 # Number of file versions to retain
|
||||
|
||||
# Upload settings
|
||||
[uploads]
|
||||
resumableuploadsenabled = false
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "32MB" # Chunk size for uploads
|
||||
allowedextensions = ["*"] # All file types are allowed for uploads
|
||||
|
||||
# Downloads settings
|
||||
[downloads]
|
||||
resumabledownloadsenabled = false
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "32MB"
|
||||
|
||||
# ClamAV settings
|
||||
[clamav]
|
||||
clamavenabled = true
|
||||
clamavsocket = "/path/to/clamav/clamd.ctl" # Path to ClamAV socket
|
||||
numscanworkers = 4 # Number of concurrent scan workers
|
||||
scanfileextensions = [
|
||||
".exe", ".dll", ".bin", ".com", ".bat",
|
||||
".sh", ".php", ".js"
|
||||
]
|
||||
|
||||
# Redis settings
|
||||
[redis]
|
||||
redisenabled = true
|
||||
redisdbindex = 0
|
||||
redisaddr = "localhost:6379" # Redis server address
|
||||
redispassword = "" # Redis password if required
|
||||
redishealthcheckinterval = "120s" # Interval for Redis health checks
|
||||
|
||||
# Worker settings
|
||||
[workers]
|
||||
numworkers = 4
|
||||
uploadqueuesize = 5000
|
||||
|
||||
# File settings
|
||||
[file]
|
||||
filerevision = 1 # Internal revision number for file handling logic
|
||||
|
||||
# Logging settings
|
||||
[logging]
|
||||
level = "info"
|
||||
file = "/var/log/hmac-file-server.log"
|
||||
max_size = 100
|
||||
max_backups = 7
|
||||
max_age = 30
|
||||
compress = true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Verification
|
||||
|
||||
The application ensures that all configuration parameters defined in `config.toml` are correctly implemented and operational. During startup, the server performs the following verification steps:
|
||||
|
||||
1. **Configuration Parsing:**
|
||||
- Parses the `config.toml` file into the application's `Config` struct.
|
||||
|
||||
2. **Validation:**
|
||||
- Checks that all required configuration parameters are set.
|
||||
- Validates the correctness of configuration values (e.g., proper duration formats, non-empty directories).
|
||||
|
||||
3. **Service Initialization:**
|
||||
- Initializes services like Redis and ClamAV based on the configuration.
|
||||
- Ensures that conditional services (e.g., Thumbnails, ISO) are only started if enabled in the configuration.
|
||||
|
||||
If any configuration parameter is missing or invalid, the server will log an error and terminate to prevent running with incorrect settings.
|
||||
|
||||
---
|
||||
|
||||
## Running the Server
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run the server with a configuration file:
|
||||
## Usage
|
||||
Start the server and access it on the configured port. Use curl or a client library to interact with the API.
|
||||
|
||||
### Example
|
||||
Upload a file:
|
||||
```bash
|
||||
./hmac-file-server -config ./config.toml
|
||||
curl -X POST -F 'file=@example.jpg' http://localhost:8080/upload
|
||||
```
|
||||
|
||||
Ensure that the `config.toml` is correctly set up before running the server.
|
||||
|
||||
---
|
||||
|
||||
### Metrics Server
|
||||
## Setup
|
||||
|
||||
If `metricsenabled` is set to `true`, the Prometheus metrics server will be available on the port specified in `metricsport` (e.g., `http://localhost:9090/metrics`).
|
||||
### Reverse Proxy
|
||||
Set up a reverse proxy using Apache2 or Nginx to handle requests.
|
||||
|
||||
Additional metrics for deduplication and ISO container operations are now available.
|
||||
#### Apache2 Example
|
||||
```apache
|
||||
<VirtualHost *:80>
|
||||
ServerName your-domain.com
|
||||
|
||||
---
|
||||
ProxyPreserveHost On
|
||||
ProxyPass / http://localhost:8080/
|
||||
ProxyPassReverse / http://localhost:8080/
|
||||
</VirtualHost>
|
||||
```
|
||||
|
||||
## Health Checks
|
||||
#### Nginx Example
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name your-domain.com;
|
||||
|
||||
The server includes built-in health checks to verify that all configured services are operational. Logs will provide detailed information about the status of each service and any issues encountered during startup.
|
||||
location / {
|
||||
proxy_pass http://localhost:8080;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
### Systemd Service
|
||||
Create a systemd service file for the HMAC File Server:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=HMAC File Server
|
||||
After=network.target
|
||||
|
||||
## Testing
|
||||
|
||||
To run the server locally for development:
|
||||
[Service]
|
||||
ExecStart=/path/to/hmac-file-server -config /path/to/config.toml
|
||||
WorkingDirectory=/path/to/hmac-file-server
|
||||
Restart=always
|
||||
User=www-data
|
||||
Group=www-data
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
Enable and start the service:
|
||||
```bash
|
||||
go run main.go -config ./config.toml
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable hmac-file-server
|
||||
sudo systemctl start hmac-file-server
|
||||
```
|
||||
|
||||
Use tools like **cURL** or **Postman** to test file uploads and downloads.
|
||||
|
||||
### Example File Upload with HMAC Token
|
||||
|
||||
```bash
|
||||
curl -X PUT -H "Authorization: Bearer <HMAC-TOKEN>" -F "file=@example.txt" http://localhost:8080/uploads/example.txt
|
||||
```
|
||||
|
||||
Replace `<HMAC-TOKEN>` with a valid HMAC signature generated using the configured `secret`.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
## Building
|
||||
To build for different architectures:
|
||||
|
||||
Prometheus metrics include:
|
||||
- **Linux (amd64)**:
|
||||
```bash
|
||||
GOOS=linux GOARCH=amd64 go build -o hmac-file-server main.go
|
||||
```
|
||||
|
||||
- **File Operations:**
|
||||
- File upload/download durations
|
||||
- Uploaded/downloaded file sizes
|
||||
- Total number of uploads/downloads
|
||||
- Total number of upload/download errors
|
||||
|
||||
- **System Metrics:**
|
||||
- Memory usage
|
||||
- CPU usage
|
||||
- Number of goroutines
|
||||
- Active connections
|
||||
|
||||
- **HTTP Requests:**
|
||||
- Total HTTP requests broken down by method and path
|
||||
|
||||
Access the metrics at `http://localhost:9090/metrics` (assuming default `metricsport`).
|
||||
|
||||
---
|
||||
|
||||
## Additional Features
|
||||
|
||||
- **Deduplication:** Automatically remove duplicate files based on SHA256 hashing to save storage space.
|
||||
- **Versioning:** Store multiple versions of files and retain a configurable number of versions.
|
||||
- **ClamAV Integration:** Scan uploaded files for viruses using ClamAV to ensure file integrity and security.
|
||||
- **Redis Caching:** Utilize Redis for caching file metadata to improve access times and performance.
|
||||
- **Auto-Adjust Worker Scaling:** Optimize the number of workers dynamically based on system resources to maintain optimal performance.
|
||||
- **Precaching:** Pre-cache file structures on startup to reduce access times for frequently accessed files.
|
||||
- **Thumbnail Creation:** Generate image thumbnails for uploaded files to provide quick previews.
|
||||
- **ISO Container Management:** Optional mounting and handling of ISO-based filesystems for specialized storage needs.
|
||||
|
||||
---
|
||||
|
||||
### Overview of Other Projects (xep0363)
|
||||
|
||||
| Feature/Project | HMAC FS | mod_http_upload_ext | xmpp-http-upload (horazont) | Prosody Filer | ngx_http_upload | xmpp-http-upload (nyovaya) |
|
||||
|-----------------------------|---------|----------------------|-----------------------------|---------------|------------------|----------------------------|
|
||||
| **Language** | Go | PHP | Python | Go | C (Nginx) | Python |
|
||||
| **Environment** | Standalone | Prosody module | Standalone | Standalone | Nginx | Standalone |
|
||||
| **XMPP** | No | Yes | Yes | Yes | No | Yes |
|
||||
| **External Storage** | Yes | No | Possible via plugins | No | No | Yes |
|
||||
| **Authentication / Security** | HMAC | Token-based | Token-based | None | Basic / None | Token-based |
|
||||
|
||||
---
|
||||
|
||||
## Build & Run
|
||||
|
||||
1. **Clone the Repository:**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
cd hmac-file-server
|
||||
```
|
||||
|
||||
2. **Build the Server:**
|
||||
|
||||
```bash
|
||||
go build -o hmac-file-server main.go
|
||||
```
|
||||
|
||||
3. **Run the Server:**
|
||||
|
||||
```bash
|
||||
./hmac-file-server -config ./config.toml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices and Recommendations
|
||||
|
||||
1. **Configuration Consistency:**
|
||||
- Ensure all configuration keys in `config.toml` match those expected in the code. For example, in the `[thumbnails]` section, use `thumbnailintervalscan` instead of `scanInterval` to align with the code.
|
||||
|
||||
2. **Security:**
|
||||
- **HMAC Secret:**
|
||||
- **Change Immediately:** The `secret` under `[security]` should be a strong, unique string. Replace `"your-secure-secret-key"` with a securely generated key.
|
||||
- **ClamAV:**
|
||||
- Ensure ClamAV is installed and the `clamavsocket` path is correct if `[clamav].clamavenabled` is `true`.
|
||||
- **Redis:**
|
||||
- Secure your Redis instance, especially if it's exposed to external networks. Use strong passwords and restrict access as needed.
|
||||
|
||||
3. **Resource Allocation:**
|
||||
- Adjust `numworkers` and `uploadqueuesize` in the `[workers]` section based on your server's hardware capabilities and expected traffic.
|
||||
- Monitor system performance to ensure that auto-adjust features are working optimally.
|
||||
|
||||
4. **Monitoring and Metrics:**
|
||||
- Regularly monitor Prometheus metrics to keep an eye on server performance, resource usage, and potential issues.
|
||||
- Set up alerts based on critical metrics to proactively handle problems.
|
||||
|
||||
5. **Logging:**
|
||||
- Ensure that log files are rotated and managed properly to prevent disk space issues.
|
||||
- Consider enabling JSON logging (`loggingjson = true`) for better integration with log management systems.
|
||||
|
||||
6. **Testing:**
|
||||
- Perform thorough testing of file uploads/downloads, especially with large files and under high load.
|
||||
- Test ClamAV scanning with both clean and malicious files to ensure security features are functioning correctly.
|
||||
|
||||
7. **Maintenance:**
|
||||
- Regularly update dependencies to benefit from security patches and performance improvements.
|
||||
- Clean up old file versions and ensure deduplication is functioning to optimize storage usage.
|
||||
|
||||
8. **Backup:**
|
||||
- Implement a backup strategy for critical data, especially if file versioning is enabled.
|
||||
|
||||
9. **Documentation:**
|
||||
- Keep the `README.md` and other documentation up-to-date with any code changes to assist in maintenance and onboarding new contributors.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Cannot Connect to Redis:**
|
||||
- Ensure Redis is running and accessible at the address specified in `redisaddr`.
|
||||
- Verify that the `redispassword` is correct if authentication is enabled.
|
||||
|
||||
- **ClamAV Scanning Fails:**
|
||||
- Check if ClamAV is installed and the `clamavsocket` path is correct.
|
||||
- Ensure that the ClamAV daemon (`clamd`) is running.
|
||||
|
||||
- **Insufficient Disk Space:**
|
||||
- Monitor the disk space and adjust `minfreebytes` in the configuration as needed.
|
||||
- Enable file TTL and cleanup to automatically remove old files.
|
||||
|
||||
- **Metrics Not Available:**
|
||||
- Ensure that `metricsenabled` is set to `true` and the server is running without errors.
|
||||
- Check if the specified `metricsport` is not blocked by a firewall.
|
||||
|
||||
- **Thumbnail Generation Not Working:**
|
||||
- Verify that the `[thumbnails].enabled` is set to `true` and the `directory` exists with appropriate permissions.
|
||||
- Check the logs for any errors related to image processing.
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please follow these steps to contribute:
|
||||
|
||||
1. **Fork the Repository**
|
||||
2. **Create a Feature Branch**
|
||||
|
||||
```bash
|
||||
git checkout -b feature/YourFeatureName
|
||||
```
|
||||
|
||||
3. **Commit Your Changes**
|
||||
|
||||
```bash
|
||||
git commit -m "Add your message here"
|
||||
```
|
||||
|
||||
4. **Push to the Branch**
|
||||
|
||||
```bash
|
||||
git push origin feature/YourFeatureName
|
||||
```
|
||||
|
||||
5. **Open a Pull Request**
|
||||
|
||||
Provide a clear description of your changes and the problem they address.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the [MIT License](LICENSE).
|
||||
- **Linux (arm64)**:
|
||||
```bash
|
||||
GOOS=linux GOARCH=arm64 go build -o hmac-file-server main.go
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
For a detailed list of changes, please refer to the [Changelog](./Changelog.md).
|
||||
### Added
|
||||
- **Deduplication Support:** Automatically remove duplicate files based on SHA256 hashing to save storage space.
|
||||
- **ISO Container Management:** Create and mount ISO containers for specialized storage needs, enhancing flexibility in file management.
|
||||
- **Prometheus Metrics Enhancements:** Added detailed metrics for deduplication and ISO container operations to improve monitoring and observability.
|
||||
- **Redis Integration Improvements:** Enhanced caching mechanisms using Redis for faster access to file metadata and application states.
|
||||
- **Precaching Feature:** Implemented precaching of file structures on startup to reduce access times for frequently used files.
|
||||
- **Configuration Options:** Updated `config.toml` to include new settings for deduplication, ISO management, and worker scaling.
|
||||
|
||||
By following this updated `README.md`, you can ensure that users and contributors have a clear understanding of the **HMAC File Server**'s capabilities, configuration options, and best practices for deployment and maintenance.
|
||||
### Changed
|
||||
- **Worker Pool Scaling:** Implemented dynamic adjustment of worker threads based on system resources to optimize performance.
|
||||
- **Logging Enhancements:** Improved logging for file operations, including detailed information on file extensions and MIME types during uploads.
|
||||
- **Temporary Path Configuration:** Replaced hardcoded temporary upload directories with a configurable `TempPath` parameter in `config.toml` for greater flexibility.
|
||||
|
||||
### Fixed
|
||||
- **Temporary File Handling:** Resolved issues where temporary `.tmp` files caused "Unsupported file type" warnings by enhancing MIME type detection logic.
|
||||
- **MIME Type Detection:** Improved MIME type detection to ensure better compatibility and accuracy during file uploads.
|
||||
|
||||
### Deprecated
|
||||
- **Thumbnail Support (Previous Implementation):** Dropped the previous thumbnail support mechanism. This feature will not return in future releases.
|
||||
|
||||
---
|
||||
|
||||
**Important Update:**
|
||||
- The minimum Go version required is now **1.20**. Please ensure your environment meets this requirement for successful compilation.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Alexander Renz
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
320
RELEASE-NOTES.MD
320
RELEASE-NOTES.MD
@ -1,294 +1,46 @@
|
||||
# HMAC File Server
|
||||
# Release Notes for HMAC File Server 2.6-Stable
|
||||
|
||||
**HMAC File Server** is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, and optional ClamAV scanning for file integrity and security. This server is built with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration.
|
||||
## Summary
|
||||
Version 2.6-Stable focuses on improving the overall stability and performance of the HMAC File Server. Significant changes have been made to prioritize reliability and scalability for production environments.
|
||||
|
||||
> **Credits:** The **HMAC File Server** is based on the source code of [Thomas Leister's prosody-filer](https://github.com/ThomasLeister/prosody-filer). Many features and design elements have been inspired or derived from this project.
|
||||
## Key Changes
|
||||
|
||||
---
|
||||
### Breaking Changes
|
||||
- **Thumbnail Generation Dropped**: Support for automatic thumbnail generation has been removed in this release. This decision was made to enhance system stability and reduce resource consumption. Users requiring thumbnails are encouraged to use external tools.
|
||||
|
||||
## Features
|
||||
### New Features
|
||||
- **ISO-Based Storage Support**: Introduced support for ISO-based storage to accommodate specialized use cases.
|
||||
- **Enhanced ClamAV Integration**: Improved ClamAV scanning with concurrent workers, providing better performance for large-scale deployments.
|
||||
- **Timeout Configuration**: Added granular timeout settings for read, write, and idle connections, improving connection management.
|
||||
- **FileNaming Configuration**: Added support for a "None" option in the `FileNaming` configuration. When set to "None", the filename remains unchanged.
|
||||
- **Example Configuration Generation**: If no configuration file is found, the server will output an example configuration for the user to copy and paste.
|
||||
- **Prometheus Metrics**: Enhanced Prometheus metrics for better monitoring and performance tracking. New metrics include upload and download durations, error counts, memory and CPU usage, and more.
|
||||
|
||||
- **HMAC Authentication:** Secure file uploads and downloads with HMAC tokens.
|
||||
- **File Versioning:** Enable versioning for uploaded files with configurable retention.
|
||||
- **Chunked and Resumable Uploads:** Handle large files efficiently with support for resumable and chunked uploads.
|
||||
- **ClamAV Scanning:** Optional virus scanning for uploaded files.
|
||||
- **Prometheus Metrics:** Monitor system and application-level metrics.
|
||||
- **Redis Integration:** Use Redis for caching or storing application states.
|
||||
- **File Expiration:** Automatically delete files after a specified TTL.
|
||||
- **Graceful Shutdown:** Handles signals and ensures proper cleanup.
|
||||
- **Deduplication:** Remove duplicate files based on hashing for storage efficiency.
|
||||
- **Auto-Adjust Worker Scaling:** Dynamically optimize HMAC and ClamAV workers based on system resources when enabled.
|
||||
- **Thumbnail Support:** Generate smaller versions of uploaded images for efficient storage and quick access.
|
||||
### Improvements
|
||||
- **Worker Management**: Auto-scaling worker threads based on system load for optimal performance.
|
||||
- **Logging Enhancements**: Improved log verbosity control, making debugging and monitoring easier.
|
||||
|
||||
---
|
||||
|
||||
## Repository
|
||||
|
||||
- **Primary Repository**: [GitHub Repository](https://github.com/PlusOne/hmac-file-server)
|
||||
- **Alternative Repository**: [uuxo.net Git Repository](https://git.uuxo.net/uuxo/hmac-file-server)
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Go 1.20+
|
||||
- Redis (optional, if Redis integration is enabled)
|
||||
- ClamAV (optional, if file scanning is enabled)
|
||||
|
||||
### Clone and Build
|
||||
|
||||
```bash
|
||||
# Clone from the primary repository
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
|
||||
# OR clone from the alternative repository
|
||||
git clone https://git.uuxo.net/uuxo/hmac-file-server.git
|
||||
|
||||
cd hmac-file-server
|
||||
go build -o hmac-file-server main.go
|
||||
```
|
||||
|
||||
### Building for Different Architectures
|
||||
|
||||
To build the HMAC File Server for different architectures, use the following commands:
|
||||
|
||||
#### Build for `arm64`
|
||||
```bash
|
||||
cd /path/to/hmac-file-server
|
||||
GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.2-stable_arm64 main.go
|
||||
```
|
||||
|
||||
#### Build for `amd64`
|
||||
```bash
|
||||
cd /path/to/hmac-file-server
|
||||
GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.2-stable_amd64 main.go
|
||||
```
|
||||
|
||||
Replace `/path/to/hmac-file-server` with the actual path to your project directory. These commands will generate the binaries for `arm64` and `amd64` architectures and place them in the `~/Temp` directory.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
The server configuration is managed through a `config.toml` file. Below are the supported configuration options:
|
||||
|
||||
### Auto-Adjust Feature
|
||||
|
||||
When `AutoAdjustWorkers` is enabled, the number of workers for HMAC operations and ClamAV scans is dynamically determined based on system resources. This ensures efficient resource utilization.
|
||||
|
||||
If `AutoAdjustWorkers = true`, the values for `NumWorkers` and `NumScanWorkers` in the configuration file will be ignored, and the server will automatically adjust these values.
|
||||
|
||||
### Network Events Monitoring
|
||||
|
||||
Setting `NetworkEvents = false` in the server configuration disables the logging and tracking of network-related events within the application. This means that functionalities such as monitoring IP changes or recording network activity will be turned off.
|
||||
|
||||
### Precaching
|
||||
|
||||
The `precaching` feature allows the server to pre-cache storage paths for faster access. This can improve performance by reducing the time needed to access frequently used storage paths.
|
||||
|
||||
### Thumbnail Support
|
||||
|
||||
- New configuration options in `[thumbnails]` to enable or disable generating image thumbnails, set the thumbnail size, and configure concurrency for thumbnail generation.
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### Deduplication Support
|
||||
|
||||
- **Description:** Added support for file deduplication to save storage space by storing a single copy of identical files.
|
||||
- **Configuration:**
|
||||
```toml
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/mnt/hmac-storage/deduplication/"
|
||||
```
|
||||
|
||||
### Thumbnail Support
|
||||
|
||||
- **Description:** Added support for thumbnail creation to generate smaller versions of uploaded images.
|
||||
- **Configuration:**
|
||||
```toml
|
||||
[thumbnails]
|
||||
enabled = true
|
||||
directory = "/mnt/hmac-storage/thumbnails/"
|
||||
size = "200x200"
|
||||
concurrency = 5
|
||||
thumbnailintervalscan = "24h"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example `config.toml`
|
||||
|
||||
```toml
|
||||
[server]
|
||||
ListenPort = "8080"
|
||||
UnixSocket = false
|
||||
StoragePath = "./uploads"
|
||||
LogLevel = "info"
|
||||
LogFile = ""
|
||||
MetricsEnabled = true
|
||||
MetricsPort = "9090"
|
||||
FileTTL = "1y"
|
||||
FileTTLEnabled = true # Enable or disable file TTL
|
||||
MinFreeBytes = "100MB"
|
||||
AutoAdjustWorkers = true # Enable auto-adjustment for worker scaling
|
||||
NetworkEvents = false # Disable logging and tracking of network-related events
|
||||
PIDFilePath = "./hmac_file_server.pid" # Path to PID file
|
||||
Precaching = true # Enable pre-caching of storage paths
|
||||
TempPath = "/tmp/uploads" # Configurable temporary upload directory
|
||||
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/mnt/nfs_vol01/hmac-storage/deduplication/"
|
||||
|
||||
[iso]
|
||||
Enabled = false
|
||||
Size = "2TB"
|
||||
MountPoint = "/mnt/iso"
|
||||
Charset = "utf-8"
|
||||
|
||||
[thumbnails]
|
||||
enabled = true
|
||||
directory = "/mnt/nfs_vol01/hmac-file-server/thumbnails/"
|
||||
size = "200x200"
|
||||
concurrency = 5
|
||||
thumbnailintervalscan = "24h"
|
||||
|
||||
[iso]
|
||||
enabled = false
|
||||
size = "1TB"
|
||||
mountpoint = "/mnt/nfs_vol01/hmac-file-server/iso/"
|
||||
charset = "utf-8"
|
||||
|
||||
[timeouts]
|
||||
ReadTimeout = "480s"
|
||||
WriteTimeout = "480s"
|
||||
IdleTimeout = "65s" # nginx/apache2 keep-alive 60s
|
||||
|
||||
[security]
|
||||
Secret = "changeme"
|
||||
|
||||
[versioning]
|
||||
EnableVersioning = false
|
||||
MaxVersions = 1
|
||||
|
||||
[uploads]
|
||||
ResumableUploadsEnabled = true
|
||||
ChunkedUploadsEnabled = true
|
||||
ChunkSize = "64MB"
|
||||
AllowedExtensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm", ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"]
|
||||
|
||||
[downloads]
|
||||
ResumableDownloadsEnabled = true
|
||||
ChunkedDownloadsEnabled = true
|
||||
ChunkSize = "64MB"
|
||||
|
||||
[clamav]
|
||||
ClamAVEnabled = false
|
||||
ClamAVSocket = "/var/run/clamav/clamd.ctl"
|
||||
NumScanWorkers = 2
|
||||
ScanFileExtensions = [".exe", ".dll", ".bin", ".com", ".bat", ".sh", ".php", ".js"]
|
||||
|
||||
[redis]
|
||||
RedisEnabled = false
|
||||
RedisAddr = "localhost:6379"
|
||||
RedisPassword = ""
|
||||
RedisDBIndex = 0
|
||||
RedisHealthCheckInterval = "120s"
|
||||
|
||||
[workers]
|
||||
NumWorkers = 4
|
||||
UploadQueueSize = 5000
|
||||
|
||||
[file]
|
||||
FileRevision = 1 # Revision number for file handling
|
||||
|
||||
[logging]
|
||||
level = "info"
|
||||
file = "/var/log/hmac-file-server.log"
|
||||
max_size = 100
|
||||
max_backups = 7
|
||||
max_age = 30
|
||||
compress = true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running the Server
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run the server with a configuration file:
|
||||
|
||||
```bash
|
||||
./hmac-file-server -config ./config.toml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Metrics Server
|
||||
|
||||
If `MetricsEnabled` is set to `true`, the Prometheus metrics server will be available on the port specified in `MetricsPort` (default: `9090`).
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
To run the server locally for development:
|
||||
|
||||
```bash
|
||||
go run main.go -config ./config.toml
|
||||
```
|
||||
|
||||
Use tools like **cURL** or **Postman** to test file uploads and downloads.
|
||||
|
||||
### Example File Upload with HMAC Token
|
||||
|
||||
```bash
|
||||
curl -X PUT -H "Authorization: Bearer <HMAC-TOKEN>" -F "file=@example.txt" http://localhost:8080/uploads/example.txt
|
||||
```
|
||||
|
||||
Replace `<HMAC-TOKEN>` with a valid HMAC signature
|
||||
|
||||
## [2.4.1] - 2025-03-10
|
||||
### Changed
|
||||
- **Configuration:** Updated `globalextensions` in `config.toml` to `["*"]`, allowing all file types globally for uploads. This change simplifies the configuration by removing the need to specify individual file extensions.
|
||||
|
||||
## [2.4.0] - 2025-02-20
|
||||
### Added
|
||||
- **Pre-Caching Support:** Introduced pre-caching of storage paths to improve access speeds.
|
||||
- **ISO Container Management:** Added functionality to create and mount ISO containers for specialized storage needs.
|
||||
- **Thumbnail Concurrency Parameter:** Users can now set the level of concurrency for thumbnail generation to optimize performance.
|
||||
|
||||
### Changed
|
||||
- **Configuration Options:** Updated `config.toml` to include new settings for pre-caching and ISO management.
|
||||
- **Documentation:** Enhanced `README.MD` with detailed instructions on new features and best practices.
|
||||
|
||||
### Fixed
|
||||
- **Bug Fixes:** Resolved minor issues related to file versioning and deduplication processes.
|
||||
|
||||
## [2.3.1] - 2025-01-15
|
||||
### Changed
|
||||
- **Configuration:** Updated `globalextensions` in `config.toml` to `["*"]`, allowing all file types globally for uploads. This change simplifies the configuration by removing the need to specify individual file extensions.
|
||||
|
||||
## [2.3.0] - 2024-12-28
|
||||
### Changed
|
||||
- **Server:** Replaced the hardcoded temporary upload directory `/tmp/uploads` with a configurable `TempPath` parameter in `config.toml`. Ensure to set `tempPath` in your configuration file accordingly.
|
||||
|
||||
## [2.2.2] - 2024-12-27
|
||||
### Bug Fixes
|
||||
- Resolved issue where temporary `.tmp` files caused "Unsupported file type" warnings by adjusting MIME type detection to use the final file extension.
|
||||
- Resolved minor issues affecting deduplication and file upload performance.
|
||||
- Fixed a rare crash scenario during high-concurrency file uploads.
|
||||
|
||||
### Enhancements
|
||||
- Improved logging for file extension and MIME type during uploads.
|
||||
## Migration Notes
|
||||
1. **Thumbnail Settings**: Remove `[thumbnails]` configuration blocks from your `config.toml` file to avoid errors.
|
||||
2. **Updated Configuration**: Review new timeout settings in `[timeouts]` and adjust as needed.
|
||||
3. **ISO Integration**: Configure the new `[iso]` block for environments utilizing ISO-based storage.
|
||||
4. **FileNaming Configuration**: Update the `FileNaming` setting in `[server]` to use the new "None" option if you want filenames to remain unchanged.
|
||||
|
||||
## [2.2.1] - 2024-12-27
|
||||
### Enhancements
|
||||
- Added detailed logging for file extensions and MIME types during file uploads to assist in diagnosing unsupported file type issues.
|
||||
[server]
|
||||
# FileNaming options: "HMAC", "None"
|
||||
FileNaming = "HMAC"
|
||||
|
||||
### Configuration
|
||||
- Updated `config.toml` to ensure necessary file extensions are allowed for uploads.
|
||||
## Recommendations
|
||||
- **Security**: Ensure that the HMAC secret key in `config.toml` is updated to a strong, unique value.
|
||||
- **Backups**: Regularly back up your `config.toml` and important data directories.
|
||||
- **Monitoring**: Leverage Prometheus metrics for real-time monitoring of server performance.
|
||||
|
||||
For a detailed guide on setting up and configuring the HMAC File Server, refer to the [README.md](./README.md).
|
||||
|
||||
---
|
||||
|
||||
Thank you for using HMAC File Server! If you encounter any issues, feel free to report them on our GitHub repository.
|
||||
|
@ -1,7 +1,6 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
@ -10,6 +9,10 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"context"
|
||||
"io"
|
||||
"sync"
|
||||
"bufio"
|
||||
|
||||
"github.com/gdamore/tcell/v2"
|
||||
"github.com/pelletier/go-toml"
|
||||
@ -20,9 +23,13 @@ import (
|
||||
"github.com/shirou/gopsutil/v3/process"
|
||||
)
|
||||
|
||||
var prometheusURL string
|
||||
var configFilePath string // Pfad der gefundenen Konfiguration
|
||||
var logFilePath string // Pfad der Logdatei aus der Konfiguration
|
||||
var (
|
||||
prometheusURL string
|
||||
configFilePath string // Pfad der gefundenen Konfiguration
|
||||
logFilePath string // Pfad der Logdatei aus der Konfiguration
|
||||
metricsEnabled bool // Neue Variable für die Aktivierung von Metriken
|
||||
bindIP string // Neue Variable für die gebundene IP-Adresse
|
||||
)
|
||||
|
||||
func init() {
|
||||
configPaths := []string{
|
||||
@ -45,7 +52,7 @@ func init() {
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
log.Fatalf("Error loading config file: %v", err)
|
||||
log.Fatalf("Error loading config file: %v\nPlease create a config.toml in one of the following locations:\n%v", err, configPaths)
|
||||
}
|
||||
|
||||
// Metricsport auslesen
|
||||
@ -69,7 +76,35 @@ func init() {
|
||||
log.Fatalf("Error: 'server.metricsport' is not of type int64 or string, got %T", v)
|
||||
}
|
||||
|
||||
prometheusURL = fmt.Sprintf("http://localhost:%d/metrics", port)
|
||||
// Lesen von 'metricsenabled' aus der Konfiguration
|
||||
metricsEnabledValue := config.Get("server.metricsenabled")
|
||||
if metricsEnabledValue == nil {
|
||||
log.Println("Warning: 'server.metricsenabled' ist in der Konfiguration nicht gesetzt. Standardmäßig deaktiviert.")
|
||||
metricsEnabled = false
|
||||
} else {
|
||||
var ok bool
|
||||
metricsEnabled, ok = metricsEnabledValue.(bool)
|
||||
if !ok {
|
||||
log.Fatalf("Konfigurationsfehler: 'server.metricsenabled' sollte ein boolescher Wert sein, aber %T wurde gefunden.", metricsEnabledValue)
|
||||
}
|
||||
}
|
||||
|
||||
// Lesen von 'bind_ip' aus der Konfiguration
|
||||
bindIPValue := config.Get("server.bind_ip")
|
||||
if bindIPValue == nil {
|
||||
log.Println("Warning: 'server.bind_ip' ist in der Konfiguration nicht gesetzt. Standardmäßig auf 'localhost' gesetzt.")
|
||||
bindIP = "localhost"
|
||||
} else {
|
||||
var ok bool
|
||||
bindIP, ok = bindIPValue.(string)
|
||||
if !ok {
|
||||
log.Fatalf("Konfigurationsfehler: 'server.bind_ip' sollte ein String sein, aber %T wurde gefunden.", bindIPValue)
|
||||
}
|
||||
}
|
||||
|
||||
// Konstruktion der prometheusURL basierend auf 'bind_ip' und 'metricsport'
|
||||
prometheusURL = fmt.Sprintf("http://%s:%d/metrics", bindIP, port)
|
||||
log.Printf("Metrics URL gesetzt auf: %s", prometheusURL)
|
||||
|
||||
// Log-Datei auslesen über server.logfile
|
||||
logFileValue := config.Get("server.logfile")
|
||||
@ -93,11 +128,17 @@ const (
|
||||
|
||||
// ProcessInfo holds information about a process
|
||||
type ProcessInfo struct {
|
||||
PID int32
|
||||
Name string
|
||||
CPUPercent float64
|
||||
MemPercent float32
|
||||
CommandLine string
|
||||
PID int32
|
||||
Name string
|
||||
CPUPercent float64
|
||||
MemPercent float32
|
||||
CommandLine string
|
||||
Uptime string // Neues Feld für die Uptime
|
||||
Status string // Neues Feld für den Status
|
||||
ErrorCount int // Neues Feld für die Anzahl der Fehler
|
||||
TotalRequests int64 // Neues Feld für die Gesamtanzahl der Anfragen
|
||||
ActiveConnections int // Neues Feld für aktive Verbindungen
|
||||
AverageResponseTime float64 // Neues Feld für die durchschnittliche Antwortzeit in Millisekunden
|
||||
}
|
||||
|
||||
// Function to fetch and parse Prometheus metrics
|
||||
@ -121,7 +162,9 @@ func fetchMetrics() (map[string]float64, error) {
|
||||
name == "memory_usage_bytes" ||
|
||||
name == "cpu_usage_percent" ||
|
||||
name == "active_connections_total" ||
|
||||
name == "goroutines_count" {
|
||||
name == "goroutines_count" ||
|
||||
name == "total_requests" ||
|
||||
name == "average_response_time_ms" {
|
||||
|
||||
for _, m := range mf.GetMetric() {
|
||||
var value float64
|
||||
@ -179,7 +222,7 @@ func fetchSystemData() (float64, float64, int, error) {
|
||||
return v.UsedPercent, cpuUsage, cores, nil
|
||||
}
|
||||
|
||||
// Function to fetch process list
|
||||
// Funktion zum Abrufen der Prozessliste mit paralleler Verarbeitung
|
||||
func fetchProcessList() ([]ProcessInfo, error) {
|
||||
processes, err := process.Processes()
|
||||
if err != nil {
|
||||
@ -187,37 +230,55 @@ func fetchProcessList() ([]ProcessInfo, error) {
|
||||
}
|
||||
|
||||
var processList []ProcessInfo
|
||||
var mu sync.Mutex
|
||||
var wg sync.WaitGroup
|
||||
|
||||
// Begrenzung der gleichzeitigen Goroutinen auf 10
|
||||
sem := make(chan struct{}, 10)
|
||||
|
||||
for _, p := range processes {
|
||||
cpuPercent, err := p.CPUPercent()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
sem <- struct{}{} // Eintritt in semaphor
|
||||
|
||||
memPercent, err := p.MemoryPercent()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
go func(p *process.Process) {
|
||||
defer wg.Done()
|
||||
defer func() { <-sem }() // Austritt aus semaphor
|
||||
|
||||
name, err := p.Name()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
cpuPercent, err := p.CPUPercent()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
cmdline, err := p.Cmdline()
|
||||
if err != nil {
|
||||
cmdline = ""
|
||||
}
|
||||
memPercent, err := p.MemoryPercent()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
processList = append(processList, ProcessInfo{
|
||||
PID: p.Pid,
|
||||
Name: name,
|
||||
CPUPercent: cpuPercent,
|
||||
MemPercent: memPercent,
|
||||
CommandLine: cmdline,
|
||||
})
|
||||
name, err := p.Name()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
cmdline, err := p.Cmdline()
|
||||
if err != nil {
|
||||
cmdline = ""
|
||||
}
|
||||
|
||||
info := ProcessInfo{
|
||||
PID: p.Pid,
|
||||
Name: name,
|
||||
CPUPercent: cpuPercent,
|
||||
MemPercent: memPercent,
|
||||
CommandLine: cmdline,
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
processList = append(processList, info)
|
||||
mu.Unlock()
|
||||
}(p)
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
return processList, nil
|
||||
}
|
||||
|
||||
@ -250,12 +311,57 @@ func fetchHmacFileServerInfo() (*ProcessInfo, error) {
|
||||
cmdline = ""
|
||||
}
|
||||
|
||||
createTime, err := p.CreateTime()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get process start time: %w", err)
|
||||
}
|
||||
uptime := time.Since(time.Unix(0, createTime*int64(time.Millisecond)))
|
||||
|
||||
status := "Running" // Standardstatus
|
||||
|
||||
// Überprüfung, ob der Prozess aktiv ist
|
||||
isRunning, err := p.IsRunning()
|
||||
if err != nil || !isRunning {
|
||||
status = "Stopped"
|
||||
}
|
||||
|
||||
errorCount, err := countHmacErrors()
|
||||
if err != nil {
|
||||
errorCount = 0
|
||||
}
|
||||
|
||||
metrics, err := fetchMetrics()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to fetch metrics: %w", err)
|
||||
}
|
||||
|
||||
totalRequests, ok := metrics["total_requests"]
|
||||
if !ok {
|
||||
totalRequests = 0
|
||||
}
|
||||
|
||||
activeConnections, ok := metrics["active_connections_total"]
|
||||
if !ok {
|
||||
activeConnections = 0
|
||||
}
|
||||
|
||||
averageResponseTime, ok := metrics["average_response_time_ms"]
|
||||
if !ok {
|
||||
averageResponseTime = 0.0
|
||||
}
|
||||
|
||||
return &ProcessInfo{
|
||||
PID: p.Pid,
|
||||
Name: name,
|
||||
CPUPercent: cpuPercent,
|
||||
MemPercent: memPercent,
|
||||
CommandLine: cmdline,
|
||||
PID: p.Pid,
|
||||
Name: name,
|
||||
CPUPercent: cpuPercent,
|
||||
MemPercent: memPercent,
|
||||
CommandLine: cmdline,
|
||||
Uptime: uptime.String(),
|
||||
Status: status,
|
||||
ErrorCount: errorCount,
|
||||
TotalRequests: int64(totalRequests),
|
||||
ActiveConnections: int(activeConnections),
|
||||
AverageResponseTime: averageResponseTime,
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
@ -263,63 +369,203 @@ func fetchHmacFileServerInfo() (*ProcessInfo, error) {
|
||||
return nil, fmt.Errorf("hmac-file-server process not found")
|
||||
}
|
||||
|
||||
// Function to update the UI with the latest data
|
||||
func updateUI(app *tview.Application, pages *tview.Pages, sysPage, hmacPage tview.Primitive) {
|
||||
// Neue Funktion zur Zählung der Fehler in den Logs
|
||||
func countHmacErrors() (int, error) {
|
||||
logFilePath := "/var/log/hmac-file-server.log" // Pfad zur Logdatei
|
||||
file, err := os.Open(logFilePath)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
scanner := bufio.NewScanner(file)
|
||||
errorCount := 0
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
if strings.Contains(line, "level=error") {
|
||||
errorCount++
|
||||
}
|
||||
}
|
||||
|
||||
if err := scanner.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return errorCount, nil
|
||||
}
|
||||
|
||||
// Funktion zur Aktualisierung der UI mit paralleler Datenbeschaffung
|
||||
func updateUI(ctx context.Context, app *tview.Application, pages *tview.Pages, sysPage, hmacPage tview.Primitive) {
|
||||
ticker := time.NewTicker(2 * time.Second)
|
||||
defer ticker.Stop()
|
||||
|
||||
for range ticker.C {
|
||||
// Fetch data for both views
|
||||
memUsage, cpuUsage, cores, err := fetchSystemData()
|
||||
if err != nil {
|
||||
log.Printf("Error fetching system data: %v\n", err)
|
||||
continue
|
||||
}
|
||||
|
||||
metrics, err := fetchMetrics()
|
||||
if err != nil {
|
||||
log.Printf("Error fetching metrics: %v\n", err)
|
||||
continue
|
||||
}
|
||||
|
||||
processes, err := fetchProcessList()
|
||||
if err != nil {
|
||||
log.Printf("Error fetching process list: %v\n", err)
|
||||
continue
|
||||
}
|
||||
|
||||
hmacInfo, err := fetchHmacFileServerInfo()
|
||||
if err != nil {
|
||||
log.Printf("Error fetching hmac-file-server info: %v\n", err)
|
||||
}
|
||||
|
||||
// Update the UI
|
||||
app.QueueUpdateDraw(func() {
|
||||
// Update system page
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
|
||||
sysFlex := sysPage.(*tview.Flex)
|
||||
|
||||
// Update system data table
|
||||
sysTable := sysFlex.GetItem(0).(*tview.Table)
|
||||
updateSystemTable(sysTable, memUsage, cpuUsage, cores)
|
||||
|
||||
// Update metrics table
|
||||
metricsTable := sysFlex.GetItem(1).(*tview.Table)
|
||||
updateMetricsTable(metricsTable, metrics)
|
||||
|
||||
// Update process table
|
||||
processTable := sysFlex.GetItem(2).(*tview.Table)
|
||||
updateProcessTable(processTable, processes)
|
||||
}
|
||||
|
||||
// Update hmac-file-server page
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "hmac" && hmacInfo != nil {
|
||||
hmacFlex := hmacPage.(*tview.Flex)
|
||||
hmacTable := hmacFlex.GetItem(0).(*tview.Table)
|
||||
updateHmacTable(hmacTable, hmacInfo, metrics)
|
||||
}
|
||||
// Einführung von Channels für verschiedene Daten
|
||||
systemDataCh := make(chan struct {
|
||||
memUsage float64
|
||||
cpuUsage float64
|
||||
cores int
|
||||
err error
|
||||
})
|
||||
var metricsCh chan struct {
|
||||
metrics map[string]float64
|
||||
err error
|
||||
}
|
||||
if metricsEnabled {
|
||||
metricsCh = make(chan struct {
|
||||
metrics map[string]float64
|
||||
err error
|
||||
})
|
||||
}
|
||||
processListCh := make(chan struct {
|
||||
processes []ProcessInfo
|
||||
err error
|
||||
})
|
||||
hmacInfoCh := make(chan struct {
|
||||
info *ProcessInfo
|
||||
metrics map[string]float64
|
||||
err error
|
||||
})
|
||||
|
||||
// Goroutine zur Datenbeschaffung
|
||||
go func() {
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
close(systemDataCh)
|
||||
if metricsEnabled {
|
||||
close(metricsCh)
|
||||
}
|
||||
close(processListCh)
|
||||
close(hmacInfoCh)
|
||||
return
|
||||
case <-ticker.C:
|
||||
// Systemdaten abrufen asynchron
|
||||
go func() {
|
||||
memUsage, cpuUsage, cores, err := fetchSystemData()
|
||||
systemDataCh <- struct {
|
||||
memUsage float64
|
||||
cpuUsage float64
|
||||
cores int
|
||||
err error
|
||||
}{memUsage, cpuUsage, cores, err}
|
||||
}()
|
||||
|
||||
if metricsEnabled {
|
||||
// Metriken abrufen asynchron
|
||||
go func() {
|
||||
metrics, err := fetchMetrics()
|
||||
metricsCh <- struct {
|
||||
metrics map[string]float64
|
||||
err error
|
||||
}{metrics, err}
|
||||
}()
|
||||
}
|
||||
|
||||
// Prozessliste abrufen asynchron
|
||||
go func() {
|
||||
processes, err := fetchProcessList()
|
||||
processListCh <- struct {
|
||||
processes []ProcessInfo
|
||||
err error
|
||||
}{processes, err}
|
||||
}()
|
||||
|
||||
// hmac-file-server Informationen abrufen asynchron
|
||||
go func() {
|
||||
hmacInfo, err := fetchHmacFileServerInfo()
|
||||
var metrics map[string]float64
|
||||
if metricsEnabled {
|
||||
metrics, err = fetchMetrics()
|
||||
}
|
||||
hmacInfoCh <- struct {
|
||||
info *ProcessInfo
|
||||
metrics map[string]float64
|
||||
err error
|
||||
}{hmacInfo, metrics, err}
|
||||
}()
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case data, ok := <-systemDataCh:
|
||||
if !ok {
|
||||
systemDataCh = nil
|
||||
continue
|
||||
}
|
||||
if data.err != nil {
|
||||
log.Printf("Fehler beim Abrufen der Systemdaten: %v\n", data.err)
|
||||
continue
|
||||
}
|
||||
// UI aktualisieren mit Systemdaten
|
||||
app.QueueUpdateDraw(func() {
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
|
||||
sysFlex := sysPage.(*tview.Flex)
|
||||
sysTable := sysFlex.GetItem(0).(*tview.Table)
|
||||
updateSystemTable(sysTable, data.memUsage, data.cpuUsage, data.cores)
|
||||
}
|
||||
})
|
||||
case data, ok := <-metricsCh:
|
||||
if !ok {
|
||||
metricsCh = nil
|
||||
continue
|
||||
}
|
||||
if data.err != nil {
|
||||
log.Printf("Fehler beim Abrufen der Metriken: %v\n", data.err)
|
||||
continue
|
||||
}
|
||||
// UI aktualisieren mit Metriken
|
||||
app.QueueUpdateDraw(func() {
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
|
||||
sysFlex := sysPage.(*tview.Flex)
|
||||
metricsTable := sysFlex.GetItem(1).(*tview.Table)
|
||||
updateMetricsTable(metricsTable, data.metrics)
|
||||
}
|
||||
})
|
||||
case data, ok := <-processListCh:
|
||||
if !ok {
|
||||
processListCh = nil
|
||||
continue
|
||||
}
|
||||
if data.err != nil {
|
||||
log.Printf("Fehler beim Abrufen der Prozessliste: %v\n", data.err)
|
||||
continue
|
||||
}
|
||||
// UI aktualisieren mit Prozessliste
|
||||
app.QueueUpdateDraw(func() {
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
|
||||
sysFlex := sysPage.(*tview.Flex)
|
||||
processTable := sysFlex.GetItem(2).(*tview.Table)
|
||||
updateProcessTable(processTable, data.processes)
|
||||
}
|
||||
})
|
||||
case data, ok := <-hmacInfoCh:
|
||||
if !ok {
|
||||
hmacInfoCh = nil
|
||||
continue
|
||||
}
|
||||
if data.err != nil {
|
||||
log.Printf("Fehler beim Abrufen der hmac-file-server Informationen: %v\n", data.err)
|
||||
continue
|
||||
}
|
||||
// UI aktualisieren mit hmac-file-server Informationen
|
||||
app.QueueUpdateDraw(func() {
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "hmac" && data.info != nil {
|
||||
hmacFlex := hmacPage.(*tview.Flex)
|
||||
hmacTable := hmacFlex.GetItem(0).(*tview.Table)
|
||||
updateHmacTable(hmacTable, data.info, data.metrics)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Abbruchbedingung, wenn alle Channels geschlossen sind
|
||||
if systemDataCh == nil && (!metricsEnabled || metricsCh == nil) && processListCh == nil && hmacInfoCh == nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Helper function to update system data table
|
||||
@ -419,9 +665,27 @@ func updateHmacTable(hmacTable *tview.Table, hmacInfo *ProcessInfo, metrics map[
|
||||
|
||||
hmacTable.SetCell(4, 0, tview.NewTableCell("Command"))
|
||||
hmacTable.SetCell(4, 1, tview.NewTableCell(hmacInfo.CommandLine))
|
||||
|
||||
hmacTable.SetCell(5, 0, tview.NewTableCell("Uptime"))
|
||||
hmacTable.SetCell(5, 1, tview.NewTableCell(hmacInfo.Uptime)) // Neue Zeile für Uptime
|
||||
|
||||
hmacTable.SetCell(6, 0, tview.NewTableCell("Status"))
|
||||
hmacTable.SetCell(6, 1, tview.NewTableCell(hmacInfo.Status)) // Neue Zeile für Status
|
||||
|
||||
hmacTable.SetCell(7, 0, tview.NewTableCell("Error Count"))
|
||||
hmacTable.SetCell(7, 1, tview.NewTableCell(fmt.Sprintf("%d", hmacInfo.ErrorCount))) // Neue Zeile für Error Count
|
||||
|
||||
hmacTable.SetCell(8, 0, tview.NewTableCell("Total Requests"))
|
||||
hmacTable.SetCell(8, 1, tview.NewTableCell(fmt.Sprintf("%d", hmacInfo.TotalRequests))) // Neue Zeile für Total Requests
|
||||
|
||||
hmacTable.SetCell(9, 0, tview.NewTableCell("Active Connections"))
|
||||
hmacTable.SetCell(9, 1, tview.NewTableCell(fmt.Sprintf("%d", hmacInfo.ActiveConnections))) // Neue Zeile für Active Connections
|
||||
|
||||
hmacTable.SetCell(10, 0, tview.NewTableCell("Avg. Response Time (ms)"))
|
||||
hmacTable.SetCell(10, 1, tview.NewTableCell(fmt.Sprintf("%.2f", hmacInfo.AverageResponseTime))) // Neue Zeile für Average Response Time
|
||||
|
||||
// Metrics related to hmac-file-server
|
||||
row := 6
|
||||
row := 12
|
||||
hmacTable.SetCell(row, 0, tview.NewTableCell("Metric").SetAttributes(tcell.AttrBold))
|
||||
hmacTable.SetCell(row, 1, tview.NewTableCell("Value").SetAttributes(tcell.AttrBold))
|
||||
row++
|
||||
@ -469,72 +733,117 @@ func createHmacPage() tview.Primitive {
|
||||
return hmacFlex
|
||||
}
|
||||
|
||||
func createLogsPage(logFilePath string) tview.Primitive {
|
||||
logsTextView := tview.NewTextView().
|
||||
SetDynamicColors(true).
|
||||
SetRegions(true).
|
||||
SetWordWrap(true)
|
||||
logsTextView.SetTitle(" [::b]Logs ").SetBorder(true)
|
||||
func createLogsPage(ctx context.Context, app *tview.Application, logFilePath string) tview.Primitive {
|
||||
logsTextView := tview.NewTextView().
|
||||
SetDynamicColors(true).
|
||||
SetRegions(true).
|
||||
SetWordWrap(true)
|
||||
logsTextView.SetTitle(" [::b]Logs ").SetBorder(true)
|
||||
|
||||
const numLines = 100 // Number of lines to read from the end of the log file
|
||||
const numLines = 100 // Number of lines to read from the end of the log file
|
||||
|
||||
// Read logs periodically
|
||||
go func() {
|
||||
for {
|
||||
content, err := readLastNLines(logFilePath, numLines)
|
||||
if err != nil {
|
||||
logsTextView.SetText(fmt.Sprintf("[red]Error reading log file: %v[white]", err))
|
||||
} else {
|
||||
// Process the log content to add colors
|
||||
lines := strings.Split(content, "\n")
|
||||
var coloredLines []string
|
||||
for _, line := range lines {
|
||||
if strings.Contains(line, "level=info") {
|
||||
coloredLines = append(coloredLines, "[green]"+line+"[white]")
|
||||
} else if strings.Contains(line, "level=warn") {
|
||||
coloredLines = append(coloredLines, "[yellow]"+line+"[white]")
|
||||
} else if strings.Contains(line, "level=error") {
|
||||
coloredLines = append(coloredLines, "[red]"+line+"[white]")
|
||||
} else {
|
||||
// Default color
|
||||
coloredLines = append(coloredLines, line)
|
||||
}
|
||||
}
|
||||
logsTextView.SetText(strings.Join(coloredLines, "\n"))
|
||||
}
|
||||
time.Sleep(2 * time.Second) // Refresh interval for logs
|
||||
}
|
||||
}()
|
||||
// Read logs periodically
|
||||
go func() {
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
default:
|
||||
content, err := readLastNLines(logFilePath, numLines)
|
||||
if err != nil {
|
||||
app.QueueUpdateDraw(func() {
|
||||
logsTextView.SetText(fmt.Sprintf("[red]Error reading log file: %v[white]", err))
|
||||
})
|
||||
} else {
|
||||
// Process the log content to add colors
|
||||
lines := strings.Split(content, "\n")
|
||||
var coloredLines []string
|
||||
for _, line := range lines {
|
||||
if strings.Contains(line, "level=info") {
|
||||
coloredLines = append(coloredLines, "[green]"+line+"[white]")
|
||||
} else if strings.Contains(line, "level=warn") {
|
||||
coloredLines = append(coloredLines, "[yellow]"+line+"[white]")
|
||||
} else if strings.Contains(line, "level=error") {
|
||||
coloredLines = append(coloredLines, "[red]"+line+"[white]")
|
||||
} else {
|
||||
// Default color
|
||||
coloredLines = append(coloredLines, line)
|
||||
}
|
||||
}
|
||||
app.QueueUpdateDraw(func() {
|
||||
logsTextView.SetText(strings.Join(coloredLines, "\n"))
|
||||
})
|
||||
}
|
||||
time.Sleep(2 * time.Second) // Refresh interval for logs
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
return logsTextView
|
||||
return logsTextView
|
||||
}
|
||||
|
||||
// Optimized readLastNLines to handle large files efficiently
|
||||
func readLastNLines(filePath string, n int) (string, error) {
|
||||
file, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer file.Close()
|
||||
file, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
var lines []string
|
||||
scanner := bufio.NewScanner(file)
|
||||
for scanner.Scan() {
|
||||
lines = append(lines, scanner.Text())
|
||||
if len(lines) > n {
|
||||
lines = lines[1:]
|
||||
}
|
||||
}
|
||||
const bufferSize = 1024
|
||||
buffer := make([]byte, bufferSize)
|
||||
var content []byte
|
||||
var fileSize int64
|
||||
|
||||
if err := scanner.Err(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
fileInfo, err := file.Stat()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
fileSize = fileInfo.Size()
|
||||
|
||||
return strings.Join(lines, "\n"), nil
|
||||
var offset int64 = 0
|
||||
for {
|
||||
if fileSize-offset < bufferSize {
|
||||
offset = fileSize
|
||||
} else {
|
||||
offset += bufferSize
|
||||
}
|
||||
|
||||
_, err := file.Seek(-offset, io.SeekEnd)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
bytesRead, err := file.Read(buffer)
|
||||
if err != nil && err != io.EOF {
|
||||
return "", err
|
||||
}
|
||||
|
||||
content = append(buffer[:bytesRead], content...)
|
||||
|
||||
if bytesRead < bufferSize || len(strings.Split(string(content), "\n")) > n+1 {
|
||||
break
|
||||
}
|
||||
|
||||
if offset >= fileSize {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
lines := strings.Split(string(content), "\n")
|
||||
if len(lines) > n {
|
||||
lines = lines[len(lines)-n:]
|
||||
}
|
||||
return strings.Join(lines, "\n"), nil
|
||||
}
|
||||
|
||||
func main() {
|
||||
app := tview.NewApplication()
|
||||
|
||||
// Create a cancellable context
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
// Create pages
|
||||
pages := tview.NewPages()
|
||||
|
||||
@ -547,14 +856,15 @@ func main() {
|
||||
pages.AddPage("hmac", hmacPage, true, false)
|
||||
|
||||
// Logs page mit dem gelesenen logFilePath
|
||||
logsPage := createLogsPage(logFilePath)
|
||||
logsPage := createLogsPage(ctx, app, logFilePath)
|
||||
pages.AddPage("logs", logsPage, true, false)
|
||||
|
||||
// Add key binding to switch views
|
||||
// Add key binding to switch views and handle exit
|
||||
app.SetInputCapture(func(event *tcell.EventKey) *tcell.EventKey {
|
||||
if event.Key() == tcell.KeyRune {
|
||||
switch event.Rune() {
|
||||
case 'q', 'Q':
|
||||
cancel()
|
||||
app.Stop()
|
||||
return nil
|
||||
case 's', 'S':
|
||||
@ -572,10 +882,11 @@ func main() {
|
||||
})
|
||||
|
||||
// Start the UI update loop in a separate goroutine
|
||||
go updateUI(app, pages, sysPage, hmacPage)
|
||||
go updateUI(ctx, app, pages, sysPage, hmacPage)
|
||||
|
||||
// Set the root and run the application
|
||||
if err := app.SetRoot(pages, true).EnableMouse(true).Run(); err != nil {
|
||||
log.Fatalf("Error running application: %v", err)
|
||||
log.Fatalf("Error running application: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
Binary file not shown.
File diff suppressed because it is too large
Load Diff
@ -52,17 +52,16 @@ func parseSize(sizeStr string) (int64, error) {
|
||||
valueStr := sizeStr[:len(sizeStr)-2]
|
||||
value, err := strconv.Atoi(valueStr)
|
||||
if err != nil {
|
||||
log.WithError(err).Errorf("Failed to parse size from input: %s", sizeStr)
|
||||
return 0, err
|
||||
return 0, fmt.Errorf("invalid size value: %v", err)
|
||||
}
|
||||
|
||||
switch unit {
|
||||
case "KB":
|
||||
return int64(value) * 1 << 10, nil
|
||||
return int64(value) * 1024, nil
|
||||
case "MB":
|
||||
return int64(value) * 1 << 20, nil
|
||||
return int64(value) * 1024 * 1024, nil
|
||||
case "GB":
|
||||
return int64(value) * 1 << 30, nil
|
||||
return int64(value) * 1024 * 1024 * 1024, nil
|
||||
default:
|
||||
return 0, fmt.Errorf("unknown size unit: %s", unit)
|
||||
}
|
||||
@ -72,7 +71,7 @@ func parseSize(sizeStr string) (int64, error) {
|
||||
func parseTTL(ttlStr string) (time.Duration, error) {
|
||||
ttlStr = strings.ToLower(strings.TrimSpace(ttlStr))
|
||||
if ttlStr == "" {
|
||||
return 0, fmt.Errorf("empty TTL string")
|
||||
return 0, fmt.Errorf("TTL string cannot be empty")
|
||||
}
|
||||
var valueStr string
|
||||
var unit rune
|
||||
@ -89,16 +88,20 @@ func parseTTL(ttlStr string) (time.Duration, error) {
|
||||
return 0, fmt.Errorf("invalid TTL value: %v", err)
|
||||
}
|
||||
switch unit {
|
||||
case 'h': // hours
|
||||
case 's':
|
||||
return time.Duration(val) * time.Second, nil
|
||||
case 'm':
|
||||
return time.Duration(val) * time.Minute, nil
|
||||
case 'h':
|
||||
return time.Duration(val) * time.Hour, nil
|
||||
case 'd': // days
|
||||
return time.Duration(val*24) * time.Hour, nil
|
||||
case 'm': // months (approx. 30 days)
|
||||
return time.Duration(val*24*30) * time.Hour, nil
|
||||
case 'y': // years (approx. 365 days)
|
||||
return time.Duration(val*24*365) * time.Hour, nil
|
||||
default: // fallback to Go's standard parsing
|
||||
return time.ParseDuration(ttlStr)
|
||||
case 'd':
|
||||
return time.Duration(val) * 24 * time.Hour, nil
|
||||
case 'w':
|
||||
return time.Duration(val) * 7 * 24 * time.Hour, nil
|
||||
case 'y':
|
||||
return time.Duration(val) * 365 * 24 * time.Hour, nil
|
||||
default:
|
||||
return 0, fmt.Errorf("unknown TTL unit: %c", unit)
|
||||
}
|
||||
}
|
||||
|
||||
@ -113,24 +116,26 @@ type LoggingConfig struct {
|
||||
}
|
||||
|
||||
type ServerConfig struct {
|
||||
BindIP string `mapstructure:"bind_ip"`
|
||||
ListenPort string `mapstructure:"listenport"`
|
||||
UnixSocket bool `mapstructure:"unixsocket"`
|
||||
StoragePath string `mapstructure:"storagepath"`
|
||||
LogFile string `mapstructure:"logfile"` // NEW field
|
||||
MetricsEnabled bool `mapstructure:"metricsenabled"`
|
||||
MetricsPort string `mapstructure:"metricsport"`
|
||||
FileTTL string `mapstructure:"filettl"`
|
||||
MinFreeBytes string `mapstructure:"minfreebytes"`
|
||||
FileTTL string `mapstructure:"filettl"`
|
||||
FileTTLEnabled bool `mapstructure:"filettlenabled"`
|
||||
AutoAdjustWorkers bool `mapstructure:"autoadjustworkers"`
|
||||
NetworkEvents bool `mapstructure:"networkevents"`
|
||||
TempPath string `mapstructure:"temppath"`
|
||||
LoggingJSON bool `mapstructure:"loggingjson"`
|
||||
PIDFilePath string `mapstructure:"pidfilepath"`
|
||||
CleanUponExit bool `mapstructure:"cleanuponexit"`
|
||||
PreCaching bool `mapstructure:"precaching"`
|
||||
FileTTLEnabled bool `mapstructure:"filettlenabled"`
|
||||
DeduplicationEnabled bool `mapstructure:"deduplicationenabled"`
|
||||
Logging LoggingConfig `mapstructure:"logging"`
|
||||
GlobalExtensions []string `mapstructure:"globalextensions"`
|
||||
FileNaming string `mapstructure:"filenaming"`
|
||||
// Removed TempPath, LoggingJSON
|
||||
}
|
||||
|
||||
type DeduplicationConfig struct {
|
||||
@ -196,7 +201,6 @@ type WorkersConfig struct {
|
||||
}
|
||||
|
||||
type FileConfig struct {
|
||||
FileRevision int `mapstructure:"filerevision"`
|
||||
}
|
||||
|
||||
type BuildConfig struct {
|
||||
@ -325,7 +329,8 @@ func writePIDFile(pidPath string) error {
|
||||
pidStr := strconv.Itoa(pid)
|
||||
err := os.WriteFile(pidPath, []byte(pidStr), 0644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to write PID file: %v", err)
|
||||
log.Errorf("Failed to write PID file: %v", err) // Improved error logging
|
||||
return err
|
||||
}
|
||||
log.Infof("PID %d written to %s", pid, pidPath)
|
||||
return nil
|
||||
@ -335,9 +340,9 @@ func writePIDFile(pidPath string) error {
|
||||
func removePIDFile(pidPath string) {
|
||||
err := os.Remove(pidPath)
|
||||
if err != nil {
|
||||
log.Warnf("failed to remove PID file %s: %v", pidPath, err)
|
||||
log.Errorf("Failed to remove PID file: %v", err) // Improved error logging
|
||||
} else {
|
||||
log.Infof("PID file %s removed", pidPath)
|
||||
log.Infof("PID file %s removed successfully", pidPath)
|
||||
}
|
||||
}
|
||||
|
||||
@ -403,7 +408,8 @@ func main() {
|
||||
os.Exit(1)
|
||||
}
|
||||
} else {
|
||||
fmt.Println("No configuration file found.")
|
||||
fmt.Println("No configuration file found. Please create a config file with the following content:")
|
||||
printExampleConfig()
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@ -411,7 +417,11 @@ func main() {
|
||||
|
||||
err := readConfig(configFile, &conf)
|
||||
if err != nil {
|
||||
log.Fatalf("Error reading config: %v", err)
|
||||
log.Fatalf("Failed to load configuration: %v\nPlease ensure your config.toml is present at one of the following paths:\n%v", err, []string{
|
||||
"/etc/hmac-file-server/config.toml",
|
||||
"../config.toml",
|
||||
"./config.toml",
|
||||
})
|
||||
}
|
||||
log.Info("Configuration loaded successfully.")
|
||||
|
||||
@ -442,13 +452,13 @@ func main() {
|
||||
log.Infof("Server MinFreeBytes: %s", conf.Server.MinFreeBytes)
|
||||
log.Infof("Server AutoAdjustWorkers: %v", conf.Server.AutoAdjustWorkers)
|
||||
log.Infof("Server NetworkEvents: %v", conf.Server.NetworkEvents)
|
||||
log.Infof("Server TempPath: %s", conf.Server.TempPath)
|
||||
log.Infof("Server LoggingJSON: %v", conf.Server.LoggingJSON)
|
||||
log.Infof("Server PIDFilePath: %s", conf.Server.PIDFilePath)
|
||||
log.Infof("Server CleanUponExit: %v", conf.Server.CleanUponExit)
|
||||
log.Infof("Server PreCaching: %v", conf.Server.PreCaching)
|
||||
log.Infof("Server FileTTLEnabled: %v", conf.Server.FileTTLEnabled)
|
||||
log.Infof("Server DeduplicationEnabled: %v", conf.Server.DeduplicationEnabled)
|
||||
log.Infof("Server BindIP: %s", conf.Server.BindIP) // Hinzugefügt: Logging für BindIP
|
||||
log.Infof("Server FileNaming: %s", conf.Server.FileNaming) // Added: Logging for FileNaming
|
||||
|
||||
err = writePIDFile(conf.Server.PIDFilePath) // Write PID file after config is loaded
|
||||
if err != nil {
|
||||
@ -569,7 +579,7 @@ func main() {
|
||||
}
|
||||
|
||||
server := &http.Server{
|
||||
Addr: ":" + conf.Server.ListenPort,
|
||||
Addr: conf.Server.BindIP + ":" + conf.Server.ListenPort, // Geändert: Nutzung von BindIP
|
||||
Handler: router,
|
||||
ReadTimeout: readTimeout,
|
||||
WriteTimeout: writeTimeout,
|
||||
@ -595,7 +605,7 @@ func main() {
|
||||
go monitorWorkerPerformance(ctx, &conf.Server, &conf.Workers, &conf.ClamAV)
|
||||
}
|
||||
|
||||
versionString = "2.5" // Updated from "2.4"
|
||||
versionString = conf.Build.Version // Updated to use Build.Version from config
|
||||
log.Infof("Running version: %s", versionString)
|
||||
|
||||
log.Infof("Starting HMAC file server %s...", versionString)
|
||||
@ -621,6 +631,97 @@ func main() {
|
||||
go handleFileCleanup(&conf)
|
||||
}
|
||||
|
||||
func printExampleConfig() {
|
||||
fmt.Print(`
|
||||
[server]
|
||||
bind_ip = "0.0.0.0"
|
||||
listenport = "8080"
|
||||
unixsocket = false
|
||||
storagepath = "./uploads"
|
||||
logfile = "/var/log/hmac-file-server.log"
|
||||
metricsenabled = true
|
||||
metricsport = "9090"
|
||||
minfreebytes = "100MB"
|
||||
filettl = "8760h"
|
||||
filettlenabled = true
|
||||
autoadjustworkers = true
|
||||
networkevents = true
|
||||
pidfilepath = "/var/run/hmacfileserver.pid"
|
||||
cleanuponexit = true
|
||||
precaching = true
|
||||
deduplicationenabled = true
|
||||
globalextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
|
||||
# FileNaming options: "HMAC", "None"
|
||||
filenaming = "HMAC"
|
||||
|
||||
[logging]
|
||||
level = "info"
|
||||
file = "/var/log/hmac-file-server.log"
|
||||
max_size = 100
|
||||
max_backups = 7
|
||||
max_age = 30
|
||||
compress = true
|
||||
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "./deduplication"
|
||||
|
||||
[iso]
|
||||
enabled = true
|
||||
size = "1GB"
|
||||
mountpoint = "/mnt/iso"
|
||||
charset = "utf-8"
|
||||
containerfile = "/mnt/iso/container.iso"
|
||||
|
||||
[timeouts]
|
||||
readtimeout = "4800s"
|
||||
writetimeout = "4800s"
|
||||
idletimeout = "4800s"
|
||||
|
||||
[security]
|
||||
secret = "changeme"
|
||||
|
||||
[versioning]
|
||||
enableversioning = false
|
||||
maxversions = 1
|
||||
|
||||
[uploads]
|
||||
resumableuploadsenabled = true
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "8192"
|
||||
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
|
||||
|
||||
[downloads]
|
||||
resumabledownloadsenabled = true
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "8192"
|
||||
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
|
||||
|
||||
[clamav]
|
||||
clamavenabled = true
|
||||
clamavsocket = "/var/run/clamav/clamd.ctl"
|
||||
numscanworkers = 2
|
||||
scanfileextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
|
||||
|
||||
[redis]
|
||||
redisenabled = true
|
||||
redisdbindex = 0
|
||||
redisaddr = "localhost:6379"
|
||||
redispassword = ""
|
||||
redishealthcheckinterval = "120s"
|
||||
|
||||
[workers]
|
||||
numworkers = 4
|
||||
uploadqueuesize = 50
|
||||
|
||||
[file]
|
||||
# Add file-specific configurations here
|
||||
|
||||
[build]
|
||||
version = "2.6-Stable"
|
||||
`)
|
||||
}
|
||||
|
||||
func max(a, b int) int {
|
||||
if a > b {
|
||||
return a
|
||||
@ -932,9 +1033,6 @@ func validateConfig(conf *Config) error {
|
||||
}
|
||||
|
||||
// Verify File Configuration
|
||||
if conf.File.FileRevision < 0 {
|
||||
return fmt.Errorf("file.filerevision cannot be negative")
|
||||
}
|
||||
|
||||
// Validate Global Extensions
|
||||
if len(conf.Server.GlobalExtensions) == 0 {
|
||||
@ -1338,7 +1436,7 @@ func processUpload(task UploadTask) error {
|
||||
exists, size := fileExists(absFilename)
|
||||
log.Debugf("Exists? %v, Size: %d", exists, size)
|
||||
|
||||
// Gajim and Dino do not require a callback or acknowledgement beyond HTTP success.
|
||||
// Gajim und Dino benötigen keinen Callback oder eine Bestätigung über HTTP-Erfolg hinaus.
|
||||
callbackURL := r.Header.Get("Callback-URL")
|
||||
if callbackURL != "" {
|
||||
log.Warnf("Callback-URL provided (%s) but not needed. Ignoring.", callbackURL)
|
||||
@ -1638,6 +1736,7 @@ func handleUpload(w http.ResponseWriter, r *http.Request, absFilename, fileStore
|
||||
} else {
|
||||
log.Warn("No HMAC attached to URL.")
|
||||
http.Error(w, "No HMAC attached to URL. Expecting 'v', 'v2', or 'token' parameter as MAC", http.StatusForbidden)
|
||||
uploadErrorsTotal.Inc()
|
||||
return
|
||||
}
|
||||
|
||||
@ -1660,12 +1759,14 @@ func handleUpload(w http.ResponseWriter, r *http.Request, absFilename, fileStore
|
||||
if err != nil {
|
||||
log.Warn("Invalid MAC encoding")
|
||||
http.Error(w, "Invalid MAC encoding", http.StatusForbidden)
|
||||
uploadErrorsTotal.Inc()
|
||||
return
|
||||
}
|
||||
|
||||
if !hmac.Equal(calculatedMAC, providedMAC) {
|
||||
log.Warn("Invalid MAC")
|
||||
http.Error(w, "Invalid MAC", http.StatusForbidden)
|
||||
uploadErrorsTotal.Inc()
|
||||
return
|
||||
}
|
||||
|
||||
@ -1688,23 +1789,43 @@ func handleUpload(w http.ResponseWriter, r *http.Request, absFilename, fileStore
|
||||
return
|
||||
}
|
||||
|
||||
// Determine the final filename based on the FileNaming configuration
|
||||
finalFilename := absFilename
|
||||
switch conf.Server.FileNaming {
|
||||
case "HMAC":
|
||||
// Use hashed/HMAC name
|
||||
tempFilename := ""
|
||||
hashVal, err := computeSHA256(context.Background(), tempFilename)
|
||||
if err != nil {
|
||||
// ...existing code...
|
||||
}
|
||||
finalFilename = filepath.Join(filepath.Dir(absFilename), hashVal)
|
||||
case "None":
|
||||
// Preserve the original filename
|
||||
finalFilename = absFilename
|
||||
default:
|
||||
// ...existing code...
|
||||
}
|
||||
|
||||
// Create temp file and write the uploaded data
|
||||
tempFilename := absFilename + ".tmp"
|
||||
tempFilename := finalFilename + ".tmp"
|
||||
err = createFile(tempFilename, r)
|
||||
if err != nil {
|
||||
log.WithFields(logrus.Fields{
|
||||
"filename": absFilename,
|
||||
"filename": finalFilename,
|
||||
}).WithError(err).Error("Error creating temp file")
|
||||
http.Error(w, "Error writing temp file", http.StatusInternalServerError)
|
||||
uploadErrorsTotal.Inc()
|
||||
return
|
||||
}
|
||||
|
||||
// Move temp file to final destination
|
||||
err = os.Rename(tempFilename, absFilename)
|
||||
err = os.Rename(tempFilename, finalFilename)
|
||||
if err != nil {
|
||||
log.Errorf("Rename failed for %s: %v", absFilename, err)
|
||||
log.Errorf("Rename failed for %s: %v", finalFilename, err)
|
||||
os.Remove(tempFilename)
|
||||
http.Error(w, "Error moving file to final destination", http.StatusInternalServerError)
|
||||
uploadErrorsTotal.Inc()
|
||||
return
|
||||
}
|
||||
|
||||
@ -1713,55 +1834,55 @@ func handleUpload(w http.ResponseWriter, r *http.Request, absFilename, fileStore
|
||||
if f, ok := w.(http.Flusher); ok {
|
||||
f.Flush()
|
||||
}
|
||||
log.Infof("Responded with 201 Created for file: %s", absFilename)
|
||||
log.Infof("Responded with 201 Created for file: %s", finalFilename)
|
||||
|
||||
// Asynchronous processing in the background
|
||||
go func() {
|
||||
var logMessages []string
|
||||
|
||||
// ClamAV scanning
|
||||
if conf.ClamAV.ClamAVEnabled && shouldScanFile(absFilename) {
|
||||
err := scanFileWithClamAV(absFilename)
|
||||
if conf.ClamAV.ClamAVEnabled && shouldScanFile(finalFilename) {
|
||||
err := scanFileWithClamAV(finalFilename)
|
||||
if err != nil {
|
||||
logMessages = append(logMessages, fmt.Sprintf("ClamAV failed for %s: %v", absFilename, err))
|
||||
logMessages = append(logMessages, fmt.Sprintf("ClamAV failed for %s: %v", finalFilename, err))
|
||||
for _, msg := range logMessages {
|
||||
log.Info(msg)
|
||||
}
|
||||
return
|
||||
} else {
|
||||
logMessages = append(logMessages, fmt.Sprintf("ClamAV scan passed for file: %s", absFilename))
|
||||
logMessages = append(logMessages, fmt.Sprintf("ClamAV scan passed for file: %s", finalFilename))
|
||||
}
|
||||
}
|
||||
|
||||
// Deduplication
|
||||
if conf.Redis.RedisEnabled && conf.Server.DeduplicationEnabled {
|
||||
err := handleDeduplication(context.Background(), absFilename)
|
||||
err := handleDeduplication(context.Background(), finalFilename)
|
||||
if err != nil {
|
||||
log.Errorf("Deduplication failed for %s: %v", absFilename, err)
|
||||
os.Remove(absFilename)
|
||||
log.Errorf("Deduplication failed for %s: %v", finalFilename, err)
|
||||
os.Remove(finalFilename)
|
||||
uploadErrorsTotal.Inc()
|
||||
return
|
||||
} else {
|
||||
logMessages = append(logMessages, fmt.Sprintf("Deduplication handled successfully for file: %s", absFilename))
|
||||
logMessages = append(logMessages, fmt.Sprintf("Deduplication handled successfully for file: %s", finalFilename))
|
||||
}
|
||||
}
|
||||
|
||||
// Versioning
|
||||
if conf.Versioning.EnableVersioning {
|
||||
if exists, _ := fileExists(absFilename); exists {
|
||||
err := versionFile(absFilename)
|
||||
if exists, _ := fileExists(finalFilename); exists {
|
||||
err := versionFile(finalFilename)
|
||||
if err != nil {
|
||||
log.Errorf("Versioning failed for %s: %v", absFilename, err)
|
||||
os.Remove(absFilename)
|
||||
log.Errorf("Versioning failed for %s: %v", finalFilename, err)
|
||||
os.Remove(finalFilename)
|
||||
uploadErrorsTotal.Inc()
|
||||
return
|
||||
} else {
|
||||
logMessages = append(logMessages, fmt.Sprintf("File versioned successfully: %s", absFilename))
|
||||
logMessages = append(logMessages, fmt.Sprintf("File versioned successfully: %s", finalFilename))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
logMessages = append(logMessages, fmt.Sprintf("Processing completed successfully for %s", absFilename))
|
||||
logMessages = append(logMessages, fmt.Sprintf("Processing completed successfully for %s", finalFilename))
|
||||
uploadsTotal.Inc()
|
||||
|
||||
// Log all messages at once
|
||||
@ -1790,6 +1911,7 @@ func handleDownload(w http.ResponseWriter, r *http.Request, absFilename, fileSto
|
||||
}
|
||||
}
|
||||
http.NotFound(w, r)
|
||||
downloadErrorsTotal.Inc()
|
||||
return
|
||||
}
|
||||
|
||||
@ -1798,6 +1920,7 @@ func handleDownload(w http.ResponseWriter, r *http.Request, absFilename, fileSto
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
w.Header().Set("Content-Type", contentType)
|
||||
w.Header().Set("Content-Length", strconv.FormatInt(fileInfo.Size(), 10))
|
||||
|
||||
if conf.Uploads.ResumableUploadsEnabled {
|
||||
handleResumableDownload(absFilename, w, r, fileInfo.Size())
|
||||
|
86
config.example.toml
Normal file
86
config.example.toml
Normal file
@ -0,0 +1,86 @@
|
||||
[server]
|
||||
bind_ip = "0.0.0.0"
|
||||
listenport = "8080"
|
||||
unixsocket = false
|
||||
storagepath = "./uploads"
|
||||
logfile = "/var/log/hmac-file-server.log"
|
||||
metricsenabled = true
|
||||
metricsport = "9090"
|
||||
minfreebytes = "100MB"
|
||||
filettl = "8760h"
|
||||
filettlenabled = true
|
||||
autoadjustworkers = true
|
||||
networkevents = true
|
||||
pidfilepath = "/var/run/hmacfileserver.pid"
|
||||
cleanuponexit = true
|
||||
precaching = true
|
||||
deduplicationenabled = true
|
||||
globalextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
|
||||
# FileNaming options: "HMAC", "None"
|
||||
filenaming = "HMAC"
|
||||
|
||||
[logging]
|
||||
level = "info"
|
||||
file = "/var/log/hmac-file-server.log"
|
||||
max_size = 100
|
||||
max_backups = 7
|
||||
max_age = 30
|
||||
compress = true
|
||||
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "./deduplication"
|
||||
|
||||
[iso]
|
||||
enabled = true
|
||||
size = "1GB"
|
||||
mountpoint = "/mnt/iso"
|
||||
charset = "utf-8"
|
||||
containerfile = "/mnt/iso/container.iso"
|
||||
|
||||
[timeouts]
|
||||
readtimeout = "4800s"
|
||||
writetimeout = "4800s"
|
||||
idletimeout = "4800s"
|
||||
|
||||
[security]
|
||||
secret = "changeme"
|
||||
|
||||
[versioning]
|
||||
enableversioning = false
|
||||
maxversions = 1
|
||||
|
||||
[uploads]
|
||||
resumableuploadsenabled = true
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "8192"
|
||||
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
|
||||
|
||||
[downloads]
|
||||
resumabledownloadsenabled = true
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "8192"
|
||||
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
|
||||
|
||||
[clamav]
|
||||
clamavenabled = true
|
||||
clamavsocket = "/var/run/clamav/clamd.ctl"
|
||||
numscanworkers = 2
|
||||
scanfileextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp"]
|
||||
|
||||
[redis]
|
||||
redisenabled = true
|
||||
redisdbindex = 0
|
||||
redisaddr = "localhost:6379"
|
||||
redispassword = ""
|
||||
redishealthcheckinterval = "120s"
|
||||
|
||||
[workers]
|
||||
numworkers = 4
|
||||
uploadqueuesize = 50
|
||||
|
||||
[file]
|
||||
# Add file-specific configurations here
|
||||
|
||||
[build]
|
||||
version = "2.6-Stable"
|
5
config.toml
Normal file
5
config.toml
Normal file
@ -0,0 +1,5 @@
|
||||
[server]
|
||||
// ...existing code...
|
||||
# FileNaming options: "HMAC", "None"
|
||||
FileNaming = "HMAC"
|
||||
// ...existing code...
|
89
config.toml.example
Normal file
89
config.toml.example
Normal file
@ -0,0 +1,89 @@
|
||||
### Issue
|
||||
In `config.toml`, comments are using `//`, which is invalid in TOML. TOML uses `#` for comments. This causes a syntax error.
|
||||
|
||||
### Corrected `config.toml`
|
||||
|
||||
```toml
|
||||
# filepath: /home/renz/source/hmac-file-server/config.toml
|
||||
|
||||
[server]
|
||||
listenport = "8080"
|
||||
unixsocket = false
|
||||
storagepath = "./uploads"
|
||||
metricsenabled = true
|
||||
metricsport = "9090"
|
||||
filettl = "8760h"
|
||||
minfreebytes = "100MB"
|
||||
autoadjustworkers = true
|
||||
networkevents = true
|
||||
temppath = "/tmp/hmac-file-server"
|
||||
loggingjson = false
|
||||
pidfilepath = "/var/run/hmacfileserver.pid"
|
||||
cleanuponexit = true
|
||||
precaching = true
|
||||
filettlenabled = true
|
||||
globalextensions = ["*"] # Allows all file types globally
|
||||
bind_ip = "0.0.0.0" # Specify the IP address to bind to (IPv4 or IPv6)
|
||||
|
||||
[logging]
|
||||
level = "info"
|
||||
file = "/var/log/hmac-file-server.log"
|
||||
max_size = 100
|
||||
max_backups = 7
|
||||
max_age = 30
|
||||
compress = true
|
||||
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "./deduplication"
|
||||
|
||||
[iso]
|
||||
enabled = true
|
||||
size = "1GB"
|
||||
mountpoint = "/mnt/iso"
|
||||
charset = "utf-8"
|
||||
containerfile = "/path/to/iso/container.iso"
|
||||
|
||||
[timeouts]
|
||||
readtimeout = "4800s"
|
||||
writetimeout = "4800s"
|
||||
idletimeout = "4800s"
|
||||
|
||||
[security]
|
||||
secret = "changeme"
|
||||
|
||||
[versioning]
|
||||
enableversioning = false
|
||||
maxversions = 5
|
||||
|
||||
[uploads]
|
||||
resumableuploadsenabled = false
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "64mb"
|
||||
allowedextensions = ["*"] # Use ["*"] to allow all or specify extensions
|
||||
|
||||
[downloads]
|
||||
resumabledownloadsenabled = false
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "64mb"
|
||||
allowedextensions = [".jpg", ".png"] # Restricts downloads to specific types
|
||||
|
||||
[clamav]
|
||||
clamavenabled = false
|
||||
clamavsocket = "/var/run/clamav/clamd.ctl"
|
||||
numscanworkers = 2
|
||||
scanfileextensions = [".exe", ".dll", ".pdf"]
|
||||
|
||||
[redis]
|
||||
redisenabled = false
|
||||
redisaddr = "localhost:6379"
|
||||
redispassword = ""
|
||||
redisdbindex = 0
|
||||
redishealthcheckinterval = "120s"
|
||||
|
||||
[workers]
|
||||
numworkers = 4
|
||||
uploadqueuesize = 5000
|
||||
|
||||
[build]
|
||||
version = "v2.5"
|
File diff suppressed because it is too large
Load Diff
BIN
dashboard/hmac_icon.png
Normal file
BIN
dashboard/hmac_icon.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 60 KiB |
@ -16,7 +16,7 @@ import (
|
||||
|
||||
const (
|
||||
serverURL = "http://[::1]:8080" // Replace with your actual server URL
|
||||
secret = "a-orc-and-a-humans-is-drinking-ale" // Replace with your HMAC secret key
|
||||
secret = "hmac-file-server-is-the-win" // Replace with your HMAC secret key
|
||||
uploadPath = "hmac_icon.png" // Test file to upload
|
||||
protocolType = "v2" // Use v2, v, or token as needed
|
||||
)
|
||||
|
834
wiki.md
Normal file
834
wiki.md
Normal file
@ -0,0 +1,834 @@
|
||||
This documentation provides detailed information on configuring, setting up, and maintaining the HMAC File Server. Whether you're a developer, system administrator, or an enthusiast, this guide will help you navigate through the server's features and configurations effectively.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Introduction](#introduction)
|
||||
2. [Configuration](#configuration)
|
||||
- [Server Configuration](#server-configuration)
|
||||
- [Deduplication Settings](#deduplication-settings)
|
||||
- [ISO Settings](#iso-settings)
|
||||
- [Timeout Settings](#timeout-settings)
|
||||
- [Security Settings](#security-settings)
|
||||
- [Versioning Settings](#versioning-settings)
|
||||
- [Uploads Settings](#uploads-settings)
|
||||
- [Downloads Settings](#downloads-settings)
|
||||
- [ClamAV Settings](#clamav-settings)
|
||||
- [Redis Settings](#redis-settings)
|
||||
- [Worker Settings](#worker-settings)
|
||||
3. [Example Configuration](#example-configuration)
|
||||
4. [Setup Instructions](#setup-instructions)
|
||||
- [1. HMAC File Server Installation](#1-hmac-file-server-installation)
|
||||
- [2. Reverse Proxy Configuration](#2-reverse-proxy-configuration)
|
||||
- [Apache2 Reverse Proxy](#apache2-reverse-proxy)
|
||||
- [Nginx Reverse Proxy](#nginx-reverse-proxy)
|
||||
- [3. ejabberd Configuration](#3-ejabberd-configuration)
|
||||
- [4. Systemd Service Setup](#4-systemd-service-setup)
|
||||
5. [Building for Different Architectures](#building-for-different-architectures)
|
||||
6. [Additional Recommendations](#additional-recommendations)
|
||||
7. [Notes](#notes)
|
||||
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
The **HMAC File Server** is a secure and efficient file management solution designed to handle file uploads, downloads, deduplication, and more. Built with a focus on security, scalability, and performance, it integrates seamlessly with various tools and services to provide a comprehensive file handling experience.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
The HMAC File Server is configured using a `config.toml` file. Below are the detailed explanations of each configuration section and their respective options.
|
||||
|
||||
### Server Configuration
|
||||
|
||||
```toml
|
||||
# Server configuration
|
||||
listenport = "8080" # TCP port for incoming requests
|
||||
unixsocket = false # Use Unix domain socket instead of TCP
|
||||
storagepath = "/path/to/hmac-file-server/data/" # Directory to store uploaded files
|
||||
loglevel = "debug" # Logging level: "debug", "info", "warn", "error"
|
||||
logfile = "/path/to/hmac-file-server.log" # Path to log file; leave empty to use stdout
|
||||
metricsenabled = true # Enable Prometheus metrics
|
||||
metricsport = "9090" # Port for Prometheus metrics
|
||||
deduplicationenabled = true
|
||||
minfreebytes = "5GB" # Minimum free disk space required
|
||||
filettl = "2Y" # Time-to-live for files
|
||||
filettlenabled = false # Enable TTL checks and cleanup
|
||||
autoadjustworkers = true # Automatically adjust worker threads based on load
|
||||
networkevents = false # Enable detailed network event logging
|
||||
pidfilepath = "./hmac-file-server.pid" # Path to PID file
|
||||
precaching = true # Pre-cache file structures on startup
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **listenport**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the TCP port on which the server listens for incoming requests.
|
||||
- *Default*: `"8080"`
|
||||
|
||||
- **unixsocket**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Determines whether to use a Unix domain socket instead of a TCP port for communication.
|
||||
- *Default*: `false`
|
||||
|
||||
- **storagepath**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the directory path where uploaded files are stored. Ensure this path exists and has appropriate permissions.
|
||||
- *Default*: `"/path/to/hmac-file-server/data/"`
|
||||
|
||||
- **loglevel**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the verbosity level of logs.
|
||||
- *Options*: `"debug"`, `"info"`, `"warn"`, `"error"`
|
||||
- *Default*: `"debug"`
|
||||
|
||||
- **logfile**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the file path for logging. If left empty, logs are output to `stdout`.
|
||||
- *Default*: `"/path/to/hmac-file-server.log"`
|
||||
|
||||
- **metricsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables the Prometheus metrics endpoint.
|
||||
- *Default*: `true`
|
||||
|
||||
- **metricsport**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the port on which Prometheus metrics are exposed.
|
||||
- *Default*: `"9090"`
|
||||
|
||||
- **deduplicationenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables file deduplication to optimize storage usage.
|
||||
- *Default*: `true`
|
||||
|
||||
- **minfreebytes**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the minimum free disk space required for the server to operate effectively.
|
||||
- *Default*: `"5GB"`
|
||||
|
||||
- **filettl**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the default Time-to-Live (TTL) for files, determining how long files are retained before deletion.
|
||||
- *Format*: Duration (e.g., `"2Y"` for two years)
|
||||
- *Default*: `"2Y"`
|
||||
|
||||
- **filettlenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables TTL checks and automatic file cleanup based on the `filettl` value.
|
||||
- *Default*: `false`
|
||||
|
||||
- **autoadjustworkers**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Automatically adjusts the number of worker threads based on server load and system resources.
|
||||
- *Default*: `true`
|
||||
|
||||
- **networkevents**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables detailed logging of network events, which can be useful for debugging but may increase log verbosity.
|
||||
- *Default*: `false`
|
||||
|
||||
- **pidfilepath**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the file path where the server writes its Process ID (PID) file. This is useful for managing the server process.
|
||||
- *Default*: `"./hmac-file-server.pid"`
|
||||
|
||||
- **precaching**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables pre-caching of file structures on startup to improve access speed and performance.
|
||||
- *Default*: `true`
|
||||
|
||||
---
|
||||
|
||||
### Deduplication Settings
|
||||
|
||||
```toml
|
||||
# Deduplication settings
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/deduplication/" # Path to deduplication metadata store
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **enabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables the deduplication feature, which helps in eliminating duplicate files to save storage space.
|
||||
- *Default*: `true`
|
||||
|
||||
- **directory**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the directory path where deduplication metadata is stored. Ensure this directory exists and has appropriate permissions.
|
||||
- *Default*: `"/path/to/hmac-file-server/deduplication/"`
|
||||
|
||||
---
|
||||
|
||||
### ISO Settings
|
||||
|
||||
```toml
|
||||
# ISO settings
|
||||
[iso]
|
||||
enabled = false
|
||||
size = "1TB" # Maximum ISO size
|
||||
mountpoint = "/path/to/hmac-file-server/iso/" # ISO mount point
|
||||
charset = "utf-8" # Filesystem character set encoding
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **enabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables the mounting of an ISO-based filesystem for specialized storage needs.
|
||||
- *Default*: `false`
|
||||
|
||||
- **size**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the maximum allowed size for the ISO container.
|
||||
- *Default*: `"1TB"`
|
||||
|
||||
- **mountpoint**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the directory path where the ISO is mounted. Ensure this path exists and has appropriate permissions.
|
||||
- *Default*: `"/path/to/hmac-file-server/iso/"`
|
||||
|
||||
- **charset**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the filesystem character set encoding for the ISO.
|
||||
- *Default*: `"utf-8"`
|
||||
|
||||
> **Note**: Ensure only one `[iso]` block is active in your `config.toml` to avoid configuration conflicts.
|
||||
|
||||
---
|
||||
|
||||
### Timeout Settings
|
||||
|
||||
```toml
|
||||
# Timeout settings
|
||||
[timeouts]
|
||||
readtimeout = "3600s" # Maximum time to read a request
|
||||
writetimeout = "3600s" # Maximum time to write a response
|
||||
idletimeout = "3600s" # Maximum keep-alive time for idle connections
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **readtimeout**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the maximum duration for reading the entire request, including the body.
|
||||
- *Format*: Duration (e.g., `"3600s"` for one hour)
|
||||
- *Default*: `"3600s"`
|
||||
|
||||
- **writetimeout**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the maximum duration before timing out writes of the response.
|
||||
- *Format*: Duration (e.g., `"3600s"` for one hour)
|
||||
- *Default*: `"3600s"`
|
||||
|
||||
- **idletimeout**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the maximum amount of time to wait for the next request when keep-alives are enabled.
|
||||
- *Format*: Duration (e.g., `"3600s"` for one hour)
|
||||
- *Default*: `"3600s"`
|
||||
|
||||
---
|
||||
|
||||
### Security Settings
|
||||
|
||||
```toml
|
||||
# Security settings
|
||||
[security]
|
||||
secret = "your-secure-secret-key" # HMAC shared secret key (change to a secure value)
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **secret**:
|
||||
- *Type*: `String`
|
||||
- *Description*: The HMAC shared secret key used for signing requests and operations.
|
||||
- *Default*: `"your-secure-secret-key"`
|
||||
- *Warning*: **Change this immediately** to a unique, strong string in production environments to ensure the security of HMAC operations.
|
||||
|
||||
---
|
||||
|
||||
### Versioning Settings
|
||||
|
||||
```toml
|
||||
# Versioning settings
|
||||
[versioning]
|
||||
enableversioning = false
|
||||
maxversions = 1 # Number of file versions to retain
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **enableversioning**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables the versioning feature, which maintains multiple versions of the same file.
|
||||
- *Default*: `false`
|
||||
|
||||
- **maxversions**:
|
||||
- *Type*: `Integer`
|
||||
- *Description*: Specifies the maximum number of versions to retain for each file.
|
||||
- *Default*: `1`
|
||||
|
||||
---
|
||||
|
||||
### Uploads Settings
|
||||
|
||||
```toml
|
||||
# Upload settings
|
||||
[uploads]
|
||||
resumableuploadsenabled = false
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "32MB" # Chunk size for uploads
|
||||
allowedextensions = [
|
||||
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
|
||||
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
|
||||
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
|
||||
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
|
||||
]
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **resumableuploadsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables support for resumable (chunked) file uploads.
|
||||
- *Default*: `false`
|
||||
|
||||
- **chunkeduploadsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Specifically enables or disables chunked uploads.
|
||||
- *Default*: `true`
|
||||
|
||||
- **chunksize**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the size of each chunk in chunked uploads.
|
||||
- *Format*: Size (e.g., `"32MB"`)
|
||||
- *Default*: `"32MB"`
|
||||
|
||||
- **allowedextensions**:
|
||||
- *Type*: `Array of Strings`
|
||||
- *Description*: Lists the file extensions permitted for upload.
|
||||
- *Default*:
|
||||
```toml
|
||||
allowedextensions = [
|
||||
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
|
||||
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
|
||||
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
|
||||
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Downloads Settings
|
||||
|
||||
```toml
|
||||
# Downloads settings
|
||||
[downloads]
|
||||
resumabledownloadsenabled = false
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "32MB"
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **resumabledownloadsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables support for resumable (chunked) downloads.
|
||||
- *Default*: `false`
|
||||
|
||||
- **chunkeddownloadsenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Specifically enables or disables chunked downloads.
|
||||
- *Default*: `true`
|
||||
|
||||
- **chunksize**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the size of each chunk in chunked downloads.
|
||||
- *Format*: Size (e.g., `"32MB"`)
|
||||
- *Default*: `"32MB"`
|
||||
|
||||
> **Note**: The `allowedextensions` key is **not** part of the `[downloads]` configuration based on the provided code. Ensure that it is omitted to prevent configuration errors.
|
||||
|
||||
---
|
||||
|
||||
### ClamAV Settings
|
||||
|
||||
```toml
|
||||
# ClamAV settings
|
||||
[clamav]
|
||||
clamavenabled = true
|
||||
clamavsocket = "/path/to/clamav/clamd.ctl" # Path to ClamAV socket
|
||||
numscanworkers = 4 # Number of concurrent scan workers
|
||||
scanfileextensions = [
|
||||
".exe", ".dll", ".bin", ".com", ".bat",
|
||||
".sh", ".php", ".js"
|
||||
]
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **clamavenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables ClamAV integration for virus scanning of uploaded files.
|
||||
- *Default*: `true`
|
||||
|
||||
- **clamavsocket**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the file path to the ClamAV socket (`.ctl` file). Ensure ClamAV is installed and the socket path is correct.
|
||||
- *Default*: `"/path/to/clamav/clamd.ctl"`
|
||||
|
||||
- **numscanworkers**:
|
||||
- *Type*: `Integer`
|
||||
- *Description*: Sets the number of concurrent workers dedicated to scanning files with ClamAV.
|
||||
- *Default*: `4`
|
||||
|
||||
- **scanfileextensions**:
|
||||
- *Type*: `Array of Strings`
|
||||
- *Description*: Lists the file extensions that should be scanned for viruses.
|
||||
- *Default*:
|
||||
```toml
|
||||
scanfileextensions = [
|
||||
".exe", ".dll", ".bin", ".com", ".bat",
|
||||
".sh", ".php", ".js"
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Redis Settings
|
||||
|
||||
```toml
|
||||
# Redis settings
|
||||
[redis]
|
||||
redisenabled = true
|
||||
redisdbindex = 0
|
||||
redisaddr = "localhost:6379" # Redis server address
|
||||
redispassword = "" # Redis password if required
|
||||
redishealthcheckinterval = "120s" # Interval for Redis health checks
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **redisenabled**:
|
||||
- *Type*: `Boolean`
|
||||
- *Description*: Enables or disables Redis integration for caching or session tracking.
|
||||
- *Default*: `true`
|
||||
|
||||
- **redisaddr**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Specifies the address of the Redis server (e.g., `"localhost:6379"`).
|
||||
- *Default*: `"localhost:6379"`
|
||||
|
||||
- **redispassword**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Sets the Redis authentication password, if required.
|
||||
- *Default*: `""`
|
||||
|
||||
- **redisdbindex**:
|
||||
- *Type*: `Integer`
|
||||
- *Description*: Specifies the Redis database index to use.
|
||||
- *Default*: `0`
|
||||
|
||||
- **redishealthcheckinterval**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the interval for performing health checks on the Redis connection.
|
||||
- *Format*: Duration (e.g., `"120s"` for two minutes)
|
||||
- *Default*: `"120s"`
|
||||
|
||||
---
|
||||
|
||||
### Worker Settings
|
||||
|
||||
```toml
|
||||
# Worker settings
|
||||
[worker]
|
||||
numworkers = 10 # Number of worker threads
|
||||
```
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **numworkers**:
|
||||
- *Type*: `Integer`
|
||||
- *Description*: Specifies the number of worker threads to handle file operations.
|
||||
- *Default*: `10`
|
||||
|
||||
---
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
- **maxfilesize**:
|
||||
- *Type*: `String`
|
||||
- *Description*: Defines the maximum allowed file size for uploads.
|
||||
- *Format*: Size (e.g., `"10GB"`)
|
||||
- *Default*: `"10GB"`
|
||||
|
||||
---
|
||||
|
||||
## Example Configuration
|
||||
|
||||
Below is an example `config.toml` file with default settings:
|
||||
|
||||
```toml
|
||||
# Example HMAC File Server configuration
|
||||
|
||||
# Server configuration
|
||||
listenport = "8080"
|
||||
unixsocket = false
|
||||
storagepath = "/path/to/hmac-file-server/data/"
|
||||
loglevel = "debug"
|
||||
logfile = "/path/to/hmac-file-server.log"
|
||||
metricsenabled = true
|
||||
metricsport = "9090"
|
||||
deduplicationenabled = true
|
||||
minfreebytes = "5GB"
|
||||
filettl = "2Y"
|
||||
filettlenabled = false
|
||||
autoadjustworkers = true
|
||||
networkevents = false
|
||||
pidfilepath = "./hmac-file-server.pid"
|
||||
precaching = true
|
||||
|
||||
# Deduplication settings
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/deduplication/"
|
||||
|
||||
# ISO settings
|
||||
[iso]
|
||||
enabled = false
|
||||
size = "1TB"
|
||||
mountpoint = "/path/to/hmac-file-server/iso/"
|
||||
charset = "utf-8"
|
||||
|
||||
# Timeout settings
|
||||
[timeouts]
|
||||
readtimeout = "3600s"
|
||||
writetimeout = "3600s"
|
||||
idletimeout = "3600s"
|
||||
|
||||
# Security settings
|
||||
[security]
|
||||
secret = "your-secure-secret-key"
|
||||
|
||||
# Versioning settings
|
||||
[versioning]
|
||||
enableversioning = false
|
||||
maxversions = 1
|
||||
|
||||
# Upload settings
|
||||
[uploads]
|
||||
resumableuploadsenabled = false
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "32MB"
|
||||
allowedextensions = [
|
||||
".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif",
|
||||
".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4",
|
||||
".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm",
|
||||
".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"
|
||||
]
|
||||
|
||||
# Download settings
|
||||
[downloads]
|
||||
resumabledownloadsenabled = false
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "32MB"
|
||||
|
||||
# ClamAV settings
|
||||
[clamav]
|
||||
clamavenabled = true
|
||||
clamavsocket = "/path/to/clamav/clamd.ctl"
|
||||
numscanworkers = 4
|
||||
scanfileextensions = [
|
||||
".exe", ".dll", ".bin", ".com", ".bat",
|
||||
".sh", ".php", ".js"
|
||||
]
|
||||
|
||||
# Redis settings
|
||||
[redis]
|
||||
redisenabled = true
|
||||
redisdbindex = 0
|
||||
redisaddr = "localhost:6379"
|
||||
redispassword = ""
|
||||
redishealthcheckinterval = "120s"
|
||||
|
||||
# Worker settings
|
||||
[worker]
|
||||
numworkers = 10
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
### 1. HMAC File Server Installation
|
||||
|
||||
To install the HMAC File Server, follow these steps:
|
||||
|
||||
1. Clone the repository:
|
||||
```sh
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
cd hmac-file-server
|
||||
```
|
||||
|
||||
2. Build the server:
|
||||
```sh
|
||||
go build -o hmac-file-server
|
||||
```
|
||||
|
||||
3. Create the necessary directories:
|
||||
```sh
|
||||
mkdir -p /path/to/hmac-file-server/data/
|
||||
mkdir -p /path/to/hmac-file-server/deduplication/
|
||||
mkdir -p /path/to/hmac-file-server/iso/
|
||||
```
|
||||
|
||||
4. Copy the example configuration file:
|
||||
```sh
|
||||
cp config.example.toml config.toml
|
||||
```
|
||||
|
||||
5. Edit the `config.toml` file to match your environment and preferences.
|
||||
|
||||
6. Start the server:
|
||||
```sh
|
||||
./hmac-file-server -config config.toml
|
||||
```
|
||||
|
||||
### 2. Reverse Proxy Configuration
|
||||
|
||||
To set up a reverse proxy for the HMAC File Server, you can use either Apache2 or Nginx. Below are the configuration examples for both.
|
||||
|
||||
#### Apache2 Reverse Proxy
|
||||
|
||||
1. Enable the necessary Apache2 modules:
|
||||
```sh
|
||||
sudo a2enmod proxy
|
||||
sudo a2enmod proxy_http
|
||||
sudo a2enmod headers
|
||||
sudo a2enmod rewrite
|
||||
```
|
||||
|
||||
2. Create a new virtual host configuration file:
|
||||
```sh
|
||||
sudo nano /etc/apache2/sites-available/hmac-file-server.conf
|
||||
```
|
||||
|
||||
3. Add the following configuration to the file:
|
||||
```apache
|
||||
<VirtualHost *:80>
|
||||
ServerName your-domain.com
|
||||
|
||||
ProxyPreserveHost On
|
||||
ProxyPass / http://localhost:8080/
|
||||
ProxyPassReverse / http://localhost:8080/
|
||||
|
||||
<Location />
|
||||
Require all granted
|
||||
Header always set X-Content-Type-Options "nosniff"
|
||||
Header always set X-Frame-Options "DENY"
|
||||
Header always set X-XSS-Protection "1; mode=block"
|
||||
</Location>
|
||||
</VirtualHost>
|
||||
```
|
||||
|
||||
4. Enable the new site and restart Apache2:
|
||||
```sh
|
||||
sudo a2ensite hmac-file-server.conf
|
||||
sudo systemctl restart apache2
|
||||
```
|
||||
|
||||
#### Nginx Reverse Proxy
|
||||
|
||||
1. Install Nginx if not already installed:
|
||||
```sh
|
||||
sudo apt-get update
|
||||
sudo apt-get install nginx
|
||||
```
|
||||
|
||||
2. Create a new server block configuration file:
|
||||
```sh
|
||||
sudo nano /etc/nginx/sites-available/hmac-file-server
|
||||
```
|
||||
|
||||
3. Add the following configuration to the file:
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name your-domain.com;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:8080;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Content-Type-Options "nosniff";
|
||||
proxy_set_header X-Frame-Options "DENY";
|
||||
proxy_set_header X-XSS-Protection "1; mode=block";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. Enable the new site and restart Nginx:
|
||||
```sh
|
||||
sudo ln -s /etc/nginx/sites-available/hmac-file-server /etc/nginx/sites-enabled/
|
||||
sudo systemctl restart nginx
|
||||
```
|
||||
|
||||
You're correct—my statement included unnecessary comments about the configuration. Here's the fully revised configuration without comments or meta-discussion:
|
||||
|
||||
---
|
||||
|
||||
#### 3. ejabberd Configuration
|
||||
|
||||
```yaml
|
||||
hosts:
|
||||
- "your-domain.com"
|
||||
|
||||
listen:
|
||||
-
|
||||
port: 5222
|
||||
module: ejabberd_c2s
|
||||
certfile: "/etc/ejabberd/ejabberd.pem"
|
||||
starttls: true
|
||||
starttls_required: true
|
||||
protocol_options:
|
||||
- "no_sslv3"
|
||||
- "no_tlsv1"
|
||||
- "no_tlsv1_1"
|
||||
ciphers: "HIGH:!aNULL:!eNULL:!3DES:@STRENGTH"
|
||||
dhfile: "/etc/ejabberd/dhparams.pem"
|
||||
max_stanza_size: 65536
|
||||
shaper: c2s_shaper
|
||||
access: c2s
|
||||
|
||||
-
|
||||
port: 5269
|
||||
module: ejabberd_s2s_in
|
||||
certfile: "/etc/ejabberd/ejabberd.pem"
|
||||
starttls: true
|
||||
starttls_required: true
|
||||
protocol_options:
|
||||
- "no_sslv3"
|
||||
- "no_tlsv1"
|
||||
- "no_tlsv1_1"
|
||||
ciphers: "HIGH:!aNULL:!eNULL:!3DES:@STRENGTH"
|
||||
dhfile: "/etc/ejabberd/dhparams.pem"
|
||||
max_stanza_size: 131072
|
||||
shaper: s2s_shaper
|
||||
access: s2s
|
||||
|
||||
acl:
|
||||
local:
|
||||
user_regexp: ""
|
||||
|
||||
access_rules:
|
||||
local:
|
||||
allow: local
|
||||
|
||||
mod_http_upload:
|
||||
max_size: 1073741824
|
||||
thumbnail: true
|
||||
put_url: https://share.uuxo.net
|
||||
get_url: https://share.uuxo.net
|
||||
external_secret: "changeme"
|
||||
custom_headers:
|
||||
"Access-Control-Allow-Origin": "*"
|
||||
"Access-Control-Allow-Methods": "GET,HEAD,PUT,OPTIONS"
|
||||
"Access-Control-Allow-Headers": "Content-Type"
|
||||
```
|
||||
|
||||
4. Restart ejabberd:
|
||||
```sh
|
||||
sudo systemctl restart ejabberd
|
||||
```
|
||||
|
||||
### 4. Systemd Service Setup
|
||||
|
||||
To set up the HMAC File Server as a systemd service, follow these steps:
|
||||
|
||||
1. Create a new systemd service file:
|
||||
```sh
|
||||
sudo nano /etc/systemd/system/hmac-file-server.service
|
||||
```
|
||||
|
||||
2. Add the following configuration to the file:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=HMAC File Server
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
ExecStart=/path/to/hmac-file-server -config /path/to/config.toml
|
||||
WorkingDirectory=/path/to/hmac-file-server
|
||||
Restart=always
|
||||
User=www-data
|
||||
Group=www-data
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
3. Reload systemd and enable the service:
|
||||
```sh
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable hmac-file-server
|
||||
sudo systemctl start hmac-file-server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Building for Different Architectures
|
||||
|
||||
To build the HMAC File Server for different architectures, you can use the following commands:
|
||||
|
||||
### Building for Linux (x86_64)
|
||||
|
||||
```sh
|
||||
GOOS=linux GOARCH=amd64 go build -o hmac-file-server-linux-amd64
|
||||
```
|
||||
|
||||
### Building for macOS (x86_64)
|
||||
|
||||
```sh
|
||||
GOOS=darwin GOARCH=amd64 go build -o hmac-file-server-darwin-amd64
|
||||
```
|
||||
|
||||
### Building for Windows (x86_64)
|
||||
|
||||
```sh
|
||||
GOOS=windows GOARCH=amd64 go build -o hmac-file-server-windows-amd64.exe
|
||||
```
|
||||
|
||||
### Building for ARM (32-bit)
|
||||
|
||||
```sh
|
||||
GOOS=linux GOARCH=arm GOARM=7 go build -o hmac-file-server-linux-arm
|
||||
```
|
||||
|
||||
### Building for ARM (64-bit)
|
||||
|
||||
```sh
|
||||
GOOS=linux GOARCH=arm64 go build -o hmac-file-server-linux-arm64
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Additional Recommendations
|
||||
|
||||
- **Security**: Ensure that the `secret` key in the `config.toml` file is changed to a unique, strong value to secure HMAC operations.
|
||||
- **Backups**: Regularly back up the `config.toml` file and any important data stored by the HMAC File Server.
|
||||
- **Monitoring**: Use monitoring tools like Prometheus and Grafana to keep track of server performance and metrics.
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- The HMAC File Server is designed to be flexible and configurable. Adjust the settings in the `config.toml` file to match your specific requirements and environment.
|
||||
- For any issues or questions, refer to the project's GitHub repository and documentation.
|
Loading…
x
Reference in New Issue
Block a user