2.5-stable release
This commit is contained in:
parent
7c87955ccf
commit
18495812fd
429
README.MD
429
README.MD
@ -1,11 +1,7 @@
|
||||
# HMAC File Server
|
||||
A secure file server with HMAC authentication and configurable features.
|
||||
|
||||
**HMAC File Server** is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, deduplication, optional ClamAV scanning for file integrity and security, and image thumbnail generation. This server is built with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration.
|
||||
|
||||
> **Credits:** The **HMAC File Server** is based on the source code of [Thomas Leister's prosody-filer](https://github.com/ThomasLeister/prosody-filer). Many features and design elements have been inspired or derived from this project.
|
||||
|
||||
---
|
||||
// ...existing code...
|
||||
|
||||
## Features
|
||||
|
||||
@ -18,429 +14,22 @@ A secure file server with HMAC authentication and configurable features.
|
||||
- **Redis Integration:** Use Redis for caching or storing application states.
|
||||
- **File Expiration:** Automatically delete files after a specified TTL.
|
||||
- **Graceful Shutdown:** Handles signals and ensures proper cleanup.
|
||||
- **Auto-Adjust Worker Scaling:** Dynamically optimize worker threads based on system resources.
|
||||
- **Precaching:** Pre-cache file structures on startup for faster access.
|
||||
- **Thumbnail Creation:** Generate image thumbnails for uploaded files.
|
||||
- **ISO Container Management:** Optional mounting and handling of ISO-based filesystems.
|
||||
- **Auto-Adjust Worker Scaling:** Dynamically optimize worker threads based on system resources when enabled.
|
||||
|
||||
---
|
||||
|
||||
## Repository
|
||||
|
||||
- **Primary Repository**: [GitHub Repository](https://github.com/PlusOne/hmac-file-server)
|
||||
- **Alternative Repository**: [uuxo.net Git Repository](https://git.uuxo.net/uuxo/hmac-file-server)
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Go 1.20+**
|
||||
- **Redis** (optional, if Redis integration is enabled)
|
||||
- **ClamAV** (optional, if file scanning is enabled)
|
||||
- **genisoimage** (optional, if ISO container management is enabled)
|
||||
|
||||
### Clone and Build
|
||||
|
||||
```bash
|
||||
# Clone from the primary repository
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
|
||||
# OR clone from the alternative repository
|
||||
git clone https://git.uuxo.net/uuxo/hmac-file-server.git
|
||||
|
||||
cd hmac-file-server
|
||||
go build -o hmac-file-server main.go
|
||||
```
|
||||
|
||||
#### Build for `arm64`
|
||||
```bash
|
||||
cd /path/to/hmac-file-server
|
||||
GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.3-stable_arm64 main.go
|
||||
```
|
||||
#### Build for `amd64`
|
||||
```bash
|
||||
cd /path/to/hmac-file-server
|
||||
GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.3-stable_amd64 main.go
|
||||
```
|
||||
|
||||
---
|
||||
// ...existing code...
|
||||
|
||||
## Configuration
|
||||
|
||||
The server configuration is managed through a `config.toml` file. Below are the supported configuration options:
|
||||
// ...existing code...
|
||||
|
||||
### Auto-Adjust Feature
|
||||
// Removed Thumbnail Creation section
|
||||
|
||||
When `AutoAdjustWorkers` is enabled, the number of workers for HMAC operations and ClamAV scans is dynamically determined based on system resources. This ensures efficient resource utilization.
|
||||
|
||||
If `AutoAdjustWorkers = true`, the values for `NumWorkers` and `NumScanWorkers` in the configuration file will be ignored, and the server will automatically adjust these values.
|
||||
|
||||
### Network Events Monitoring
|
||||
|
||||
Setting `NetworkEvents = false` in the server configuration disables the logging and tracking of network-related events within the application. This means that functionalities such as monitoring IP changes or recording network activity will be turned off.
|
||||
|
||||
### Precaching
|
||||
|
||||
The `precaching` feature allows the server to pre-cache storage paths for faster access. This can improve performance by reducing the time needed to access frequently used storage paths.
|
||||
|
||||
### Thumbnail Creation
|
||||
|
||||
Set `enabled = true` in the `[thumbnails]` section of `config.toml` to enable image thumbnail generation.
|
||||
|
||||
---
|
||||
|
||||
## Example `config.toml`
|
||||
|
||||
Below is an example configuration file (`config.toml`) you can use as a reference (with sensitive data replaced by placeholder/example data):
|
||||
|
||||
```toml
|
||||
# Server configuration
|
||||
listenport = "8080" # TCP port for incoming requests
|
||||
unixsocket = false # Use Unix domain socket instead of TCP
|
||||
storagepath = "/path/to/hmac-file-server/data/" # Directory to store uploaded files
|
||||
loglevel = "debug" # Logging level: "debug", "info", "warn", "error"
|
||||
logfile = "/path/to/hmac-file-server.log" # Path to log file; leave empty to use stdout
|
||||
metricsenabled = true # Enable Prometheus metrics
|
||||
metricsport = "9090" # Port for Prometheus metrics
|
||||
deduplicationenabled = true
|
||||
minfreebytes = "5GB" # Minimum free disk space required
|
||||
filettl = "2Y" # Time-to-live for files (2 years)
|
||||
filettlenabled = false # Enable TTL checks and cleanup
|
||||
autoadjustworkers = true # Automatically adjust worker threads based on load
|
||||
networkevents = false # Enable detailed network event logging
|
||||
pidfilepath = "./hmac-file-server.pid" # Path to PID file
|
||||
precaching = true # Pre-cache file structures on startup
|
||||
tempPath = "/tmp/hmac-file-server" # Path for temporary file uploads
|
||||
|
||||
# Deduplication settings
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/deduplication/" # Path to deduplication metadata store
|
||||
|
||||
# Thumbnails settings
|
||||
[thumbnails]
|
||||
enabled = true
|
||||
directory = "/path/to/hmac-file-server/thumbnails/" # Directory for storing thumbnails
|
||||
size = "200x200" # Thumbnail dimensions
|
||||
thumbnailintervalscan = "1h" # Interval for scheduled thumbnail generation
|
||||
concurrency = 5 # Number of concurrent thumbnail generation tasks
|
||||
|
||||
# ISO settings
|
||||
[iso]
|
||||
enabled = false
|
||||
size = "1TB" # Maximum ISO size
|
||||
mountpoint = "/path/to/hmac-file-server/iso/" # ISO mount point
|
||||
charset = "utf-8" # Filesystem character set encoding
|
||||
|
||||
# Timeout settings
|
||||
[timeouts]
|
||||
readtimeout = "3600s" # Maximum time to read a request (1 hour)
|
||||
writetimeout = "3600s" # Maximum time to write a response (1 hour)
|
||||
idletimeout = "3600s" # Maximum keep-alive time for idle connections (1 hour)
|
||||
|
||||
# Security settings
|
||||
[security]
|
||||
secret = "your-secure-secret-key" # HMAC shared secret key (change to a secure value)
|
||||
|
||||
# Versioning settings
|
||||
[versioning]
|
||||
enableversioning = false
|
||||
maxversions = 1 # Number of file versions to retain
|
||||
|
||||
# Upload settings
|
||||
[uploads]
|
||||
resumableuploadsenabled = false
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "32MB" # Chunk size for uploads
|
||||
allowedextensions = ["*"] # All file types are allowed for uploads
|
||||
|
||||
# Downloads settings
|
||||
[downloads]
|
||||
resumabledownloadsenabled = false
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "32MB"
|
||||
|
||||
# ClamAV settings
|
||||
[clamav]
|
||||
clamavenabled = true
|
||||
clamavsocket = "/path/to/clamav/clamd.ctl" # Path to ClamAV socket
|
||||
numscanworkers = 4 # Number of concurrent scan workers
|
||||
scanfileextensions = [
|
||||
".exe", ".dll", ".bin", ".com", ".bat",
|
||||
".sh", ".php", ".js"
|
||||
]
|
||||
|
||||
# Redis settings
|
||||
[redis]
|
||||
redisenabled = true
|
||||
redisdbindex = 0
|
||||
redisaddr = "localhost:6379" # Redis server address
|
||||
redispassword = "" # Redis password if required
|
||||
redishealthcheckinterval = "120s" # Interval for Redis health checks
|
||||
|
||||
# Worker settings
|
||||
[workers]
|
||||
numworkers = 4
|
||||
uploadqueuesize = 5000
|
||||
|
||||
# File settings
|
||||
[file]
|
||||
filerevision = 1 # Internal revision number for file handling logic
|
||||
|
||||
# Logging settings
|
||||
[logging]
|
||||
level = "info"
|
||||
file = "/var/log/hmac-file-server.log"
|
||||
max_size = 100
|
||||
max_backups = 7
|
||||
max_age = 30
|
||||
compress = true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Verification
|
||||
|
||||
The application ensures that all configuration parameters defined in `config.toml` are correctly implemented and operational. During startup, the server performs the following verification steps:
|
||||
|
||||
1. **Configuration Parsing:**
|
||||
- Parses the `config.toml` file into the application's `Config` struct.
|
||||
|
||||
2. **Validation:**
|
||||
- Checks that all required configuration parameters are set.
|
||||
- Validates the correctness of configuration values (e.g., proper duration formats, non-empty directories).
|
||||
|
||||
3. **Service Initialization:**
|
||||
- Initializes services like Redis and ClamAV based on the configuration.
|
||||
- Ensures that conditional services (e.g., Thumbnails, ISO) are only started if enabled in the configuration.
|
||||
|
||||
If any configuration parameter is missing or invalid, the server will log an error and terminate to prevent running with incorrect settings.
|
||||
|
||||
---
|
||||
|
||||
## Running the Server
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run the server with a configuration file:
|
||||
|
||||
```bash
|
||||
./hmac-file-server -config ./config.toml
|
||||
```
|
||||
|
||||
Ensure that the `config.toml` is correctly set up before running the server.
|
||||
|
||||
---
|
||||
|
||||
### Metrics Server
|
||||
|
||||
If `metricsenabled` is set to `true`, the Prometheus metrics server will be available on the port specified in `metricsport` (e.g., `http://localhost:9090/metrics`).
|
||||
|
||||
Additional metrics for deduplication and ISO container operations are now available.
|
||||
|
||||
---
|
||||
|
||||
## Health Checks
|
||||
|
||||
The server includes built-in health checks to verify that all configured services are operational. Logs will provide detailed information about the status of each service and any issues encountered during startup.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
To run the server locally for development:
|
||||
|
||||
```bash
|
||||
go run main.go -config ./config.toml
|
||||
```
|
||||
|
||||
Use tools like **cURL** or **Postman** to test file uploads and downloads.
|
||||
|
||||
### Example File Upload with HMAC Token
|
||||
|
||||
```bash
|
||||
curl -X PUT -H "Authorization: Bearer <HMAC-TOKEN>" -F "file=@example.txt" http://localhost:8080/uploads/example.txt
|
||||
```
|
||||
|
||||
Replace `<HMAC-TOKEN>` with a valid HMAC signature generated using the configured `secret`.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
Prometheus metrics include:
|
||||
|
||||
- **File Operations:**
|
||||
- File upload/download durations
|
||||
- Uploaded/downloaded file sizes
|
||||
- Total number of uploads/downloads
|
||||
- Total number of upload/download errors
|
||||
|
||||
- **System Metrics:**
|
||||
- Memory usage
|
||||
- CPU usage
|
||||
- Number of goroutines
|
||||
- Active connections
|
||||
|
||||
- **HTTP Requests:**
|
||||
- Total HTTP requests broken down by method and path
|
||||
|
||||
Access the metrics at `http://localhost:9090/metrics` (assuming default `metricsport`).
|
||||
|
||||
---
|
||||
// ...existing code...
|
||||
|
||||
## Additional Features
|
||||
|
||||
- **Deduplication:** Automatically remove duplicate files based on SHA256 hashing to save storage space.
|
||||
- **Versioning:** Store multiple versions of files and retain a configurable number of versions.
|
||||
- **ClamAV Integration:** Scan uploaded files for viruses using ClamAV to ensure file integrity and security.
|
||||
- **Redis Caching:** Utilize Redis for caching file metadata to improve access times and performance.
|
||||
- **Auto-Adjust Worker Scaling:** Optimize the number of workers dynamically based on system resources to maintain optimal performance.
|
||||
- **Precaching:** Pre-cache file structures on startup to reduce access times for frequently accessed files.
|
||||
- **Thumbnail Creation:** Generate image thumbnails for uploaded files to provide quick previews.
|
||||
- **ISO Container Management:** Optional mounting and handling of ISO-based filesystems for specialized storage needs.
|
||||
// ...existing features...
|
||||
|
||||
---
|
||||
// Removed Thumbnail Creation details
|
||||
|
||||
### Overview of Other Projects (xep0363)
|
||||
|
||||
| Feature/Project | HMAC FS | mod_http_upload_ext | xmpp-http-upload (horazont) | Prosody Filer | ngx_http_upload | xmpp-http-upload (nyovaya) |
|
||||
|-----------------------------|---------|----------------------|-----------------------------|---------------|------------------|----------------------------|
|
||||
| **Language** | Go | PHP | Python | Go | C (Nginx) | Python |
|
||||
| **Environment** | Standalone | Prosody module | Standalone | Standalone | Nginx | Standalone |
|
||||
| **XMPP** | No | Yes | Yes | Yes | No | Yes |
|
||||
| **External Storage** | Yes | No | Possible via plugins | No | No | Yes |
|
||||
| **Authentication / Security** | HMAC | Token-based | Token-based | None | Basic / None | Token-based |
|
||||
|
||||
---
|
||||
|
||||
## Build & Run
|
||||
|
||||
1. **Clone the Repository:**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
cd hmac-file-server
|
||||
```
|
||||
|
||||
2. **Build the Server:**
|
||||
|
||||
```bash
|
||||
go build -o hmac-file-server main.go
|
||||
```
|
||||
|
||||
3. **Run the Server:**
|
||||
|
||||
```bash
|
||||
./hmac-file-server -config ./config.toml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices and Recommendations
|
||||
|
||||
1. **Configuration Consistency:**
|
||||
- Ensure all configuration keys in `config.toml` match those expected in the code. For example, in the `[thumbnails]` section, use `thumbnailintervalscan` instead of `scanInterval` to align with the code.
|
||||
|
||||
2. **Security:**
|
||||
- **HMAC Secret:**
|
||||
- **Change Immediately:** The `secret` under `[security]` should be a strong, unique string. Replace `"your-secure-secret-key"` with a securely generated key.
|
||||
- **ClamAV:**
|
||||
- Ensure ClamAV is installed and the `clamavsocket` path is correct if `[clamav].clamavenabled` is `true`.
|
||||
- **Redis:**
|
||||
- Secure your Redis instance, especially if it's exposed to external networks. Use strong passwords and restrict access as needed.
|
||||
|
||||
3. **Resource Allocation:**
|
||||
- Adjust `numworkers` and `uploadqueuesize` in the `[workers]` section based on your server's hardware capabilities and expected traffic.
|
||||
- Monitor system performance to ensure that auto-adjust features are working optimally.
|
||||
|
||||
4. **Monitoring and Metrics:**
|
||||
- Regularly monitor Prometheus metrics to keep an eye on server performance, resource usage, and potential issues.
|
||||
- Set up alerts based on critical metrics to proactively handle problems.
|
||||
|
||||
5. **Logging:**
|
||||
- Ensure that log files are rotated and managed properly to prevent disk space issues.
|
||||
- Consider enabling JSON logging (`loggingjson = true`) for better integration with log management systems.
|
||||
|
||||
6. **Testing:**
|
||||
- Perform thorough testing of file uploads/downloads, especially with large files and under high load.
|
||||
- Test ClamAV scanning with both clean and malicious files to ensure security features are functioning correctly.
|
||||
|
||||
7. **Maintenance:**
|
||||
- Regularly update dependencies to benefit from security patches and performance improvements.
|
||||
- Clean up old file versions and ensure deduplication is functioning to optimize storage usage.
|
||||
|
||||
8. **Backup:**
|
||||
- Implement a backup strategy for critical data, especially if file versioning is enabled.
|
||||
|
||||
9. **Documentation:**
|
||||
- Keep the `README.md` and other documentation up-to-date with any code changes to assist in maintenance and onboarding new contributors.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Cannot Connect to Redis:**
|
||||
- Ensure Redis is running and accessible at the address specified in `redisaddr`.
|
||||
- Verify that the `redispassword` is correct if authentication is enabled.
|
||||
|
||||
- **ClamAV Scanning Fails:**
|
||||
- Check if ClamAV is installed and the `clamavsocket` path is correct.
|
||||
- Ensure that the ClamAV daemon (`clamd`) is running.
|
||||
|
||||
- **Insufficient Disk Space:**
|
||||
- Monitor the disk space and adjust `minfreebytes` in the configuration as needed.
|
||||
- Enable file TTL and cleanup to automatically remove old files.
|
||||
|
||||
- **Metrics Not Available:**
|
||||
- Ensure that `metricsenabled` is set to `true` and the server is running without errors.
|
||||
- Check if the specified `metricsport` is not blocked by a firewall.
|
||||
|
||||
- **Thumbnail Generation Not Working:**
|
||||
- Verify that the `[thumbnails].enabled` is set to `true` and the `directory` exists with appropriate permissions.
|
||||
- Check the logs for any errors related to image processing.
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please follow these steps to contribute:
|
||||
|
||||
1. **Fork the Repository**
|
||||
2. **Create a Feature Branch**
|
||||
|
||||
```bash
|
||||
git checkout -b feature/YourFeatureName
|
||||
```
|
||||
|
||||
3. **Commit Your Changes**
|
||||
|
||||
```bash
|
||||
git commit -m "Add your message here"
|
||||
```
|
||||
|
||||
4. **Push to the Branch**
|
||||
|
||||
```bash
|
||||
git push origin feature/YourFeatureName
|
||||
```
|
||||
|
||||
5. **Open a Pull Request**
|
||||
|
||||
Provide a clear description of your changes and the problem they address.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the [MIT License](LICENSE).
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
For a detailed list of changes, please refer to the [Changelog](./Changelog.md).
|
||||
|
||||
By following this updated `README.md`, you can ensure that users and contributors have a clear understanding of the **HMAC File Server**'s capabilities, configuration options, and best practices for deployment and maintenance.
|
||||
// ...existing code...
|
||||
|
||||
294
RELEASE-NOTES.MD
294
RELEASE-NOTES.MD
@ -1,294 +1,6 @@
|
||||
# HMAC File Server
|
||||
## [2.6.0] - 2025-05-01
|
||||
|
||||
**HMAC File Server** is a secure, scalable, and feature-rich file server with advanced capabilities like HMAC authentication, resumable uploads, chunked uploads, file versioning, and optional ClamAV scanning for file integrity and security. This server is built with extensibility and operational monitoring in mind, including Prometheus metrics support and Redis integration.
|
||||
|
||||
> **Credits:** The **HMAC File Server** is based on the source code of [Thomas Leister's prosody-filer](https://github.com/ThomasLeister/prosody-filer). Many features and design elements have been inspired or derived from this project.
|
||||
### Removed
|
||||
- **Thumbnail Support:** The thumbnail generation feature has been removed from the HMAC File Server. This includes the elimination of all related configuration options and dependencies to streamline server operations.
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
- **HMAC Authentication:** Secure file uploads and downloads with HMAC tokens.
|
||||
- **File Versioning:** Enable versioning for uploaded files with configurable retention.
|
||||
- **Chunked and Resumable Uploads:** Handle large files efficiently with support for resumable and chunked uploads.
|
||||
- **ClamAV Scanning:** Optional virus scanning for uploaded files.
|
||||
- **Prometheus Metrics:** Monitor system and application-level metrics.
|
||||
- **Redis Integration:** Use Redis for caching or storing application states.
|
||||
- **File Expiration:** Automatically delete files after a specified TTL.
|
||||
- **Graceful Shutdown:** Handles signals and ensures proper cleanup.
|
||||
- **Deduplication:** Remove duplicate files based on hashing for storage efficiency.
|
||||
- **Auto-Adjust Worker Scaling:** Dynamically optimize HMAC and ClamAV workers based on system resources when enabled.
|
||||
- **Thumbnail Support:** Generate smaller versions of uploaded images for efficient storage and quick access.
|
||||
|
||||
---
|
||||
|
||||
## Repository
|
||||
|
||||
- **Primary Repository**: [GitHub Repository](https://github.com/PlusOne/hmac-file-server)
|
||||
- **Alternative Repository**: [uuxo.net Git Repository](https://git.uuxo.net/uuxo/hmac-file-server)
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Go 1.20+
|
||||
- Redis (optional, if Redis integration is enabled)
|
||||
- ClamAV (optional, if file scanning is enabled)
|
||||
|
||||
### Clone and Build
|
||||
|
||||
```bash
|
||||
# Clone from the primary repository
|
||||
git clone https://github.com/PlusOne/hmac-file-server.git
|
||||
|
||||
# OR clone from the alternative repository
|
||||
git clone https://git.uuxo.net/uuxo/hmac-file-server.git
|
||||
|
||||
cd hmac-file-server
|
||||
go build -o hmac-file-server main.go
|
||||
```
|
||||
|
||||
### Building for Different Architectures
|
||||
|
||||
To build the HMAC File Server for different architectures, use the following commands:
|
||||
|
||||
#### Build for `arm64`
|
||||
```bash
|
||||
cd /path/to/hmac-file-server
|
||||
GOOS=linux GOARCH=arm64 go build -o ~/Temp/hmac-file-server-2.2-stable_arm64 main.go
|
||||
```
|
||||
|
||||
#### Build for `amd64`
|
||||
```bash
|
||||
cd /path/to/hmac-file-server
|
||||
GOOS=linux GOARCH=amd64 go build -o ~/Temp/hmac-file-server-2.2-stable_amd64 main.go
|
||||
```
|
||||
|
||||
Replace `/path/to/hmac-file-server` with the actual path to your project directory. These commands will generate the binaries for `arm64` and `amd64` architectures and place them in the `~/Temp` directory.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
The server configuration is managed through a `config.toml` file. Below are the supported configuration options:
|
||||
|
||||
### Auto-Adjust Feature
|
||||
|
||||
When `AutoAdjustWorkers` is enabled, the number of workers for HMAC operations and ClamAV scans is dynamically determined based on system resources. This ensures efficient resource utilization.
|
||||
|
||||
If `AutoAdjustWorkers = true`, the values for `NumWorkers` and `NumScanWorkers` in the configuration file will be ignored, and the server will automatically adjust these values.
|
||||
|
||||
### Network Events Monitoring
|
||||
|
||||
Setting `NetworkEvents = false` in the server configuration disables the logging and tracking of network-related events within the application. This means that functionalities such as monitoring IP changes or recording network activity will be turned off.
|
||||
|
||||
### Precaching
|
||||
|
||||
The `precaching` feature allows the server to pre-cache storage paths for faster access. This can improve performance by reducing the time needed to access frequently used storage paths.
|
||||
|
||||
### Thumbnail Support
|
||||
|
||||
- New configuration options in `[thumbnails]` to enable or disable generating image thumbnails, set the thumbnail size, and configure concurrency for thumbnail generation.
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### Deduplication Support
|
||||
|
||||
- **Description:** Added support for file deduplication to save storage space by storing a single copy of identical files.
|
||||
- **Configuration:**
|
||||
```toml
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/mnt/hmac-storage/deduplication/"
|
||||
```
|
||||
|
||||
### Thumbnail Support
|
||||
|
||||
- **Description:** Added support for thumbnail creation to generate smaller versions of uploaded images.
|
||||
- **Configuration:**
|
||||
```toml
|
||||
[thumbnails]
|
||||
enabled = true
|
||||
directory = "/mnt/hmac-storage/thumbnails/"
|
||||
size = "200x200"
|
||||
concurrency = 5
|
||||
thumbnailintervalscan = "24h"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example `config.toml`
|
||||
|
||||
```toml
|
||||
[server]
|
||||
ListenPort = "8080"
|
||||
UnixSocket = false
|
||||
StoragePath = "./uploads"
|
||||
LogLevel = "info"
|
||||
LogFile = ""
|
||||
MetricsEnabled = true
|
||||
MetricsPort = "9090"
|
||||
FileTTL = "1y"
|
||||
FileTTLEnabled = true # Enable or disable file TTL
|
||||
MinFreeBytes = "100MB"
|
||||
AutoAdjustWorkers = true # Enable auto-adjustment for worker scaling
|
||||
NetworkEvents = false # Disable logging and tracking of network-related events
|
||||
PIDFilePath = "./hmac_file_server.pid" # Path to PID file
|
||||
Precaching = true # Enable pre-caching of storage paths
|
||||
TempPath = "/tmp/uploads" # Configurable temporary upload directory
|
||||
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "/mnt/nfs_vol01/hmac-storage/deduplication/"
|
||||
|
||||
[iso]
|
||||
Enabled = false
|
||||
Size = "2TB"
|
||||
MountPoint = "/mnt/iso"
|
||||
Charset = "utf-8"
|
||||
|
||||
[thumbnails]
|
||||
enabled = true
|
||||
directory = "/mnt/nfs_vol01/hmac-file-server/thumbnails/"
|
||||
size = "200x200"
|
||||
concurrency = 5
|
||||
thumbnailintervalscan = "24h"
|
||||
|
||||
[iso]
|
||||
enabled = false
|
||||
size = "1TB"
|
||||
mountpoint = "/mnt/nfs_vol01/hmac-file-server/iso/"
|
||||
charset = "utf-8"
|
||||
|
||||
[timeouts]
|
||||
ReadTimeout = "480s"
|
||||
WriteTimeout = "480s"
|
||||
IdleTimeout = "65s" # nginx/apache2 keep-alive 60s
|
||||
|
||||
[security]
|
||||
Secret = "changeme"
|
||||
|
||||
[versioning]
|
||||
EnableVersioning = false
|
||||
MaxVersions = 1
|
||||
|
||||
[uploads]
|
||||
ResumableUploadsEnabled = true
|
||||
ChunkedUploadsEnabled = true
|
||||
ChunkSize = "64MB"
|
||||
AllowedExtensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm", ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"]
|
||||
|
||||
[downloads]
|
||||
ResumableDownloadsEnabled = true
|
||||
ChunkedDownloadsEnabled = true
|
||||
ChunkSize = "64MB"
|
||||
|
||||
[clamav]
|
||||
ClamAVEnabled = false
|
||||
ClamAVSocket = "/var/run/clamav/clamd.ctl"
|
||||
NumScanWorkers = 2
|
||||
ScanFileExtensions = [".exe", ".dll", ".bin", ".com", ".bat", ".sh", ".php", ".js"]
|
||||
|
||||
[redis]
|
||||
RedisEnabled = false
|
||||
RedisAddr = "localhost:6379"
|
||||
RedisPassword = ""
|
||||
RedisDBIndex = 0
|
||||
RedisHealthCheckInterval = "120s"
|
||||
|
||||
[workers]
|
||||
NumWorkers = 4
|
||||
UploadQueueSize = 5000
|
||||
|
||||
[file]
|
||||
FileRevision = 1 # Revision number for file handling
|
||||
|
||||
[logging]
|
||||
level = "info"
|
||||
file = "/var/log/hmac-file-server.log"
|
||||
max_size = 100
|
||||
max_backups = 7
|
||||
max_age = 30
|
||||
compress = true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running the Server
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run the server with a configuration file:
|
||||
|
||||
```bash
|
||||
./hmac-file-server -config ./config.toml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Metrics Server
|
||||
|
||||
If `MetricsEnabled` is set to `true`, the Prometheus metrics server will be available on the port specified in `MetricsPort` (default: `9090`).
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
To run the server locally for development:
|
||||
|
||||
```bash
|
||||
go run main.go -config ./config.toml
|
||||
```
|
||||
|
||||
Use tools like **cURL** or **Postman** to test file uploads and downloads.
|
||||
|
||||
### Example File Upload with HMAC Token
|
||||
|
||||
```bash
|
||||
curl -X PUT -H "Authorization: Bearer <HMAC-TOKEN>" -F "file=@example.txt" http://localhost:8080/uploads/example.txt
|
||||
```
|
||||
|
||||
Replace `<HMAC-TOKEN>` with a valid HMAC signature
|
||||
|
||||
## [2.4.1] - 2025-03-10
|
||||
### Changed
|
||||
- **Configuration:** Updated `globalextensions` in `config.toml` to `["*"]`, allowing all file types globally for uploads. This change simplifies the configuration by removing the need to specify individual file extensions.
|
||||
|
||||
## [2.4.0] - 2025-02-20
|
||||
### Added
|
||||
- **Pre-Caching Support:** Introduced pre-caching of storage paths to improve access speeds.
|
||||
- **ISO Container Management:** Added functionality to create and mount ISO containers for specialized storage needs.
|
||||
- **Thumbnail Concurrency Parameter:** Users can now set the level of concurrency for thumbnail generation to optimize performance.
|
||||
|
||||
### Changed
|
||||
- **Configuration Options:** Updated `config.toml` to include new settings for pre-caching and ISO management.
|
||||
- **Documentation:** Enhanced `README.MD` with detailed instructions on new features and best practices.
|
||||
|
||||
### Fixed
|
||||
- **Bug Fixes:** Resolved minor issues related to file versioning and deduplication processes.
|
||||
|
||||
## [2.3.1] - 2025-01-15
|
||||
### Changed
|
||||
- **Configuration:** Updated `globalextensions` in `config.toml` to `["*"]`, allowing all file types globally for uploads. This change simplifies the configuration by removing the need to specify individual file extensions.
|
||||
|
||||
## [2.3.0] - 2024-12-28
|
||||
### Changed
|
||||
- **Server:** Replaced the hardcoded temporary upload directory `/tmp/uploads` with a configurable `TempPath` parameter in `config.toml`. Ensure to set `tempPath` in your configuration file accordingly.
|
||||
|
||||
## [2.2.2] - 2024-12-27
|
||||
### Bug Fixes
|
||||
- Resolved issue where temporary `.tmp` files caused "Unsupported file type" warnings by adjusting MIME type detection to use the final file extension.
|
||||
|
||||
### Enhancements
|
||||
- Improved logging for file extension and MIME type during uploads.
|
||||
|
||||
## [2.2.1] - 2024-12-27
|
||||
### Enhancements
|
||||
- Added detailed logging for file extensions and MIME types during file uploads to assist in diagnosing unsupported file type issues.
|
||||
|
||||
### Configuration
|
||||
- Updated `config.toml` to ensure necessary file extensions are allowed for uploads.
|
||||
|
||||
@ -1,7 +1,6 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
@ -10,6 +9,10 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"context"
|
||||
"io"
|
||||
"sync"
|
||||
"bufio"
|
||||
|
||||
"github.com/gdamore/tcell/v2"
|
||||
"github.com/pelletier/go-toml"
|
||||
@ -20,9 +23,13 @@ import (
|
||||
"github.com/shirou/gopsutil/v3/process"
|
||||
)
|
||||
|
||||
var prometheusURL string
|
||||
var configFilePath string // Pfad der gefundenen Konfiguration
|
||||
var logFilePath string // Pfad der Logdatei aus der Konfiguration
|
||||
var (
|
||||
prometheusURL string
|
||||
configFilePath string // Pfad der gefundenen Konfiguration
|
||||
logFilePath string // Pfad der Logdatei aus der Konfiguration
|
||||
metricsEnabled bool // Neue Variable für die Aktivierung von Metriken
|
||||
bindIP string // Neue Variable für die gebundene IP-Adresse
|
||||
)
|
||||
|
||||
func init() {
|
||||
configPaths := []string{
|
||||
@ -69,7 +76,35 @@ func init() {
|
||||
log.Fatalf("Error: 'server.metricsport' is not of type int64 or string, got %T", v)
|
||||
}
|
||||
|
||||
prometheusURL = fmt.Sprintf("http://localhost:%d/metrics", port)
|
||||
// Lesen von 'metricsenabled' aus der Konfiguration
|
||||
metricsEnabledValue := config.Get("server.metricsenabled")
|
||||
if metricsEnabledValue == nil {
|
||||
log.Println("Warning: 'server.metricsenabled' ist in der Konfiguration nicht gesetzt. Standardmäßig deaktiviert.")
|
||||
metricsEnabled = false
|
||||
} else {
|
||||
var ok bool
|
||||
metricsEnabled, ok = metricsEnabledValue.(bool)
|
||||
if !ok {
|
||||
log.Fatalf("Konfigurationsfehler: 'server.metricsenabled' sollte ein boolescher Wert sein, aber %T wurde gefunden.", metricsEnabledValue)
|
||||
}
|
||||
}
|
||||
|
||||
// Lesen von 'bind_ip' aus der Konfiguration
|
||||
bindIPValue := config.Get("server.bind_ip")
|
||||
if bindIPValue == nil {
|
||||
log.Println("Warning: 'server.bind_ip' ist in der Konfiguration nicht gesetzt. Standardmäßig auf 'localhost' gesetzt.")
|
||||
bindIP = "localhost"
|
||||
} else {
|
||||
var ok bool
|
||||
bindIP, ok = bindIPValue.(string)
|
||||
if !ok {
|
||||
log.Fatalf("Konfigurationsfehler: 'server.bind_ip' sollte ein String sein, aber %T wurde gefunden.", bindIPValue)
|
||||
}
|
||||
}
|
||||
|
||||
// Konstruktion der prometheusURL basierend auf 'bind_ip' und 'metricsport'
|
||||
prometheusURL = fmt.Sprintf("http://%s:%d/metrics", bindIP, port)
|
||||
log.Printf("Metrics URL gesetzt auf: %s", prometheusURL)
|
||||
|
||||
// Log-Datei auslesen über server.logfile
|
||||
logFileValue := config.Get("server.logfile")
|
||||
@ -93,11 +128,17 @@ const (
|
||||
|
||||
// ProcessInfo holds information about a process
|
||||
type ProcessInfo struct {
|
||||
PID int32
|
||||
Name string
|
||||
CPUPercent float64
|
||||
MemPercent float32
|
||||
CommandLine string
|
||||
PID int32
|
||||
Name string
|
||||
CPUPercent float64
|
||||
MemPercent float32
|
||||
CommandLine string
|
||||
Uptime string // Neues Feld für die Uptime
|
||||
Status string // Neues Feld für den Status
|
||||
ErrorCount int // Neues Feld für die Anzahl der Fehler
|
||||
TotalRequests int64 // Neues Feld für die Gesamtanzahl der Anfragen
|
||||
ActiveConnections int // Neues Feld für aktive Verbindungen
|
||||
AverageResponseTime float64 // Neues Feld für die durchschnittliche Antwortzeit in Millisekunden
|
||||
}
|
||||
|
||||
// Function to fetch and parse Prometheus metrics
|
||||
@ -121,7 +162,9 @@ func fetchMetrics() (map[string]float64, error) {
|
||||
name == "memory_usage_bytes" ||
|
||||
name == "cpu_usage_percent" ||
|
||||
name == "active_connections_total" ||
|
||||
name == "goroutines_count" {
|
||||
name == "goroutines_count" ||
|
||||
name == "total_requests" ||
|
||||
name == "average_response_time_ms" {
|
||||
|
||||
for _, m := range mf.GetMetric() {
|
||||
var value float64
|
||||
@ -179,7 +222,7 @@ func fetchSystemData() (float64, float64, int, error) {
|
||||
return v.UsedPercent, cpuUsage, cores, nil
|
||||
}
|
||||
|
||||
// Function to fetch process list
|
||||
// Funktion zum Abrufen der Prozessliste mit paralleler Verarbeitung
|
||||
func fetchProcessList() ([]ProcessInfo, error) {
|
||||
processes, err := process.Processes()
|
||||
if err != nil {
|
||||
@ -187,37 +230,55 @@ func fetchProcessList() ([]ProcessInfo, error) {
|
||||
}
|
||||
|
||||
var processList []ProcessInfo
|
||||
var mu sync.Mutex
|
||||
var wg sync.WaitGroup
|
||||
|
||||
// Begrenzung der gleichzeitigen Goroutinen auf 10
|
||||
sem := make(chan struct{}, 10)
|
||||
|
||||
for _, p := range processes {
|
||||
cpuPercent, err := p.CPUPercent()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
sem <- struct{}{} // Eintritt in semaphor
|
||||
|
||||
memPercent, err := p.MemoryPercent()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
go func(p *process.Process) {
|
||||
defer wg.Done()
|
||||
defer func() { <-sem }() // Austritt aus semaphor
|
||||
|
||||
name, err := p.Name()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
cpuPercent, err := p.CPUPercent()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
cmdline, err := p.Cmdline()
|
||||
if err != nil {
|
||||
cmdline = ""
|
||||
}
|
||||
memPercent, err := p.MemoryPercent()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
processList = append(processList, ProcessInfo{
|
||||
PID: p.Pid,
|
||||
Name: name,
|
||||
CPUPercent: cpuPercent,
|
||||
MemPercent: memPercent,
|
||||
CommandLine: cmdline,
|
||||
})
|
||||
name, err := p.Name()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
cmdline, err := p.Cmdline()
|
||||
if err != nil {
|
||||
cmdline = ""
|
||||
}
|
||||
|
||||
info := ProcessInfo{
|
||||
PID: p.Pid,
|
||||
Name: name,
|
||||
CPUPercent: cpuPercent,
|
||||
MemPercent: memPercent,
|
||||
CommandLine: cmdline,
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
processList = append(processList, info)
|
||||
mu.Unlock()
|
||||
}(p)
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
return processList, nil
|
||||
}
|
||||
|
||||
@ -250,12 +311,57 @@ func fetchHmacFileServerInfo() (*ProcessInfo, error) {
|
||||
cmdline = ""
|
||||
}
|
||||
|
||||
createTime, err := p.CreateTime()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get process start time: %w", err)
|
||||
}
|
||||
uptime := time.Since(time.Unix(0, createTime*int64(time.Millisecond)))
|
||||
|
||||
status := "Running" // Standardstatus
|
||||
|
||||
// Überprüfung, ob der Prozess aktiv ist
|
||||
isRunning, err := p.IsRunning()
|
||||
if err != nil || !isRunning {
|
||||
status = "Stopped"
|
||||
}
|
||||
|
||||
errorCount, err := countHmacErrors()
|
||||
if err != nil {
|
||||
errorCount = 0
|
||||
}
|
||||
|
||||
metrics, err := fetchMetrics()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to fetch metrics: %w", err)
|
||||
}
|
||||
|
||||
totalRequests, ok := metrics["total_requests"]
|
||||
if !ok {
|
||||
totalRequests = 0
|
||||
}
|
||||
|
||||
activeConnections, ok := metrics["active_connections_total"]
|
||||
if !ok {
|
||||
activeConnections = 0
|
||||
}
|
||||
|
||||
averageResponseTime, ok := metrics["average_response_time_ms"]
|
||||
if !ok {
|
||||
averageResponseTime = 0.0
|
||||
}
|
||||
|
||||
return &ProcessInfo{
|
||||
PID: p.Pid,
|
||||
Name: name,
|
||||
CPUPercent: cpuPercent,
|
||||
MemPercent: memPercent,
|
||||
CommandLine: cmdline,
|
||||
PID: p.Pid,
|
||||
Name: name,
|
||||
CPUPercent: cpuPercent,
|
||||
MemPercent: memPercent,
|
||||
CommandLine: cmdline,
|
||||
Uptime: uptime.String(),
|
||||
Status: status,
|
||||
ErrorCount: errorCount,
|
||||
TotalRequests: int64(totalRequests),
|
||||
ActiveConnections: int(activeConnections),
|
||||
AverageResponseTime: averageResponseTime,
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
@ -263,63 +369,203 @@ func fetchHmacFileServerInfo() (*ProcessInfo, error) {
|
||||
return nil, fmt.Errorf("hmac-file-server process not found")
|
||||
}
|
||||
|
||||
// Function to update the UI with the latest data
|
||||
func updateUI(app *tview.Application, pages *tview.Pages, sysPage, hmacPage tview.Primitive) {
|
||||
// Neue Funktion zur Zählung der Fehler in den Logs
|
||||
func countHmacErrors() (int, error) {
|
||||
logFilePath := "/var/log/hmac-file-server.log" // Pfad zur Logdatei
|
||||
file, err := os.Open(logFilePath)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
scanner := bufio.NewScanner(file)
|
||||
errorCount := 0
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
if strings.Contains(line, "level=error") {
|
||||
errorCount++
|
||||
}
|
||||
}
|
||||
|
||||
if err := scanner.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return errorCount, nil
|
||||
}
|
||||
|
||||
// Funktion zur Aktualisierung der UI mit paralleler Datenbeschaffung
|
||||
func updateUI(ctx context.Context, app *tview.Application, pages *tview.Pages, sysPage, hmacPage tview.Primitive) {
|
||||
ticker := time.NewTicker(2 * time.Second)
|
||||
defer ticker.Stop()
|
||||
|
||||
for range ticker.C {
|
||||
// Fetch data for both views
|
||||
memUsage, cpuUsage, cores, err := fetchSystemData()
|
||||
if err != nil {
|
||||
log.Printf("Error fetching system data: %v\n", err)
|
||||
continue
|
||||
}
|
||||
|
||||
metrics, err := fetchMetrics()
|
||||
if err != nil {
|
||||
log.Printf("Error fetching metrics: %v\n", err)
|
||||
continue
|
||||
}
|
||||
|
||||
processes, err := fetchProcessList()
|
||||
if err != nil {
|
||||
log.Printf("Error fetching process list: %v\n", err)
|
||||
continue
|
||||
}
|
||||
|
||||
hmacInfo, err := fetchHmacFileServerInfo()
|
||||
if err != nil {
|
||||
log.Printf("Error fetching hmac-file-server info: %v\n", err)
|
||||
}
|
||||
|
||||
// Update the UI
|
||||
app.QueueUpdateDraw(func() {
|
||||
// Update system page
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
|
||||
sysFlex := sysPage.(*tview.Flex)
|
||||
|
||||
// Update system data table
|
||||
sysTable := sysFlex.GetItem(0).(*tview.Table)
|
||||
updateSystemTable(sysTable, memUsage, cpuUsage, cores)
|
||||
|
||||
// Update metrics table
|
||||
metricsTable := sysFlex.GetItem(1).(*tview.Table)
|
||||
updateMetricsTable(metricsTable, metrics)
|
||||
|
||||
// Update process table
|
||||
processTable := sysFlex.GetItem(2).(*tview.Table)
|
||||
updateProcessTable(processTable, processes)
|
||||
}
|
||||
|
||||
// Update hmac-file-server page
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "hmac" && hmacInfo != nil {
|
||||
hmacFlex := hmacPage.(*tview.Flex)
|
||||
hmacTable := hmacFlex.GetItem(0).(*tview.Table)
|
||||
updateHmacTable(hmacTable, hmacInfo, metrics)
|
||||
}
|
||||
// Einführung von Channels für verschiedene Daten
|
||||
systemDataCh := make(chan struct {
|
||||
memUsage float64
|
||||
cpuUsage float64
|
||||
cores int
|
||||
err error
|
||||
})
|
||||
var metricsCh chan struct {
|
||||
metrics map[string]float64
|
||||
err error
|
||||
}
|
||||
if metricsEnabled {
|
||||
metricsCh = make(chan struct {
|
||||
metrics map[string]float64
|
||||
err error
|
||||
})
|
||||
}
|
||||
processListCh := make(chan struct {
|
||||
processes []ProcessInfo
|
||||
err error
|
||||
})
|
||||
hmacInfoCh := make(chan struct {
|
||||
info *ProcessInfo
|
||||
metrics map[string]float64
|
||||
err error
|
||||
})
|
||||
|
||||
// Goroutine zur Datenbeschaffung
|
||||
go func() {
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
close(systemDataCh)
|
||||
if metricsEnabled {
|
||||
close(metricsCh)
|
||||
}
|
||||
close(processListCh)
|
||||
close(hmacInfoCh)
|
||||
return
|
||||
case <-ticker.C:
|
||||
// Systemdaten abrufen asynchron
|
||||
go func() {
|
||||
memUsage, cpuUsage, cores, err := fetchSystemData()
|
||||
systemDataCh <- struct {
|
||||
memUsage float64
|
||||
cpuUsage float64
|
||||
cores int
|
||||
err error
|
||||
}{memUsage, cpuUsage, cores, err}
|
||||
}()
|
||||
|
||||
if metricsEnabled {
|
||||
// Metriken abrufen asynchron
|
||||
go func() {
|
||||
metrics, err := fetchMetrics()
|
||||
metricsCh <- struct {
|
||||
metrics map[string]float64
|
||||
err error
|
||||
}{metrics, err}
|
||||
}()
|
||||
}
|
||||
|
||||
// Prozessliste abrufen asynchron
|
||||
go func() {
|
||||
processes, err := fetchProcessList()
|
||||
processListCh <- struct {
|
||||
processes []ProcessInfo
|
||||
err error
|
||||
}{processes, err}
|
||||
}()
|
||||
|
||||
// hmac-file-server Informationen abrufen asynchron
|
||||
go func() {
|
||||
hmacInfo, err := fetchHmacFileServerInfo()
|
||||
var metrics map[string]float64
|
||||
if metricsEnabled {
|
||||
metrics, err = fetchMetrics()
|
||||
}
|
||||
hmacInfoCh <- struct {
|
||||
info *ProcessInfo
|
||||
metrics map[string]float64
|
||||
err error
|
||||
}{hmacInfo, metrics, err}
|
||||
}()
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case data, ok := <-systemDataCh:
|
||||
if !ok {
|
||||
systemDataCh = nil
|
||||
continue
|
||||
}
|
||||
if data.err != nil {
|
||||
log.Printf("Fehler beim Abrufen der Systemdaten: %v\n", data.err)
|
||||
continue
|
||||
}
|
||||
// UI aktualisieren mit Systemdaten
|
||||
app.QueueUpdateDraw(func() {
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
|
||||
sysFlex := sysPage.(*tview.Flex)
|
||||
sysTable := sysFlex.GetItem(0).(*tview.Table)
|
||||
updateSystemTable(sysTable, data.memUsage, data.cpuUsage, data.cores)
|
||||
}
|
||||
})
|
||||
case data, ok := <-metricsCh:
|
||||
if !ok {
|
||||
metricsCh = nil
|
||||
continue
|
||||
}
|
||||
if data.err != nil {
|
||||
log.Printf("Fehler beim Abrufen der Metriken: %v\n", data.err)
|
||||
continue
|
||||
}
|
||||
// UI aktualisieren mit Metriken
|
||||
app.QueueUpdateDraw(func() {
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
|
||||
sysFlex := sysPage.(*tview.Flex)
|
||||
metricsTable := sysFlex.GetItem(1).(*tview.Table)
|
||||
updateMetricsTable(metricsTable, data.metrics)
|
||||
}
|
||||
})
|
||||
case data, ok := <-processListCh:
|
||||
if !ok {
|
||||
processListCh = nil
|
||||
continue
|
||||
}
|
||||
if data.err != nil {
|
||||
log.Printf("Fehler beim Abrufen der Prozessliste: %v\n", data.err)
|
||||
continue
|
||||
}
|
||||
// UI aktualisieren mit Prozessliste
|
||||
app.QueueUpdateDraw(func() {
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "system" {
|
||||
sysFlex := sysPage.(*tview.Flex)
|
||||
processTable := sysFlex.GetItem(2).(*tview.Table)
|
||||
updateProcessTable(processTable, data.processes)
|
||||
}
|
||||
})
|
||||
case data, ok := <-hmacInfoCh:
|
||||
if !ok {
|
||||
hmacInfoCh = nil
|
||||
continue
|
||||
}
|
||||
if data.err != nil {
|
||||
log.Printf("Fehler beim Abrufen der hmac-file-server Informationen: %v\n", data.err)
|
||||
continue
|
||||
}
|
||||
// UI aktualisieren mit hmac-file-server Informationen
|
||||
app.QueueUpdateDraw(func() {
|
||||
if currentPage, _ := pages.GetFrontPage(); currentPage == "hmac" && data.info != nil {
|
||||
hmacFlex := hmacPage.(*tview.Flex)
|
||||
hmacTable := hmacFlex.GetItem(0).(*tview.Table)
|
||||
updateHmacTable(hmacTable, data.info, data.metrics)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Abbruchbedingung, wenn alle Channels geschlossen sind
|
||||
if systemDataCh == nil && (!metricsEnabled || metricsCh == nil) && processListCh == nil && hmacInfoCh == nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Helper function to update system data table
|
||||
@ -419,9 +665,27 @@ func updateHmacTable(hmacTable *tview.Table, hmacInfo *ProcessInfo, metrics map[
|
||||
|
||||
hmacTable.SetCell(4, 0, tview.NewTableCell("Command"))
|
||||
hmacTable.SetCell(4, 1, tview.NewTableCell(hmacInfo.CommandLine))
|
||||
|
||||
hmacTable.SetCell(5, 0, tview.NewTableCell("Uptime"))
|
||||
hmacTable.SetCell(5, 1, tview.NewTableCell(hmacInfo.Uptime)) // Neue Zeile für Uptime
|
||||
|
||||
hmacTable.SetCell(6, 0, tview.NewTableCell("Status"))
|
||||
hmacTable.SetCell(6, 1, tview.NewTableCell(hmacInfo.Status)) // Neue Zeile für Status
|
||||
|
||||
hmacTable.SetCell(7, 0, tview.NewTableCell("Error Count"))
|
||||
hmacTable.SetCell(7, 1, tview.NewTableCell(fmt.Sprintf("%d", hmacInfo.ErrorCount))) // Neue Zeile für Error Count
|
||||
|
||||
hmacTable.SetCell(8, 0, tview.NewTableCell("Total Requests"))
|
||||
hmacTable.SetCell(8, 1, tview.NewTableCell(fmt.Sprintf("%d", hmacInfo.TotalRequests))) // Neue Zeile für Total Requests
|
||||
|
||||
hmacTable.SetCell(9, 0, tview.NewTableCell("Active Connections"))
|
||||
hmacTable.SetCell(9, 1, tview.NewTableCell(fmt.Sprintf("%d", hmacInfo.ActiveConnections))) // Neue Zeile für Active Connections
|
||||
|
||||
hmacTable.SetCell(10, 0, tview.NewTableCell("Avg. Response Time (ms)"))
|
||||
hmacTable.SetCell(10, 1, tview.NewTableCell(fmt.Sprintf("%.2f", hmacInfo.AverageResponseTime))) // Neue Zeile für Average Response Time
|
||||
|
||||
// Metrics related to hmac-file-server
|
||||
row := 6
|
||||
row := 12
|
||||
hmacTable.SetCell(row, 0, tview.NewTableCell("Metric").SetAttributes(tcell.AttrBold))
|
||||
hmacTable.SetCell(row, 1, tview.NewTableCell("Value").SetAttributes(tcell.AttrBold))
|
||||
row++
|
||||
@ -469,72 +733,117 @@ func createHmacPage() tview.Primitive {
|
||||
return hmacFlex
|
||||
}
|
||||
|
||||
func createLogsPage(logFilePath string) tview.Primitive {
|
||||
logsTextView := tview.NewTextView().
|
||||
SetDynamicColors(true).
|
||||
SetRegions(true).
|
||||
SetWordWrap(true)
|
||||
logsTextView.SetTitle(" [::b]Logs ").SetBorder(true)
|
||||
func createLogsPage(ctx context.Context, app *tview.Application, logFilePath string) tview.Primitive {
|
||||
logsTextView := tview.NewTextView().
|
||||
SetDynamicColors(true).
|
||||
SetRegions(true).
|
||||
SetWordWrap(true)
|
||||
logsTextView.SetTitle(" [::b]Logs ").SetBorder(true)
|
||||
|
||||
const numLines = 100 // Number of lines to read from the end of the log file
|
||||
const numLines = 100 // Number of lines to read from the end of the log file
|
||||
|
||||
// Read logs periodically
|
||||
go func() {
|
||||
for {
|
||||
content, err := readLastNLines(logFilePath, numLines)
|
||||
if err != nil {
|
||||
logsTextView.SetText(fmt.Sprintf("[red]Error reading log file: %v[white]", err))
|
||||
} else {
|
||||
// Process the log content to add colors
|
||||
lines := strings.Split(content, "\n")
|
||||
var coloredLines []string
|
||||
for _, line := range lines {
|
||||
if strings.Contains(line, "level=info") {
|
||||
coloredLines = append(coloredLines, "[green]"+line+"[white]")
|
||||
} else if strings.Contains(line, "level=warn") {
|
||||
coloredLines = append(coloredLines, "[yellow]"+line+"[white]")
|
||||
} else if strings.Contains(line, "level=error") {
|
||||
coloredLines = append(coloredLines, "[red]"+line+"[white]")
|
||||
} else {
|
||||
// Default color
|
||||
coloredLines = append(coloredLines, line)
|
||||
}
|
||||
}
|
||||
logsTextView.SetText(strings.Join(coloredLines, "\n"))
|
||||
}
|
||||
time.Sleep(2 * time.Second) // Refresh interval for logs
|
||||
}
|
||||
}()
|
||||
// Read logs periodically
|
||||
go func() {
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
default:
|
||||
content, err := readLastNLines(logFilePath, numLines)
|
||||
if err != nil {
|
||||
app.QueueUpdateDraw(func() {
|
||||
logsTextView.SetText(fmt.Sprintf("[red]Error reading log file: %v[white]", err))
|
||||
})
|
||||
} else {
|
||||
// Process the log content to add colors
|
||||
lines := strings.Split(content, "\n")
|
||||
var coloredLines []string
|
||||
for _, line := range lines {
|
||||
if strings.Contains(line, "level=info") {
|
||||
coloredLines = append(coloredLines, "[green]"+line+"[white]")
|
||||
} else if strings.Contains(line, "level=warn") {
|
||||
coloredLines = append(coloredLines, "[yellow]"+line+"[white]")
|
||||
} else if strings.Contains(line, "level=error") {
|
||||
coloredLines = append(coloredLines, "[red]"+line+"[white]")
|
||||
} else {
|
||||
// Default color
|
||||
coloredLines = append(coloredLines, line)
|
||||
}
|
||||
}
|
||||
app.QueueUpdateDraw(func() {
|
||||
logsTextView.SetText(strings.Join(coloredLines, "\n"))
|
||||
})
|
||||
}
|
||||
time.Sleep(2 * time.Second) // Refresh interval for logs
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
return logsTextView
|
||||
return logsTextView
|
||||
}
|
||||
|
||||
// Optimized readLastNLines to handle large files efficiently
|
||||
func readLastNLines(filePath string, n int) (string, error) {
|
||||
file, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer file.Close()
|
||||
file, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
var lines []string
|
||||
scanner := bufio.NewScanner(file)
|
||||
for scanner.Scan() {
|
||||
lines = append(lines, scanner.Text())
|
||||
if len(lines) > n {
|
||||
lines = lines[1:]
|
||||
}
|
||||
}
|
||||
const bufferSize = 1024
|
||||
buffer := make([]byte, bufferSize)
|
||||
var content []byte
|
||||
var fileSize int64
|
||||
|
||||
if err := scanner.Err(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
fileInfo, err := file.Stat()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
fileSize = fileInfo.Size()
|
||||
|
||||
return strings.Join(lines, "\n"), nil
|
||||
var offset int64 = 0
|
||||
for {
|
||||
if fileSize-offset < bufferSize {
|
||||
offset = fileSize
|
||||
} else {
|
||||
offset += bufferSize
|
||||
}
|
||||
|
||||
_, err := file.Seek(-offset, io.SeekEnd)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
bytesRead, err := file.Read(buffer)
|
||||
if err != nil && err != io.EOF {
|
||||
return "", err
|
||||
}
|
||||
|
||||
content = append(buffer[:bytesRead], content...)
|
||||
|
||||
if bytesRead < bufferSize || len(strings.Split(string(content), "\n")) > n+1 {
|
||||
break
|
||||
}
|
||||
|
||||
if offset >= fileSize {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
lines := strings.Split(string(content), "\n")
|
||||
if len(lines) > n {
|
||||
lines = lines[len(lines)-n:]
|
||||
}
|
||||
return strings.Join(lines, "\n"), nil
|
||||
}
|
||||
|
||||
func main() {
|
||||
app := tview.NewApplication()
|
||||
|
||||
// Create a cancellable context
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
// Create pages
|
||||
pages := tview.NewPages()
|
||||
|
||||
@ -547,14 +856,15 @@ func main() {
|
||||
pages.AddPage("hmac", hmacPage, true, false)
|
||||
|
||||
// Logs page mit dem gelesenen logFilePath
|
||||
logsPage := createLogsPage(logFilePath)
|
||||
logsPage := createLogsPage(ctx, app, logFilePath)
|
||||
pages.AddPage("logs", logsPage, true, false)
|
||||
|
||||
// Add key binding to switch views
|
||||
// Add key binding to switch views and handle exit
|
||||
app.SetInputCapture(func(event *tcell.EventKey) *tcell.EventKey {
|
||||
if event.Key() == tcell.KeyRune {
|
||||
switch event.Rune() {
|
||||
case 'q', 'Q':
|
||||
cancel()
|
||||
app.Stop()
|
||||
return nil
|
||||
case 's', 'S':
|
||||
@ -572,10 +882,11 @@ func main() {
|
||||
})
|
||||
|
||||
// Start the UI update loop in a separate goroutine
|
||||
go updateUI(app, pages, sysPage, hmacPage)
|
||||
go updateUI(ctx, app, pages, sysPage, hmacPage)
|
||||
|
||||
// Set the root and run the application
|
||||
if err := app.SetRoot(pages, true).EnableMouse(true).Run(); err != nil {
|
||||
log.Fatalf("Error running application: %v", err)
|
||||
log.Fatalf("Error running application: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
84
config.toml
Normal file
84
config.toml
Normal file
@ -0,0 +1,84 @@
|
||||
[server]
|
||||
listenport = "8080"
|
||||
unixsocket = false
|
||||
storagepath = "./uploads"
|
||||
metricsenabled = true
|
||||
metricsport = "9090"
|
||||
filettl = "8760h"
|
||||
minfreebytes = "100MB"
|
||||
autoadjustworkers = true
|
||||
networkevents = true
|
||||
temppath = "/tmp/hmac-file-server"
|
||||
loggingjson = false
|
||||
pidfilepath = "/var/run/hmacfileserver.pid"
|
||||
cleanuponexit = true
|
||||
precaching = true
|
||||
filettlenabled = true
|
||||
globalextensions = ["*"] # Allows all file types globally
|
||||
bind_ip = "0.0.0.0" # New option: Specify the IP address to bind to (IPv4 or IPv6)
|
||||
|
||||
[logging]
|
||||
level = "info"
|
||||
file = "/var/log/hmac-file-server.log"
|
||||
max_size = 100
|
||||
max_backups = 7
|
||||
max_age = 30
|
||||
compress = true
|
||||
|
||||
[deduplication]
|
||||
enabled = true
|
||||
directory = "./deduplication"
|
||||
|
||||
[iso]
|
||||
enabled = true
|
||||
size = "1GB"
|
||||
mountpoint = "/mnt/iso"
|
||||
charset = "utf-8"
|
||||
containerfile = "/path/to/iso/container.iso"
|
||||
|
||||
[timeouts]
|
||||
readtimeout = "4800s"
|
||||
writetimeout = "4800s"
|
||||
idletimeout = "4800s"
|
||||
|
||||
[security]
|
||||
secret = "changeme"
|
||||
|
||||
[versioning]
|
||||
enableversioning = true
|
||||
maxversions = 5
|
||||
|
||||
[uploads]
|
||||
resumableuploadsenabled = true
|
||||
chunkeduploadsenabled = true
|
||||
chunksize = "8192"
|
||||
allowedextensions = ["*"] # Use ["*"] to allow all or specify extensions
|
||||
|
||||
[downloads]
|
||||
resumabledownloadsenabled = true
|
||||
chunkeddownloadsenabled = true
|
||||
chunksize = "8192"
|
||||
allowedextensions = [".jpg", ".png"] # Restricts downloads to specific types
|
||||
|
||||
[clamav]
|
||||
clamavenabled = true
|
||||
clamavsocket = "/var/run/clamav/clamd.ctl"
|
||||
numscanworkers = 2
|
||||
scanfileextensions = [".exe", ".dll", ".pdf"]
|
||||
|
||||
[redis]
|
||||
redisenabled = true
|
||||
redisaddr = "localhost:6379"
|
||||
redispassword = ""
|
||||
redisdbindex = 0
|
||||
redishealthcheckinterval = "120s"
|
||||
|
||||
[workers]
|
||||
numworkers = 4
|
||||
uploadqueuesize = 50
|
||||
|
||||
[file]
|
||||
filerevision = 1
|
||||
|
||||
[build]
|
||||
version = "v2.5"
|
||||
Loading…
x
Reference in New Issue
Block a user