uuxo/hmac-file-server
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Update configuration and README files

Browse Source
This commit is contained in:
Alexander Renz 2024-12-23 11:23:26 +01:00
parent 1152fa28cc
commit 0933482f64
9 changed files with 608 additions and 277 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
README.MD
View File

@ -65,6 +65,10 @@ If `AutoAdjustWorkers = true`, the values for `NumWorkers` and `NumScanWorkers`
Setting `NetworkEvents = false` in the server configuration disables the logging and tracking of network-related events within the application. This means that functionalities such as monitoring IP changes or recording network activity will be turned off.
### Precaching
The `precaching` feature allows the server to pre-cache storage paths for faster access. This can improve performance by reducing the time needed to access frequently used storage paths.
---
## Example `config.toml`
@ -81,13 +85,15 @@ MetricsPort = "9090"
FileTTL = "1y"
DeduplicationEnabled = true
MinFreeBytes = "100MB"
AutoAdjustWorkers = true # Enable auto-adjustment for worker scaling
NetworkEvents = false # IP changes or recording network activity will be turned off.
AutoAdjustWorkers = true # Enable auto-adjustment for worker scaling
NetworkEvents = false # Disable logging and tracking of network-related events
PIDFilePath = "./hmac_file_server.pid" # Path to PID file
Precaching = true # Enable pre-caching of storage paths
[timeouts]
ReadTimeout = "480s"
WriteTimeout = "480s"
IdleTimeout = "65s" # nginx/apache2 keep-a-live 60s
IdleTimeout = "65s" # nginx/apache2 keep-alive 60s
[security]
Secret = "changeme"
@ -99,9 +105,14 @@ MaxVersions = 1
[uploads]
ResumableUploadsEnabled = true
ChunkedUploadsEnabled = true
ChunkSize = "32MB"
ChunkSize = "64MB"
AllowedExtensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm", ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"]
[downloads]
ResumableDownloadsEnabled = true
ChunkedDownloadsEnabled = true
ChunkSize = "64MB"
[clamav]
ClamAVEnabled = false
ClamAVSocket = "/var/run/clamav/clamd.ctl"
@ -119,11 +130,8 @@ RedisHealthCheckInterval = "120s"
NumWorkers = 4
UploadQueueSize = 5000
[iso]
Enabled = false
Size = "2TB"
MountPoint = "/mnt/iso"
Charset = "utf-8"
[file]
FileRevision = 1 # Revision number for file handling
```
---
@ -183,4 +191,13 @@ Prometheus metrics include:
- **Versioning**: Store multiple versions of files and keep a maximum of `MaxVersions` versions.
- **ClamAV Integration**: Scan uploaded files for viruses using ClamAV.
- **Redis Caching**: Utilize Redis for caching file metadata for faster access.
- **Auto-Adjust Worker Scaling**: Optimize the number of workers dynamically based on system resources.
- **Auto-Adjust Worker Scaling**: Optimize the number of workers dynamically based on system resources.
---
## Build & Run
1. Clone the repository.
2. Build the server:
go build -o hmac-file-server cmd/server/main.go
3. Run the server:
./hmac-file-server --config ./cmd/server/config.toml

156
RELEASE-NOTES.md Normal file
View File

@ -0,0 +1,156 @@
# Release Notes - hmac-file-server v2.1-stable
**Release Date:** April 27, 2024
## Overview
We are excited to announce the release of **hmac-file-server v2.1-stable**. This version brings significant enhancements, new features, and important bug fixes to improve the performance, security, and usability of the HMAC File Server. Below are the detailed changes and updates included in this release.
## New Features
### 1. **ClamAV Integration**
- **Description:** Integrated ClamAV for enhanced malware scanning capabilities.
- **Benefits:**
- Improved security by scanning uploaded files for viruses and malware.
- Configurable number of scan workers to optimize performance.
- **Configuration:**
```toml
[clamav]
clamavenabled = true
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 4
scanfileextensions = [".exe", ".dll", ".bin", ".com", ".bat", ".sh", ".php", ".js"]
```
### 2. **Redis Support**
- **Description:** Added support for Redis to enhance caching and data management.
- **Benefits:**
- Improved performance with Redis as an external cache.
- Enhanced scalability for handling high loads.
- **Configuration:**
```toml
[redis]
redisenabled = true
redisdbindex = 0
redisaddr = "localhost:6379"
redispassword = ""
redishealthcheckinterval = "120s"
```
### 3. **Enhanced Configuration Management**
- **Description:** Expanded configuration options for greater flexibility.
- **New Configuration Options:**
- `precaching`: Enables pre-caching of frequently accessed files.
- `networkevents`: Toggles the logging of network events for better monitoring.
- **Updated `config.toml`:**
```toml
[server]
precaching = true
networkevents = false
```
### 4. **Improved Logging**
- **Description:** Enhanced logging capabilities with configurable log levels and formats.
- **Benefits:**
- Better insights into server operations and issues.
- Support for JSON-formatted logs for easier integration with log management systems.
- **Configuration:**
```toml
[server]
loglevel = "debug"
logfile = "/var/log/hmac-file-server.log"
loggingjson = false
```
## Enhancements
### 1. **Graceful Shutdown**
- **Description:** Implemented graceful shutdown procedures to ensure all ongoing processes complete before the server stops.
- **Benefits:**
- Prevents data corruption and ensures consistency.
- Enhances reliability during server restarts and shutdowns.
### 2. **Auto-Adjusting Worker Pools**
- **Description:** Introduced auto-adjustment for worker pools based on current load and resource availability.
- **Benefits:**
- Optimizes resource usage.
- Maintains optimal performance under varying loads.
### 3. **Extended Timeout Configurations**
- **Description:** Added configurable timeouts for read, write, and idle connections.
- **Configuration:**
```toml
[timeouts]
readtimeout = "3600s"
writetimeout = "3600s"
idletimeout = "3600s"
```
## Bug Fixes
- **Fixed:** Resolved issues with unused parameters and function calls in the handler modules.
- **Fixed:** Addressed syntax errors related to constant declarations and import paths in `main.go`.
- **Fixed:** Corrected configuration parsing to handle new and updated configuration fields effectively.
- **Fixed:** Improved error handling during Redis and ClamAV client initialization to prevent server crashes.
## Performance Improvements
- **Optimized:** Enhanced the upload and download handlers for faster file processing and reduced latency.
- **Optimized:** Improved caching mechanisms to decrease load times and increase throughput.
## Security Enhancements
- **Enhanced:** Strengthened security configurations by integrating ClamAV and enabling secure Redis connections.
- **Improved:** Secured sensitive information handling, ensuring secrets are managed appropriately.
## Configuration Changes
### Removed Deprecated Options
- **Deprecated:** Removed outdated configuration options that are no longer supported to streamline the configuration process.
### Updated Configuration Structure
- **Change:** Updated the configuration structure to align with the new features and enhancements, ensuring better clarity and maintainability.
## Known Issues
- **Issue:** Some users may experience delays in file processing when auto-adjusting worker pools under extreme loads. We are actively working on optimizing this feature.
- **Issue:** JSON-formatted logs may require additional parsing tools for integration with certain logging systems.
## Upgrade Instructions
1. **Backup Configuration:**
- Ensure you have a backup of your current `config.toml` before proceeding with the upgrade.
2. **Update Application:**
- Pull the latest version from the repository:
```sh
git pull origin v2.1-stable
```
- Alternatively, download the latest release from the [releases page](https://github.com/PlusOne/hmac-file-server/releases).
3. **Update Dependencies:**
- Navigate to the project directory and run:
```sh
go mod tidy
```
4. **Review Configuration:**
- Compare your existing `config.toml` with the updated configuration file to incorporate new settings.
5. **Restart the Server:**
- Restart the HMAC File Server to apply the updates:
```sh
systemctl restart hmac-file-server
```
## Acknowledgments
We would like to thank our contributors and the community for their continuous support and valuable feedback, which have been instrumental in shaping this release.
## Support
For any issues or questions regarding this release, please open an issue on our [GitHub repository](https://github.com/PlusOne/hmac-file-server/issues) or contact our support team.
---
*Thank you for using hmac-file-server! We hope this release enhances your experience and meets your needs effectively.*

206
cmd/monitor/monitor.go
View File

@ -1,10 +1,13 @@
package main
import (
"bufio"
"fmt"
"log"
"net/http"
"os"
"sort"
"strconv"
"strings"
"time"
@ -18,19 +21,25 @@ import (
)
var prometheusURL string
var configFilePath string // Pfad der gefundenen Konfiguration
var logFilePath string // Pfad der Logdatei aus der Konfiguration
func init() {
configPaths := []string{
"/etc/hmac-file-server/config.toml",
"../config.toml",
"./config.toml",
}
var config *toml.Tree
var err error
// Lade die config.toml aus den definierten Pfaden
for _, path := range configPaths {
config, err = toml.LoadFile(path)
if err == nil {
configFilePath = path
log.Printf("Using config file: %s", configFilePath)
break
}
}
@ -39,8 +48,41 @@ func init() {
log.Fatalf("Error loading config file: %v", err)
}
port := config.Get("server.metrics_port").(int64)
// Metricsport auslesen
portValue := config.Get("server.metricsport")
if portValue == nil {
log.Println("Warning: 'server.metricsport' is missing in the configuration, using default port 9090")
portValue = int64(9090)
}
var port int64
switch v := portValue.(type) {
case int64:
port = v
case string:
parsedPort, err := strconv.ParseInt(v, 10, 64)
if err != nil {
log.Fatalf("Error parsing 'server.metricsport' as int64: %v", err)
}
port = parsedPort
default:
log.Fatalf("Error: 'server.metricsport' is not of type int64 or string, got %T", v)
}
prometheusURL = fmt.Sprintf("http://localhost:%d/metrics", port)
// Log-Datei auslesen über server.logfile
logFileValue := config.Get("server.logfile")
if logFileValue == nil {
log.Println("Warning: 'server.logfile' is missing, using default '/var/log/hmac-file-server.log'")
logFilePath = "/var/log/hmac-file-server.log"
} else {
lf, ok := logFileValue.(string)
if !ok {
log.Fatalf("Error: 'server.logfile' is not of type string, got %T", logFileValue)
}
logFilePath = lf
}
}
// Thresholds for color coding
@ -393,50 +435,6 @@ func updateHmacTable(hmacTable *tview.Table, hmacInfo *ProcessInfo, metrics map[
}
}
func main() {
app := tview.NewApplication()
// Create pages
pages := tview.NewPages()
// System page
sysPage := createSystemPage()
pages.AddPage("system", sysPage, true, true)
// hmac-file-server page
hmacPage := createHmacPage()
pages.AddPage("hmac", hmacPage, true, false)
// Add key binding to switch views
app.SetInputCapture(func(event *tcell.EventKey) *tcell.EventKey {
if event.Key() == tcell.KeyRune {
switch event.Rune() {
case 'q', 'Q':
app.Stop()
return nil
case 's', 'S':
// Switch to system page
pages.SwitchToPage("system")
return nil
case 'h', 'H':
// Switch to hmac-file-server page
pages.SwitchToPage("hmac")
return nil
}
}
return event
})
// Start the UI update loop in a separate goroutine
go updateUI(app, pages, sysPage, hmacPage)
// Set the root and run the application
if err := app.SetRoot(pages, true).EnableMouse(true).Run(); err != nil {
log.Fatalf("Error running application: %v", err)
}
}
// Function to create the system page
func createSystemPage() tview.Primitive {
// Create system data table
sysTable := tview.NewTable().SetBorders(false)
@ -453,14 +451,13 @@ func createSystemPage() tview.Primitive {
// Create a flex layout to hold the tables
sysFlex := tview.NewFlex().
SetDirection(tview.FlexRow).
AddItem(sysTable, 7, 0, false). // Fixed height for system data
AddItem(metricsTable, 0, 1, false). // Proportional height for metrics
AddItem(processTable, 0, 2, false) // Proportional height for process list
AddItem(sysTable, 7, 0, false).
AddItem(metricsTable, 0, 1, false).
AddItem(processTable, 0, 2, false)
return sysFlex
}
// Function to create the hmac-file-server page
func createHmacPage() tview.Primitive {
hmacTable := tview.NewTable().SetBorders(false)
hmacTable.SetTitle(" [::b]hmac-file-server Details ").SetBorder(true)
@ -471,3 +468,114 @@ func createHmacPage() tview.Primitive {
return hmacFlex
}
func createLogsPage(logFilePath string) tview.Primitive {
logsTextView := tview.NewTextView().
SetDynamicColors(true).
SetRegions(true).
SetWordWrap(true)
logsTextView.SetTitle(" [::b]Logs ").SetBorder(true)
const numLines = 100 // Number of lines to read from the end of the log file
// Read logs periodically
go func() {
for {
content, err := readLastNLines(logFilePath, numLines)
if err != nil {
logsTextView.SetText(fmt.Sprintf("[red]Error reading log file: %v[white]", err))
} else {
// Process the log content to add colors
lines := strings.Split(content, "\n")
var coloredLines []string
for _, line := range lines {
if strings.Contains(line, "level=info") {
coloredLines = append(coloredLines, "[green]"+line+"[white]")
} else if strings.Contains(line, "level=warn") {
coloredLines = append(coloredLines, "[yellow]"+line+"[white]")
} else if strings.Contains(line, "level=error") {
coloredLines = append(coloredLines, "[red]"+line+"[white]")
} else {
// Default color
coloredLines = append(coloredLines, line)
}
}
logsTextView.SetText(strings.Join(coloredLines, "\n"))
}
time.Sleep(2 * time.Second) // Refresh interval for logs
}
}()
return logsTextView
}
func readLastNLines(filePath string, n int) (string, error) {
file, err := os.Open(filePath)
if err != nil {
return "", err
}
defer file.Close()
var lines []string
scanner := bufio.NewScanner(file)
for scanner.Scan() {
lines = append(lines, scanner.Text())
if len(lines) > n {
lines = lines[1:]
}
}
if err := scanner.Err(); err != nil {
return "", err
}
return strings.Join(lines, "\n"), nil
}
func main() {
app := tview.NewApplication()
// Create pages
pages := tview.NewPages()
// System page
sysPage := createSystemPage()
pages.AddPage("system", sysPage, true, true)
// hmac-file-server page
hmacPage := createHmacPage()
pages.AddPage("hmac", hmacPage, true, false)
// Logs page mit dem gelesenen logFilePath
logsPage := createLogsPage(logFilePath)
pages.AddPage("logs", logsPage, true, false)
// Add key binding to switch views
app.SetInputCapture(func(event *tcell.EventKey) *tcell.EventKey {
if event.Key() == tcell.KeyRune {
switch event.Rune() {
case 'q', 'Q':
app.Stop()
return nil
case 's', 'S':
// Switch to system page
pages.SwitchToPage("system")
case 'h', 'H':
// Switch to hmac-file-server page
pages.SwitchToPage("hmac")
case 'l', 'L':
// Switch to logs page
pages.SwitchToPage("logs")
}
}
return event
})
// Start the UI update loop in a separate goroutine
go updateUI(app, pages, sysPage, hmacPage)
// Set the root and run the application
if err := app.SetRoot(pages, true).EnableMouse(true).Run(); err != nil {
log.Fatalf("Error running application: %v", err)
}
}

43
cmd/server/config.toml
View File

@ -1,14 +1,20 @@
[server]
listenport = "8080"
unixsocket = false
storagepath = "./upload"
loglevel = "info"
storagepath = "./uploads"
loglevel = "debug"
logfile = "./hmac-file-server.log"
metricsenabled = true
metricsport = "9090"
DeduplicationEnabled = true
filettl = "1y"
minfreebytes = "100GB"
metricsport = "8081"
filettl = "180d"
minfreebytes = "2GB"
deduplicationenabled = true
autoadjustworkers = true
networkevents = false
temppath = "/tmp/hmac"
loggingjson = false
pidfilepath = "./hmac_file_server.pid"
cleanuponexit = true
[iso]
enabled = false
@ -17,9 +23,9 @@ mountpoint = "/mnt/nfs_vol01/hmac-file-server/iso/"
charset = "utf-8"
[timeouts]
readtimeout = "3600s"
writetimeout = "3600s"
idletimeout = "3600s"
readtimeout = "1800s"
writetimeout = "1800s"
idletimeout = "1800s"
[security]
secret = "a-orc-and-a-humans-is-drinking-ale"
@ -29,23 +35,28 @@ enableversioning = false
maxversions = 1
[uploads]
resumableuploadsenabled = true
chunkeduploadsenabled = true
chunksize = "32MB"
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm", ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg"]
resumableuploadsenabled = false
chunkeduploadsenabled = false
chunksize = "64MB"
allowedextensions = [".txt", ".pdf", ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".svg", ".webp", ".wav", ".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".webm", ".mpeg", ".mpg", ".m4v", ".3gp", ".3g2", ".mp3", ".ogg", ".zip", ".rar"]
[downloads]
resumabledownloadsenabled = true
chunkeddownloadsenabled = true
chunksize = "64MB"
[clamav]
clamavenabled = false
clamavsocket = "/var/run/clamav/clamd.ctl"
numscanworkers = 4
ScanFileExtensions = [".exe", ".dll", ".bin", ".com", ".bat", ".sh", ".php", ".js"]
scanfileextensions = [".exe", ".dll", ".bin", ".com", ".bat", ".sh", ".php", ".js"]
[redis]
redisenabled = false
redisdbindex = 0
redisdbindex = 1
redisaddr = "localhost:6379"
redispassword = ""
redishealthcheckinterval = "120s"
redishealthcheckinterval = "60s"
[workers]
numworkers = 4

BIN
cmd/server/hmac-file-server Executable file
View File

Binary file not shown.

119
cmd/server/hmac-file-server.log
View File

@ -1,119 +0,0 @@
time="2024-12-04T17:22:02+01:00" level=info msg="========================================"
time="2024-12-04T17:22:02+01:00" level=info msg=" HMAC File Server - v2.0-dev "
time="2024-12-04T17:22:02+01:00" level=info msg=" Secure File Handling with HMAC Auth "
time="2024-12-04T17:22:02+01:00" level=info msg="========================================"
time="2024-12-04T17:22:02+01:00" level=info msg="Features: Prometheus Metrics, Chunked Uploads, ClamAV Scanning"
time="2024-12-04T17:22:02+01:00" level=info msg="Build Date: 2024-10-28"
time="2024-12-04T17:22:02+01:00" level=info msg="Operating System: linux"
time="2024-12-04T17:22:02+01:00" level=info msg="Architecture: amd64"
time="2024-12-04T17:22:02+01:00" level=info msg="Number of CPUs: 8"
time="2024-12-04T17:22:02+01:00" level=info msg="Go Version: go1.22.0"
time="2024-12-04T17:22:02+01:00" level=info msg="Total Memory: 15684 MB"
time="2024-12-04T17:22:02+01:00" level=info msg="Free Memory: 2378 MB"
time="2024-12-04T17:22:02+01:00" level=info msg="Used Memory: 5770 MB"
time="2024-12-04T17:22:02+01:00" level=info msg="CPU Model: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz, Cores: 1, Mhz: 4900.000000"
time="2024-12-04T17:22:02+01:00" level=info msg="CPU Model: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz, Cores: 1, Mhz: 4900.000000"
time="2024-12-04T17:22:02+01:00" level=info msg="CPU Model: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz, Cores: 1, Mhz: 4900.000000"
time="2024-12-04T17:22:02+01:00" level=info msg="CPU Model: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz, Cores: 1, Mhz: 4900.000000"
time="2024-12-04T17:22:02+01:00" level=info msg="CPU Model: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz, Cores: 1, Mhz: 4900.000000"
time="2024-12-04T17:22:02+01:00" level=info msg="CPU Model: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz, Cores: 1, Mhz: 4900.000000"
time="2024-12-04T17:22:02+01:00" level=info msg="CPU Model: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz, Cores: 1, Mhz: 4900.000000"
time="2024-12-04T17:22:02+01:00" level=info msg="CPU Model: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz, Cores: 1, Mhz: 4900.000000"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /, Total: 465 GB, Free: 58 GB, Used: 383 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/backupz/6, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/brave/458, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/brave/456, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/bare/5, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/canonical-livepatch/282, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/canonical-livepatch/286, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/cameractrls/35, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/cameractrls/34, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/cmake/1425, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/cmake/1429, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/core/17200, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/core18/2846, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/core18/2829, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/core20/2379, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/core20/2434, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/core24/490, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/core22/1663, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/core24/609, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/drawio/228, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/duplicity/513, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/drawio/230, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/duplicity/517, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/dynahack/32, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/firefox/5273, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/firefox/5361, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/firmware-updater/127, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/firmware-updater/147, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/gaming-graphics-core22/166, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/gaming-graphics-core22/184, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/gnome-3-28-1804/198, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/gnome-3-34-1804/93, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/gnome-3-38-2004/143, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/gnome-42-2204/176, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/godot/45, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/gnome-46-2404/48, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/hello-world/29, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/gtk-common-themes/1535, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/kf5-5-110-qt-5-15-11-core22/3, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/kf5-5-113-qt-5-15-11-core22/1, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/marble/32, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/master-pdf-editor-5/20, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/master-pdf-editor-5/21, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/mesa-2404/143, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/mesa-core22/311, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/micropolis/168, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/mkcron/2, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/obs-studio/1302, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/qownnotes/11552, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/qownnotes/11560, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/qt513/24, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/remmina/6419, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/retroarch/2879, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/snap-store/1244, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/snapd/21759, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/snapd/23258, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/snapd-desktop-integration/247, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/steam/200, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/snapd-desktop-integration/253, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/sublime-text/156, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/steam/206, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/sublime-text/177, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/suckit/49, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/teleguard-desktop/91, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/thunderbird/571, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/thunderbird/581, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/wine-platform-7-stable-core20/6, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /var/snap/firefox/common/host-hunspell, Total: 465 GB, Free: 58 GB, Used: 383 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/winbox/171, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/video-downloader/1212, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/winbox/170, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/wine-platform-runtime-core20/146, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/wine-platform-runtime-core20/147, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /boot, Total: 1 GB, Free: 1 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /boot/efi, Total: 1 GB, Free: 1 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /media/renz/external, Total: 116 GB, Free: 80 GB, Used: 29 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/video-downloader/1221, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/core22/1722, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/snap-store/1247, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Disk Mountpoint: /snap/teleguard-desktop/94, Total: 0 GB, Free: 0 GB, Used: 0 GB"
time="2024-12-04T17:22:02+01:00" level=info msg="Hostname: xps7390"
time="2024-12-04T17:22:02+01:00" level=info msg="Uptime: 35037 seconds"
time="2024-12-04T17:22:02+01:00" level=info msg="Boot Time: 2024-12-04 07:38:06 +0100 CET"
time="2024-12-04T17:22:02+01:00" level=info msg="Platform: ubuntu"
time="2024-12-04T17:22:02+01:00" level=info msg="Platform Family: debian"
time="2024-12-04T17:22:02+01:00" level=info msg="Platform Version: 24.04"
time="2024-12-04T17:22:02+01:00" level=info msg="Kernel Version: 6.8.0-49-generic"
time="2024-12-04T17:22:02+01:00" level=info msg="Prometheus metrics initialized."
time="2024-12-04T17:22:02+01:00" level=info msg="Upload queue initialized with size: 1000"
time="2024-12-04T17:22:02+01:00" level=info msg="Upload, scan, and network event channels initialized."
time="2024-12-04T17:22:02+01:00" level=info msg="Initialized 4 upload workers"
time="2024-12-04T17:22:02+01:00" level=info msg="Upload worker 2 started."
time="2024-12-04T17:22:02+01:00" level=info msg="Upload worker 1 started."
time="2024-12-04T17:22:02+01:00" level=info msg="Upload worker 3 started."
time="2024-12-04T17:22:02+01:00" level=info msg="Upload worker 0 started."
time="2024-12-04T17:22:02+01:00" level=info msg="Starting HMAC file server v2.0-dev..."
time="2024-12-04T17:22:02+01:00" level=info msg="Metrics server started on port 9090"
time="2024-12-04T17:22:03+01:00" level=info msg="Received signal interrupt. Initiating shutdown..."

272
cmd/server/main.go
View File

@ -37,6 +37,7 @@ import (
"github.com/shirou/gopsutil/mem"
"github.com/sirupsen/logrus"
"github.com/spf13/viper"
"gopkg.in/natefinch/lumberjack.v2"
)
// parseSize converts a human-readable size string to bytes
@ -108,6 +109,8 @@ type ServerConfig struct {
MinFreeByte string `mapstructure:"MinFreeByte"`
AutoAdjustWorkers bool `mapstructure:"AutoAdjustWorkers"`
NetworkEvents bool `mapstructure:"NetworkEvents"` // Added field
PrecachingEnabled bool `mapstructure:"precaching"` // Added field
PIDFilePath string `mapstructure:"pidfilepath"` // Added field
}
type TimeoutConfig struct {
@ -163,17 +166,29 @@ type ISOConfig struct {
Charset string `mapstructure:"charset"`
}
type PasteConfig struct {
Enabled bool `mapstructure:"enabled"`
StoragePath string `mapstructure:"storagePath"`
}
type DownloadsConfig struct {
ResumableDownloadEnabled bool `mapstructure:"ResumableDownloadEnabled"`
ChunkSize string `mapstructure:"ChunkSize"`
}
type Config struct {
Server ServerConfig `mapstructure:"server"`
Timeouts TimeoutConfig `mapstructure:"timeouts"`
Security SecurityConfig `mapstructure:"security"`
Versioning VersioningConfig `mapstructure:"versioning"`
Uploads UploadsConfig `mapstructure:"uploads"`
Downloads DownloadsConfig `mapstructure:"downloads"`
ClamAV ClamAVConfig `mapstructure:"clamav"`
Redis RedisConfig `mapstructure:"redis"`
Workers WorkersConfig `mapstructure:"workers"`
File FileConfig `mapstructure:"file"`
ISO ISOConfig `mapstructure:"iso"`
Paste PasteConfig `mapstructure:"paste"`
}
type UploadTask struct {
@ -194,7 +209,7 @@ type NetworkEvent struct {
var (
conf Config
versionString string = "v2.0-stable"
versionString string = "v2.1-stable"
log = logrus.New()
uploadQueue chan UploadTask
networkEvents chan NetworkEvent
@ -237,6 +252,46 @@ const maxConcurrentOperations = 10
var semaphore = make(chan struct{}, maxConcurrentOperations)
var logMessages []string
var logMu sync.Mutex
func cumulateLogMessage(level logrus.Level, msg string) {
logMu.Lock()
defer logMu.Unlock()
logMessages = append(logMessages, fmt.Sprintf("time=%q level=%s msg=%q", time.Now().Format(time.RFC3339), level, msg))
}
func flushLogMessages() {
logMu.Lock()
defer logMu.Unlock()
for _, msg := range logMessages {
log.Info(msg)
}
logMessages = []string{}
}
// writePIDFile writes the current process ID to the specified pid file
func writePIDFile(pidPath string) error {
pid := os.Getpid()
pidStr := strconv.Itoa(pid)
err := os.WriteFile(pidPath, []byte(pidStr), 0644)
if err != nil {
return fmt.Errorf("failed to write PID file: %v", err)
}
log.Infof("PID %d written to %s", pid, pidPath)
return nil
}
// removePIDFile removes the PID file
func removePIDFile(pidPath string) {
err := os.Remove(pidPath)
if err != nil {
log.Warnf("failed to remove PID file %s: %v", pidPath, err)
} else {
log.Infof("PID file %s removed", pidPath)
}
}
func main() {
setDefaults()
@ -250,6 +305,11 @@ func main() {
}
log.Info("Configuration loaded successfully.")
err = writePIDFile(conf.Server.PIDFilePath) // Write PID file after config is loaded
if err != nil {
log.Fatalf("Error writing PID file: %v", err)
}
initializeWorkerSettings(&conf.Server, &conf.Workers, &conf.ClamAV)
if conf.ISO.Enabled {
@ -260,6 +320,17 @@ func main() {
}
fileInfoCache = cache.New(5*time.Minute, 10*time.Minute)
if conf.Server.PrecachingEnabled { // Conditionally perform pre-caching
// Starting pre-caching of storage path
log.Info("Starting pre-caching of storage path...")
err = precacheStoragePath(conf.Server.StoragePath)
if err != nil {
log.Warnf("Pre-caching storage path failed: %v", err)
} else {
log.Info("Pre-cached all files in the storage path.")
}
}
err = os.MkdirAll(conf.Server.StoragePath, os.ModePerm)
if err != nil {
@ -335,11 +406,12 @@ func main() {
}
server := &http.Server{
Addr: ":" + conf.Server.ListenPort,
Handler: router,
ReadTimeout: readTimeout,
WriteTimeout: writeTimeout,
IdleTimeout: idleTimeout,
Addr: ":" + conf.Server.ListenPort,
Handler: router,
ReadTimeout: readTimeout,
WriteTimeout: writeTimeout,
IdleTimeout: idleTimeout,
MaxHeaderBytes: 1 << 20, // 1 MB
}
if conf.Server.MetricsEnabled {
@ -390,8 +462,10 @@ func autoAdjustWorkers() (int, int) {
cpuCores, _ := cpu.Counts(true)
numWorkers := cpuCores * 2
if v.Available < 2*1024*1024*1024 {
if v.Available < 4*1024*1024*1024 { // Less than 4GB available
numWorkers = max(numWorkers/2, 1)
} else if v.Available < 8*1024*1024*1024 { // Less than 8GB available
numWorkers = max(numWorkers*3/4, 1)
}
queueSize := numWorkers * 10
@ -473,6 +547,8 @@ func setDefaults() {
viper.SetDefault("server.MinFreeBytes", "100MB")
viper.SetDefault("server.AutoAdjustWorkers", true)
viper.SetDefault("server.NetworkEvents", true) // Set default
viper.SetDefault("server.precaching", true) // Set default for precaching
viper.SetDefault("server.pidfilepath", "/var/run/hmacfileserver.pid") // Set default for PID file path
_, err := parseTTL("1D")
if err != nil {
log.Warnf("Failed to parse TTL: %v", err)
@ -559,22 +635,89 @@ func validateConfig(conf *Config) error {
}
}
if conf.Paste.Enabled && conf.Paste.StoragePath == "" {
return fmt.Errorf("paste is enabled but 'storagePath' is not set in '[paste]' section")
}
// Validate Downloads Configuration
if conf.Downloads.ResumableDownloadEnabled {
if conf.Downloads.ChunkSize == "" {
return fmt.Errorf("downloads.chunkSize must be set when resumable downloads are enabled")
}
if _, err := parseSize(conf.Downloads.ChunkSize); err != nil {
return fmt.Errorf("invalid downloads.chunkSize: %v", err)
}
}
// Validate Uploads Configuration
if conf.Uploads.ResumableUploadsEnabled {
if conf.Uploads.ChunkSize == "" {
return fmt.Errorf("uploads.chunkSize must be set when resumable uploads are enabled")
}
if _, err := parseSize(conf.Uploads.ChunkSize); err != nil {
return fmt.Errorf("invalid uploads.chunkSize: %v", err)
}
}
// Validate Workers Configuration
if conf.Workers.NumWorkers <= 0 {
return fmt.Errorf("workers.numWorkers must be greater than 0")
}
if conf.Workers.UploadQueueSize <= 0 {
return fmt.Errorf("workers.uploadQueueSize must be greater than 0")
}
// Validate ClamAV Configuration
if conf.ClamAV.ClamAVEnabled {
if conf.ClamAV.ClamAVSocket == "" {
return fmt.Errorf("clamav.clamAVSocket must be set when ClamAV is enabled")
}
if conf.ClamAV.NumScanWorkers <= 0 {
return fmt.Errorf("clamav.numScanWorkers must be greater than 0")
}
}
// Validate ISO Configuration
if conf.ISO.Enabled {
if conf.ISO.Size == "" {
return fmt.Errorf("iso.size must be set when ISO is enabled")
}
if _, err := parseSize(conf.ISO.Size); err != nil {
return fmt.Errorf("invalid iso.size: %v", err)
}
if conf.ISO.MountPoint == "" {
return fmt.Errorf("iso.mountPoint must be set when ISO is enabled")
}
if conf.ISO.Charset == "" {
return fmt.Errorf("iso.charset must be set when ISO is enabled")
}
}
// Validate Paste Configuration
if conf.Paste.Enabled {
if conf.Paste.StoragePath == "" {
return fmt.Errorf("paste.storagePath must be set when paste is enabled")
}
}
return nil
}
func setupLogging() {
level, err := logrus.ParseLevel(conf.Server.LogLevel)
if err != nil {
if (err != nil) {
log.Fatalf("Invalid log level: %s", conf.Server.LogLevel)
}
log.SetLevel(level)
if conf.Server.LogFile != "" {
logFile, err := os.OpenFile(conf.Server.LogFile, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
if err != nil {
log.Fatalf("Failed to open log file: %v", err)
}
log.SetOutput(io.MultiWriter(os.Stdout, logFile))
log.SetOutput(&lumberjack.Logger{
Filename: conf.Server.LogFile,
MaxSize: 100, // megabytes
MaxBackups: 3,
MaxAge: 28, // days
Compress: true, // compress old log files
})
} else {
log.SetOutput(os.Stdout)
}
@ -604,8 +747,12 @@ func logSystemInfo() {
log.Infof("Used Memory: %v MB", v.Used/1024/1024)
cpuInfo, _ := cpu.Info()
uniqueCPUModels := make(map[string]bool)
for _, info := range cpuInfo {
log.Infof("CPU Model: %s, Cores: %d, Mhz: %f", info.ModelName, info.Cores, info.Mhz)
if !uniqueCPUModels[info.ModelName] {
log.Infof("CPU Model: %s, Cores: %d, Mhz: %f", info.ModelName, info.Cores, info.Mhz)
uniqueCPUModels[info.ModelName] = true
}
}
partitions, _ := disk.Partitions(false)
@ -864,9 +1011,9 @@ func createFile(tempFilename string, r *http.Request) error {
bufWriter := bufio.NewWriter(file)
defer bufWriter.Flush()
bufPtr := bufferPool.Get().(*[]byte) // Correct type assertion
defer bufferPool.Put(bufPtr)
defer bufferPool.Put(bufPtr)
_, err = io.CopyBuffer(bufWriter, r.Body, *bufPtr)
if err != nil {
@ -887,7 +1034,6 @@ func shouldScanFile(filename string) bool {
}
func uploadWorker(ctx context.Context, workerID int) {
log.Infof("Upload worker %d started.", workerID)
defer log.Infof("Upload worker %d stopped.", workerID)
for {
select {
@ -897,7 +1043,6 @@ func uploadWorker(ctx context.Context, workerID int) {
if !ok {
return
}
log.Infof("Worker %d processing file: %s", workerID, task.AbsFilename)
err := processUpload(task)
if err != nil {
log.Errorf("Worker %d failed to process file %s: %v", workerID, task.AbsFilename, err)
@ -910,26 +1055,24 @@ func uploadWorker(ctx context.Context, workerID int) {
}
func initializeUploadWorkerPool(ctx context.Context, w *WorkersConfig) {
var workerIDs []int
for i := 0; i < w.NumWorkers; i++ {
go uploadWorker(ctx, i)
log.Infof("Upload worker %d started.", i)
workerIDs = append(workerIDs, i)
}
log.Infof("Initialized %d upload workers", w.NumWorkers)
log.Infof("Initialized %d upload workers: %v", w.NumWorkers, workerIDs)
}
func scanWorker(ctx context.Context, workerID int) {
log.WithField("worker_id", workerID).Info("Scan worker started")
defer log.WithField("worker_id", workerID).Info("Scan worker stopping")
for {
select {
case <-ctx.Done():
log.WithField("worker_id", workerID).Info("Scan worker stopping")
return
case task, ok := <-scanQueue:
if !ok {
log.WithField("worker_id", workerID).Info("Scan queue closed")
return
}
log.WithFields(logrus.Fields{"worker_id": workerID, "file": task.AbsFilename}).Info("Processing scan task")
err := scanFileWithClamAV(task.AbsFilename)
if err != nil {
log.WithFields(logrus.Fields{"worker_id": workerID, "file": task.AbsFilename, "error": err}).Error("Failed to scan file")
@ -943,10 +1086,12 @@ func scanWorker(ctx context.Context, workerID int) {
}
func initializeScanWorkerPool(ctx context.Context) {
var workerIDs []int
for i := 0; i < conf.ClamAV.NumScanWorkers; i++ {
go scanWorker(ctx, i)
workerIDs = append(workerIDs, i)
}
log.Infof("Initialized %d scan workers", conf.ClamAV.NumScanWorkers)
log.Infof("Initialized %d scan workers: %v", conf.ClamAV.NumScanWorkers, workerIDs)
}
func setupRouter() http.Handler {
@ -1037,8 +1182,9 @@ func handleRequest(w http.ResponseWriter, r *http.Request) {
fileStorePath := strings.TrimPrefix(p, "/")
if fileStorePath == "" || fileStorePath == "/" {
log.Warn("Access to root directory is forbidden")
cumulateLogMessage(logrus.WarnLevel, "Access to root directory is forbidden")
http.Error(w, "Forbidden", http.StatusForbidden)
flushLogMessages()
return
} else if fileStorePath[0] == '/' {
fileStorePath = fileStorePath[1:]
@ -1158,14 +1304,19 @@ func handleUpload(w http.ResponseWriter, r *http.Request, absFilename, fileStore
// Asynchronous processing in the background
go func() {
var logMessages []string
// ClamAV scanning
if conf.ClamAV.ClamAVEnabled && shouldScanFile(absFilename) {
err := scanFileWithClamAV(absFilename)
if err != nil {
log.Errorf("ClamAV failed for %s: %v", absFilename, err)
os.Remove(absFilename)
uploadErrorsTotal.Inc()
logMessages = append(logMessages, fmt.Sprintf("ClamAV failed for %s: %v", absFilename, err))
for _, msg := range logMessages {
log.Info(msg)
}
return
} else {
logMessages = append(logMessages, fmt.Sprintf("ClamAV scan passed for file: %s", absFilename))
}
}
@ -1177,6 +1328,8 @@ func handleUpload(w http.ResponseWriter, r *http.Request, absFilename, fileStore
os.Remove(absFilename)
uploadErrorsTotal.Inc()
return
} else {
logMessages = append(logMessages, fmt.Sprintf("Deduplication handled successfully for file: %s", absFilename))
}
}
@ -1189,12 +1342,19 @@ func handleUpload(w http.ResponseWriter, r *http.Request, absFilename, fileStore
os.Remove(absFilename)
uploadErrorsTotal.Inc()
return
} else {
logMessages = append(logMessages, fmt.Sprintf("File versioned successfully: %s", absFilename))
}
}
}
log.Infof("Processing completed successfully for %s", absFilename)
logMessages = append(logMessages, fmt.Sprintf("Processing completed successfully for %s", absFilename))
uploadsTotal.Inc()
// Log all messages at once
for _, msg := range logMessages {
log.Info(msg)
}
}()
}
@ -1460,7 +1620,7 @@ func handleNetworkEvents(ctx context.Context) {
log.Info("Stopping network event handler.")
return
case event, ok := <-networkEvents:
if !ok {
if (!ok) {
log.Info("Network events channel closed.")
return
}
@ -1502,33 +1662,21 @@ func setupGracefulShutdown(server *http.Server, cancel context.CancelFunc) {
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
go func() {
sig := <-quit
log.Infof("Received signal %s. Initiating shutdown...", sig)
ctxShutdown, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer shutdownCancel()
if err := server.Shutdown(ctxShutdown); err != nil {
log.Errorf("Server shutdown failed: %v", err)
} else {
log.Info("Server shutdown gracefully.")
}
log.Infof("Received signal %s. Initiating graceful shutdown...", sig)
removePIDFile(conf.Server.PIDFilePath) // Ensure PID file is removed
cancel()
close(uploadQueue)
log.Info("Upload queue closed.")
close(scanQueue)
log.Info("Scan queue closed.")
close(networkEvents)
log.Info("Network events channel closed.")
log.Info("Shutdown process completed. Exiting application.")
os.Exit(0)
ctx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer shutdownCancel()
if err := server.Shutdown(ctx); err != nil {
log.Errorf("Graceful shutdown failed: %v", err)
} else {
log.Info("Server gracefully stopped")
}
}()
}
func initRedis() {
if !conf.Redis.RedisEnabled {
if (!conf.Redis.RedisEnabled) {
log.Info("Redis is disabled in configuration.")
return
}
@ -1543,7 +1691,7 @@ func initRedis() {
defer cancel()
_, err := redisClient.Ping(ctx).Result()
if err != nil {
if (err != nil) {
log.Fatalf("Failed to connect to Redis: %v", err)
}
log.Info("Connected to Redis successfully")
@ -1566,12 +1714,12 @@ func MonitorRedisHealth(ctx context.Context, client *redis.Client, checkInterval
err := client.Ping(ctx).Err()
mu.Lock()
if err != nil {
if redisConnected {
if (redisConnected) {
log.Errorf("Redis health check failed: %v", err)
}
redisConnected = false
} else {
if !redisConnected {
if (!redisConnected) {
log.Info("Redis reconnected successfully")
}
redisConnected = true
@ -2017,3 +2165,17 @@ func handleCorruptedISOFile(isoPath string, files []string, size string, charset
}
return nil
}
func precacheStoragePath(dir string) error {
return filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
log.Warnf("Error accessing path %s: %v", path, err)
return nil // Continue walking
}
if !info.IsDir() {
fileInfoCache.Set(path, info, cache.DefaultExpiration)
log.Debugf("Cached file info for %s", path)
}
return nil
})
}

13
go.mod
View File

@ -3,37 +3,36 @@ module github.com/PlusOne/hmac-file-server
go 1.21
require (
github.com/gdamore/tcell/v2 v2.7.4
github.com/go-redis/redis/v8 v8.11.5
github.com/pelletier/go-toml v1.9.5
github.com/prometheus/client_golang v1.20.5
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/shirou/gopsutil/v3 v3.24.5
github.com/sirupsen/logrus v1.9.3
github.com/spf13/viper v1.19.0
)
require (
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/gdamore/encoding v1.0.0 // indirect
github.com/gdamore/tcell/v2 v2.7.4 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646 // indirect
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sagikazarmark/locafero v0.4.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/shirou/gopsutil/v3 v3.24.5 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/cast v1.6.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.19.0 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.9.0 // indirect
@ -45,7 +44,6 @@ require (
)
require (
github.com/BurntSushi/toml v1.4.0
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/dutchcoders/go-clamd v0.0.0-20170520113014-b970184f4d9e
@ -54,7 +52,7 @@ require (
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.61.0 // indirect
github.com/prometheus/common v0.61.0
github.com/prometheus/procfs v0.15.1 // indirect
github.com/rivo/tview v0.0.0-20241103174730-c76f7879f592
github.com/tklauser/go-sysconf v0.3.14 // indirect
@ -62,4 +60,5 @@ require (
github.com/yusufpapurcu/wmi v1.2.4 // indirect
golang.org/x/sys v0.28.0 // indirect
google.golang.org/protobuf v1.35.2 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1
)

39
go.sum
View File

@ -1,25 +1,21 @@
github.com/BurntSushi/toml v1.4.0 h1:kuoIxZQy2WRRk1pttg9asf+WVv6tWQuBNVmK8+nqPr0=
github.com/BurntSushi/toml v1.4.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/dutchcoders/go-clamd v0.0.0-20170520113014-b970184f4d9e h1:rcHHSQqzCgvlwP0I/fQ8rQMn/MpHE5gWSLdtpxtP6KQ=
github.com/dutchcoders/go-clamd v0.0.0-20170520113014-b970184f4d9e/go.mod h1:Byz7q8MSzSPkouskHJhX0er2mZY/m0Vj5bMeMCkkyY4=
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
github.com/gdamore/encoding v1.0.0 h1:+7OoQ1Bc6eTm5niUzBa0Ctsh6JbMW6Ra+YNuAtDBdko=
github.com/gdamore/encoding v1.0.0/go.mod h1:alR0ol34c49FCSBLjhosxzcPHQbf2trDkoo5dl+VrEg=
github.com/gdamore/tcell/v2 v2.7.1 h1:TiCcmpWHiAU7F0rA2I3S2Y4mmLmO9KHxJ7E1QhYzQbc=
github.com/gdamore/tcell/v2 v2.7.1/go.mod h1:dSXtXTSK0VsW1biw65DZLZ2NKr7j0qP/0J7ONmsraWg=
github.com/gdamore/tcell/v2 v2.7.4 h1:sg6/UnTM9jGpZU+oFYAsDahfchWAFW8Xx2yFinNSAYU=
github.com/gdamore/tcell/v2 v2.7.4/go.mod h1:dSXtXTSK0VsW1biw65DZLZ2NKr7j0qP/0J7ONmsraWg=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
@ -34,6 +30,10 @@ github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lucasb-eyer/go-colorful v1.2.0 h1:1nnpGOrhyZZuNyfu1QjKiUICQ74+3FNCN69Aj6K7nkY=
@ -48,8 +48,6 @@ github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyua
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646 h1:zYyBkD/k9seD2A7fsi6Oo2LfFZAehjjQMERAvZLEDnQ=
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646/go.mod h1:jpp1/29i3P1S/RLdc7JQKbRpFeM1dOBd8T9ki5s+AY8=
github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
@ -62,17 +60,15 @@ github.com/pelletier/go-toml v1.9.5 h1:4yBQzkHv+7BHq2PQUZF3Mx0IYxG7LsP222s7Agd3v
github.com/pelletier/go-toml v1.9.5/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c h1:ncq/mPwQF4JjgDlrVEn3C11VoGHZN7m8qihwgMEtzYw=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/prometheus/client_golang v1.20.5 h1:cxppBPuYhUnsO6yo/aoRol4L7q7UFfdm+bR9r+8l63Y=
github.com/prometheus/client_golang v1.20.5/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE=
github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
github.com/prometheus/common v0.60.1 h1:FUas6GcOw66yB/73KC+BOZoFJmbo/1pojoILArPAaSc=
github.com/prometheus/common v0.60.1/go.mod h1:h0LYf1R1deLSKtD4Vdg8gy4RuOvENW2J/h19V5NADQw=
github.com/prometheus/common v0.61.0 h1:3gv/GThfX0cV2lpO7gkTUwZru38mxevy90Bj8YFSRQQ=
github.com/prometheus/common v0.61.0/go.mod h1:zr29OCN/2BsJRaFwG8QOBr41D6kkchKbpeNH7pAjb/s=
github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
@ -83,6 +79,8 @@ github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJ
github.com/rivo/uniseg v0.4.3/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/sagikazarmark/locafero v0.4.0 h1:HApY1R9zGo4DBgr7dqsTH/JJxLTTsOt7u6keLGt6kNQ=
github.com/sagikazarmark/locafero v0.4.0/go.mod h1:Pe1W6UlPYUk/+wc/6KFhbORCfqzgYEpgQ3O5fPuL3H4=
github.com/sagikazarmark/slog-shim v0.1.0 h1:diDBnUNK9N/354PgrxMywXnAwEr1QZcOr6gto+ugjYE=
@ -93,6 +91,8 @@ github.com/shirou/gopsutil/v3 v3.24.5 h1:i0t8kL+kQTvpAYToeuiVk3TgDeKOFioZO3Ztz/i
github.com/shirou/gopsutil/v3 v3.24.5/go.mod h1:bsoOS1aStSs9ErQ1WWfxllSeS1K5D+U30r2NfcubMVk=
github.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFtM=
github.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ=
github.com/shoenig/test v0.6.4 h1:kVTaSd7WLz5WZ2IaoM0RSzRsUD+m8wRR+5qvntpn4LU=
github.com/shoenig/test v0.6.4/go.mod h1:byHiCGXqrVaflBLAMq/srcZIHynQPQgeyvkvXnjqq0k=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo=
@ -114,9 +114,9 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/tklauser/go-sysconf v0.3.14 h1:g5vzr9iPFFz24v2KZXs/pvpvh8/V9Fw6vQK5ZZb78yU=
@ -140,9 +140,8 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.29.0 h1:5ORfpBpCs4HzDYoodCDBbwHzdR5UrLBZ3sOnUJmFoHo=
golang.org/x/net v0.29.0/go.mod h1:gLkgy8jTGERgjzMic6DS9+SP0ajcu6Xu3Orq/SpETg0=
golang.org/x/net v0.32.0 h1:ZqPmj8Kzc+Y6e0+skZsuACbx+wzMgo5MQsJh9Qd6aYI=
golang.org/x/net v0.32.0/go.mod h1:CwU0IoeOlnQQWJ6ioyFrfRuomB8GKF6KbYXZVyeXNfs=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@ -157,8 +156,6 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo=
golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0 h1:Fksou7UEQUWlKvIdsqzJmUmCX3cZuD2+P3XyyzwMhlA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@ -171,8 +168,6 @@ golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224=
golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
golang.org/x/text v0.21.0 h1:zyQAAkrwaneQ066sspRyJaG9VNi/YJ1NfzcGB3hZ/qo=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@ -181,13 +176,15 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.35.1 h1:m3LfL6/Ca+fqnjnlqQXNpFPABW1UD7mjh8KO2mKFytA=
google.golang.org/protobuf v1.35.1/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
google.golang.org/protobuf v1.35.2 h1:8Ar7bF+apOIoThw1EdZl0p1oWvMqTHmpA2fRTyZO8io=
google.golang.org/protobuf v1.35.2/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc=
gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=