From 495ee31247c785cbf03413b74f850a4d8f27510b Mon Sep 17 00:00:00 2001 From: Alexander Renz Date: Wed, 7 Jan 2026 11:55:20 +0100 Subject: [PATCH] docs: add comprehensive SYSTEMD.md installation guide - Create dedicated SYSTEMD.md with full manual installation steps - Cover security hardening, multiple instances, troubleshooting - Document Prometheus exporter manual setup - Simplify README systemd section with link to detailed guide - Add SYSTEMD.md to documentation list --- README.md | 43 ++--- SYSTEMD.md | 529 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 545 insertions(+), 27 deletions(-) create mode 100644 SYSTEMD.md diff --git a/README.md b/README.md index 780b6ac..284fa17 100644 --- a/README.md +++ b/README.md @@ -684,46 +684,34 @@ dbbackup rto analyze mydb --target-rto 4h --target-rpo 1h Install dbbackup as a systemd service for automated scheduled backups: ```bash -# Install as cluster backup service (daily at midnight) -sudo dbbackup install --backup-type cluster --schedule daily - -# Install for specific instance with custom schedule -sudo dbbackup install --instance production --schedule "*-*-* 02:00:00" - # Install with Prometheus metrics exporter -sudo dbbackup install --backup-type cluster --with-metrics --metrics-port 9399 +sudo dbbackup install --backup-type cluster --with-metrics -# Preview installation (dry-run) -sudo dbbackup install --dry-run +# Preview what would be installed +dbbackup install --dry-run --backup-type cluster # Check installation status dbbackup install --status # Uninstall -sudo dbbackup uninstall +sudo dbbackup uninstall cluster --purge +``` -# Uninstall specific instance and remove config -sudo dbbackup uninstall production --purge +**Schedule options:** +```bash +--schedule daily # Every day at midnight (default) +--schedule weekly # Every Monday at midnight +--schedule "*-*-* 02:00:00" # Every day at 2am +--schedule "Mon *-*-* 03:00" # Every Monday at 3am ``` **What gets installed:** -- Systemd service unit (dbbackup@.service or dbbackup-cluster.service) -- Systemd timer unit for scheduling -- Dedicated `dbbackup` user and group +- Systemd service and timer units +- Dedicated `dbbackup` user with security hardening - Directories: `/var/lib/dbbackup/`, `/etc/dbbackup/` -- Optional: Prometheus metrics exporter service +- Optional: Prometheus HTTP exporter on port 9399 -**Security features:** -- NoNewPrivileges, ProtectSystem=strict -- ReadWritePaths limited to backup directories -- CapabilityBoundingSet restricted -- OOMScoreAdjust=-100 to protect backups - -**Schedule format (OnCalendar):** -- `daily` - Every day at midnight -- `weekly` - Every Monday at midnight -- `*-*-* 02:00:00` - Every day at 2am -- `Mon *-*-* 03:00` - Every Monday at 3am +📖 **Full documentation:** [SYSTEMD.md](SYSTEMD.md) - Manual setup, security hardening, multiple instances, troubleshooting ## Prometheus Metrics @@ -870,6 +858,7 @@ Workload types: ## Documentation +- [SYSTEMD.md](SYSTEMD.md) - Systemd installation & scheduling - [DOCKER.md](DOCKER.md) - Docker deployment - [CLOUD.md](CLOUD.md) - Cloud storage configuration - [PITR.md](PITR.md) - Point-in-Time Recovery diff --git a/SYSTEMD.md b/SYSTEMD.md new file mode 100644 index 0000000..5f42ee5 --- /dev/null +++ b/SYSTEMD.md @@ -0,0 +1,529 @@ +# Systemd Integration Guide + +This guide covers installing dbbackup as a systemd service for automated scheduled backups. + +## Quick Start (Installer) + +The easiest way to set up systemd services is using the built-in installer: + +```bash +# Install as cluster backup service (daily at midnight) +sudo dbbackup install --backup-type cluster --schedule daily + +# Check what would be installed (dry-run) +dbbackup install --dry-run --backup-type cluster + +# Check installation status +dbbackup install --status + +# Uninstall +sudo dbbackup uninstall cluster --purge +``` + +## Installer Options + +| Flag | Description | Default | +|------|-------------|---------| +| `--instance NAME` | Instance name for named backups | - | +| `--backup-type TYPE` | Backup type: `cluster`, `single`, `sample` | `cluster` | +| `--schedule SPEC` | Timer schedule (see below) | `daily` | +| `--with-metrics` | Install Prometheus metrics exporter | false | +| `--metrics-port PORT` | HTTP port for metrics exporter | 9399 | +| `--dry-run` | Preview changes without applying | false | + +### Schedule Format + +The `--schedule` option accepts systemd OnCalendar format: + +| Value | Description | +|-------|-------------| +| `daily` | Every day at midnight | +| `weekly` | Every Monday at midnight | +| `hourly` | Every hour | +| `*-*-* 02:00:00` | Every day at 2:00 AM | +| `*-*-* 00/6:00:00` | Every 6 hours | +| `Mon *-*-* 03:00` | Every Monday at 3:00 AM | +| `*-*-01 00:00:00` | First day of every month | + +Test schedule with: `systemd-analyze calendar "Mon *-*-* 03:00"` + +## What Gets Installed + +### Directory Structure + +``` +/etc/dbbackup/ +├── dbbackup.conf # Main configuration +└── env.d/ + └── cluster.conf # Instance credentials (mode 0600) + +/var/lib/dbbackup/ +├── catalog/ +│ └── backups.db # SQLite backup catalog +├── backups/ # Default backup storage +└── metrics/ # Prometheus textfile metrics + +/var/log/dbbackup/ # Log files + +/usr/local/bin/dbbackup # Binary copy +``` + +### Systemd Units + +**For cluster backups:** +- `/etc/systemd/system/dbbackup-cluster.service` - Backup service +- `/etc/systemd/system/dbbackup-cluster.timer` - Backup scheduler + +**For named instances:** +- `/etc/systemd/system/dbbackup@.service` - Template service +- `/etc/systemd/system/dbbackup@.timer` - Template timer + +**Metrics exporter (optional):** +- `/etc/systemd/system/dbbackup-exporter.service` + +### System User + +A dedicated `dbbackup` user and group are created: +- Home: `/var/lib/dbbackup` +- Shell: `/usr/sbin/nologin` +- Purpose: Run backup services with minimal privileges + +## Manual Installation + +If you prefer to set up systemd services manually without the installer: + +### Step 1: Create User and Directories + +```bash +# Create system user +sudo useradd --system --home-dir /var/lib/dbbackup --shell /usr/sbin/nologin dbbackup + +# Create directories +sudo mkdir -p /etc/dbbackup/env.d +sudo mkdir -p /var/lib/dbbackup/{catalog,backups,metrics} +sudo mkdir -p /var/log/dbbackup + +# Set ownership +sudo chown -R dbbackup:dbbackup /var/lib/dbbackup /var/log/dbbackup +sudo chown root:dbbackup /etc/dbbackup +sudo chmod 750 /etc/dbbackup + +# Copy binary +sudo cp dbbackup /usr/local/bin/ +sudo chmod 755 /usr/local/bin/dbbackup +``` + +### Step 2: Create Configuration + +```bash +# Main configuration +sudo tee /etc/dbbackup/dbbackup.conf << 'EOF' +# DBBackup Configuration +db-type=postgres +host=localhost +port=5432 +user=postgres +backup-dir=/var/lib/dbbackup/backups +compression=6 +retention-days=30 +min-backups=7 +EOF + +# Instance credentials (secure permissions) +sudo tee /etc/dbbackup/env.d/cluster.conf << 'EOF' +PGPASSWORD=your_secure_password +# Or for MySQL: +# MYSQL_PWD=your_secure_password +EOF +sudo chmod 600 /etc/dbbackup/env.d/cluster.conf +sudo chown dbbackup:dbbackup /etc/dbbackup/env.d/cluster.conf +``` + +### Step 3: Create Service Unit + +```bash +sudo tee /etc/systemd/system/dbbackup-cluster.service << 'EOF' +[Unit] +Description=DBBackup Cluster Backup +Documentation=https://github.com/PlusOne/dbbackup +After=network.target postgresql.service mysql.service +Wants=network.target + +[Service] +Type=oneshot +User=dbbackup +Group=dbbackup + +# Load configuration +EnvironmentFile=-/etc/dbbackup/env.d/cluster.conf + +# Working directory +WorkingDirectory=/var/lib/dbbackup + +# Execute backup +ExecStart=/usr/local/bin/dbbackup backup cluster \ + --config /etc/dbbackup/dbbackup.conf \ + --backup-dir /var/lib/dbbackup/backups \ + --allow-root + +# Security hardening +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +PrivateDevices=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectControlGroups=yes +RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +MemoryDenyWriteExecute=yes +LockPersonality=yes + +# Allow write to specific paths +ReadWritePaths=/var/lib/dbbackup /var/log/dbbackup + +# Capability restrictions +CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_CONNECT +AmbientCapabilities= + +# Resource limits +MemoryMax=4G +CPUQuota=80% + +# Prevent OOM killer from terminating backups +OOMScoreAdjust=-100 + +# Logging +StandardOutput=journal +StandardError=journal +SyslogIdentifier=dbbackup + +[Install] +WantedBy=multi-user.target +EOF +``` + +### Step 4: Create Timer Unit + +```bash +sudo tee /etc/systemd/system/dbbackup-cluster.timer << 'EOF' +[Unit] +Description=DBBackup Cluster Backup Timer +Documentation=https://github.com/PlusOne/dbbackup + +[Timer] +# Run daily at midnight +OnCalendar=daily + +# Randomize start time within 15 minutes to avoid thundering herd +RandomizedDelaySec=900 + +# Run immediately if we missed the last scheduled time +Persistent=true + +# Run even if system was sleeping +WakeSystem=false + +[Install] +WantedBy=timers.target +EOF +``` + +### Step 5: Enable and Start + +```bash +# Reload systemd +sudo systemctl daemon-reload + +# Enable timer (auto-start on boot) +sudo systemctl enable dbbackup-cluster.timer + +# Start timer +sudo systemctl start dbbackup-cluster.timer + +# Verify timer is active +sudo systemctl status dbbackup-cluster.timer + +# View next scheduled run +sudo systemctl list-timers dbbackup-cluster.timer +``` + +### Step 6: Test Backup + +```bash +# Run backup manually +sudo systemctl start dbbackup-cluster.service + +# Check status +sudo systemctl status dbbackup-cluster.service + +# View logs +sudo journalctl -u dbbackup-cluster.service -f +``` + +## Prometheus Metrics Exporter (Manual) + +### Service Unit + +```bash +sudo tee /etc/systemd/system/dbbackup-exporter.service << 'EOF' +[Unit] +Description=DBBackup Prometheus Metrics Exporter +Documentation=https://github.com/PlusOne/dbbackup +After=network.target + +[Service] +Type=simple +User=dbbackup +Group=dbbackup + +# Working directory +WorkingDirectory=/var/lib/dbbackup + +# Start HTTP metrics server +ExecStart=/usr/local/bin/dbbackup metrics serve --port 9399 + +# Restart on failure +Restart=on-failure +RestartSec=10 + +# Security hardening +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +PrivateDevices=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectControlGroups=yes +RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +LockPersonality=yes + +# Catalog access +ReadWritePaths=/var/lib/dbbackup + +# Capability restrictions +CapabilityBoundingSet=CAP_NET_BIND_SERVICE +AmbientCapabilities= + +# Logging +StandardOutput=journal +StandardError=journal +SyslogIdentifier=dbbackup-exporter + +[Install] +WantedBy=multi-user.target +EOF +``` + +### Enable Exporter + +```bash +sudo systemctl daemon-reload +sudo systemctl enable dbbackup-exporter +sudo systemctl start dbbackup-exporter + +# Test +curl http://localhost:9399/health +curl http://localhost:9399/metrics +``` + +### Prometheus Configuration + +Add to `prometheus.yml`: + +```yaml +scrape_configs: + - job_name: 'dbbackup' + static_configs: + - targets: ['localhost:9399'] + scrape_interval: 60s +``` + +## Security Hardening + +The systemd units include comprehensive security hardening: + +| Setting | Purpose | +|---------|---------| +| `NoNewPrivileges=yes` | Prevent privilege escalation | +| `ProtectSystem=strict` | Read-only filesystem except allowed paths | +| `ProtectHome=yes` | Block access to /home, /root, /run/user | +| `PrivateTmp=yes` | Isolated /tmp namespace | +| `PrivateDevices=yes` | No access to physical devices | +| `RestrictAddressFamilies` | Only Unix and IP sockets | +| `MemoryDenyWriteExecute=yes` | Prevent code injection | +| `CapabilityBoundingSet` | Minimal Linux capabilities | +| `OOMScoreAdjust=-100` | Protect backup from OOM killer | + +### Database Access + +For PostgreSQL with peer authentication: +```bash +# Add dbbackup user to postgres group +sudo usermod -aG postgres dbbackup + +# Or create a .pgpass file +sudo -u dbbackup tee /var/lib/dbbackup/.pgpass << EOF +localhost:5432:*:postgres:password +EOF +sudo chmod 600 /var/lib/dbbackup/.pgpass +``` + +For PostgreSQL with password authentication: +```bash +# Store password in environment file +echo "PGPASSWORD=your_password" | sudo tee /etc/dbbackup/env.d/cluster.conf +sudo chmod 600 /etc/dbbackup/env.d/cluster.conf +``` + +## Multiple Instances + +Run different backup configurations as separate instances: + +```bash +# Install multiple instances +sudo dbbackup install --instance production --schedule "*-*-* 02:00:00" +sudo dbbackup install --instance staging --schedule "*-*-* 04:00:00" +sudo dbbackup install --instance analytics --schedule "weekly" + +# Manage individually +sudo systemctl status dbbackup@production.timer +sudo systemctl start dbbackup@staging.service +``` + +Each instance has its own: +- Configuration: `/etc/dbbackup/env.d/.conf` +- Timer schedule +- Journal logs: `journalctl -u dbbackup@.service` + +## Troubleshooting + +### View Logs + +```bash +# Real-time logs +sudo journalctl -u dbbackup-cluster.service -f + +# Last backup run +sudo journalctl -u dbbackup-cluster.service -n 100 + +# All dbbackup logs +sudo journalctl -t dbbackup + +# Exporter logs +sudo journalctl -u dbbackup-exporter -f +``` + +### Timer Not Running + +```bash +# Check timer status +sudo systemctl status dbbackup-cluster.timer + +# List all timers +sudo systemctl list-timers --all | grep dbbackup + +# Check if timer is enabled +sudo systemctl is-enabled dbbackup-cluster.timer +``` + +### Service Fails to Start + +```bash +# Check service status +sudo systemctl status dbbackup-cluster.service + +# View detailed error +sudo journalctl -u dbbackup-cluster.service -n 50 --no-pager + +# Test manually as dbbackup user +sudo -u dbbackup /usr/local/bin/dbbackup backup cluster --config /etc/dbbackup/dbbackup.conf + +# Check permissions +ls -la /var/lib/dbbackup/ +ls -la /etc/dbbackup/ +``` + +### Permission Denied + +```bash +# Fix ownership +sudo chown -R dbbackup:dbbackup /var/lib/dbbackup + +# Check SELinux (if enabled) +sudo ausearch -m avc -ts recent + +# Check AppArmor (if enabled) +sudo aa-status +``` + +### Exporter Not Accessible + +```bash +# Check if running +sudo systemctl status dbbackup-exporter + +# Check port binding +sudo ss -tlnp | grep 9399 + +# Test locally +curl -v http://localhost:9399/health + +# Check firewall +sudo ufw status +sudo iptables -L -n | grep 9399 +``` + +## Uninstallation + +### Using Installer + +```bash +# Remove cluster backup (keeps config) +sudo dbbackup uninstall cluster + +# Remove and purge configuration +sudo dbbackup uninstall cluster --purge + +# Remove named instance +sudo dbbackup uninstall production --purge +``` + +### Manual Removal + +```bash +# Stop and disable services +sudo systemctl stop dbbackup-cluster.timer dbbackup-cluster.service dbbackup-exporter +sudo systemctl disable dbbackup-cluster.timer dbbackup-exporter + +# Remove unit files +sudo rm /etc/systemd/system/dbbackup-cluster.service +sudo rm /etc/systemd/system/dbbackup-cluster.timer +sudo rm /etc/systemd/system/dbbackup-exporter.service +sudo rm /etc/systemd/system/dbbackup@.service +sudo rm /etc/systemd/system/dbbackup@.timer + +# Reload systemd +sudo systemctl daemon-reload + +# Optional: Remove user and directories +sudo userdel dbbackup +sudo rm -rf /var/lib/dbbackup +sudo rm -rf /etc/dbbackup +sudo rm -rf /var/log/dbbackup +sudo rm /usr/local/bin/dbbackup +``` + +## See Also + +- [README.md](README.md) - Main documentation +- [DOCKER.md](DOCKER.md) - Docker deployment +- [CLOUD.md](CLOUD.md) - Cloud storage configuration +- [PITR.md](PITR.md) - Point-in-Time Recovery