From 190d8ea39f3342920247e70459a6f10347842ece Mon Sep 17 00:00:00 2001 From: Renz Date: Wed, 12 Nov 2025 07:08:28 +0000 Subject: [PATCH] Fix corrupted README.md - clean professional version - Removed duplicate merged content - Clean, properly formatted markdown - Conservative professional style - All sections properly structured - 22KB clean documentation --- README.md | 1132 ++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 809 insertions(+), 323 deletions(-) diff --git a/README.md b/README.md index 3ab2141..5183d12 100644 --- a/README.md +++ b/README.md @@ -1,489 +1,975 @@ -# dbbackup# dbbackup +# dbbackup# dbbackup# dbbackup + +![dbbackup](dbbackup.png) + Professional database backup and restore utility for PostgreSQL, MySQL, and MariaDB with support for large databases and cluster operations.Database backup utility for PostgreSQL and MySQL with support for large databases. +Professional database backup and restore utility for PostgreSQL, MySQL, and MariaDB with support for large databases and cluster operations. + +## Overview + ## Overview## Recent Changes (November 2025) +`dbbackup` is a production-ready database backup tool designed for reliability, performance, and ease of use. It provides both interactive and command-line interfaces for single database backups, cluster-wide operations, and disaster recovery scenarios. + +### Key Features + `dbbackup` is a production-ready database backup tool designed for reliability, performance, and ease of use. It provides both interactive and command-line interfaces for single database backups, cluster-wide operations, and disaster recovery scenarios.### 🎯 ETA Estimation for Long Operations -- Real-time progress tracking with time estimates +- **Multi-Database Support**: PostgreSQL, MySQL, and MariaDB -### Key Features- Shows elapsed time and estimated time remaining - -- Format: "X/Y (Z%) | Elapsed: 25m | ETA: ~40m remaining" - -- **Multi-Database Support**: PostgreSQL, MySQL, and MariaDB- Particularly useful for 2+ hour cluster backups - -- **Backup Modes**: Single database, sample data, and full cluster- Works with both CLI and TUI modes +- **Backup Modes**: Single database, sample data, and full cluster- Real-time progress tracking with time estimates - **Restore Operations**: Single database and full cluster restore with safety checks -- **Performance**: Automatic CPU detection, parallel processing, and streaming compression### 🔐 Authentication Detection & Smart Guidance +- **Performance**: Automatic CPU detection, parallel processing, and streaming compression### Key Features- Shows elapsed time and estimated time remaining -- **Large Database Handling**: Optimized for databases from gigabytes to terabytes- Detects OS user vs DB user mismatches +- **Large Database Handling**: Optimized for databases from gigabytes to terabytes -- **Interactive Interface**: Full-featured terminal UI with real-time progress tracking- Identifies PostgreSQL authentication methods (peer/ident/md5) +- **Interactive Interface**: Full-featured terminal UI with real-time progress tracking- Format: "X/Y (Z%) | Elapsed: 25m | ETA: ~40m remaining" + +- **Cross-Platform**: Pre-compiled binaries for Linux, macOS, FreeBSD, OpenBSD, NetBSD + +- **Production Ready**: Comprehensive error handling, logging, and safety checks- **Multi-Database Support**: PostgreSQL, MySQL, and MariaDB- Particularly useful for 2+ hour cluster backups + + + +## Installation- **Backup Modes**: Single database, sample data, and full cluster- Works with both CLI and TUI modes + + + +### Pre-compiled Binaries- **Restore Operations**: Single database and full cluster restore with safety checks + + + +Download the appropriate binary for your platform:- **Performance**: Automatic CPU detection, parallel processing, and streaming compression### 🔐 Authentication Detection & Smart Guidance + + + +```bash- **Large Database Handling**: Optimized for databases from gigabytes to terabytes- Detects OS user vs DB user mismatches + +# Linux (x86_64) + +curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_linux_amd64 -o dbbackup- **Interactive Interface**: Full-featured terminal UI with real-time progress tracking- Identifies PostgreSQL authentication methods (peer/ident/md5) + +chmod +x dbbackup - **Cross-Platform**: Pre-compiled binaries for Linux, macOS, FreeBSD, OpenBSD, NetBSD- Shows helpful error messages with 4 solutions before connection attempt -- **Production Ready**: Comprehensive error handling, logging, and safety checks- Auto-loads passwords from `~/.pgpass` file +# Linux (ARM64) + +curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_linux_arm64 -o dbbackup- **Production Ready**: Comprehensive error handling, logging, and safety checks- Auto-loads passwords from `~/.pgpass` file + +chmod +x dbbackup - Prevents confusing TLS/authentication errors in TUI mode -## Installation- Works across all Linux distributions +# macOS (Intel) + +curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_amd64 -o dbbackup## Installation- Works across all Linux distributions + +chmod +x dbbackup -### Pre-compiled Binaries### 🗄️ MariaDB Support +# macOS (Apple Silicon) + +curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_arm64 -o dbbackup### Pre-compiled Binaries### 🗄️ MariaDB Support + +chmod +x dbbackup - MariaDB now selectable as separate database type in interactive mode -Download the appropriate binary for your platform:- Press Enter to cycle: PostgreSQL → MySQL → MariaDB +# FreeBSD, OpenBSD, NetBSD - see bin/ directory for other platforms -- Stored as distinct type in configuration +```Download the appropriate binary for your platform:- Press Enter to cycle: PostgreSQL → MySQL → MariaDB -```bash -# Linux (x86_64)### 🎨 UI Improvements -curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_linux_amd64 -o dbbackup- Conservative terminal colors for better compatibility +### Build from Source- Stored as distinct type in configuration -chmod +x dbbackup- Fixed operation history navigation (arrow keys, viewport scrolling) -- Clean plain text display without styling artifacts -# Linux (ARM64)- 15-item viewport with scroll indicators +Requirements: Go 1.19 or later```bash -curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_linux_arm64 -o dbbackup -chmod +x dbbackup### Large Database Handling -- Streaming compression reduces memory usage by ~90% - -# macOS (Intel)- Native pgx v5 driver reduces memory by ~48% compared to lib/pq - -curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_amd64 -o dbbackup- Automatic format selection based on database size - -chmod +x dbbackup- Per-database timeout configuration (default: 240 minutes) - -- Parallel compression support via pigz when available - -# macOS (Apple Silicon) - -curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_arm64 -o dbbackup### Memory Usage - -chmod +x dbbackup - -| Database Size | Memory Usage | - -# FreeBSD, OpenBSD, NetBSD - see bin/ directory for other platforms|---------------|--------------| - -```| 10GB | ~850MB | - -| 25GB | ~920MB | - -### Build from Source| 50GB | ~940MB | - -| 100GB+ | <1GB | - -Requirements: Go 1.19 or later - -### Progress Tracking - -```bash - -git clone https://git.uuxo.net/uuxo/dbbackup.git- Real-time progress indicators - -cd dbbackup- Step-by-step operation tracking - -go build -o dbbackup- Structured logging with timestamps - -```- Operation history - - - -## Quick Start## Features - - - -### Interactive Mode (Recommended)- PostgreSQL and MySQL support - -- Single database, sample, and cluster backup modes - -The interactive terminal interface provides guided backup and restore operations:- CPU detection and parallel job optimization - -- Interactive terminal interface - -```bash- Cross-platform binaries (Linux, macOS, Windows, BSD) - -# PostgreSQL (requires peer authentication)- SSL/TLS support - -sudo -u postgres ./dbbackup interactive- Configurable compression levels - - - -# MySQL/MariaDB## Installation - -./dbbackup interactive --db-type mysql --user root --password - -```### Pre-compiled Binaries - - - -### Command Line InterfaceDownload the binary for your platform: - - - -```bash```bash - -# Single database backup# Linux (Intel/AMD) - -./dbbackup backup single myapp_productioncurl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_linux_amd64 -o dbbackup - -chmod +x dbbackup - -# Full cluster backup (PostgreSQL only) - -./dbbackup backup cluster# macOS (Intel) - -curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_amd64 -o dbbackup - -# Restore from backupchmod +x dbbackup - -./dbbackup restore single /path/to/backup.dump --target myapp_production - -# macOS (Apple Silicon) - -# Cluster restore with safety checkscurl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_arm64 -o dbbackup - -./dbbackup restore cluster /path/to/cluster_backup.tar.gz --confirmchmod +x dbbackup - -`````` - - - -## Usage### Build from Source - - - -### Backup Operations```bash +```bash# Linux (x86_64)### 🎨 UI Improvements git clone https://git.uuxo.net/uuxo/dbbackup.git -#### Single Database Backupcd dbbackup +cd dbbackupcurl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_linux_amd64 -o dbbackup- Conservative terminal colors for better compatibility -go build -o dbbackup main.go +go build -```bash``` +```chmod +x dbbackup- Fixed operation history navigation (arrow keys, viewport scrolling) + + + +## Quick Start- Clean plain text display without styling artifacts + + + +### Interactive Mode (Recommended)# Linux (ARM64)- 15-item viewport with scroll indicators + + + +The interactive terminal interface provides guided backup and restore operations:curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_linux_arm64 -o dbbackup + + + +```bashchmod +x dbbackup### Large Database Handling + +# PostgreSQL (requires peer authentication) + +sudo -u postgres ./dbbackup interactive- Streaming compression reduces memory usage by ~90% + + + +# MySQL/MariaDB# macOS (Intel)- Native pgx v5 driver reduces memory by ~48% compared to lib/pq + +./dbbackup interactive --db-type mysql --user root --password + +```curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_amd64 -o dbbackup- Automatic format selection based on database size + + + +### Command Line Interfacechmod +x dbbackup- Per-database timeout configuration (default: 240 minutes) + + + +```bash- Parallel compression support via pigz when available + +# Single database backup + +./dbbackup backup single myapp_production# macOS (Apple Silicon) + + + +# Full cluster backup (PostgreSQL only)curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_arm64 -o dbbackup### Memory Usage + +./dbbackup backup cluster + +chmod +x dbbackup + +# Restore from backup + +./dbbackup restore single /path/to/backup.dump --target myapp_production| Database Size | Memory Usage | + + + +# Cluster restore with safety checks# FreeBSD, OpenBSD, NetBSD - see bin/ directory for other platforms|---------------|--------------| + +./dbbackup restore cluster /path/to/cluster_backup.tar.gz --confirm + +``````| 10GB | ~850MB | + + + +## Usage| 25GB | ~920MB | + + + +### Backup Operations### Build from Source| 50GB | ~940MB | + + + +#### Single Database Backup| 100GB+ | <1GB | + + + +```bashRequirements: Go 1.19 or later ./dbbackup backup single [options] -## Usage +### Progress Tracking Options: - --host string Database host (default "localhost")### Interactive Mode + --host string Database host (default "localhost")```bash --port int Database port (default 5432 for PostgreSQL, 3306 for MySQL) - --user string Database user (default "postgres")```bash + --user string Database user (default "postgres")git clone https://git.uuxo.net/uuxo/dbbackup.git- Real-time progress indicators - --password string Database password# PostgreSQL - must match OS user for peer authentication + --password string Database password - --backup-dir string Backup directory (default "/var/lib/pgsql/db_backups")sudo -u postgres dbbackup interactive + --backup-dir string Backup directory (default "/var/lib/pgsql/db_backups")cd dbbackup- Step-by-step operation tracking --compression int Compression level 0-9 (default 6) - --db-type string Database type: postgres, mysql, mariadb (default "postgres")# Or specify user explicitly + --db-type string Database type: postgres, mysql, mariadb (default "postgres")go build -o dbbackup- Structured logging with timestamps - --insecure Disable SSL/TLSsudo -u postgres dbbackup interactive --user postgres + --insecure Disable SSL/TLS -``` +``````- Operation history -# MySQL/MariaDB -Example:dbbackup interactive --db-type mysql --user root -```bash``` +Example: -./dbbackup backup single production_db \ +```bash - --host db.example.com \Interactive mode provides menu navigation with arrow keys and automatic status updates. +./dbbackup backup single production_db \## Quick Start## Features + + --host db.example.com \ --user backup_user \ - --password \**Authentication Note:** For PostgreSQL with peer authentication, run as the postgres user to avoid connection errors. + --password \ - --compression 9 \ + --compression 9 \### Interactive Mode (Recommended)- PostgreSQL and MySQL support - --backup-dir /mnt/backups### Command Line + --backup-dir /mnt/backups -``` +```- Single database, sample, and cluster backup modes -```bash -#### Cluster Backup (PostgreSQL)# Single database backup -dbbackup backup single myapp_db +#### Cluster Backup (PostgreSQL)The interactive terminal interface provides guided backup and restore operations:- CPU detection and parallel job optimization -Backs up all databases in a PostgreSQL cluster including roles and tablespaces: -# Sample backup (10% of data) -```bashdbbackup backup sample myapp_db --sample-ratio 10 +Backs up all databases in a PostgreSQL cluster including roles and tablespaces:- Interactive terminal interface + + + +```bash```bash- Cross-platform binaries (Linux, macOS, Windows, BSD) ./dbbackup backup cluster [options] -# Full cluster backup (PostgreSQL) - -Options:dbbackup backup cluster - - --max-cores int Maximum CPU cores to use (default: auto-detect) - - --cpu-workload string Workload type: cpu-intensive, io-intensive, balanced (default "balanced")# With custom settings - - --jobs int Number of parallel jobs (default: auto-detect)dbbackup backup single myapp_db \ - -``` --host db.example.com \ - - --port 5432 \ - -Example: --user backup_user \ - -```bash --ssl-mode require - -sudo -u postgres ./dbbackup backup cluster \``` - - --compression 3 \ - - --max-cores 16 \### System Commands - - --cpu-workload cpu-intensive - -``````bash - -# Check connection status - -#### Sample Backupdbbackup status - - - -Create backups with reduced data for testing/development:# Run preflight checks - -dbbackup preflight - -```bash - -./dbbackup backup sample [options]# List databases and backups - -dbbackup list +# PostgreSQL (requires peer authentication)- SSL/TLS support Options: - --sample-strategy string Strategy: ratio, percent, count (default "ratio")# Show CPU information + --max-cores int Maximum CPU cores to use (default: auto-detect)sudo -u postgres ./dbbackup interactive- Configurable compression levels - --sample-value float Sample value based on strategy (default 10)dbbackup cpu + --cpu-workload string Workload type: cpu-intensive, io-intensive, balanced (default "balanced") + + --jobs int Number of parallel jobs (default: auto-detect) + +``` + +# MySQL/MariaDB## Installation + +Example: + +```bash./dbbackup interactive --db-type mysql --user root --password + +sudo -u postgres ./dbbackup backup cluster \ + + --compression 3 \```### Pre-compiled Binaries + + --max-cores 16 \ + + --cpu-workload cpu-intensive + +``` + +### Command Line InterfaceDownload the binary for your platform: + +#### Sample Backup + + + +Create backups with reduced data for testing/development: + +```bash```bash + +```bash + +./dbbackup backup sample [options]# Single database backup# Linux (Intel/AMD) + + + +Options:./dbbackup backup single myapp_productioncurl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_linux_amd64 -o dbbackup + + --sample-strategy string Strategy: ratio, percent, count (default "ratio") + + --sample-value float Sample value based on strategy (default 10)chmod +x dbbackup + +``` + +# Full cluster backup (PostgreSQL only) + +Examples: + +```bash./dbbackup backup cluster# macOS (Intel) + +# Keep 10% of rows + +./dbbackup backup sample myapp_db --sample-strategy percent --sample-value 10curl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_amd64 -o dbbackup + + + +# Keep 1 in 100 rows# Restore from backupchmod +x dbbackup + +./dbbackup backup sample myapp_db --sample-strategy ratio --sample-value 100 + +./dbbackup restore single /path/to/backup.dump --target myapp_production + +# Keep 10,000 rows per table + +./dbbackup backup sample myapp_db --sample-strategy count --sample-value 10000# macOS (Apple Silicon) + +``` + +# Cluster restore with safety checkscurl -L https://git.uuxo.net/uuxo/dbbackup/raw/branch/main/bin/dbbackup_darwin_arm64 -o dbbackup + +### Restore Operations + +./dbbackup restore cluster /path/to/cluster_backup.tar.gz --confirmchmod +x dbbackup + +#### Single Database Restore `````` - - -Examples:## Configuration - ```bash -# Keep 10% of rows### Command Line Flags - -./dbbackup backup sample myapp_db --sample-strategy percent --sample-value 10 - -| Flag | Description | Default | - -# Keep 1 in 100 rows|------|-------------|---------| - -./dbbackup backup sample myapp_db --sample-strategy ratio --sample-value 100| `--host` | Database host | `localhost` | - -| `--port` | Database port | `5432` (PostgreSQL), `3306` (MySQL) | - -# Keep 10,000 rows per table| `--user` | Database user | `postgres` | - -./dbbackup backup sample myapp_db --sample-strategy count --sample-value 10000| `--database` | Database name | `postgres` | - -```| `-d`, `--db-type` | Database type | `postgres` | - -| `--ssl-mode` | SSL mode | `prefer` | - -### Restore Operations| `--jobs` | Parallel jobs | Auto-detected | - -| `--dump-jobs` | Parallel dump jobs | Auto-detected | - -#### Single Database Restore| `--compression` | Compression level (0-9) | `6` | - -| `--backup-dir` | Backup directory | `/var/lib/pgsql/db_backups` | - -```bash - -./dbbackup restore single [options]### PostgreSQL +./dbbackup restore single [options] -Options:#### Authentication Methods +Options:## Usage### Build from Source --target string Target database name (required) - --create Create database if it doesn't existPostgreSQL uses different authentication methods depending on your system configuration: + --create Create database if it doesn't exist --clean Drop and recreate database before restore - --jobs int Number of parallel restore jobs (default 4)**Peer Authentication (most common on Linux):** + --jobs int Number of parallel restore jobs (default 4)### Backup Operations```bash + +``` + +git clone https://git.uuxo.net/uuxo/dbbackup.git + +Example: + +```bash#### Single Database Backupcd dbbackup + +./dbbackup restore single /backups/myapp_20250111.dump \ + + --target myapp_restored \go build -o dbbackup main.go + + --create \ + + --jobs 8```bash``` + +``` + +./dbbackup backup single [options] + +#### Cluster Restore (PostgreSQL) + +## Usage + +Restore an entire PostgreSQL cluster from backup: + +Options: + +```bash + +./dbbackup restore cluster [options] --host string Database host (default "localhost")### Interactive Mode + + + +Options: --port int Database port (default 5432 for PostgreSQL, 3306 for MySQL) + + --confirm Confirm and execute restore (required for safety) + + --dry-run Show what would be done without executing --user string Database user (default "postgres")```bash + + --force Skip safety checks + + --jobs int Number of parallel decompression jobs (default: auto) --password string Database password# PostgreSQL - must match OS user for peer authentication + +``` + + --backup-dir string Backup directory (default "/var/lib/pgsql/db_backups")sudo -u postgres dbbackup interactive + +Example: + +```bash --compression int Compression level 0-9 (default 6) + +sudo -u postgres ./dbbackup restore cluster /backups/cluster_20250111.tar.gz --confirm + +``` --db-type string Database type: postgres, mysql, mariadb (default "postgres")# Or specify user explicitly + + + +**Safety Features:** --insecure Disable SSL/TLSsudo -u postgres dbbackup interactive --user postgres + +- Pre-restore validation of archive integrity + +- Disk space checks``` + +- Verification of required tools (psql, pg_restore, tar, gzip) + +- Automatic detection and cleanup of existing databases (interactive mode)# MySQL/MariaDB + +- Progress tracking with ETA estimation + +Example:dbbackup interactive --db-type mysql --user root + +### Disaster Recovery + +```bash``` + +For complete disaster recovery scenarios, use the included script: + +./dbbackup backup single production_db \ + +```bash + +sudo ./disaster_recovery_test.sh --host db.example.com \Interactive mode provides menu navigation with arrow keys and automatic status updates. + +``` + + --user backup_user \ + +This script performs: + +1. Full cluster backup with maximum performance --password \**Authentication Note:** For PostgreSQL with peer authentication, run as the postgres user to avoid connection errors. + +2. Documentation of current state + +3. Controlled destruction of all user databases (with confirmation) --compression 9 \ + +4. Complete cluster restoration + +5. Verification of database integrity --backup-dir /mnt/backups### Command Line + + + +**Warning:** This is a destructive operation. Only use in test environments or genuine disaster recovery scenarios.``` + + + +## Configuration```bash + + + +### PostgreSQL Authentication#### Cluster Backup (PostgreSQL)# Single database backup + + + +PostgreSQL authentication varies by system configuration. The tool automatically detects issues and provides solutions.dbbackup backup single myapp_db + + + +#### Peer/Ident Authentication (Default on Linux)Backs up all databases in a PostgreSQL cluster including roles and tablespaces: + + + +Run as the PostgreSQL system user:# Sample backup (10% of data) + + + +```bash```bashdbbackup backup sample myapp_db --sample-ratio 10 + +sudo -u postgres ./dbbackup backup cluster + +```./dbbackup backup cluster [options] + + + +#### Password Authentication# Full cluster backup (PostgreSQL) + + + +Option 1 - Using .pgpass file (recommended for automation):Options:dbbackup backup cluster + +```bash + +echo "localhost:5432:*:postgres:password" > ~/.pgpass --max-cores int Maximum CPU cores to use (default: auto-detect) + +chmod 0600 ~/.pgpass + +./dbbackup backup single mydb --user postgres --cpu-workload string Workload type: cpu-intensive, io-intensive, balanced (default "balanced")# With custom settings + +``` + + --jobs int Number of parallel jobs (default: auto-detect)dbbackup backup single myapp_db \ + +Option 2 - Environment variable: + +```bash``` --host db.example.com \ + +export PGPASSWORD=your_password + +./dbbackup backup single mydb --user postgres --port 5432 \ + +``` + +Example: --user backup_user \ + +Option 3 - Command line flag: + +```bash```bash --ssl-mode require + +./dbbackup backup single mydb --user postgres --password your_password + +```sudo -u postgres ./dbbackup backup cluster \``` + + + +### MySQL/MariaDB Authentication --compression 3 \ + + + +```bash --max-cores 16 \### System Commands + +# Command line + +./dbbackup backup single mydb --db-type mysql --user root --password your_password --cpu-workload cpu-intensive + + + +# Environment variable``````bash + +export MYSQL_PWD=your_password + +./dbbackup backup single mydb --db-type mysql --user root# Check connection status + + + +# Configuration file#### Sample Backupdbbackup status + +cat > ~/.my.cnf << EOF + +[client] + +user=backup_user + +password=your_passwordCreate backups with reduced data for testing/development:# Run preflight checks + +host=localhost + +EOFdbbackup preflight + +chmod 0600 ~/.my.cnf ``````bash -# Must run as postgres user -Example:sudo -u postgres dbbackup backup cluster + +### Environment Variables./dbbackup backup sample [options]# List databases and backups + + + +```bashdbbackup list + +# PostgreSQL + +export PG_HOST=localhostOptions: + +export PG_PORT=5432 + +export PG_USER=postgres --sample-strategy string Strategy: ratio, percent, count (default "ratio")# Show CPU information + +export PGPASSWORD=password + + --sample-value float Sample value based on strategy (default 10)dbbackup cpu + +# MySQL/MariaDB + +export MYSQL_HOST=localhost`````` + +export MYSQL_PORT=3306 + +export MYSQL_USER=root + +export MYSQL_PWD=password + +Examples:## Configuration + +# General + +export BACKUP_DIR=/var/backups/databases```bash + +export COMPRESS_LEVEL=6 + +export CLUSTER_TIMEOUT_MIN=240# Keep 10% of rows### Command Line Flags + +``` + +./dbbackup backup sample myapp_db --sample-strategy percent --sample-value 10 + +## Performance + +| Flag | Description | Default | + +### CPU Optimization + +# Keep 1 in 100 rows|------|-------------|---------| + +The tool automatically detects CPU configuration and optimizes parallel operations: + +./dbbackup backup sample myapp_db --sample-strategy ratio --sample-value 100| `--host` | Database host | `localhost` | ```bash -./dbbackup restore single /backups/myapp_20250111.dump \# If you see this error: "Ident authentication failed for user postgres" +./dbbackup cpu| `--port` | Database port | `5432` (PostgreSQL), `3306` (MySQL) | + +``` + +# Keep 10,000 rows per table| `--user` | Database user | `postgres` | + +Manual override: + +```bash./dbbackup backup sample myapp_db --sample-strategy count --sample-value 10000| `--database` | Database name | `postgres` | + +./dbbackup backup cluster --max-cores 32 --jobs 32 --cpu-workload cpu-intensive + +``````| `-d`, `--db-type` | Database type | `postgres` | + + + +### Memory Usage| `--ssl-mode` | SSL mode | `prefer` | + + + +Streaming architecture maintains constant memory usage:### Restore Operations| `--jobs` | Parallel jobs | Auto-detected | + + + +| Database Size | Memory Usage || `--dump-jobs` | Parallel dump jobs | Auto-detected | + +|---------------|--------------| + +| 1-10 GB | ~800 MB |#### Single Database Restore| `--compression` | Compression level (0-9) | `6` | + +| 10-50 GB | ~900 MB | + +| 50-100 GB | ~950 MB || `--backup-dir` | Backup directory | `/var/lib/pgsql/db_backups` | + +| 100+ GB | <1 GB | + +```bash + +### Large Database Support + +./dbbackup restore single [options]### PostgreSQL + +- Databases >5GB automatically use optimized plain format with streaming compression + +- Parallel compression via pigz (if available) for maximum throughput + +- Per-database timeout configuration (default: 4 hours) + +- Automatic format selection based on sizeOptions:#### Authentication Methods + + + +## System Commands --target string Target database name (required) + + + +```bash --create Create database if it doesn't existPostgreSQL uses different authentication methods depending on your system configuration: + +# Check database connection and configuration + +./dbbackup status --clean Drop and recreate database before restore + + + +# Run pre-flight checks before backup --jobs int Number of parallel restore jobs (default 4)**Peer Authentication (most common on Linux):** + +./dbbackup preflight + +``````bash + +# List available databases + +./dbbackup list# Must run as postgres user + + + +# Display CPU informationExample:sudo -u postgres dbbackup backup cluster + +./dbbackup cpu + +```bash + +# Show version information + +./dbbackup version./dbbackup restore single /backups/myapp_20250111.dump \# If you see this error: "Ident authentication failed for user postgres" + +``` --target myapp_restored \# Use one of these solutions: +## Troubleshooting + --create \``` +### Connection Issues + --jobs 8 -```**Solution 1: Use matching OS user (recommended)** +Test connectivity: + +```bash```**Solution 1: Use matching OS user (recommended)** + +./dbbackup status + +``````bash + + + +For PostgreSQL peer authentication errors:#### Cluster Restore (PostgreSQL)sudo -u postgres dbbackup status --user postgres ```bash -#### Cluster Restore (PostgreSQL)sudo -u postgres dbbackup status --user postgres +sudo -u postgres ./dbbackup status``` ``` Restore an entire PostgreSQL cluster from backup: -**Solution 2: Configure ~/.pgpass file** +For SSL/TLS issues: -```bash```bash +```bash**Solution 2: Configure ~/.pgpass file** -./dbbackup restore cluster [options]echo "localhost:5432:*:postgres:your_password" > ~/.pgpass +./dbbackup status --insecure -chmod 0600 ~/.pgpass +``````bash```bash -Options:dbbackup status --user postgres - --confirm Confirm and execute restore (required for safety)``` - --dry-run Show what would be done without executing +### Out of Memory./dbbackup restore cluster [options]echo "localhost:5432:*:postgres:your_password" > ~/.pgpass - --force Skip safety checks**Solution 3: Set PGPASSWORD environment variable** - --jobs int Number of parallel decompression jobs (default: auto)```bash -```export PGPASSWORD=your_password +If experiencing memory issues with very large databases:chmod 0600 ~/.pgpass -dbbackup status --user postgres -Example:``` + +1. Check available memory:Options:dbbackup status --user postgres ```bash -sudo -u postgres ./dbbackup restore cluster /backups/cluster_20250111.tar.gz --confirm**Solution 4: Use --password flag** +free -h --confirm Confirm and execute restore (required for safety)``` -``````bash +dmesg | grep -i oom -dbbackup status --user postgres --password your_password +``` --dry-run Show what would be done without executing -**Safety Features:**``` -- Pre-restore validation of archive integrity -- Disk space checks#### SSL Configuration - -- Verification of required tools (psql, pg_restore, tar, gzip) - -- Automatic detection and cleanup of existing databases (interactive mode)SSL modes: `disable`, `prefer`, `require`, `verify-ca`, `verify-full` - -- Progress tracking with ETA estimation - -Cluster operations (backup/restore/verify) are PostgreSQL-only. - -### Disaster Recovery - -### MySQL / MariaDB - -For complete disaster recovery scenarios, use the included script: - -Set `--db-type mysql` or `--db-type mariadb`: - -```bash```bash - -sudo ./disaster_recovery_test.shdbbackup backup single mydb \ - -``` --db-type mysql \ - - --host 127.0.0.1 \ - -This script performs: --user backup_user \ - -1. Full cluster backup with maximum performance --password **** - -2. Documentation of current state``` - -3. Controlled destruction of all user databases (with confirmation) - -4. Complete cluster restorationMySQL backups are created as `.sql.gz` files. - -5. Verification of database integrity - -### Environment Variables - -**Warning:** This is a destructive operation. Only use in test environments or genuine disaster recovery scenarios. +2. Add swap space: --force Skip safety checks**Solution 3: Set PGPASSWORD environment variable** ```bash -## Configuration# Database +sudo fallocate -l 16G /swapfile --jobs int Number of parallel decompression jobs (default: auto)```bash -export PG_HOST=localhost +sudo chmod 600 /swapfile -### PostgreSQL Authenticationexport PG_PORT=5432 +sudo mkswap /swapfile```export PGPASSWORD=your_password -export PG_USER=postgres +sudo swapon /swapfile -PostgreSQL authentication varies by system configuration. The tool automatically detects issues and provides solutions.export PGPASSWORD=secret - -export MYSQL_HOST=localhost - -#### Peer/Ident Authentication (Default on Linux)export MYSQL_PWD=secret +```dbbackup status --user postgres -Run as the PostgreSQL system user:# Backup +3. Reduce parallelism:Example:``` -export BACKUP_DIR=/var/backups +```bash -```bashexport COMPRESS_LEVEL=6 - -sudo -u postgres ./dbbackup backup clusterexport CLUSTER_TIMEOUT_MIN=240 # Cluster timeout in minutes +./dbbackup backup cluster --jobs 4 --dump-jobs 4```bash ``` +sudo -u postgres ./dbbackup restore cluster /backups/cluster_20250111.tar.gz --confirm**Solution 4: Use --password flag** + +### Debug Mode + +``````bash + +Enable detailed logging: + +```bashdbbackup status --user postgres --password your_password + +./dbbackup backup single mydb --debug + +```**Safety Features:**``` + + + +### Common Error Messages- Pre-restore validation of archive integrity + + + +**"Ident authentication failed"** - Run as matching OS user or configure password authentication- Disk space checks#### SSL Configuration + + + +**"Permission denied"** - Check database user privileges or run with appropriate system user- Verification of required tools (psql, pg_restore, tar, gzip) + + + +**"Disk space check failed"** - Ensure sufficient space in backup directory (4x archive size recommended)- Automatic detection and cleanup of existing databases (interactive mode)SSL modes: `disable`, `prefer`, `require`, `verify-ca`, `verify-full` + + + +**"Archive validation failed"** - Backup file may be corrupted or incomplete- Progress tracking with ETA estimation + + + +## Building All Platform BinariesCluster operations (backup/restore/verify) are PostgreSQL-only. + + + +To build binaries for all supported platforms:### Disaster Recovery + + + +```bash### MySQL / MariaDB + +./build_all.sh + +```For complete disaster recovery scenarios, use the included script: + + + +Binaries will be created in the `bin/` directory.Set `--db-type mysql` or `--db-type mariadb`: + + + +## Project Structure```bash```bash + + + +```sudo ./disaster_recovery_test.shdbbackup backup single mydb \ + +dbbackup/ + +├── main.go # Application entry point``` --db-type mysql \ + +├── cmd/ # CLI command implementations + +├── internal/ --host 127.0.0.1 \ + +│ ├── backup/ # Backup engine + +│ ├── restore/ # Restore engine with safety checksThis script performs: --user backup_user \ + +│ ├── config/ # Configuration management + +│ ├── database/ # Database drivers (PostgreSQL, MySQL)1. Full cluster backup with maximum performance --password **** + +│ ├── cpu/ # CPU detection and optimization + +│ ├── logger/ # Structured logging2. Documentation of current state``` + +│ ├── progress/ # Progress tracking and ETA estimation + +│ └── tui/ # Interactive terminal interface3. Controlled destruction of all user databases (with confirmation) + +├── bin/ # Pre-compiled binaries + +├── disaster_recovery_test.sh # Disaster recovery testing script4. Complete cluster restorationMySQL backups are created as `.sql.gz` files. + +└── build_all.sh # Multi-platform build script + +```5. Verification of database integrity + + + +## Requirements### Environment Variables + + + +### System Requirements**Warning:** This is a destructive operation. Only use in test environments or genuine disaster recovery scenarios. + + + +- Linux, macOS, FreeBSD, OpenBSD, or NetBSD```bash + +- 1 GB RAM minimum (2 GB recommended for large databases) + +- Sufficient disk space for backups (typically 30-50% of database size)## Configuration# Database + + + +### Software Requirementsexport PG_HOST=localhost + + + +#### PostgreSQL### PostgreSQL Authenticationexport PG_PORT=5432 + +- PostgreSQL client tools (psql, pg_dump, pg_dumpall, pg_restore) + +- PostgreSQL 10 or later recommendedexport PG_USER=postgres + + + +#### MySQL/MariaDBPostgreSQL authentication varies by system configuration. The tool automatically detects issues and provides solutions.export PGPASSWORD=secret + +- MySQL/MariaDB client tools (mysql, mysqldump) + +- MySQL 5.7+ or MariaDB 10.3+ recommendedexport MYSQL_HOST=localhost + + + +#### Optional#### Peer/Ident Authentication (Default on Linux)export MYSQL_PWD=secret + +- pigz (for parallel compression) + +- pv (for progress monitoring) + + + +## Best PracticesRun as the PostgreSQL system user:# Backup + + + +1. **Test Restores Regularly** - Verify backups can be restored successfullyexport BACKUP_DIR=/var/backups + +2. **Monitor Disk Space** - Ensure adequate space for backup operations + +3. **Use Compression** - Balance between speed and space (level 3-6 recommended)```bashexport COMPRESS_LEVEL=6 + +4. **Automate Backups** - Schedule regular backups via cron or systemd timers + +5. **Secure Credentials** - Use .pgpass or .my.cnf files with proper permissions (0600)sudo -u postgres ./dbbackup backup clusterexport CLUSTER_TIMEOUT_MIN=240 # Cluster timeout in minutes + +6. **Version Control** - Keep multiple backup versions for point-in-time recovery + +7. **Off-Site Storage** - Copy backups to remote storage for disaster recovery``` + +8. **Document Procedures** - Maintain runbooks for restore operations + # Swap file management (Linux + root only) +## Support + #### Password Authenticationexport AUTO_SWAP=false +For issues, questions, or contributions: + export SWAP_FILE_SIZE_GB=8 -Option 1 - Using .pgpass file (recommended for automation):export SWAP_FILE_PATH=/tmp/dbbackup_swap +- Repository: https://git.uuxo.net/uuxo/dbbackup -```bash``` +- Report issues via the repository issue trackerOption 1 - Using .pgpass file (recommended for automation):export SWAP_FILE_PATH=/tmp/dbbackup_swap + + + +## License```bash``` + + + +MIT License - see repository for detailsecho "localhost:5432:*:postgres:password" > ~/.pgpass -echo "localhost:5432:*:postgres:password" > ~/.pgpass chmod 0600 ~/.pgpass## Architecture