Fixed:
- Native restore now connects to target database correctly (was connecting to source)
- Sequences now properly exported (fixed type mismatch in information_schema query)
- COPY FROM stdin protocol now properly handled using pgx CopyFrom
- Tool verification skipped when --native flag is used
- Fixed slice bounds panic on short SQL statements
Changes:
- internal/engine/native/manager.go: Create engine with target database for restore
- internal/engine/native/postgresql.go: COPY handling, sequence type casting
- cmd/restore.go: Skip VerifyTools in native mode
- internal/tui/restore_preview.go: Native engine mode bypass
Tested: 100k row backup/restore cycle verified working
NEW FEATURES:
- --native flag for cluster backup creates SQL format (.sql.gz) using pure Go
- --native flag for cluster restore uses pure Go engine for .sql.gz files
- Zero external tool dependencies when using native mode
- Single-binary deployment now possible without pg_dump/pg_restore
CLUSTER BACKUP (--native):
- Creates .sql.gz files instead of .dump files
- Uses pgx wire protocol for data export
- Parallel gzip compression with pgzip
- Automatic fallback with --fallback-tools
CLUSTER RESTORE (--native):
- Restores .sql.gz files using pure Go (pgx CopyFrom)
- No psql or pg_restore required
- Automatic detection: native for .sql.gz, pg_restore for .dump
FILES MODIFIED:
- cmd/backup.go: Added --native and --fallback-tools flags
- cmd/restore.go: Added --native and --fallback-tools flags
- internal/backup/engine.go: Native engine path in BackupCluster()
- internal/restore/engine.go: Added restoreWithNativeEngine()
- NATIVE_ENGINE_SUMMARY.md: Complete rewrite with accurate docs
- CHANGELOG.md: v5.5.0 release notes
CRITICAL FIX:
- Progress only updated after DB completed, not during restore
- For 100GB DB taking 4+ hours, TUI showed 0% the whole time
CHANGES:
- Heartbeat now reports estimated progress every 5s (was 15s text-only)
- Time-based estimation: ~10MB/s throughput, capped at 95%
- TUI shows spinner + elapsed time when byte-level progress unavailable
- Better visual feedback that restore is actively running
- Use 1.2x multiplier for cluster .tar.gz (pre-compressed dumps)
- Use 5x multiplier for single .sql.gz files (was 7x)
- New CheckSystemMemoryWithType() for archive-aware estimation
- 119GB archive now estimates ~143GB instead of ~833GB
- profile.go only had: conservative, balanced, aggressive, potato
- 'turbo' profile returned ERROR and silently fell back to 'balanced'
- 'balanced' has Jobs=0 which became Jobs=1 after default fallback
- Result: --profile turbo was IGNORED, restore ran single-threaded
Added:
- turbo profile: Jobs=8, ParallelDBs=2
- max-performance profile: Jobs=8, ParallelDBs=4
NOW --profile turbo correctly uses pg_restore --jobs=8
ROOT CAUSE FOUND AND FIXED:
- BuildRestoreCommand() had condition 'if options.Parallel > 1'
- This meant --jobs flag was NEVER added when Parallel was 1 or less
- Changed to 'if options.Parallel > 0' so --jobs is ALWAYS set
- This was THE root cause why restores took 12+ hours instead of ~4 hours
- Now pg_restore --jobs=8 is correctly generated for turbo profile
PROBLEM: User's profile Jobs setting was being overridden in multiple places:
1. restoreSection() for phased restores had NO --jobs flag at all
2. Auto-fallback forced Jobs=1 when PostgreSQL locks couldn't be boosted
3. Auto-fallback forced Jobs=1 on low memory detection
FIX:
- Added --jobs flag to restoreSection() for phased restores
- Removed auto-override of Jobs=1 - now only warns user
- User's profile choice (turbo, performance, etc.) is now respected
- This was causing restores to take 9+ hours instead of ~4 hours
- TUI Settings now persist to .dbbackup.conf file (was only in-memory)
- Native Engine (pure Go) is now the default instead of external tools
- Added FallbackToTools=true for graceful degradation
- Environment variables: USE_NATIVE_ENGINE, FALLBACK_TO_TOOLS
CRITICAL BUG FIX: The --jobs flag and profile Jobs setting were completely
ignored for pg_restore. The code had hardcoded Parallel: 1 instead of using
e.cfg.Jobs, causing all restores to run single-threaded regardless of
configuration.
This fix enables restores to match native pg_restore -j8 performance:
- 12h 38m -> ~4h for 119.5GB cluster backup
- Throughput: 2.7 MB/s -> ~8 MB/s
Affected functions:
- restorePostgreSQLDump()
- restorePostgreSQLDumpWithOwnership()
Now logs parallel_jobs value for visibility. Turbo profile with Jobs: 8
now correctly passes --jobs=8 to pg_restore.
- Update version string to 5.1.14
- Update CHANGELOG with v5.1.10-v5.1.14 features
- Update README with new enterprise features
- Remove development files from repository
- Add sensitive files to .gitignore
- Enhanced professional tone across all markdown files
- Replaced visual indicators with clear text-based labels
- Standardized formatting for business-appropriate documentation
- Updated CHANGELOG.md, README.md, and technical documentation
- Improved readability for enterprise deployment scenarios
- Maintained all technical content while enhancing presentation
Fixes from Opus code review:
- PostgreSQL: Use native TEXT format for COPY (matches FROM stdin header)
- MySQL: Escape backticks in restore to prevent SQL injection
- MySQL: Add SHOW BINARY LOG STATUS fallback for MySQL 8.0.22+
- Fix duration calculation to accurately track backup time
Updated messaging: We built our own machines - really big step.
🚀 MAJOR RELEASE - Complete Independence from External Tools
This release implements native Go database engines that eliminate
ALL external dependencies (pg_dump, mysqldump, pg_restore, etc.).
Major Changes:
- Native PostgreSQL engine with pgx protocol support
- Native MySQL engine with go-sql-driver implementation
- Advanced data type handling for all database types
- Zero external tool dependencies
- New CLI flags: --native, --fallback-tools, --native-debug
- Comprehensive architecture for future enhancements
Technical Impact:
- Pure Go implementation for all backup operations
- Direct database protocol communication
- Improved performance and reliability
- Simplified deployment with single binary
- Backward compatibility with all existing features
- SEC#1: Removed --password CLI flag (prevents password in ps aux)
- SEC#2: All backup files now created with 0600 permissions
- #4: Fixed directory race conditions in parallel backups
- Added internal/fs/secure.go for secure file operations
- Added internal/exitcode/codes.go for standard exit codes
- Updated CHANGELOG.md with comprehensive release notes
- Pre-extract cluster archive once when listing databases
- Reuse extracted directory for restore (avoids second extraction)
- Add ListDatabasesFromExtractedDir() for fast DB listing from disk
- Automatic cleanup of temp directory after restore
- Performance: 50GB cluster now processes 1x instead of 2x (saves 5-15min)
- Add CopyWithContext to all long-running I/O operations
- Fix restore/extract.go: single DB extraction from cluster
- Fix wal/compression.go: WAL compression/decompression
- Fix restore/engine.go: SQL restore streaming
- Fix backup/engine.go: pg_dump/mysqldump streaming
- Fix cloud/s3.go, azure.go, gcs.go: cloud transfers
- Fix drill/engine.go: DR drill decompression
- All operations now check context every 1MB for responsive cancellation
- Partial files cleaned up on interruption
Version 4.2.4
- ValidateAndExtractCluster no longer calls ValidateArchive internally
- Added CopyWithContext for context-aware file copying during extraction
- Ctrl+C now immediately interrupts large file extractions
- Partial files cleaned up on cancellation
Version 4.2.3
- New internal/tui/health.go (644 lines)
- 10 health checks with async execution
- Added to Tools menu as 'System Health Check'
- Color-coded results + recommendations
- Updated CHANGELOG.md for v4.2.0
- Added --socket flag for explicit socket path
- Auto-detect socket from --host if path starts with /
- Updated mysqldump/mysql commands to use -S flag
- Works for both backup and restore operations
- Added dbbackup_build_info{server,version,commit} metric for fleet tracking
- Fixed docs: pitr_base is auto-assigned by 'dbbackup pitr base', not CLI flag value
- Updated EXPORTER.md and METRICS.md with build_info documentation
Tools submenu expansion:
- Table Sizes: view top 100 tables sorted by size (PG/MySQL)
- Kill Connections: list/kill active DB connections
- Drop Database: safe drop with double confirmation
v3.42.108
- New 'Tools' submenu in TUI with utility functions
- Blob Statistics: scan database for bytea/blob columns with size analysis
- New 'dbbackup blob stats' CLI command
- Supports PostgreSQL (bytea, oid) and MySQL (blob types)
- Shows row counts, total size, avg size, max size per column
- Recommendations for databases with >100MB blob data
FIXES:
- Switch from mattn/go-sqlite3 (CGO) to modernc.org/sqlite (pure Go)
Binaries compiled with CGO_ENABLED=0 now work correctly
- Fix MySQL positional database argument being ignored
'dbbackup backup single gitea --db-type mysql' now uses 'gitea' correctly
Feature requested by DBA: Limit upload/download speed during business hours.
- New --bandwidth-limit flag for cloud operations (S3, GCS, Azure, MinIO, B2)
- Supports human-readable formats: 10MB/s, 50MiB/s, 100Mbps, unlimited
- Environment variable: DBBACKUP_BANDWIDTH_LIMIT
- Token-bucket style throttling with 100ms windows for smooth limiting
- Reduces multipart concurrency when throttled for better rate control
- Unit tests for parsing and throttle behavior
- Remove ALL remaining exec.Command tar/gzip/gunzip calls from internal code
- diagnose.go: Replace 'tar -tzf' test with direct file open check
- large_restore_check.go: Replace 'gzip -t' and 'gzip -l' with in-process pgzip verification
- pitr/restore.go: Replace 'tar -xf' with in-process archive/tar extraction
- All backup/restore operations now 100% in-process using github.com/klauspost/pgzip
- Benefits: No external tool dependencies, 2-4x faster on multi-core, reliable error handling
- Note: Docker drill container commands still use gunzip for in-container ops (intentional)
- Archive now extracted once and reused for validation + restore
- Saves 3-6 min on 50GB clusters, 1-2 min on 10GB clusters
- New ValidateAndExtractCluster() combines validation + extraction
- RestoreCluster() accepts optional preExtractedPath parameter
- Enhanced tar.gz validation with fast stream-based header checks
- Disk space checks intelligently skipped for pre-extracted directories
- Fully backward compatible, optimization auto-enabled with --diagnose
Bubbletea v1.3+ sends InterruptMsg for SIGINT signals instead of
KeyMsg with 'ctrl+c', causing cancellation to not work properly.
- Add tea.InterruptMsg handling to restore_exec.go
- Add tea.InterruptMsg handling to backup_exec.go
- Add tea.InterruptMsg handling to menu.go
- Call cleanup.KillOrphanedProcesses on all interrupt paths
- No zombie pg_dump/pg_restore/gzip processes left behind
Fixes Ctrl+C not working during cluster restore/backup operations.
v3.42.50
