dbbackup/docs/NATIVE_ENGINE_ROADMAP.md

Native Engine Implementation Roadmap

Complete Elimination of External Tool Dependencies

Current Status (Updated January 2026)

• External tools to eliminate: pg_dump, pg_dumpall, pg_restore, psql, mysqldump, mysql, mysqlbinlog
• Target: 100% pure Go implementation with zero external dependencies
• Benefit: Self-contained binary, better integration, enhanced control
• Status: Phase 1 and Phase 2 largely complete, Phase 3-5 in progress

Phase 1: Core Native Engines (8-12 weeks) - COMPLETE

PostgreSQL Native Engine (4-6 weeks) - COMPLETE

Week 1-2: Foundation

• Basic engine architecture and interfaces
• Connection management with pgx/v5
• SQL format backup implementation
• Basic table data export using COPY TO STDOUT
• Schema extraction from information_schema

Week 3-4: Advanced Features

• Complete schema object support (tables, views, functions, sequences)
• Foreign key and constraint handling
• PostgreSQL data type support (arrays, JSON, custom types)
• Transaction consistency and locking
• Parallel table processing

Week 5-6: Formats and Polish

• Custom format implementation (PostgreSQL binary format)
• Directory format support
• Tar format support
• Compression integration (pgzip, lz4, zstd)
• Progress reporting and metrics

MySQL Native Engine (4-6 weeks) - COMPLETE

Week 1-2: Foundation

• Basic engine architecture
• Connection management with go-sql-driver/mysql
• SQL script generation
• Table data export with SELECT and INSERT statements
• Schema extraction from information_schema

Week 3-4: MySQL Specifics

• Storage engine handling (InnoDB, MyISAM, etc.)
• MySQL data type support (including BLOB, TEXT variants)
• Character set and collation handling
• AUTO_INCREMENT and foreign key constraints
• Stored procedures, functions, triggers, events

Week 5-6: Enterprise Features

• Binary log position capture (SHOW MASTER STATUS / SHOW BINARY LOG STATUS)
• GTID support for MySQL 5.6+
• Single transaction consistent snapshots
• Extended INSERT optimization
• MySQL-specific optimizations (DISABLE KEYS, etc.)

Phase 2: Advanced Protocol Features (6-8 weeks) - COMPLETE

PostgreSQL Advanced (3-4 weeks) - COMPLETE

• Custom format parser/writer: Implement PostgreSQL's custom archive format
• Large object (BLOB) support: Handle pg_largeobject system catalog
• Parallel processing: Multiple worker goroutines for table dumping
• Incremental backup support: Track LSN positions (partial)
• Point-in-time recovery: WAL file integration (partial)

MySQL Advanced (3-4 weeks) - COMPLETE

• Binary log parsing: Native implementation replacing mysqlbinlog
• PITR support: Binary log position tracking and replay
• MyISAM vs InnoDB optimizations: Engine-specific dump strategies
• Parallel dumping: Multi-threaded table processing
• Incremental support: Binary log-based incremental backups (partial)

Phase 3: Restore Engines (4-6 weeks) - IN PROGRESS

PostgreSQL Restore Engine

• SQL script execution: Native psql replacement
• Custom format restore: Parse and restore from binary format
• Selective restore: Schema-only, data-only, table-specific
• Parallel restore: Multi-worker restoration
• Error handling: Continue on error, skip existing objects

MySQL Restore Engine

• SQL script execution: Native mysql client replacement
• Batch processing: Efficient INSERT statement execution
• Error recovery: Handle duplicate key, constraint violations
• Progress reporting: Track restoration progress
• Point-in-time restore: Apply binary logs to specific positions

Phase 4: Integration & Migration (2-4 weeks) - COMPLETE

Engine Selection Framework

• Configuration option: --native flag enables native engines
• Automatic fallback: --fallback-tools uses tools if native engine fails
• Performance comparison: Benchmarking native vs tools
• Feature parity validation: Ensure native engines match tool behavior

Code Integration

• Update backup engine: Integrate native engines into existing flow
• Update restore engine: Replace tool-based restore logic
• Update PITR: Native binary log processing (partial)
• Update verification: Native dump file analysis

Legacy Code Removal - DEFERRED

• Remove tool validation: Keep ValidateBackupTools() for fallback mode
• Remove subprocess execution: Keep exec.Command for fallback mode
• Remove tool-specific error handling: Maintain for compatibility
• Update documentation: Native engine docs complete

Phase 5: Testing & Validation (4-6 weeks) - IN PROGRESS

Comprehensive Test Suite

• Unit tests: All native engine components
• Integration tests: End-to-end backup/restore cycles
• Performance tests: Compare native vs tool-based approaches
• Compatibility tests: Various PostgreSQL/MySQL versions
• Edge case tests: Large databases, complex schemas, exotic data types

Data Validation

• Schema comparison: Verify restored schema matches original
• Data integrity: Checksum validation of restored data
• Foreign key consistency: Ensure referential integrity
• Performance benchmarks: Backup/restore speed comparisons

Technical Implementation Details

Key Components to Implement

PostgreSQL Protocol Details:

// Core SQL generation for schema objects
func (e *PostgreSQLNativeEngine) generateTableDDL(ctx context.Context, schema, table string) (string, error)
func (e *PostgreSQLNativeEngine) generateViewDDL(ctx context.Context, schema, view string) (string, error)
func (e *PostgreSQLNativeEngine) generateFunctionDDL(ctx context.Context, schema, function string) (string, error)

// Custom format implementation
func (e *PostgreSQLNativeEngine) writeCustomFormatHeader(w io.Writer) error
func (e *PostgreSQLNativeEngine) writeCustomFormatTOC(w io.Writer, objects []DatabaseObject) error
func (e *PostgreSQLNativeEngine) writeCustomFormatData(w io.Writer, obj DatabaseObject) error

MySQL Protocol Details:

// Binary log processing
func (e *MySQLNativeEngine) parseBinlogEvent(data []byte) (*BinlogEvent, error)
func (e *MySQLNativeEngine) applyBinlogEvent(ctx context.Context, event *BinlogEvent) error

// Storage engine optimization
func (e *MySQLNativeEngine) optimizeForEngine(engine string) *DumpStrategy
func (e *MySQLNativeEngine) generateOptimizedInserts(rows [][]interface{}) []string

Performance Targets

• Backup Speed: Match or exceed external tools (within 10%)
• Memory Usage: Stay under 500MB for large database operations
• Concurrency: Support 4-16 parallel workers based on system cores
• Compression: Achieve 2-4x speedup with native pgzip integration

Compatibility Requirements

• PostgreSQL: Support versions 10, 11, 12, 13, 14, 15, 16
• MySQL: Support versions 5.7, 8.0, 8.1+ and MariaDB 10.3+
• Platforms: Linux, macOS, Windows (ARM64 and AMD64)
• Go Version: Go 1.24+ for latest features and performance

Rollout Strategy

Gradual Migration Approach

1. Phase 1: Native engines available as --engine=native option
2. Phase 2: Native engines become default, tools as fallback
3. Phase 3: Tools deprecated with warning messages
4. Phase 4: Tools completely removed, native only

Risk Mitigation

• Extensive testing on real-world databases before each phase
• Performance monitoring to ensure native engines meet expectations
• User feedback collection during preview phases
• Rollback capability to tool-based engines if issues arise

Success Metrics

• Zero external dependencies: Native engines work without pg_dump, mysqldump, etc.
• Performance parity: Native engines >= 90% speed of external tools
• Feature completeness: All current functionality preserved
• Reliability: <0.1% failure rate in production environments (monitoring)
• Binary size: Single self-contained executable ~55MB

This roadmap achieves the goal of complete elimination of external tool dependencies while maintaining all current functionality and performance characteristics.

---

Implementation Summary (v5.1.14)

The native engine implementation is production-ready with the following components:

Component	File	Functions	Status
PostgreSQL Engine	postgresql.go	37	Complete
MySQL Engine	mysql.go	40	Complete
Advanced Engine	advanced.go	17	Complete
Engine Manager	manager.go	12	Complete
Restore Engine	restore.go	8	Partial
Integration	integration_example.go	6	Complete

Total: 120 functions across 6 files

Usage:

# Use native engines (no external tools required)
dbbackup backup single mydb --native

# Use native with fallback to tools if needed
dbbackup backup single mydb --native --fallback-tools

# Enable debug output for native engines
dbbackup backup single mydb --native --native-debug