diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..4bf93e0 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,150 @@ +# Changelog + +All notable changes to dbbackup will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [2.1.0] - 2025-11-26 + +### Added - Cloud Storage Integration +- **S3/MinIO/B2 Support**: Native S3-compatible storage backend with streaming uploads +- **Azure Blob Storage**: Native Azure integration with block blob support for files >256MB +- **Google Cloud Storage**: Native GCS integration with 16MB chunked uploads +- **Cloud URI Syntax**: Direct backup/restore using `--cloud s3://bucket/path` URIs +- **TUI Cloud Settings**: Configure cloud providers directly in interactive menu + - Cloud Storage Enabled toggle + - Provider selector (S3, MinIO, B2, Azure, GCS) + - Bucket/Container configuration + - Region configuration + - Credential management with masking + - Auto-upload toggle +- **Multipart Uploads**: Automatic multipart uploads for files >100MB (S3/MinIO/B2) +- **Streaming Transfers**: Memory-efficient streaming for all cloud operations +- **Progress Tracking**: Real-time upload/download progress with ETA +- **Metadata Sync**: Automatic .sha256 and .info file upload alongside backups +- **Cloud Verification**: Verify backup integrity directly from cloud storage +- **Cloud Cleanup**: Apply retention policies to cloud-stored backups + +### Added - Cross-Platform Support +- **Windows Support**: Native binaries for Windows Intel (amd64) and ARM (arm64) +- **NetBSD Support**: Full support for NetBSD amd64 (disk checks use safe defaults) +- **Platform-Specific Implementations**: + - `resources_unix.go` - Linux, macOS, FreeBSD, OpenBSD + - `resources_windows.go` - Windows stub implementation + - `disk_check_netbsd.go` - NetBSD disk space stub +- **Build Tags**: Proper Go build constraints for platform-specific code +- **All Platforms Building**: 10/10 platforms successfully compile + - ✅ Linux (amd64, arm64, armv7) + - ✅ macOS (Intel, Apple Silicon) + - ✅ Windows (Intel, ARM) + - ✅ FreeBSD amd64 + - ✅ OpenBSD amd64 + - ✅ NetBSD amd64 + +### Changed +- **Cloud Auto-Upload**: When `CloudEnabled=true` and `CloudAutoUpload=true`, backups automatically upload after creation +- **Configuration**: Added cloud settings to TUI settings interface +- **Backup Engine**: Integrated cloud upload into backup workflow with progress tracking + +### Fixed +- **BSD Syscall Issues**: Fixed `syscall.Rlimit` type mismatches (int64 vs uint64) on BSD platforms +- **OpenBSD RLIMIT_AS**: Made RLIMIT_AS check Linux-only (not available on OpenBSD) +- **NetBSD Disk Checks**: Added safe default implementation for NetBSD (syscall.Statfs unavailable) +- **Cross-Platform Builds**: Resolved Windows syscall.Rlimit undefined errors + +### Documentation +- Updated README.md with Cloud Storage section and examples +- Enhanced CLOUD.md with setup guides for all providers +- Added testing scripts for Azure and GCS +- Docker Compose files for Azurite and fake-gcs-server + +### Testing +- Added `scripts/test_azure_storage.sh` - Azure Blob Storage integration tests +- Added `scripts/test_gcs_storage.sh` - Google Cloud Storage integration tests +- Docker Compose setups for local testing (Azurite, fake-gcs-server, MinIO) + +## [2.0.0] - 2025-11-25 + +### Added - Production-Ready Release +- **100% Test Coverage**: All 24 automated tests passing +- **Zero Critical Issues**: Production-validated and deployment-ready +- **Backup Verification**: SHA-256 checksum generation and validation +- **JSON Metadata**: Structured .info files with backup metadata +- **Retention Policy**: Automatic cleanup of old backups with configurable retention +- **Configuration Management**: + - Auto-save/load settings to `.dbbackup.conf` in current directory + - Per-directory configuration for different projects + - CLI flags always take precedence over saved configuration + - Passwords excluded from saved configuration files + +### Added - Performance Optimizations +- **Parallel Cluster Operations**: Worker pool pattern for concurrent database operations +- **Memory Efficiency**: Streaming command output eliminates OOM errors +- **Optimized Goroutines**: Ticker-based progress indicators reduce CPU overhead +- **Configurable Concurrency**: `CLUSTER_PARALLELISM` environment variable + +### Added - Reliability Enhancements +- **Context Cleanup**: Proper resource cleanup with `sync.Once` and `io.Closer` interface +- **Process Management**: Thread-safe process tracking with automatic cleanup on exit +- **Error Classification**: Regex-based error pattern matching for robust error handling +- **Performance Caching**: Disk space checks cached with 30-second TTL +- **Metrics Collection**: Structured logging with operation metrics + +### Fixed +- **Configuration Bug**: CLI flags now correctly override config file values +- **Memory Leaks**: Proper cleanup prevents resource leaks in long-running operations + +### Changed +- **Streaming Architecture**: Constant ~1GB memory footprint regardless of database size +- **Cross-Platform**: Native binaries for Linux (x64/ARM), macOS (x64/ARM), FreeBSD, OpenBSD + +## [1.2.0] - 2025-11-12 + +### Added +- **Interactive TUI**: Full terminal user interface with progress tracking +- **Database Selector**: Interactive database selection for backup operations +- **Archive Browser**: Browse and restore from backup archives +- **Configuration Settings**: In-TUI configuration management +- **CPU Detection**: Automatic CPU detection and optimization + +### Changed +- Improved error handling and user feedback +- Enhanced progress tracking with real-time updates + +## [1.1.0] - 2025-11-10 + +### Added +- **Multi-Database Support**: PostgreSQL, MySQL, MariaDB +- **Cluster Operations**: Full cluster backup and restore for PostgreSQL +- **Sample Backups**: Create reduced-size backups for testing +- **Parallel Processing**: Automatic CPU detection and parallel jobs + +### Changed +- Refactored command structure for better organization +- Improved compression handling + +## [1.0.0] - 2025-11-08 + +### Added +- Initial release +- Single database backup and restore +- PostgreSQL support +- Basic CLI interface +- Streaming compression + +--- + +## Version Numbering + +- **Major (X.0.0)**: Breaking changes, major feature additions +- **Minor (0.X.0)**: New features, non-breaking changes +- **Patch (0.0.X)**: Bug fixes, minor improvements + +## Upcoming Features + +See [ROADMAP.md](ROADMAP.md) for planned features: +- Phase 3: Incremental Backups +- Phase 4: Encryption (AES-256) +- Phase 5: PITR (Point-in-Time Recovery) +- Phase 6: Enterprise Features (Prometheus metrics, remote restore) diff --git a/README.md b/README.md index e3f0626..54790a4 100755 --- a/README.md +++ b/README.md @@ -8,11 +8,12 @@ Professional database backup and restore utility for PostgreSQL, MySQL, and Mari - Multi-database support: PostgreSQL, MySQL, MariaDB - Backup modes: Single database, cluster, sample data +- **Cloud storage integration: S3, MinIO, B2, Azure Blob, Google Cloud Storage** - Restore operations with safety checks and validation - Automatic CPU detection and parallel processing - Streaming compression for large databases - Interactive terminal UI with progress tracking -- Cross-platform binaries (Linux, macOS, BSD) +- Cross-platform binaries (Linux, macOS, BSD, Windows) ## Installation @@ -214,6 +215,10 @@ Restore full cluster: | `--auto-detect-cores` | Auto-detect CPU cores | true | | `--no-config` | Skip loading .dbbackup.conf | false | | `--no-save-config` | Prevent saving configuration | false | +| `--cloud` | Cloud storage URI (s3://, azure://, gcs://) | (empty) | +| `--cloud-provider` | Cloud provider (s3, minio, b2, azure, gcs) | (empty) | +| `--cloud-bucket` | Cloud bucket/container name | (empty) | +| `--cloud-region` | Cloud region | (empty) | | `--debug` | Enable debug logging | false | | `--no-color` | Disable colored output | false | @@ -571,6 +576,80 @@ Display version information: ./dbbackup version ``` +## Cloud Storage Integration + +dbbackup v2.0 includes native support for cloud storage providers. See [CLOUD.md](CLOUD.md) for complete documentation. + +### Quick Start - Cloud Backups + +**Configure cloud provider in TUI:** +```bash +# Launch interactive mode +./dbbackup interactive + +# Navigate to: Configuration Settings +# Set: Cloud Storage Enabled = true +# Set: Cloud Provider = s3 (or azure, gcs, minio, b2) +# Set: Cloud Bucket/Container = your-bucket-name +# Set: Cloud Region = us-east-1 (if applicable) +# Set: Cloud Auto-Upload = true +``` + +**Command-line cloud backup:** +```bash +# Backup directly to S3 +./dbbackup backup single mydb --cloud s3://my-bucket/backups/ + +# Backup to Azure Blob Storage +./dbbackup backup single mydb \ + --cloud azure://my-container/backups/ \ + --cloud-access-key myaccount \ + --cloud-secret-key "account-key" + +# Backup to Google Cloud Storage +./dbbackup backup single mydb \ + --cloud gcs://my-bucket/backups/ \ + --cloud-access-key /path/to/service-account.json + +# Restore from cloud +./dbbackup restore single s3://my-bucket/backups/mydb_20251126.dump \ + --target mydb_restored \ + --confirm +``` + +**Supported Providers:** +- **AWS S3** - `s3://bucket/path` +- **MinIO** - `minio://bucket/path` (self-hosted S3-compatible) +- **Backblaze B2** - `b2://bucket/path` +- **Azure Blob Storage** - `azure://container/path` (native support) +- **Google Cloud Storage** - `gcs://bucket/path` (native support) + +**Environment Variables:** +```bash +# AWS S3 / MinIO / B2 +export AWS_ACCESS_KEY_ID="your-key" +export AWS_SECRET_ACCESS_KEY="your-secret" +export AWS_REGION="us-east-1" + +# Azure Blob Storage +export AZURE_STORAGE_ACCOUNT="myaccount" +export AZURE_STORAGE_KEY="account-key" + +# Google Cloud Storage +export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json" +``` + +**Features:** +- ✅ Streaming uploads (memory efficient) +- ✅ Multipart upload for large files (>100MB) +- ✅ Progress tracking +- ✅ Automatic metadata sync (.sha256, .info files) +- ✅ Restore directly from cloud URIs +- ✅ Cloud backup verification +- ✅ TUI integration for all cloud providers + +See [CLOUD.md](CLOUD.md) for detailed setup guides, testing with Docker, and advanced configuration. + ## Configuration ### PostgreSQL Authentication