diff --git a/CLOUD.md b/CLOUD.md new file mode 100644 index 0000000..b9a63c4 --- /dev/null +++ b/CLOUD.md @@ -0,0 +1,758 @@ +# Cloud Storage Guide for dbbackup + +## Overview + +dbbackup v2.0 includes comprehensive cloud storage integration, allowing you to backup directly to S3-compatible storage providers and restore from cloud URIs. + +**Supported Providers:** +- AWS S3 +- MinIO (self-hosted S3-compatible) +- Backblaze B2 +- Google Cloud Storage (via S3 compatibility) +- Any S3-compatible storage + +**Key Features:** +- ✅ Direct backup to cloud with `--cloud` URI flag +- ✅ Restore from cloud URIs +- ✅ Verify cloud backup integrity +- ✅ Apply retention policies to cloud storage +- ✅ Multipart upload for large files (>100MB) +- ✅ Progress tracking for uploads/downloads +- ✅ Automatic metadata synchronization +- ✅ Streaming transfers (memory efficient) + +--- + +## Quick Start + +### 1. Set Up Credentials + +```bash +# For AWS S3 +export AWS_ACCESS_KEY_ID="your-access-key" +export AWS_SECRET_ACCESS_KEY="your-secret-key" +export AWS_REGION="us-east-1" + +# For MinIO +export AWS_ACCESS_KEY_ID="minioadmin" +export AWS_SECRET_ACCESS_KEY="minioadmin123" +export AWS_ENDPOINT_URL="http://localhost:9000" + +# For Backblaze B2 +export AWS_ACCESS_KEY_ID="your-b2-key-id" +export AWS_SECRET_ACCESS_KEY="your-b2-application-key" +export AWS_ENDPOINT_URL="https://s3.us-west-002.backblazeb2.com" +``` + +### 2. Backup with Cloud URI + +```bash +# Backup to S3 +dbbackup backup single mydb --cloud s3://my-bucket/backups/ + +# Backup to MinIO +dbbackup backup single mydb --cloud minio://my-bucket/backups/ + +# Backup to Backblaze B2 +dbbackup backup single mydb --cloud b2://my-bucket/backups/ +``` + +### 3. Restore from Cloud + +```bash +# Restore from cloud URI +dbbackup restore single s3://my-bucket/backups/mydb_20260115_120000.dump --confirm + +# Restore to different database +dbbackup restore single s3://my-bucket/backups/mydb.dump \ + --target mydb_restored \ + --confirm +``` + +--- + +## URI Syntax + +Cloud URIs follow this format: + +``` +://// +``` + +**Supported Providers:** +- `s3://` - AWS S3 or S3-compatible storage +- `minio://` - MinIO (auto-enables path-style addressing) +- `b2://` - Backblaze B2 +- `gs://` or `gcs://` - Google Cloud Storage +- `azure://` - Azure Blob Storage (coming soon) + +**Examples:** +```bash +s3://production-backups/databases/postgres/ +minio://local-backups/dev/mydb/ +b2://offsite-backups/daily/ +gs://gcp-backups/prod/ +``` + +--- + +## Configuration Methods + +### Method 1: Cloud URIs (Recommended) + +```bash +dbbackup backup single mydb --cloud s3://my-bucket/backups/ +``` + +### Method 2: Individual Flags + +```bash +dbbackup backup single mydb \ + --cloud-auto-upload \ + --cloud-provider s3 \ + --cloud-bucket my-bucket \ + --cloud-prefix backups/ +``` + +### Method 3: Environment Variables + +```bash +export CLOUD_ENABLED=true +export CLOUD_AUTO_UPLOAD=true +export CLOUD_PROVIDER=s3 +export CLOUD_BUCKET=my-bucket +export CLOUD_PREFIX=backups/ +export CLOUD_REGION=us-east-1 + +dbbackup backup single mydb +``` + +### Method 4: Config File + +```toml +# ~/.dbbackup.conf +[cloud] +enabled = true +auto_upload = true +provider = "s3" +bucket = "my-bucket" +prefix = "backups/" +region = "us-east-1" +``` + +--- + +## Commands + +### Cloud Upload + +Upload existing backup files to cloud storage: + +```bash +# Upload single file +dbbackup cloud upload /backups/mydb.dump \ + --cloud-provider s3 \ + --cloud-bucket my-bucket + +# Upload with cloud URI flags +dbbackup cloud upload /backups/mydb.dump \ + --cloud-provider minio \ + --cloud-bucket local-backups \ + --cloud-endpoint http://localhost:9000 + +# Upload multiple files +dbbackup cloud upload /backups/*.dump \ + --cloud-provider s3 \ + --cloud-bucket my-bucket \ + --verbose +``` + +### Cloud Download + +Download backups from cloud storage: + +```bash +# Download to current directory +dbbackup cloud download mydb.dump . \ + --cloud-provider s3 \ + --cloud-bucket my-bucket + +# Download to specific directory +dbbackup cloud download backups/mydb.dump /restore/ \ + --cloud-provider s3 \ + --cloud-bucket my-bucket \ + --verbose +``` + +### Cloud List + +List backups in cloud storage: + +```bash +# List all backups +dbbackup cloud list \ + --cloud-provider s3 \ + --cloud-bucket my-bucket + +# List with prefix filter +dbbackup cloud list \ + --cloud-provider s3 \ + --cloud-bucket my-bucket \ + --cloud-prefix postgres/ + +# Verbose output with details +dbbackup cloud list \ + --cloud-provider s3 \ + --cloud-bucket my-bucket \ + --verbose +``` + +### Cloud Delete + +Delete backups from cloud storage: + +```bash +# Delete specific backup (with confirmation prompt) +dbbackup cloud delete mydb_old.dump \ + --cloud-provider s3 \ + --cloud-bucket my-bucket + +# Delete without confirmation +dbbackup cloud delete mydb_old.dump \ + --cloud-provider s3 \ + --cloud-bucket my-bucket \ + --confirm +``` + +### Backup with Auto-Upload + +```bash +# Backup and automatically upload +dbbackup backup single mydb --cloud s3://my-bucket/backups/ + +# With individual flags +dbbackup backup single mydb \ + --cloud-auto-upload \ + --cloud-provider s3 \ + --cloud-bucket my-bucket \ + --cloud-prefix backups/ +``` + +### Restore from Cloud + +```bash +# Restore from cloud URI (auto-download) +dbbackup restore single s3://my-bucket/backups/mydb.dump --confirm + +# Restore to different database +dbbackup restore single s3://my-bucket/backups/mydb.dump \ + --target mydb_restored \ + --confirm + +# Restore with database creation +dbbackup restore single s3://my-bucket/backups/mydb.dump \ + --create \ + --confirm +``` + +### Verify Cloud Backups + +```bash +# Verify single cloud backup +dbbackup verify-backup s3://my-bucket/backups/mydb.dump + +# Quick verification (size check only) +dbbackup verify-backup s3://my-bucket/backups/mydb.dump --quick + +# Verbose output +dbbackup verify-backup s3://my-bucket/backups/mydb.dump --verbose +``` + +### Cloud Cleanup + +Apply retention policies to cloud storage: + +```bash +# Cleanup old backups (dry-run) +dbbackup cleanup s3://my-bucket/backups/ \ + --retention-days 30 \ + --min-backups 5 \ + --dry-run + +# Actual cleanup +dbbackup cleanup s3://my-bucket/backups/ \ + --retention-days 30 \ + --min-backups 5 + +# Pattern-based cleanup +dbbackup cleanup s3://my-bucket/backups/ \ + --retention-days 7 \ + --min-backups 3 \ + --pattern "mydb_*.dump" +``` + +--- + +## Provider-Specific Setup + +### AWS S3 + +**Prerequisites:** +- AWS account +- S3 bucket created +- IAM user with S3 permissions + +**IAM Policy:** +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:DeleteObject", + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::my-bucket/*", + "arn:aws:s3:::my-bucket" + ] + } + ] +} +``` + +**Configuration:** +```bash +export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE" +export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" +export AWS_REGION="us-east-1" + +dbbackup backup single mydb --cloud s3://my-bucket/backups/ +``` + +### MinIO (Self-Hosted) + +**Setup with Docker:** +```bash +docker run -d \ + -p 9000:9000 \ + -p 9001:9001 \ + -e "MINIO_ROOT_USER=minioadmin" \ + -e "MINIO_ROOT_PASSWORD=minioadmin123" \ + --name minio \ + minio/minio server /data --console-address ":9001" + +# Create bucket +docker exec minio mc alias set local http://localhost:9000 minioadmin minioadmin123 +docker exec minio mc mb local/backups +``` + +**Configuration:** +```bash +export AWS_ACCESS_KEY_ID="minioadmin" +export AWS_SECRET_ACCESS_KEY="minioadmin123" +export AWS_ENDPOINT_URL="http://localhost:9000" + +dbbackup backup single mydb --cloud minio://backups/db/ +``` + +**Or use docker-compose:** +```bash +docker-compose -f docker-compose.minio.yml up -d +``` + +### Backblaze B2 + +**Prerequisites:** +- Backblaze account +- B2 bucket created +- Application key generated + +**Configuration:** +```bash +export AWS_ACCESS_KEY_ID="" +export AWS_SECRET_ACCESS_KEY="" +export AWS_ENDPOINT_URL="https://s3.us-west-002.backblazeb2.com" +export AWS_REGION="us-west-002" + +dbbackup backup single mydb --cloud b2://my-bucket/backups/ +``` + +### Google Cloud Storage + +**Prerequisites:** +- GCP account +- GCS bucket with S3 compatibility enabled +- HMAC keys generated + +**Enable S3 Compatibility:** +1. Go to Cloud Storage > Settings > Interoperability +2. Create HMAC keys + +**Configuration:** +```bash +export AWS_ACCESS_KEY_ID="" +export AWS_SECRET_ACCESS_KEY="" +export AWS_ENDPOINT_URL="https://storage.googleapis.com" + +dbbackup backup single mydb --cloud gs://my-bucket/backups/ +``` + +--- + +## Features + +### Multipart Upload + +Files larger than 100MB automatically use multipart upload for: +- Faster transfers with parallel parts +- Resume capability on failure +- Better reliability for large files + +**Configuration:** +- Part size: 10MB +- Concurrency: 10 parallel parts +- Automatic based on file size + +### Progress Tracking + +Real-time progress for uploads and downloads: + +```bash +Uploading backup to cloud... +Progress: 10% +Progress: 20% +Progress: 30% +... +Upload completed: /backups/mydb.dump (1.2 GB) +``` + +### Metadata Synchronization + +Automatically uploads `.meta.json` with each backup containing: +- SHA-256 checksum +- Database name and type +- Backup timestamp +- File size +- Compression info + +### Automatic Verification + +Downloads from cloud include automatic checksum verification: + +```bash +Downloading backup from cloud... +Download completed +Verifying checksum... +Checksum verified successfully: sha256=abc123... +``` + +--- + +## Testing + +### Local Testing with MinIO + +**1. Start MinIO:** +```bash +docker-compose -f docker-compose.minio.yml up -d +``` + +**2. Run Integration Tests:** +```bash +./scripts/test_cloud_storage.sh +``` + +**3. Manual Testing:** +```bash +# Set credentials +export AWS_ACCESS_KEY_ID=minioadmin +export AWS_SECRET_ACCESS_KEY=minioadmin123 +export AWS_ENDPOINT_URL=http://localhost:9000 + +# Test backup +dbbackup backup single mydb --cloud minio://test-backups/test/ + +# Test restore +dbbackup restore single minio://test-backups/test/mydb.dump --confirm + +# Test verify +dbbackup verify-backup minio://test-backups/test/mydb.dump + +# Test cleanup +dbbackup cleanup minio://test-backups/test/ --retention-days 7 --dry-run +``` + +**4. Access MinIO Console:** +- URL: http://localhost:9001 +- Username: `minioadmin` +- Password: `minioadmin123` + +--- + +## Best Practices + +### Security + +1. **Never commit credentials:** + ```bash + # Use environment variables or config files + export AWS_ACCESS_KEY_ID="..." + ``` + +2. **Use IAM roles when possible:** + ```bash + # On EC2/ECS, credentials are automatic + dbbackup backup single mydb --cloud s3://bucket/ + ``` + +3. **Restrict bucket permissions:** + - Minimum required: GetObject, PutObject, DeleteObject, ListBucket + - Use bucket policies to limit access + +4. **Enable encryption:** + - S3: Server-side encryption enabled by default + - MinIO: Configure encryption at rest + +### Performance + +1. **Use multipart for large backups:** + - Automatic for files >100MB + - Configure concurrency based on bandwidth + +2. **Choose nearby regions:** + ```bash + --cloud-region us-west-2 # Closest to your servers + ``` + +3. **Use compression:** + ```bash + --compression gzip # Reduces upload size + ``` + +### Reliability + +1. **Test restores regularly:** + ```bash + # Monthly restore test + dbbackup restore single s3://bucket/latest.dump --target test_restore + ``` + +2. **Verify backups:** + ```bash + # Daily verification + dbbackup verify-backup s3://bucket/backups/*.dump + ``` + +3. **Monitor retention:** + ```bash + # Weekly cleanup check + dbbackup cleanup s3://bucket/ --retention-days 30 --dry-run + ``` + +### Cost Optimization + +1. **Use lifecycle policies:** + - S3: Transition old backups to Glacier + - Configure in AWS Console or bucket policy + +2. **Cleanup old backups:** + ```bash + dbbackup cleanup s3://bucket/ --retention-days 30 --min-backups 10 + ``` + +3. **Choose appropriate storage class:** + - Standard: Frequent access + - Infrequent Access: Monthly restores + - Glacier: Long-term archive + +--- + +## Troubleshooting + +### Connection Issues + +**Problem:** Cannot connect to S3/MinIO + +```bash +Error: failed to create cloud backend: failed to load AWS config +``` + +**Solution:** +1. Check credentials: + ```bash + echo $AWS_ACCESS_KEY_ID + echo $AWS_SECRET_ACCESS_KEY + ``` + +2. Test connectivity: + ```bash + curl $AWS_ENDPOINT_URL + ``` + +3. Verify endpoint URL for MinIO/B2 + +### Permission Errors + +**Problem:** Access denied + +```bash +Error: failed to upload to S3: AccessDenied +``` + +**Solution:** +1. Check IAM policy includes required permissions +2. Verify bucket name is correct +3. Check bucket policy allows your IAM user + +### Upload Failures + +**Problem:** Large file upload fails + +```bash +Error: multipart upload failed: connection timeout +``` + +**Solution:** +1. Check network stability +2. Retry - multipart uploads resume automatically +3. Increase timeout in config +4. Check firewall allows outbound HTTPS + +### Verification Failures + +**Problem:** Checksum mismatch + +```bash +Error: checksum mismatch: expected abc123, got def456 +``` + +**Solution:** +1. Re-download the backup +2. Check if file was corrupted during upload +3. Verify original backup integrity locally +4. Re-upload if necessary + +--- + +## Examples + +### Full Backup Workflow + +```bash +#!/bin/bash +# Daily backup to S3 with retention + +# Backup all databases +for db in db1 db2 db3; do + dbbackup backup single $db \ + --cloud s3://production-backups/daily/$db/ \ + --compression gzip +done + +# Cleanup old backups (keep 30 days, min 10 backups) +dbbackup cleanup s3://production-backups/daily/ \ + --retention-days 30 \ + --min-backups 10 + +# Verify today's backups +dbbackup verify-backup s3://production-backups/daily/*/$(date +%Y%m%d)*.dump +``` + +### Disaster Recovery + +```bash +#!/bin/bash +# Restore from cloud backup + +# List available backups +dbbackup cloud list \ + --cloud-provider s3 \ + --cloud-bucket disaster-recovery \ + --verbose + +# Restore latest backup +LATEST=$(dbbackup cloud list \ + --cloud-provider s3 \ + --cloud-bucket disaster-recovery | tail -1) + +dbbackup restore single "s3://disaster-recovery/$LATEST" \ + --target restored_db \ + --create \ + --confirm +``` + +### Multi-Cloud Strategy + +```bash +#!/bin/bash +# Backup to both AWS S3 and Backblaze B2 + +# Backup to S3 +dbbackup backup single production_db \ + --cloud s3://aws-backups/prod/ \ + --output-dir /tmp/backups + +# Also upload to B2 +BACKUP_FILE=$(ls -t /tmp/backups/*.dump | head -1) +dbbackup cloud upload "$BACKUP_FILE" \ + --cloud-provider b2 \ + --cloud-bucket b2-offsite-backups \ + --cloud-endpoint https://s3.us-west-002.backblazeb2.com + +# Verify both locations +dbbackup verify-backup s3://aws-backups/prod/$(basename $BACKUP_FILE) +dbbackup verify-backup b2://b2-offsite-backups/$(basename $BACKUP_FILE) +``` + +--- + +## FAQ + +**Q: Can I use dbbackup with my existing S3 buckets?** +A: Yes! Just specify your bucket name and credentials. + +**Q: Do I need to keep local backups?** +A: No, use `--cloud` flag to upload directly without keeping local copies. + +**Q: What happens if upload fails?** +A: Backup succeeds locally. Upload failure is logged but doesn't fail the backup. + +**Q: Can I restore without downloading?** +A: No, backups are downloaded to temp directory, then restored and cleaned up. + +**Q: How much does cloud storage cost?** +A: Varies by provider: +- AWS S3: ~$0.023/GB/month + transfer +- Backblaze B2: ~$0.005/GB/month + transfer +- MinIO: Self-hosted, hardware costs only + +**Q: Can I use multiple cloud providers?** +A: Yes! Use different URIs or upload to multiple destinations. + +**Q: Is multipart upload automatic?** +A: Yes, automatically used for files >100MB. + +**Q: Can I use S3 Glacier?** +A: Yes, but restore requires thawing. Use lifecycle policies for automatic archival. + +--- + +## Related Documentation + +- [README.md](README.md) - Main documentation +- [ROADMAP.md](ROADMAP.md) - Feature roadmap +- [docker-compose.minio.yml](docker-compose.minio.yml) - MinIO test setup +- [scripts/test_cloud_storage.sh](scripts/test_cloud_storage.sh) - Integration tests + +--- + +## Support + +For issues or questions: +- GitHub Issues: [Create an issue](https://github.com/yourusername/dbbackup/issues) +- Documentation: Check README.md and inline help +- Examples: See `scripts/test_cloud_storage.sh` diff --git a/docker-compose.minio.yml b/docker-compose.minio.yml new file mode 100644 index 0000000..169e054 --- /dev/null +++ b/docker-compose.minio.yml @@ -0,0 +1,101 @@ +version: '3.8' + +services: + # MinIO S3-compatible object storage for testing + minio: + image: minio/minio:latest + container_name: dbbackup-minio + ports: + - "9000:9000" # S3 API + - "9001:9001" # Web Console + environment: + MINIO_ROOT_USER: minioadmin + MINIO_ROOT_PASSWORD: minioadmin123 + MINIO_REGION: us-east-1 + volumes: + - minio-data:/data + command: server /data --console-address ":9001" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + interval: 30s + timeout: 20s + retries: 3 + networks: + - dbbackup-test + + # PostgreSQL database for backup testing + postgres: + image: postgres:16-alpine + container_name: dbbackup-postgres-test + environment: + POSTGRES_USER: testuser + POSTGRES_PASSWORD: testpass123 + POSTGRES_DB: testdb + POSTGRES_INITDB_ARGS: "-E UTF8 --locale=C" + ports: + - "5433:5432" + volumes: + - postgres-data:/var/lib/postgresql/data + - ./test_data:/docker-entrypoint-initdb.d + healthcheck: + test: ["CMD-SHELL", "pg_isready -U testuser"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - dbbackup-test + + # MySQL database for backup testing + mysql: + image: mysql:8.0 + container_name: dbbackup-mysql-test + environment: + MYSQL_ROOT_PASSWORD: rootpass123 + MYSQL_DATABASE: testdb + MYSQL_USER: testuser + MYSQL_PASSWORD: testpass123 + ports: + - "3307:3306" + volumes: + - mysql-data:/var/lib/mysql + - ./test_data:/docker-entrypoint-initdb.d + command: --default-authentication-plugin=mysql_native_password + healthcheck: + test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-prootpass123"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - dbbackup-test + + # MinIO Client (mc) for bucket management + minio-mc: + image: minio/mc:latest + container_name: dbbackup-minio-mc + depends_on: + minio: + condition: service_healthy + entrypoint: > + /bin/sh -c " + sleep 5; + /usr/bin/mc alias set myminio http://minio:9000 minioadmin minioadmin123; + /usr/bin/mc mb --ignore-existing myminio/test-backups; + /usr/bin/mc mb --ignore-existing myminio/production-backups; + /usr/bin/mc mb --ignore-existing myminio/dev-backups; + echo 'MinIO buckets created successfully'; + exit 0; + " + networks: + - dbbackup-test + +volumes: + minio-data: + driver: local + postgres-data: + driver: local + mysql-data: + driver: local + +networks: + dbbackup-test: + driver: bridge diff --git a/go.mod b/go.mod index ef77cc5..9bb07d6 100755 --- a/go.mod +++ b/go.mod @@ -20,9 +20,10 @@ require ( filippo.io/edwards25519 v1.1.0 // indirect github.com/aws/aws-sdk-go-v2 v1.40.0 // indirect github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.3 // indirect - github.com/aws/aws-sdk-go-v2/config v1.32.1 // indirect - github.com/aws/aws-sdk-go-v2/credentials v1.19.1 // indirect + github.com/aws/aws-sdk-go-v2/config v1.32.2 // indirect + github.com/aws/aws-sdk-go-v2/credentials v1.19.2 // indirect github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.14 // indirect + github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.12 // indirect github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.14 // indirect github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.14 // indirect github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect @@ -31,11 +32,11 @@ require ( github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.5 // indirect github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.14 // indirect github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.14 // indirect - github.com/aws/aws-sdk-go-v2/service/s3 v1.92.0 // indirect - github.com/aws/aws-sdk-go-v2/service/signin v1.0.1 // indirect - github.com/aws/aws-sdk-go-v2/service/sso v1.30.4 // indirect - github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.9 // indirect - github.com/aws/aws-sdk-go-v2/service/sts v1.41.1 // indirect + github.com/aws/aws-sdk-go-v2/service/s3 v1.92.1 // indirect + github.com/aws/aws-sdk-go-v2/service/signin v1.0.2 // indirect + github.com/aws/aws-sdk-go-v2/service/sso v1.30.5 // indirect + github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.10 // indirect + github.com/aws/aws-sdk-go-v2/service/sts v1.41.2 // indirect github.com/aws/smithy-go v1.23.2 // indirect github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect github.com/charmbracelet/colorprofile v0.2.3-0.20250311203215-f60798e515dc // indirect diff --git a/go.sum b/go.sum index f29abbd..0519d5d 100755 --- a/go.sum +++ b/go.sum @@ -8,10 +8,16 @@ github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.3 h1:DHctwEM8P8iTXFxC github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.3/go.mod h1:xdCzcZEtnSTKVDOmUZs4l/j3pSV6rpo1WXl5ugNsL8Y= github.com/aws/aws-sdk-go-v2/config v1.32.1 h1:iODUDLgk3q8/flEC7ymhmxjfoAnBDwEEYEVyKZ9mzjU= github.com/aws/aws-sdk-go-v2/config v1.32.1/go.mod h1:xoAgo17AGrPpJBSLg81W+ikM0cpOZG8ad04T2r+d5P0= +github.com/aws/aws-sdk-go-v2/config v1.32.2 h1:4liUsdEpUUPZs5WVapsJLx5NPmQhQdez7nYFcovrytk= +github.com/aws/aws-sdk-go-v2/config v1.32.2/go.mod h1:l0hs06IFz1eCT+jTacU/qZtC33nvcnLADAPL/XyrkZI= github.com/aws/aws-sdk-go-v2/credentials v1.19.1 h1:JeW+EwmtTE0yXFK8SmklrFh/cGTTXsQJumgMZNlbxfM= github.com/aws/aws-sdk-go-v2/credentials v1.19.1/go.mod h1:BOoXiStwTF+fT2XufhO0Efssbi1CNIO/ZXpZu87N0pw= +github.com/aws/aws-sdk-go-v2/credentials v1.19.2 h1:qZry8VUyTK4VIo5aEdUcBjPZHL2v4FyQ3QEOaWcFLu4= +github.com/aws/aws-sdk-go-v2/credentials v1.19.2/go.mod h1:YUqm5a1/kBnoK+/NY5WEiMocZihKSo15/tJdmdXnM5g= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.14 h1:WZVR5DbDgxzA0BJeudId89Kmgy6DIU4ORpxwsVHz0qA= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.14/go.mod h1:Dadl9QO0kHgbrH1GRqGiZdYtW5w+IXXaBNCHTIaheM4= +github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.12 h1:Zy6Tme1AA13kX8x3CnkHx5cqdGWGaj/anwOiWGnA0Xo= +github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.12/go.mod h1:ql4uXYKoTM9WUAUSmthY4AtPVrlTBZOvnBJTiCUdPxI= github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.14 h1:PZHqQACxYb8mYgms4RZbhZG0a7dPW06xOjmaH0EJC/I= github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.14/go.mod h1:VymhrMJUWs69D8u0/lZ7jSB6WgaG/NqHi3gX0aYf6U0= github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.14 h1:bOS19y6zlJwagBfHxs0ESzr1XCOU2KXJCWcq3E2vfjY= @@ -30,14 +36,24 @@ github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.14 h1:FzQE21lNtUor0 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.14/go.mod h1:s1ydyWG9pm3ZwmmYN21HKyG9WzAZhYVW85wMHs5FV6w= github.com/aws/aws-sdk-go-v2/service/s3 v1.92.0 h1:8FshVvnV2sr9kOSAbOnc/vwVmmAwMjOedKH6JW2ddPM= github.com/aws/aws-sdk-go-v2/service/s3 v1.92.0/go.mod h1:wYNqY3L02Z3IgRYxOBPH9I1zD9Cjh9hI5QOy/eOjQvw= +github.com/aws/aws-sdk-go-v2/service/s3 v1.92.1 h1:OgQy/+0+Kc3khtqiEOk23xQAglXi3Tj0y5doOxbi5tg= +github.com/aws/aws-sdk-go-v2/service/s3 v1.92.1/go.mod h1:wYNqY3L02Z3IgRYxOBPH9I1zD9Cjh9hI5QOy/eOjQvw= github.com/aws/aws-sdk-go-v2/service/signin v1.0.1 h1:BDgIUYGEo5TkayOWv/oBLPphWwNm/A91AebUjAu5L5g= github.com/aws/aws-sdk-go-v2/service/signin v1.0.1/go.mod h1:iS6EPmNeqCsGo+xQmXv0jIMjyYtQfnwg36zl2FwEouk= +github.com/aws/aws-sdk-go-v2/service/signin v1.0.2 h1:MxMBdKTYBjPQChlJhi4qlEueqB1p1KcbTEa7tD5aqPs= +github.com/aws/aws-sdk-go-v2/service/signin v1.0.2/go.mod h1:iS6EPmNeqCsGo+xQmXv0jIMjyYtQfnwg36zl2FwEouk= github.com/aws/aws-sdk-go-v2/service/sso v1.30.4 h1:U//SlnkE1wOQiIImxzdY5PXat4Wq+8rlfVEw4Y7J8as= github.com/aws/aws-sdk-go-v2/service/sso v1.30.4/go.mod h1:av+ArJpoYf3pgyrj6tcehSFW+y9/QvAY8kMooR9bZCw= +github.com/aws/aws-sdk-go-v2/service/sso v1.30.5 h1:ksUT5KtgpZd3SAiFJNJ0AFEJVva3gjBmN7eXUZjzUwQ= +github.com/aws/aws-sdk-go-v2/service/sso v1.30.5/go.mod h1:av+ArJpoYf3pgyrj6tcehSFW+y9/QvAY8kMooR9bZCw= github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.9 h1:LU8S9W/mPDAU9q0FjCLi0TrCheLMGwzbRpvUMwYspcA= github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.9/go.mod h1:/j67Z5XBVDx8nZVp9EuFM9/BS5dvBznbqILGuu73hug= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.10 h1:GtsxyiF3Nd3JahRBJbxLCCdYW9ltGQYrFWg8XdkGDd8= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.10/go.mod h1:/j67Z5XBVDx8nZVp9EuFM9/BS5dvBznbqILGuu73hug= github.com/aws/aws-sdk-go-v2/service/sts v1.41.1 h1:GdGmKtG+/Krag7VfyOXV17xjTCz0i9NT+JnqLTOI5nA= github.com/aws/aws-sdk-go-v2/service/sts v1.41.1/go.mod h1:6TxbXoDSgBQ225Qd8Q+MbxUxUh6TtNKwbRt/EPS9xso= +github.com/aws/aws-sdk-go-v2/service/sts v1.41.2 h1:a5UTtD4mHBU3t0o6aHQZFJTNKVfxFWfPX7J0Lr7G+uY= +github.com/aws/aws-sdk-go-v2/service/sts v1.41.2/go.mod h1:6TxbXoDSgBQ225Qd8Q+MbxUxUh6TtNKwbRt/EPS9xso= github.com/aws/smithy-go v1.23.2 h1:Crv0eatJUQhaManss33hS5r40CG3ZFH+21XSkqMrIUM= github.com/aws/smithy-go v1.23.2/go.mod h1:LEj2LM3rBRQJxPZTB4KuzZkaZYnZPnvgIhb4pu07mx0= github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k= diff --git a/internal/cloud/s3.go b/internal/cloud/s3.go index 52f40ea..b72b340 100644 --- a/internal/cloud/s3.go +++ b/internal/cloud/s3.go @@ -11,6 +11,7 @@ import ( "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/credentials" + "github.com/aws/aws-sdk-go-v2/feature/s3/manager" "github.com/aws/aws-sdk-go-v2/service/s3" ) @@ -92,7 +93,7 @@ func (s *S3Backend) buildKey(filename string) string { return filepath.Join(s.prefix, filename) } -// Upload uploads a file to S3 +// Upload uploads a file to S3 with multipart support for large files func (s *S3Backend) Upload(ctx context.Context, localPath, remotePath string, progress ProgressCallback) error { // Open local file file, err := os.Open(localPath) @@ -108,17 +109,30 @@ func (s *S3Backend) Upload(ctx context.Context, localPath, remotePath string, pr } fileSize := stat.Size() + // Build S3 key + key := s.buildKey(remotePath) + + // Use multipart upload for files larger than 100MB + const multipartThreshold = 100 * 1024 * 1024 // 100 MB + + if fileSize > multipartThreshold { + return s.uploadMultipart(ctx, file, key, fileSize, progress) + } + + // Simple upload for smaller files + return s.uploadSimple(ctx, file, key, fileSize, progress) +} + +// uploadSimple performs a simple single-part upload +func (s *S3Backend) uploadSimple(ctx context.Context, file *os.File, key string, fileSize int64, progress ProgressCallback) error { // Create progress reader var reader io.Reader = file if progress != nil { reader = NewProgressReader(file, fileSize, progress) } - // Build S3 key - key := s.buildKey(remotePath) - // Upload to S3 - _, err = s.client.PutObject(ctx, &s3.PutObjectInput{ + _, err := s.client.PutObject(ctx, &s3.PutObjectInput{ Bucket: aws.String(s.bucket), Key: aws.String(key), Body: reader, @@ -131,6 +145,40 @@ func (s *S3Backend) Upload(ctx context.Context, localPath, remotePath string, pr return nil } +// uploadMultipart performs a multipart upload for large files +func (s *S3Backend) uploadMultipart(ctx context.Context, file *os.File, key string, fileSize int64, progress ProgressCallback) error { + // Create uploader with custom options + uploader := manager.NewUploader(s.client, func(u *manager.Uploader) { + // Part size: 10MB + u.PartSize = 10 * 1024 * 1024 + + // Upload up to 10 parts concurrently + u.Concurrency = 10 + + // Leave parts on failure for debugging + u.LeavePartsOnError = false + }) + + // Wrap file with progress reader + var reader io.Reader = file + if progress != nil { + reader = NewProgressReader(file, fileSize, progress) + } + + // Upload with multipart + _, err := uploader.Upload(ctx, &s3.PutObjectInput{ + Bucket: aws.String(s.bucket), + Key: aws.String(key), + Body: reader, + }) + + if err != nil { + return fmt.Errorf("multipart upload failed: %w", err) + } + + return nil +} + // Download downloads a file from S3 func (s *S3Backend) Download(ctx context.Context, remotePath, localPath string, progress ProgressCallback) error { // Build S3 key diff --git a/scripts/test_cloud_storage.sh b/scripts/test_cloud_storage.sh new file mode 100755 index 0000000..763cd90 --- /dev/null +++ b/scripts/test_cloud_storage.sh @@ -0,0 +1,253 @@ +#!/bin/bash +set -e + +# Color output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +echo -e "${BLUE}========================================${NC}" +echo -e "${BLUE}dbbackup Cloud Storage Integration Test${NC}" +echo -e "${BLUE}========================================${NC}" +echo + +# Configuration +MINIO_ENDPOINT="http://localhost:9000" +MINIO_ACCESS_KEY="minioadmin" +MINIO_SECRET_KEY="minioadmin123" +MINIO_BUCKET="test-backups" +POSTGRES_HOST="localhost" +POSTGRES_PORT="5433" +POSTGRES_USER="testuser" +POSTGRES_PASS="testpass123" +POSTGRES_DB="cloudtest" + +# Export credentials +export AWS_ACCESS_KEY_ID="$MINIO_ACCESS_KEY" +export AWS_SECRET_ACCESS_KEY="$MINIO_SECRET_KEY" +export AWS_ENDPOINT_URL="$MINIO_ENDPOINT" +export AWS_REGION="us-east-1" + +# Check if dbbackup binary exists +if [ ! -f "./dbbackup" ]; then + echo -e "${YELLOW}Building dbbackup...${NC}" + go build -o dbbackup . + echo -e "${GREEN}✓ Build successful${NC}" +fi + +# Function to wait for service +wait_for_service() { + local service=$1 + local host=$2 + local port=$3 + local max_attempts=30 + local attempt=1 + + echo -e "${YELLOW}Waiting for $service to be ready...${NC}" + + while ! nc -z $host $port 2>/dev/null; do + if [ $attempt -ge $max_attempts ]; then + echo -e "${RED}✗ $service did not start in time${NC}" + return 1 + fi + echo -n "." + sleep 1 + attempt=$((attempt + 1)) + done + + echo -e "${GREEN}✓ $service is ready${NC}" +} + +# Step 1: Start services +echo -e "${BLUE}Step 1: Starting services with Docker Compose${NC}" +docker-compose -f docker-compose.minio.yml up -d + +# Wait for services +wait_for_service "MinIO" "localhost" "9000" +wait_for_service "PostgreSQL" "localhost" "5433" + +sleep 5 + +# Step 2: Create test database +echo -e "\n${BLUE}Step 2: Creating test database${NC}" +PGPASSWORD=$POSTGRES_PASS psql -h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER -c "DROP DATABASE IF EXISTS $POSTGRES_DB;" postgres 2>/dev/null || true +PGPASSWORD=$POSTGRES_PASS psql -h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER -c "CREATE DATABASE $POSTGRES_DB;" postgres +PGPASSWORD=$POSTGRES_PASS psql -h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER -d $POSTGRES_DB << EOF +CREATE TABLE users ( + id SERIAL PRIMARY KEY, + name VARCHAR(100), + email VARCHAR(100), + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +INSERT INTO users (name, email) VALUES + ('Alice', 'alice@example.com'), + ('Bob', 'bob@example.com'), + ('Charlie', 'charlie@example.com'); + +CREATE TABLE products ( + id SERIAL PRIMARY KEY, + name VARCHAR(200), + price DECIMAL(10,2) +); + +INSERT INTO products (name, price) VALUES + ('Widget', 19.99), + ('Gadget', 29.99), + ('Doohickey', 39.99); +EOF + +echo -e "${GREEN}✓ Test database created with sample data${NC}" + +# Step 3: Test local backup +echo -e "\n${BLUE}Step 3: Creating local backup${NC}" +./dbbackup backup single $POSTGRES_DB \ + --db-type postgres \ + --host $POSTGRES_HOST \ + --port $POSTGRES_PORT \ + --user $POSTGRES_USER \ + --password $POSTGRES_PASS \ + --output-dir /tmp/dbbackup-test + +LOCAL_BACKUP=$(ls -t /tmp/dbbackup-test/${POSTGRES_DB}_*.dump 2>/dev/null | head -1) +if [ -z "$LOCAL_BACKUP" ]; then + echo -e "${RED}✗ Local backup failed${NC}" + exit 1 +fi +echo -e "${GREEN}✓ Local backup created: $LOCAL_BACKUP${NC}" + +# Step 4: Test cloud upload +echo -e "\n${BLUE}Step 4: Uploading backup to MinIO (S3)${NC}" +./dbbackup cloud upload "$LOCAL_BACKUP" \ + --cloud-provider minio \ + --cloud-bucket $MINIO_BUCKET \ + --cloud-endpoint $MINIO_ENDPOINT + +echo -e "${GREEN}✓ Upload successful${NC}" + +# Step 5: Test cloud list +echo -e "\n${BLUE}Step 5: Listing cloud backups${NC}" +./dbbackup cloud list \ + --cloud-provider minio \ + --cloud-bucket $MINIO_BUCKET \ + --cloud-endpoint $MINIO_ENDPOINT \ + --verbose + +# Step 6: Test backup with cloud URI +echo -e "\n${BLUE}Step 6: Testing backup with cloud URI${NC}" +./dbbackup backup single $POSTGRES_DB \ + --db-type postgres \ + --host $POSTGRES_HOST \ + --port $POSTGRES_PORT \ + --user $POSTGRES_USER \ + --password $POSTGRES_PASS \ + --output-dir /tmp/dbbackup-test \ + --cloud minio://$MINIO_BUCKET/uri-test/ + +echo -e "${GREEN}✓ Backup with cloud URI successful${NC}" + +# Step 7: Drop database for restore test +echo -e "\n${BLUE}Step 7: Dropping database for restore test${NC}" +PGPASSWORD=$POSTGRES_PASS psql -h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER -c "DROP DATABASE $POSTGRES_DB;" postgres + +# Step 8: Test restore from cloud URI +echo -e "\n${BLUE}Step 8: Restoring from cloud URI${NC}" +CLOUD_URI="minio://$MINIO_BUCKET/$(basename $LOCAL_BACKUP)" +./dbbackup restore single "$CLOUD_URI" \ + --target $POSTGRES_DB \ + --create \ + --confirm \ + --db-type postgres \ + --host $POSTGRES_HOST \ + --port $POSTGRES_PORT \ + --user $POSTGRES_USER \ + --password $POSTGRES_PASS + +echo -e "${GREEN}✓ Restore from cloud successful${NC}" + +# Step 9: Verify data +echo -e "\n${BLUE}Step 9: Verifying restored data${NC}" +USER_COUNT=$(PGPASSWORD=$POSTGRES_PASS psql -h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER -d $POSTGRES_DB -t -c "SELECT COUNT(*) FROM users;") +PRODUCT_COUNT=$(PGPASSWORD=$POSTGRES_PASS psql -h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER -d $POSTGRES_DB -t -c "SELECT COUNT(*) FROM products;") + +if [ "$USER_COUNT" -eq 3 ] && [ "$PRODUCT_COUNT" -eq 3 ]; then + echo -e "${GREEN}✓ Data verification successful (users: $USER_COUNT, products: $PRODUCT_COUNT)${NC}" +else + echo -e "${RED}✗ Data verification failed (users: $USER_COUNT, products: $PRODUCT_COUNT)${NC}" + exit 1 +fi + +# Step 10: Test verify command +echo -e "\n${BLUE}Step 10: Verifying cloud backup integrity${NC}" +./dbbackup verify-backup "$CLOUD_URI" + +echo -e "${GREEN}✓ Backup verification successful${NC}" + +# Step 11: Test cloud cleanup +echo -e "\n${BLUE}Step 11: Testing cloud cleanup (dry-run)${NC}" +./dbbackup cleanup "minio://$MINIO_BUCKET/" \ + --retention-days 0 \ + --min-backups 1 \ + --dry-run + +# Step 12: Create multiple backups for cleanup test +echo -e "\n${BLUE}Step 12: Creating multiple backups for cleanup test${NC}" +for i in {1..5}; do + echo "Creating backup $i/5..." + ./dbbackup backup single $POSTGRES_DB \ + --db-type postgres \ + --host $POSTGRES_HOST \ + --port $POSTGRES_PORT \ + --user $POSTGRES_USER \ + --password $POSTGRES_PASS \ + --output-dir /tmp/dbbackup-test \ + --cloud minio://$MINIO_BUCKET/cleanup-test/ + sleep 1 +done + +echo -e "${GREEN}✓ Multiple backups created${NC}" + +# Step 13: Test actual cleanup +echo -e "\n${BLUE}Step 13: Testing cloud cleanup (actual)${NC}" +./dbbackup cleanup "minio://$MINIO_BUCKET/cleanup-test/" \ + --retention-days 0 \ + --min-backups 2 + +echo -e "${GREEN}✓ Cloud cleanup successful${NC}" + +# Step 14: Test large file upload (multipart) +echo -e "\n${BLUE}Step 14: Testing large file upload (>100MB for multipart)${NC}" +echo "Creating 150MB test file..." +dd if=/dev/zero of=/tmp/large-test-file.bin bs=1M count=150 2>/dev/null + +echo "Uploading large file..." +./dbbackup cloud upload /tmp/large-test-file.bin \ + --cloud-provider minio \ + --cloud-bucket $MINIO_BUCKET \ + --cloud-endpoint $MINIO_ENDPOINT \ + --verbose + +echo -e "${GREEN}✓ Large file multipart upload successful${NC}" + +# Cleanup +echo -e "\n${BLUE}Cleanup${NC}" +rm -f /tmp/large-test-file.bin +rm -rf /tmp/dbbackup-test + +echo -e "\n${GREEN}========================================${NC}" +echo -e "${GREEN}✓ ALL TESTS PASSED!${NC}" +echo -e "${GREEN}========================================${NC}" +echo +echo -e "${YELLOW}To stop services:${NC}" +echo -e " docker-compose -f docker-compose.minio.yml down" +echo +echo -e "${YELLOW}To view MinIO console:${NC}" +echo -e " http://localhost:9001 (minioadmin / minioadmin123)" +echo +echo -e "${YELLOW}To keep services running for manual testing:${NC}" +echo -e " export AWS_ACCESS_KEY_ID=minioadmin" +echo -e " export AWS_SECRET_ACCESS_KEY=minioadmin123" +echo -e " export AWS_ENDPOINT_URL=http://localhost:9000" +echo -e " ./dbbackup cloud list --cloud-provider minio --cloud-bucket test-backups"