UUXO/dbbackup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 28 Wiki Activity

feat: Sprint 4 - Azure Blob Storage and Google Cloud Storage support

Browse Source 
Implemented full native support for Azure Blob Storage and Google Cloud Storage:

**Azure Blob Storage (internal/cloud/azure.go):**
- Native Azure SDK integration (github.com/Azure/azure-sdk-for-go)
- Block blob upload for large files (>256MB with 100MB blocks)
- Azurite emulator support for local testing
- Production Azure authentication (account name + key)
- SHA-256 integrity verification with metadata
- Streaming uploads with progress tracking

**Google Cloud Storage (internal/cloud/gcs.go):**
- Native GCS SDK integration (cloud.google.com/go/storage)
- Chunked upload for large files (16MB chunks)
- fake-gcs-server emulator support for local testing
- Application Default Credentials support
- Service account JSON key file support
- SHA-256 integrity verification with metadata
- Streaming uploads with progress tracking

**Backend Integration:**
- Updated NewBackend() factory to support azure/azblob and gs/gcs providers
- Added Name() methods to both backends
- Fixed ProgressReader usage across all backends
- Updated Config comments to document Azure/GCS support

**Testing Infrastructure:**
- docker-compose.azurite.yml: Azurite + PostgreSQL + MySQL test environment
- docker-compose.gcs.yml: fake-gcs-server + PostgreSQL + MySQL test environment
- scripts/test_azure_storage.sh: 8 comprehensive Azure integration tests
- scripts/test_gcs_storage.sh: 8 comprehensive GCS integration tests
- Both test scripts validate upload/download/verify/cleanup/restore operations

**Documentation:**
- AZURE.md: Complete guide (600+ lines) covering setup, authentication, usage
- GCS.md: Complete guide (600+ lines) covering setup, authentication, usage
- Updated CLOUD.md with Azure and GCS sections
- Updated internal/config/config.go with Azure/GCS field documentation

**Test Coverage:**
- Large file uploads (300MB for Azure, 200MB for GCS)
- Block/chunked upload verification
- Backup verification with SHA-256 checksums
- Restore from cloud URIs
- Cleanup and retention policies
- Emulator support for both providers

**Dependencies Added:**
- Azure: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3
- GCS: cloud.google.com/go/storage v1.57.2
- Plus transitive dependencies (~50+ packages)

**Build:**
- Compiles successfully: 68MB binary
- All imports resolved
- No compilation errors

Sprint 4 closes the multi-cloud gap identified in Sprint 3 evaluation.
Users can now use Azure and GCS URIs that were previously parsed but unsupported.
This commit is contained in:
Renz
2025-11-25 21:31:21 +00:00
parent 8929004abc
commit 64f1458e9a
13 changed files with 2990 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

83
CLOUD.md
View File

@@ -8,7 +8,8 @@ dbbackup v2.0 includes comprehensive cloud storage integration, allowing you to
- AWS S3
- MinIO (self-hosted S3-compatible)
- Backblaze B2
- Google Cloud Storage (via S3 compatibility)
- **Azure Blob Storage** (native support)
- **Google Cloud Storage** (native support)
- Any S3-compatible storage
**Key Features:**
@@ -83,8 +84,8 @@ Cloud URIs follow this format:
- `s3://` - AWS S3 or S3-compatible storage
- `minio://` - MinIO (auto-enables path-style addressing)
- `b2://` - Backblaze B2
- `gs://` or `gcs://` - Google Cloud Storage
- `azure://` - Azure Blob Storage (coming soon)
- `gs://` or `gcs://` - Google Cloud Storage (native support)
- `azure://` or `azblob://` - Azure Blob Storage (native support)
**Examples:**
```bash
@@ -381,26 +382,68 @@ export AWS_REGION="us-west-002"
dbbackup backup single mydb --cloud b2://my-bucket/backups/
```
### Azure Blob Storage
**Native Azure support with comprehensive features:**
See **[AZURE.md](AZURE.md)** for complete documentation.
**Quick Start:**
```bash
# Using account name and key
dbbackup backup postgres \
--host localhost \
--database mydb \
--cloud "azure://container/backups/db.sql?account=myaccount&key=ACCOUNT_KEY"
# With Azurite emulator for testing
dbbackup backup postgres \
--host localhost \
--database mydb \
--cloud "azure://test-backups/db.sql?endpoint=http://localhost:10000"
```
**Features:**
- Native Azure SDK integration
- Block blob upload for large files (>256MB)
- Azurite emulator support for local testing
- SHA-256 integrity verification
- Comprehensive test suite
### Google Cloud Storage
**Prerequisites:**
- GCP account
- GCS bucket with S3 compatibility enabled
- HMAC keys generated
**Native GCS support with full features:**
**Enable S3 Compatibility:**
1. Go to Cloud Storage > Settings > Interoperability
2. Create HMAC keys
See **[GCS.md](GCS.md)** for complete documentation.
**Configuration:**
**Quick Start:**
```bash
export AWS_ACCESS_KEY_ID="<your-hmac-access-id>"
export AWS_SECRET_ACCESS_KEY="<your-hmac-secret>"
export AWS_ENDPOINT_URL="https://storage.googleapis.com"
# Using Application Default Credentials
dbbackup backup postgres \
--host localhost \
--database mydb \
--cloud "gs://mybucket/backups/db.sql"
dbbackup backup single mydb --cloud gs://my-bucket/backups/
# With service account
dbbackup backup postgres \
--host localhost \
--database mydb \
--cloud "gs://mybucket/backups/db.sql?credentials=/path/to/key.json"
# With fake-gcs-server emulator for testing
dbbackup backup postgres \
--host localhost \
--database mydb \
--cloud "gs://test-backups/db.sql?endpoint=http://localhost:4443/storage/v1"
```
**Features:**
- Native GCS SDK integration
- Chunked upload for large files (16MB chunks)
- fake-gcs-server emulator support
- Application Default Credentials support
- Workload Identity for GKE
---
## Features
@@ -727,6 +770,8 @@ A: No, backups are downloaded to temp directory, then restored and cleaned up.
**Q: How much does cloud storage cost?**
A: Varies by provider:
- AWS S3: ~$0.023/GB/month + transfer
- Azure Blob Storage: ~$0.018/GB/month (Hot tier)
- Google Cloud Storage: ~$0.020/GB/month (Standard)
- Backblaze B2: ~$0.005/GB/month + transfer
- MinIO: Self-hosted, hardware costs only
@@ -744,9 +789,15 @@ A: Yes, but restore requires thawing. Use lifecycle policies for automatic archi
## Related Documentation
- [README.md](README.md) - Main documentation
- [AZURE.md](AZURE.md) - **Azure Blob Storage guide** (comprehensive)
- [GCS.md](GCS.md) - **Google Cloud Storage guide** (comprehensive)
- [ROADMAP.md](ROADMAP.md) - Feature roadmap
- [docker-compose.minio.yml](docker-compose.minio.yml) - MinIO test setup
- [scripts/test_cloud_storage.sh](scripts/test_cloud_storage.sh) - Integration tests
- [docker-compose.azurite.yml](docker-compose.azurite.yml) - Azure Azurite test setup
- [docker-compose.gcs.yml](docker-compose.gcs.yml) - GCS fake-gcs-server test setup
- [scripts/test_cloud_storage.sh](scripts/test_cloud_storage.sh) - S3 integration tests
- [scripts/test_azure_storage.sh](scripts/test_azure_storage.sh) - Azure integration tests
- [scripts/test_gcs_storage.sh](scripts/test_gcs_storage.sh) - GCS integration tests
---