feat: Sprint 4 - Azure Blob Storage and Google Cloud Storage support

Implemented full native support for Azure Blob Storage and Google Cloud Storage:

**Azure Blob Storage (internal/cloud/azure.go):**
- Native Azure SDK integration (github.com/Azure/azure-sdk-for-go)
- Block blob upload for large files (>256MB with 100MB blocks)
- Azurite emulator support for local testing
- Production Azure authentication (account name + key)
- SHA-256 integrity verification with metadata
- Streaming uploads with progress tracking

**Google Cloud Storage (internal/cloud/gcs.go):**
- Native GCS SDK integration (cloud.google.com/go/storage)
- Chunked upload for large files (16MB chunks)
- fake-gcs-server emulator support for local testing
- Application Default Credentials support
- Service account JSON key file support
- SHA-256 integrity verification with metadata
- Streaming uploads with progress tracking

**Backend Integration:**
- Updated NewBackend() factory to support azure/azblob and gs/gcs providers
- Added Name() methods to both backends
- Fixed ProgressReader usage across all backends
- Updated Config comments to document Azure/GCS support

**Testing Infrastructure:**
- docker-compose.azurite.yml: Azurite + PostgreSQL + MySQL test environment
- docker-compose.gcs.yml: fake-gcs-server + PostgreSQL + MySQL test environment
- scripts/test_azure_storage.sh: 8 comprehensive Azure integration tests
- scripts/test_gcs_storage.sh: 8 comprehensive GCS integration tests
- Both test scripts validate upload/download/verify/cleanup/restore operations

**Documentation:**
- AZURE.md: Complete guide (600+ lines) covering setup, authentication, usage
- GCS.md: Complete guide (600+ lines) covering setup, authentication, usage
- Updated CLOUD.md with Azure and GCS sections
- Updated internal/config/config.go with Azure/GCS field documentation

**Test Coverage:**
- Large file uploads (300MB for Azure, 200MB for GCS)
- Block/chunked upload verification
- Backup verification with SHA-256 checksums
- Restore from cloud URIs
- Cleanup and retention policies
- Emulator support for both providers

**Dependencies Added:**
- Azure: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3
- GCS: cloud.google.com/go/storage v1.57.2
- Plus transitive dependencies (~50+ packages)

**Build:**
- Compiles successfully: 68MB binary
- All imports resolved
- No compilation errors

Sprint 4 closes the multi-cloud gap identified in Sprint 3 evaluation.
Users can now use Azure and GCS URIs that were previously parsed but unsupported.
This commit is contained in:
2025-11-25 21:31:21 +00:00
parent 8929004abc
commit 64f1458e9a
13 changed files with 2990 additions and 28 deletions

59
docker-compose.gcs.yml Normal file
View File

@@ -0,0 +1,59 @@
version: '3.8'
services:
# fake-gcs-server - Google Cloud Storage Emulator
gcs-emulator:
image: fsouza/fake-gcs-server:latest
container_name: dbbackup-gcs
ports:
- "4443:4443"
command: -scheme http -public-host localhost:4443 -external-url http://localhost:4443
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:4443/storage/v1/b"]
interval: 5s
timeout: 3s
retries: 30
networks:
- dbbackup-net
# PostgreSQL 16 for testing
postgres:
image: postgres:16-alpine
container_name: dbbackup-postgres-gcs
environment:
POSTGRES_USER: testuser
POSTGRES_PASSWORD: testpass
POSTGRES_DB: testdb
ports:
- "5435:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U testuser -d testdb"]
interval: 5s
timeout: 3s
retries: 10
networks:
- dbbackup-net
# MySQL 8.0 for testing
mysql:
image: mysql:8.0
container_name: dbbackup-mysql-gcs
environment:
MYSQL_ROOT_PASSWORD: rootpass
MYSQL_DATABASE: testdb
MYSQL_USER: testuser
MYSQL_PASSWORD: testpass
ports:
- "3309:3306"
command: --default-authentication-plugin=mysql_native_password
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-prootpass"]
interval: 5s
timeout: 3s
retries: 10
networks:
- dbbackup-net
networks:
dbbackup-net:
driver: bridge