feat: Sprint 4 - Azure Blob Storage and Google Cloud Storage support
Implemented full native support for Azure Blob Storage and Google Cloud Storage: **Azure Blob Storage (internal/cloud/azure.go):** - Native Azure SDK integration (github.com/Azure/azure-sdk-for-go) - Block blob upload for large files (>256MB with 100MB blocks) - Azurite emulator support for local testing - Production Azure authentication (account name + key) - SHA-256 integrity verification with metadata - Streaming uploads with progress tracking **Google Cloud Storage (internal/cloud/gcs.go):** - Native GCS SDK integration (cloud.google.com/go/storage) - Chunked upload for large files (16MB chunks) - fake-gcs-server emulator support for local testing - Application Default Credentials support - Service account JSON key file support - SHA-256 integrity verification with metadata - Streaming uploads with progress tracking **Backend Integration:** - Updated NewBackend() factory to support azure/azblob and gs/gcs providers - Added Name() methods to both backends - Fixed ProgressReader usage across all backends - Updated Config comments to document Azure/GCS support **Testing Infrastructure:** - docker-compose.azurite.yml: Azurite + PostgreSQL + MySQL test environment - docker-compose.gcs.yml: fake-gcs-server + PostgreSQL + MySQL test environment - scripts/test_azure_storage.sh: 8 comprehensive Azure integration tests - scripts/test_gcs_storage.sh: 8 comprehensive GCS integration tests - Both test scripts validate upload/download/verify/cleanup/restore operations **Documentation:** - AZURE.md: Complete guide (600+ lines) covering setup, authentication, usage - GCS.md: Complete guide (600+ lines) covering setup, authentication, usage - Updated CLOUD.md with Azure and GCS sections - Updated internal/config/config.go with Azure/GCS field documentation **Test Coverage:** - Large file uploads (300MB for Azure, 200MB for GCS) - Block/chunked upload verification - Backup verification with SHA-256 checksums - Restore from cloud URIs - Cleanup and retention policies - Emulator support for both providers **Dependencies Added:** - Azure: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3 - GCS: cloud.google.com/go/storage v1.57.2 - Plus transitive dependencies (~50+ packages) **Build:** - Compiles successfully: 68MB binary - All imports resolved - No compilation errors Sprint 4 closes the multi-cloud gap identified in Sprint 3 evaluation. Users can now use Azure and GCS URIs that were previously parsed but unsupported.
This commit is contained in:
59
docker-compose.gcs.yml
Normal file
59
docker-compose.gcs.yml
Normal file
@@ -0,0 +1,59 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# fake-gcs-server - Google Cloud Storage Emulator
|
||||
gcs-emulator:
|
||||
image: fsouza/fake-gcs-server:latest
|
||||
container_name: dbbackup-gcs
|
||||
ports:
|
||||
- "4443:4443"
|
||||
command: -scheme http -public-host localhost:4443 -external-url http://localhost:4443
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--spider", "-q", "http://localhost:4443/storage/v1/b"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
networks:
|
||||
- dbbackup-net
|
||||
|
||||
# PostgreSQL 16 for testing
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: dbbackup-postgres-gcs
|
||||
environment:
|
||||
POSTGRES_USER: testuser
|
||||
POSTGRES_PASSWORD: testpass
|
||||
POSTGRES_DB: testdb
|
||||
ports:
|
||||
- "5435:5432"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U testuser -d testdb"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
networks:
|
||||
- dbbackup-net
|
||||
|
||||
# MySQL 8.0 for testing
|
||||
mysql:
|
||||
image: mysql:8.0
|
||||
container_name: dbbackup-mysql-gcs
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: rootpass
|
||||
MYSQL_DATABASE: testdb
|
||||
MYSQL_USER: testuser
|
||||
MYSQL_PASSWORD: testpass
|
||||
ports:
|
||||
- "3309:3306"
|
||||
command: --default-authentication-plugin=mysql_native_password
|
||||
healthcheck:
|
||||
test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-prootpass"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
networks:
|
||||
- dbbackup-net
|
||||
|
||||
networks:
|
||||
dbbackup-net:
|
||||
driver: bridge
|
||||
Reference in New Issue
Block a user