feat: Add enterprise DBA features for production reliability
New features implemented: 1. Backup Catalog (internal/catalog/) - SQLite-based backup tracking - Gap detection and RPO monitoring - Search and statistics - Filesystem sync 2. DR Drill Testing (internal/drill/) - Automated restore testing in Docker containers - Database validation with custom queries - Catalog integration for drill-tested status 3. Smart Notifications (internal/notify/) - Event batching with configurable intervals - Time-based escalation policies - HTML/text/Slack templates 4. Compliance Reports (internal/report/) - SOC2, GDPR, HIPAA, PCI-DSS, ISO27001 frameworks - Evidence collection from catalog - JSON, Markdown, HTML output formats 5. RTO/RPO Calculator (internal/rto/) - Recovery objective analysis - RTO breakdown by phase - Recommendations for improvement 6. Replica-Aware Backup (internal/replica/) - Topology detection for PostgreSQL/MySQL - Automatic replica selection - Configurable selection strategies 7. Parallel Table Backup (internal/parallel/) - Concurrent table dumps - Worker pool with progress tracking - Large table optimization 8. MySQL/MariaDB PITR (internal/pitr/) - Binary log parsing and replay - Point-in-time recovery support - Transaction filtering CLI commands added: catalog, drill, report, rto All changes support the goal: reliable 3 AM database recovery.
This commit is contained in:
401
MYSQL_PITR.md
Normal file
401
MYSQL_PITR.md
Normal file
@@ -0,0 +1,401 @@
|
||||
# MySQL/MariaDB Point-in-Time Recovery (PITR)
|
||||
|
||||
This guide explains how to use dbbackup for Point-in-Time Recovery with MySQL and MariaDB databases.
|
||||
|
||||
## Overview
|
||||
|
||||
Point-in-Time Recovery (PITR) allows you to restore your database to any specific moment in time, not just to when a backup was taken. This is essential for:
|
||||
|
||||
- Recovering from accidental data deletion or corruption
|
||||
- Restoring to a state just before a problematic change
|
||||
- Meeting regulatory compliance requirements for data recovery
|
||||
|
||||
### How MySQL PITR Works
|
||||
|
||||
MySQL PITR uses binary logs (binlogs) which record all changes to the database:
|
||||
|
||||
1. **Base Backup**: A full database backup with the binlog position recorded
|
||||
2. **Binary Log Archiving**: Continuous archiving of binlog files
|
||||
3. **Recovery**: Restore base backup, then replay binlogs up to the target time
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ Base Backup │ --> │ binlog.00001 │ --> │ binlog.00002 │ --> │ binlog.00003 │
|
||||
│ (pos: 1234) │ │ │ │ │ │ (current) │
|
||||
└─────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
|
||||
│ │ │ │
|
||||
▼ ▼ ▼ ▼
|
||||
10:00 AM 10:30 AM 11:00 AM 11:30 AM
|
||||
↑
|
||||
Target: 11:15 AM
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### MySQL Configuration
|
||||
|
||||
Binary logging must be enabled in MySQL. Add to `my.cnf`:
|
||||
|
||||
```ini
|
||||
[mysqld]
|
||||
# Enable binary logging
|
||||
log_bin = mysql-bin
|
||||
server_id = 1
|
||||
|
||||
# Recommended: Use ROW format for PITR
|
||||
binlog_format = ROW
|
||||
|
||||
# Optional but recommended: Enable GTID for easier replication and recovery
|
||||
gtid_mode = ON
|
||||
enforce_gtid_consistency = ON
|
||||
|
||||
# Keep binlogs for at least 7 days (adjust as needed)
|
||||
expire_logs_days = 7
|
||||
# Or for MySQL 8.0+:
|
||||
# binlog_expire_logs_seconds = 604800
|
||||
```
|
||||
|
||||
After changing configuration, restart MySQL:
|
||||
```bash
|
||||
sudo systemctl restart mysql
|
||||
```
|
||||
|
||||
### MariaDB Configuration
|
||||
|
||||
MariaDB configuration is similar:
|
||||
|
||||
```ini
|
||||
[mysqld]
|
||||
log_bin = mariadb-bin
|
||||
server_id = 1
|
||||
binlog_format = ROW
|
||||
|
||||
# MariaDB uses different GTID implementation (auto-enabled with log_slave_updates)
|
||||
log_slave_updates = ON
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Check PITR Status
|
||||
|
||||
```bash
|
||||
# Check if MySQL is properly configured for PITR
|
||||
dbbackup pitr mysql-status
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
MySQL/MariaDB PITR Status (mysql)
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
PITR Status: ❌ NOT CONFIGURED
|
||||
Binary Logging: ✅ ENABLED
|
||||
Binlog Format: ROW
|
||||
GTID Mode: ON
|
||||
Current Position: mysql-bin.000042:1234
|
||||
|
||||
PITR Requirements:
|
||||
✅ Binary logging enabled
|
||||
✅ Row-based logging (recommended)
|
||||
```
|
||||
|
||||
### 2. Enable PITR
|
||||
|
||||
```bash
|
||||
# Enable PITR and configure archive directory
|
||||
dbbackup pitr mysql-enable --archive-dir /backups/binlog_archive
|
||||
```
|
||||
|
||||
### 3. Create a Base Backup
|
||||
|
||||
```bash
|
||||
# Create a PITR-capable backup
|
||||
dbbackup backup single mydb --pitr
|
||||
```
|
||||
|
||||
### 4. Start Binlog Archiving
|
||||
|
||||
```bash
|
||||
# Run binlog archiver in the background
|
||||
dbbackup binlog watch --binlog-dir /var/lib/mysql --archive-dir /backups/binlog_archive --interval 30s
|
||||
```
|
||||
|
||||
Or set up a cron job for periodic archiving:
|
||||
```bash
|
||||
# Archive new binlogs every 5 minutes
|
||||
*/5 * * * * dbbackup binlog archive --binlog-dir /var/lib/mysql --archive-dir /backups/binlog_archive
|
||||
```
|
||||
|
||||
### 5. Restore to Point in Time
|
||||
|
||||
```bash
|
||||
# Restore to a specific time
|
||||
dbbackup restore pitr mydb_backup.sql.gz --target-time '2024-01-15 14:30:00'
|
||||
```
|
||||
|
||||
## Commands Reference
|
||||
|
||||
### PITR Commands
|
||||
|
||||
#### `pitr mysql-status`
|
||||
Show MySQL/MariaDB PITR configuration and status.
|
||||
|
||||
```bash
|
||||
dbbackup pitr mysql-status
|
||||
```
|
||||
|
||||
#### `pitr mysql-enable`
|
||||
Enable PITR for MySQL/MariaDB.
|
||||
|
||||
```bash
|
||||
dbbackup pitr mysql-enable \
|
||||
--archive-dir /backups/binlog_archive \
|
||||
--retention-days 7 \
|
||||
--require-row-format \
|
||||
--require-gtid
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--archive-dir`: Directory to store archived binlogs (required)
|
||||
- `--retention-days`: Days to keep archived binlogs (default: 7)
|
||||
- `--require-row-format`: Require ROW binlog format (default: true)
|
||||
- `--require-gtid`: Require GTID mode enabled (default: false)
|
||||
|
||||
### Binlog Commands
|
||||
|
||||
#### `binlog list`
|
||||
List available binary log files.
|
||||
|
||||
```bash
|
||||
# List binlogs from MySQL data directory
|
||||
dbbackup binlog list --binlog-dir /var/lib/mysql
|
||||
|
||||
# List archived binlogs
|
||||
dbbackup binlog list --archive-dir /backups/binlog_archive
|
||||
```
|
||||
|
||||
#### `binlog archive`
|
||||
Archive binary log files.
|
||||
|
||||
```bash
|
||||
dbbackup binlog archive \
|
||||
--binlog-dir /var/lib/mysql \
|
||||
--archive-dir /backups/binlog_archive \
|
||||
--compress
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--binlog-dir`: MySQL binary log directory
|
||||
- `--archive-dir`: Destination for archived binlogs (required)
|
||||
- `--compress`: Compress archived binlogs with gzip
|
||||
- `--encrypt`: Encrypt archived binlogs
|
||||
- `--encryption-key-file`: Path to encryption key file
|
||||
|
||||
#### `binlog watch`
|
||||
Continuously monitor and archive new binlog files.
|
||||
|
||||
```bash
|
||||
dbbackup binlog watch \
|
||||
--binlog-dir /var/lib/mysql \
|
||||
--archive-dir /backups/binlog_archive \
|
||||
--interval 30s \
|
||||
--compress
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--interval`: How often to check for new binlogs (default: 30s)
|
||||
|
||||
#### `binlog validate`
|
||||
Validate binlog chain integrity.
|
||||
|
||||
```bash
|
||||
dbbackup binlog validate --binlog-dir /var/lib/mysql
|
||||
```
|
||||
|
||||
Output shows:
|
||||
- Whether the chain is complete (no missing files)
|
||||
- Any gaps in the sequence
|
||||
- Server ID changes (indicating possible failover)
|
||||
- Total size and file count
|
||||
|
||||
#### `binlog position`
|
||||
Show current binary log position.
|
||||
|
||||
```bash
|
||||
dbbackup binlog position
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
Current Binary Log Position
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
File: mysql-bin.000042
|
||||
Position: 123456
|
||||
GTID Set: 3E11FA47-71CA-11E1-9E33-C80AA9429562:1-1000
|
||||
|
||||
Position String: mysql-bin.000042:123456
|
||||
```
|
||||
|
||||
## Restore Scenarios
|
||||
|
||||
### Restore to Specific Time
|
||||
|
||||
```bash
|
||||
# Restore to January 15, 2024 at 2:30 PM
|
||||
dbbackup restore pitr mydb_backup.sql.gz \
|
||||
--target-time '2024-01-15 14:30:00'
|
||||
```
|
||||
|
||||
### Restore to Specific Position
|
||||
|
||||
```bash
|
||||
# Restore to a specific binlog position
|
||||
dbbackup restore pitr mydb_backup.sql.gz \
|
||||
--target-position 'mysql-bin.000042:12345'
|
||||
```
|
||||
|
||||
### Dry Run (Preview)
|
||||
|
||||
```bash
|
||||
# See what SQL would be replayed without applying
|
||||
dbbackup restore pitr mydb_backup.sql.gz \
|
||||
--target-time '2024-01-15 14:30:00' \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
### Restore to Backup Point Only
|
||||
|
||||
```bash
|
||||
# Restore just the base backup without replaying binlogs
|
||||
dbbackup restore pitr mydb_backup.sql.gz --immediate
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Archiving Strategy
|
||||
|
||||
- Archive binlogs frequently (every 5-30 minutes)
|
||||
- Use compression to save disk space
|
||||
- Store archives on separate storage from the database
|
||||
|
||||
### 2. Retention Policy
|
||||
|
||||
- Keep archives for at least as long as your oldest valid base backup
|
||||
- Consider regulatory requirements for data retention
|
||||
- Use the cleanup command to purge old archives:
|
||||
|
||||
```bash
|
||||
dbbackup binlog cleanup --archive-dir /backups/binlog_archive --retention-days 30
|
||||
```
|
||||
|
||||
### 3. Validation
|
||||
|
||||
- Regularly validate your binlog chain:
|
||||
```bash
|
||||
dbbackup binlog validate --binlog-dir /var/lib/mysql
|
||||
```
|
||||
|
||||
- Test restoration periodically on a test environment
|
||||
|
||||
### 4. Monitoring
|
||||
|
||||
- Monitor the `dbbackup binlog watch` process
|
||||
- Set up alerts for:
|
||||
- Binlog archiver failures
|
||||
- Gaps in binlog chain
|
||||
- Low disk space on archive directory
|
||||
|
||||
### 5. GTID Mode
|
||||
|
||||
Enable GTID for:
|
||||
- Easier tracking of replication position
|
||||
- Automatic failover in replication setups
|
||||
- Simpler point-in-time recovery
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Binary Logging Not Enabled
|
||||
|
||||
**Error**: "Binary logging appears to be disabled"
|
||||
|
||||
**Solution**: Add to my.cnf and restart MySQL:
|
||||
```ini
|
||||
[mysqld]
|
||||
log_bin = mysql-bin
|
||||
server_id = 1
|
||||
```
|
||||
|
||||
### Missing Binlog Files
|
||||
|
||||
**Error**: "Gaps detected in binlog chain"
|
||||
|
||||
**Causes**:
|
||||
- `RESET MASTER` was executed
|
||||
- `expire_logs_days` is too short
|
||||
- Binlogs were manually deleted
|
||||
|
||||
**Solution**:
|
||||
- Take a new base backup immediately
|
||||
- Adjust retention settings to prevent future gaps
|
||||
|
||||
### Permission Denied
|
||||
|
||||
**Error**: "Failed to read binlog directory"
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Add dbbackup user to mysql group
|
||||
sudo usermod -aG mysql dbbackup_user
|
||||
|
||||
# Or set appropriate permissions
|
||||
sudo chmod g+r /var/lib/mysql/mysql-bin.*
|
||||
```
|
||||
|
||||
### Wrong Binlog Format
|
||||
|
||||
**Warning**: "binlog_format = STATEMENT (ROW recommended)"
|
||||
|
||||
**Impact**: STATEMENT format may not capture all changes accurately
|
||||
|
||||
**Solution**: Change to ROW format (requires restart):
|
||||
```ini
|
||||
[mysqld]
|
||||
binlog_format = ROW
|
||||
```
|
||||
|
||||
### Server ID Changes
|
||||
|
||||
**Warning**: "server_id changed from X to Y (possible master failover)"
|
||||
|
||||
This warning indicates the binlog chain contains events from different servers, which may happen during:
|
||||
- Failover in a replication setup
|
||||
- Restoring from a different server's backup
|
||||
|
||||
This is usually informational but review your topology if unexpected.
|
||||
|
||||
## MariaDB-Specific Notes
|
||||
|
||||
### GTID Format
|
||||
|
||||
MariaDB uses a different GTID format than MySQL:
|
||||
- **MySQL**: `3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5`
|
||||
- **MariaDB**: `0-1-100` (domain-server_id-sequence)
|
||||
|
||||
### Tool Detection
|
||||
|
||||
dbbackup automatically detects MariaDB and uses:
|
||||
- `mariadb-binlog` if available (MariaDB 10.4+)
|
||||
- Falls back to `mysqlbinlog` for older versions
|
||||
|
||||
### Encrypted Binlogs
|
||||
|
||||
MariaDB supports binlog encryption. If enabled, ensure the key is available during archive and restore operations.
|
||||
|
||||
## See Also
|
||||
|
||||
- [PITR.md](PITR.md) - PostgreSQL PITR documentation
|
||||
- [DOCKER.md](DOCKER.md) - Running in Docker environments
|
||||
- [CLOUD.md](CLOUD.md) - Cloud storage for archives
|
||||
Reference in New Issue
Block a user