CRITICAL FIX: Remove --single-transaction and --exit-on-error from pg_restore
- Disabled --single-transaction to prevent lock table exhaustion with large objects - Removed --exit-on-error to allow PostgreSQL to skip ignorable errors - Fixes 'could not open large object' errors (lock exhaustion with 35K+ BLOBs) - Fixes 'already exists' errors causing complete restore failure - Each object now restored in its own transaction (locks released incrementally) - PostgreSQL default behavior (continue on ignorable errors) is correct Per PostgreSQL docs: --single-transaction incompatible with large object restores and causes ALL locks to be held until commit, exhausting lock table with 1000+ objects
This commit is contained in:
165
LARGE_OBJECT_RESTORE_FIX.md
Normal file
165
LARGE_OBJECT_RESTORE_FIX.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# Large Object Restore Fix
|
||||
|
||||
## Problem Analysis
|
||||
|
||||
### Error 1: "type backup_state already exists" (postgres database)
|
||||
**Root Cause**: `--single-transaction` combined with `--exit-on-error` causes entire restore to fail when objects already exist in target database.
|
||||
|
||||
**Why it fails**:
|
||||
- `--single-transaction` wraps restore in BEGIN/COMMIT
|
||||
- `--exit-on-error` aborts on ANY error (including ignorable ones)
|
||||
- "already exists" errors are IGNORABLE - PostgreSQL should continue
|
||||
|
||||
### Error 2: "could not open large object 9646664" + 2.5M errors (resydb database)
|
||||
**Root Cause**: `--single-transaction` takes locks on ALL restored objects simultaneously, exhausting lock table.
|
||||
|
||||
**Why it fails**:
|
||||
- Single transaction locks ALL large objects at once
|
||||
- With 35,000+ large objects, exceeds max_locks_per_transaction
|
||||
- Lock exhaustion → "could not open large object" errors
|
||||
- Cascading failures → millions of errors
|
||||
|
||||
## PostgreSQL Documentation (Verified)
|
||||
|
||||
### From pg_restore docs:
|
||||
> **"pg_restore cannot restore large objects selectively"** - All large objects restored together
|
||||
|
||||
> **"-j / --jobs: Only custom and directory formats supported"**
|
||||
|
||||
> **"multiple jobs cannot be used together with --single-transaction"**
|
||||
|
||||
### From Section 19.5 (Resource Consumption):
|
||||
> **"max_locks_per_transaction × max_connections = total locks"**
|
||||
- Lock table is SHARED across all sessions
|
||||
- Single transaction consuming all locks blocks everything
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. Disabled `--single-transaction` (CRITICAL FIX)
|
||||
**File**: `internal/restore/engine.go`
|
||||
- Line 186: `SingleTransaction: false` (was: true)
|
||||
- Line 210: `SingleTransaction: false` (was: true)
|
||||
|
||||
**Impact**:
|
||||
- No longer wraps entire restore in one transaction
|
||||
- Each object restored in its own transaction
|
||||
- Locks released incrementally (not held until end)
|
||||
- Prevents lock table exhaustion
|
||||
|
||||
### 2. Removed `--exit-on-error` (CRITICAL FIX)
|
||||
**File**: `internal/database/postgresql.go`
|
||||
- Line 375-378: Removed `cmd.append("--exit-on-error")`
|
||||
|
||||
**Impact**:
|
||||
- PostgreSQL continues on ignorable errors (correct behavior)
|
||||
- "already exists" errors logged but don't stop restore
|
||||
- Final error count reported at end
|
||||
- Only real errors cause failure
|
||||
|
||||
### 3. Kept Sequential Parallelism Detection
|
||||
**File**: `internal/restore/engine.go`
|
||||
- Lines 552-565: `detectLargeObjectsInDumps()` still active
|
||||
- Automatically reduces cluster parallelism to 1 when BLOBs detected
|
||||
|
||||
**Impact**:
|
||||
- Prevents multiple databases with large objects from competing for locks
|
||||
- Sequential cluster restore = only one DB's large objects in lock table at a time
|
||||
|
||||
## Why This Works
|
||||
|
||||
### Before (BROKEN):
|
||||
```
|
||||
START TRANSACTION; -- Single transaction begins
|
||||
CREATE TABLE ... -- Lock acquired
|
||||
CREATE INDEX ... -- Lock acquired
|
||||
RESTORE BLOB 1 -- Lock acquired
|
||||
RESTORE BLOB 2 -- Lock acquired
|
||||
...
|
||||
RESTORE BLOB 35000 -- Lock acquired → EXHAUSTED!
|
||||
ERROR: max_locks_per_transaction exceeded
|
||||
ROLLBACK; -- Everything fails
|
||||
```
|
||||
|
||||
### After (FIXED):
|
||||
```
|
||||
BEGIN; CREATE TABLE ...; COMMIT; -- Lock released
|
||||
BEGIN; CREATE INDEX ...; COMMIT; -- Lock released
|
||||
BEGIN; RESTORE BLOB 1; COMMIT; -- Lock released
|
||||
BEGIN; RESTORE BLOB 2; COMMIT; -- Lock released
|
||||
...
|
||||
BEGIN; RESTORE BLOB 35000; COMMIT; -- Each only holds ~100 locks max
|
||||
SUCCESS: All objects restored
|
||||
```
|
||||
|
||||
## Testing Recommendations
|
||||
|
||||
### 1. Test with postgres database (backup_state error)
|
||||
```bash
|
||||
./dbbackup restore cluster /path/to/backup.tar.gz
|
||||
# Should now skip "already exists" errors and continue
|
||||
```
|
||||
|
||||
### 2. Test with resydb database (large objects)
|
||||
```bash
|
||||
# Check dump for large objects first
|
||||
pg_restore -l resydb.dump | grep -i "blob\|large object"
|
||||
|
||||
# Restore should now work without lock exhaustion
|
||||
./dbbackup restore cluster /path/to/backup.tar.gz
|
||||
```
|
||||
|
||||
### 3. Monitor locks during restore
|
||||
```sql
|
||||
-- In another terminal while restore runs:
|
||||
SELECT count(*) FROM pg_locks;
|
||||
-- Should stay well below max_locks_per_transaction × max_connections
|
||||
```
|
||||
|
||||
## Expected Behavior Now
|
||||
|
||||
### For "already exists" errors:
|
||||
```
|
||||
pg_restore: warning: object already exists: TYPE backup_state
|
||||
pg_restore: warning: object already exists: FUNCTION ...
|
||||
... (continues restoring) ...
|
||||
pg_restore: total errors: 10 (all ignorable)
|
||||
SUCCESS
|
||||
```
|
||||
|
||||
### For large objects:
|
||||
```
|
||||
Restoring database resydb...
|
||||
Large objects detected - using sequential restore
|
||||
Restoring 35,000 large objects... (progress)
|
||||
✓ Database resydb restored successfully
|
||||
```
|
||||
|
||||
## Configuration Settings (Still Valid)
|
||||
|
||||
These PostgreSQL settings help but are NO LONGER REQUIRED with the fix:
|
||||
|
||||
```ini
|
||||
# Still recommended for performance, not required for correctness:
|
||||
max_locks_per_transaction = 256 # Provides headroom
|
||||
maintenance_work_mem = 1GB # Faster index creation
|
||||
shared_buffers = 8GB # Better caching
|
||||
```
|
||||
|
||||
## Commit This Fix
|
||||
|
||||
```bash
|
||||
git add internal/restore/engine.go internal/database/postgresql.go
|
||||
git commit -m "CRITICAL FIX: Remove --single-transaction and --exit-on-error from pg_restore
|
||||
|
||||
- Disabled --single-transaction to prevent lock table exhaustion with large objects
|
||||
- Removed --exit-on-error to allow PostgreSQL to skip ignorable errors
|
||||
- Fixes 'could not open large object' errors (lock exhaustion)
|
||||
- Fixes 'already exists' errors causing complete restore failure
|
||||
- Each object now restored in its own transaction (locks released incrementally)
|
||||
- PostgreSQL default behavior (continue on ignorable errors) is correct for restores
|
||||
|
||||
Per PostgreSQL docs: --single-transaction incompatible with large object restores
|
||||
and causes lock table exhaustion with 1000+ objects."
|
||||
|
||||
git push
|
||||
```
|
||||
@@ -371,9 +371,9 @@ func (p *PostgreSQL) BuildRestoreCommand(database, inputFile string, options Res
|
||||
cmd = append(cmd, "--single-transaction")
|
||||
}
|
||||
|
||||
// CRITICAL: Exit on first error (by default pg_restore continues on errors)
|
||||
// This ensures we catch failures immediately instead of at the end
|
||||
cmd = append(cmd, "--exit-on-error")
|
||||
// NOTE: --exit-on-error removed because it causes entire restore to fail on
|
||||
// "already exists" errors. PostgreSQL continues on ignorable errors by default
|
||||
// and reports error count at the end, which is correct behavior for restores.
|
||||
|
||||
// Skip data restore if table creation fails (prevents duplicate data errors)
|
||||
cmd = append(cmd, "--no-data-for-failed-tables")
|
||||
|
||||
@@ -183,8 +183,8 @@ func (e *Engine) restorePostgreSQLDump(ctx context.Context, archivePath, targetD
|
||||
Clean: cleanFirst,
|
||||
NoOwner: true,
|
||||
NoPrivileges: true,
|
||||
SingleTransaction: true,
|
||||
Verbose: true, // Enable verbose for single database restores (not cluster)
|
||||
SingleTransaction: false, // CRITICAL: Disabled to prevent lock exhaustion with large objects
|
||||
Verbose: true, // Enable verbose for single database restores (not cluster)
|
||||
}
|
||||
|
||||
cmd := e.db.BuildRestoreCommand(targetDB, archivePath, opts)
|
||||
@@ -205,7 +205,7 @@ func (e *Engine) restorePostgreSQLDumpWithOwnership(ctx context.Context, archive
|
||||
Clean: false, // We already dropped the database
|
||||
NoOwner: !preserveOwnership, // Preserve ownership if we're superuser
|
||||
NoPrivileges: !preserveOwnership, // Preserve privileges if we're superuser
|
||||
SingleTransaction: true,
|
||||
SingleTransaction: false, // CRITICAL: Disabled to prevent lock exhaustion with large objects
|
||||
Verbose: false, // CRITICAL: disable verbose to prevent OOM on large restores
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user