Alexander Renz
3887feb12c
- Introduced a comprehensive configuration template (config-adaptive.toml) for adaptive I/O, enabling improved upload/download dual stack with various performance optimizations, security settings, and network resilience features. - Created a test configuration template (test-config.toml) mirroring the adaptive configuration for testing purposes. - Added a simple test configuration (test-simple-config.toml) for basic adaptive features testing with essential parameters. - Included an empty Jupyter notebook (xep0363_analysis.ipynb) for future analysis related to XEP-0363.
392 lines
13 KiB
Markdown
392 lines
13 KiB
Markdown
# Adaptive I/O Integration Guide
|
|
|
|
## Overview
|
|
|
|
This guide explains how to integrate the new adaptive I/O engine into the existing HMAC file server without breaking existing functionality.
|
|
|
|
## Integration Strategy
|
|
|
|
### Phase 1: Add Adaptive Components (Backward Compatible)
|
|
|
|
1. **Add the adaptive I/O file** - Already created as `adaptive_io.go`
|
|
2. **Update main.go imports and initialization**
|
|
3. **Add new configuration options**
|
|
4. **Enable gradual rollout**
|
|
|
|
### Phase 2: Gradual Migration
|
|
|
|
1. **Enable adaptive mode via configuration flag**
|
|
2. **Run both old and new handlers in parallel**
|
|
3. **Monitor performance differences**
|
|
4. **Migrate users progressively**
|
|
|
|
### Phase 3: Full Adoption
|
|
|
|
1. **Default to adaptive mode**
|
|
2. **Maintain fallback options**
|
|
3. **Remove old code paths (optional)**
|
|
|
|
## Implementation Steps
|
|
|
|
### Step 1: Update main.go Initialization
|
|
|
|
Add to the main function in `cmd/server/main.go`:
|
|
|
|
```go
|
|
// Add after existing initialization, before starting the server
|
|
if conf.Performance.AdaptiveBuffers {
|
|
initStreamingEngine()
|
|
log.Info("Adaptive I/O engine enabled")
|
|
}
|
|
|
|
// Initialize multi-interface support if enabled
|
|
if conf.NetworkResilience.MultiInterfaceEnabled {
|
|
log.Info("Multi-interface network switching enabled")
|
|
}
|
|
```
|
|
|
|
### Step 2: Update Configuration Structure
|
|
|
|
Add to the configuration structures in `main.go`:
|
|
|
|
```go
|
|
// Add new configuration sections
|
|
type PerformanceConfig struct {
|
|
AdaptiveBuffers bool `toml:"adaptive_buffers" mapstructure:"adaptive_buffers"`
|
|
MinBufferSize string `toml:"min_buffer_size" mapstructure:"min_buffer_size"`
|
|
MaxBufferSize string `toml:"max_buffer_size" mapstructure:"max_buffer_size"`
|
|
BufferOptimizationInterval string `toml:"buffer_optimization_interval" mapstructure:"buffer_optimization_interval"`
|
|
InitialBufferSize string `toml:"initial_buffer_size" mapstructure:"initial_buffer_size"`
|
|
ClientProfiling bool `toml:"client_profiling" mapstructure:"client_profiling"`
|
|
ConnectionTypeDetection bool `toml:"connection_type_detection" mapstructure:"connection_type_detection"`
|
|
PerformanceHistorySamples int `toml:"performance_history_samples" mapstructure:"performance_history_samples"`
|
|
}
|
|
|
|
type ClientOptimizationConfig struct {
|
|
Enabled bool `toml:"enabled" mapstructure:"enabled"`
|
|
LearningEnabled bool `toml:"learning_enabled" mapstructure:"learning_enabled"`
|
|
AdaptationSpeed string `toml:"adaptation_speed" mapstructure:"adaptation_speed"`
|
|
UserAgentAnalysis bool `toml:"user_agent_analysis" mapstructure:"user_agent_analysis"`
|
|
ConnectionFingerprinting bool `toml:"connection_fingerprinting" mapstructure:"connection_fingerprinting"`
|
|
PerformanceClassification bool `toml:"performance_classification" mapstructure:"performance_classification"`
|
|
StrategyMobile ClientOptimizationStrategy `toml:"strategy_mobile" mapstructure:"strategy_mobile"`
|
|
StrategyDesktop ClientOptimizationStrategy `toml:"strategy_desktop" mapstructure:"strategy_desktop"`
|
|
StrategyServer ClientOptimizationStrategy `toml:"strategy_server" mapstructure:"strategy_server"`
|
|
}
|
|
|
|
type ClientOptimizationStrategy struct {
|
|
BufferSize string `toml:"buffer_size" mapstructure:"buffer_size"`
|
|
ChunkSize string `toml:"chunk_size" mapstructure:"chunk_size"`
|
|
RetryMultiplier float64 `toml:"retry_multiplier" mapstructure:"retry_multiplier"`
|
|
TimeoutMultiplier float64 `toml:"timeout_multiplier" mapstructure:"timeout_multiplier"`
|
|
}
|
|
|
|
// Add to main Config struct
|
|
type Config struct {
|
|
Server ServerConfig `toml:"server" mapstructure:"server"`
|
|
Performance PerformanceConfig `toml:"performance" mapstructure:"performance"` // New
|
|
ClientOptimization ClientOptimizationConfig `toml:"client_optimization" mapstructure:"client_optimization"` // New
|
|
NetworkInterfaces NetworkInterfacesConfig `toml:"network_interfaces" mapstructure:"network_interfaces"` // New
|
|
Handoff HandoffConfig `toml:"handoff" mapstructure:"handoff"` // New
|
|
Uploads UploadsConfig `toml:"uploads" mapstructure:"uploads"`
|
|
Downloads DownloadsConfig `toml:"downloads" mapstructure:"downloads"`
|
|
// ... existing fields
|
|
}
|
|
|
|
// Add network interface configuration
|
|
type NetworkInterfacesConfig struct {
|
|
Ethernet NetworkInterfaceSettings `toml:"ethernet" mapstructure:"ethernet"`
|
|
WiFi NetworkInterfaceSettings `toml:"wifi" mapstructure:"wifi"`
|
|
LTE NetworkInterfaceSettings `toml:"lte" mapstructure:"lte"`
|
|
Cellular NetworkInterfaceSettings `toml:"cellular" mapstructure:"cellular"`
|
|
VPN NetworkInterfaceSettings `toml:"vpn" mapstructure:"vpn"`
|
|
}
|
|
|
|
type NetworkInterfaceSettings struct {
|
|
BufferSize string `toml:"buffer_size" mapstructure:"buffer_size"`
|
|
ChunkSize string `toml:"chunk_size" mapstructure:"chunk_size"`
|
|
TimeoutMultiplier float64 `toml:"timeout_multiplier" mapstructure:"timeout_multiplier"`
|
|
Priority int `toml:"priority" mapstructure:"priority"`
|
|
}
|
|
|
|
type HandoffConfig struct {
|
|
SeamlessSwitching bool `toml:"seamless_switching" mapstructure:"seamless_switching"`
|
|
ChunkRetryOnSwitch bool `toml:"chunk_retry_on_switch" mapstructure:"chunk_retry_on_switch"`
|
|
PauseTransfersOnSwitch bool `toml:"pause_transfers_on_switch" mapstructure:"pause_transfers_on_switch"`
|
|
SwitchNotificationEnabled bool `toml:"switch_notification_enabled" mapstructure:"switch_notification_enabled"`
|
|
InterfaceQualityHistory int `toml:"interface_quality_history" mapstructure:"interface_quality_history"`
|
|
PerformanceComparisonWindow string `toml:"performance_comparison_window" mapstructure:"performance_comparison_window"`
|
|
}
|
|
```
|
|
|
|
### Step 3: Add Route Handlers
|
|
|
|
Add new route handlers that can coexist with existing ones:
|
|
|
|
```go
|
|
// Add to the route setup in main.go
|
|
func setupRoutes() {
|
|
// Existing routes
|
|
http.HandleFunc("/upload", handleUpload)
|
|
http.HandleFunc("/download/", handleDownload)
|
|
|
|
// New adaptive routes (optional, for testing)
|
|
if conf.Performance.AdaptiveBuffers {
|
|
http.HandleFunc("/upload/adaptive", handleUploadWithAdaptiveIO)
|
|
http.HandleFunc("/download/adaptive/", handleDownloadWithAdaptiveIO)
|
|
}
|
|
|
|
// Override default handlers if adaptive mode is fully enabled
|
|
if conf.Performance.AdaptiveBuffers && conf.Performance.FullyAdaptive {
|
|
http.HandleFunc("/upload", handleUploadWithAdaptiveIO)
|
|
http.HandleFunc("/download/", handleDownloadWithAdaptiveIO)
|
|
}
|
|
}
|
|
```
|
|
|
|
### Step 4: Update Existing Handlers (Optional Hybrid Approach)
|
|
|
|
Modify existing handlers to use adaptive components when available:
|
|
|
|
```go
|
|
// In the existing handleUpload function, add adaptive streaming option:
|
|
func handleUpload(w http.ResponseWriter, r *http.Request) {
|
|
// ... existing authentication and file handling code ...
|
|
|
|
// Choose I/O method based on configuration
|
|
if conf.Performance.AdaptiveBuffers && globalStreamingEngine != nil {
|
|
// Use adaptive streaming
|
|
clientIP := getClientIP(r)
|
|
sessionID := generateSessionID()
|
|
|
|
written, err := globalStreamingEngine.StreamWithAdaptation(
|
|
dst, file, header.Size, sessionID, clientIP,
|
|
)
|
|
|
|
if err != nil {
|
|
http.Error(w, fmt.Sprintf("Error saving file: %v", err), http.StatusInternalServerError)
|
|
uploadErrorsTotal.Inc()
|
|
os.Remove(absFilename)
|
|
return
|
|
}
|
|
} else {
|
|
// Use traditional buffer pool method
|
|
bufPtr := bufferPool.Get().(*[]byte)
|
|
defer bufferPool.Put(bufPtr)
|
|
buf := *bufPtr
|
|
|
|
written, err := io.CopyBuffer(dst, file, buf)
|
|
if err != nil {
|
|
http.Error(w, fmt.Sprintf("Error saving file: %v", err), http.StatusInternalServerError)
|
|
uploadErrorsTotal.Inc()
|
|
os.Remove(absFilename)
|
|
return
|
|
}
|
|
}
|
|
|
|
// ... rest of existing code ...
|
|
}
|
|
```
|
|
|
|
## Configuration Migration
|
|
|
|
### Gradual Configuration Rollout
|
|
|
|
1. **Start with adaptive buffers disabled**:
|
|
```toml
|
|
[performance]
|
|
adaptive_buffers = false
|
|
```
|
|
|
|
2. **Enable for testing**:
|
|
```toml
|
|
[performance]
|
|
adaptive_buffers = true
|
|
client_profiling = true
|
|
```
|
|
|
|
3. **Full adaptive mode**:
|
|
```toml
|
|
[performance]
|
|
adaptive_buffers = true
|
|
client_profiling = true
|
|
connection_type_detection = true
|
|
fully_adaptive = true
|
|
```
|
|
|
|
### Feature Flags
|
|
|
|
Add feature flags for gradual rollout:
|
|
|
|
```go
|
|
type PerformanceConfig struct {
|
|
AdaptiveBuffers bool `toml:"adaptive_buffers"`
|
|
FullyAdaptive bool `toml:"fully_adaptive"` // Replace default handlers
|
|
AdaptiveUploads bool `toml:"adaptive_uploads"` // Enable adaptive uploads only
|
|
AdaptiveDownloads bool `toml:"adaptive_downloads"` // Enable adaptive downloads only
|
|
TestingMode bool `toml:"testing_mode"` // Parallel testing mode
|
|
}
|
|
```
|
|
|
|
## Testing Strategy
|
|
|
|
### Parallel Testing Mode
|
|
|
|
Enable both old and new handlers for A/B testing:
|
|
|
|
```go
|
|
if conf.Performance.TestingMode {
|
|
// Setup both handlers with different paths
|
|
http.HandleFunc("/upload", handleUpload) // Original
|
|
http.HandleFunc("/upload/adaptive", handleUploadWithAdaptiveIO) // New
|
|
|
|
// Route 50% of traffic to each (example)
|
|
http.HandleFunc("/upload/auto", func(w http.ResponseWriter, r *http.Request) {
|
|
if rand.Intn(2) == 0 {
|
|
handleUpload(w, r)
|
|
} else {
|
|
handleUploadWithAdaptiveIO(w, r)
|
|
}
|
|
})
|
|
}
|
|
```
|
|
|
|
### Performance Comparison
|
|
|
|
Create benchmarking endpoints:
|
|
|
|
```go
|
|
http.HandleFunc("/benchmark/upload/original", benchmarkOriginalUpload)
|
|
http.HandleFunc("/benchmark/upload/adaptive", benchmarkAdaptiveUpload)
|
|
```
|
|
|
|
## Monitoring and Rollback
|
|
|
|
### Enhanced Metrics
|
|
|
|
Add comparative metrics:
|
|
|
|
```go
|
|
var (
|
|
// Original metrics
|
|
uploadDuration = prometheus.NewHistogram(...)
|
|
uploadErrorsTotal = prometheus.NewCounter(...)
|
|
|
|
// Adaptive metrics
|
|
adaptiveUploadDuration = prometheus.NewHistogram(...)
|
|
adaptiveUploadErrorsTotal = prometheus.NewCounter(...)
|
|
adaptiveBufferOptimizations = prometheus.NewCounter(...)
|
|
adaptivePerformanceGains = prometheus.NewHistogram(...)
|
|
)
|
|
```
|
|
|
|
### Rollback Strategy
|
|
|
|
1. **Configuration-based rollback**:
|
|
```toml
|
|
[performance]
|
|
adaptive_buffers = false # Immediate rollback
|
|
```
|
|
|
|
2. **Automatic rollback on high error rates**:
|
|
```go
|
|
func monitorAdaptivePerformance() {
|
|
if adaptiveErrorRate > originalErrorRate * 1.1 {
|
|
log.Warn("Adaptive mode showing higher error rate, reverting to original")
|
|
conf.Performance.AdaptiveBuffers = false
|
|
}
|
|
}
|
|
```
|
|
|
|
## Migration Timeline
|
|
|
|
### Week 1: Infrastructure Setup
|
|
- Add adaptive I/O code
|
|
- Add configuration options
|
|
- Set up monitoring
|
|
|
|
### Week 2: Internal Testing
|
|
- Enable testing mode
|
|
- Run performance comparisons
|
|
- Collect initial metrics
|
|
|
|
### Week 3: Limited Rollout
|
|
- Enable for 10% of traffic
|
|
- Monitor performance and errors
|
|
- Gather feedback
|
|
|
|
### Week 4: Gradual Expansion
|
|
- Increase to 50% of traffic
|
|
- Fine-tune optimization algorithms
|
|
- Address any issues
|
|
|
|
### Week 5: Full Deployment
|
|
- Enable for all traffic
|
|
- Set as default configuration
|
|
- Plan for old code removal
|
|
|
|
## Best Practices
|
|
|
|
### 1. Monitoring
|
|
- Always monitor both performance and error rates
|
|
- Set up alerts for performance degradation
|
|
- Track buffer optimization effectiveness
|
|
|
|
### 2. Configuration
|
|
- Start with conservative settings
|
|
- Enable features gradually
|
|
- Maintain rollback options
|
|
|
|
### 3. Testing
|
|
- Test with various file sizes
|
|
- Test with different network conditions
|
|
- Test with various client types
|
|
|
|
### 4. Documentation
|
|
- Document performance improvements
|
|
- Update user guides
|
|
- Maintain troubleshooting guides
|
|
|
|
## Backward Compatibility
|
|
|
|
The adaptive I/O system is designed to be fully backward compatible:
|
|
|
|
1. **Existing APIs remain unchanged**
|
|
2. **Configuration is additive** (new sections, existing ones unchanged)
|
|
3. **Default behavior is preserved** when adaptive features are disabled
|
|
4. **No changes to client protocols** required
|
|
|
|
## Performance Expectations
|
|
|
|
Based on the adaptive optimizations:
|
|
|
|
- **High-speed networks**: 30-50% throughput improvement
|
|
- **Mobile networks**: 20-30% improvement in reliability
|
|
- **Variable conditions**: Better adaptation to changing network conditions
|
|
- **Memory usage**: Optimized buffer allocation reduces memory pressure
|
|
- **CPU usage**: Minimal overhead from optimization algorithms
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **Higher memory usage**: Adjust `max_buffer_size`
|
|
2. **CPU overhead**: Reduce `buffer_optimization_interval`
|
|
3. **Poor adaptation**: Enable more detailed logging
|
|
4. **Compatibility issues**: Disable specific adaptive features
|
|
|
|
### Debug Configuration
|
|
|
|
```toml
|
|
[logging]
|
|
level = "debug"
|
|
|
|
[performance]
|
|
adaptive_buffers = true
|
|
detailed_logging = true
|
|
optimization_logging = true
|
|
client_profile_logging = true
|
|
```
|
|
|
|
This integration guide ensures a smooth transition to the improved dual stack while maintaining system stability and providing clear rollback options.
|