hmac-file-server/ADAPTIVE_IO_INTEGRATION.md

Adaptive I/O Integration Guide

Overview

This guide explains how to integrate the new adaptive I/O engine into the existing HMAC file server without breaking existing functionality.

Integration Strategy

Phase 1: Add Adaptive Components (Backward Compatible)

1. Add the adaptive I/O file - Already created as adaptive_io.go
2. Update main.go imports and initialization
3. Add new configuration options
4. Enable gradual rollout

Phase 2: Gradual Migration

1. Enable adaptive mode via configuration flag
2. Run both old and new handlers in parallel
3. Monitor performance differences
4. Migrate users progressively

Phase 3: Full Adoption

1. Default to adaptive mode
2. Maintain fallback options
3. Remove old code paths (optional)

Implementation Steps

Step 1: Update main.go Initialization

Add to the main function in cmd/server/main.go:

// Add after existing initialization, before starting the server
if conf.Performance.AdaptiveBuffers {
    initStreamingEngine()
    log.Info("Adaptive I/O engine enabled")
}

// Initialize multi-interface support if enabled
if conf.NetworkResilience.MultiInterfaceEnabled {
    log.Info("Multi-interface network switching enabled")
}

Step 2: Update Configuration Structure

Add to the configuration structures in main.go:

// Add new configuration sections
type PerformanceConfig struct {
    AdaptiveBuffers            bool   `toml:"adaptive_buffers" mapstructure:"adaptive_buffers"`
    MinBufferSize              string `toml:"min_buffer_size" mapstructure:"min_buffer_size"`
    MaxBufferSize              string `toml:"max_buffer_size" mapstructure:"max_buffer_size"`
    BufferOptimizationInterval string `toml:"buffer_optimization_interval" mapstructure:"buffer_optimization_interval"`
    InitialBufferSize          string `toml:"initial_buffer_size" mapstructure:"initial_buffer_size"`
    ClientProfiling            bool   `toml:"client_profiling" mapstructure:"client_profiling"`
    ConnectionTypeDetection    bool   `toml:"connection_type_detection" mapstructure:"connection_type_detection"`
    PerformanceHistorySamples  int    `toml:"performance_history_samples" mapstructure:"performance_history_samples"`
}

type ClientOptimizationConfig struct {
    Enabled                  bool                      `toml:"enabled" mapstructure:"enabled"`
    LearningEnabled          bool                      `toml:"learning_enabled" mapstructure:"learning_enabled"`
    AdaptationSpeed          string                   `toml:"adaptation_speed" mapstructure:"adaptation_speed"`
    UserAgentAnalysis        bool                      `toml:"user_agent_analysis" mapstructure:"user_agent_analysis"`
    ConnectionFingerprinting bool                      `toml:"connection_fingerprinting" mapstructure:"connection_fingerprinting"`
    PerformanceClassification bool                     `toml:"performance_classification" mapstructure:"performance_classification"`
    StrategyMobile           ClientOptimizationStrategy `toml:"strategy_mobile" mapstructure:"strategy_mobile"`
    StrategyDesktop          ClientOptimizationStrategy `toml:"strategy_desktop" mapstructure:"strategy_desktop"`
    StrategyServer           ClientOptimizationStrategy `toml:"strategy_server" mapstructure:"strategy_server"`
}

type ClientOptimizationStrategy struct {
    BufferSize        string  `toml:"buffer_size" mapstructure:"buffer_size"`
    ChunkSize         string  `toml:"chunk_size" mapstructure:"chunk_size"`
    RetryMultiplier   float64 `toml:"retry_multiplier" mapstructure:"retry_multiplier"`
    TimeoutMultiplier float64 `toml:"timeout_multiplier" mapstructure:"timeout_multiplier"`
}

// Add to main Config struct
type Config struct {
    Server             ServerConfig             `toml:"server" mapstructure:"server"`
    Performance        PerformanceConfig        `toml:"performance" mapstructure:"performance"`        // New
    ClientOptimization ClientOptimizationConfig `toml:"client_optimization" mapstructure:"client_optimization"` // New
    NetworkInterfaces  NetworkInterfacesConfig  `toml:"network_interfaces" mapstructure:"network_interfaces"` // New
    Handoff           HandoffConfig            `toml:"handoff" mapstructure:"handoff"`              // New
    Uploads            UploadsConfig            `toml:"uploads" mapstructure:"uploads"`
    Downloads          DownloadsConfig          `toml:"downloads" mapstructure:"downloads"`
    // ... existing fields
}

// Add network interface configuration
type NetworkInterfacesConfig struct {
    Ethernet NetworkInterfaceSettings `toml:"ethernet" mapstructure:"ethernet"`
    WiFi     NetworkInterfaceSettings `toml:"wifi" mapstructure:"wifi"`
    LTE      NetworkInterfaceSettings `toml:"lte" mapstructure:"lte"`
    Cellular NetworkInterfaceSettings `toml:"cellular" mapstructure:"cellular"`
    VPN      NetworkInterfaceSettings `toml:"vpn" mapstructure:"vpn"`
}

type NetworkInterfaceSettings struct {
    BufferSize        string  `toml:"buffer_size" mapstructure:"buffer_size"`
    ChunkSize         string  `toml:"chunk_size" mapstructure:"chunk_size"`
    TimeoutMultiplier float64 `toml:"timeout_multiplier" mapstructure:"timeout_multiplier"`
    Priority          int     `toml:"priority" mapstructure:"priority"`
}

type HandoffConfig struct {
    SeamlessSwitching           bool   `toml:"seamless_switching" mapstructure:"seamless_switching"`
    ChunkRetryOnSwitch         bool   `toml:"chunk_retry_on_switch" mapstructure:"chunk_retry_on_switch"`
    PauseTransfersOnSwitch     bool   `toml:"pause_transfers_on_switch" mapstructure:"pause_transfers_on_switch"`
    SwitchNotificationEnabled  bool   `toml:"switch_notification_enabled" mapstructure:"switch_notification_enabled"`
    InterfaceQualityHistory    int    `toml:"interface_quality_history" mapstructure:"interface_quality_history"`
    PerformanceComparisonWindow string `toml:"performance_comparison_window" mapstructure:"performance_comparison_window"`
}

Step 3: Add Route Handlers

Add new route handlers that can coexist with existing ones:

// Add to the route setup in main.go
func setupRoutes() {
    // Existing routes
    http.HandleFunc("/upload", handleUpload)
    http.HandleFunc("/download/", handleDownload)
    
    // New adaptive routes (optional, for testing)
    if conf.Performance.AdaptiveBuffers {
        http.HandleFunc("/upload/adaptive", handleUploadWithAdaptiveIO)
        http.HandleFunc("/download/adaptive/", handleDownloadWithAdaptiveIO)
    }
    
    // Override default handlers if adaptive mode is fully enabled
    if conf.Performance.AdaptiveBuffers && conf.Performance.FullyAdaptive {
        http.HandleFunc("/upload", handleUploadWithAdaptiveIO)
        http.HandleFunc("/download/", handleDownloadWithAdaptiveIO)
    }
}

Step 4: Update Existing Handlers (Optional Hybrid Approach)

Modify existing handlers to use adaptive components when available:

// In the existing handleUpload function, add adaptive streaming option:
func handleUpload(w http.ResponseWriter, r *http.Request) {
    // ... existing authentication and file handling code ...
    
    // Choose I/O method based on configuration
    if conf.Performance.AdaptiveBuffers && globalStreamingEngine != nil {
        // Use adaptive streaming
        clientIP := getClientIP(r)
        sessionID := generateSessionID()
        
        written, err := globalStreamingEngine.StreamWithAdaptation(
            dst, file, header.Size, sessionID, clientIP,
        )
        
        if err != nil {
            http.Error(w, fmt.Sprintf("Error saving file: %v", err), http.StatusInternalServerError)
            uploadErrorsTotal.Inc()
            os.Remove(absFilename)
            return
        }
    } else {
        // Use traditional buffer pool method
        bufPtr := bufferPool.Get().(*[]byte)
        defer bufferPool.Put(bufPtr)
        buf := *bufPtr

        written, err := io.CopyBuffer(dst, file, buf)
        if err != nil {
            http.Error(w, fmt.Sprintf("Error saving file: %v", err), http.StatusInternalServerError)
            uploadErrorsTotal.Inc()
            os.Remove(absFilename)
            return
        }
    }
    
    // ... rest of existing code ...
}

Configuration Migration

Gradual Configuration Rollout

1. Start with adaptive buffers disabled:

[performance]
adaptive_buffers = false

2. Enable for testing:

[performance]
adaptive_buffers = true
client_profiling = true

3. Full adaptive mode:

[performance]
adaptive_buffers = true
client_profiling = true
connection_type_detection = true
fully_adaptive = true

Feature Flags

Add feature flags for gradual rollout:

type PerformanceConfig struct {
    AdaptiveBuffers     bool `toml:"adaptive_buffers"`
    FullyAdaptive       bool `toml:"fully_adaptive"`       // Replace default handlers
    AdaptiveUploads     bool `toml:"adaptive_uploads"`     // Enable adaptive uploads only
    AdaptiveDownloads   bool `toml:"adaptive_downloads"`   // Enable adaptive downloads only
    TestingMode         bool `toml:"testing_mode"`         // Parallel testing mode
}

Testing Strategy

Parallel Testing Mode

Enable both old and new handlers for A/B testing:

if conf.Performance.TestingMode {
    // Setup both handlers with different paths
    http.HandleFunc("/upload", handleUpload)                    // Original
    http.HandleFunc("/upload/adaptive", handleUploadWithAdaptiveIO) // New
    
    // Route 50% of traffic to each (example)
    http.HandleFunc("/upload/auto", func(w http.ResponseWriter, r *http.Request) {
        if rand.Intn(2) == 0 {
            handleUpload(w, r)
        } else {
            handleUploadWithAdaptiveIO(w, r)
        }
    })
}

Performance Comparison

Create benchmarking endpoints:

http.HandleFunc("/benchmark/upload/original", benchmarkOriginalUpload)
http.HandleFunc("/benchmark/upload/adaptive", benchmarkAdaptiveUpload)

Monitoring and Rollback

Enhanced Metrics

Add comparative metrics:

var (
    // Original metrics
    uploadDuration     = prometheus.NewHistogram(...)
    uploadErrorsTotal  = prometheus.NewCounter(...)
    
    // Adaptive metrics
    adaptiveUploadDuration     = prometheus.NewHistogram(...)
    adaptiveUploadErrorsTotal  = prometheus.NewCounter(...)
    adaptiveBufferOptimizations = prometheus.NewCounter(...)
    adaptivePerformanceGains   = prometheus.NewHistogram(...)
)

Rollback Strategy

1. Configuration-based rollback:

[performance]
adaptive_buffers = false  # Immediate rollback

2. Automatic rollback on high error rates:

func monitorAdaptivePerformance() {
    if adaptiveErrorRate > originalErrorRate * 1.1 {
        log.Warn("Adaptive mode showing higher error rate, reverting to original")
        conf.Performance.AdaptiveBuffers = false
    }
}

Migration Timeline

Week 1: Infrastructure Setup

• Add adaptive I/O code
• Add configuration options
• Set up monitoring

Week 2: Internal Testing

• Enable testing mode
• Run performance comparisons
• Collect initial metrics

Week 3: Limited Rollout

• Enable for 10% of traffic
• Monitor performance and errors
• Gather feedback

Week 4: Gradual Expansion

• Increase to 50% of traffic
• Fine-tune optimization algorithms
• Address any issues

Week 5: Full Deployment

• Enable for all traffic
• Set as default configuration
• Plan for old code removal

Best Practices

1. Monitoring

• Always monitor both performance and error rates
• Set up alerts for performance degradation
• Track buffer optimization effectiveness

2. Configuration

• Start with conservative settings
• Enable features gradually
• Maintain rollback options

3. Testing

• Test with various file sizes
• Test with different network conditions
• Test with various client types

4. Documentation

• Document performance improvements
• Update user guides
• Maintain troubleshooting guides

Backward Compatibility

The adaptive I/O system is designed to be fully backward compatible:

1. Existing APIs remain unchanged
2. Configuration is additive (new sections, existing ones unchanged)
3. Default behavior is preserved when adaptive features are disabled
4. No changes to client protocols required

Performance Expectations

Based on the adaptive optimizations:

• High-speed networks: 30-50% throughput improvement
• Mobile networks: 20-30% improvement in reliability
• Variable conditions: Better adaptation to changing network conditions
• Memory usage: Optimized buffer allocation reduces memory pressure
• CPU usage: Minimal overhead from optimization algorithms

Troubleshooting

Common Issues

1. Higher memory usage: Adjust max_buffer_size
2. CPU overhead: Reduce buffer_optimization_interval
3. Poor adaptation: Enable more detailed logging
4. Compatibility issues: Disable specific adaptive features

Debug Configuration

[logging]
level = "debug"

[performance]
adaptive_buffers = true
detailed_logging = true
optimization_logging = true
client_profile_logging = true

This integration guide ensures a smooth transition to the improved dual stack while maintaining system stability and providing clear rollback options.