Implement comprehensive TOML configuration system with validation and constructor parameter alignment

[Copilot]

This PR implements a brand new TOML configuration system for pycloudlib that addresses long-standing configuration inconsistencies and limitations identified in issues canonical#457 and canonical#466.

Problem Statement

The existing configuration system had several critical issues:

1. Issue [enhancement] Update handling of "required_values" in cloud class canonical/pycloudlib#466: If all required values were provided to a cloud constructor, the TOML file was completely ignored, preventing users from using TOML as a base configuration with runtime overrides.

2. Issue feature request: completely align toml and class instantiation configuration options canonical/pycloudlib#457: Inconsistent configuration options between TOML and constructor parameters. SSH keys could not be passed at runtime and had to be configured in TOML or set after instantiation using the use_key method.

3. No validation: TOML configuration errors were only discovered at runtime when attempting to use cloud services.

Solution

🔧 Fixed Issue canonical#466 - TOML Always Parsed

• Modified _check_and_get_config to always parse TOML configuration when available
• TOML now serves as base configuration with constructor parameters providing overrides
• Previous behavior: if all required values provided: ignore TOML completely
• New behavior: always parse TOML + merge with constructor parameters

🔧 Fixed Issue canonical#457 - Complete Configuration Alignment

• Any setting available in TOML can now be passed as constructor parameter
• SSH keys can be configured at runtime via constructor parameters
• Enhanced BaseCloud to accept and merge all constructor parameters with TOML config
• Updated cloud implementations (EC2, Azure) to properly pass constructor parameters through

✨ Added TOML Validation

• Created comprehensive JSON schemas for all cloud types (EC2, Azure, GCE, IBM, OCI, OpenStack, QEMU, VMware, LXD)
• Validation catches configuration errors immediately using JSONSchema
• Balanced approach: catches invalid additional properties without being overly restrictive about required fields
• Cloud-specific validation adapts per cloud implementation

🛠️ Enhanced Configuration System

• New validate_cloud_config() function for immediate error detection
• New merge_configs() function for proper parameter merging with None value handling
• Enhanced parse_config() with optional validation support
• Maintains full backward compatibility with existing code

Key Changes

# Before: TOML ignored if required values provided
ec2 = EC2('test', region='us-east-1', access_key_id='KEY', secret_access_key='SECRET')
# Result: Only constructor values used, TOML completely ignored

# After: TOML serves as base configuration with constructor overrides
ec2 = EC2('test', config_file='config.toml', region='us-east-1')  
# Result: All TOML settings loaded + region overridden by constructor

# Before: SSH keys couldn't be passed at runtime
# Had to use: ec2.use_key() after instantiation

# After: SSH keys work as constructor parameters
ec2 = EC2('test', public_key_path='/path/to/key.pub', private_key_path='/path/to/key')

# New: TOML validation catches errors immediately
# Invalid TOML with extra fields will raise ValueError with clear error message

Demonstration

A comprehensive demonstration script (demo_new_config_system.py) shows all features working:

• TOML parsing with constructor overrides
• SSH key configuration at runtime
• TOML validation catching invalid configurations
• Proper configuration merging with None value handling

Testing

• 29 new comprehensive tests covering all new functionality
• All existing tests pass (full backward compatibility maintained)
• Integration tests with real cloud implementations (Azure, GCE)
• Edge case testing (empty configs, None values, invalid schemas)

Files Changed

• Core: config.py, cloud.py - Enhanced configuration system
• New: config_schemas.py - JSON schemas for validation
• Cloud implementations: ec2/cloud.py, azure/cloud.py - Parameter passing
• Template: pycloudlib.toml.template - Updated documentation
• Demo: demo_new_config_system.py - Comprehensive examples
• Tests: Multiple new test files with extensive coverage

Backward Compatibility

✅ Fully maintained - All existing code continues to work unchanged. The new features are additive and the configuration behavior is enhanced rather than replaced.

Closes canonical#457
Closes canonical#466

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.