[TEMP] Tmp/0.14.1 with migration #884
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds the foundational structure for database migrations: - migration/mod.rs: Main migration runner with status checking, execution, backup creation, and crash recovery - migration/error.rs: Comprehensive error types for migration failures - migration/context.rs: Migration context with progress reporting - migration/registry.rs: Migration registry for version-to-function mapping - migration/revisions/mod.rs: Placeholder for individual revision files The migration system supports: - Version detection and comparison - Sequential migration execution - Pre-migration backups via RocksDB checkpoints - Crash recovery via state checkpointing - Progress reporting for long-running migrations - Lock files to prevent concurrent migrations Tests: 13 unit tests covering all core functionality
This commit integrates the migration system with the database opening logic: Changes: - Update build.rs to also export DB_BASE_VERSION for minimum migration version - Add base_version field to .db-versions.yml - Integrate MigrationRunner into MadaraBackend::open_rocksdb() - Add initialize_fresh_database() method for new databases - Add run_migrations_with_storage() for running with RocksDBStorage - Add inner_db() accessor to RocksDBStorage for migration access - Remove old db_version.rs module (replaced by migration system) - Use MultiThreaded DB type in migration module for compatibility The flow is now: 1. Check migration status (file-based check) 2. For fresh DB: write version file 3. For migration needed: open DB, run migrations, close 4. Open DB normally with RocksDBStorage wrapper Tests: All 36 tests pass
Add comprehensive integration tests for database migration: - test_fresh_database_creates_version_file: Verify fresh DB creates version file - test_fresh_database_can_be_reopened: Test database can be reopened - test_same_version_opens_without_migration: No migration for same version - test_newer_database_version_fails: Detect database newer than binary - test_too_old_database_version_fails: Detect database too old to migrate - test_migration_state_file_operations: Test state file persistence - test_version_file_read_write: Test version file operations - test_migration_lock_file_creation: Test lock file mechanism - test_backup_directory_creation: Verify backup path - test_full_database_lifecycle: Full open/reopen/verify cycle - test_database_with_data_survives_reopen: Data persistence test - test_invalid_version_file_format: Error handling for bad version - test_version_file_with_whitespace: Handle whitespace in version file Tests: 49 tests now pass (14 new migration tests)
Documentation updates: 1. .db-versions.yml: - Added comprehensive header documentation - Explained purpose of current_version and base_version - Added description field to version entries - Added instructions for adding new versions 2. MIGRATION.md: - Complete guide for the migration system - Architecture overview with ASCII diagram - Step-by-step guide for adding new migrations - MigrationContext API documentation - Error handling guide - Recovery procedures for failed/interrupted migrations - Testing best practices - Troubleshooting section
- Remove indeterminate() from MigrationProgress (always know steps) - Improve NoMigrationPath error message to clarify it's a registry bug - Merge resume_migration into execute_migrations (single unified flow) - Trim redundant doc comments and simplify code structure - Add comment explaining RocksDB checkpoint is lightweight (hard links) - Export DB_VERSION_FILE, DB_MIGRATION_LOCK, DB_MIGRATION_STATE constants - Make read_version_file/write_version_file public for testing - Refactor tests: use rstest for parameterized tests, remove redundant ones - Use strong assertions with exact version checks
- Add task-test-migration.yml: reusable workflow for migration testing - Downloads base DB fixture from GitHub releases - Starts madara (migration runs automatically) - Verifies .db-version matches expected version - Runs migration validation tests - Graceful handling when fixture doesn't exist - Add pull-request-migration.yml: trigger on migration file changes - Triggers on changes to migration/revisions/** or .db-versions.yml - Calls the task workflow with version parameters
- Add migration.rs with test structure: - test_migration_validation_rpc_healthy: verify RPC responds - test_migration_validation_blocks_accessible: verify block data - test_migration_validation_state_update_intact: verify state roots - Tests are #[ignore] by default, run with: cargo test -p mc-e2e-tests migration_validation -- --ignored - Tests connect to external madara (configurable via MIGRATION_TEST_RPC_URL) - Add v9_snip34 module placeholder for SNIP-34 specific tests - Add url dependency to mc-e2e-tests
- Triggers on push to main when .db-versions.yml changes
- Also supports manual dispatch with force_update option
- Workflow steps:
1. Detect if base_version actually changed
2. Build madara from current main
3. Sync configurable number of blocks (default 50)
4. Package DB as tarball
5. Upload to GitHub releases as db-fixtures-v{VERSION}
- Release includes usage instructions
- Used by task-test-migration.yml to download fixtures
- If base DB fixture doesn't exist in releases, create one on-the-fly - Useful for PRs that change both base_version and add migration code - Added blocks_to_sync input parameter (default 50) - Increased timeout to 60 min to accommodate DB creation - Added db_source output to track if DB was downloaded or created - Improved logging and summary output
pull-request-migration.yml: - Split into two dependent jobs: create-base-db → migration-test - Job 1 downloads fixture or creates DB, uploads as artifact - Job 2 downloads artifact, runs migration, runs tests - Clear separation of concerns task-test-migration.yml: - Simplified to match same two-job pattern - Reusable for manual dispatch or other workflows - Cleaner, more focused implementation
- Remove task-update-base-db.yml (no automation) - Simplify task-test-migration.yml (just download, no fallback) - Simplify pull-request-migration.yml (just calls task) - Add scripts/create-base-db.sh for manual DB creation and upload Manual workflow: 1. Run: ./scripts/create-base-db.sh 8 50 2. Script builds madara, syncs blocks, uploads to releases 3. Migration tests download fixture from releases
Remove path filter - migration tests now run on all PRs, not just those that modify migration files. This ensures migrations don't break from any code changes.
- Remove standalone pull-request-migration.yml - Add test-migration job to pull-request-main-ci.yml - Runs after test-madara completes - Update task-test-migration.yml to read versions from .db-versions.yml - No more hardcoded version parameters needed
- Download pre-built madara binary instead of rebuilding - Run test-migration after build-madara (parallel with test-madara) - Add madara-binary-hash input parameter - Faster CI execution
- Add task-do-nothing-test-migration.yml for merge queue - Add test-migration to pull-request-merge.yml - Fix markdown lint errors in MIGRATION.md (line length, code block language)
- Store DB fixtures as Docker images at ghcr.io/madara-alliance/db-fixtures:v{VERSION}
- Consistent with existing artifact storage pattern
- No release tags cluttering the releases page
- Add debug_assert validation for MigrationProgress (current_step <= total_steps) - Handle target_version mismatch when resuming interrupted migrations
Treat locks with unreadable metadata as stale instead of ignoring
- Use stronger memory ordering for cross-thread visibility - Explicitly handle system time errors in lock age check
- Use subshell for cd to preserve working directory - Add binary existence check before running - Add comment explaining || true usage - Clarify current_step range in MigrationProgress docs
These documentation files were accidentally included. Removing from git tracking.
Similar to clippy, madara needs separate formatting since it's not in the toplevel workspace.
Use docker buildx to build for both linux/amd64 (CI) and linux/arm64 (Mac)
- Switch from sepolia to mainnet for consistency - Fix --no-l1-sync to --l1-sync-disabled in script - Add --sync-stop-at 50 in workflow to prevent unnecessary syncing
- Use .github/actions/setup-rust which installs LLVM 19 - Use blacksmith runner for consistency - Add LDFLAGS for Cairo Native compatibility
…14.1 - blockifier: 57052466 -> 0f818e01 - starknet_api: 57052466 -> 0f818e01 - generate-pie: v0.14.0-alpha.4 -> 4f1b79f7 - starknet-types-core: 0.2.3 -> 0.2.4 (required by new blockifier)
- Add get_compiled_class_hash_v2 to BlockifierStateAdapter - First checks DB for stored v2 hash, then computes on-the-fly - Add compiled_class_hash_v2() method to ClassInfo - Uses HashableCompiledClass trait from starknet_api for BLAKE hash computation
… block close - Extract migrated class hashes from BlockExecutionSummary - Partition declared_classes into newly declared and migrated - Populate migrated_compiled_classes in StateDiff - Add logging for migration statistics
- Call update_class_v2_hashes for migrated classes before closing block - This persists computed BLAKE hashes so future lookups don't need to recompute
Milestone 0-5 implementation: - Update blockifier, starknet_api, generate-pie dependencies - Bump latest_protocol_version to 0.14.1 - Add VERSIONED_CONSTANTS_V0_14_1 to chain config - Implement get_compiled_class_hash_v2 in BlockifierStateAdapter and LayeredStateAdapter - Handle compiled_class_hashes_for_migration in block close - Persist v2 hashes to DB via update_class_v2_hashes - Update devnet to use BLAKE hashes for v0.14.1 compatibility Note: Some L1 handler tests still failing, needs further investigation
- Return Felt::ZERO for legacy (Cairo 0) classes which don't have compiled_class_hash - Fix test failures caused by blockifier calling get_compiled_class_hash on UDC (legacy class) - Update devnet to use BLAKE hash (v2) for Sierra class storage - Fix devnet classes.rs to properly set compiled_class_hash_v2 for v0.14.1+ All mc-block-production tests now pass (9 passed, 1 ignored)
- Use compile_to_casm_with_blake_hash() in test_declare for v0.14.1 - Update assertion to check compiled_class_hash_v2 instead of compiled_class_hash - Handle legacy classes (Cairo 0) in get_compiled_class_hash by returning Felt::ZERO All mc-block-production and mc-devnet tests pass
- Remove 0.14.1/ folder from git (kept locally) - Add 0.14.1/ to .gitignore to prevent future commits
- Move 0.14.1/ to AI_docs/0.14.1/ - Update .gitignore to ignore AI_docs/ instead
- Add skip_migration_backup to MadaraBackendConfig - Add --skip-migration-backup CLI flag (default: backup enabled) - Add cleanup_backup() after successful migration - Backup only persists on failure (for recovery)
- starknet-rust-accounts: 0.16.0 -> 0.17.0-rc.0 - starknet-rust-contract: 0.16.0 -> 0.17.0-rc.0 - starknet-rust-core: 0.16 -> 0.17.0-rc.0 - starknet-rust-crypto: 0.8.1 -> 0.9.0-rc.0 - starknet-rust-providers: 0.16 -> 0.17.0-rc.0 - starknet-rust-signers: 0.14 -> 0.15.0-rc.0 - starknet-rust: 0.17.0 -> 0.18.0-rc.0 - Added starknet-rust-core-derive: 0.1.0 - Added starknet-rust-macros: 0.2.6-rc.0 - Updated orchestrator squash.rs for migrated_compiled_classes
- starknet-rust-accounts: 0.16.0 -> 0.17.0-rc.0 - starknet-rust-core: 0.16 -> 0.17.0-rc.0 - starknet-rust-crypto: 0.8 -> 0.9.0-rc.0 - starknet-rust-providers: 0.16 -> 0.17.0-rc.0 - starknet-rust-signers: 0.14 -> 0.15.0-rc.0 - starknet-rust: 0.17.0 -> 0.18.0-rc.0 - Added starknet-rust-core-derive: 0.1.0 - Added starknet-rust-macros: 0.2.6-rc.0
- starknet-rust: 0.17.0 -> 0.18.0-rc.0 Note: bootstrapper-v2 has a pre-existing build issue with alloy 1.0.25 and unlinked Factory contract bytecode (not related to this update)
- Add migrated_compiled_classes field to StateDiff in tests - Add event_index and transaction_index fields to EmittedEvent - Fix old_root type from Felt to Option<Felt> in PreConfirmedStateUpdate
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request type
Please add the labels corresponding to the type of changes your PR introduces:
What is the current behavior?
Resolves: #NA
What is the new behavior?
Does this introduce a breaking change?
Other information