feat: Add option to dump and load database snapshots, while keeping relationships on foreign tables.

[alanpeixinho]

Include a command to dump and load database snapshots.

Summary

• Add snapshot and restore subcommands to update_db management command, replacing the previous direct DB-to-DB migration flow
• Snapshots are stored as .tar.gz archives containing per-table CSV files, preserving foreign key relationships on restore

How to Test

Important

For these tests, especially for writing, use a local database, instead of pointing to production.

1. Populate a dabase with poetry run python3 manage.py seed_test_data --clear --yes, or use an existing database.
2. Run the snapshot command, with desired parameters, as in poetry run python3 manage.py update_db snapshot --start-interval "100 days" --end-interval "0 minute" --filepath ./my-snapshot.tar.gz
3. Check that data is properly saved inside tables on my-snapshot.tar.gz.
4. Clear the database with run python3 manage.py seed_test_data --clear --yes --checkouts 0 --issues 0
5. Run the restore command with desired parameters, as poetry run python3 manage.py restore snapshot --start-interval "100 days" --end-interval "0 minute" --filepath ./my-snapshot.tar.gz
6. Check that the data on the snapshot was properly inserted in the database.

Closes #1884

[MarceloRobert]

looks ok, I'll test later

[MarceloRobert]

it worked for a small interval, but for an interval of 1 day I got an error of

Processed 1610 tests records: 1.53 MiB
Error updating database: read
CommandError: Command failed

I assume it's because it went over the size limit of 32MiB?

Also left some other comments

[gustavobtflores]

I think we could remove https://github.com/kernelci/dashboard/blob/main/backend/scripts/copy_db_data.sh as it no longer makes sense after the changes on the command

[gustavobtflores]

Wouldn't be better to restore incidents last here? restore_incidents calls insert_incidents_data that checks for existing issues, builds and tests and could skip incidents incorrectly

[alanpeixinho]

Good catch. Done.

[alanpeixinho]

it worked for a small interval, but for an interval of 1 day I got an error of

Processed 1610 tests records: 1.53 MiB
Error updating database: read
CommandError: Command failed

I assume it's because it went over the size limit of 32MiB?

Also left some other comments

I have managed to find the problem, when spilling the 32MiB, the temporary file is moved to disk, and this was causing the later reads to fail.
The best solution is to actually open the file with read and write permissions.
But I also included a little more buffer (64MiB) to avoid (for most of the cases) the spilling.

[MarceloRobert]

Seems to be all working, just left some nits

[MarceloRobert]

Tested again and is working well