Downgrade Support

[TheBlueMatt]

For most enterprise deployments of software, the ability to roll back a change if things aren't working correctly during a deploy is a critical feature. It seems BDK's current wallet serialization logic doesn't provide this as a hard guarantee to downstream users. It would be nice if this were provided as an explicit supported feature.

[notmandatory]

I agree we should support a downgrade policy but we need to define more precisely what that means.

Currently bdk_wallet and bdk_chain types that are persisted implement the Merge: Default trait. The docs for Merge could use improvement, currently says "Trait that makes an object mergeable." In practice because this trait requires Default if a new version of persistence code loads old data nothing breaks because newly required data is defaulted.

But the question here is what if a user downgrades to an old version of persistence code and tries to load and "Merge" data created with a newer version of the persistence code. In this case we need to guarantee the new persistence code didn't:

1. remove or rename struct fields or enum variants expected by the old code
2. change how a field or variant is used in a way that would break old code
3. add new data that can't be ignored by the old code

I don't know a way to enforce the above as we do with Default for forward compatibility. But here's what I think we can do:

• Document somewhere that all persisted data must support the Merge trait
• Document for Merge that new fields and enum variants must not be removed between minor/patch versions
• Document a process to deprecate and then remove/rename fields/variants in future major versions
• Add CI process to verify data created with the prior release version can be loaded with a new version
• Add CI process to verify data created with new version can be loaded with the prior release version

For the deprecate policy I'm thinking we only support roll back for deprecated fields/variants for one major release before removing rollback support for it. This would mean users can upgrade or downgrade between say 1.x and 2.x, but not 1.x and 3.x. Would that be sufficient downgrade support?

[TheBlueMatt]

For the deprecate policy I'm thinking we only support roll back for deprecated fields/variants for one major release before removing rollback support for it. This would mean users can upgrade or downgrade between say 1.x and 2.x, but not 1.x and 3.x. Would that be sufficient downgrade support?

Depends on the time between versions, but likely yes. Certainly if there's a year or more between major versions that seems fine.

On the LDK end, we do this with TLVs - each new bit of data we write is tagged with a lightning TLV where the type being even indicates that a downgrade to a version that doesn't understand it will fail, and the type being even indicates that a previous version will simply ignore it. Then, for new features we can document that use of new features will break downgrade, but use of existing features will not. That lets us worry a bit less about it for new features that we don't want old versions to ignore.

[notmandatory]

Quick update, I have a summer of bitcoin student (@110CodingP) who's working on some utilities/test suite for persisting BDK change set data that will be a step in addressing this issue. But it won't make it into the bdk_wallet 2.1 release so removing from the milestone. See also #14.