buildpack s3 cleanup #1403
buildpack s3 cleanup #1403
Conversation
| 2. Copy each blob to its new namespaced location: | ||
| ```bash | ||
| # Example for ruby-buildpack | ||
| aws s3 cp \ | ||
| s3://buildpacks.cloudfoundry.org/29a57fe6-b667-4261-6725-124846b7bb47 \ | ||
| s3://buildpacks.cloudfoundry.org/ruby-buildpack/29a57fe6-b667-4261-6725-124846b7bb47 | ||
| ``` | ||
| 3. Keep original files in root temporarily for rollback capability | ||
| 4. After successful verification (30-day grace period), archive or delete root-level blobs |
There was a problem hiding this comment.
I would have two questions:
- Are DNS updates for option 1 needed?
- How we will make sure that the migrated blobs are used and not the ones in the root folder? We could use bucket polices, ACLs, etc. but we need to make sure that no root blobs are used.
There was a problem hiding this comment.
- i do not think dns updates are necessary
- we would change the bosh-release
config/final.yml
stephanme
left a comment
stephanme
left a comment
There was a problem hiding this comment.
Will older buildpack releases be deployable after the restructuring?
E.g. when I want to deploy cf-deployment v46.3.0 from Jan 2025 using javabuildpack v4.71.0 - will this work after the old blobs have been deleted (after phase 4 : cleanup)?
| 2. **Orphan Detection:** No automated way to identify unused blobs when `blobs.yml` diverges from S3 | ||
| 3. **Audit Challenges:** Cannot identify file types, owners, or purposes without external mappings | ||
| 4. **Cost Inefficiency:** Potentially storing obsolete blobs (estimated 30-40% orphan rate in some analyses) | ||
| 5. **Multi-Team Collision:** Multiple buildpack teams share the same flat namespace, increasing collision risk and confusion |
There was a problem hiding this comment.
The technical separation makes sense. But I don't think that there is a multi-team collision. All buildpacks are handled by the same team / WG area "Buildpacks and Stacks": https://github.com/cloudfoundry/community/blob/main/toc/working-groups/app-runtime-interfaces.md?plain=1#L75
There was a problem hiding this comment.
only creating a bosh release from a tag will create a issue. but still possible with minimal change.
when checking out a tag. the user should only add a folder_name: xxx in the config/final.yml
There was a problem hiding this comment.
Just adding more context. The final release tarballs are available on bosh.io (e.g. https://bosh.io/releases/github.com/cloudfoundry/java-buildpack-release?all=1) uploaded by this task, that is why they are not impacted. If you want to build from source for an older tag will require the workaround @ramonskie mentioned above.
|
As long as the older releases are still supported, I like option 1. |
beyhan
left a comment
beyhan
left a comment
There was a problem hiding this comment.
During the TOC meeting we discussed this and the feedback is
- Define a retention period for the orphaned blobs
- Option 1 has been selected
|
Given that there don't seem to be any open issues with this RFC and work for this is progressing at great speed, the TOC decided to move this to the final comment period. You now have one week to leave any comments or forever hold your peace. |
4 participants
This PR adds the RFC "buildpacks s3 bucket cleanup".
For easier viewing, you can see the full RFC as preview.