Flyte 2 WIP by EngHabu · Pull Request #6583 · flyteorg/flyte

EngHabu · 2025-08-22T06:38:23Z

Add flyteidl and minimal publishing workflows

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Please add one or more of the following labels to categorize your PR:

added: For new features.
changed: For changes in existing functionality.
deprecated: For soon-to-be-removed features.
removed: For features being removed.
fixed: For any bug fixed.
security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

main
- Flyte 2 WIP 👈

* Add flyteidl and minimal publishing workflows Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * pretend version for buf python Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

flyte-bot · 2025-08-22T06:38:58Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* python version fix and use OIDC to authenticate crates Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix python versioning Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Set version before setting up env Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * echo version Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Write to GITHUB_ENV Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Use env: Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf generate Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

flyte-bot · 2025-08-22T18:38:23Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

flyte-bot · 2025-08-25T20:39:37Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Mark go package as /v2 to be able to import both at the same time Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Upgrade to go 1.24 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Rename python, rust and npm to flyteidl2 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

flyte-bot · 2025-08-26T01:51:47Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

…6610) * trying idl2 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update trace def Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: rename idl2 to flyteidl2 Signed-off-by: machichima <nary12321@gmail.com> * generate -> buf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: rename idl2 to flyteidl2 Signed-off-by: machichima <nary12321@gmail.com> * build: re-gen Signed-off-by: machichima <nary12321@gmail.com> * fix: exclude python venv Signed-off-by: machichima <nary12321@gmail.com> * Add translator service Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Regen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf gen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * remove node execution service proto Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Breakup task service into a separate package Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Help strings for make file targets Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update TaskLog with read bool Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Trying the updated protovalidate plugin Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * protovalidate with semver Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try claude :shrug Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add buf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update deps Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * build: remove generate validate for python Signed-off-by: machichima <nary12321@gmail.com> * build: use managed mode on buf.gen.yaml Signed-off-by: machichima <nary12321@gmail.com> * Update validate go plugin Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update imagebuild Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix oneof buf lint Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add deps to generated python code Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Be explicit about the index Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * updated idl Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Catch up to main Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Generate mocks for workflow package Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * generate mocks for other packages Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf improvements Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * updates Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * feat: generate mock for common Signed-off-by: machichima <nary12321@gmail.com> * use make gen in GH Actions Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Do not setup python env in generation CI Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * setup uv Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * set working directory Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Try to reference files by path Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Explicitly move .venv to root Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try exclude in buf.yaml Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * set UV_PROJECT_ENVIRONMENT Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * download tooling Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add which mockery Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com>

flyte-bot · 2025-09-22T20:23:04Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Add Trigger APIs Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update triggers Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

bito-code-review · 2025-09-30T18:23:19Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Fix cargo build and add CI check Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint lib.rs Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update Readme and contributing guides Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * signed? Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build-crate target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix included ts files in npm Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

bito-code-review · 2025-09-30T22:29:13Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

bito-code-review · 2025-10-03T18:55:43Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Add device class to `GpuAccelerator` Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * make gen Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add HABANA_GAUDI as supported device class Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

bito-code-review · 2025-10-04T00:01:40Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

bito-code-review · 2025-10-04T04:21:41Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

bito-code-review · 2025-10-07T18:11:43Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

bito-code-review · 2025-10-08T19:51:43Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Clean up triggers api Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * make gen Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> --------- Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

bito-code-review · 2025-10-09T18:11:16Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

Signed-off-by: Alex Wu <c.alexwu@gmail.com>

Signed-off-by: machichima <nary12321@gmail.com>

* abort reconciller Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * mocs added Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix: workers run before startup scan Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * review fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

Signed-off-by: Alex Wu <c.alexwu@gmail.com>

…ons (#7089) Signed-off-by: Kevin Su <pingsutw@apache.org>

…#7058) Signed-off-by: WangWang0226 <eeha8834@gmail.com>

Signed-off-by: SZL741023 <shengzhe.lin@icloud.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: machichima <nary12321@gmail.com>

Signed-off-by: Kevin Su <pingsutw@apache.org>

…mpt (#7093) * retry added Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * rename pod to task Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * output uri fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix: produce outputuri in same format as the executor Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix: review Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * added comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

…rn (#7108) The docker-registry Helm subchart generates a random haSharedSecret on every helm template run when the value is not set. This causes complete.yaml and dev.yaml to always show as modified after running `make sandbox-build`, even with no actual deployment changes. Pin the secret to a static value in flyte-sandbox values.yaml since this is a local development sandbox environment. Signed-off-by: Kevin Su <pingsutw@apache.org>

… init (#6932) Move clientSet, connectorRegistry, and plugin struct creation from newConnectorPlugin into the PluginLoader callback to enable lazy initialization. Also use cfg.SupportedTaskTypes directly instead of fetching from the registry at startup time. Signed-off-by: Kevin Su <pingsutw@apache.org>

Add Abort APIs (#6590)

82a41b6

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Vendor in python buf dependencies (#6651)

1993238

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

EngHabu added 5 commits October 3, 2025 21:20

wip

7bb0db0

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

queue service implementation

075f324

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Support sqllite

987d536

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Runs service

aa24751

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Connect RunService to QueueService

3ef9544

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Triggers API adjustments (#6655)

83acecf

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

RunService.CreateRun from_automation flags (#6653)

cda285a

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

Cleanup triggers api (#6666)

a480286

* Clean up triggers api Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * make gen Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> --------- Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

[V2] Add RunLogsService for streaming pod logs (#7072)

162e684

Signed-off-by: Kevin Su <pingsutw@apache.org>

github-actions bot mentioned this pull request Mar 25, 2026

Add connectrpc Python stubs for flyteidl2 #7086

Merged

3 tasks

Add connectrpc Python stubs for flyteidl2 (#7086)

141745c

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

github-actions bot mentioned this pull request Mar 25, 2026

[V2] Add crd sync logic #7087

Merged

3 tasks

add crd sync logic (#7087)

9561ff2

Signed-off-by: Alex Wu <c.alexwu@gmail.com>

This was referenced Mar 25, 2026

[V2] Fix single binary and proxy pod slow start issue #7088

Merged

fix: send actual EndTime in UpdateActionStatus to fix inflated durations #7089

Merged

popojk and others added 2 commits March 24, 2026 22:49

decrease readness and liveness prob of sandbx and proxy pod (#7088)

da1fc32

Signed-off-by: Alex Wu <c.alexwu@gmail.com>

[v2] Event update time bug (#7090)

a8d8f23

Signed-off-by: machichima <nary12321@gmail.com>

This was referenced Mar 25, 2026

[v2] Event watcher #7091

Open

feat(executor): in-place task retry with scoped output paths per attempt #7093

Merged

This was referenced Mar 25, 2026

fix: adjust name for ActionIdentifier to a fixed value "a0" #7092

Merged

[V2]Fix secret injection error in sandbox #7101

Merged

[V2] Set global secret if both project and domain not provided #7102

Open

[V2]Fix secret injection error in sandbox (#7101)

3b803b9

Signed-off-by: Alex Wu <c.alexwu@gmail.com>

github-actions bot mentioned this pull request Mar 27, 2026

[V2] Pass run context to subaction + return in GetRunDetails #7094

Open

3 tasks

fix: send actual EndTime in UpdateActionStatus to fix inflated durati…

0e78c27

…ons (#7089) Signed-off-by: Kevin Su <pingsutw@apache.org>

github-actions bot mentioned this pull request Mar 27, 2026

[v2] DB migration with version #7103

Merged

3 tasks

WangWang0226 and others added 2 commits March 27, 2026 10:49

[Run Service] Add CreateRun default-input and response contract tests (…

dfb45ae

…#7058) Signed-off-by: WangWang0226 <eeha8834@gmail.com>

fix: adjust name for ActionIdentifier to a fixed value "a0" (#7092)

4ba6739

Signed-off-by: SZL741023 <shengzhe.lin@icloud.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>

github-actions bot mentioned this pull request Mar 27, 2026

fix: suppress pflag usage output on startup errors #7106

Merged

2 tasks

pingsutw and others added 2 commits March 27, 2026 11:30

fix: suppress pflag usage output on startup errors (#7106)

0e0bbd3

Signed-off-by: Kevin Su <pingsutw@apache.org>

[v2] DB migration with version (#7103)

12de8a2

Signed-off-by: machichima <nary12321@gmail.com>

github-actions bot mentioned this pull request Mar 27, 2026

Use dedicated connection for NOTIFY to avoid pool contention #7107

Merged

8 tasks

pingsutw and others added 2 commits March 27, 2026 22:39

Use dedicated connection for NOTIFY to avoid pool contention (#7107)

90ec0f3

Signed-off-by: Kevin Su <pingsutw@apache.org>

github-actions bot mentioned this pull request Mar 28, 2026

fix: pin docker-registry haSharedSecret to avoid sandbox manifest churn #7108

Merged

3 tasks

pingsutw added 2 commits March 28, 2026 02:13

Conversation

EngHabu commented Aug 22, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

Uh oh!

flyte-bot commented Aug 22, 2025

Uh oh!

flyte-bot commented Aug 22, 2025

Uh oh!

flyte-bot commented Aug 25, 2025

Uh oh!

flyte-bot commented Aug 26, 2025

Uh oh!

flyte-bot commented Sep 22, 2025

Uh oh!

bito-code-review bot commented Sep 30, 2025

Uh oh!

bito-code-review bot commented Sep 30, 2025

Uh oh!

bito-code-review bot commented Oct 3, 2025

Uh oh!

bito-code-review bot commented Oct 4, 2025

Uh oh!

bito-code-review bot commented Oct 4, 2025

Uh oh!

bito-code-review bot commented Oct 7, 2025

Uh oh!

bito-code-review bot commented Oct 8, 2025

Uh oh!

bito-code-review bot commented Oct 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

EngHabu commented Aug 22, 2025 •

edited by github-actions bot

Loading