Conversation
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
| ) | ||
| return task | ||
|
|
||
| def loader_args(self, task: TaskTemplate, root_dir: pathlib.Path | None) -> list[str]: |
There was a problem hiding this comment.
we could load it using the task_name itself.
Ideally honestly i would love to support tasks to be like container task. Why do we need to load the Notebook task? Can we not just send the notebook path and just run that directly? you might need the task for inputs / outputs maybe.
async def my_task():
t = NotebookTask("my_nb.ipynb")
await t(...)Ideally i would like ^ this to work. if you use module level resolver this cannot work.
There was a problem hiding this comment.
updated the resolver. it now bakes the notebook path and i/o type schemas into the task spec at registration time so it doesn't have to be a module-level resolver. this works?
@env.task
async def async_inline_workflow(x: int = 3, y: float = 1.5) -> int:
from flyteplugins.papermill import NotebookTask
nb = NotebookTask(
name="add_numbers_async_inline",
notebook_path="notebooks/basic_math.ipynb",
task_environment=env,
inputs={"x": int, "y": float},
outputs={"result": int},
)
# Use .aio() when calling a NotebookTask from an async task function
return await nb.aio(x=x, y=y)
|
looks pretty cool, but one comment, about resolver |
…o schema Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
fiedlerNr9
left a comment
fiedlerNr9
left a comment
There was a problem hiding this comment.
Works like a charm! Thanks for building this 🚀
Somehow, I want the user to see this is a Notebook task when clicking on summary. Maybe showing a Integration link with a JN icon, similiar to what we have for spark and ray. The link could just be a redirect to the report tab?
What do you think?
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
|
|
||
| inputs = _schema_to_types(args_dict.get("input-schema", "{}")) | ||
| outputs = _schema_to_types(args_dict.get("output-schema", "{}")) | ||
| config: dict = json.loads(args_dict.get("config", "{}")) |
There was a problem hiding this comment.
ooh this is dangerous. hmm we might have very big right, does it work always?
There was a problem hiding this comment.
it does work with the examples i tested. why would it be big? it's just the types and the papermill config, no?
|
|
||
| inputs = _schema_to_types(args_dict.get("input-schema", "{}")) | ||
| outputs = _schema_to_types(args_dict.get("output-schema", "{}")) | ||
| config: dict = json.loads(args_dict.get("config", "{}")) |
There was a problem hiding this comment.
I think if you want to pass it so, just pickle the entire task and send it no?
There was a problem hiding this comment.
hmm this should work too, right? papermill doesn’t support a wide range of types: only JSON-serializable ones (int, str, bool, float, list, dict, None), along with File, Dir, and DataFrame. these should work well with the plugin.
…nd add a type-check validator Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
|
@kumare3 let me know what you think. |
Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
3 participants
No description provided.