Can the Matbench submission process be handled entirely within a Google Colab notebook?

[sgbaird]

Not necessarily meaning let's change how the Matbench submission system works (I like the thoroughness/provenance), but rather if Google Colab has what's necessary to programatically follow the three submission steps:

1. "Create 3 required files"
2. "Put files in appropriate folder"
3. "Create a PR to the Matbench repository"

[sgbaird]

Save notebook programmatically to harddisk:

• Save the notebook from command line or line magic jupyterlab/jupyterlab#6161
• https://stackoverflow.com/questions/66769956/how-to-get-raw-ipynb-file-from-google-colab-using-sharing-url-command-line

Write info.json file to temporary Colab storage:

• %%writefile IPython magic https://stackoverflow.com/questions/48687091/how-to-edit-and-save-text-files-py-in-google-colab

Put this in a directory structure using os or similar

Make pull request via API or CLI:

• https://github.com/PyGithub/PyGithub
• https://opensource.com/article/19/7/create-pull-request-github ("Open and PR" could be a link that Colab automatically opens)
• https://stackoverflow.com/questions/4037928/can-you-issue-pull-requests-from-the-command-line-on-github

Short answer seems to be yes.

This could reduce the barrier to getting an initial "win" (e.g. verifying it works for a dummy model), and also opens up a direct opportunity for low-barrier even with bigger models if someone uses Colab Pro or Pro+. This would also make it easier for someone to mock-up a notebook with some dummy data and then ask someone with a Colab Pro account to click run. Anyway, just some thoughts based on some recent internal discussion with a colleague about Matbench.

[sgbaird]

The tough part is programmatically downloading a snapshot of the current notebook with outputs. This is even harder if this requires the Colab notebook to be the active window on the user's machine.

• https://stackoverflow.com/questions/51121647/get-the-current-contents-of-the-entire-jupyter-notebook
• https://stackoverflow.com/questions/12544056/how-do-i-get-the-current-ipython-jupyter-notebook-name/44589075#44589075
• https://stackoverflow.com/questions/54350254/get-only-the-code-out-of-jupyter-notebook/54350786#54350786
• https://stackoverflow.com/questions/32237275/save-an-ipython-notebook-programmatically-from-within-itself
• https://stackoverflow.com/questions/58132658/how-to-use-javascript-in-google-colab
• https://stackoverflow.com/questions/44961557/how-can-i-save-a-jupyter-notebook-ipython-notebook-programmatically
• https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb
• https://stackoverflow.com/questions/66880698/how-to-cause-jupyter-lab-to-save-notebook-programmatically
• https://stackoverflow.com/questions/69010980/save-an-jupyterlab-notebook-programmatically-from-within-itself
• https://stackoverflow.com/questions/57631541/how-to-trigger-save-command-in-jupyter-notebook

In the end, I think the user will have to "Save a copy to Drive" after clicking an "Open in Colab" link. From there, it should be possible to programmatically extract the file ID and download the (hopefully recently autosaved) notebook. Probably good to include save an extra notebook with the input history.

Some of the relevant code:

from IPython.display import display, Javascript
display(Javascript('IPython.notebook.save_checkpoint();')) # save, but probably only if window is active
%notebook -e notebook.ipynb

based on some modifications I'm doing for xtal2png + imagen-pytorch:

import time
from IPython.display import display, Javascript
display(Javascript('IPython.notebook.save_checkpoint();')) # save, but probably only if window is active
timestr = time.strftime("%Y%m%d-%H%M%S")
notebook_savepath = path.join(results_folder, f"notebook.ipynb")
print(notebook_savepath) # no output cells
%notebook -e notebook-input-history.ipynb
!mv notebook-input-history.ipynb $notebook_savepath

WIP at https://colab.research.google.com/drive/15YLOWHB_NkIIqKLO0ik784fsK2xJD08l?usp=sharing

Adding an "Open in Colab" badge could be accomplished ad-hoc in the Matbench actions. Still, maybe it's better to request that the user downloads the notebook and manually adds a version with outputs, markdown cells, and the "open in Colab" badge (which the Colab UI makes very straightforward).

Haven't fleshed out the details of creating the GitHub PR. Not sure if authentication will cause more problems than just following normal instructions.

once it's ready, then planning to share notebook with @jae3goals #141