Updated documentation for the upcoming data release by chiarabellenghi · Pull Request #294 · icecube/skyllh

chiarabellenghi · 2026-04-08T15:41:05Z

Smaller, task-focused notebooks :)

Before merging:

Run all notebooks
Double check that text matches results
Release notes should include a note that tells the user that the background PDF treatment changed and they should expect slightly different results compared to the previous skyllh release

…lease

…tion

This reverts commit 8ccb49b.

…lease

Copilot

Pull request overview

This PR refreshes SkyLLH’s documentation and tutorial notebooks for the upcoming IceCube public data release, reorganizing the Sphinx site structure and adding task-focused examples/tutorials while making small docstring/code robustness tweaks in core utilities.

Changes:

Reworked Sphinx docs structure (new landing pages, tutorials index, FAQ formatting, API reference entry) and added/updated example & tutorial notebooks.
Updated packaging/docs tooling (new Sphinx extensions in pyproject.toml/conf.py, Makefile parallel build option, pre-commit formatting for Jupyter).
Improved dataset post-transfer archive extraction behavior by logging/removing corrupted archives on extraction errors.

Reviewed changes

Copilot reviewed 41 out of 47 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
skyllh/core/signal_generator.py	Docstring formatting tweak to render examples as literal blocks.
skyllh/core/interpolate.py	Docstring formatting for `**kwargs`.
skyllh/core/dataset.py	Adds extraction error handling/logging and archive cleanup for post-transfer unarchiving.
skyllh/core/background_generation.py	Docstring list formatting cleanup.
skyllh/analyses/i3/publicdata_ps/utils.py	Docstring spacing improvement.
skyllh/analyses/i3/publicdata_ps/time_integrated_ps.py	Docstring indentation fix for deprecated option note.
skyllh/analyses/i3/publicdata_ps/time_integrated_ps_function_energy_spectrum.py	Docstring indentation fix for deprecated option note.
skyllh/analyses/i3/publicdata_ps/time_dependent_ps.py	Docstring indentation fix for deprecated option note.
README.md	Adds “Quick links”, clarifies install instructions and Python requirement.
pyproject.toml	Adds Sphinx extension deps to `docs` extra.
doc/sphinx/tutorials/sensitivity_study.ipynb	New tutorial on sensitivity/discovery potential workflow.
doc/sphinx/tutorials/publicdata_ps_timedep.ipynb	Removes outdated time-dependent tutorial notebook.
doc/sphinx/tutorials/publicdata_ps_14yr.ipynb	Removes temporary/check notebook.
doc/sphinx/tutorials/index.rst	Replaces/expands tutorials landing page and toctree.
doc/sphinx/tutorials/getting_started.ipynb	Removes older “getting started” notebook.
doc/sphinx/tutorials/fitting_a_source.ipynb	New steady point-source fitting tutorial (IceTracks-DR2).
doc/sphinx/tutorials/dataset_collections.ipynb	New tutorial introducing dataset collections and selection.
doc/sphinx/notes.rst	Fixes wording/formatting in docstring notes.
doc/sphinx/Makefile	Enables parallel Sphinx builds and copies `user_manual.pdf` into build output.
doc/sphinx/installation.rst	Rewrites install page (pip/uv/conda tabs, extras, i3skyllh note).
doc/sphinx/index.rst	Updates main docs landing page structure and navigation.
doc/sphinx/faq/signal_generator.ipynb	Removes old FAQ notebook in favor of inline dropdown.
doc/sphinx/faq/index.rst	Converts FAQ to a dropdown-based page with embedded content.
doc/sphinx/examples/timepdf.ipynb	Cleans up example notebook formatting/markdown cells and outputs.
doc/sphinx/examples/scrambling.ipynb	Adds new data scrambling example notebook.
doc/sphinx/examples/multiproc.ipynb	Adds new multiprocessing example notebook.
doc/sphinx/examples/index.rst	Adds new examples to the examples toctree.
doc/sphinx/dev_docs/unit_tests.rst	Expands unit test docs (pytest runner usage, structure, examples).
doc/sphinx/dev_docs/logging.ipynb	Minor formatting cleanup in logging dev doc notebook.
doc/sphinx/contributing.rst	Adds contributing guide (issues, PRs, tutorials, style).
doc/sphinx/conf.py	Enables new Sphinx extensions and sets copybutton/theme nav options.
doc/sphinx/concepts/source_definition.ipynb	Removes autoreload cell/empty code cell from concept notebook.
doc/sphinx/concepts/pdf.ipynb	Removes autoreload cell from concept notebook.
doc/sphinx/concepts/parameters.ipynb	Removes autoreload cell from concept notebook.
doc/sphinx/concepts/index.rst	Fixes spelling/grammar.
doc/sphinx/concepts/flux_definition.ipynb	Removes autoreload cell and fixes wording (right-ascension).
doc/sphinx/api_reference.rst	Adds an API reference entry point for the Sphinx toctree.
.pre-commit-config.yaml	Runs Ruff hooks on Jupyter notebooks as well.
.gitignore	Ignores `.DS_Store`.
.github/workflows/documentation.yml	Builds docs on `documentation` branch too.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tomaskontrimas · 2026-05-21T10:01:24Z

+    try:
+        if filename_lower.endswith('.zip'):
+            with zipfile.ZipFile(archive_path) as zf:
+                zf.extractall(dst_path)
+        elif filename_lower.endswith(('.tar.gz', '.tgz', '.tar.bz2', '.tar.xz')):
+            with tarfile.open(archive_path) as tf:
+                tf.extractall(dst_path)
+        else:


We trust the dataset sources

tomaskontrimas · 2026-05-21T10:09:04Z

+    except (zipfile.BadZipFile, tarfile.ReadError, tarfile.StreamError) as err:
+        logger = get_logger(f'{__name__}.post_transfer_unarchive')
+        logger.warning(
+            f'Failed to extract archive file "{archive_path}"! The file might be corrupted and will be removed. Error message: {err!s}'
+        )
+        os.remove(archive_path)
+        raise err


Fixed in eb406ad

tomaskontrimas · 2026-05-21T10:00:49Z

 # Build html documentation for each branch separately.
 html-multiversion:
-	sphinx-multiversion "$(SOURCEDIR)" "$(BUILDDIR)"
+	sphinx-multiversion "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS)


It does accept it and speeds up documentation builds where multiple cpus are available

+    {
+     "data": {
+      "text/plain": [
+       "{'IceTracks-DR1': <module 'skyllh.datasets.i3.PublicData_10y_ps' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/PublicData_10y_ps.py'>,\n",
+       " 'IceTracks-DR2': <module 'skyllh.datasets.i3.PublicData_14y_ps' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/PublicData_14y_ps.py'>,\n",
+       " 'IceTracks-DR1_wMC': <module 'skyllh.datasets.i3.PublicData_10y_ps_wMC' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/PublicData_10y_ps_wMC.py'>,\n",
+       " 'TestData': <module 'skyllh.datasets.i3.TestData' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/TestData.py'>}"
+      ]


+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'multiproc': {'ncpu': None},\n",
+       " 'logging': {'log_level': 'INFO',\n",
+       "  'log_format': '%(asctime)s %(processName)s %(name)s %(levelname)s: %(message)s',\n",
+       "  'enable_tracing': False},\n",
+       " 'project': {'working_directory': '.'},\n",
+       " 'repository': {'base_path': '/Users/tomas/.cache/skyllh',\n",
+       "  'download_from_origin': True},\n",
+       " 'units': {'internal': {'angle': Unit(\"rad\"),\n",
+       "   'energy': Unit(\"GeV\"),\n",
+       "   'length': Unit(\"cm\"),\n",
+       "   'time': Unit(\"s\")},\n",
+       "  'defaults': {'fluxes': {'angle': Unit(\"rad\"),\n",
+       "    'energy': Unit(\"GeV\"),\n",
+       "    'length': Unit(\"cm\"),\n",
+       "    'time': Unit(\"s\")}}},\n",
+       " 'datafields': {'run': 4,\n",
+       "  'ra': 4,\n",
+       "  'dec': 4,\n",
+       "  'ang_err': 4,\n",
+       "  'time': 4,\n",
+       "  'log_energy': 4,\n",
+       "  'true_ra': 8,\n",
+       "  'true_dec': 8,\n",
+       "  'true_energy': 8,\n",
+       "  'mcweight': 8},\n",
+       " 'caching': {'pdf': {'MultiDimGridPDF': False}}}"
+      ]
+     },


tomaskontrimas · 2026-05-21T09:18:52Z

+The tutorials below walk through common SkyLLH analysis tasks using IceCube public data.
+They progress from a basic steady-state point-source fit to more specialised techniques.
+
+The tutorials support both the `10-year <https://doi.org/10.7910/DVN/VKL316>`_ (IceTracks-DR1) and the `14-year <https://doi.org/10.7910/DVN/MMIIZA>`_ (IceTracks-DR2) IceCube public point-source datasets. They are automatically downloaded from `dataverse.harvard.edu <https://dataverse.harvard.edu/dataverse/icecube>`_ to a local cache directory (``~/.skyllh/cache``). To use custom dataset locations, set the ``cfg['repository']['base_path']`` to the desired path.


Fixed in 596cf83

tomaskontrimas · 2026-05-21T09:18:39Z

+  {
+   "cell_type": "markdown",
+   "id": "e8921c82",
+   "metadata": {},
+   "source": [
+    "The datasets are automatically downloaded from [dataverse.harvard.edu](https://dataverse.harvard.edu/dataverse/icecube) to a local cache directory (`~/.skyllh/cache`) on first run. To use a custom dataset location, set `cfg['repository']['base_path']` to the desired path."
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "d6a539f5",
+   "metadata": {
+    "raw_mimetype": "text/restructuredtext"
+   },
+   "source": [
+    ".. note::\n",
+    "    IceTracks-DR2 is a pretty heavy dataset (~5GB) and it might take a while to initialize the analysis when it has to be downloaded for the first time. Be patient, and the next time it will be loaded from `~/.skyllh/cache` :)"
+   ]


Fixed in 596cf83

tomaskontrimas

🚀

tomaskontrimas and others added 30 commits November 7, 2024 03:28

Add dataset definition

92a62c4

Add temp notebook

7a1a9bb

Add custom irfs path.

ebf75a1

Change basepath

469adae

Save the script used to link irfs folders for testing

82a6c37

Setting the default max gamma to 4.0

e5cfd31

Fix for missing reco energies in some true neutrino energy bins.

255cabf

Merge branch 'data_release' of github.com:icecube/skyllh into data_re…

74e3358

…lease

Add warning about fitting gamma values beyond 4.

5bb6f2b

Fix for missing reco energies in some true neutrino energy bins.

98671cc

Update 14y dataset collection name

b0da467

Change to the energy spline creation method for flexibility

9a83c8e

Adapt the rest to work with the new energy spline

0914f78

Bug fix.

44046dd

Bug fix

8dabba2

Check validity of lowest energy bin for detection probability calcula…

96c8f2a

…tion

change dataset energy bins to match csky

231ef25

Fix IC79 binning definitions for bkg PDF construction

8fa0edf

Spline bkg histogragram for SoB ratio

6958d7c

Backward compatibility fixes

8ccb49b

Revert "Backward compatibility fixes"

6b51e19

This reverts commit 8ccb49b.

Make old dataset future-compatible

690ce7d

Merge branch 'data_release' of github.com:icecube/skyllh into data_re…

0398b57

…lease

Merge remote-tracking branch 'origin/master' into data_release

fb08766

Merge remote-tracking branch 'origin/master' into data_release

485ab02

Merge remote-tracking branch 'origin' into data_release

cc77cd2

remove energy PDF ratio cap_ratio for zero bkg PDF values

89924ad

Updated tutorials

6eff13f

made changes

2bec6a6

removed unneccessary files

6218051

tomaskontrimas temporarily deployed to github-pages May 19, 2026 12:07 — with GitHub Actions Inactive

fix some more notebooks

f8039b5

chiarabellenghi temporarily deployed to github-pages May 19, 2026 13:55 — with GitHub Actions Inactive

tomaskontrimas and others added 2 commits May 19, 2026 16:37

Remove .zip file if it appears to be corupted

be1e90e

fix the sky scan notebook

cb32cb4

chiarabellenghi temporarily deployed to github-pages May 19, 2026 14:51 — with GitHub Actions Inactive

chiarabellenghi and others added 2 commits May 19, 2026 17:58

Remove sm/effA notebook

2992a04

Remove sm/effA notebook link

afd81b5

tomaskontrimas temporarily deployed to github-pages May 19, 2026 16:12 — with GitHub Actions Inactive

Add links to 14 year dataset

7c823d7

tomaskontrimas temporarily deployed to github-pages May 20, 2026 07:47 — with GitHub Actions Inactive

chiarabellenghi marked this pull request as ready for review May 21, 2026 08:29

Copilot AI review requested due to automatic review settings May 21, 2026 08:29

Copilot started reviewing on behalf of chiarabellenghi May 21, 2026 08:30 View session

Use IceTracks-DR2 in all tutorials

53f9cd7

Copilot AI reviewed May 21, 2026

View reviewed changes

chiarabellenghi temporarily deployed to github-pages May 21, 2026 08:43 — with GitHub Actions Inactive

Fix time dep dataset

808b95d

chiarabellenghi temporarily deployed to github-pages May 21, 2026 09:18 — with GitHub Actions Inactive

Fix wrong default cache path in text

596cf83

tomaskontrimas temporarily deployed to github-pages May 21, 2026 09:27 — with GitHub Actions Inactive

Fix exception handler and log errors of os.remove

eb406ad

tomaskontrimas temporarily deployed to github-pages May 21, 2026 10:19 — with GitHub Actions Inactive

Merge branch 'master' into documentation

20379aa

tomaskontrimas temporarily deployed to github-pages May 21, 2026 15:12 — with GitHub Actions Inactive

rerun all notebooks

403d188

tomaskontrimas approved these changes May 21, 2026

View reviewed changes

chiarabellenghi temporarily deployed to github-pages May 21, 2026 15:54 — with GitHub Actions Inactive

tomaskontrimas merged commit b7edc05 into master May 21, 2026
7 checks passed

tomaskontrimas deleted the documentation branch May 21, 2026 15:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated documentation for the upcoming data release#294

Updated documentation for the upcoming data release#294
tomaskontrimas merged 125 commits into
masterfrom
documentation

chiarabellenghi commented Apr 8, 2026 •

edited by tomaskontrimas

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

tomaskontrimas May 21, 2026

Uh oh!

tomaskontrimas May 21, 2026

Uh oh!

tomaskontrimas May 21, 2026

Uh oh!

tomaskontrimas May 21, 2026

Uh oh!

tomaskontrimas May 21, 2026

Uh oh!

tomaskontrimas left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

chiarabellenghi commented Apr 8, 2026 • edited by tomaskontrimas Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

tomaskontrimas May 21, 2026

Choose a reason for hiding this comment

Uh oh!

tomaskontrimas May 21, 2026

Choose a reason for hiding this comment

Uh oh!

tomaskontrimas May 21, 2026

Choose a reason for hiding this comment

Uh oh!

tomaskontrimas May 21, 2026

Choose a reason for hiding this comment

Uh oh!

tomaskontrimas May 21, 2026

Choose a reason for hiding this comment

Uh oh!

tomaskontrimas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chiarabellenghi commented Apr 8, 2026 •

edited by tomaskontrimas

Loading