Merge pull request #178 from iiasa/enh/test-report

Test genno vs. legacy reporting

Author: khaeru

doc/api/report/index.rst

@@ -9,7 +9,9 @@ On this page:
 Elsewhere:
 
 - ``global.yaml``, the :doc:`default-config`.
-- Documentation for :mod:`genno` (:doc:`genno:index`), :mod:`ixmp.report`, and :mod:`message_ix.report`.
+- Documentation for :mod:`genno` (:doc:`genno:index`),
+  :mod:`ixmp.report`, and
+  :mod:`message_ix.report`.
 - Reporting for specific model variants:
 
   - :mod:`.water.reporting`
@@ -31,41 +33,85 @@ Introduction
 See :doc:`the discussion in the MESSAGEix docs <message-ix:reporting>` about the stack.
 In short, for instance:
 
-- :mod:`message_ix` **must not** contain reporting code that references :py:`technology="coal_ppl"`, because not every model built on the MESSAGE framework will have a technology with this name.
-- Any model in the MESSAGEix-GLOBIOM family—built with :mod:`message_ix_models` and/or :mod:`message_data`—**should**, with few exceptions, have a :py:`technology="coal_ppl"`, since this appears in the common list of :ref:`technology-yaml`.
-  Reporting specific to this technology ID, *as it is represented* in this model family, should be in :mod:`message_ix_models` or user code.
+- :mod:`message_ix` **must not** contain reporting code that references :py:`technology="coal_ppl"`,
+  because not every model built on the MESSAGE framework will have a technology with this name.
+- Any model in the MESSAGEix-GLOBIOM family
+  —built with :mod:`message_ix_models` and/or :mod:`message_data`—
+  **should**, with few exceptions, have a :py:`technology="coal_ppl"`,
+  since this appears in the common list of :ref:`technology-yaml`.
+  Reporting specific to this technology ID,
+  *as it is represented* in this model family,
+  should be in :mod:`message_ix_models` or user code.
 
 The basic **design pattern** of :mod:`message_ix_models.report` is:
 
-- :func:`~.report.prepare_reporter` populates a new :class:`~.message_ix.Reporter` for a given :class:`.Scenario` with many keys to report all quantities of interest in a MESSAGEix-GLOBIOM–family model.
-- This function relies on *callbacks* defined in multiple submodules to add keys and tasks for general or tailored reporting calculations and actions.
-  Additional modules **should** define callback functions and register them with :func:`~report.register` when they are to be used.
+- :func:`~.report.prepare_reporter` populates a new :class:`~.message_ix.Reporter`
+  for a given :class:`.Scenario` with many tasks
+  to report all quantities of interest in a MESSAGEix-GLOBIOM–family model.
+- This function relies on *callbacks* defined in multiple submodules
+  to add keys and tasks for general or tailored reporting calculations and actions.
+  Additional modules **should** define callback functions
+  and register them with :func:`~report.register` when they are to be used.
   For example:
 
-  1. The module :mod:`message_ix_models.report.plot` defines :func:`.plot.callback` that adds standard plots to the Reporter.
-  2. The module :mod:`message_data.model.transport.report` defines :func:`~.message_data.model.transport.report.callback` that adds tasks specific to MESSAGEix-Transport.
-  3. The module :mod:`message_data.projects.navigate.report` defines :func:`~.message_data.projects.navigate.report.callback` that add tasks specific to the ‘NAVIGATE’ research project.
-
-  The callback (1) is always registered, because these plots are always applicable and can be expected to function correctly for all models in the family. In contrast, (2) and (3) **should** only be registered and run for the specific model variants for which they are developed/intended.
-
-  Modules with tailored reporting configuration **may** also be indicated on the :ref:`command line <report-cli>` by using the :program:`-m/--modules` option: :program:`mix-models report -m model.transport`.
-
-- A file :file:`global.yaml` file (in `YAML <https://en.wikipedia.org/wiki/YAML#Example>`_ format) contains a description of some of the reporting computations needed for a MESSAGE-GLOBIOM model.
-  :func:`~.report.prepare_reporter` uses the :doc:`configuration handlers <genno:config>` built into :mod:`genno` (and some extensions specific to :mod:`message_ix_models`) to handle the different sections of the file.
+  1. The module :mod:`message_ix_models.report.plot` defines :func:`.plot.callback`
+     that adds standard plots to the Reporter.
+  2. The module :mod:`message_ix_models.model.transport.report` defines :func:`~.transport.report.callback`
+     that adds tasks specific to the MESSAGEix-Transport model variant.
+  3. The module :mod:`message_ix_models.project.navigate.report` defines :func:`~.navigate.report.callback`
+       that add tasks specific to the ‘NAVIGATE’ research project.
+
+  The callback (1) is always registered,
+  because these plots are always applicable and can be expected to function correctly
+  for all models in the family.
+  In contrast, (2) and (3) **should** only be registered and run
+  for the specific model variants for which they are developed/intended.
+
+  Modules with tailored reporting configuration **may** also be indicated
+  on the :ref:`command line <report-cli>`
+  by using the :program:`-m/--modules` option:
+  :program:`mix-models report -m model.transport`.
+
+- A file :file:`global.yaml`
+  (in `YAML <https://en.wikipedia.org/wiki/YAML#Example>`_ format)
+  contains a description of some of the reporting computations
+  needed for a MESSAGE-GLOBIOM model.
+  :func:`~.report.prepare_reporter` uses the :doc:`configuration handlers <genno:config>`
+  built into :mod:`genno`
+  (and some extensions specific to :mod:`message_ix_models`)
+  to handle the different sections of the file.
 
 Features
 ========
 
-By combining these genno, ixmp, message_ix, and message_ix_models features, the following functionality is provided.
+By combining these genno, ixmp, message_ix, and message_ix_models features,
+the following functionality is provided.
 
 .. note:: If any of this does not appear to work as advertised, file a bug!
 
+Scope of :mod:`genno`-based vs. :mod:`.legacy` reporting
+--------------------------------------------------------
+
+:func:`.prepare_reporter` currently handles only a subset
+of the IAMC-structured data produced by :mod:`.report.legacy`.
+The module globals :data:`.NOT_IMPLEMENTED_MEASURE` and :mod:`NOT_IMPLEMENTED_IAMC`
+express the measures and IAMC ‘variables’ that are either not implemented
+or not exactly matching the legacy reporting output.
+The test :func:`.test_report.test_compare` compares outputs
+from the two forms of reporting using :func:`.iamc.compare`,
+ensuring that data match for all entries *except* these exclusions.
+
+See GitHub issues and pull requests with the
+`'report' label <https://github.com/iiasa/message-ix-models/issues?q=state%3Aopen%20label%3Areport>`_
+for ongoing work to expand the scope of :func:`prepare_reporter`.
+
 Units
 -----
 
 - Are read automatically for ixmp parameters.
 - Pass through calculations/are derived automatically.
-- Are recognized based on the definitions of non-SI units from `IAMconsortium/units <https://github.com/IAMconsortium/units/>`_.
+- Are recognized based on the definitions of non-SI units from
+  `IAMconsortium/units <https://github.com/IAMconsortium/units/>`_.
 - Are discarded when inconsistent.
 - Can be overridden for entire parameters:
 
@@ -133,7 +179,9 @@ Operators
       remove_ts
       share_curtailment
 
-   The following functions, defined elsewhere, are exposed through :mod:`.operator` and so can also be referenced by name:
+   The following functions, defined elsewhere,
+   are exposed through :mod:`.operator`
+   and so can also be referenced by name:
 
    .. autosummary::
 
@@ -255,11 +303,20 @@ Testing
 Continuous reporting
 --------------------
 
-As part of the :ref:`test-suite`, reporting is run on the same events (pushes and daily schedule) on publicly-available :doc:`model snapshots </api/model-snapshot>`.
-One goal of these tests *inter alia* is to ensure that adjustments and improvements to the reporting code do not disturb manually-verified model outputs.
-
-As part of the (private) :mod:`message_data` test suite, multiple workflows run on regular schedules; some of these include a combination of :mod:`message_ix_models`-based and :ref:`‘legacy’ reporting <report-legacy>`.
+As part of the :ref:`test-suite`, reporting is run on the same events
+(pushes and daily schedule)
+on publicly-available :doc:`model snapshots </api/model-snapshot>`.
+One goal of these tests *inter alia* is to ensure that adjustments
+and improvements to the reporting code
+do not disturb manually-verified model outputs.
+
+As part of the (private) :mod:`message_data` test suite,
+multiple workflows run on regular schedules;
+some of these include a combination of :mod:`message_ix_models`-based
+and :ref:`‘legacy’ reporting <report-legacy>`.
 These workflows:
 
 - Operate on specific scenarios within IIASA databases.
-- Create files in CSV, Excel, and/or PDF formats that are that are preserved and made available as 'build artifacts' via the GitHub Actions web interface and API.
+- Create files in CSV, Excel, and/or PDF formats
+  that are that are preserved and made available as 'build artifacts'
+  via the GitHub Actions web interface and API.

doc/whatsnew.rst

@@ -1,8 +1,19 @@
 What's new
 **********
 
-.. Next release
-.. ============
+Next release
+============
+
+- Improvements to :class:`.report.Config` (:pull:`178`):
+
+  - New attribute :attr:`~.report.Config.modules`.
+  - New method :meth:`~.report.Config.iter_callbacks`.
+
+- New function :func:`.tools.iamc.compare` (:pull:`178`).
+- Expand :doc:`api/report/index` documentation (:pull:`178`)
+  to cover features implemented/not implemented by :mod:`genno`-based reporting.
+  Module globals :data:`.NOT_IMPLEMENTED_MEASURE` and :data:`.NOT_IMPLEMENTED_IAMC`
+  record not-yet-implemented measures and IAMC ‘variables’.
 
 v2025.10.31
 ===========

message_ix_models/cli.py

@@ -57,7 +57,7 @@ def main(click_ctx, **kwargs):
 
     # Check for a non-trivial execution of the CLI
     non_trivial = (
-        not any(s in sys.argv for s in {"last-log", "--help"})
+        not any(s in sys.argv for s in {"config", "last-log", "--help"})
         and click_ctx.invoked_subcommand != "_test"
         and "pytest" not in sys.argv[0]
     )

message_ix_models/data/report/global.yaml

@@ -685,7 +685,7 @@ general:
 # All other technologies not in out::h2
 - key: out:*:se_0
   comp: select
-  inputs: [out]
+  inputs: [ out ]
   args:
     indexers:
       t: [h2_coal, h2_coal_ccs, h2_smr, h2_smr_ccs, h2_bio, h2_bio_ccs]
@@ -862,12 +862,12 @@ iamc:
 #   <<: *pe_iamc
 
 - variable: Primary Energy|Hydro
-  base: out:nl-t-ya-m-c-l:se
+  base: out:nl-t-ya-m-c-l:se_1+se
   select: {l: [secondary], t: [hydro]}
   <<: *pe_iamc
 
 - variable: Primary Energy|Nuclear
-  base: out:nl-t-ya-m-c-l:se
+  base: out:nl-t-ya-m-c-l:se_1+se
   select: {l: [secondary], t: [nuclear]}
   <<: *pe_iamc
 
@@ -1043,7 +1043,7 @@ iamc:
 report:
 - key: pe test
   members:
-#  - Primary Energy|Biomass::iamc
+  # - Primary Energy|Biomass::iamc
   - Primary Energy|Coal::iamc
   - Primary Energy|Gas::iamc
   - Primary Energy|Hydro::iamc

message_ix_models/data/report/legacy/default_units.yaml

@@ -41,14 +41,18 @@ conversion_factors:
             ZJ: .00003154
             km3/yr: 1.
             Index (2005 = 1): 1
+        GWyr/yr:
+            EJ/yr: 0.03154
+            GWa: 1.
+            km3/yr: 1.
         EJ/yr:
             ZJ: .001
-        y:
+        "y":
             years: 1.
         # New units from unit-revision
         "Mt C/GWyr/yr":
             Mt CO2/yr: "float(f\"{mu['conv_c2co2']}\")"
-            Mt CO2-equiv/yr: "float(f\"{mu['conv_c2co2']}\")" 
+            Mt CO2-equiv/yr: "float(f\"{mu['conv_c2co2']}\")"
         # Emissions currently have the units ???
         -:
             Mt CO2/yr: "float(f\"{mu['conv_c2co2']}\")"
@@ -57,7 +61,7 @@ conversion_factors:
             # NB this values implies that whatever quantity it is applied to is
             #    internally [Mt C/yr]
             Mt CO2/yr: "float(f\"{mu['conv_c2co2']}\")"
-            Mt CO2-equiv/yr: "float(f\"{mu['conv_c2co2']}\")" 
+            Mt CO2-equiv/yr: "float(f\"{mu['conv_c2co2']}\")"
             # N2O is always left in kt
             kt N2O/yr: 1.
             # All other units are in kt
@@ -139,7 +143,7 @@ conversion_factors:
             Mt C/yr: "float(f\"{mu['conv_co22c']}\")"
         Mt C/yr:
             Mt CO2eq/yr: "float(f\"{mu['conv_c2co2']}\")"
-            Mt CO2/yr: "float(f\"{mu['conv_c2co2']}\")" 
+            Mt CO2/yr: "float(f\"{mu['conv_c2co2']}\")"
             Mt CO2-equiv/yr: "float(f\"{mu['conv_c2co2']}\")"
         Mt CO2/yr:
             Mt CO2/yr: 1.

message_ix_models/data/test/report/snapshot-1.csv.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b096505d79154852870cba8ebe0404a68ac754cd7022d38602444f21156870fc
+size 2884451

message_ix_models/report/__init__.py

@@ -25,6 +25,8 @@
     from .config import Callback
 
 __all__ = [
+    "NOT_IMPLEMENTED_IAMC",
+    "NOT_IMPLEMENTED_MEASURE",
     "Config",
     "prepare_reporter",
     "register",
@@ -46,6 +48,66 @@ def _(c: Reporter, info):
         pass
 
 
+PE0 = r"Primary Energy\|(Coal|Gas|Hydro|Nuclear|Solar|Wind)"
+PE1 = r"Primary Energy\|(Coal|Gas|Solar|Wind)"
+E = (
+    r"Emissions\|CO2\|Energy\|Demand\|Transportation\|Road Rail and Domestic "
+    "Shipping"
+)
+
+#: Measures for which reporting is not implemented. See :data:`NOT_IMPLEMENTED_IAMC`.
+NOT_IMPLEMENTED_MEASURE = {
+    "(Agricultural|Forestry) (Demand|Production)",
+    "Capacity Additions",
+    "Capacity",
+    "Capital Cost",
+    "Carbon Sequestration",
+    "Consumption",
+    r"Cost\|Cost Nodal Net",
+    "Cumulative Capacity",
+    "Efficiency",
+    r"Emissions\|(BC|CF4|CH4|CO|CO2|F-Gases|HFC|Kyoto Gases|N2O|NH3|NOx|OC|SF6|Sulfur)",
+    r"Emissions\|(VOC)",
+    "Fertilizer Use",
+    "Final Energy",
+    "Food Demand",
+    "GDP",
+    "GLOBIOM",
+    "Investment",
+    "Land Cover",
+    "Lifetime",
+    "OM Cost",
+    "Population",
+    "Price",
+    r"Resource\|(Cumulative )?Extraction",
+    "Secondary Energy",
+    "Trade",
+    "Useful Energy",
+    "Water Consumption",
+    "Water Withdrawal",
+    "Yield",
+}
+
+
+#: Expressions for IAMC variable names not implemented by :func:`prepare_reporter`.
+NOT_IMPLEMENTED_IAMC = [
+    # Other 'variable' codes are missing from `obs`
+    rf"variable='({'|'.join(NOT_IMPLEMENTED_MEASURE)})(\|.*)?': no right data",
+    # 'variable' codes with further parts are missing from `obs`
+    f"variable='{PE0}.*': no right data",
+    # For `pe1` (NB: not Hydro or Solar) units and most values differ
+    f"variable='{PE1}.*': units mismatch .*EJ/yr.*'', nan",
+    r"variable='Primary Energy|Coal': 220 of 240 values with \|diff",
+    r"variable='Primary Energy|Gas': 234 of 240 values with \|diff",
+    r"variable='Primary Energy|Solar': 191 of 240 values with \|diff",
+    r"variable='Primary Energy|Wind': 179 of 240 values with \|diff",
+    # For `e` units and most values differ
+    f"variable='{E}': units mismatch: .*Mt CO2/yr.*Mt / a",
+    rf"variable='{E}': 20 missing right entries",
+    rf"variable='{E}': 220 of 240 values with \|diff",
+]
+
+
 @genno.config.handles("iamc")
 def iamc(c: Reporter, info):
     """Handle one entry from the ``iamc:`` config section.
@@ -74,6 +136,11 @@ def iamc(c: Reporter, info):
     # Common
     base_key = Key(info["base"])
 
+    # First part of the 'Variable' name
+    name = info.pop("variable", base_key.name)
+    # Parts (string literals or dimension names) to concatenate into variable name
+    var_parts = info.pop("var", [name])
+
     # Use message_ix_models custom collapse() method
     info.setdefault("collapse", {})
 
@@ -96,17 +163,15 @@ def iamc(c: Reporter, info):
         # TODO allow iterable of str
         dims = dims.split("-")
 
-        label = f"{info['variable']} {'-'.join(dims) or 'full'}"
+        label = f"{name} {'-'.join(dims) or 'full'}"
 
         # Modified copy of `info` for this invocation
         _info = info.copy()
         # Base key: use the partial sum over any `dims`. Use a distinct variable name.
         _info.update(base=base_key.drop(*dims), variable=label)
         # Exclude any summed dimensions from the IAMC Variable to be constructed
         _info["collapse"].update(
-            callback=partial(
-                collapse, var=list(filter(lambda v: v not in dims, info.get("var", [])))
-            )
+            callback=partial(collapse, var=[v for v in var_parts if v not in dims])
         )
 
         # Invoke the genno built-in handler
@@ -115,7 +180,7 @@ def iamc(c: Reporter, info):
         keys.append(f"{label}::iamc")
 
     # Concatenate together the multiple tables
-    c.add("concat", f"{info['variable']}::iamc", *keys)
+    c.add("concat", f"{name}::iamc", *keys)
 
 
 def register(name_or_callback: "Callback | str") -> str | None:
@@ -324,8 +389,11 @@ def prepare_reporter(
     )
     rep.configure(model=deepcopy(context.model))
 
+    # Add a placeholder task to concatenate IAMC-structured data
+    rep.add("all::iamc", "concat")
+
     # Apply callbacks for other modules which define additional reporting computations
-    for callback in context.report.callback:
+    for callback in context.report.iter_callbacks():
         callback(rep, context)
 
     key = context.report.key