Document #991

- Reorder, expand, and copyedit docs of default reporter contents.
- Add #988, #989 to release notes.

Author: khaeru

RELEASE_NOTES.rst

@@ -47,6 +47,11 @@ All changes
   - :mod:`message_ix.message` includes :class:`.MESSAGE`.
   - :mod:`message_ix.message_macro` includes :class:`.MESSAGE_MACRO`.
 
+- Improve :class:`.Reporter` and its documentation (:pull:`991`).
+
+  - Handle older scenarios—for instance, those without |input_cap|—in :meth:`.Reporter.from_scenario` (:issue:`988`).
+  - Expand and test keys for multiple methods of calculating :ref:`reporter-historical` (:issue:`989`).
+
 - Document the :ref:`minimum version of Java <install-java>` required for :class:`ixmp.JDBCBackend <ixmp.backend.jdbc.JDBCBackend>` (:pull:`962`).
 - Document :ref:`how to run a local PostgreSQL instance <install-postgres>`
   for local testing using :class:`ixmp.IXMP4Backend <ixmp.backend.ixmp4.IXMP4Backend>` (:pull:`981`).

doc/reporting.rst

@@ -21,7 +21,7 @@ Each layer of this “stack” builds on the features in the level below:
      - Calculations for specific technologies
    * - ``message_ix``
      - Energy model framework
-     - Common sets/parameters (``output``)
+     - MESSAGE-specific sets/parameters (``output``)
      - Derived quantities (``tom``)
    * - ``ixmp``
      - Optimization models & data
@@ -37,7 +37,7 @@ Each layer of this “stack” builds on the features in the level below:
 These features are accessible through :class:`.Reporter`, which can produce multiple **reports** from one or more Scenarios.
 A report and the quantities that enter it is identified by a **key**, and may…
 
-- perform arbitrarily complex calculations while intelligently handling units;
+- perform arbitrarily complex calculations, handling dimensionality and units;
 - read and make use of data that is ‘exogenous’ to (not included in) a Scenario;
 - produce output as Python or R objects (in code), or write to files or databases;
 - calculate only a requested subset of quantities; and
@@ -57,10 +57,16 @@ See :doc:`genno:usage` in the genno documentation for an introduction to concept
 In :mod:`message_ix.report`:
 
 - The :class:`.Reporter` class is an extended version of the :class:`genno.Computer` class.
-- :mod:`ixmp` parameters, scalars, equations, and time-series data all become quantities for the purpose of reporting.
-- For example, the |MESSAGEix| parameter ``resource_cost``, defined with the dimensions (node `n`, commodity `c`, grade `g`, year `y`) is identified by the key ``resource_cost:n-c-g-y``.
-  When summed across the grade/`g` dimension, it has dimensions `n`, `c`, `y` and is identified by the key ``resource_cost:n-c-y``.
-- :meth:`.Reporter.from_scenario` automatically sets up keys and tasks (such as ``resource_cost:n-c-g-y``) that simply retrieve raw/unprocessed data from a :class:`~message_ix.Scenario` and return it as a :any:`genno.Quantity`.
+- :mod:`ixmp` parameters, scalars, equations, and time-series data
+  all become ‘quantities’ for the purpose of reporting.
+- For example, the |MESSAGEix| parameter ``resource_cost``
+  defined with the dimensions (node `n`, commodity `c`, grade `g`, year `y`)
+  is identified by the key ``resource_cost:n-c-g-y``.
+  When summed across the grade/`g` dimension, it has dimensions `n`, `c`, `y`
+  and is identified by the key ``resource_cost:n-c-y``.
+- :meth:`.Reporter.from_scenario` automatically sets up keys and tasks
+  (such as ``resource_cost:n-c-g-y``) that simply retrieve raw/unprocessed data from a :class:`~message_ix.Scenario`
+  and return it as a :any:`genno.Quantity`.
 - Operators are defined as functions in modules including:
   :mod:`message_ix.report.operator`,
   :mod:`ixmp.report.operator`, and
@@ -72,10 +78,14 @@ Usage
 
 A |MESSAGEix| reporting workflow has the following steps:
 
-1. Obtain a :class:`.Scenario` object from an :class:`~ixmp.Platform`.
-2. Use :meth:`.Reporter.from_scenario` to prepare a Reporter object with many calculations automatically prepared.
-3. (optionally) Use the built-in features of :class:`.Reporter` to describe additional calculations.
-4. Use :meth:`.get` 1 or more times to execute tasks, including all the calculations on which they depend:
+1. Obtain a :class:`.Scenario` object from a :class:`~ixmp.Platform`.
+2. Use :meth:`.Reporter.from_scenario` to prepare a Reporter object
+   with many calculations automatically prepared.
+   See below.
+3. (optionally) Use the built-in features of :class:`.Reporter`
+   to describe additional calculations.
+4. Use :meth:`.get` 1 or more times to execute tasks,
+   including all the calculations on which they depend:
 
 .. code-block:: python
 
@@ -88,12 +98,173 @@ A |MESSAGEix| reporting workflow has the following steps:
     rep.get("all")
 
 Note that keys and tasks are **described** in steps (2–3), but they are not **executed** until :meth:`.get` is called—or the results of one task are required by another.
-This design allows the Reporter to skip unneeded (and potentially slow) computations and deliver good performance.
+This design allows the Reporter to skip unneeded (and potentially slow)
+tasks and re-use intermediate results to deliver good performance.
 The Reporter's :attr:`~genno.Computer.graph` may contain thousands of tasks for retrieving model quantities and calculating derived quantities, but a particular call to :meth:`.get` may only execute a few of these.
 
+.. _reporter-default:
+
+Default reporter contents
+-------------------------
+
+:meth:`message_ix.Reporter.from_scenario <.Reporter.from_scenario>`
+returns a new Reporter instance pre-filled with many keys and tasks
+based on the contents of the :class:`.Scenario` argument.
+
+These include the contents added by 
+:meth:`ixmp.Reporter.from_scenario <ixmp.report.Reporter.from_scenario>`
+—that is, every :mod:`ixmp` set, parameter, variable, and equation available
+in the Scenario—
+plus additional keys for derived quantities specific to the MESSAGE model structure.
+These automatic contents are prepared using:
+
+.. autosummary::
+
+   ~message_ix.report.TASKS0
+   ~message_ix.report.PYAM_CONVERT
+   ~message_ix.report.TASKS1
+   ~message_ix.report.get_tasks
+
+…and include the following:
+
+Sets
+   - Keys matching the standard short symbols for |MESSAGEix| sets/dimensions,
+     according to  :data:`.common.DIMS`.
+     Thus for instance:
+
+     .. code-block:: python
+
+        rep.get("n")
+
+     …returns a Python :class:`list`
+     with the elements of the |MESSAGEix| set named "node".
+     These lists can be used as input to other tasks,
+     or to construct data structures for indexing, aggregation, and other operators.
+   - ``y0``: the ``firstmodelyear`` or :math:`y_0` (:class:`int`).
+   - ``y::model``, :class:`list` of :class:`int`:
+     only the periods in the `year` set (``y``/:math:`Y`)
+     that are equal to or greater than ``y0``.
+   - ``map_<name>``: "one-hot" or indicator quantities
+     for the respective |MESSAGEix| mapping sets ``cat_<name>``.
+
+Model solution data
+   The following quantities combine |MESSAGEix| parameter data (exogenous)
+   and variable data (endogenous; the optimal solution to the MESSAGE linear program)
+   in commonly-used ways.
+
+   Each is available in its full dimensionality
+   (the union of the dimensions of its operands)
+   and as partial sums over all combinations of 1 or more dimensions.
+   Use :meth:`~genno.Computer.full_key` to retrieve the full-dimensionality
+   :class:`~genno.Key` for any of these quantities.
+
+   - ``in``              = ``input`` × ``ACT``;
+     that is, the product of ``input`` (input intensity) and ``ACT`` (activity).
+   - ``out``             = ``output`` × ``ACT``
+   - ``emi``             = ``emission_factor`` × ``ACT``
+   - ``inv``             = ``inv_cost`` × ``CAP_NEW``
+   - ``fom``             = ``fix_cost`` × ``CAP``.
+     The name is an abbreviation for "Fixed Operation and Maintenance costs".
+   - ``vom``             = ``var_cost`` × ``ACT``,
+     "Variable Operation and Maintenance costs".
+   - ``tom``             = ``fom`` + ``vom``,
+     "Total Operation and Maintenance costs".
+   - ``land_out``        = ``land_output`` × ``LAND``
+   - ``land_use_qty``    = ``land_use`` × ``LAND``
+   - ``land_emi``        = ``land_emission`` × ``LAND``
+   - ``addon conversion``,
+     the model parameter ``addon_conversion`` (note space versus underscore),
+     except broadcast across individual add-on technologies (`ta`)
+     rather than add-on types (`type_addon`).
+   - ``addon up``, which is ``addon_up`` similarly broadcast.
+   - ``addon ACT``       = ``addon conversion`` × ``ACT``
+   - ``addon in``        = ``input`` × ``addon ACT``
+   - ``addon out``       = ``output`` × ``addon ACT``
+   - ``addon potential`` = ``addon up`` × ``addon ACT``,
+     the maximum potential activity by add-on technology.
+   - ``price emission``,
+     the model variable ``PRICE_EMISSION``
+     broadcast across emission species (`e`) *and* technologies (`t`)
+     rather than types (`type_emission`, `type_tec`).
+
+.. _reporter-historical:
+
+Historical and reference values
+   The following quantities combine |MESSAGEix| parameter data.
+   Because they do not include variable data,
+   they are entirely exogenous and do not depend on the model solution.
+   Their names correspond to the keys above:
+   for example, ``in:*:historical+current`` and ``in:*:historical+weighted``
+   correspond to ``in:*``: the latter is *within* the model time horizon,
+   and the former are *before* the first model period.
+
+   - ``in:*:historical+full``  = ``input`` × ``historical_activity``.
+     This is an intermediate quantity,
+     and in most cases should not be used directly.
+     It is used to calculate ``{…}:*:historical+current``
+     and ``{…}:*:historical+weighted``. [#key-notation]_
+
+     Similar quantities exist for ``out``, ``emi``, and ``inv``,
+     with keys like ``out:*:historical+full`` etc.
+   - ``in:*:historical+current``:
+     The subset of ``in:*:historical+full`` where :math:`y^V = y^A`.
+     Use this quantity to represent an assumption that
+     *all activity in each historical period* :math:`y^A < y_0` *is attributable to
+     the technology vintage constructed in the same ('current') period*.
+     In other words, this uses ``input`` intensity values for :math:`(y^V=y^A, y^A)`,
+     and omits/ignores values for earlier vintages  :math:`(y^V < y^A, y^A)`,
+     even if the |technical_lifetime| and |historical_new_capacity| of those earlier vintages
+     implies they could be active in the period |yA|.
+
+     Similar quantities exist for ``out``, ``emi``, and ``inv``.
+   - ``in:*:historical+weighted``:
+     a sum of ``in:*:historical+full`` across the ``yv`` dimension,
+     weighted by ``share:*:in+historical``.
+     The latter quantity **must** be supplied exogenously by the user.
+     This can be done by overriding the placeholder task created by
+     :meth:`.from_scenario`:
+
+     .. code-block:: python
+
+        k = rep.full_key("share:*:in+historical")
+        rep.add(k, ...)  # Some task that returns share values, e.g. from file
+
+     ``share:*:{…}+historical`` values give the fractional contribution
+     of each technology vintage :math:`y^V`
+     to the ``historical_activity`` in each :math:`y^A`.
+     Therefore they **should** sum to 1.0 across the ``yv`` dimension
+     for each combination of other indices.
+
+     Similar quantities exist for ``out``, ``emi``, and ``inv``.
+   - ``in:*:ref+current``, ``in:*:ref+weighted``, and ``in:*:ref+full``:
+     Same as above, but computed using ``ref_activity`` instead of ``historical_activity``.
+
+     Similar quantities exist for ``out``, ``emi``, and ``inv``.
+
+.. [#key-notation] These string representations of :class:`genno.Key` use
+   an asterisk for the dimensions (such as ``name:*:tag``)
+   to indicate “all dimensions for this ``name`` and ``tag``”;
+   and ``{…}`` to indicate “any string”.
+
+.. _default-reports:
+
+Time series data (:data:`.PYAM_CONVERT`, :data:`.TASKS1`)
+   Tasks that transform :class:`~genno.Quantity` with varying number of dimensions
+   to the :mod:`ixmp` :ref:`structure for time-series data <ixmp:data-tsdata>`
+   (also called the “IAMC data structure”),
+   specifically as instances of :class:`pyam.IamDataFrame`.
+
+   These include:
+
+   - ``<name>::pyam`` for most of the above derived quantities.
+   - ``CAP::pyam`` (from ``CAP``)
+   - ``CAP_NEW::pyam`` (from ``CAP_NEW``)
+   - ``message::system``, ``message::costs``, and ``message::emissions``:
+     concatenation of subsets of the above time series data.
+   - ``message::default``: concatenation of all of the above time series data.
 
 Customization
-=============
+-------------
 
 A Reporter prepared with :meth:`.from_scenario` always contains a key
 ``scenario``, referring to the Scenario to be reported.
@@ -112,8 +283,12 @@ The method :meth:`.Reporter.add` can be used to add *arbitrary* Python code that
     rep.add("custom", (my_custom_report, "scenario"))
     rep.get("custom")
 
-In this example, the function ``my_custom_report()`` *could* run to thousands of lines; read to and write from multiple files; invoke other programs or Python scripts; etc.
-In order to take advantage of the performance-optimizing features of the Reporter, such calculations can instead be composed from atomic (i.e. small, indivisible) operators or functions.
+In this example, the function ``my_custom_report()`` **may** run to thousands of lines;
+read to and write from multiple files;
+invoke other programs or Python scripts; etc.
+In order to take advantage of the performance-optimizing features of the Reporter,
+such calculations **should** instead be composed from atomic (small, indivisible) operators
+or functions.
 See the :mod:`genno` documentation for more.
 
 API reference
@@ -146,66 +321,6 @@ Their documentation is repeated below for convenience.
    configure
 .. currentmodule:: message_ix.report
 
-:meth:`ixmp.Reporter.from_scenario <ixmp.report.Reporter.from_scenario>` automatically adds keys based on the contents of the :class:`.Scenario` argument;
-that is, every :mod:`ixmp` set, parameter, variable, and equation available in the Scenario.
-:meth:`message_ix.Reporter.from_scenario <.Reporter.from_scenario>` extends this to add additional keys for derived quantities specific to the MESSAGEix model framework.
-These include:
-
-.. tip:: Use :meth:`~genno.Computer.full_key` to retrieve the full-dimensionality :class:`~genno.Key` for any of these quantities.
-
-- ``out``          = ``output`` × ``ACT``; that is, the product of ``output`` (output efficiency) and ``ACT`` (activity)
-- ``out_hist``     = ``output`` × ``ref_activity`` (historical reference activity)
-- ``in``           = ``input`` × ``ACT``
-- ``in_hist``      = ``input`` × ``ref_activity``
-- ``emi``          = ``emission_factor`` × ``ACT``
-- ``emi_hist``     = ``emission_factor`` × ``ref_activity``
-- ``inv``          = ``inv_cost`` × ``CAP_NEW``
-- ``inv_hist``     = ``inv_cost`` × ``ref_new_capacity``
-- ``fom``          = ``fix_cost`` × ``CAP``; the name refers to "Fixed Operation and Maintenance costs"
-- ``fom_hist``     = ``fix_cost`` × ``ref_capacity``
-- ``vom``          = ``var_cost`` × ``ACT``; "Variable Operation and Maintenance costs"
-- ``vom_hist``     = ``var_cost`` × ``ref_activity``
-- ``tom``          = ``fom`` + ``vom``; "Total Operation and Maintenance costs"
-- ``land_out``     = ``land_output`` × ``LAND``
-- ``land_use_qty`` = ``land_use`` × ``LAND``
-- ``land_emi``     = ``land_emission`` × ``LAND``
-- ``addon conversion``, the model parameter ``addon_conversion`` (note space versus underscore), except broadcast across individual add-on technologies (`ta`) rather than add-on types (`type_addon`).
-- ``addon up``, which is ``addon_up`` similarly broadcast.
-- ``addon ACT``    = ``addon conversion`` × ``ACT``
-- ``addon in``     = ``input`` × ``addon ACT``
-- ``addon out``    = ``output`` × ``addon ACT``
-- ``addon potential`` = ``addon up`` × ``addon ACT``, the maximum potential activity by add-on technology.
-- ``price emission``, the model variable ``PRICE_EMISSION`` broadcast across emission species (`e`) *and* technologies (`t`) rather than types (`type_emission`, `type_tec`).
-
-Other added keys include:
-
-- :mod:`message_ix` adds the standard short symbols for |MESSAGEix| dimensions (sets) based on :data:`.common.DIMS`.
-  Each of these is also available in a Reporter: for example :py:`rep.get("n")` returns a list with the elements of the |MESSAGEix| set named "node";
-  :py:`rep.get("t")` returns the elements of the set "technology", and so on.
-  These keys can be used as input to other computations.
-- ``y0`` = the ``firstmodelyear`` or :math:`y_0` (:class:`int`).
-- ``y::model`` = only the periods in the `year` set (``y``) that are equal to or greater than ``y0``.
-
-.. _default-reports:
-
-- Computations to convert internal :class:`~genno.Quantity` data format to the IAMC data format, specifically as :class:`pyam.IamDataFrame` objects.
-  These include:
-
-  - ``<name>::pyam`` for most of the above derived quantities.
-  - ``CAP::pyam`` (from ``CAP``)
-  - ``CAP_NEW::pyam`` (from ``CAP_NEW``)
-
-- ``map_<name>`` as "one-hot" or indicator quantities for the respective |MESSAGEix| mapping sets ``cat_<name>``.
-- Standard reports ``message::system``, ``message::costs``, and ``message::emissions`` per :data:`TASKS1`.
-- The report ``message::default``, collecting all of the above reports.
-
-These automatic contents are prepared using:
-
-.. autosummary::
-
-   TASKS0
-   PYAM_CONVERT
-   TASKS1
 
 .. autoclass:: Reporter
    :show-inheritance: