Add Zppy compatibility, logging, and other v1.0 features #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft

mkstratos wants to merge 32 commits into main from add-logs

.github/workflows/python-package.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -16,18 +16,28 @@ jobs:
  
        strategy:

          fail-fast: false

          matrix:

            python-version: ["3.10", "3.11", "3.12", "3.13",]

            include:

              - python-version: "3.10"

                pixi-env: py310dev

              - python-version: "3.11"

                pixi-env: py311dev

              - python-version: "3.12"

                pixi-env: py312dev

              - python-version: "3.13"

                pixi-env: py313dev

              - python-version: "3.14"

                pixi-env: py314dev

        steps:

        - uses: actions/checkout@v4

        - name: Set up Python ${{ matrix.python-version }}

          uses: actions/setup-python@v5

        - uses: actions/checkout@v6

        - name: Set up Pixi ${{ matrix.python-version }}

          uses: prefix-dev/setup-pixi@v0.9.4

          with:

            python-version: ${{ matrix.python-version }}

        - name: Install dependencies

            pixi-version: v0.62.2

            # Project environment caching uses pixi.lock as the cache key input.

            # If you don't commit a lockfile, disable caching to avoid ENOENT.

            # cache: ${{ hashFiles('pixi.lock') != '' }}

            environments: ${{ matrix.pixi-env }}

        - name: Run tests

          run: |

            python3 -m pip install --upgrade pip

            python3 -m pip install -e .\[dev\]

        - name: Test with pytest

          run: |

            pytest -vvv

            pixi run -e ${{ matrix.pixi-env }} pytest -vvv

.pre-commit-config.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -10,6 +10,7 @@ repos: @@
           - id: debug-statements
           - id: destroyed-symlinks
           - id: detect-private-key
       - repo: https://github.com/astral-sh/ruff-pre-commit
         # Ruff version.
         rev: v0.13.0
@@ Expand All / @@ -18,7 +19,3 @@ repos: @@
           - id: ruff-check
           # Run the formatter.
           - id: ruff-format
-      - repo: https://github.com/pycqa/isort
-        rev: 6.0.1
-        hooks:
-          - id: isort

README.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,6 +1,6 @@
  
    ![](https://raw.githubusercontent.com/wiki/LIVVkit/LIVVkit/imgs/livvkit.png)

    # LIVVkit Extensions (LEX)

    # LIVVkit Extensions (LivvEX)

    This repository holds a collection of extensions to

    [LIVVkit](https://livvkit.github.io/Docs/index.html) for validation and

    @@ -19,41 +19,43 @@ machines, including Perlmutter at NERSC, and Chrysalis at ANL\'s LCRC.
  
    The Python package itself is described in `pyproject.toml`, which is used by

    `pip` to install this package

    Currently, LEX is designed to run on Perlmutter, but future work is

    planned to support other machines.

    Currently, LEX is designed to run on NERSC's Perlmutter, and ANL-LCRC's Chrysalis,

    but future work is planned to support other machines where E3SM runs.

    ## Environment setup

    For setting up an environment to which lex and dependencies will be

    installed, conda and Python virtualenv are documented here. **NB** this

    will only currently work on NERSC's Perlmutter, the environment should be

    installed, Pixi and conda are documented here. **NB** this

    will only currently work on Perlmutter and Chrysalis, the environment should be

    created there.

    ### Conda environment

    ### Pixi environment

    [pixi](https://pixi.sh/latest/) is a package management tool, and the primary

    environment management tool for LEX.

    First, it must be [installed](https://pixi.prefix.dev/latest/installation/) locally.

    Then an enviornment for lex development can be created:

    ```bash

    $ git clone https://github.com/LIVVkit/lex.git

    $ cd lex

    $ {conda, mamba} create -n lex_env python --file requirements.txt

    $ {conda, mamba} activate lex_env

    $ pip install -e .    # Installs the lex module as an editable Python package to the lex_env environment.

    $ pixi install          # This will install the default environment

    $ pixi shell -e default # To activate the default (runtime) environment

    ```

    ### Python virtualenv

    If development tools are needed (e.g. `pytest`), install and activate the `dev` environment:

    ```bash

    $ git clone https://github.com/LIVVkit/lex.git

    $ cd lex python -m venv .env

    $ source .env/bin/activate

    $ pip install --upgrade pip # Needed if the system pip version < 21.3

    $ pip install -e .

    $ pixi install -e dev   # Installs the environment

    $ pixi shell -e dev     # Activates the dev environment (for code testing and checks)

    ```

    **N.B.** it is not recommended to do `pixi install --all`, as this will install the environment versions used for testing.

    This will create a virtual environment at `lex/.env`, and install the

    LEX package as editable with all its Python requirements to run.

    ### Other available environment management solutions

    Not documented here, but also available for environment management

    - [uv](https://docs.astral.sh/uv/)

    - [pixi](https://pixi.sh/latest/)

    ### Conda environment

    ```bash

    $ git clone https://github.com/LIVVkit/lex.git

    $ cd lex

    $ {conda, mamba} env create -n lex_env python --file env.yml

    $ {conda, mamba} activate lex_env

    $ pip install -e .    # Installs the lex module as an editable Python package to the lex_env environment.

    ```

    ## Basic usage

    @@ -72,7 +74,9 @@ website you\'d run this command:
  
    ```bash

    $ livv -V config/example/example.yml -o vv_test -s

    ```

    This will create a directory in the current directory called `vv_test` (~7.5 MB), and spawn an HTTP server, which should only be used for testing purposes. (This works best if the output is in the current directory)

    This will create a directory in the current directory called `vv_test` (~7.5 MB), and

    spawn an HTTP server, which should only be used for testing purposes.

    (This works best if the output is in the current directory)

    *Note:* All the extension configurations files assume you are working

    from the top level `lex` directory. You *can* run any of these

    @@ -103,20 +107,24 @@ $ sbatch run_lex_pm-cpu.sbatch
  
    ## Running new cases on PM-CPU

    ### Generate a single timeseries file from ELM h0 outputs

    - `ncrcat -v topo,landfrac,QSNOFRZ,FSRND,FSRVD,FSDSVD,FSDSND,EFLX_LH_TOT,FIRA,FLDS,FSA,FSDS,FSH,QICE,QRUNOFF,QSNOMELT,QSOIL,RAIN,SNOW,TSA elm*h0*.nc -o ${CASE}.nc`

    - `CASE="The case name"`

    - `ncrcat -v topo,landfrac,QSNOFRZ,FSRND,FSRVD,FSDSVD,FSDSND,EFLX_LH_TOT,FIRA,FLDS,FSA,FSDS,FSH,QICE,QRUNOFF,QSNOMELT,QSOIL,RAIN,SNOW,TSA,SNOWICE,SNOWLIQ,H2OSNO ${CASE}.elm.h0*.nc -o ${CASE}.nc`

    ### Perform post-processing on a single time series ELM h0 output

    - Edit the `lex/lex/postproc/e3sm/postproc.sbatch` batch file to mach the new run

        Key variables:

        - `INDIR`: Path which contains single output time series file

        - `OUTCASE`: Name of the new case which is the name of the netCDF file without extension (e.g. `v2.1.r025.IGERA5ELM_MLI-deep_firn_1980_2020`)

        - `OUTCASE`: Name of the new case which is the name of the netCDF file without

            extension (e.g. `v2.1.r025.IGERA5ELM_MLI-deep_firn_1980_2020`)

        - `RES`: ELM output resolution (currently accepts `R05` and `R025`)

        - `OUTDIR`: Scratch directory into which climatology files will be written, defaults to `${SCRATCH}/lex/data/e3sm/${OUTCASE}`

        - `OUTDIR`: Scratch directory into which climatology files will be written,

            defaults to `${SCRATCH}/lex/data/e3sm/${OUTCASE}`

    - Run the post-processing script:

      - `cd lex/lex/postproc/e3sm; sbatch postproc.sbatch`

    **NB**: the `postproc.sbatch` script will create the configuration for your case (based on `OUTCASE` and `OUTDIR`), then run LIVVkit on it with `lex/run_livv.sh`

    **NB**: the `postproc.sbatch` script will create the configuration for your case

    (based on `OUTCASE` and `OUTDIR`), then run LIVVkit on it with `lex/run_livv.sh`

    ## Developing a custom extension

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Zppy compatibility, logging, and other v1.0 features #1

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Add Zppy compatibility, logging, and other v1.0 features #1

Are you sure you want to change the base?

Uh oh!

Add Zppy compatibility, logging, and other v1.0 features #1

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!