Try updating gmx to modern versions by lilyminium · Pull Request #313 · leeping/forcebalance

lilyminium · 2026-02-26T21:42:51Z

This PR updates ForceBalance compatibility to 2019, 2022, 2023, 2024, 2025. (with many caveats for 2022 and 2023 -- the bugs present in gromacs themselves mean it's probably a bad idea to use them with forcebalance).

I haven't yet rebased this one off #312, just because of the non-trivial complexity here. So CI is likely to fail sporadically on TestEvaluatorBromineStudy with a ConnectionRefusedError. I'd suggest the following process: 1. reviewing and merging #312 first, and 2. merging main into here, fixing merge conflicts, and 3. fingers crossed nothing explodes on Python 3.9 or 3.10 .

Changes

There are quite a few changes here. Broadly, they touch four areas: gmxio.py, MDP files, test infrastructure, and CI. The easy ones first:

CI

Added gromacs to the testing matrix

Test infrastructure

I modified setup_method to always chdir to the base/root directory at the start of a test. Occasionally a test failure could cause a failure cascade because the teardown method wouldn't work, so the next test would start in the wrong directory.
added gmx version detection for test skipping

In the end, I didn't actually wind up needing to modify the test reference values themselves -- although one of the bromine tests can be a bit flaky.

gmxio.py

Main changes:

vacuum simulations are no longer supported. Instead, we now create a large box and center molecules so they're far from the boundaries. Boxes are created dynamically from the molecule's existing box which should correspond roughly to its size. The cutoff section is picked to be a bit under half the box length. Because of this, optimize() adds a -pbc whole call as well in case something gets split across the periodic boundary
cutoff=group no longer supported, everything is Verlet now.
interaction energy: the old approach ran two separate mdrun -rerun calls (one fully interacting, one with energygrp-excl = A B) and subtracted. energygrp-excl was removed in GROMACS 2021. The rewrite runs a single mdrun -rerun with energygrps = A B and reads the four cross-group columns (Coul-SR:A-B, LJ-SR:A-B, Coul-14:A-B, LJ-14:A-B) from the .edr via gmx energy . There's a bug in 2022/2023 where this will be incorrect, see https://gitlab.com/gromacs/gromacs/-/issues/5109
changes to evaluate_ -- gromacs 2025 added some validations here that made things fail, so some modifications are made here to get around that. We delete stale checkpoints that could make grompp fail and set nstlist=1 -- there's a check when any trajectory frame is smaller than 2xrlist and a test was failing on that.

MDPs

MDPs have been updated to be compatible with modern gromacs. Main changes:

pbc = no --> pbc = xyz
removed deprecated options ns_type, nstxtcout, xtc_groups, and cutoffs of 0 which were only valid with cutoff=group
studies/003_liquid_bromine/targets/LiquidBromine/liquid.mdp sets nstpcouple = 10 and tau_p = 4.0 due to new validation checks in gmx 2025 requiring tau_p to be at least 200 integration steps -- https://manual.gromacs.org/current/release-notes/2025/2025.4.html . The defaults are nstpcoupl=100 as of gmx2023, I tested with both locally and they both worked, so I stuck with nstpcouple=10 as the more original value, see https://gitlab.com/gromacs/gromacs/-/issues/5424

…d a previously failing test affecting others

.github/workflows/ci.yml

lilyminium · 2026-03-05T10:35:29Z

.github/workflows/ci.yml

-          # - "3.13"
+        gmx-version:
+          - "2019"
+          - "2022"


2020 and 2021 are not on conda-forge. 2022 is not available for Mac, weirdly enough

lilyminium · 2026-03-05T10:35:45Z

.github/workflows/ci.yml

-        which gmx_d
-        gmx help
-        gmx dump -h
+    # - name: Install Gromacs


Or could be deleted

.github/workflows/ci.yml

lilyminium · 2026-03-06T00:17:31Z

src/gmxio.py

+        # gmxversion == 5 means "has the unified gmx wrapper" — covers GROMACS 5.x
+        # and all 20xx releases, so have left this alone for now.
+        # Old-style program names (g_energy, gmxdump, …)
+        # are mapped to their modern subcommand equivalents (energy, dump, …).
+        # gmxversion == 4 means only a standalone mdrun was found (GROMACS 4.x).
        if self.gmxversion == 5:
            csplit[0] = csplit[0].replace('g_','').replace('gmxdump','dump')
            csplit = ['gmx' + self.gmxsuffix] + csplit
        elif self.gmxversion == 4:
            csplit[0] = prog + self.gmxsuffix
        else:
-            raise RuntimeError('gmxversion can only be 4 or 5')
-        return _exec(' '.join(csplit), stdin=stdin, print_to_screen=print_to_screen, print_command=print_command, **kwargs)
+            raise RuntimeError('gmxversion must be 4 (standalone mdrun) or 5 (gmx wrapper, valid for GROMACS 5.x and all 20xx releases)')
+        return _exec(' '.join(csplit), stdin=stdin, print_to_screen=print_to_screen, print_command=print_command, copy_stderr=copy_stderr, **kwargs)


This may not make actual sense for gmx versions 2019+ -- as usual I tried to make as minimal changes as possible here.

Thanks, does gmx version 4 still work? (I assume that would require version 4 to support newer keywords such as verlet list, nstxout-compressed, coulomb-modifier etc.)

Hmm, could have sworn I left a reply to this earlier ... but no, the hardcoded keywords mean that gmx4 no longer works. To get around that, we could have a version check before swapping out the keywords.

Could this part be updated to raise an error if gmx version is 4? (So the only valid value of gmxversion is 5).

Thanks! Yeah I agree, it shouldn't get that far due to the new check but I've updated the code here and elsewhere as well to simply not accept gmxversion == 4.

src/gmxio.py

leeping

Thanks so much for making these extensive updates. I have a number of questions below.

src/tests/files/amber_alaglu/shot.mdp

.github/workflows/ci.yml

src/tests/__init__.py

src/tests/test_molecule.py

leeping · 2026-03-07T07:17:22Z

src/gmxio.py

+
+            BOX_LENGTH = (maxbox + 20) * 2
+            NSTLIST = int(1e2)
+            CUTOFF = BOX_LENGTH / 25 # nm


What's the reason for these choices of parameters?

Thanks, this is one thing that would be valuable to talk about!

One approach is to hard-code a box size, similar to Jeff's old PR (99.9 or 999.9 nm variously) or the Konermann Lab's approach (333.3 nm). I found with Jeff's larger boxes that I started running into memory issues, and 99.9 was small enough that I was still getting numerical differences from the previous vacuum results. (Actually, this might have been why the tests were hanging in CI initially and I put in that timeout flag).

The Konermann lab's 333.3 nm size is a good midpoint, but I suppose I was wary that it might not generalize to all systems due to running into either one or the other of the problems above. The approach here is based on the size of the molecule instead:

CUTOFF (which is in nm): meant to cover the entire molecule but be slightly under half the box length. Actually, now that I look at this again, I might have a flaw in my maths and logic there, thanks for the catch. I've updated this to 1) scope out every box instead of just the first one, 2) quadruple it in case of conformational change, and 3) have a minimum value of 1.0 nm just in case.

BOX_LENGTH (which is in angstroms): This is now also set dependent on CUTOFF, to quadruple just in case, and adds a 1 nm additional buffer.

NSTLIST: the neighbourlist here doesn't really need to be updated, so I've just set it to an arbitrarily high number.

The warning doesn't have to be there except that once the molecule is that big the box will become truly enormous and crash my Mac. :-)

leeping · 2026-03-07T07:19:39Z

src/gmxio.py

+        # gmxversion == 5 means "has the unified gmx wrapper" — covers GROMACS 5.x
+        # and all 20xx releases, so have left this alone for now.
+        # Old-style program names (g_energy, gmxdump, …)
+        # are mapped to their modern subcommand equivalents (energy, dump, …).
+        # gmxversion == 4 means only a standalone mdrun was found (GROMACS 4.x).
        if self.gmxversion == 5:
            csplit[0] = csplit[0].replace('g_','').replace('gmxdump','dump')
            csplit = ['gmx' + self.gmxsuffix] + csplit
        elif self.gmxversion == 4:
            csplit[0] = prog + self.gmxsuffix
        else:
-            raise RuntimeError('gmxversion can only be 4 or 5')
-        return _exec(' '.join(csplit), stdin=stdin, print_to_screen=print_to_screen, print_command=print_command, **kwargs)
+            raise RuntimeError('gmxversion must be 4 (standalone mdrun) or 5 (gmx wrapper, valid for GROMACS 5.x and all 20xx releases)')
+        return _exec(' '.join(csplit), stdin=stdin, print_to_screen=print_to_screen, print_command=print_command, copy_stderr=copy_stderr, **kwargs)


Thanks, does gmx version 4 still work? (I assume that would require version 4 to support newer keywords such as verlet list, nstxout-compressed, coulomb-modifier etc.)

src/gmxio.py

leeping · 2026-03-07T07:23:37Z

src/gmxio.py

+                except ValueError:
+                    continue
+        energies = np.array(energies)
+        return energies / 4.184  # kJ/mol → kcal/mol


Wow, this looks like it was challenging, thanks...

If Buckingham or other uncommon nonbonded functional forms were used, would this function still work?

No, good catch -- I fixated on LJ systems. I modified this function to look for all A-B cross terms which should pick up Buckingham potentials, although I haven't added a test case for this...

Yes, it would be ideal to have a test case because it seems the interaction energy calculation with Gromacs isn't tested at all. Would it be straightforward to add one? If not, perhaps it should be a different PR.

studies/004_thermo_liquid_bromine/targets/LiquidBromine/1/sim1.mdp

studies/001_water_tutorial/targets.tar.bz2

.github/workflows/ci.yml

src/tests/__init__.py

src/tests/test_molecule.py

leeping · 2026-03-17T05:57:46Z

src/gmxio.py

+        # gmxversion == 5 means "has the unified gmx wrapper" — covers GROMACS 5.x
+        # and all 20xx releases, so have left this alone for now.
+        # Old-style program names (g_energy, gmxdump, …)
+        # are mapped to their modern subcommand equivalents (energy, dump, …).
+        # gmxversion == 4 means only a standalone mdrun was found (GROMACS 4.x).
        if self.gmxversion == 5:
            csplit[0] = csplit[0].replace('g_','').replace('gmxdump','dump')
            csplit = ['gmx' + self.gmxsuffix] + csplit
        elif self.gmxversion == 4:
            csplit[0] = prog + self.gmxsuffix
        else:
-            raise RuntimeError('gmxversion can only be 4 or 5')
-        return _exec(' '.join(csplit), stdin=stdin, print_to_screen=print_to_screen, print_command=print_command, **kwargs)
+            raise RuntimeError('gmxversion must be 4 (standalone mdrun) or 5 (gmx wrapper, valid for GROMACS 5.x and all 20xx releases)')
+        return _exec(' '.join(csplit), stdin=stdin, print_to_screen=print_to_screen, print_command=print_command, copy_stderr=copy_stderr, **kwargs)


Could this part be updated to raise an error if gmx version is 4? (So the only valid value of gmxversion is 5).

src/gmxio.py

leeping · 2026-03-17T06:03:54Z

src/gmxio.py

+                except ValueError:
+                    continue
+        energies = np.array(energies)
+        return energies / 4.184  # kJ/mol → kcal/mol


Yes, it would be ideal to have a test case because it seems the interaction energy calculation with Gromacs isn't tested at all. Would it be straightforward to add one? If not, perhaps it should be a different PR.

src/molecule.py

studies/004_thermo_liquid_bromine/targets/LiquidBromine/1/sim1.mdp

lilyminium added 30 commits February 26, 2026 12:58

get interaction energies working

d6f95d2

update mdps of water tutorial

d270512

use double precision

ebd297f

add gmx to matrix

16fbbc1

add suffix

e7714dc

add fixes and updates

5f243bb

more gmxsuffix

d3058cf

refactor so we always start in the "files" path before a test to avoi…

588beea

…d a previously failing test affecting others

fix gro writing

f1595e0

mass modify pbc to xyz

f8b281e

mass modify mdp to pbc=xyz

109a683

install gmx via conda

8b8517e

remove accidental path

b95a604

name tests

9f23c23

also hard to install gmx2026 atm

fc87225

add explanatory comment

cf20ed7

rm unsupported options

76988dd

update teardown

ea5dbb4

update name

adb8d9a

add space

314e0b6

add timeout and double precision

46bc759

allow for no single precision

21e08cd

update mdps again

119400d

add timeout package

8c12820

double quotes?

7903ac5

try no quotes

e296046

try nompi

ea4a0b0

try to free up some memory

7af3e32

expand timeout of engine tests

06f3e88

make box length much smaller

42c8d8f

lilyminium added 2 commits March 5, 2026 19:09

edit the right mdp, oops

6fed976

reset file parsing

8e19625

lilyminium commented Mar 5, 2026

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

lilyminium commented Mar 5, 2026

View reviewed changes

.github/workflows/ci.yml Outdated

which gmx_d

gmx help

gmx dump -h

# - name: Install Gromacs

Copy link

Collaborator Author

lilyminium Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or could be deleted

lilyminium commented Mar 5, 2026

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

lilyminium added 2 commits March 5, 2026 21:38

rm accidental test from other PR

86286fc

clean up gmxio a bit

b806dcb

lilyminium marked this pull request as ready for review March 6, 2026 00:05

lilyminium commented Mar 6, 2026

View reviewed changes

src/gmxio.py Show resolved Hide resolved

clarify comment

89c617d

leeping reviewed Mar 7, 2026

View reviewed changes

lilyminium and others added 6 commits March 8, 2026 11:49

Merge branch 'master' into update-gmx

5bf4da9

update cutoff

51856f2

test gmx 5

ee64839

add timeout to py39 env

f002731

try testing v4

880a51a

fix v4 conda

d18d91e

lilyminium force-pushed the update-gmx branch from 280bb91 to d18d91e Compare March 8, 2026 03:18

lilyminium and others added 5 commits March 8, 2026 18:50

adjust box size

708c26a

fix box length and A-B cross terms

1c0f018

clarify comment

d7dbb1e

remove gmx 4 from matrix

8e14168

add gmx version error and test

4c2bcac

leeping approved these changes Mar 17, 2026

View reviewed changes

leeping self-requested a review March 17, 2026 15:18

lilyminium added 2 commits March 18, 2026 05:38

remove lines that appear to allow gmx4

1b76f47

oops, add 'else' back

ea577f3

leeping merged commit 54f7ea8 into leeping:master Mar 17, 2026
36 checks passed

Conversation

lilyminium commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI

Test infrastructure

gmxio.py

MDPs

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

leeping left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lilyminium Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lilyminium commented Feb 26, 2026 •

edited

Loading

lilyminium Mar 8, 2026 •

edited

Loading