Benchmark modified WilsonGaugeAction code on GPU

[qiUip]

To check if the changes in the gauge action deriv branch make sense, we should benchmark the code on GPU(s) before and after the change for the sp2n and SU3 cases.

I'm not 100% clear which cases are best fit for this, but we should try the two cases we have profiled for start.

[qiUip]

@asifsamiarain once they are done, could you please add the links to the benchmarks here?

[asifsamiarain]

Here are below the results w.r.t. develop branch:

[dc-asif1@tursa-login2 pgda007]$ mlc gdchk defb
                                                                        LOG_NAME   SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-001_sp2n-16.16.16.16-1.1.1.1_defb00_38972.log   148         0.282725093001668 tu-c0r0n90
               gd_tursa-gnucuda_seq-001_su3-16.16.16.16-1.1.1.1_defb00_38986.log    85        0.2561569001390701 tu-c0r0n87

and gague_actions_deriv branch:

[dc-asif1@tursa-login2 pgda008]$ mlc gdchk defb
                                                                        LOG_NAME   SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-001_sp2n-16.16.16.16-1.1.1.1_defb00_38975.log   120         0.282725093001668 tu-c0r0n90
               gd_tursa-gnucuda_seq-001_su3-16.16.16.16-1.1.1.1_defb00_38974.log    63        0.2561569001390701 tu-c0r0n87

[qiUip]

Oh, that's interesting! The sp2n test certainly shows improvements and the result is the same, but SU3 takes longer and the result is wrong!

I certainly need to investigate this deeper then. Thanks!

EDIT: After inspecting the logs (thanks for providing them!), the develop branch SU3 case was run with
Grid : Message : 1.155066 s : [Integrator] Number of MD steps : 20 instead of 36 like the gague_actions_deriv branch, so that explains the differences. Could we please rerun this one?

[asifsamiarain]

Updated the results above.

[qiUip]

I missed this before, but we did not use --accelerator-threads here. From what Ed mentioned, they normally set it to 8. Lets redo these 4 tests with --accelerator-threads 8 and remove the OMP --threads 1.

If we want to make a PR directly to the upstream for the branch, then potentially we should test it with a larger case too.

[asifsamiarain]

The --threads 1 updated with --accelerator-threads 8 and repeated above tests:

develop branch:

[dc-asif1@tursa-login2 pgda007]$ mlc gdchk defb03
                                                                        LOG_NAME   SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-008_sp2n-16.16.16.16-1.1.1.1_defb03_39986.log    79         0.282725093001668 tu-c0r0n87
               gd_tursa-gnucuda_seq-008_su3-16.16.16.16-1.1.1.1_defb03_39985.log    58        0.2561569001390701 tu-c0r0n93

and gague_actions_deriv branch:

[dc-asif1@tursa-login2 pgda008]$ mlc gdchk defb03
                                                                        LOG_NAME   SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-008_sp2n-16.16.16.16-1.1.1.1_defb03_39982.log    75         0.282725093001668 tu-c0r0n90
               gd_tursa-gnucuda_seq-008_su3-16.16.16.16-1.1.1.1_defb03_39981.log    67        0.2561569001390701 tu-c0r0n87

[qiUip]

Time went UP in the new branch for the su3 case and they are worse than with accelerator_threads = 1? The results are identical so I doubt it's the setup... lets double check the logs to make sure it worked as expected as this is a bit strange.

[asifsamiarain]

Instead of picking the time calculated via date command in deploy.sh, here are below the results as the run time is grepped w.r.t. string "Grid Finalize" in log:

develop branch:

[dc-asif1@tursa-login2 pgda007]$ mlc gdchk defb00
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-001_sp2n-16.16.16.16-1.1.1.1_defb00_38972.log      129.710517         0.282725093001668 tu-c0r0n90
               gd_tursa-gnucuda_seq-001_su3-16.16.16.16-1.1.1.1_defb00_38986.log       69.344686        0.2561569001390701 tu-c0r0n87

[dc-asif1@tursa-login2 pgda007]$ mlc gdchk defb03
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-008_sp2n-16.16.16.16-1.1.1.1_defb03_39986.log       75.988421         0.282725093001668 tu-c0r0n87
               gd_tursa-gnucuda_seq-008_su3-16.16.16.16-1.1.1.1_defb03_39985.log       54.606163        0.2561569001390701 tu-c0r0n93

and gague_actions_deriv branch:

[dc-asif1@tursa-login2 pgda008]$ mlc gdchk defb00
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-001_sp2n-16.16.16.16-1.1.1.1_defb00_38975.log      117.528248         0.282725093001668 tu-c0r0n90
               gd_tursa-gnucuda_seq-001_su3-16.16.16.16-1.1.1.1_defb00_38974.log       60.166884        0.2561569001390701 tu-c0r0n87

[dc-asif1@tursa-login2 pgda008]$ mlc gdchk defb03
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-008_sp2n-16.16.16.16-1.1.1.1_defb03_39982.log       71.589940         0.282725093001668 tu-c0r0n90
               gd_tursa-gnucuda_seq-008_su3-16.16.16.16-1.1.1.1_defb03_39981.log       50.250853        0.2561569001390701 tu-c0r0n87

Note: defb00 is with --threads 1 & defb03 with --accelerator-threads 8

[asifsamiarain]

Having the -p gpu-a100-80 -q dev --gres=gpu:4 repeated tests with all --accelerator-threads possibilities (so far w.r.t. sp2n config yet).

develop branch:

[dc-asif1@tursa-login2 pgda007]$ mlc gdchk defb03 | grep -v su3
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-001_sp2n-16.16.16.16-1.1.1.1_defb03_40047.log      194.279084         0.282725093001668 tu-c0r0n87
              gd_tursa-gnucuda_seq-002_sp2n-16.16.16.16-1.1.1.1_defb03_40048.log      128.314100         0.282725093001668 tu-c0r0n90
              gd_tursa-gnucuda_seq-004_sp2n-16.16.16.16-1.1.1.1_defb03_40059.log       94.333143         0.282725093001668 tu-c0r0n93
              gd_tursa-gnucuda_seq-008_sp2n-16.16.16.16-1.1.1.1_defb03_39986.log       75.988421         0.282725093001668 tu-c0r0n87
              gd_tursa-gnucuda_seq-016_sp2n-16.16.16.16-1.1.1.1_defb03_40062.log       70.165318         0.282725093001668 tu-c0r0n87
              gd_tursa-gnucuda_seq-032_sp2n-16.16.16.16-1.1.1.1_defb03_40110.log       68.801651         0.282725093001668 tu-c0r0n90
              gd_tursa-gnucuda_seq-064_sp2n-16.16.16.16-1.1.1.1_defb03_40111.log       70.502582         0.282725093001668 tu-c0r0n93

and gague_actions_deriv branch:

[dc-asif1@tursa-login2 pgda008]$ mlc gdchk defb03 | grep -v su3
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_seq-001_sp2n-16.16.16.16-1.1.1.1_defb03_40130.log      172.610941         0.282725093001668 tu-c0r0n87
              gd_tursa-gnucuda_seq-002_sp2n-16.16.16.16-1.1.1.1_defb03_40131.log      115.338775         0.282725093001668 tu-c0r0n90
              gd_tursa-gnucuda_seq-004_sp2n-16.16.16.16-1.1.1.1_defb03_40132.log       85.860179         0.282725093001668 tu-c0r0n93
              gd_tursa-gnucuda_seq-008_sp2n-16.16.16.16-1.1.1.1_defb03_39982.log       71.589940         0.282725093001668 tu-c0r0n90
              gd_tursa-gnucuda_seq-016_sp2n-16.16.16.16-1.1.1.1_defb03_40133.log       66.174500         0.282725093001668 tu-c0r0n87
              gd_tursa-gnucuda_seq-032_sp2n-16.16.16.16-1.1.1.1_defb03_40134.log       64.453990         0.282725093001668 tu-c0r0n90
              gd_tursa-gnucuda_seq-064_sp2n-16.16.16.16-1.1.1.1_defb03_40135.log       66.288794         0.282725093001668 tu-c0r0n93

[qiUip]

Moving forward, we are going to use 2 tests for official benchmarking of code changes for upstream PRs:

1. HMC/Mobius2p1f - this is a familiar production run.
2. tests/smearing/WilsonFlow + patches from the 3 commits on Ed's fork + using the configuration from the Mobius2p1f benchmark through an input.xml file (details below) - this is still SU3, but using almost 100% calls to gauge action.

Both tests should be run with these 2 configuration: --grid 24.24.24.48 --mpi 1.1.1.1 and --grid 32.32.32.64 --mpi 1.1.1.4 and both with --accelerator-threads 8 and --Trajectories 5 --Thermalizations 3.

We created a patch for the 3 commits from Ed's branch which is found in /home/dp208/dp208/shared/RAC16/patches/WilsonFlow.patch - not sure if it would be worth merging this to develop as I believe it will not need to go upstream (check with Ed?).

To load the configuration from Mobius2p1f into WilsonFlow, an additional input.xml file is required with the following:

 <grid>
  <WilsonFlow>
    <steps>100</steps>
    <step_size>0.01</step_size>
    <meas_interval>10</meas_interval>
  </WilsonFlow>
  <Configurations>
    <conf_prefix>./ckpoint_EODWF_lat</conf_prefix>
    <rng_prefix>./ckpoint_EODWF_rng</rng_prefix>
    <StartConfiguration>7</StartConfiguration>
    <EndConfiguration>8</EndConfiguration>
    <Skip>1</Skip>
  </Configurations>
 </grid>

and the ckpoint_EODWF_rng.x and ckpoint_EODWF_lat.x (where x=7,8) need to be copied over to the WilsonFlow directory (should we try changing the path in the input.xml file instead?).
Note that for some reason, when the code loads the configuration, the .rng file name is not read correctly and the ckpoint_EODWF_rng.* need to be renamed just to rng.* (@asifsamiarain please confirm if this is still correct?).

For internal interest on sp2n, we can continue using the same test as we did before, but it does not need to be included in the upstream PR data.

[asifsamiarain]

That's "the .rng file name is not read correctly and the ckpoint_EODWF_rng.* need to be renamed just to rng.*" correct.

[asifsamiarain]

In connection with congif1 --grid 24.24.24.48 --mpi 1.1.1.1 --Trajectories 5 --Thermalizations 3

develop branch:

[dc-asif1@tursa-login2 pgda018]$ mlc gdchk defe
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_001-008_mobi-24.24.24.48-1.1.1.1_defe00_45339.log    16758.760737        0.5974304307694565 tu-c0r4n21
              gd_tursa-gnucuda_001-008_prod-24.24.24.48-1.1.1.1_defe01_45012.log      170.332597         0.996634196056962 tu-c0r1n30
               gd_tursa-gnucuda_001-008_su3-24.24.24.48-1.1.1.1_defe02_45803.log     1977.824533        0.4949517921780855 tu-c0r0n72

and gague_actions_deriv branch:

[dc-asif1@tursa-login2 pgda019]$ mlc gdchk defe
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_001-008_mobi-24.24.24.48-1.1.1.1_defe00_44980.log    16761.844173        0.5974304307694565 tu-c0r1n24
              gd_tursa-gnucuda_001-008_prod-24.24.24.48-1.1.1.1_defe01_45113.log      150.759162         0.996634196056962 tu-c0r0n72
               gd_tursa-gnucuda_001-008_su3-24.24.24.48-1.1.1.1_defe02_45825.log     1829.613212        0.4949517921780855 tu-c0r0n72

In connection with config2 --grid 32.32.32.64 --mpi 1.1.1.4 --Trajectories 5 --Thermalizations 3

develop branch:

[dc-asif1@tursa-login2 pgda018]$ mlc gdchk deff
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_004-008_mobi-32.32.32.64-1.1.1.4_deff00_45340.log    17867.537452        0.5973682800185346 tu-c0r1n69
              gd_tursa-gnucuda_004-008_prod-32.32.32.64-1.1.1.4_deff01_44982.log      226.403670         0.996599127390424 tu-c0r1n27
               gd_tursa-gnucuda_004-008_su3-32.32.32.64-1.1.1.4_deff02_45802.log     2443.877700        0.4948530689679816 tu-c0r0n72

and gague_actions_deriv branch:

[dc-asif1@tursa-login2 pgda019]$ mlc gdchk deff
                                                                        LOG_NAME             SEC                 Plaquette      NODES
              gd_tursa-gnucuda_004-008_mobi-32.32.32.64-1.1.1.4_deff00_45011.log    17806.483462        0.5973682800185346 tu-c0r4n03
              gd_tursa-gnucuda_004-008_prod-32.32.32.64-1.1.1.4_deff01_45112.log      209.341596         0.996599127390424 tu-c0r0n72
               gd_tursa-gnucuda_004-008_su3-32.32.32.64-1.1.1.4_deff02_45814.log     2322.403084        0.4948530689679816 tu-c0r0n72

As expressed above by @qiUip, the mobi refer to the HMC/Mobius2p1f and prod to tests/smearing/Test_WilsonFlow. Additionally also gathered the results against tests/hmc/Test_hmc_WilsonFermionGauge referred as su3.

[asifsamiarain]

Moreover as far as running the mobi with config2 was concerned particularly met with errors such as:

Mobius2p1f: ../../Grid/communicator/SharedMemory.cc:103: void* Grid::SharedMemory::ShmBufferMalloc(size_t): Assertion `heap_bytes<heap_size' failed.
[tu-c0r0n87:357262] *** Process received signal ***
[tu-c0r0n87:357262] Signal: Aborted (6)
[tu-c0r0n87:357262] Signal code:  (-6)
[tu-c0r0n87:357262] [ 0] /lib64/libpthread.so.0(+0x12d10)[0x1549fb566d10]
[tu-c0r0n87:357262] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x1549fb1cc52f]
[tu-c0r0n87:357262] [ 2] /lib64/libc.so.6(abort+0x127)[0x1549fb19fe65]
[tu-c0r0n87:357262] [ 3] /lib64/libc.so.6(+0x21d39)[0x1549fb19fd39]
[tu-c0r0n87:357262] [ 4] /lib64/libc.so.6(+0x46e86)[0x1549fb1c4e86]
[tu-c0r0n87:357262] [ 5] ./Mobius2p1f[0x51d172]
[tu-c0r0n87:357262] [ 6] ./Mobius2p1f[0x63e1e6]
[tu-c0r0n87:357262] [ 7] ./Mobius2p1f[0x663aae]
[tu-c0r0n87:357262] [ 8] ./Mobius2p1f[0x5290d2]
[tu-c0r0n87:357262] [ 9] ./Mobius2p1f[0x4818df]
[tu-c0r0n87:357262] [10] ./Mobius2p1f[0x40c911]
[tu-c0r0n87:357262] [11] /lib64/libc.so.6(__libc_start_main+0xe5)[0x1549fb1b87e5]
[tu-c0r0n87:357262] [12] ./Mobius2p1f[0x414c9e]
[tu-c0r0n87:357262] *** End of error message ***

And the solution was to pass --shm additional flag as shown below:
srun --hint=nomultithread --distribution=block:block gpu_launch.sh ./Mobius2p1f --grid 32.32.32.64 --Thermalizations 5 --Trajectories 3 --accelerator-threads 8 --mpi 1.1.1.4 --shm 4096MB