WIP: Feature/links dev merge

[crkrenn]

@FrankD412 & @jsemler,

This is a beta version of a user defined linking system.

It needs more documentation and tests, but I'd like your initial impressions. And, I also need to add some additional logging. There are also some minor differences (e.g. 'apply dag' vs. 'apply environment') that I will resolve before removing the "WIP" label.

A question: should I keep the new code in utils.py, or should I move it to a new module? Would it count as an "interface" (e.g. interfaces/linker.py)?

And, more jinja templates could be easily added (e.g. 'time', 'username', etc.).

Some miscellaneous thoughts:

• I think I should add an automatic suffix if there are link conflict
• I could consider a step index as well as a maestro execution index, but I don't really want to encourage that kind of naming.
• And, I could write a tool that would process a set of maestro runs and create links after the simulations are run.

There are 3 new maestro run options:

  --make-links          Automatically make customizable, human-readable 
                        links to run directories. [Default: False]
  --link-directory LINK_DIRECTORY
                        Jinja template for path where links to run directories are made
                        [Default: {{output_root}}/links]
  --link-template LINK_TEMPLATE
                        Jinja template for links to run directories
                        [Default: {{link_directory}}/{{date}}/run-{{INDEX}}/{{instance}}/{{step}}]
                         
                        Currently supported Jinja variables:
                        {{output_root}} - Parent directory for this maestro study
                        {{link_directory}} - Link directory for this maestro study
                        {{date}} - Human-readable date (e.g. '2020_07_28')
                        {{instance}} - Maestro label for a set of parameters
                                       (e.g. 'X1.5.X2.5.X3.20')
                                       [maximum length: 255 characters]
                        {{step}} - Maestro label for a given step (e.g. 'run')
                        {{INDEX}} - Unique number for each maestro execution (e.g. '0001')

A simple example of links that are automatically created is below:

(venv) ➜  /tmp tree sample_output                                                                                                   
sample_output
├── hello_bye_world_20201015-124125
│   ├── batch.info
│   ├── bye_world
│   │   ├── bye.txt
│   │   ├── bye_world.13912.err
│   │   ├── bye_world.13912.out
│   │   └── bye_world.sh
│   ├── hello_bye_parameterized.yaml
│   ├── hello_bye_world.pkl
│   ├── hello_bye_world.study.pkl
│   ├── hello_bye_world.txt
│   ├── hello_world
│   │   ├── GREETING.Ciao.NAME.Jim
│   │   │   ├── Ciao_Jim.txt
│   │   │   ├── hello_world_GREETING.Ciao.NAME.Jim.13909.err
│   │   │   ├── hello_world_GREETING.Ciao.NAME.Jim.13909.out
│   │   │   └── hello_world_GREETING.Ciao.NAME.Jim.sh
│   │   └── GREETING.Hello.NAME.Pam
│   │       ├── Hello_Pam.txt
│   │       ├── hello_world_GREETING.Hello.NAME.Pam.13908.err
│   │       ├── hello_world_GREETING.Hello.NAME.Pam.13908.out
│   │       └── hello_world_GREETING.Hello.NAME.Pam.sh
│   ├── logs
│   │   └── hello_bye_world.log
│   ├── maestro_index_file
│   ├── meta
│   │   ├── environment.yaml
│   │   ├── metadata.yaml
│   │   ├── parameters.yaml
│   │   └── study
│   │       └── env.pkl
│   ├── status.csv
│   └── whats_up_world
│       ├── NAME.Jim
│       │   ├── whats_up.txt
│       │   ├── whats_up_world_NAME.Jim.13907.err
│       │   ├── whats_up_world_NAME.Jim.13907.out
│       │   └── whats_up_world_NAME.Jim.sh
│       └── NAME.Pam
│           ├── whats_up.txt
│           ├── whats_up_world_NAME.Pam.13906.err
│           ├── whats_up_world_NAME.Pam.13906.out
│           └── whats_up_world_NAME.Pam.sh
├── hello_bye_world_20201015-124137
│   ├── batch.info
│   ├── bye_world
│   │   ├── bye.txt
│   │   ├── bye_world.13978.err
│   │   ├── bye_world.13978.out
│   │   └── bye_world.sh
│   ├── hello_bye_parameterized.yaml
│   ├── hello_bye_world.pkl
│   ├── hello_bye_world.study.pkl
│   ├── hello_bye_world.txt
│   ├── hello_world
│   │   ├── GREETING.Ciao.NAME.Jim
│   │   │   ├── Ciao_Jim.txt
│   │   │   ├── hello_world_GREETING.Ciao.NAME.Jim.13966.err
│   │   │   ├── hello_world_GREETING.Ciao.NAME.Jim.13966.out
│   │   │   └── hello_world_GREETING.Ciao.NAME.Jim.sh
│   │   └── GREETING.Hello.NAME.Pam
│   │       ├── Hello_Pam.txt
│   │       ├── hello_world_GREETING.Hello.NAME.Pam.13965.err
│   │       ├── hello_world_GREETING.Hello.NAME.Pam.13965.out
│   │       └── hello_world_GREETING.Hello.NAME.Pam.sh
│   ├── logs
│   │   └── hello_bye_world.log
│   ├── maestro_index_file
│   ├── meta
│   │   ├── environment.yaml
│   │   ├── metadata.yaml
│   │   ├── parameters.yaml
│   │   └── study
│   │       └── env.pkl
│   ├── status.csv
│   └── whats_up_world
│       ├── NAME.Jim
│       │   ├── whats_up.txt
│       │   ├── whats_up_world_NAME.Jim.13964.err
│       │   ├── whats_up_world_NAME.Jim.13964.out
│       │   └── whats_up_world_NAME.Jim.sh
│       └── NAME.Pam
│           ├── whats_up.txt
│           ├── whats_up_world_NAME.Pam.13963.err
│           ├── whats_up_world_NAME.Pam.13963.out
│           └── whats_up_world_NAME.Pam.sh
└── links
    └── 2020-10-15
        ├── run-0001
        │   ├── GREETING.Ciao.NAME.Jim
        │   │   └── hello_world -> /private/tmp/sample_output/hello_bye_world_20201015-124125/hello_world/GREETING.Ciao.NAME.Jim
        │   ├── GREETING.Hello.NAME.Pam
        │   │   └── hello_world -> /private/tmp/sample_output/hello_bye_world_20201015-124125/hello_world/GREETING.Hello.NAME.Pam
        │   ├── NAME.Jim
        │   │   └── whats_up_world -> /private/tmp/sample_output/hello_bye_world_20201015-124125/whats_up_world/NAME.Jim
        │   ├── NAME.Pam
        │   │   └── whats_up_world -> /private/tmp/sample_output/hello_bye_world_20201015-124125/whats_up_world/NAME.Pam
        │   └── all_records
        │       └── bye_world -> /private/tmp/sample_output/hello_bye_world_20201015-124125/bye_world
        └── run-0002
            ├── GREETING.Ciao.NAME.Jim
            │   └── hello_world -> /private/tmp/sample_output/hello_bye_world_20201015-124137/hello_world/GREETING.Ciao.NAME.Jim
            ├── GREETING.Hello.NAME.Pam
            │   └── hello_world -> /private/tmp/sample_output/hello_bye_world_20201015-124137/hello_world/GREETING.Hello.NAME.Pam
            ├── NAME.Jim
            │   └── whats_up_world -> /private/tmp/sample_output/hello_bye_world_20201015-124137/whats_up_world/NAME.Jim
            ├── NAME.Pam
            │   └── whats_up_world -> /private/tmp/sample_output/hello_bye_world_20201015-124137/whats_up_world/NAME.Pam
            └── all_records
                └── bye_world -> /private/tmp/sample_output/hello_bye_world_20201015-124137/bye_world

[FrankD412]

@crkrenn -- Sorry for taking so long to get to this PR. I'm looking at testing it this morning.

[FrankD412]

Alright -- after some review, this appears to be functioning on my end. I tested the LULESH example locally. One thing that I did notice is that parameters are repeated if they're in combination with other values. For examples:

(maestrowf-JqadW55J-py3.7)  ~/Documents/Code/Python/maestrowf/sample_output/lulesh   feature/links_dev_merge  ll links/2020-11-09/run-0001
total 0
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:08 ITER.10.SIZE.10
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:08 ITER.10.SIZE.20
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:08 ITER.10.SIZE.30
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:08 ITER.20.SIZE.10
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:08 ITER.20.SIZE.20
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:08 ITER.20.SIZE.30
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:08 ITER.30.SIZE.10
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:08 ITER.30.SIZE.20
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:08 ITER.30.SIZE.30
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 SIZE.10
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 SIZE.20
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 SIZE.30
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 TRIAL.1
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 TRIAL.2
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 TRIAL.3
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 TRIAL.4
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 TRIAL.5
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 TRIAL.6
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 TRIAL.7
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 TRIAL.8
drwxr-xr-x@ 3 frank  staff    96B Nov  9 11:09 TRIAL.9
drwxr-xr-x@ 4 frank  staff   128B Nov  9 11:09 all_records

This structure isn't a tree so you see that SIZE appears multiple times. Is that intended? It's perfectly fine if this is an iterative step too. I just want to verify that this is working as you intended, @crkrenn.

[thecivicscenter]

Hello Frank,

Thanks for taking a look!

Could you post the commands you used to generate the commands above? Your output may or may not be what I intended.

Thanks!

-Chris

[FrankD412]

@crkrenn @thecivicscenter -- I generated the above links from the root of the Maestro repository by running the following:

maestro run --make-links ./samples/lulesh/lulesh_sample1_macosx.yaml

That creates timestamped directories in the ./sample_output/lulesh/lulesh_sample1* with a links directory at ./sample_output/lulesh/links

You should be able to substitute the macosx spec with the unix version.

[crkrenn]

Hello Frank,

Thanks for giving me a challenging example! And, this does work the way I intended.

I could implement tree links if there was interest, but I think that this meets our current needs. Personally, I prefer the flat directory naming.

-Chris

sample_output/lulesh/links/2020-11-11/run-0001
├── ITER.10.SIZE.10
│   └── run-lulesh -> maestro/run-lulesh/ITER.10.SIZE.10
├── ITER.10.SIZE.20
│   └── run-lulesh -> maestro/run-lulesh/ITER.10.SIZE.20
├── ITER.10.SIZE.30
│   └── run-lulesh -> maestro/run-lulesh/ITER.10.SIZE.30
├── ITER.20.SIZE.10
│   └── run-lulesh -> maestro/run-lulesh/ITER.20.SIZE.10
├── ITER.20.SIZE.20
│   └── run-lulesh -> maestro/run-lulesh/ITER.20.SIZE.20
├── ITER.20.SIZE.30
│   └── run-lulesh -> maestro/run-lulesh/ITER.20.SIZE.30
├── ITER.30.SIZE.10
│   └── run-lulesh -> maestro/run-lulesh/ITER.30.SIZE.10
├── ITER.30.SIZE.20
│   └── run-lulesh -> maestro/run-lulesh/ITER.30.SIZE.20
├── ITER.30.SIZE.30
│   └── run-lulesh -> maestro/run-lulesh/ITER.30.SIZE.30
├── SIZE.10
│   └── post-process-lulesh-size -> maestro/post-process-lulesh-size/SIZE.10
├── SIZE.20
│   └── post-process-lulesh-size -> maestro/post-process-lulesh-size/SIZE.20
├── SIZE.30
│   └── post-process-lulesh-size -> maestro/post-process-lulesh-size/SIZE.30
├── TRIAL.1
│   └── post-process-lulesh-trials -> maestro/post-process-lulesh-trials/TRIAL.1
├── TRIAL.2
│   └── post-process-lulesh-trials -> maestro/post-process-lulesh-trials/TRIAL.2
├── TRIAL.3
│   └── post-process-lulesh-trials -> maestro/post-process-lulesh-trials/TRIAL.3
├── TRIAL.4
│   └── post-process-lulesh-trials -> maestro/post-process-lulesh-trials/TRIAL.4
├── TRIAL.5
│   └── post-process-lulesh-trials -> maestro/post-process-lulesh-trials/TRIAL.5
├── TRIAL.6
│   └── post-process-lulesh-trials -> maestro/post-process-lulesh-trials/TRIAL.6
├── TRIAL.7
│   └── post-process-lulesh-trials -> maestro/post-process-lulesh-trials/TRIAL.7
├── TRIAL.8
│   └── post-process-lulesh-trials -> maestro/post-process-lulesh-trials/TRIAL.8
├── TRIAL.9
│   └── post-process-lulesh-trials -> maestro/post-process-lulesh-trials/TRIAL.9
└── all_records
├── make-lulesh -> maestro/make-lulesh
└── post-process-lulesh -> maestro/post-process-lulesh

45 directories, 0 files

[crkrenn]

All:
I'm only working one day per month now, so it is unlikely that I will be able to push this over the finish line by myself.
I did refactor so that the linking takes advantage of _StepRecord and ExecutionGraph. The only current conflict is poetry.lock, which should be easy to resolve.
V/r,
-Chris