Memory optimization PYNXTOOLS validate

[mkuehbach]

Planned for v0.14.0

The other aspect of #737:

For HDF5 file base validate we wish to reduce the number of cases when large h5py.Dataset payload needs to be loaded into main memory, especially when using chunked storage layout that would allow iterating over chunks.

Motivation was usage of the standalone HDF validator on a 32GiB file which has essentially only one dataset (a large image stack uncompressed approx 32GiB payload) from CENEM on a 128GiB RAM workstation. Despite that image stack is stored chunked, all data gets loaded somewhen and eventually generating multiple copies in the process into main memory, causing the validation process to terminated after trying to allocate past all the memory available on the workstation.

• Adds vscode launch configuration to debug the validation
• Applies two key optimization steps i) chunk-by-chunk reading while validating datasets and enums and ii) a reorganization of the execution flow whilst validating enums to remove unnecessary or inefficient loading and unpacking entire dataset into main memory.
• Drafts a further optimization for streamed reading of the positivity test in cases where reading a dataset in full is still required as in the case of contiguous storage layout, fact that contiguous storage still can take advantage of hyperslab-based reading allows streaming to approximately I/O chunks, a simple heuristic was defined to set the block_size, which could profit from further tuning for specific type of hardware and I/O patterns.
• Evaluates cases of possible situations where the assumption that there exists always only one concept with highest score or none breaks. Now a score_board is used, the old implementation is carried along, and the user is warned if best_namefit fits multiple concepts with the same highest score. Inspecting the matter revealed an imprecision in the constraining of NXem_* base classes that was fixed with a separate feature branch of nexus_definitions, here pulled in for testing purposes and to show the full picture. Be careful that unfixed was observed to lead to different execution paths followed for the same file and code when uncorrected (inspected via Hunt nondeterminism #768).
• Tightens Iterator to Sequence in validate
• Fixing tests
• New tests for evaluating NX_POSINT and other elementary NeXus datatypes for contiguous, chunked_uncompressed, and chunked_compressed storage

Keys benefits:

• Validation substantially faster as unnecessary I/O time and time for prepping payload as np.array removed
• Validation with substantially smaller and smoother memory footprint for arbitrarily large HDF5 files provided chunked based storage layout is used.

[rettigl]

I appreciate the effort, but for me this PR includes too many changes to properly review. The connection of many of these changes to the issue at hand (large memory consumption) is not clear to me. I suggest to split it up into several PRs.
Also, it changes the behavior of the validator by removing the acceptance of ints as floats, which we explicitly decided to allow at some point, as far as I remember.

[mkuehbach]

I appreciate the effort, but for me this PR includes too many changes to properly review. The connection of many of these changes to the issue at hand (large memory consumption) is not clear to me. I suggest to split it up into several PRs. Also, it changes the behavior of the validator by removing the acceptance of ints as floats, which we explicitly decided to allow at some point, as far as I remember.

Thank you @rettigl

Wrt to the last point, there is a good reason why in NeXus we distinguish NX_NUMBER, NX_(U,POS)INT, and NX_FLOAT. I do not agree that this decision in the past was a good one. IMHO a validator should check a report where there are inconsistencies it should not for convenience purposes do conversion of types. This should be the pynxtools-readers responsibility to feed to pynxtools straightaway. I can see if people may be fine with just using the default precision of data types, like python inbuild float, int but I disagree with cross-accepting what are different elementary data types. Not only for the fact that currently our implementation then would also need to check if really every precision of an int is exactly mappable on a floating with certain precision.

[lukaspie]

As it stands, I would not merge this PR.

• I agree that we need separate function for validity checks (HDF5 vs, template), since we don't always want to load full HDF5 datasets.
• Probably, we should split up the HDF5 and template validation in separate modules eventually. helpers.py is already rather overloaded. But let's do this in a separate PR.
• Regarding namefitting, I left a longer comment. I don't agree with the idea that sorting by optionality is really better than what we had before.
• Changes to the definitions do not belong here.
• vscode settings are too specific: as long as the NeXus files that are mentioned there are not part of the repo, we cannot use the launch.json. I am not suggesting to add these files, rather the settings should be kept simpler.
• Regarding "validation, not conversion" -> on a technical design level, I would agree with this. However, my rationale for allowing simple conversion (or other workarounds, like adding the target attribute automatically) was always to allow for easier reader development. Maybe one compromise could be that we allow such conversions in the the template validation, whereas for the HDF5 validation (where also non-pynxtools-generated files can be used), we are more strict. Curious about your opinion @rettigl.

[mkuehbach]

As it stands, I would not merge this PR.

* I agree that we need separate function for validity checks (HDF5 vs, template), since we don't always want to load full HDF5 datasets. ([x] DONE)

* Probably, we should split up the HDF5 and template validation in separate modules eventually. `helpers.py` is already rather overloaded. But let's do this in a separate PR. ([x] FUTURE PR)

* Regarding namefitting, I left a longer comment. I don't agree with the idea that sorting by optionality is really better than what we had before. ([x] COMPROMISE, original solution recovered, monitoring of multiple solutions via logger.debug)

* Changes to the definitions do not belong here. ([x] REMOVED)

* vscode settings are too specific: as long as the NeXus files that are mentioned there are not part of the repo, we cannot use the `launch.json`. I am not suggesting to add these files, rather the settings should be kept simpler. ([x] REMOVED)

* Regarding "validation, not conversion" -> on a technical design level, I would agree with this. However, my rationale for allowing simple conversion (or other workarounds, like adding the target attribute automatically) was always to allow for easier reader development. Maybe one compromise could be that we allow such conversions in the the template validation, whereas for the HDF5 validation (where also non-pynxtools-generated files can be used), we are more strict. Curious about your opinion @rettigl. ([x] FOLLOWING exactly the here said compromise, added technical arguments why I think this is the best approach)

[lukaspie]

@RubelMozumder note how this PR has another decoding step, similar to what you have in #778. We should align this in one central function.

[lukaspie]

Probably okay, but I will have to test thisin action to understand its full implications