Custom indexes

[ilan-gold]

Description of feature

Related to but somehwat different than #43

Currently we just blindly exhaust the entire on-disk obs index but there are multiple cases where you might want to break this model:

1. Different weighted sampling schemes Planned Feat: groupby per dataset #186 CategoricalSampler support #119
2. "Mapped indexing" i.e., from a neighborhood graph
3. Masked indexing i.e., you only want the current data loader to focus on a fixed subset of the data Implemented

1 and 3 probably break the idea that we should pre-shuffle the entire dataset (although you should probably still do some degree of shuffling, and perhaps this motivates more flexible shuffling) but 2 probably could be worked with pre-shuffling in some way or another.

The flipside is that these could be used to "weight" preshuffling so the order of the on-disk data gives the randomness scheme desired (i.e., shuffling in a certain way to achieve 2 or 3 without actually changing the loading code).

[ilan-gold]

2 probably could be worked with pre-shuffling in some way or another.

We definitely, based on discussion, would want a graph-like (maybe just simply morton?) ordering on-disk similar to what we are doing for categorical orderings in #186 but for KNN graphs/spatial locality

[selmanozleyen]

Sorry did we implement weighted sampling? I don't think we did

[ilan-gold]

Sorry, shouldn't have been typing when talking. Should have said "planned".