Deal with different streams across blocks in Open Ephys

[alejoe91]

Hi guys,

We have some user that modify the active probes across experiments. While this is totally fine in Open Ephys (you get different settings file), this is not supported in NEo. https://github.com/NeuralEnsemble/python-neo/blob/master/neo/rawio/openephysbinaryrawio.py#L1127-L1142

Is there a reason why we can't have different streams across blocks?

@samuelgarcia @h-mayorquin @zm711

[h-mayorquin]

I am not familiar with the block data model so I think @samuelgarcia is the best person to describe it or point us to the documentation but as a workaround meanwhile.

As a temporal workaround can they poin the reader to some specific sub-folder of the structure?

python-neo/neo/rawio/openephysbinaryrawio.py

Lines 53 to 81 in a376b4a

	Expected OpenEphys Folder Structure:
	dirname/
	├── Record Node 102/ # Recording hardware node (optional, v5.x+)
	│ ├── settings.xml # Main settings file
	│ ├── settings_2.xml # Additional experiment settings
	│ ├── experiment1/ # Experiment folder (Neo Block 0)
	│ │ ├── recording1/ # Recording session (Neo Segment 0)
	│ │ │ ├── structure.oebin # Required: JSON metadata file
	│ │ │ ├── continuous/ # Signal data streams
	│ │ │ │ └── AP_band/ # Stream folder (becomes "Record Node 102#AP_band")
	│ │ │ │ ├── continuous.dat # Raw binary signal data
	│ │ │ │ ├── timestamps.npy # Sample timestamps (pre-v0.6)
	│ │ │ │ └── sample_numbers.npy # Sample numbers (v0.6+)
	│ │ │ └── events/ # Event data streams (optional)
	│ │ │ └── TTL_1/ # Event stream folder
	│ │ │ ├── timestamps.npy
	│ │ │ ├── channels.npy
	│ │ │ └── states.npy
	│ │ └── recording2/ # Recording session (Neo Segment 1)
	│ │ └── [same structure as recording1]
	│ └── experiment2/ # Experiment folder (Neo Block 1)
	│ └── [same structure as experiment1]
	└── Record Node 103/ # Second recording hardware node
	├── settings.xml # Must have identical experiment structure
	├── experiment1/ # Same experiments as Record Node 102
	│ └── recording1/ # Streams become "Record Node 103#AP_band"
	│ └── [same structure]
	└── experiment2/
	└── [same structure]

[alejoe91]

That could work!

[samuelgarcia]

Is there a reason why we can't have different streams across blocks?

It is by design, stream must be across blocks and segments.

Eventualy, you can create stream that are silent during some segment/block. But you will loose the buffer abtraction.

[alejoe91]

One option could be that streams missing from a block could have a "virtual" size of 0 and pointer to no file.

Here: https://github.com/NeuralEnsemble/python-neo/blob/master/neo/rawio/openephysbinaryrawio.py#L273-L295
We can create a fake buffer with shape (0, num_channels). That should be enough