v2: Combining experiment and condition tables

[elbert5770]

Problem: The current condition table only allows entry of the conditionId, targetId and targetValue. In v1, multiple targets could be specified for each condition. This allowed description of a whole experiment in one line, but the changes had to occur at time = 0.

v2 allows different conditions to occur at different times. This is an improvement as this reflects experimental protocols better. It also allows introduction of a 'forcing function' of sorts with stepwise changes in the parameters.

I propose that the change from wide condition tables to long condition tables obviates the need for the condition table and experiment tables should have the following columns:
experimentId, time, targetId, targetValue

Example:
v1 Alkan_SciSig2018 condition table (first two entries):
conditionId | Dox_level | Gem_level | SN38_level
model1_data1 | 0.0 | 0.0 | 0.0
model1_data2 | 0.0 | 0.0003 | 0.0

v2 Alkan_SciSig2018 condition table:
conditionId | targatID | targetValue
model1_data1 | Dox_level | 0.0
model1_data1 | Gem_level | 0.0
model1_data1 | SN38_level| 0.0
model1_data2 | Dox_level | 0.0
model1_data2 | Gem_level | 0.0003
model1_data2 | SN38_level| 0.0

v2 Alkan_SciSig2018 experiment table:
experimentId | time | conditionId
experiment1 | 0 | model1_data1
experiment2 | 0 | model1_data2

Proposed v2 Alkan_SciSig2018 experiment table:
experimentId | time | targatID | targetValue
experiment1 | 0 | Dox_level | 0.0
experiment1 | 0 | Gem_level | 0.0
experiment1 | 0 | SN38_level| 0.0
experiment2 | 0 | Dox_level | 0.0
experiment2 | 0 | Gem_level | 0.0003
experiment2 | 0 | SN38_level| 0.0

If, for example, experiment2 was a pulse of Gem_level, then it would be:
experimentId | time | targatID | targetValue
experiment1 | 0 | Dox_level | 0.0
experiment1 | 0 | Gem_level | 0.0
experiment1 | 0 | SN38_level| 0.0
experiment2 | 0 | Dox_level | 0.0
experiment2 | 0 | Gem_level | 0.0003
experiment2 | 10 | Gem_level | 0.0
experiment2 | 0 | SN38_level| 0.0

The current equivalent would be:
v2 Alkan_SciSig2018 experiment table:
experimentId | time | conditionId
experiment1 | 0 | model1_data1
experiment2 | 0 | model1_data2
experiment2 | 10 | model1_data1

which would introduce two unneeded events or a new condition. Because the conditionId is used nowhere else besides the experiment table, you will be creating a bunch of conditionIds for minor benefits. I agree that the benefits would be greater if the condition table was still wide.

[dweindl]

Hi @elbert5770, thanks for your input.

I see that the PEtab v2 experiment/condition table format is somewhat inconvenient for certain problems and I agree that your suggestion would work in principle.

In particular, in situations where we have some simple dose-response dataset it's tedious to assign some condition Id to each change. For more complex experimental setups or larger datasets, the current PEtab v2 format allows for a more concise and maintainable description, though. This decision was a trade-off.

When looking for example at Froehlich_CellSystems2018, the benefits of the new format become quite noticeable - I outlined that in #615. In short, the indirection via "conditions" simplifies the encoding of factorial designs.

An in-between solution could be supporting targetId, targetValue in the experiment table in addition to conditionId. So simple problems could be encoded in the way you suggested, but more complex problems could optionally use the indirection via the condition table. However, this is a step backwards in terms of normalization, because the different experiment table rows would then have different meanings. (I think, something like that has been proposed and rejected elsewhere, but I couldn't find the discussion right away.)