Support for censored measurements

[dweindl]

Hi all,

I think it would be great if PEtab would support censored measurements via truncated noise distributions.

One question would be how to best encode the additional distribution parameters (truncation limits), since this won't fit the current location (observableFormula & observableParameters) + scale (noiseFormula & noiseParameters) schema.

Another question would be whether the truncation limits should be "hard-coded" in the observable definition, or whether those should be overridable via the measurements table to account for measurement-/condition-specific limits.

I didn't fully think that through yet, but I'd be happy about any feedback on whether that would be reasonable to have in PEtab-core, or whether that should be an extension, or about ideas on how to best represent that in PEtab.

[JanHasenauer]

agreed, this would be a very useful feature!

[dweindl]

One question would be how to best encode the additional distribution parameters (truncation limits), since this won't fit the current location (observableFormula & observableParameters) + scale (noiseFormula & noiseParameters) schema.

The most intuitive option seems to be adding additional optional columns for censoring (lower;upper) and truncation (lower;upper) to the observable table:

observableId	…	noiseDistribution	censoring	truncation
obs1	…	normal	20;80	0;100

Not sure if it's reasonable to allow arbitrary math expressions there, as in observableFormula. If so, those would better be named censoringFormula, truncationFormula (or plural).

If the truncation/censoring bounds should be allowed to be measurement-specific, additional censoringParameters and truncationParameters would be required in the measurement table.

[plakrisenko]

I suggested adding "left-truncated-normal", "right-truncated-normal" and "interval-truncated-normal" to the list of supported noiseDistribution values + an additional column for noiseDistributionParameters, thinking that maybe there could be another case where additional noise distribution parameters may be required. However, may be your suggestion is clearer.

[matthiaskoenig]

this sounds like a great idea!

If I understand correctly, this would handle measurements that are either below the limit of quantification (common in pharmacokinetics) or above the maximum measurable value (e.g. sensor saturation).

Could you briefly explain the difference between censoring and truncation, and what the typical use cases are for each?

Also, how would we encode these kinds of measurements? In practice, they’re often represented with strings like BLQ, <LQ, or similar. But we probably don’t want to use NaN—so what’s the best way to represent these values in the measurement table?

[dweindl]

If I understand correctly, this would handle measurements that are either below the limit of quantification (common in pharmacokinetics) or above the maximum measurable value (e.g. sensor saturation).

Exactly.

Could you briefly explain the difference between censoring and truncation, and what the typical use cases are for each?

Censoring would mean, that the value is somewhere beyond the censoring limit, but it can't be determined more accurately.

Truncation would mean, that the measurement domain is restricted. For example, age, mass, concentrations, ... must be non-negative. Some instrument might only generate values within a certain range.

Truncation and censoring can also be combined.

Also, how would we encode these kinds of measurements? In practice, they’re often represented with strings like BLQ, <LQ, or similar. But we probably don’t want to use NaN—so what’s the best way to represent these values in the measurement table?

For the sake of computing the likelihood, any value between the lower (upper) truncation limit and the lower (upper) censoring limit would be fine. If you have concentration measurement but you know that it is below the limit of quantification, use that. If you only know "<LOD" (limit of detection), I'd say use 0 or the LOD.

[matthiaskoenig]

Makes sense. So I would just use some values outside of the censoring limits. I am just a bit afraid that tools which do not support censoring would then use these numeric values resulting in incorrect results.

[dweindl]

I think if tools just silently ignore parts of the specs, this one won't be our only problem...

[matthiaskoenig]

I agree. But still would be nice to have some way to indicate this. How about using -inf and +inf in the measurement table for values below or above the censorship. This would clearly indicate these are special measurements and could be easily handled by tools even if not wanting to implement censorship/truncation.
These would still be numeric values.

[plakrisenko]

There can also be interval censoring, where a data point is somewhere on an interval between two values. I guess we could still use inf to that it's a special case and not a regular measurement.

[plakrisenko]

Another question would be whether the truncation limits should be "hard-coded" in the observable definition, or whether those should be overridable via the measurements table to account for measurement-/condition-specific limits.

Maybe start with "hard-coded" for now. For now it seems reasonable if one has to describe an observable with censoring and/or truncation limits.