DTO analysis

[cmatKhan]

1. Select all binding samples (set up all binding samples in VirtualDB config)

   If there is a dataset that cannot be configured (no annotatedfeature dataset),
   put that in the issue discussion. @cmatKhan will address it. Do the analysis
   with available binding data.

2. After selecting all binding samples from all datasets with DTO P<=0.01 compared
   to either Hackett-2020-ZEV or Kemmeren-2014-TFKO, investigate whether some TFs
   pass in most datasets while others pass in almost none.

   Challenge: Deciding which Hackett condition requires examining Hackett data
   and setting filters such that there is 1 hackett sample per regulator, OR explaining how multiple
   conditions per regulator affects results. In particular, we are interested in the effect over time. Which
   timepoint is best? What is the effect of time? DTO distribution over time is a good output here. Additionally,
   it should be possible to set different filters on a per regulator basis such that if there is a ZEV and GEV sample,
   then we can choose the one that performs best.

   Analysis steps:

   i. Select binding samples from all datasets with DTO vs. Hackett-2020-ZEV P<=0.01

   ii. Select binding samples from all datasets with DTO vs. Kemmeren-2014-TFKO P<=0.01

   iii. Intersect the previous two sets (this is probably a composition of the filter above)

   iv. For regulators in any active-set sample, present number of active samples:
   - As a table: one row per regulator + count
   - As a distribution: across TFs of the count above

[CalvinFang-code]

The readme file for the barkai dataset indicates that it does not have an annotated_features configuration.

[cmatKhan]

barkai doesn't have a control dataset, which we have for the other binding datasets. I have been meaning to calculate normalized counts over promoters for each TF sample. That would give us a distribution over promoters, and we could calculate a z-score. The Barkai lab did that in one of their papers.

Another option is to call significance using the Hahn lab controls.

In any case, leave Barkai out for now. I'll make an issue for it and assign it to myself, but it is something I am already tracking

edit: see #64

[CalvinFang-code]

After several rounds of debugging, I'm very excited. I find the SQL design approach very flexible and easy to use. I submitted the DTO analysis and YAML file; the ID is: bf46a19

The target is the dev branch.

[CalvinFang-code]

Meanwhile, I encountered some issues:

1. The Man2025 dataset seems to lack carbon source information?

2. In the datacard readme file for the hugging face in the Hu 2007 dataset, true and false were not enclosed in quotes, causing an error：
   DataCardValidationError: Dataset card validation failed for BrentLab/hu_2007_reimand_2010:

• Field 'configs -> 0 -> dataset_info -> features -> 8 -> definitions -> 1 -> [key]': Input should be a valid string (got: True)
• Field 'configs -> 0 -> dataset_info -> features -> 8 -> definitions -> 0 -> [key]': Input should be a valid string (got: False)
• Field 'configs -> 0 -> dataset_info -> features -> 9 -> definitions -> 1 -> [key]': Input should be a valid string (got: True)
• Field 'configs -> 0 -> dataset_info -> features -> 9 -> definitions -> 0 -> [key]': Input should be a valid string (got: False)

3. The Barkai dataset lacks annotated_features.

4. The annotated_features for calling cards uses the id column instead of sample_id, which may be the cause of the error.

[cmatKhan]

1. I think you mean mahendrawada 2025. Synthetic complete media will have glucose. Try looking in the paper/methods, please

2. The datacard specifies it as a bool. The error looks like it is being read in as a bool. One of the dtypes that can be used in the virtualDB config is bool. I don't see hu reimand in your config, so I'm not sure if that was specified. If you did try specifying it, and it still tried to parse it as a string, then use breakpoints and the debugger feature in jupyterlab in vscode to step through the code and try to pinpoint where the parsing issue is occurring. Pull requests to fix this are welcome

3. I'm aware. Ignore it for now

4. it should be possible to specify the field in VirtualDB -- see the docs

repositories:
  # Each repository defines a "table" in the virtual database
  BrentLab/callingcards:
    sample_id:
      field: id

if that doesn't work, then same as #2, try to figure out if it is a parsing issue in tfbpapi or if the datacard is malformed/doesn't reflect the underlying data,

[CalvinFang-code]

For question 2:
I initiated a pull request on huggingface
The program crashes when parsing the datacard and encounters true and false. Using dtype in datasets.yaml doesn't solve the problem because the crash occurs before dtype is used. Therefore, I tried adding quotes around true and false.

DataCardValidationError: Dataset card validation failed for BrentLab/hu_2007_reimand_2010:

Field 'configs -> 0 -> dataset_info -> features -> 8 -> definitions -> 1 -> [key]': Input should be a valid string (got: True)
Field 'configs -> 0 -> dataset_info -> features -> 8 -> definitions -> 0 -> [key]': Input should be a valid string (got: False)
Field 'configs -> 0 -> dataset_info -> features -> 9 -> definitions -> 1 -> [key]': Input should be a valid string (got: True)
Field 'configs -> 0 -> dataset_info -> features -> 9 -> definitions -> 0 -> [key]': Input should be a valid string (got: False)

[cmatKhan]

For question 2: I initiated a pull request on huggingface The program crashes when parsing the datacard and encounters true and false. Using dtype in datasets.yaml doesn't solve the problem because the crash occurs before dtype is used. Therefore, I tried adding quotes around true and false.

DataCardValidationError: Dataset card validation failed for BrentLab/hu_2007_reimand_2010:

Field 'configs -> 0 -> dataset_info -> features -> 8 -> definitions -> 1 -> [key]': Input should be a valid string (got: True) Field 'configs -> 0 -> dataset_info -> features -> 8 -> definitions -> 0 -> [key]': Input should be a valid string (got: False) Field 'configs -> 0 -> dataset_info -> features -> 9 -> definitions -> 1 -> [key]': Input should be a valid string (got: True) Field 'configs -> 0 -> dataset_info -> features -> 9 -> definitions -> 0 -> [key]': Input should be a valid string (got: False)

good -- merged. thank you!