Disease-specificity benchmark
Preliminary results
Status Exploratory, not peer reviewed
What the first run shows
Logistic regression was chosen as the primary model before any results were seen. The other two are sensitivity analyses.
Cancer-model alerts in other conditions
Each row compares adults with a condition to healthy adult controls from the same study and country. Dots show the difference in alert rates, and lines show 95% intervals. A filled dot means the interval excludes zero, without adjustment for multiple comparisons. Cancer status usually isn't confirmed, so an alert isn't a confirmed false positive.
View as a table
Detection by cancer stage
Bars show the share of people with cancer flagged by a model trained on other studies, with 95% intervals. Samples were collected at or after diagnosis, so this shows detection at a stage, not prediction before diagnosis.
View as a table
Performance on studies the model never saw
Did teaching the model about other diseases help?
Which microbes the model relied on
These species had the largest weights in the primary model. They show what the model used, not what causes cancer.
What these results can't show
- Why alerts happen: differences could reflect a condition, age, medication, geography, or how a study was run.
- Confirmed false positives: comparison cohorts usually weren't confirmed to be cancer-free.
- Clinical performance: these models aren't approved tests and shouldn't guide care.
- Prediction before diagnosis: samples were collected at or after diagnosis.
- Reliable results for small groups: some rows include only a few people, and the Crohn's disease pilot has 43 adults.
- Truly healthy controls in every study: one Crohn's study's "healthy" metadata label covers non-IBD participants, including some with symptoms.
- One-study independence: several conditions come from one study, and one large cohort supplies most of the type 2 diabetes data.
How it was done
Data came from curatedMetagenomicData, using species-level profiles and one sample per person. Models were tested by leaving out each of 11 cancer studies in turn. The alert threshold aimed for 90% specificity using training data only.
An analysis plan was recorded before any profiles were downloaded or models were run. Two data errors were found by checks during the run: mixed taxonomic ranks and repeated sample IDs. Those results were discarded, the errors were fixed, and the changes were logged.