Academic research is full of fuff, and sometimes peer-review fails in discriminating between trendy topics from actual, rigorously controlled science.

Equally frustratingly, industry is also full of fuff, and public regulatory bodies sometimes fail in discriminating between a not very remarkable research result and a commercial product of high interest for public health.

The problem is - this shouldn’t happen. While it’s easier for me to excuse the peer-reviewers, who commit their time for free in evaluating someone else’s work*, the governmental bodies regulating commercial medical products have a decision-making role potentially affecting the life of thousands or millions of people. They should not fail.

And still, we have Ziwig. Zigwig is a France-based company making “Ziwig Endotest®, the first salivary diagnostic test for endometriosis”. Here is the webpage of their product, as of 2024-08-21. As written in bold characters, this test would allow “reducing the diagnosis time to just a few days”.

Now, having known people who struggled with endometriosis or with reaching a diagnosis, this statement triggered my interest to say the least.

For the uninitiated, endometriosis is a term to define a poorly understood condition that the FDA estimates to affect more than 1 in 10 women [1]. It is characterised by the growth of endometrium-like tissue outside the uterus. It doesn’t kill, yet it can bring complete havoc in a person’s personal and professional life, with the most evident symptom being intense, debilitating pain. Because the awareness of this condition is so scarce even in the medical community, it is common that many women suffer for years before they even hear the term “endometriosis” from their doctors. Only in the recent years private and public initiatives have started to accelerate the research and societal response to this reality.

So far the only reliable way to diagnose endometriosis is through laparoscopy, a surgical procedure performed under anaesthesia. So, needless to say, considering a women’s health care standard still lacking behind, a potentially very vast market, and the public savings due to health care improvements, a non-invasive and fast new gold standard to diagnose endometriosis could not be given light fast enough. The market opportunity is huge.

Ziwig Endotest® needs a patient’s saliva sample from which the micro-RNA (miRNA) content is extracted and sequenced. It makes a prediction about the disease status (endometriosis / not endometriosis) using a machine learning classifier trained on microRNAome features [2]. While the company doesn’t share the details behind the method, they state they trained a random forest classifier on 109 selected miRNAs, presumably to only include miRNA species that contribute the most to the prediction performance.

At first glance, their claims and reported metrics are very promising and made me internally cheer for that biotech ingenuity that is currently changing our world. However, in this case my enthusiasm diminished at closer inspection.

Their sample size is small and the data is highly unbalanced towards the positive label. There is no investigation of confounding factors and covariates. No mention of comorbidities that could influence the RNA signature. No analysis of degradation of the classification performance as a function of microRNA data quality or subsampling. No mention of biological insights into the oral miRNAome, or why it would be even informative of the presence of endometriosis. No feature importance analysis. Most crucially, no comparison with a null model, generated for example by permuting the labels in the dataset before going through the same feature selection pipeline and model training. As it is presented, the computational part of the study looks like a machine learning exercise that a student, given readily available data, could have done one afternoon on a Jupyter Notebook. Not wrong, but there is no content actually convincing me that the resulting miRNA signature holds any biological or diagnostic relevance for endometriosis.

Anyway, advancements in endometriosis care are urgently needed. Endometriosis is astonishingly prevalent. Progress requires investment. Investments require risks. There would be no progress without risks. So let’s be bold, and assume for a moment that all the reported metrics in Zigwig’s work on the small dataset would, just as they are, generalise to the world’s population. Would Ziwig Endotest® actually translate in a diagnostic tool?

Short answer

Nope.

Long answer

Let’s compute some probabilities to figure out the chances that if you - assuming you are a random woman alive today - were to take Ziwig Endotest®, woud face a correct result, under the condition that you either have or not have endometriosis.

Definitions:

• Event \(E\): woman has endometriosis
• Event \(T\): test returns a positive result
• Event \(\neg T\): test returns a negative result
• \(P(E)\): probability of a random woman to have endometriosis
• \(P(T)\): probability of a test to return a positive result
• \(P(\neg T)\): probability of a test to return a negative result

We want to know:

• \(P(E|T)\): probability of a random woman to have endometriosis given that her test returned a positive result
• \(P(\neg E|\neg T)\): probability of a random woman to not have endometriosis given that her test returned a negative result

Ziwig claims:

• The sensitivity of the test is \(96.2\%\) (\(95\%\) confidence interval [CI], \(93.7\) to \(97.3\%\)), therefore

  \(P(T|E) = 0.962\)

• The specificity (Or true negative rate, \(TNR)\) of the test is \(95.1\%\) (\(95\%\) CI, \(85.2\) to \(99.1\%\)). Since the false positive rate, \(FPR\), equals \(1 - TPR\), it follows that \(FPR=4.9\%\), and therefore

  \(P(T|\neg E) = 0.049\)

Also, from previous studies, we assume:

• The prevalence of endometriosis in the general population of women is \(10\%\), i.e.

  \(P(E) = 0.1\)

For the Bayes theorem: $$ P(E|T)\ P(T) = P(T|E)\ P(E) $$ $$ P(E|T) = \frac{P(T|E)\ P(E)}{P(T)} $$

As beautifully illustrated by skobelevs, we can compute \(P(E|T)\) by thinking about the Bayes theorem geometrically:

$$ P(E|T)=\frac{P(T|E)P(E)}{P(T|E)P(E)+P(T|\neg E)P(\neg E)} = \frac{0.0962}{0.0962+0.9*0.049} = \frac{0.0962}{0.1403}=0.686 $$

So if the test returns positive, there is a probability of less than \(70\%\) for the tested person to actually have endometriosis. That gives rise to quite a lot of false positives. As a comparison, for commercially available COVID-19 antigen lateral flow tests used to screen symptomatic people, this probability would be around \(89\%\) [3]. Importantly, their purpose is also more tolerant to inaccuracies, since they would be used to recommend someone to stay at home for a few days. When your aim is to diagnose an incurable and life-changing condition such as endometriosis, you definitely want more certainty.

What if the test returns negative? What is the probability of the tested woman to NOT have endometriosis given a negative test result, i.e. what is \(P(\neg E|\neg T)\)?

$$ P(\neg E|\neg T) = \frac{P(\neg T|\neg E)\ P(\neg E)}{P(\neg T|\neg E)\ P(\neg E)+P(\neg T|E)\ P(E)} = \frac{0.951 * 0.9}{0.951 * 0.9 + 0.038 * 0.1} = \frac{0.8559}{0.8559 + 0.0038} = 0.9956 $$

So we can conclude that, based on the Ziwig’s published performance metrics, a woman taking their test will face the following scenarios:

• the test returns negative - there is a probability smaller than \(0.5\%\) that the test is wrong. Therefore, the subject can pretty confidently assume they don’t have endometriosis.
• the test returns positive - the subject is relatively likely to have endometriosis, but in almost one case out of three, this won’t be the case.

Therefore, while I could be comfortable with considering Ziwig Endotest® a tool to rule out endometriosis, since a false negative result is extremely rare, I would have a hard time calling it a diagnostic tool. Having a positive test doesn’t give a high enough confidence to diagnose endometriosis, and further assessments, likely with more invasive methods, are required.

I am in no way a clinical practitioner, but just in the light of the minimal statistical arguments presented here, I think a sensible stance from public authorities would be to prevent companies from claiming they are “reducing the diagnosis time to just a few days” when it’s clearly not the case.

Source: https://ziwig.com/en/endometriosis/

References

1. https://www.fda.gov/consumers/knowledge-and-news-women-owh-blog/understanding-endometriosis-symptoms-treatment
2. Bendifallah, S., Dabi, Y., Suisse, S., Delbos, L., Spiers, A., Poilblanc, M., Golfier, F., Jornea, L., Bouteiller, D., Fernandez, H., Madar, A., Petit, E., Perotte, F., Fauvet, R., Benjoar, M., Akladios, C., Lavoué, V., Darnaud, T., Merlot, B., … Descamps, P. (2023). Validation of a Salivary miRNA Signature of Endometriosis — Interim Data. In NEJM Evidence (Vol. 2, Issue 7). Massachusetts Medical Society. https://doi.org/10.1056/evidoa2200282
3. Dinnes, J., Sharma, P., Berhane, S., van Wyk, S. S., Nyaaba, N., Domen, J., Taylor, M., Cunningham, J., Davenport, C., Dittrich, S., Emperador, D., Hooft, L., Leeflang, M. M., McInnes, M. D., Spijker, R., Verbakel, J. Y., Takwoingi, Y., Taylor-Phillips, S., … Van den Bruel, A. (2022). Rapid, point-of-care antigen tests for diagnosis of SARS-CoV-2 infection. In Cochrane Database of Systematic Reviews (Vol. 2022, Issue 7). Wiley. https://doi.org/10.1002/14651858.cd013705.pub3. The table below referes to a population of 1000 symptomatic people taking a commercially available antigen lateral flow test:

Notes

*by perpetuating the giant scam that is for-profit scientific publishing