# Biomarker evaluation - c-statistic (AUC) and alternatives

**URL:** <https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956>\
**Category:** data analysis\
**Created:** [July 25, 2023, 11:30am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956 "2023-07-25T11:30:22Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [April 10, 2024, 12:55pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/21 "2024-04-10T12:55:10Z")

</div>

The reviewer has made very common but tragic mistakes. There are many papers showing extreme problems with all the methods the reviewer likes. May need to go to a journal that has qualified statistical reviewers. Start with papers by Margaret Pepe. Bad power from studying changes in c-index can be easily simulated; can’t put my finger on a paper. But if you keep trying that journal you might mention that the original developer of the c-index backs you up.

---

<div class="post-metadata">

**Author:** ![JuliaRose](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/juliarose/32/5585_2.png) [@JuliaRose](https://discourse.datamethods.org/u/JuliaRose)\
**Post date:** [April 10, 2024, 3:04pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/22 "2024-04-10T15:04:57Z")

</div>

Thank you very much for your reply. I will definitely start with the Pepe ones. I wish all journals had qualified statistical reviewers…  
Do you perhaps have a good reference to cite, where is stated that p-values for differences in c-statistics and p-values for NRI and IDI are problematic? Would really much help me.  
I am currently trying to familiarise myself with the literature on this, but as I only have a layman’s background in statistics, this is of course very challenging.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [April 11, 2024, 11:49am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/23 "2024-04-11T11:49:42Z")

</div>

I’m sure someone can find such a reference for us. Think about it this way: Differences in c equate to differences in Wilcoxon statistics. If doing Wilcoxon tests to compare A with B and B with C no one computes the difference in two Wilcoxon stats to compare A with C. That would detect weird alternatives and have poor power.

---

<div class="post-metadata">

**Author:** ![kiwiskiNZ](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/kiwiskinz/32/42_2.png) [@kiwiskiNZ](https://discourse.datamethods.org/u/kiwiskiNZ)\
**Post date:** [April 18, 2024, 4:17am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/24 "2024-04-18T04:17:31Z")

</div>

Ugh… I feel for you Julia.  
@f2harrell giving permission to name drop him is great!

Margaret Pepe’s papers about NRI:

1. Pepe MS, Fan J, Feng Z, et al. The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets. _Stat Biosci_ 2015; 7: 282–295.

2. Pepe MS. Problems with risk reclassification methods for evaluating prediction models. _Am J Epidemiol_ 2011; 173: 1327–1335.

I think Jorgen Gerds wrote about the IDI and issues when there was miscalibration of models.

---

<div class="post-metadata">

**Author:** ![llynn](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/llynn/32/2827_2.png) [@llynn](https://discourse.datamethods.org/u/llynn)\
**Post date:** [April 18, 2024, 4:09pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/25 "2024-04-18T16:09:58Z")

</div>

Julia, I defer to the world class experts here but there are clinical considerations which I have learned over the past 4 decades of studying the clinical behavior of relational time series matricies of biomarkers, matrix patterns and derivatives. These affect the validity of the statistics.

It is first important to consider:

1. What type of biomarker(s) you are investigating?

Biomarkers may be derived directly from the condition under test (hcg) or indirectly from a complication (creatinine), a general mortality signal (lactate) or a physiologic response to the condition (WBC)?

1. What type of condition you are investigating?

A well defined disease or a broadly defined synthetic syndrome comprised of many diseases?

1. What is the volitility of the biomarker. (what is the derivative range of the biomarker with and without the condition under test )?

2. What factors affect the biomarker volitility and is this volitility a function of the condition?

3. Is the biomarker depletable or biphasic. (WBC is a depletable biomarker often high early and low late)?

4. Does the biomarker have a high normal phenotypic range. (eg platelets)?

IMHO, these are pivotal considerations and it is intuitive that these account for much of the failure of AUROC as a stand alone biomarker test.

In an example, bands (a type of immature neutrophil), were abandoned in many hospitals because biomarker studies demonstrated that the absolute neutrophils had a better AUC. However both are “response biomarkers”, moderately volatile biomarkers, and bands are late markers whereas absolute neutrophils are both early markers and depletable.

Had the authors of these studies engaged in due considerations of the above relational times series matrix factors they would not have determined that absolute neutrophils were superior to (and could replace) bands.

The loss of bands as a biomarker in many hospital systems remains and has echoed in the halls of the ICU as a function of very late detection of life-threatening infection for decades.

![1000012855](https://discourse.datamethods.org/uploads/default/original/2X/6/6894e03087f5606d99d1a8dd0b7ea1991828ec9d.jpeg)

---

<div class="post-metadata">

**Author:** ![JuliaRose](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/juliarose/32/5585_2.png) [@JuliaRose](https://discourse.datamethods.org/u/JuliaRose)\
**Post date:** [February 12, 2026, 8:33pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/26 "2026-02-12T20:33:47Z")

</div>

Published yesterday in Circulation: [https://doi.org/10.1161/CIR.0000000000001401](https://doi.org/10.1161/CIR.0000000000001401)  
It’s an AHA Scientific Statement: _“Criteria to Assess the Predictive and Clinical Utility of Novel Models, Biomarkers, and Tools for Risk of Cardiovascular Disease: A Scientific Statement From the American Heart Association”_

I am wondering, why the “changes in the C-statistic” is again highlighted as well as “Categorical NRI […] at event rate […]” and in the whole statement there is not even one sentence about Likelihood ratio importance?? 🙄

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [February 13, 2026, 8:56pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/27 "2026-02-13T20:56:18Z")

</div>

This is way behind the times, and NRI has been particularly criticized in biomarker research.

---

<div class="post-metadata">

**Author:** ![JuliaRose](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/juliarose/32/5585_2.png) [@JuliaRose](https://discourse.datamethods.org/u/JuliaRose)\
**Post date:** [February 14, 2026, 12:32am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/28 "2026-02-14T00:32:13Z")

</div>

Way too sad, that this statement got published and will now always be referenced by clinicians/cardiologists for „best practice“ 😓  
As kind of a junior researcher it is very hurtful to see this and already knowing that the fights against senior clinicians regarding LR and not doing NRI and testing delta c-statistics, will now be even more painful….

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [February 14, 2026, 1:49pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/29 "2026-02-14T13:49:26Z")

</div>

Human nature disappoints me in so many ways. One of them is the laziness involved in not wanting to learn better approaches to problem solving and life in general. Too many researchers continue to rely on the first method they learned, or trust what their colleagues have published or what their favorite journal allows people to get away with.

---

<div class="post-metadata">

**Author:** ![JiaqiLi](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/j/74df32/32.png) [@JiaqiLi](https://discourse.datamethods.org/u/JiaqiLi)\
**Post date:** [March 24, 2026, 4:13am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/30 "2026-03-24T04:13:46Z")

</div>

This is a complaint.

I used decision curve analysis in my paper and wrote the following expression. A collaborator told me that I was treating clinicians like idiots. I might lack sufficient knowledge, but I don’t understand why saying this is considered treating clinicians like idiots. Is this sentence offensive?

> However, the C statistics are challenging for clinicians to interpret. The incremental predictive value of triglyceride levels requires interpretable metrics for quantification.

---

<div class="post-metadata">

**Author:** ![ESMD](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/e/9e8a1a/32.png) [@ESMD](https://discourse.datamethods.org/u/ESMD)\
**Post date:** [March 24, 2026, 12:08pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/31 "2026-03-24T12:08:52Z")

</div>

Hi Jiaqi

As a clinician who just googled “C-statistic,” I’m not offended by what you wrote. Most clinicians have only rudimentary statistical training at best. If someone with statistical training considers that a statistic could be tricky for clinicians to understand, it almost certainly _will_ be.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [March 24, 2026, 1:26pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/32 "2026-03-24T13:26:45Z")

</div>

In line with that I would challenge the offended physician to interpret a difference in two concordance probabilities. I’m not holding my breath for a correct answer.

---

<div class="post-metadata">

**Author:** ![Uriah](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/uriah/32/1280_2.png) [@Uriah](https://discourse.datamethods.org/u/Uriah)\
**Post date:** [March 24, 2026, 4:17pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/33 "2026-03-24T16:17:46Z")

</div>

Honestly I do think c-statistics are quite interpretable, but not very useful for clinical decision-making.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [March 24, 2026, 6:36pm UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/34 "2026-03-24T18:36:11Z")

</div>

Yes I think they are interpretable by statisticians and some others but are not on the needed clinical scale (as opposed to distributions of assigned risk e.g. [this](https://fharrell.com/post/add-value)).

---

<div class="post-metadata">

**Author:** ![JiaqiLi](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/j/74df32/32.png) [@JiaqiLi](https://discourse.datamethods.org/u/JiaqiLi)\
**Post date:** [March 25, 2026, 1:53am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/35 "2026-03-25T01:53:02Z")

</div>

My understanding is that the C statistic of a simple model is interpretable and is an essential metric. What I intend to convey is that comparing C statistics is difficult to interpret. Adding a new factor to a model typically yields small improvements. For example, from 0.70 to 0.72, the practical interpretation of such a result is unclear. Or is this difference interpretable?

---

<div class="post-metadata">

**Author:** ![Uriah](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/uriah/32/1280_2.png) [@Uriah](https://discourse.datamethods.org/u/Uriah)\
**Post date:** [March 25, 2026, 6:46am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/36 "2026-03-25T06:46:24Z")

</div>

The difference between c-index of 0.72/0.70 means that the first model has 2% higher chance to guess the event in an imaginary game that samples one event and one non-event from the validated data.

The real issue is not the interpretability, but the relevance for decision making.

Considering new biomarkers must come with cost, c-index can’t help you with these kind of decisions.

PS: That’s my attempt to explain c-index (and why I don’t like “area-under-curve” interpertation):

> **[discrimination](https://predictionperformancediscrimination.netlify.app/#/why-i-dont-like-roc-curve-8)**

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [March 25, 2026, 11:30am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/37 "2026-03-25T11:30:24Z")

</div>

Yes, the AUC being the concordance probability is the only good thing about the curve. But it should only be used when there is a single model under consideration, as concordance probabilities are too insensitive to detect added information. I’ve never seen a helpful ROC curve.

---

<div class="post-metadata">

**Author:** ![JiaqiLi](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/j/74df32/32.png) [@JiaqiLi](https://discourse.datamethods.org/u/JiaqiLi)\
**Post date:** [March 26, 2026, 1:38am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/38 "2026-03-26T01:38:56Z")

</div>

Methodologically this interpretation is valid. My point regarding inexplicability refers to the limited clinical interpretability or practical utility. I agree with your reasoning and think your slides are great and very easy to understand.  
I would like to add that the C statistic yields the same concordance when comparing non-cases and cases whether the predicted probabilities are 0.3 and 0.7 or 0.1 and 0.9 because it only evaluates the rank order. The latter may represent better prediction as the values approach 0 and 1. . This is another reason why I find a 2% difference between two C difficult to interpret. Even disregarding costs a 2% margin can hardly be characterized as a correct improvement in prediction.

---

<div class="post-metadata">

**Author:** ![Uriah](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/uriah/32/1280_2.png) [@Uriah](https://discourse.datamethods.org/u/Uriah)\
**Post date:** [March 26, 2026, 10:09am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/39 "2026-03-26T10:09:41Z")

</div>

Honestly the only performance metrics I care about from the “Discrimination” category are PPV and Lift - but they also hardly related to patient-specific decision making and more about organization-resource-constraint type of decision making.

---

<div class="post-metadata">

**Author:** ![JiaqiLi](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/j/74df32/32.png) [@JiaqiLi](https://discourse.datamethods.org/u/JiaqiLi)\
**Post date:** [March 26, 2026, 10:39am UTC](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956/40 "2026-03-26T10:39:37Z")

</div>

The final goal of prediction is to provide information that supports clinical decision-making, and I believe we are aligned on this point. This is also why I am angry. I am trying to provide information that is suitable for explaining to physicians and patients and for facilitating decisions (at least that is what I think), then I am being accused of treating physicians as if they lack expertise.😩

[Previous page](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956.md?page=1)

[Next page](https://discourse.datamethods.org/t/biomarker-evaluation-c-statistic-auc-and-alternatives/6956.md?page=3)