A framework for deciding to measure a costly predictor for prognostication

1

Disease A is always preceded by a variably long non-disease state that is defined by an elevation in relatively cheap and widely available blood biomarker X. A relatively small proportion of those with an elevation in biomarker X progress to disease A, and those that do might do so anywhere between one and 30 years from initial measurement. An elevation in X is fairly common and it is not feasible to monitor all persons with an elevation with regards to progression.

Several other cheap and widely available biomarkers and demographic variables, together with the absolute value of X, are useful for prognostication with regards to progression to disease A. However, an expensive and/or otherwise difficult to obtain biomarker Y exists, which strongly predicts progression to disease A. Biomarker Y correlates strongly (but not perfectly) with X and the other easily available prognostic variables.

A clinician is consulted regarding a patient with an elevation in X and tasked with deciding whether the patient should be monitored for progression to disease A, and if so, how frequently. The results of X and the other easily obtainable prognostic variables are available. The decision could be seen to have multiple sub-components:

  • what is the clinician’s (and patient’s) risk threshold taking into account the potential costs and benefits of monitoring?
  • how close is the predicted risk of progression to this risk threshold based on the results of a multivariable prognostic model using the cheap and widely available prognostic variables?
  • how likely is it that the risk threshold would be crossed (in either direction) given the results of a multivariable prognostic model that also incorporates biomarker Y?
  • does this likelihood justify the additional cost of obtaining Y?

I have been tasked with developing a multivariable prognostic model of progression to disease A on a large prospective cohort of patients with an elevation in biomarker X. All individuals in this cohort have undergone testing of biomarker Y, regardless of underlying risk. I therefore believe I have an unbiased sample of the relationship between X & other widely available biomarkers, and Y. Instead of just developing a single multivariable prognostic model that includes X, Y, and the other prognostic variables, I propose developing a set of models:

  1. A multivariable prognostic model of progression to disease A, given X and the other easily obtainable prognostic variables.
  2. A multivariable prognostic model that predicts disease A, given X, Y and the other easily obtainable prognostic variables.
  3. A multivariable model that predicts the difference in predicted prognosis of disease A, between the prognostic model excluding Y and the prognostic model including Y.

Given that the results of these models are clearly presented, it seems to me that this approach could inform all the decisions the clinician faces in this common clinical scenario. As long as the models are jointly evaluated both for internal and external validation, I don’t see any downsides to this approach.

Are there any obvious flaws in my logic? Is this a common approach to prognostic models, has it been done before and is there any literature on the topic? I haven’t been able to find anything.

3 Likes
2

There is a fairly extensive value-of-information literature that you might like to avail yourself of here, Elias. @Drew_Levy and I wrote an article 7 years back that attempted to lift the VOI literature of the time into the domain of precision medicine—with specific reference to heterogeneity not only of physiologic kinds but also regarding “individuals’ values and life circumstances” [1]. The latter would be relevant to the individual risk thresholds you’re attempting to address. Basu & Meltzer as I recall have discussed VOI relative to individualized care [2]. You might find an up-to-date bibliography on VOI as applied in health econ policy in [3] (which I haven’t read), but maybe an operations-research outlook on VOI would be better suited to your application.

  1. Norris, David C., and Drew G. Levy. “Precision Medicine Is a Value-of-Information Proposition—and Vice Versa.” The Journal of Precision Medicine 1, no. 1 (June 2015): 57–63.

  2. Basu, Anirban, and David Meltzer. “Value of Information on Preference Heterogeneity and Individualized Care.” Medical Decision Making 27, no. 2 (March 1, 2007): 112–27. https://doi.org/10.1177/0272989X06297393.

  3. Tuffaha H, Rothery C, Kunst N, Jackson C, Strong M, Birch S. A Review of Web-Based Tools for Value-of-Information Analysis. Appl Health Econ Health Policy. 2021;19(5):645-651. doi:10.1007/s40258-021-00662-4

3 Likes
3

Hi,

In reading your post, my mind immediately went to prostate cancer (PCa) as a parallel situation, if indeed, your domain of interest here is not actually PCa.

As you likely are aware, the cheap and widely accessible test in PCa is the PSA test, sometimes augmented by evaluating the free PSA level. Elevation over a number of years, which happens for most men, is one signal for possible PCa, albeit, it can also be associated with benign prostatic hyperplasia. Of course, physical exam, the digital rectal exam (DRE), is also a first step in the diagnosis process, as well as evaluating family history.

The gold standard test today is still the prostate biopsy, whether via trans-rectal or trans-perineal approach. However, with the standard 12 core biopsy alone, the risk of false negatives can still be ~30%.

In recent years, the multi-parametric MRI (mpMRI) has become more popular in use (especially outside the US) as a first possible next step in PCa diagnosis, and importantly, as a possible screen for whether or not to do a biopsy, given the cost, inconvenience, infection and bleeding risks. With current 3T MRI units, in the hands of competent readers and an imaging team (not always available), the false negative rate is ~15%, and when it misses lesions, they tend to not be clinically significant. However, if the mpMRI identifies densities, those can be used in a fusion 3D biopsy to enhance the accuracy of the biopsy, since the needles can be targeted to the densities in the MRI imaging.

If the mpMRI is negative, and certainly if the biopsy is negative, in the face of an elevated PSA, the patient will likely end up on a semi-annual active monitoring plan, which will include the DRE and PSA testing. That will serve as the screening plan until parameters give rise, at some point, to considering a repeat mpMRI and biopsy.

In the mean time, a number of biomarker based tests have evolved, in an effort to better screen men for PCa. The objective being able to better identify those with elevated PSA levels that also have a high risk of having or progressing to PCa, and importantly, trying to differentiate the likelihood of aggressive versus non-aggressive PCA, to avoid the more expensive and invasive mpMRI and biopsy tests. These include the 4K Score, the PHI, the Decipher score and the Stockholm 3 score. However, these are early in their development, not yet widely or consistently available, and many urologists are not yet convinced of their utility.

Thus, bottom line, even if PCa is not your application, I would take time to review the numerous models that are evolving in that domain, given the similarities in the decision making process to your description. Many of the same considerations apply regarding the ease and costs of obtaining some of the selected predictors, that can potentially enhance clinical decision making.

From a general model development context, I would review Frank’s book, “Regression Modeling Strategies”, and Ewout Steyerberg’s book, “Clinical Prediction Models”. Both are excellent references.

Lastly, it was not clear to me, if in your cohort, a meaningful proportion of the patients already have the disease of interest, or that you expect them to develop it, as you monitor them over time, which presumably, could be decades. I presume that your plan is to be able to determine if the new biomarker enhances the prediction of the presence of the disease over the existing methods, such that the increased cost/inconvenience is justified?

2 Likes
4

Thank you for these references. I was unable to find a copy of " 1. Norris, David C., and Drew G. Levy. “Precision Medicine Is a Value-of-Information Proposition—and Vice Versa.” The Journal of Precision Medicine 1, no. 1 (June 2015): 57–63." but the others were informative.

My main point was that at the time of clinical decision making the clinician is faced with two decisions: i) Can I recommend no clinical monitoring given my current knowledge of the patient? and ii) if not, do I believe that an additional test will enable me to make this decision?

Presumably, for a subset of patients, the likelihood of progression to disease A is so small and the correlation between the currently known predictors and the costly unknown predictor Y so strong, that a measurement of Y can safely be deferred and a decision to not monitor be made without collecting Y. Furthermore, for a subset of patients, the likelihood of progression to disease A is so strong, that a decision to monitor can already be made without measuring Y. In only a subset of patients who fall into a category in which the additional information contained within Y has a reasonable probability of changing the clinicians decision, would a measurement of Y be of benefit.

What I am wondering, is whether there is any literature on how to approach this. My idea is to present the decision maker / clinician with the predicted probability of progression given the available data and the probability that measuring Y will change predicted probability by a “significant amount”. Given the risk threshold of the clinician they can then decide whether a decision can be made given the available data, and if not, whether measuring Y is likely to make a decision possible.

5

Thank you for this very informative discussion. Prostate cancer is a similar clinical scenario to mine that I hadn’t thought of, and it seems that there is quite some literature on this exact topic in the prostate cancer domain (even some by Andrew Vickers!), as you have laid out. I will study this and share any references that I find to this thread.

6

This approach introduces arbitrary thresholds (you note one of these yourself inside ‘scare quotes’) that circumvent a fully Bayesian decision-theoretic framing of the problem. What is the (patient-centered) utility of monitoring for disease progression? Is there some intervention which the monitoring enables us to apply before disease becomes clinically evident? What is the value of intervening earlier vs later? Sidestepping these crucial questions will make a coherent decision analysis impossible, I think.

1 Like
7

Early treatment has been shown to prolong survival. But the precursor condition (elevation in X) is too common to offer monitoring to everyone. Thus, a prognostic model is needed.

8

Can you quantify the extension of life? If so, then you would obtain a common denominator for evaluating the various decisions. This would avoid the introduction of dubious additional concepts such as changing a predicted probability by a “significant amount” etc. What features of the problem are persuading you to depart from the standard practice of modeling decisions as utility maximization, and to substitute a novel change-in-probability doctrine?

1 Like
9

This sounds like a situation where the crucial decision point is whether to test for the presence of X in the first place.

I can imagine that you might be describing a scenario in which a patient has a family history of a certain disease, for which the biomarker X is invariably present. Presumably, the patient is anxious about his potential to develop the disease and the physician decided to test for X in the hope that a negative test might put the patient’s mind at ease (i.e., in the absence of X, he can now stop worrying that he will develop the disease). But the physician’s plan has now backfired, since the patient has now been found to have biomarker X and his anxiety has unfortunately increased. Now, there are three options:

  1. Subject the patient to intensive monitoring for disease development, even though his absolute risk for developing the disease might actually be very low (i.e., forgo testing for biomarker Y and proceed with intensive monitoring for disease development). Whether this is a reasonable decision will depend on monitoring cost/invasiveness, and effectiveness. If the patient’s health system will not cover the cost of such surveillance (which might be very likely given the high prevalence of X among people destined never to get the disease), then this approach will not be feasible.

  2. Do nothing further. This option is not ideal, since the patient is left feeling anxious about his unclear prospect of developing the disease. The ordering physician should have carefully counselled the patient about this possible outcome prior to testing him for the presence of X.

  3. Offer the patient testing for biomarker Y, to further delineate his risk for developing the disease. At this point, there are several possible scenarios:

  • The patient can not afford testing for biomarker Y and his health system will not cover its cost. This is an undesirable scenario, as it leaves the patient in a kind of limbo, constantly worrying about a disease that he might never be destined to develop. The feasibility, purpose, and potential outcomes of testing for Y should have been considered at the initial decision point (i.e., prior to testing the patient for biomarker X);

  • The patient is able to access testing for Y, but he can not afford the disease surveillance that would be recommended if he were to test positive (and his health care system would not cover the cost of surveillance). This is an undesirable scenario, as the patient has a strong reason to believe he will one day develop the disease, and yet feels powerless to decrease his risk. The only foreseeable upside to this scenario is that the patient could plan his life around the expectation that he is likely to develop the disease at some point;

  • The patient is able to access testing for Y and his health system will pay for disease surveillance if he tests positive. Provided that there is good evidence that the disease in question is not so aggressive that periodic monitoring has limited ability to detect an early stage of the disease AND that early treatment indeed improves prognosis (rather than simply reflecting length time bias), this scenario would be the optimal one.

All of the above scenarios should have been considered by the physician prior to ordering the test for X.

2 Likes