Handling nonsense ordinal outcome predictions

[martinspn]

Hi everyone,

I am developing a predictive model for the 90-day modified Rankin Score (mRS) among hospitalized patients who had strokes. One of the selection criteria was pre-stroke mRS ≤ 2 (patient lives independently). The ordinal model considers age, pre-stroke mRS and some other predictors.

It is very, very unlikely, not to say semi-impossible, for a patient to improve their mRS to the point of getting better than pre-stroke mRS after a hospitalization due to stroke. Whenever that happens, chances are there was measurement error of pre-stroke mRS or 90-day mRS. Even in this case the “improvement” would mean no more than 1 point.

However, some patients with pre-stroke mRS of 2 but favorable distributions of other predictors, like age and other clinical variables, still have quite high estimated probabilities of 90-day mRS 0 or 1, particularly 1. Even for usual predictor combinations, the probabilities of 90-day mRS better than pre-stroke mRS float around 4-5%.

To address that, would it be sound to manually assign a intercept of -Inf/+Inf, depending on the original or reversed scale being used, to 90d mRS < 2 for patients with pre-stroke mRS 2? Then, the new estimated probabilities of mRS = 2 would reflect the current estimated probabilities of mRS 0, 1 and 2 combined. A similar procedure would be done for pre-stroke mRS 1.

Otherwise, how can I handle such issue? Should I just leave it as it is?

Thanks in advance!

[FurlanLeo]

@martinspn

It is very, very unlikely, not to say semi-impossible, for a patient to improve their mRS after a hospitalization due to stroke for any reason other than measurement error of pre-stroke mRS or 90-day mRS. Even when there is measurement error the “improvement” would mean no more than 1 point.

Hi Pedro, I’m intrigued by this. What do you mean exactly? That stroke pts do not improve their mRS score after the stroke?

[martinspn]

@FurlanLeo It is likely that they will improve when compared to discharge mRS, but not to the point of getting better than their pre-stroke mRS. The probabilities of 90d mRS better than pre-stroke mRS are the ones annoying me. I’ve edited the question to make it clearer, thanks

[f2harrell]

I would wager that this fix does not bring an improvement that is proportional to the risk it adds of not fitting. Just be sure that the baseline mRS is modeled as categorical even though the follow-up mRS is modeled as ordinal.

[Elias_Eythorsson]

Why not model baseline mRS as ordinal?

[f2harrell]

Rarely, a baseline ordinal variable will not be monotonically related to the follow-up measurement. But the real challenge is we don’t have a good frequentist way to model predictors as ordinal. Using the brms package in R you can handle ordinal predictors elegantly. If there are only a few distinct values of baseline mRS it’s not worth the trouble and you can use use it as nominal.

[martinspn]

@f2harrell

I would wager that this fix does not bring an improvement that is proportional to the risk it adds of not fitting.

I don’t see why it would hinder fitting - I would modify the intercepts after getting the model fit, not before. It’s the mathematical equivalent of collapsing the estimated probabilities of mRS 0, 1 and 2 for patients with pre-stroke mRS 2, for example.

[f2harrell]

Haven’t thought about that -

[EpiMD5]

Here is a common definition of what the mRS is meant to do and a description of the 7 categories of the mRS copied from MEDCALC.

Modified Rankin Scale for Neurologic Disability

“Measures the degree of disability or dependence in the daily activities of people who have suffered a stroke or other causes of neurological disability.”

mRS Category Description

|0| No symptoms at all
|1| No significant disability despite symptoms; able to carry out all usual duties and activities
|2| Slight disability; unable to carry out all previous activities, but able to look after own affairs without assistance
|3| Moderate disability; requiring some help, but able to walk without assistance
|4| Moderately severe disability; unable to walk and attend to bodily needs without assistance
|5| Severe disability; bedridden, incontinent and requiring constant nursing care and attention
|6| Dead

The mRS was developed as a global measure of disability in patients who have had a stroke. The specific wording for categorization at, for example, a time 3 months after an acute stroke refers, implicitly (category 1) and explicitly (category 2), to a time before the acute stroke.

You posit that:

It is very, very unlikely, not to say semi-impossible, for a patient to improve their mRS to the point of getting better than pre-stroke mRS after a hospitalization due to stroke.”

This statement is surely true considering the mRS category of 6 (Dead) pre-stroke. Indeed, the mRS category of 6 obviously cannot exist pre-stroke.

It also seems, as you point out, highly unlikely even impossible, for a person admitted to the hospital with an acute stroke (or suffering an acute stroke while in the hospital for something else) and who was, pre-stroke, “severely disabled; bedridden; incontinent and requiring constant nursing care and attention”–(mRS category 5)–to move to a lower mRS category after an acute stroke.

However, as I understand it, patients eligible for your analysis have been selected to have a mRS<=2 at the time of their acute stroke.

You describe these patients as:

“patient lives independently”

But this is not the definition of mRS categories <=2.

In my opinion, a clinician or someone else who attempts to “assign” a mRS category of 0, 1, or 2 using the standard descriptions in the table to a patient with an acute stroke is going to be more than a bit confused.

What does it mean to have “no symptoms at all”–the descriptor for mRS category 0—at the time of an acute stroke? Does it mean no symptoms of neurologic disease? More importantly, a person with no symptoms of neurologic disease probably has not had an acute stroke.

Category 2 doesn’t make a lot of sense when considering status at the time of admission to the hospital with an acute stroke because it says “….unable to carry out all previous activities.” The patient’s functional status will evolve over time after an acute stroke. It would be impossible for a rater assessing the patient at the time of an acute stroke to know at this “baseline” time whether, at later time, the patient will be able to carry “all previous activities.”

If mRS categories 0, 1, and 2 can be redefined as “patient lives independently” at the time of the acute stroke based on empiric data, in my opinion, it logical to simply combine them for the purposes of your modeling.

Or simply point out that applying the standard descriptors of categories of the mRS to patients with an acute stroke at the time of the acute stroke does not make sense and may have confused the raters, leading to misclassification that invalidates the use of the individual categories 0, 1, and 2 as “baseline” measures of functional status.

I agree, the categories 0, 1, 2 as separate categories at some “baseline” are nonsense.

[martinspn]

Thanks for such a thoughtful comment, @EpiMD5.

@EpiMD5

The mRS was developed as a global measure of disability in patients who have had a stroke.

@EpiMD5

The specific wording for categorization at, for example, a time 3 months after an acute stroke refers, implicitly (category 1) and explicitly (category 2), to a time before the acute stroke.

I completely agree with the first sentence, but only partially with the second. Although the mRS original idea was a pre-post stroke comparison, it is reasonable to assume most patients will not have any pre-stroke disability. This is the reason why the scale is often used as a disability scale per se, not only as a comparison of post- versus pre-stroke functional status. Except for the transitions from 0 to 1 and 1 to 2, every change in the scale is defined by an easy to evaluate deficit that does not rely on the previous functional status of the patients.

@EpiMD5

However, as I understand it, patients eligible for your analysis have been selected to have a mRS<=2 at the time of their acute stroke.

You describe these patients as:

“patient lives independently”

But this is not the definition of mRS categories <=2.

In my opinion, a clinician or someone else who attempts to “assign” a mRS category of 0, 1, or 2 using the standard descriptions in the table to a patient with an acute stroke is going to be more than a bit confused.

The baseline classification refers to the state of the patient in the week or month before the stroke, not to the acute setting. In the emergency, we use the NIHSS for severity assessment; however, knowing the level of pre-existent disability is important for determining treatment and expectations of recovery. There are other possible scales like GOS, but, since the mRS is how the outcome will be assessed, I think it’s sound to use it to assess baseline dependency.

Regarding the definition of mRS ≤ 2, “Requiring some help” is the core of the transition between mRS 2 and 3 which defines the outcome of functional independence (e.g., as a coprimary endpoint in the DAWN Trial). That is why I used “patient lives independently” in a broader sense, but I agree it is not the exact definition.

@EpiMD5

You posit that:

It is very, very unlikely, not to say semi-impossible, for a patient to improve their mRS to the point of getting better than pre-stroke mRS after a hospitalization due to stroke.”

This statement is surely true considering the mRS category of 6 (Dead) pre-stroke. Indeed, the mRS category of 6 obviously cannot exist pre-stroke.

It also seems, as you point out, highly unlikely even impossible, for a person admitted to the hospital with an acute stroke (or suffering an acute stroke while in the hospital for something else) and who was, pre-stroke, “severely disabled; bedridden; incontinent and requiring constant nursing care and attention”–(mRS category 5)–to move to a lower mRS category after an acute stroke.

@EpiMD5

I agree, the categories 0, 1, 2 as separate categories at some “baseline” are nonsense.

I agree that these cases are very clear, but milder pre-existent disabilities (e.g, a patient who had a previous stroke with some residual deficits or a patient who is otherwise independent but cannot drive due to arthritis) are not expected to improve after a stroke either. If anything, they remain the same. This is why I thought about collapsing the outcome probabilities mRS 0, 1 and 2 for patients with pre-stroke mRS 2 and collapsing the probabilities of mRS 0 and 1 for patients whose pre-stroke mRS was 1.

[EpiMD5]

As I now understand your plan, for patients who pre-stroke have “a slight disability but are able to look after own affairs without assistance (definition of mRS 2),” the following mRS outcome categories would be collapsed.

|0| No symptoms at all
|1| No significant disability despite symptoms; able to carry out all usual duties and activities
|2| Slight disability; unable to carry out all previous activities, but able to look after own affairs without assistance

Are patients with collapsed outcomes of mRS 0, 1, and 2 assigned a value of 2 for the outcome? Is this collapsed outcome category going to be called an mRS or will it be given another name?

As I understand it, the second part of the plan is as follows. For patients who pre-stroke have “no significant disability and are able to carry out all usual duties and activities (definition of mRS 1),” the following mRS outcome categories are collapsed:

|0| No symptoms at all
|1| No significant disability despite symptoms; able to carry out all usual duties and activities

Are patients with collapsed outcomes of mRS 0 and 1 assigned a value of 1 for the outcome? Is this collapsed category going to be called an mRS or will it be given another name?

If the collapsed outcome categories are assigned values of 2 and 1, the plan assures that patients with pre-stroke slight disability do not “get better” after their stroke. Patients with “symptoms” who are able to carry all usual duties and activities do not become asymptomatic.

But collapsing the values of a measured outcome depending on pre-stroke values of the mRS may be a “hard sell” in the sophisticated world of stroke outcomes research.

In my opinion, the problem is with trying to apply definitions meant to assess outcome (i.e., the mRS) to the assessment of pre-stroke disability status. Exactly who is entering a score for pre-stroke mRS and what definitions are these people being told to use in order to enter that pre-stroke score?

Not an easy problem to handle!

[martinspn]

Thanks for the reply!

I will use the case for pre-stroke mRS 2 to avoid being repetitive, but a similar reasoning could be applied to a pre-stroke mRS of 1.

@EpiMD5

Are patients with collapsed outcomes of mRS 0, 1, and 2 assigned a value of 2 for the outcome? Is this collapsed outcome category going to be called an mRS or will it be given another name?

The patients would be modeled with their actual outcome values - there are no cases with pre-stroke mRS 2 and outcome mRS < 2. However, the ordinal model considering pre-stroke mRS as a fixed effect will still assign non-null probabilities of mRS 0 and 1 to such patients based off the estimated intercepts for outcome mRS 0 and 1 in the ordinal logistic model.

I would then either manually set the intercepts (corresponding to mRS 0 and mRS 0-1) to -Inf/+Inf or replace the estimated probability of mRS = 2 in these cases with the sum of the estimated probabilities of mRS 0, 1 and 2 - such approaches would be mathematically equivalent as the procedures would be applied after model fitting. This estimated outcome would be mRS 2 (no recoding).

@EpiMD5

But collapsing the values of a measured outcome depending on pre-stroke values of the mRS may be a “hard sell” in the sophisticated world of stroke outcomes research.

The collapsed values would not be the measured outcomes’, as we do not have measured outcomes at the two lowest levels. The mismatch between this observation and the persistent estimation of non-null probabilities to these levels despite strong empirical data and pathophysiological rationale for believing they should be null is exactly what bothers me.

While I agree collapsing seems like oversimplifying something that may require deeper inspection, it seemed a reasonable approach and I could not think of any potentially better one. Now I wonder if a partial proportional odds model with PPO for pre-stroke mRS would be a better option, but I am really not sure - at the bottom line, it may do pretty much the same of setting the intercepts to -Inf/+Inf or collapsing the probabilities.

@EpiMD5

In my opinion, the problem is with trying to apply definitions meant to assess outcome (i.e., the mRS) to the assessment of pre-stroke disability status. Exactly who is entering a score for pre-stroke mRS and what definitions are these people being told to use in order to enter that pre-stroke score?

Emergency/stroke doctors assess the baseline mRS using answers from relatives to questions like “Could the patient walk by him/herself before the stroke?”, “Could he/she be stay alone at home?”, “What kind of tasks did he/she need help to do?” and many more possible. Although not very direct, these are usually enough for the doctor to know how dependent the patient is. While I second the opinion that using the outcome definition to assess baseline status is not ideal, I doubt that any assessment just as feasible in the acute setting would differ enough from the mRS to justify switching.

@EpiMD5

Not an easy problem to handle!

Absolutely!

[EpiMD5]

So, ER/stroke doctors are collecting data from family members about pre-stroke functional status without using a structured instrument that asks questions that line up exactly with the definitions of the mRS categories. Then somehow, the responses to these questions get put into mRS categories and the data are called “pre-stroke mRS.”

Pretty easy to put a person into a pre-stroke “mRS” category of 3, 4, or 5 reliably based solely on information about whether the patient was able to walk without assistance (mRS 3); was unable to walk without assistance (mRS 4); or was bedridden (mRS 5).

|3| Moderate disability; requiring some help, but able to walk without assistance
|4| Moderately severe disability; unable to walk and attend to bodily needs without assistance
|5| Severe disability; bedridden, incontinent and requiring constant nursing care and attention

Sort of mysterious how responses using unstructured data collection get put into what is called “pre-stroke mRS” categories 0, 1, or 2 reliably.

But you already know that the data on “pre-stroke mRS” for patients placed in mRS categories 0, 1, or 2 are “not ideal” (putting it sort of mildly).

Explaining your approach, you state that:

“it seemed a reasonable approach and I could not think of any potentially better one.”

I agree reasonable. I cannot think of any potentially better one. Recognizing the data problem, perhaps others reading this can come with a better statistical “fix.”

Collecting data in acute settings using busy doctors whose job is not data collection for patients who are suspected of having a stroke in whom treatment decisions must be prompt is obviously a huge challenge.

I hope this “discussion” has been helpful.

[martinspn]

@EpiMD5

Sort of mysterious how responses using unstructured data collection get put into what is called “pre-stroke mRS” categories 0, 1, or 2 reliably.

But you already know that the data on “pre-stroke mRS” for patients placed in mRS categories 0, 1, or 2 are “not ideal” (putting it sort of mildly).

I agree. Wonderful insight - this practice is so widespread that I had never thought twice about its perils. Despite this relevant measurement error issue, pre-stroke mRS has been consistently linked to worse 90d mRS, which may be explained by it actually having some utility or bias in the assessment of the outcome due to researchers/staff often having access to the admission records where pre-stroke mRS is described. I think both play a role.

@EpiMD5

Explaining your approach, you state that:

“it seemed a reasonable approach and I could not think of any potentially better one.”

I agree reasonable. I cannot think of any potentially better one. Recognizing the data problem, perhaps others reading this can come with a better statistical “fix.”

Collecting data in acute settings using busy doctors whose job is not data collection for patients who are suspected of having a stroke in whom treatment decisions must be prompt is obviously a huge challenge.

I hope this “discussion” has been helpful.

There are several ongoing trials for distal medium vessel occlusions (DMVOs) which are expected to have a lower impact on the mRS - many DMVOs can’t even cause dependence - and usually have lower NIHSS, although most of the studies select based on the NIHSS. I think it is a matter of time until we start using something different from the mRS to evaluate the outcome for these patients. Not sure for pre-stroke status, though.

It has been absolutely helpful and I feel privileged that you invested time discussing this with me.