Thank you very much for your response. Is there a reason you suggest to treat trial as a random effect instead of imposing a compound symmetry correlation matrix for trial via Gls? I was under the impression from the course that gls is a more preferred approach? Also, is there a function in rms package which allows for random effects?
GLS (generalized least squares) is a good approach for continuous longitudinal outcomes. It provides nice ways to handle within-subject serial correlation as well as compound symmetry (which typically fits less well than say an AR(1) serial correlation pattern). The correlations we’re talking about are continuous in time, which is much different from how one models a categorical variable such as clinical site.
Hi Dr. Harrell. Sorry for the very basic question. I was wondering how to run model validity and calibration after using multiple imputation (using Hmisc and rms).
A good question with no easy answer. There are some notes from Ewout Steyerberg and possibly some references on the website of papers I pointed class participants to.
I found this link on stack exchange. I also found the following paper which seems consistent with the response in stack exchange: Wahl, Simone, et al. “Assessment of predictive performance in incomplete data by combining internal validation and multiple imputation.” BMC medical research methodology 16.1 (2016): 144.
However, intuitively, I’d think one should be able to perform multiple imputation on the full dataset prior to validation and not worry about information leakage, just as we aren’t concerned about using Y to impute X in training set. Am I missing something (don’t mind the pun)? I would love to hear your insights.
Thanks.
Thank you for the referral. I have ordered @Stephen 's book and look forward to reading up more about this topic.
I am wondering, in a different scenario, where lets say there were only 2 doses (instead of
daily dosing) in the above schedule (Days 1 and 20 with samples collects at several intervals in between like above) is there a necessity to treat Day as a separate categorical variable with hours-post-recent-dose nested within Day, or can I just have hours-post-initial-dose (let’s call it “time”) as one time variable and just allow for additional knots in my “time” variable to allow for the extra fluctuation after the subsequent dosing at 480 hours post original dose. I guess what I am not clear about is, are there reasons to prefer treating time as continuous spline with many knots instead of as a categorical variable to allow for the flexibility?
Thanks.
At the end of the course we had a brief discussion about different ways of handling time-dependent covariates. One involved extensions to the Cox model, and I think there are plenty of resources available on that. Dr. Harrell also mentioned an alternative idea which I believe he called a moving landmark design (or something similar). I believe he made reference to a study of his that used this kind of approach. In this case, I am thinking about a prediction model (no inference) that would have a moving window where predictions from a logistic or ordinal regression are made forward in time using covariates from a look-back period that moves along with the window. If Dr. Harrell or anyone else could point me to any papers using this kind of approach I would appreciate it. I do have one example of the kind of design I am referring to here (https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2728625).
Time is a simple fixed effect (the only complexity with time is allowing for a within-subject correlation pattern that is a function of time differences) so model it flexibly as a spline with more knots than usual if there are many distinct time points. Use the actual measurement times, not intended measurement windows.
See https://pediatrics.aappublications.org/content/116/5/1070 and
https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9469.2006.00529.x
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5711994/
Thank you for the references Dr. Harrell.
1 Like
Hi Dr. Harrell. Can varclus (Hmisc) accommodate a mixture of categorical and continuous data (with bothpos)? If not, would you use ClustOfVar instead?
varclus doesn’t officially support that but we tend to forge ahead anyway. ClustOfVar is made for this but check if it is still being supported.
In the first lecture, about a half hour in, Frank mentions how specifying a simple model and then assessing the deficiencies of the model via diagnostic plots increases uncertainty in the modelling processes. Interestingly, this process sounds very similar to Gelman’s Continuous Model Expansion, where a model is specified and slowly improved upon through posterior predictive checks or similar.
Does the Bayesian perspective get around these concerns, or might we still be worried about this as Bayesians?
2 Likes
Great question. The Bayesian approach only gets around this is you set priors before looking at the data, the priors cover all aspects of what you are about to model, and you don’t change the priors after looking at the data. In general, Bayesian modeling has a similar problem as frequentism: if you try a lot of models and the priors are not well pre-specified the posterior distributions will be too wide. If you add a parameter to the model for what you don’t know (e.g., normality or equal variances) the posterior gets a little wider.
1 Like
I’d appreciate some clarification on the chunk test. Assume fit a model like
Y = \beta_0 + \beta_1 X_1 + \beta_2 X'_1 + \beta_3 X_1''
If I tested \beta_1 = \beta_2 =\beta_3=0, that would test if X_1 has an effect on Y. If I tested \beta_2 =\beta_3=0, that would test if the effect is non-linear (since if these coefficients were 0, then the effect would be linear).
Suppose I tested \beta_2 =\beta_3=0 and failed to reject the null of that test. I would conclude that the effect of X_1 is linear. Would I report in my paper the estimate of \beta_1 as the effect of X_1, or would I still be required to report the entire spline? Some clarity on reporting protocol when we can not conclude non-linearity is present would be appreciated.
Good questions. Yes you have the hypotheses correct. But the null hypothesis of linearity, because of the Grambsch-O’Brien issue, is used only for curiousity and not for changing the model. Otherwise final p-values and confidence intervals are invalid. The course strategy in a nutshell is to allow for linearity for variables that are not known to operate linearly and for which the sample size allows you to estimate all parameters (in your case, 3).
Not also that “conclude that the effect of X_1 is linear” is an “absence of evidence is not evidence of absence” problem. Failure to reject H_0 does not mean that one can conclude that H_0 is true; it only means that you need more data.
So, I assume that applying the validate function to the model built using fit.mult.impute, as shown here https://thomaselove.github.io/432-notes/linear-regression-and-the-smartcle1-data.html (bottom) is incorrect, right? (Steyerberg performs a bootstrap procedure within each data set).
As I understand it, Thomas did a simple imputation in order to be able to do validation via resampling. We often take this short cut and still get a good approximation to the full validation. Or did I miss what you are referring to?
Thank you, that was the question. I thought the approach was philosophically incorrect.
Hi there, not sure if this is the right place but I was directed here for questions related to the rms package. I wondered how variables are retained in the validate()/ fastbw() functions? I know we can specify the stopping rule and significance level etc, but if different variables are selected in each bootstrap sample, is there a threshold for the amount of times this occurs in order for a factor to be retained in the final model? If yes, what is this?
Thanks!
It is not a good idea to set such a threshold. The proportion of times that a variable is selected in bootstrap samples tells you nothing that the original P-value doesn’t tell you. You need to let the bootstrap “fly” just as if you acquired a new dataset you’ll see great disagreements in which variables are selected when compared to your original sample. The bootstrap gives an honest representation of feature stability and points out the futility of labeling the final model as the final model as opposed of more correctly labeling it as an example model.