I have received a fairly large grant to create electronic infrastructure at my institution to facilitate clinical trial work. We have an in-house electronic health record and I want to build in modules that support clinical trial work (automatic identification of eligible patients, screening inclusion exclusion criteria, consenting, ordering necessary measurements and giving correct medications, safe data storage etc.) within the EHR as much as possible. One very tempting route we could take is to use REDCap for most of the above communicating through the REDCap API as our EHR uses FHIR (Fast Healthcare Interoperability Resources) Specification which REDCap supports. We already have a local REDCap instance.
I wanted to ask whether datamethodsâ readers have used REDCap for clinical trials they have participated in and what their thoughts are regarding quality and usability? Does anyone have experience with something similar to what I am suggesting above?
We at Vanderbilt University Medical Center (where REDCap is developed) have used REDCap extensively in clinical trials, and many users have used the special tie-ins with EPIC that the REDCap team has developed. New advanced randomization capabilities (including for adaptive trials) have recently be added by the community and central team.
The main disadvantage of REDCap is that itâs not a real normalized database system but is rather a case report form capture system. For many applications this doesnât matter but there have been two challenges.
Itâs a little awkward handling longitudinal data
Whether using the REDCap API package in R (which my department supports) to export data or using a web site to more manually export data, the system is not really exporting a hierarchical structure of case report forms. Itâs really exporting one huge wide form. To get the database back to what looks like the forms that users entered takes a LOT of work. Without this extra work, youâll have happen what happened to me on our COVID-19 platform ACTIV-6: If you have 10,000 patients and a total of 200,000 longitudinal measurements, one form was entered for 2 patients but I got 200,000 records for it in the exported file using the API. The system has a hard time recognizing what is a real record and one that was never touched by human hands.
Thank you for your reply and information. In the clinical trials you have participated in and lead, how has the REDCap implementation been managed? I imagine Vanderbilt University Medical Center is an outlier in its ability to provide IT support for setting up projects and necessary intergrations due to it being the center of REDCap development. Do researchers have access to REDCap IT support through the overhead generated by grants? Is this service provided by the clinical trials unit? Are there any in-house documentation that you can provide links to that I could peruse?
As someone working at an institution (and country) with historically and contemporarily little clinical trial participation, it is striking how little of clinical trial logistics and infrastructure is documented in an easily accessible manner. I am terrified of building something that does not adhere to best-practices, but I am unable to even confirm that âbest-practicesâ exist.
Briefly, we have a Data Coordinating Center which manages such trials (this one was actually joint between two institutions) and the grants supporting our large COVID-19 platform trial was of a type that had all the project support as line items in the budget. The cost of database development, administrative reports, QC, and creating of analysis files is indeed enormous, far higher than you might imagine, especially for administrative reports and QC. REDCap provides a lot of facilities for this, but to someone who is not directly engaged in this the process seems quite involved. It would be interesting to compare the total cost with the cost of a purely manual system. I think it would be higher but of course the resulting data quality is better.
Some examples from my work in oncology. What Iâve found is EHR data is not suitable for trials without a lot of processing. E.g., in our system (Switzerland) ICD diagnoses are unreliable, so we need to somehow be able to automatically structure free text oncology diagnoses form the EHR or make the doctors directly capture the diagnoses in a structured format. The latter works so so (lots of missig data), so weâre exploring classification methods to turn free text into structured data.
Patients are prescribed various treatments, various lines of therapy, etc. For eligibility you will need to identify which of these therapies are for which diagnosis, which line etc. Data from pathology will need to linked to diagnoses (in our case thatâs pretty difficult), lab values might be analysed on different machines (i.e., different machines for the same test) which need to merged into column, etc.
My point is, before thinking about REDCap, imho, you need a pipeline that creates data that is suitable for trials. A pipeline that accurately prepares data in a way that is suitable for the trials you care about. I think even for one specialty this is already a gigantic task.
I would then try to store the âcleanâ data in a relational data base.
Iâm not really experienced with REDCap modules. Whatâs worked quite well for me is to have an R package that allows for easy communication of the data warehouse (relational database) and REDCap. I can then setup simple R programs, that pull the data we need from the data warehouse into trial specific REDCap databases on a scheduled basis. E.g., for screening lists, lab-values, pathology reports, imaging reports, etc.
Edit: I might be involved in the process of building all of the above at hour healthsystem (likely not before 2027), so Iâd be interested in the approaches you want to / will take. Iâve been currently thinking about using NextFlow for the pipeline.
Though never done, it is important to use probabilistic methods to capture the uncertainty in such approaches and carry it forward into all analyses. If an estimation rule results in a 0.85 chance that a certain diagnosis is present, we need to quit classifying that patient as having the diagnosis and instead carry the 0.85 forward. Bayesian modeling or multiple imputation can come to the rescue. You might find that your N=1000 patient study really has an effective sample size of N=700 which is nothing to be ashamed of. But as of now there is a lot of false confidence in using forced-choice classification in decision rules based on EHR data.
This implies that for every estimation rule you need an unbiased audit that provides sliver-standard determinations to model against, and the sample size needed for such audits needs to be N > 300.
Agreed. Weâre currently submitting a study, where we tried to create a gold-standard (but yeah itâs more of a silver standard, duplicate classification by humans + expert adjudication) for classifying the radiology reports into response, remission, etc, among other.
LLMs compare quite favourably to this standard, but are not ready yet imo. The issue is that thereâs so much to classify that itâs probably too much work to create training data for every task, e.g., if we wanted to use something more lightweight, like fine-tuning BERT. I had some issue exporting the logits from ollama when using LLMs (I think there currently is a bug), so we would not be able to carry forward the uncertainty, which is a problem (unless weâre >99.x% accurate Iâd say).
In any case, Iâve not yet thought about how Iâd approach an analysis, where predictors have uncertainty.
Multiple imputation is the easiest approach, e.g. if you knew that P(X=1) = 0.7 you could create 30 imputations per patient with 0.7 of them = 1 and 0.3 = 0. Also take a look at SIMEX. You can think of this as a measurement error problem.
Interesting to hear that others are (possibly) developing similar software support and I would be absolutely delighted to collaborate on any possible even tangentially related aspect in the future. I will switch any and all documentation from Icelandic to English right now â it didnât even occur to me that someone could possibly benefit from what I and my team will be doing.
ICD-10 codes are almost worthless in Iceland as well. We have a single payer system with 90% fixed budgets for our healthcare institutions. Only 10% of the budget is DRG based and the contracts are terrible â even with minimal coding we always hit our DRG cap. This means there is absolutely no institutional incentive to use ICD-10 codes directly. We have very few medical coders and physicians frankly, do not care. Myself included.
I agree 100% with your approach of collecting clean data locally within the EHR, which then gets pushed to REDCap to reap the benefits of all of REDCapâs features. Our EHR is pretty nifty, as it has an underlying engine that can read and (mostly) write anything that the EHR has access to, on an âif this then thatâ workflow basis. This engine even has access to, as an input, an in-house neural network that has been trained on and has access to all medical text in real-time. I can expand a bit on how I have pitched this being utilized here, or it may just be easier to translate my grant proposal to english and share it with datamethodâs readers.
I am not a software engineer, but we are hiring one full time for the grant money to implement this.
As of yet, I have no experience working in large scale clinical trials. I find this interesting and important but for some reason my first thought is to wonder how well adjudicated baseline covariates are in non-EHR clinical trials? I imagine that in many such trials, it would be based on interviews between research staff and participants. Presumably, accuracy is not perfect there either? And if not, what should be the gold standard?
I would be very interested in the grant proposal. We also have a very capable in-house GPU cluster, so we would be capable to relatively sophisticated models locally. If all goes well, I would be fully responsible for this project towards the end of 2026 at the earliest, so I would profit a lot from your experiences (+ would probably also need grant money for the implementation to hire someone).
Such a radical idea! At analysis-time, perhaps it would make sense to identify the uncertainties with greatest leverage over the results, and systematically re-adjudicate those.
To give you some idea what I am thinking, this is a translation of the non-technical introduction to the proposed infrastructure:
âThis document details the proposed electronic infrastructure for clinical trials at Landspitali University Hospital (LandspĂtali HĂĄskĂłlasjĂșkrahĂșs, LSH). The proposal consists of ten interconnecting modules, which together form usable infrastructure. Unless otherwise stated, all modules will be built within HeilsugĂĄtt, the inhouse electronic health record (EHR) of LSH. First, an overview is presented and then each module is described in detail. Finally, a timeline is presented detailing which modules will be developed and put into production. Within each module-subchapter, the potential of module for clinical use will be described.
Overview
The first module is a tool to perform feasibility checks for recruitment. A researcher will be able to specify a list of variables (including demographic, location i.e. department, vital sign measurements, laboratory results and diagnostic codes) that define the inclusion criteria of their potential study, and see the number of individuals who meet these criteria historically at LSH. This tool will not need be built into the EHR but will have access to the underlying EHR data. It provides only aggregate counts by calendar month. The second module is an editor that allows the researcher to specify the settings for the other modules and how they work together, such as screening inclusion criteria, which individuals should be notified, formal inclusion and exclusion criteria, etc. The third module screens patients who seek the services of LSH for specified screening inclusion criteria of active studies in real time. The module will keep track of the number of individuals who fulfill screening criteria, regardless of whether they were later actively evaluated for study participation, and will keep track of aggregate demographic and selected baseline covariate data to allow researchers to evaluate and report the proportion and representativeness of those later included in the study. The fourth module notifies the correct individual of the potential participant whom the previous module has identified. Who the correct individual is can be specified and will vary depending on the study design. For some embedded pragmatic clinical trials, this may be the nurse or physician responsible for the clinical care of the patient and for others, this may be a research nurse or study coordinator. The fifth module provides interface to formally document whether inclusion or exclusion criteria are met and, if applicable, helps the user order the necessary studies to determine this. This module also keeps track of the aggregate demographic and selected baseline covariate data of individuals who are formally evaluated for study participation, again so that researchers can report the proportion and representativeness of those later included in the study. If the patient meets all the criteria, the sixth module supports the process of informed consent with as many support tools as possible, including electronic signature and storage of consent forms. Should the patient agree to participate, the seventh module handles randomized allocation and assigns patients to the arms of the study and notifies the user to which arm the patient was assigned. Next, the eight module opens, which helps the user perform the necessary tasks for the arm of the study to which the patient was assigned, as well as ordering the appropriate studies or placing appropriate orders. Module nine runs behind the scenes and automatically collects pre-specified interim and outcome data. Finally, module ten facilitates data extraction for interim analysis or at study completion. â
In my experience the quality and completeness of data collected by trained research staff far surpasses that of the EHR. This is especially true for symptoms, functional status, and quality of life variables.
Merry Christmas! Why not implement this first module within your EHR, so that it becomes a general capability accessible to all clinicians? If individual clinicians could roll their own clinical epidemiology (âHmm.., I wonder how many times I might have missed diagnosis Y in clinic, in patients presenting with X..â) then maybe there would be a meaningful incentive to curate those pesky ICD codes. An ad hoc query language âjust for researchersâ is much less likely to be well designed than a more general query facility.
Excellent point and I agree. Module 1 is actually in production and accessible to all staff with access to the computer system of our hospital. It can be activated within the EHR but can also be activated as a stand alone program. This allows non-clinical staff to use it as well. I will say that itâs use had been disappointingly⊠Marginal. Probably because I havenât advertised it well enough.