I am an PGY3 internal medicine resident who has a background in biostatistics (MA) and epidemiology (PhD).
I have decided to set up a Journal Club in the ~ 50 resident IM program in a midsize University hospital (~700 beds). The goal of the Journal Club is to train residents in critical appraisal of the literature and its application to clinical medicine. I eventually want to formalise the Journal Club and have it funded and incorporated into the residency program. For this to happen, I would need to demonstrate efficacy. To be honest, if that were the only goal I could likely use a (likely) worthless outcome measures like the number of journal articles read, or the participants self-scored competency, and poor pre-post methodology to p-hack the outcome I want to get the funding. But I figure if Iām going to spend time on this anyway I should evaluate the Journal Club in a pre-specified and statistically sound fashion, with the goal of making the dataset available and publishing the results.
My concern is how to structure the outcome measure so that it is i) appropriate, ii) valid and iii) generalisable. There seem to be three instruments in general usage; the ACE tool (1), the Fresno test (2) and the Berlin questionnaire (3). However, these instruments seem to emphasise undesirable concepts like the absolute interpretation of P-values, sensitivity, specificity and number needed to treat. I was wondering if anyone here knew of more appropriate instruments for the purpose of demonstrating understanding.