Statistics Reform

#21

Let’s say that if the work the biostatistician is doing will impact medical decision-making, then going on rounds to understand medical decision-making as it actually happens, at least once in a while, is important in my opinion.

4 Likes

#22

This is very nicely said. What was your motivation, and what did it take to set up the experiences?

0 Likes

#23

This is a great thread with great suggestions. One REALLY important one that is buried in your Medium piece: the statistics community and academic stats departments need to place more value on strong applied statistics as a genuine career path. In my experience I hear a lot about statisticians not being “real researchers” if they don’t get their OWN grants. But their own grants are usually on methodology and take away time from collaboration (even preventing collaboration).

We (stats community) need to recognize that a statistician who may never write a first-author paper in a stats journal, but who raises the level of statistical work in an entire department (like the embedded groups referred to above) is making a valuable contribution not just to science but the field of statistics. And, this person should be developed and promoted as a statistician.

I know there are good examples of this role/person out there, but I don’t think it’s the norm, and it should be.

13 Likes

#24

In seconding Robert’s excellent thoughts, I want to add that in the health field the number of active large statistical methods grants is in the 200 range from US NIH, whereas the number of large NIH biomedical topic grants is in the many thousand range. So the majority of funding for biostatistics comes from topical work. I’ve built a career on doing biostatistical methods research to meet the goals of biomedical research questions, with the methods development nested inside topical grants. Methods development goes hand in hand with collaborative work, and makes both better. Though you can’t get the sustained efforts funded in a single topical grant as you would in a single much harder-to-get R01 methods grant, by connecting methods development across a variety of topical grants you can certainly get a lot done.

6 Likes

#25

This is a great informative thread. I’m a PhD researcher in the Psychology field, looking into the whole narrative of statistics in the Psychology research world with UK samples, and it’s very enlightening seeing the comments of others. I’m not convinced that outsourcing is the answer within my particular field as statistics is so enmeshed with research design - something that many researchers aren’t really aware of - so outsourcing would potentially encourage less researchers to learn how to justify and explore their research decisions, as well as make it easier to misunderstand outcomes.

0 Likes

#26

I’ve always seen your research program as a model, Frank! Your methods work has a very strong connection to relevant topical projects, and is therefore relevant and useful for actual statistical practice! I also really like the way you and colleagues developed things at Vanderbilt with embedded statisticians. The optimal setting is synergy between applied work and methods development with credit given for both.

I strongly agree about funding. Getting methods work funded through substantive/topical grants is very effective, and leads to more relevant methods problems. It means that we’re answering methods questions that move substantive research forward as well. This is particularly relevant for large network grants. When we developed CNODES (www.cnodes.ca), Samy Suissa insisted that we retain ~5-10% of the project funding for methods work. This has been a great source of methods funding, and there is a nice synergy between our methods work for CNODES and the drug safety work we do for Health Canada. It’s also a fantastic way to train applied statistics students – they see early in their career what it’s like to participate in substantive projects and develop methods problems organically.

3 Likes

#27

i did a phd in medicine instead of a phd in statistics for that reason; my supervisor was a cardiologist, not a statistician. I’m not sure young statisticians are aware of this option, i’m not sure how long the option has existed(?), but there seem to be only a a few uni’s around the world that offer a phd in medicine for biostatisticians. You have to be confident in biostats, obviously, but you are completely free to do applies stats and develop methodology simultaneously. And the outcome is papers in medical journals and a wider readshership and the potential to influence the practice of stats, rather than something buried in stats in medicine. I think stats in med and smmr are coveted, but why? it’s just for prestige, no clinician will ever read it and they are the decision makers, they are the ones, mostly, who run the AROs

3 Likes

#28

I have no idea! :slight_smile: I have to say that I am generally wary of the push to publish undergraduate research. It’s just impossible to have any kind of quality control in that scenario. By all means, teach research and provide opportunities for experience, but we need to stop trying to pretending that most of it isn’t clumsy (as it should be!).

1 Like

#29

This is my understanding. At the end of the day, any call for better research will inevitably lead to less research, which is an unacceptable outcome for any group relying on politicians to send money their way.

1 Like

#30

Yes, I have found myself in a really lucky position where I have some job security without having to chase my own grants (we still do, but I don’t live or die on it) so I am free to collaborate on other people’s stuff, and conducting training for things like open science, FAIR data, R workflows, etc. The problem is down the road though - will I eventually be in a position for promotion to full professor, based on how it’s done here? Right now, I’d call it a long shot. And I have no interest in developing novel methods (probably due to lack of aptitude!) so I can’t hang my hat on that either. So in most cases, asking someone to be a purely applied, collaborative statistician is to ask them to committ career suicide - plus people don’t want to fund ~permanent positions but would rather treat it like they do Postdoctoral Researchers, and nobody is going to choose that life over industry/govt/ngo.

2 Likes

#31

Editorial suggestion: highlight text you are replying to and see the Quote button appear. This will paste highlighted text with appropriate markup into the active window, or create a new active “reply” window.

1 Like

#32

The reason why I think this happens, as I’ve pointed out above, is that there is no pressure to employ statisticians as permanent or long-term members of a research lab/team. Compare this with Bioinformatics: research groups that rely heavily on 'omics data can’t do without dedicated bioinformaticians, because they would be unable to process and analyse all that data in a reasonable timeframe without those with the necessary skillset. So they have to put money into it. They have no option. But in a publishing environment in which most editors and reviewers alike know little of statistics, all you need is p < 0.05. No pressure to spend money on statisticians when study design and data analysis isn’t seriously evaluated when submitting a paper to a journal. The situation for applied statisticians working specifically on data analysis as opposed to method development will only change when publishing culture changes, if ever.

2 Likes

#33

i’m in two minds about whether editors should demand the programming code from authors. It would reveal that the authors are using point-and-click from drop down menus in spss to do analyses and don’t really understand the code behind it, and it would then put pressure on researchers to employ statisticians to minimise embarrassment and ensure data integrity

3 Likes

#34

My observation based on reading so many statistics related articles is that there is a need for a logic handbook designed specifically for statistics and even mathematics. I don’t think their logic is covered sufficiently in training.

1 Like

#35

Not sure if I agree. We live in a rational world where everything needs to be quantifiable, so institutions/funders/government can evaluate whether the investment payed off. We won’t be able to change that, but we may be able to change how things are quantified. Why aren’t we weighting publications with a quality score?

A research quality score could give higher weights to publications with published study protocols. Your research could get a higher weight if you can demonstrate an independent group succesfully replicated your findings. We may even give a higher score to validation or replication studies, to compensate for the fact that these are hard to publish.

I think such a score would have to have a broad outlook, applicabile in very different scientific fields (so not just focus on stats methods). For things to improve, doing the right thing must be made more attractive…

2 Likes

#36

Who should conduct the score-keeping? Thanks.

1 Like

#37

There is too much research being done. As Doug Altman famously said we need to concentrate funding on the better research. https://www.bmj.com/content/308/6924/283

5 Likes

#38

In a recent lecture at NIH, John Ioannidis mentioned that there were an estimated 180 million research articles out there: from about 2009 onward.

1 Like

#39

Yes, to quote him “Abandoning using the number of publications as a measure of ability would be a start.” Hence my thinking: if one high quality paper (quality in methods, regardless of stat significance) counts as much as five bad ones, numbers of publications will drop, and funders will have an incentive to stop favoring hyped topics over good methods. What is wrong with my reasoning?

3 Likes

#40

If the goal is to improve statistical training in non-statisticians, my suggestion would be to create a framework or approach that can teach important concepts and applications with less formal mathematics. Can we educate someone whose knowledge base is equivalent to a high school (or secondary school) graduate?

Here is an interesting example where elementary school children in Uganda were taught to assess reliability of health claims:
https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(17)31226-6/fulltext

4 Likes