Precision as the goal for sample size estimation

sample-size
precision
#1

I have seen arguments for estimating sample size based on precision as a goal pop up in a few different places. In BBR, @f2harrell provides the following summary of why this may be of interest:

There are many reasons for preferring to run estimation studies instead of hypothesis testing studies. A null hypothesis may be irrelevant, and when there is adequate precision one can learn from a study regardless of the magnitude of a P-value.

Some other places I have seen this include

  1. John Krushke’s blog post on optional stopping arguing stopping based on precision targets results in unbiased point estimates.

    • One issue I have with this post is my understanding that if the prior correctly matches the data-generating process then the point estimate should be unbiased.

  2. Planning future studies based on the precision of a meta-analysis

And most recently:

  1. This paper on planning epidemiological study study size based on precision rather than power.

I am attracted to this idea in my own work where I am primarily interested in estimation (and gradually trying to link this to loss functions or probabilities to get people away from p-values), but outside of methods papers I literally never see this done. it seems particularly powerful when combined with Bayesian analysis with sensible priors. Some of the bigger challenges that come to mind for me are:

  1. Shift in thinking required to think in terms of precision… With a loss function/economic model it’s easier to explain in terms of value of information, but not sure how to settle on “How precise is precise enough.” Is a power curve type visualization the best bet here?
  2. Typically the examples given result in larger sample sizes (though maybe the curve comes into play here again). Without a loss function, how do you convince people the larger sample is worth while?

Does anyone have any experience actually implementing this approach? Did you find it useful? Do people think it actually provides any benefit?

#2

We have used precision-based sample size calculations in multiple grant proposals to NIH with good success. It is a bit of an art to set the level of acceptable margin of error. I would like to find some principles on which to base that.

#3

with regard to implementation, i recently had to do a power calculation based on the CI width of a correlation coefficient and noticed that although there is an R function that can do it (https://people.ucsc.edu/~dgbonett/psyc181.html - function 6 in the word doc) it was not possible in sas proc power (surprisingly). The paper @f2harrell mentioned in another thread is worth reading too: Sample size, more than calculations

#4

I use precision of correlation coefficients all the time in my work. The R Hmisc package plotCorrPrecision function makes sample size calculations easy.

#5

i’ve tried to keep this tidbit in my memory: https://twitter.com/f2harrell/status/936230071219707913?lang=en :grinning:

#6
  1. All my biomarker pilot studies (including NIH applications) are designed this way.
  2. Our current project LUMIFY-PD that examines the utility of a point of care device in fluid management for dialysis patients

I was introduced into the idea during my 4 year stint in Pharma (Abbott Labs and then Abbvie). We were very interested in prospective cohorts designs for post marketing single arm studies and this is effectively the best way to design those , draft budgets etc

By the way there is a paper from the 50s by 2 Naval officers that can be used as a citation in a grant

#7

Could you share an example of language used in one of your grants? Also, what was the principal rationale for choosing the confidence interval that you did?

#8

Here is a piece.

Sample size calculations are often based on assuming a biologically relevant effect to detect with a given power. Rejecting, or even more so not rejecting, a null hypothesis may not provide sufficient information to elucidate a biological problem. It is often the case that the true research goal is the estimation of a population parameter such as a mean or the difference in two means (i.e., a treatment effect). In line with the goal of estimation, the sample size can be justified in terms of the expected precision of the estimate, as discussed by Borenstein and Parker and Berman . They argue that the precision (margin of error) of estimates is often more relevant than the power to detect any non-zero effect. One advantage of precision-guided planning is the increased likelihood that even a negative study will be informative (i.e., that confidence intervals including no effect can exclude large effects).

Sample size formulas based on precision are in BBR Sections 5.5.4 5.6.3 5.8.4 5.10 6.7 8.5.2.

#9

I’ve often used this as a complement, but personally I think situations where it should be the only consideration are rare. It’s the ability to distinguish some clinically interesting scenario, such as benefit vs harm, or possible inferiority by some delta, that we most often want to design for. Then we must consider both CI width and point estimate. It’s not that often that just how wide the CI is is more important than where its end points are.

#10

I’ve used precision exclusive of power successfully on a number of occasions. Power is too arbitrary to me, because the effect size to detect is too arbitrary, and because I want to be able to have an interpretation should the p-value be large.

#11

This is a topic in which my interest goes back 30 years:

https://www.sciencedirect.com/science/article/pii/S0140673689913561?via%3Dihub
https://www.jstor.org/stable/2684165?seq=1#page_scan_tab_contents

and in particular

https://www.jstor.org/stable/2532411?seq=1#page_scan_tab_contents

which uses prior information on sigma to calculate an assurance type expectation.

#12

Late in the game, but to my knowledge the first one who have used precision to determine sample size appears to have been a paper by the US Navy
Journal of the American Statistical Association, Vol. 45, No. 250 (Jun., 1950), pp. 257-260

#13

I had no idea this paper existed. Very nice. I like to emphasize the n needed to estimate the SD because in normal linear models the SD is used all over the place. This was part of the recent modeling sample size papers by Richard Riley et al. I have a derivation of the multiplicative margin of error in BBR Section 5.10.1. WIth 0.95 confidence n=70 achieves a multiplicative margin of error of 1.2 in estimating \sigma.