Sample Size Calculation with GPower

• Goal

  • to introduce key ideas
  • to offer a perspective for reasoning
  • to offer first practical experience

• Target audience

  • primarily the research community at VUB / UZ Brussel

• Feedback

  • help us improve this document
    wilfried.cools@vub.be

Program

Sample size calculation: demarcation

• How many observations will be sufficient ?

  • avoid too many, because typically observations imply a cost
    • money / time → limited resources
    • risk / harm → ethical constraints
  • depends on the aim of the study
    • research aim → statistical inference
      
      

• Linked to statistical inference (using standard error)

  • testing → power [probability to detect effect]
  • estimation → accuracy [size of confidence interval]

Sample size calculation: a difficult design issue

• Before data collection, during design of study

  • requires understanding: what is a relevant outcome ?!
  • requires understanding: future data, analysis, inference (effect size, focus, ...)
  • decision based on (highly) incomplete information, based on (strong) assumptions
    

• Not always possible nor meaningful !

  • easier for confirmatory studies, much less for exploratory studies
  • easier for experiments (control), less for observational studies
  • not possible for predictive models, because no standard error
  • NO retrospective power analyses → OK for future study only
    Hoenig, J., & Heisey, D. (2001). The Abuse of Power:
    The Pervasive Fallacy of Power Calculations for Data Analysis. The American Statistician, 55, 19–24.
    

• Alternative justifications often more realistic:

  • common practice, feasibility, ... or a change of research aim (description, pilot, ...)
  • less strong, puts more weight on non-statistical justification (importance, low cost, ...)

Simple example confirmatory experiment

• Example: does this method work for reducing tumor size ?

  • evaluation of radiotherapy to reduce a tumor in mice

  • comparing treatment group with control (=conditions)

    • tumor induced, random assignment treatment or control (equal if no effect)
    • after 20 days, measurement of tumor size (=observations)
    • happy if 20% more reduction in treatment !! (=minimal clinically relevant difference)

  • intended analysis: unpaired t-test to compare averages for treatment and control

  • SAMPLE SIZE CALCULATION:

    • IF average tumor size for treatment at least 20% less than control (4 vs. 5 mm)
    • THEN how many observations sufficient to detect that difference (significance) ?

Reference example

• Reference example used throughout the workshop !!

Formula you could use

GPower: the building blocks in action

GPower: a useful tool

GPower statistical tests

GPower input

GPower output

GPower protocol

• Summary for future reference or communication
• File/Edit save or print file (copy-paste)

t tests - Means: Difference between two independent means (two groups)
Analysis: A priori: Compute required sample size

Non-centrality parameter ( $\delta$ ), shift Ha from Ho

Note: Ho and Ha, asymmetry in statistical testing

• Ha is NOT interchangeable with Ho

• Cut-off at Ho using $\alpha$

  • in statistics → observe test statistics (Ha unknown)
  • in sample size calculation → assume Ha

• If fail to reject then remain in doubt

  • absence of evidence $\neq$ evidence of absence
    • p-value → P(statistic|Ho) != P(Ho|statistic)
    • example: evidence for insignificant $\eta$ same as for $\eta$ * 2

• Equivalence testing → Ha for 'no effect'

  • reject Ho that smaller than 0 - | $\Delta$ | AND bigger than 0 + | $\Delta$ |
  • acts as two superiority tests with margin, combined

Alternative: divide by N

Type I/II error probability

Create plot

• create plot
  • X-Y plot for range of values
  • Y-axis / X-axis / curves and constant

Exercise on errors, interpret plot

• Understand the building blocks, interpret the plot

Decide Type I/II error probability

Control Type I error

• Defined on the Ho, known

  • assumes only sampling variability

• Multiple testing

  • typically used to explore effects in more detail
  • inflates type I error $\alpha$ (each peak possible error)
  • family of tests: $1-(1- \alpha)^k$ → correct, eg., Bonferroni ( $\alpha/k$)

• Interim analysis

  • interim analysis (analyze and conditionally proceed)
  • plan in advance
  • alpha spending, eg., O'Brien-Flemming bounds
  • NOT GPower
    • our own simulation tool (Susanne Blotwijk):
      http://apps.icds.be/simAlphaSpending/
  • determine boundaries with PASS, R (ldbounds), ...

For fun: P(effect exists | test says so)

• Using $\alpha$, $\beta$ and power or $1-\beta$
  • $P(infer=Ha|truth=Ha) = power$ → $P$(test says there is effect | effect exists)
  • $P(infer=Ha|truth=Ho) = \alpha$
  • $P(infer=Ho|truth=Ha) = \beta$
  • $P(\underline{truth}=Ha|\underline{infer}=Ha) = \frac{P(infer=Ha|truth=Ha) * P(truth=Ha)}{P(infer=Ha)}$ → Bayes Theorem
  • __ = $\frac{P(infer=Ha|truth=Ha) * P(truth=Ha)}{P(infer=Ha|truth=Ha) * P(truth=Ha) + P(infer=Ha|truth=Ho) * P(truth=Ho)}$
  • __ = $\frac{power * P(truth=Ha)}{power * P(truth=Ha) + \alpha * P(truth=Ho)}$ → depends on prior probabilities

• IF very low probability model is true (eg., .01) → $P(truth=Ha) = .01$
• THEN probability effect exists if test says so is low, in this case only .14 !!
  
• $P(truth=Ha|infer=Ha) = \frac{.8 * .01}{.8 * .01 + .05 * .99} = .14$

Effect sizes, in principle

• Estimate/guestimate of minimal magnitude of interest

• Typically standardized: signal to noise ratio (noise provides scale)

  • eg., effect size $d$ = .5 means .5 standard deviations
  • eg., difference on scale of pooled standard deviation

• Part of non-centrality (as is sample size) → pushing away Ha

• ~ practical relevance (not statistical significance)

  • NOT p-value ~ partly effect size, but also partly sample size

• 2 main families of effect sizes (test specific)

  • d-family (differences) and r-family (associations)
  • transform one into other, eg., d = .5 → r = .243
    $\hspace{20 mm}d = \frac{2r}{\sqrt{1-r^2}}$ $\hspace{20 mm}r = \frac{d}{\sqrt{d^2+4}}$ $\hspace{20 mm}d = ln(OR) * \frac{\sqrt{3}}{\pi}$

Effect sizes, in literature

Effect sizes, in GPower (Determine)

Exercise on effect sizes, ingredients Cohen's d

Exercise on effect sizes, plot

Exercise on effect size, imbalance

Effect sizes, how to determine them in theory

• Choice of effect size matters → justify choice !!

• Choice of effect size depends on aim of the study

  • realistic (eg., previously observed effect) → replicate
  • important (eg., minimally relevant effect)
  • NOT significant → meaningless, dependent on sample size

• Choice of effect size dependent on statistical test of interest

  • for independent t-test → means and standard deviations
  • possible alternative: variance explained, eg., 1 versus 16+1
    • with one-way ANOVA ( $f$ = .25 instead of d = .5)
    • with linear regression ( $f^2$ = .0625 instead of d = .5)
    • https://www.psychometrica.de/effect_size.html#transform

Effect sizes, how to determine them in practice

• Experts / patients → use if possible → importance
  minimally clinically relevant effect

• Literature (earlier study / systematic review) → beware of publication bias → realistic

• Pilot → guestimate dispersion estimate (not effect size → small sample)

• Internal pilot → conditional power (sequential)

• Guestimate uncertainty...

  • sd from assumed range, assume normal and divide by 6
  • sd for proportions at conservative .5
  • sd from control, assume treatment the same
  • ...

• Turn to Cohen → use if everything else fails (rules of thumb)

  • eg., .2 - .5 - .8 for Cohen's d

Relation sample & effect size, type I & II errors

Exercise on type of power analysis

• For the reference example:
  • retrieve power given n, $\alpha$ and $\Delta$
  • then, for power .8, take half the sample size, how does $\Delta$ change ?
  • then, set $\beta$/ $\alpha$ ratio to 4, what is $\alpha$ & $\beta$ ? what is the critical value ?
  • then, keep $\beta$/ $\alpha$ ratio to 4 for effect size .7, what is $\alpha$ & $\beta$ ? critical value ?

Solution for type of power analysis

• For the reference example:
  • retrieve power given n, $\alpha$ and $\Delta$ of reference case
    • use post-hoc 64x2 → .8
  • then, for power .8, take half the sample size, how does $\Delta$ change ?
    • use sensitivity 32x2 (d=.7114)
    • $\Delta$ from .5 to .7115 = .2115
    • bigger effect $\Delta$ compensates loss of sample size n
  • then, set $\beta$ / $\alpha$ ratio to 4, what is $\alpha$ & $\beta$ ? what is the critical value ?
    • use compromise 32x2
    • $\alpha$ =.09 and $\beta$ =.38, critical value 1.6994
  • then, keep $\beta$ / $\alpha$ ratio to 4 for effect size .7
    • use compromise 32x2
    • $\alpha$ =.05 and $\beta$ =.2, critical value 1.9990

Getting your hands dirty

.n <- 64
.df <- 2*.n-2
.ncp <- 2 / (4 * sqrt(2)) * sqrt(.n)
.power <- 1 -
    pt(
        qt(.975,df=.df),
        df=.df,    ncp=.ncp
    ) - 
    pt( qt(.025,df=.df), df=.df, ncp=.ncp)
round(.power,4)

## [1] 0.8015

GPower, beyond the independent t-test

• So far, comparing two independent means

• From now on, selected topics beyond independent t-test
  with small exercises

  • dependent instead of independent
  • non-parametric instead of assuming normality
  • relations instead of groups (regression)
  • correlations
  • proportions, dependent and independent
  • more than 2 groups (compare jointly, pairwise, focused)
  • more than 1 predictor
  • repeated measures

• Look into GPower manual
  27 tests → effect size, non-centrality parameter and example !!

Dependence between groups

• If 2 dependent groups (eg., before/after treatment) → account for correlations

• Correlation typically obtained from pilot data, earlier research

• GPower: matched pairs (t-test / means, difference 2 dependent means)

  • use reference example,
    assume correlation .5 to compare with reference effect size, ncp, n !?
  • how many observations if no correlation exists (think then try) ? effect size ?
  • what changes with correlation .875 (think: more or less n, higher or lower effect size) ?
  • what would the power be with the reference sample size, n=128, but now cor=.5 ?

Solution for dependence between groups

• GPower: matched pairs (t-test / means, difference 2 dependent means)
• Assume correlation .5 to compare with reference effect size, ncp, n
  • $\Delta$ looks same, n much smaller = 34 (note: 34 x 2)
  • different type of effect size: dz ~ d / $\sqrt{2*(1-\rho)}$
• How many observations if no correlation exists (think then try) ? effect size ?
  • 65, approx. same as INdependent means → 64 (*2=128) but also estimate the correlation
  • $\Delta$ = dz = .3535 (~ d = .5)
• What changes with correlation .875 (think: more or less n, higher or lower effect size) ?
  • effect size * 2 → sample size from 34 to 10 (almost / 4)
• What would the power be with the reference sample size, correlation .5 ? what is the ncp ?
  • post - hoc power, 64 * 2 measurements, with .5 correlation
  • power > .976, ncp > 4,

Non-parametric distribution

• Expect non-normally distributed residuals, not possible to avoid (eg., transformations)

• Only considers ranks or uses permutations → price is efficiency and flexibility

• Requires parent distribution (alternative hypothesis), 'min ARE' should be default

• GPower: two groups → Wilcoxon-Mann-Whitney (t-test / means, diff. 2 indep. means)

  • use reference example
    with normal parent distribution, how much efficiency is lost ?
  • for a parent distribution 'min ARE', how much efficiency is lost ?

Solution for non-parametric distribution

• GPower: two groups → Wilcoxon-Mann-Whitney (t-test / means, diff. 2 indep. means)
• Use reference example, with normal parent distribution, how much efficiency is lost ?
  • requires a few more observations (3 more per group)
  • less than 5 % loss (~134/128)
• For a parent distribution 'min ARE', how much efficiency is lost ?
  • requires several more observations
  • more than 15 % loss (~148/128)
  • min ARE is safest choice without extra information, least efficient

A relations perspective, regression analysis

• Differences between groups → relation observations & grouping (categorization)

• Example → d = .5 → r = .243 (note: slope $\beta = {r*\sigma_y} / {\sigma_x}$)

  • .243*sqrt( $4^2+1$ )/sqrt( $.25$ ) = 2
  • note: total variance = residual variance + model variance (2 or 0 for all observations)
    var((2-1),(0-1),(2-1),(0-1),...)
  • note: design variance = variance -.5 and .5 for all observations
    var((1-.5),(0-.5),(1-.5),(0-.5),...)
• GPower: regression coefficient (t-test / regression, one group size of slope)
  • determine slope $\beta$ and $\sigma_y$ for reference values, d=.5 (hint:d~r), SD = 4 and $\sigma_x$ = .5 (1/0)
  • calculate sample size
  • what happens with slope and sample size if predictor values are taken as 1/-1 ?
  • determine $\sigma_y$ for slope 6, $\sigma_x$ = .5, and SD = 4, would it increase the sample size ?

Solution on a relations perspective

• GPower: regression coefficient (t-test / regression, one group size of slope)
• Determine slope $\beta$ and $\sigma_y$ for reference values, d=.5, SD = 4 and $\sigma_x$ = .5 (1/0)
  • $\sigma_x$ = $\sqrt{.25}$ = .5 (binary, 2 groups: 0 and 1) → slope = 2, $\sigma_y$ = 4.12 = $\sqrt{4^2+1^2}$
• Calculate sample size
  • 128, same as for reference example, now with effect size slope H1 given 1/0 predictor values
• What happens with slope and sample size if predictor values are taken as 1/-1 ?
  • $\beta$ is 1, a difference of 2 over 2 units instead of 1
  • no difference in sample size, compensated by variance of design
• Determine $\sigma_y$ for slope 6, $\sigma_x$ = .5, and SD = 4, would it increase the sample size ?
  • $\sigma_y$ = 5 = $\sqrt{4^2+3^2}$ (assuming balanced data)
  • bigger effect → smaller sample size, only 17

A variance ratio perspective, ANOVA

• Difference between groups or relation → ratio between and within group variance

• GPower: regression coefficient (t-test / regression, fixed model single regression coef)

  • use reference example, regression style (sd of effect and error, but squared)
  • calculate sample size, compare effect sizes ?
  • what if also other predictors in the model ?
  • what if 3 predictors extra reduce residual variance to 50% ?

• Note:

  • partial $R^2$ = variance predictor / total variance
  • $f^2$ = variance predictor / residual variance = ${R^2/{(1-R^2)}}$

Solution on a variance ratio perspective

• GPower: regression coefficient (t-test / regression, fixed model single regression coef)
  • use reference example, regression style (sd of effect and error, but squared)
• Calculate sample size, compare effect sizes ?
  • 128, same as for reference example, now with $f^2$ = $.25^2$ = .0625 (d=.5,r=.243)
• What if also other predictors in the model ?
  • very little impact → loss of degree of freedom
  • ignore that predictors explain variance → reduce residual variance
• What if 3 predictors extra reduce residual variance to 50% ?
  • control for confounding variables: less noise → bigger effect size
  • sample size much less (65)

A variance ratio perspective on multiple groups

Multiple groups: omnibus

• Difference between some groups → at least two differ

• GPower: one-way Anova (F-test / Means, ANOVA - fixed effects, omnibus, one way)

  • effect size f, with numerator/denominator df
  • obtain sample size for reference example, just 2 groups C and T1 (size=64)!
  • play with sizes, how does size matter ?
  • include third group, with mean 2, what are sample sizes (compare with 2 groups)?
  • set third group mean to 0, how does it compare with mean 2 (think and try)?
  • set third group mean to 4, but also vary middle group (eg., 1 or 3), does that have an effect ?
  • change procedure: repeat for between variance 2.67 (balanced: 0, 2, 4) and within variance 16 ?

Solution for multiple groups omnibus

• GPower: one-way Anova (F-test / Means, ANOVA - fixed effects, omnibus, one way)
• Obtain sample size for reference example, just 2 groups C and T1 (size=64)!
  • 128, same again, despite different effect size (f) and distribution
  • size used only to include imbalance
• Include third group, with mean 2, what are sample sizes (compare with 2 groups)?
  • effect sizes f = .236; sample size 177 (59*3), requires more observations
• Set third group mean to 0, how does it compare with mean 2 (think and try)?
  • effect and sample size same, no difference whether big 0 group or big 2 group.
• Set third group mean to 4, but also vary middle group (eg., 1 or 3), does that have an effect ?
  • effect sizes f = .408 (4), .425 (1/3), increase with middle group away from middle.
• Change procedure: repeat for between variance 2.67 (balanced: 0, 2, 4) and within variance 16 ?
  • sample size 21*3=63, for f = .408 (1/7th explained = 1 between / 6 within)

Multiple groups: pairwise

• Assume one control, and two treatments

  • interested in all three pairwise comparisons → maybe Tukey
    • typically run aposteriori, after omnibus shows effect
  • use multiple t-tests with corrected $\alpha$ for multiple testing
    GPower: t-tests/means difference two independent groups

• Apply Bonferroni correction for original 3 group example (0, 2, 4)

  • what samples sizes are necessary for all three pairwise tests ?
  • what if biggest difference ignored (C-T2), because know that easier to detect ?
  • with original 64 sized groups, what is the power to detect a difference group (C-T1) (both situations above) ?

Solution for multiple groups pairwise

• GPower: t-tests/means difference two independent groups
• Apply Bonferroni correction for original 3 group example (0, 2, 4)
• What samples sizes are necessary for all three pairwise tests ?
  • 0-2 and 2-4 → d=.5, 0-4 → d=1
  • divide $\alpha$ by 3 → .05/3=.0167
  • sample size 86 2 for 0-2 and 2-4, 23 2 for 0-4 → 86 * 3 = 258
• What if biggest difference ignored (C-T2), because know that easier to detect ?
  • divide $\alpha$ by 2 → .05/2=.025
  • sample size 78 2 for 0-2 and 2-4 → 78 3 = 234 (24 less)
• With original 64 sized groups, what is the power (both situations above) ?
  • .6562 for 3 tests ( $\alpha$ =.0167)
  • .7118 for 2 tests ( $\alpha$ =.0250)
  • post-hoc test → power-loss (lower $\alpha$ → higher $\beta$)

Multiple groups: contrasts

Multiple groups: contrasts (continued)

• GPower: one-way ANOVA (F-test / Means, ANOVA-fixed effects,special,main,interaction)

• Obtain effect sizes for contrasts (assume equally sized for convenience)

  • $\sigma_{contrast}$ T1-C: $\frac{(-1*0 + 1*2 + 0*4)}{\sqrt(2*((-1)^2+1^2+0^2))} = 1$; $\sigma_{error}$ = 4 → $f$=.25
  • $\sigma_{contrast}$ T2-C: $\frac{(-1*0 + 0*2 + 1*4)}{\sqrt(2*((-1)^2+0^2+1^2))} = 2$; $\sigma_{error}$ = 4 → $f$=.5
  • $\sigma_{contrast}$ (T1+T2)/2-C: $\frac{(-1*0 + (1/2)*2 + (1/2)*4)}{\sqrt(3*((-1)^2+(1/2)^2+(1/2)^2))} = 1.414214$; $\sigma_{error}$ = 4 → $f$=.3535

• Sample size for each contrast, each 1 df

  • what samples sizes for either contrast 1 or contrast 2 ?
  • what samples sizes for both contrast 1 and contrast 2 combined ?
  • if taking that sample size, what will be the power for T1-T2 ?
  • what samples size for contrast 3 ?

Solution for multiple groups contrasts

• GPower: one-way ANOVA (F-test / Means, ANOVA-fixed effects,special,main,interaction)
• What samples sizes for either contrast 1 or contrast 2 ?
  • variance explained $1^2$ or $2^2$
  • for T1-C $f$ = $\sqrt{1^2/4^2}$ = .25 = d/2 → 128 (64 C - 64 T1)
  • for T2-C $f$ = $\sqrt{2^2/4^2}$ = .50 = d/2 → 34 (17 C - 17 T2)
• What samples sizes for both contrast 1 and contrast 2 combined ?
  • multiple testing, consider Bonferroni correction → /2
  • for T1-C 155, for T2-C 41 → total 175 (78 C, 77 T1, 20 T2)
• If taking that sample size, what will be the power for T1-T2 ?
  • post-hoc, 77 and 20, with d=.5 and $\alpha$ = .5 → power $\approx$ .5
• What samples size for contrast 3 ?
  • variance contrast $1.4142^2$
  • 3 groups, little impact if any
  • for .5*(T1+T2) - C $f$ = $\sqrt{2/16}$ = .3535 → 65 (22 C, 21 T1, 22 T2)

Multiple factors

• Multiple main effects and possibly interaction effects (eg., treatment and type)

  • main effects (average effects, additive) & interaction (factor level specific effects)
  • note: numerator degrees of freedom → main effect (nr-1), interaction (nr1-1)*(nr2-1)
  • $\eta^2$ = $f^2 / (1+f^2)$, remember $f = d/2$ for two groups
  • note: get effect sizes for two way anova: http://apps.icds.be/effectSizes/

• GPower: multiway ANOVA (F-test / Means, ANOVA-fixed effects,special,main,interaction)

  • determine $\eta^2$ and sample size for reference example,
    remember the between group variance ?
  • use the app: use for means only values 0 and 2, and 4 and 6 if necessary
    for treatment use C-T1-T2, for type (second predictor) use B1-B2
    • get $\eta^2$ for treatment effect but no type effect ? recognize $f$ ?
    • specify such that types differ, not treatment → $f$ and sample size ?
    • specify such that treatment effect only for one type → $f$ and sample size ?
    • specify effect for both treatment and type, without interaction → $f$ and sample size ?

Solution for multiple factors

• GPower: multiway ANOVA (F-test / Means, ANOVA-fixed effects,special,main,interaction)
• Determine sample size for reference example,
  remember the between group variance ?
  • between group variance 1, within 16, sample size 128 (numerator df = 2-1)
  • 2 x 2 with 0-2 → $\eta^2$ as expected = .0588
• Get $\eta^2$ for treatment effect but no type effect ? recognize $f$ ?
  • 0-2-4 for both types → $f$ = .4082 of the omnibus F-test (compare all groups)
• Specify such that types differ, not treatment → $f$ and sample size ?
  • 0-0-0 versus 2-2-2 → $f$ = .25 of t-test (compare two groups)
• Specify such that treatment effect only for one type → $f$ and sample size ?
  • 0-2-4 versus 0-0-0 → $f$ = .2041, .25 and .2041
    • detect interaction (num df = 2) = 235 total (40 per combination)
    • detect only treatment effect (num df = 2) = 235 total (79 each group, 79/2 per combination)
    • detect only type effect (num df = 1) = 128 total (64 each group, 64/3 per combination)
    • detect both both main effects = 40 each combination ~ max(79/2,64/3)
• Specify effect for both treatment and type, without interaction → $f$ and sample size ?
  • 0-2-4 versus 2-4-6 → $f$ = .4082, .25 and 0, sample size = 21 per combination

Repeated measures

• If repeated measures → account for correlations within

• Possible to focus on:

  • within: similar to dependent t-test for multiple measurements
  • between: group comparison, each based on multiple measurements
  • interaction: difference between changes over measurements (within)

• Correlation within unit (eg., within subject)

  • informative within unit (like paired t-test)
  • redundancy on information between units (observations less informative)

• Beware: effect size could include or exclude correlation

• GPower: repeated measures (F-test / Means, repeated measures...)

  • correlation not yet included → Options: 'as in GPower 3.0'
  • correlation already included → Options: 'as in SPSS'

Repeated measures within

• GPower: repeated measures (F-test / Means, repeated measures within factors)

• Use effect size f = .25 (1/16 explained versus unexplained)

  • mimic dependent t-test, correlation .5 !
  • mimic independent t-test, but only use 1 group !
  • double number of groups to 2, or 4 (cor = .5), what changes ?
  • double number of measurements to 4 (cor = .5), impact ?
  • compare impact double number of measurements for correlations .5 with .25 ?

Solution for repeated measures within

• GPower: repeated measures (F-test / Means, repeated measures within factors)
• Mimic dependent t-test, correlation .5 !
  • only 1 group, 2 repeated measures, correlation .5 → 34 x 2 measurements
• Mimic independent t-test, but only use 1 group !
  • only 1 group, 2 repeated measures, correlation 0 → 65 x 2 measurements
• Double number of groups to 2, or 4 (cor = .5), what changes ?
  • number of groups not relevant for within group comparison
  • but requires estimation, changed degrees of freedom
• Double number of measurements to 4 (cor = .5), impact ?
  • sample size reduces from 34 to 24, but 34x2=68, 24*4=96
• With 4 measurements (double) take halve the correlation (0.25), impact ?
  • sample size 35, nearly 34
  • 2 repeated measurements with corr .5, about same sample size as 4 repeats with corr .25

Repeated measures between

• GPower: repeated measures (F-test / Means, repeated measures between factors)

• Use effect size f = .25 (1/16 explained versus unexplained)

  • compare 2 groups, each 2 measurements...
    impact on sample size when correlation 0, .25 and .5 ?
  • double number of groups to 2, or 4 (cor = .5), what changes ?
  • double number of measurements to 4 (cor = .5), impact ?
  • compare impact number of measurements for different correlations .5 with .25 ?
  • mimic independent t-test ?

Solution for repeated measures between

• GPower: repeated measures (F-test / Means, repeated measures between factors)
• Use effect size f = .25 (1/16 explained versus unexplained)
• Compare 2 groups, each 2 measurements...
  impact on sample size when correlation 0, .25 and .5 ?
  • increase in correlations results in increase in sample size (redundancy)
• Double number of groups to 2, or 4 (cor = .5), what changes ?
  • increase in number of groups, small increase (estimation required) IF same effect size $f$
• Double number of measurements to 4 (cor = .5), impact ?
  • increase in number of measurements, increases total number, but reduces number of units
• Compare impact number of measurements for different correlations .5 with .25 ?
  • increase stronger if correlations stronger
• Mimic independent t-test ?
  • 128 units, if .99 correlation with fully redundant second set
  • 132 units (66*2), if 0 correlation with need to estimate four group (2x2) averages and correlation

Repeated measures interaction within x between

• GPower: repeated measures (F-test / Means, repeated measures within-between factors)

• Option: calculate effect sizes: http://apps.icds.be/effectSizes/

  • for sd = 4, with group with average 0-2-4, and with non-responsive (all 0):
  • compare effect sizes for interaction with correlation .5 and 0, conclude ?
  • compare sample sizes for those 2 effect sizes with correlation .5 or 0 ?

Solution for repeated measures interaction within x between

• GPower: repeated measures (F-test / Means, repeated measures within-between factors)
• Option: calculate effect sizes: http://apps.icds.be/effectSizes/
• For sd = 4, with group with average 0-2-4, and with non-responsive (all 0):
• Compare effect sizes for interaction with correlation .5 and 0, conclude ?
  • with 0 correlation → $f$ for interaction = .25
  • with .5 correlation → $f$ = .3536
• Compare sample sizes for those 2 effect sizes with correlation .5 or 0 ?
  • for $f$ = .25, sample sizes are 54x2 (cor=0) and 28x2 (cor=.5)
  • for $f$ = .3535, sample sizes are 28x2 (cor=0) and 16x2 (cor=.5)
  • either include .5 correlation to calculate effect size OR sample size

Correlations

• If comparing two independent correlations

• Use Fisher Z transformations to normalize first

  • z = .5 * log( $\frac{1+r}{1-r}$ ) → q = z1-z2

• GPower: z-tests / correlation & regressions: 2 indep. Pearson r's

  • with correlation coefficients .7844 and .5, what are the effect & sample sizes ?
  • with the same difference, but stronger correlations, eg., .9844 and .7, what changes ?
  • with the same difference, but weaker correlations, eg., .1 and .3844, what changes ?

• Note that dependent correlations are more difficult, see manual

Solution for correlations

• GPower: z-tests / correlation & regressions: 2 indep. Pearson r's
• With correlation coefficients .7844 and .5, what are the effect & sample sizes ?
  • effect size q = 0.5074, sample size 64*2 = 128
  • $.5*log((1+.7844)/(1-.7844)) - .5*log((1+.5)/(1-.5))$
  • notice: effect size q $\approx$ d, same sample size
• With the same difference, but stronger correlations, eg., .9844 and .7, what changes ?
  • effect size q = 1.5556, sample size 10*2 = 20
  • same difference but bigger effect (higher correlations more easy to differentiate)
• With the same difference, but weaker correlations, eg., .1 and .3844, what changes ?
  • effect size q = 0.3048, sample size 172*2 = 344
  • same difference, negative, and smaller effect (lower correlations more difficult to differentiate)

Proportions

• If comparing two independent proportions → bounded between 0 and 1

• GPower: Fisher Exact Test (exact / proportions, difference 2 independent proportions)

• Effect sizes in odds ratio, relative risk, difference proportion

  • for odds ratio 3 and p2 = .50, what is p1 ? and for odds ratio 1/3 ?
  • what is the sample size to detect a difference for both situations ?
  • for odds ratio 3 and p2 = .75, determine p1 and sample size,
    how does it compare with before ?
  • for odds ratio 1/3 and p2 = .25, determine p1 and sample size,
    how does it compare with before ?
  • compare sample size for a .15 difference, at p1=.5 ?

Solution for proportions

• GPower: Fisher Exact Test (exact / proportions, difference 2 independent proportions)
• For odds ratio 3 and p2 = .50, what is p1 ? and for odds ratio 1/3 ?
  • odds ratio 3 → with p2 = .5 or odds_2 = 1, odds_1 = 3 thus p1 = 3/(3+1) = .75
• What is the sample size to detect a difference for both situations ?
  • 128, same for .5 versus .25 or .75 (unlike correlation)
• For odds ratio 3 and p2 = .75, determine p1 and sample size, how does it compare with before ?
  • p1 to .9, difference of .15, sample size increases to 220
• For odds ratio 1/3 and p2 = .25, determine p1 and sample size, how does it compare with before ?
  • p1 to .1, difference of .15, sample size increases to 220
• Compare sample size for a .15 difference, at p1=.5 ?
  • sample size even higher, to 366, increase not because smaller difference

Exercise proportions

• GPower: Fisher Exact Test (exact / proportions, difference 2 independent proportions)

Solution for proportions

• GPower: Fisher Exact Test (exact / proportions, difference 2 independent proportions)

Dependent proportions

• If comparing two dependent proportions → categorical shift

  • if only two categories, McNemar test: compare $p_{12}$ with $p_{21}$
  • information from changes only → discordant pairs
  • effect size as odds ratio → ratio of discordance
  • like other exact tests, choice assignment alpha

• GPower: McNemar test (exact / proportions, difference 2 dependent proportions)

  • assume odds ratio equal to 2, equal sized, type I and II errors .05 and .2, two-way !
  • what is the sample size for .25 proportion discordant, .5, and 1 ?
  • odds ratio 1 versus .5, (prop discordant = .25), what are $p_12$ and $p_21$ and sample sizes ?
  • repeat for third alpha option, and consider total sample size, what happens ?

Solution for dependent proportions

• GPower: McNemar test (exact / proportions, difference 2 dependent proportions)
• Assume odds ratio equal to 2, equal sized, type I and II errors .05 and .2, two-way !
• What is the sample size for .25 proportion discordant, .5, and 1 ?
  • 288 (.25), 144 (.5), 73~144/2 (.99) → decrease in sample size with increased discordance
• Odds ratio .5 or 4, (prop discordant = .25), what are $p_{12}$ and $p_{21}$ and sample sizes ?
  • same as 2 but reverse $p_{12}$ and $p_{21}$, with sample size 288
  • with 4 as odds ratio, larger effect, requires smaller sample size, only 80
  • odds ratio = $p_{12}$ / $p_{21}$
• Repeat for third alpha option, with odds ratio 4, what happens ?
  • changed lower / upper critical N, lower sample size
  • BUT, is because lower power, closer to requested .8

Not included

• Various statistical tests difficult to specify in GPower

  • various statistics / parametervalues that are difficult to guestimate
  • manual for more complex tests not always very elaborate

• Various statistical tests not included in GPower

  • eg., survival analysis
  • many tools online, most dedicated to a particular model

• Various statistical tests no formula to offer sample size

  • simulation may be the only tool
    • iterate many times: generate and analyze → proportion of rejections
    • generate: simulated outcome ← model and uncertainties
    • analyze: simulated outcome → model and parameter estimates + statistics

Simulation example t-test

gr <- rep(c('T','C'),64)
y <- ifelse(gr=='C',0,2)
dta <- data.frame(y=y,X=gr)
cutoff <- qt(.025,nrow(dta))
my_sim_function <- function(){
    dta$y <- dta$y+rnorm(length(dta$X),0,4)        # generate (with sd=4)
    res <- t.test(data=dta,y~X)                    # analyze
    c(res$estimate %*% c(-1,1),res$statistic,res$p.value)
}
sims <- replicate(10000,my_sim_function())        # many iterations
dimnames(sims)[[1]] <- c('diff','t.stat','p.val')
mean(sims['p.val',] < .05)    # p-values    0.8029
mean(sims['t.stat',] < cutoff)    # t-statistics 0.8029
mean(sims['diff',] > sd(sims['diff',])*cutoff*(-1))    # differences 0.8024

Focus / simplify

• Complex statistical models

  • simulate BUT it requires programming and a thorough understanding of the model
  • alternative: focus on essential elements → simplify the aim

• Sample size calculations (design) for simpler research aim

  • not necessarily equivalent to final statistical testing / estimation
  • requires justification to convince yourself and/or reviewers
    • successful already if simple aim is satisfied
    • ignored part is not too costly

• Example:

  • statistics: group difference evolution 4 repeated measurements → mixed model
  • focus: difference treatment and control last time point is essential → t-test
  • argument: first 3 measurements low cost, interesting to see change

Conclusion

• Sample size calculation is a design issue, not a statistical one

• Building blocks: sample & effect sizes, type I & II errors

  • establish any of these building blocks, conditional on the rest

• Effect sizes express the amount of signal compared to the background noise

• GPower deals with not too complex models

  • more complex complex models imply more complex specification
  • simplify using a focus, if justifiable → then GPower can get you a long way

Methodological and statistical support to help make a difference

• SQUARE provides complementary support in methodology and statistics to our research community, for both individual researchers and research groups, in order to get the best out of them

• SQUARE aims to address all questions related to quantitative research, and to further enhance the quality of both the research and how it is communicated

website: https://square.research.vub.be/ includes information on who we serve, and how

booking: https://square.research.vub.be/bookings for individual consultations