Statistical power and precision of experimental studies originated in the Republic of Turkey from 2010 to 2020: Current practices and some recommendations

Authors : Metin Bulus, Ilhan Koyuncu

Pages : 24-43

Doi:10.17275/per.21.77.8.4

View : 24 | Download : 7

Publication Date : 2021-12-01

Article Type : Research

Abstract :This study systematically reviews randomly selected 155 experimental studies in education field originated in the Republic of Turkey between 2010 and 2020. Indiscriminate choice of sample size in recent publications prompted us to evaluate their statistical power and precision. First, above and beyond our review, we could not identify any large-scale experiments such as cluster-randomized or multisite randomized trials, which overcome shortcomings of small-scale experiments, better suit to the organizational structure of the education field, nevertheless require far greater effort and financial resources. Second, none of the small-scale experiments has reported or conducted ex-ante power analysis. Third, results indicate that studies are sufficiently powered to detect medium effects and above (Cohen’s d ≥ 0.50), however they are underpowered to detect small effects (Cohen’s d ≤ 0.20). Trends in the past ten years indicate precision remained unchanged. We made several recommendations to increase the precision of experimental designs and improve their evidential values: Determine sample size prior to an experiment with power analysis routine, randomize subjects / clusters to obtain unbiased estimates, collect pre-test information and other relevant covariates, adjust for baseline differences beyond covariate control, document attrition, report standardized treatment effect and standardized variance parameters. Findings should be interpreted considering minimum effects in education that are relevant to education policy and practice.
Keywords : experimental design, systematic review, minimum detectable effect size, statistical power, precision