What are the advantages and disadvantages of using standardized effect sizes in experimental research?

Effect size measures are numerical indicators of the magnitude and direction of the difference between two groups or the strength of the relationship between two variables. In experimental research, effect size measures can help researchers to evaluate the practical significance and generalizability of their results, beyond the statistical significance. However, there are different types of effect size measures, each with its own advantages and disadvantages. In this article, we will compare and contrast the most common types of effect size measures in experimental research: standardized and unstandardized.

1 Standardized effect sizes

Standardized effect sizes are calculated by dividing the difference between two groups or the strength of the relationship between two variables by a standard deviation or a pooled standard deviation. The most widely used standardized effect sizes are Cohen's d for mean differences, Pearson's r for correlations, and eta-squared for variance explained. Standardized effect sizes have the advantage of being unitless and scale-free, meaning that they can be easily compared across different studies and contexts. They also have established guidelines for interpreting their magnitude, such as small, medium, and large effects. However, standardized effect sizes also have some drawbacks, such as being sensitive to sample size, variability, and measurement error, and ignoring the meaningfulness and relevance of the units of measurement.

Add your perspective

Nirmal Ghimire, Ph.D.

Assistant Professor @ Watson College of Education, UNCW. Integrating #BigData, #ML & #AI to advance #Literacy, #TeacherEducation & #EducationalEquity through #Research, #Teaching & #CommunityEngagement.
Report contribution
The standardized effect size assess the size and/or magnitude of the difference between two groups of entities being studied by scaling the effect size relative to a SD or another unit of measurement. They are commonly used measures in experimental research especially to measure the strength of the effect of the treatment under study. They are also used to determine the statistical power-the probability of detecting a difference between groups if one exists. Some common types of standardized effect sizes are Cohen's d, Hedges' g, and Pearson's r. Sources: - American Psychological Association. (2019). - Mallinckrodt, B., & Leong, F. T. L. (2012).

Like
Tushar Verma

Market Researcher at Industrial Info Resources ll Photographer ll Writer ll Chef ll Physics ll Mentor II
Report contribution
Advantages include: They allow for the evaluation of the magnitude of an effect when the units of measurement aren’t intuitive. They enable comparison of results across different studies. They are unitless, which can be beneficial when the original units are not inherently meaningful or potentially confusing. However, there are also disadvantages: Standardized effect sizes are less easily understood. The use of standardized effect sizes in primary research and their interpretation can be hampered if done without considering the historical and research context. They should be used with caution in sample size calculations. It’s important to consider these factors when deciding to use standardized effect sizes in experimental research.

Like

2 Unstandardized effect sizes

Unstandardized effect sizes are calculated by using the original units of measurement of the variables or the outcomes of interest. Examples of unstandardized effect sizes are mean difference, risk difference, risk ratio, odds ratio, and slope coefficient. Unstandardized effect sizes have the advantage of being more intuitive and meaningful, as they reflect the actual change or difference that occurs in the phenomenon under study. They also have the advantage of being less affected by sample size, variability, and measurement error, as they do not rely on a standard deviation. However, unstandardized effect sizes also have some limitations, such as being dependent on the scale and units of measurement, making them harder to compare across different studies and contexts. They also have no clear guidelines for interpreting their magnitude, as they depend on the substantive and practical significance of the outcome.

Add your perspective

3 Choosing between standardized and unstandardized effect sizes

When deciding which type of effect size measure is more appropriate for experimental research, there is no definite answer. The choice depends on several factors, such as the research question, the design of the study, the nature of the variables, the audience of the report, and the conventions of the field. When choosing between standardized and unstandardized effect sizes, researchers should consider the purpose and scope of the study, how to interpret and communicate the results, and any assumptions and limitations associated with each type of measure. For instance, standardized effect sizes assume that the standard deviation is a valid measure of variability; however, this may not be true for skewed or ordinal data. On the other hand, unstandardized effect sizes assume that the units of measurement are meaningful; yet this may not be true for arbitrary or transformed data.

Add your perspective

Sat Virk, PhD, JD

AI Mixed Methods User Researcher / Design Strategist
Report contribution
Cohen's D (standardized) effect size tells you the % of users in the control group below the mean in the experimental group. It doesn't tell you practically the delta for that effect which is very important. If you run an A/B test on two buttons and the Cohen's D is 1.4, 92% of users exhibit the effect where button 2 has many more clicks than button 1. But, how many clicks? Let's say the difference is a mean of 20 clicks for Button A and 80 clicks for Button B. 60 clicks could translate to 7 million dollars ROI / month for a mobile app for 92% of users! That is strategically very important data (build button 2!). Standardized and unstandardized measures should be given together to see the whole picture.

Like

4 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

Add your perspective

Nirmal Ghimire, Ph.D.

Assistant Professor @ Watson College of Education, UNCW. Integrating #BigData, #ML & #AI to advance #Literacy, #TeacherEducation & #EducationalEquity through #Research, #Teaching & #CommunityEngagement.
Report contribution
Odds Ratio: The odds ratio is one of a very useful tools in experimental research. Odds ratio higher than 1 suggests that the treatment group is more likely to experience statistically significant difference compared to control group, and vice-versa. For example, in a medical research like effectiveness of a certain COVID-19 vaccine, the odds ratio can help researchers identify risk factors associated to COVID-19 and if the proposed vaccine is more effective for certain demographic groups. The choice of method for measuring the odds ratio depend on many factors like the research question and the data type. Some of the popular odds ratio is called Mantel-Haenszel, conditional logistic regression, and unconditional logistic regression.

Like

What are the advantages and disadvantages of using standardized effect sizes in experimental research?

1

2

3

4

1 Standardized effect sizes

2 Unstandardized effect sizes

3 Choosing between standardized and unstandardized effect sizes

4 Here’s what else to consider

Experimental Research

Rate this article

Thanks for your feedback

More articles on Experimental Research

More relevant reading