Why i get annoyed at "research led" design/reform within education.

Inappropriate Use of Statistical Tests

Improper application of statistical tests can lead to incorrect conclusions. This occurs when researchers choose statistical methods that are not suitable for their data or research design.

Example:

Misusing Parametric Tests: A researcher might use a t-test (a parametric test assuming normal distribution of data) to compare test scores between two groups of students. However, if the test scores are not normally distributed (which is common in educational data), the use of a t-test could lead to incorrect conclusions. A non-parametric test like the Mann-Whitney U test would be more appropriate in such cases.

Lack of Replication and Cross-Validation

Educational research often suffers from a lack of replication, meaning the studies are not repeated to verify results. Cross-validation involves testing the findings in different settings or populations to ensure they are universally applicable.

Example:

Single-Context Study: A study conducted in an urban, high-income school district finds that integrating tablets into the classroom significantly improves student engagement. However, this study is not replicated in rural or lower-income districts. Without replication, it's unclear if the positive effect of tablets on student engagement is universally applicable or if it's specific to certain types of school districts.

Data Dredging or P-hacking

Data dredging or p-hacking involves manipulating data analysis to find statistically significant results. This practice can lead to false-positive findings or overstate the effectiveness of an intervention or educational technique.

Example:

Searching for Significance in Subgroups: An educational researcher analyzes data from a new teaching method intended to improve math skills. The initial analysis shows no significant improvement overall. However, the researcher then begins to test numerous subgroups (e.g., boys vs. girls, different age groups, students from different schools) until a statistically significant improvement is found in one small subgroup. This approach can lead to a misleading conclusion about the effectiveness of the teaching method, as the significant result may be due to chance rather than a real effect.