Aim. This work aims to quantify the rate of false positive trials due to poor pre-registration of outcomes. It also proposes solutions to mitigate this phenomenon.
Methods. 20 spine trials using ODI as a primary outcome, pre-registered on ClinicalTrials.gov in 2022, were included. Based on the description of the outcome included in the registration (e.g. "ODI", "change in ODI", "at least 5 points change in ODI"), each trial was simulated 100,000 times using R under the null hypothesis (where the intervention has no effect). In scenario A, each set of simulation results was analysed using seven of the most common statistical tests used in spine research: ANCOVA, t-test on the post scores, t-test on the change in scores, dichotomisation by fixed optimal cut-point, dichotomisation by arbitrary post hoc cut-point, dichotomisation by fixed optimal percentage, and dichotomisation by arbitrary post hoc percentage. In scenario B, the same tests were performed but dichotomisation was restricted to cut-points found in the literature.
Results. 18 of the 20 trials (90%) did not specify their analytical strategy for ODI with sufficient detail to be limited to a single statistical test. Scenario A showed that when all allowable tests are performed, p-values below 0.05, which are expected to arise 5% of the time under the null hypothesis, were instead returned between 19% and 48% of the time. Scenario B found that when dichotomisation is limited to previously published cut-points, this reduces the prevalence of significant trials by 20 to 50%. Finally, using a single dichotomisation cut-point calculated to match the expected mean change reduces the inflation of significant results by up to 83%.
Conclusions. Current registration practices allow for an analytical freedom that impacts the meaningfulness of trial results. Most of the trials simulated returned a significant p-values from at least one test at least 20% of the time. The cherry picking of dichotomisation cut-points, in particular, inflates false positives up to 9 times and we conclude that limiting dichotomisation, when chosen as the analytical strategy, to a single cut-point chosen at the time of pre-registration, would significantly reduce false positives.