Methods Change Raises New Questions of Bias in Cass Review
Questions surround changes to the methodology of several systematic reviews of gender-affirming treatments for minors by York University.
by Evan Urquhart
The release of the Cass Report in April has had wide reaching political and policy implications. While misinformation has dogged the Report’s release, with anti-trans activists and right-wing outlets misrepresenting Cass’ findings, the Report’s core finding, stating the body of medical evidence supporting puberty blockers and hormone therapy for trans youth was “weak,” is sure to cast a pall over the provision of such care in the UK and beyond, and to do so for the foreseeable future.
This finding of weak evidence was based on the conclusions of systematic reviews into puberty blockers and cross-sex hormones. These SRs were conducted by a team of researchers from York University in England, led by Jo Taylor, and published on April 9 in the British Medical Journal. However, an unacknowledged change from the pre-registered methodology of the reviews is raising new questions about whether these reviews can truly be relied upon as unbiased assessments of the literature.
A systematic review is a type of study that examines other studies in a rigorous fashion. SRs such as the two in question provide a comprehensive look at research conducted on a scientific question. They describe what, if any, conclusions can be drawn from taking all of it together. SRs are vitally important for the advancement of scientific knowledge, however, as with all science research, SRs involve human judgment at many key moments, such as when choosing the methods, criteria, and scoring or evaluating studies. This opens the door to bias in much the same way that anything else humans do is vulnerable to human biases.
In order to limit the potential bias, good SRs employ something called pre-registration. In pre-registration, a prospective study’s authors write down exactly what they plan to do and how they plan to do it beforehand. They lock themselves into their methods, preventing them from yielding to the temptation to select some particular method in order to make it easier to reach the result they’re hoping for.
That’s why it’s notable when the methods of a study don’t match those that were laid out in pre-registration. And, in the case of the York team's studies of puberty blockers and hormone treatments, their methods don’t match pre-registration. The York team’s pre-registered documentation is a document that broadly describes all the research questions and systematic reviews that were planned at the outset of their research. When it comes to how they plan to evaluate the studies in their SRs it states, “The Mixed Methods Appraisal Tool (MMAT) will be used to assess the quality of studies.”
screenshot from PROSPERO pre-registration for the York team’s research
However, when you read the methods sections for the York team’s SRs on puberty blockers and hormone treatments Taylor et al. make no mention of the MMAT. Instead, both state that a modified version of the Newcastle Ottowa Scale (NOS) was used to evaluate study quality. So Taylor et al. used a different instrument to rate study quality from the one originally planned, and then they modified it.
This discrepancy between the pre-registered plan in PROSPERO (an international register for exactly these types of research plans that is maintained by York University) versus the methods in the final papers was first remarked on by Simon Whitten. Soon after having read Whitten’s thread on twitter.com, we contacted the York team to ask about it. We asked if we could interview a representative of the team or, if no one was available, if they could tell us exactly when and why the decision to move from MMAT to the NOS happened.
A representative of the York team replied by email, but without responding to our questions. This spokesperson wrote, “We undertook a linked series of systematic reviews and for each review we selected the most appropriate appraisal tool for the study designs included. For the review assessing the outcomes of psychosocial support interventions we used the Mixed Methods Appraisal Tool. For the review of clinical guidelines, we used the Appraisal of Guidelines for Research and Evaluation II tool. For the reviews assessing the outcomes of social transition, puberty suppression, and masculinising or feminising hormones, we used the Newcastle Ottawa Scale. The Newcastle Ottawa Scale provides a more detailed set of questions for non-randomised studies than the Mixed Methods Appraisal Tool.”
Because the York team were not responsive to our questions, we sought further expert input from Dr. Jack Turban, the Director of the Gender Psychiatry Program at the University of California. As his title indicates, Dr. Turban is a practitioner in the field of gender-affirming treatments. He’s also a researcher whose own work was included in the scope of the York team’s SRs, and evaluated as low quality.
Dr. Turban’s answers to our questions about the role and importance of pre-registration informed Assigned’s research and reporting into the purpose and importance of pre-registration for this sort of study, discussed above. Asked specifically about the unacknowledged shift by the York team from MMAT to NOS, Dr. Turban told us, via email, “Pre-registration is done so that authors can’t meddle with their methodology later if they don’t like their results. While there could be valid reasons to change methodology after pre-registration, this should be discussed in the manuscript and reviewed by peer-reviewers.The fact that the authors of this study used an adapted scale, which they customized themselves, raises particular concern. Ideally that adapted customized scale would have been pre-registered.”
Evan Urquhart is the founder of Assigned Media and an incoming member of the 2024-2025 Knight Science Journalism fellowship class at MIT.