Continue Learning with FreeCME
Clinician.com and FreeCME Account Successfully Created
Your new Clinician account also gives you access to FreeCME, where you can take free accredited for CE credit
IRBs Inconsistently Assess Scientific Merit
July 1, 2025 9 minutes read
By Stacey Kusterbeck
In the United States, thousands of people volunteer for clinical research trials each year. “They put their bodies on the line, often seeking benefits from experimental treatments because existing care is ineffective or because they are altruistic,” says Michael S. Wilkes, MD, director of global health and professor of medicine at University of California, Davis. Institutional Review Boards (IRBs) are ethically obligated to assess the scientific merit of studies to ensure high-quality trials that justify the risks to participants. Wilkes and colleagues conducted a study to see how well IRBs were fulfilling this obligation.1 The researchers analyzed IRB standard operating procedures and other relevant materials from 35 nonprofit institutions and one for-profit IRB to determine whether they provided adequate guidance on assessing scientific merit.
Basic study elements, such as study design, study recruitment, and intervention justification, were mentioned in the vast majority of IRB materials. However, there were significant gaps in critical aspects related to study quality. These included bias reduction and outcome measurement tools (53% and 57% of IRB materials included these, respectively). Only 10% of IRB materials mentioned confounder control.
“It turns out that, frequently, no one is examining the science behind a proposed clinical trial,” says Wilkes.
Many IRBs routinely overlook the legal requirement to assess scientific merit, the study findings suggest. This grants clinical trials a stamp of ethical approval without this essential evaluation. “Without scientific assessments, IRBs are failing to meet their ethical duty to protect trial participants,” warns Wilkes. This means that vulnerable people may be asked to participate in low-quality trials that have no chance of yielding a clinically meaningful outcome.
Participants need to have assurance that the clinical trials for which they are being recruited employ scientific methods that can address the hypothesis suggested. “Without this reassurance, the trial is a betrayal of the trust that trial volunteers place in the research enterprise,” argues Wilkes.
One reason for lack of consistency in how IRBs review study protocols is that new technologies or approaches that never have been studied previously continually come up. Thus, each IRB must figure out independently how to oversee it. “Every time we are looking at a new type of study with something novel, we are looking at new questions. And old guidelines are hard to fit into new questions,” says Sara Huston, MS, a research assistant professor at Ann & Robert H. Lurie Children’s Hospital of Chicago and Northwestern University.
More than a decade ago, Huston, a geneticist by training, voiced concerns over IRBs managing genomic research data inconsistently. IRB members had different views on what results should be returned and what risks needed to be explained to participants. “Some IRBs were being overly lenient and some were overly protective,” recalls Huston. Ultimately, Huston joined the IRB to provide expertise on how to consistently address those and other ethical considerations in genomic data studies. “Managing genomics in research studies was a mess when I started out in the 2010s — that decade was quite messy. Now, IRBs are managing genomic data pretty consistently across different institutions. They’re providing information in consent forms that were not being done consistently, they are returning results that participants might be interested in having more consistently. You see a shift, but it took a lot of time to get those changes into play,” says Huston.
When encountering novel types of research, IRBs share information at conferences or through word of mouth. Eventually, this leads to greater consistency in how IRBs review those studies. “When technologies are brand new and we don’t understand them, IRBs need to do their best. Ultimately, the board has to make a decision based on the information that’s in front of them. Sometimes the study team doesn’t explain it well and we have to make a judgment call,” says Huston.
In some cases, IRBs need to obtain outside expertise because a question comes up that is beyond the scope of the IRB members. An ethical concern recently came up in a pediatric clinical trial involving using cameras on a crib to look at the causes of sudden infant death syndrome. This raised questions about what researchers were obligated to do to address privacy and confidentiality of the data, whether artificial intelligence (AI) could be used to re-identify people who were recorded, and whether the data could be used by the courts in child abuse cases. Another question was who the researchers needed consent from — whether it was just the occupants of the home or also visitors who might be recorded, such as nannies. The IRB deferred the protocol until legal and privacy experts could weigh in on the risks and protections needed.
“IRBs will defer a study if we don’t have enough information to make a call. IRBs aren’t perfect, but they are democratic. If you vote to approve, or you oppose, or you abstain, that is on the record. It’s the best we can do in terms of trying to protect participants and convey risks in research studies that have never been done previously,” says Huston.
There always is the possibility that IRB members may have biases and potential conflicts of interest. In part, this is because members often know the principal investigators of the study protocols being reviewed. Also, IRB members want to support the institution they work for. “There is potential conflict there, too. I don’t know that there is a perfect solution, but IRB members are generally pencil-pushing, honest people and are willing to speak up if there’s a concern,” says Huston.
Years ago, a study was going down the path of approval. Huston had concerns that it did not seem like a research study but more so an opportunity for the institution to make money by expanding the standard of care. She spoke up just before the motion to approve and, ultimately, the IRB opposed the study.
“We want to be sure we are doing right by participants — which is why IRBs have a reputation for being burdensome: ‘You need me to fill out another form?’” says Huston. “It’s an imperfect system. It is a watchdog of sorts, but it’s the best tool we’ve come up with as a research enterprise to have an external entity, with experts’ eyes that are focused on participant’s rights.”
Some IRBs work hard to require simple language in study protocols and consent forms and might tell the principal investigator (PI) to use the word “teaspoon” instead of “milliliter,” for instance. “You really need a good IRB staff that understands all these nuances,” says Huston. Many staff are novices who are not trained in medicine. The learning curve for new IRB staff can be frustrating for study teams and is another reason there can be inconsistencies in how study protocols are reviewed.
“It’s never going to be a perfect system, but it’s a pretty good system. We can get as close as we can to compliance and optimal protections for each study. And each study does need independent review, with humans involved and thinking about these difficult and novel questions. Because who else is going to do it?” asks Huston.
Federal regulations provide IRBs with a general structure on what criteria researchers have to meet if doing research on human subjects. “But they leave it to IRBs to decide how best to do that. Is there concern about inconsistency? Yes, I think so,” says Thomas D. Harter, PhD, director of the Department of Bioethics and Humanities and chair of the IRB at Emplify Health-Gundersen. “As chair of an IRB, my job is to say, ‘If we are going to do research on human subjects, what are we doing to protect their rights and interests? How are you protecting them?’”
The IRB’s role is to determine if there are concerns about participant safety that are not clearly addressed in the study protocol. For review of research determined to be minimal risk, the IRB members’ main focus is whether researchers are using the best methodology for conducting the study. Most of the time, the answer is yes, but there may be other concerns. Recently, the IRB reviewed a behavioral health survey study, and members wanted to know what researchers would do if the survey revealed that a particular participant has a high risk for self-harm. “If so, the protocol should describe the plan for unblinding the data, to try to help that person and reach out, to be sure we are protecting them,” says Harter.
When it comes to investigator-initiated, “homegrown” research, larger institutions are going to be better equipped to review the scientific merit than smaller institutions. That can lead to inconsistencies in how those study protocols are reviewed. “Vetting the scientific pieces of a protocol can be challenging for those smaller IRBs. That does raise ethical issues,” says Harter. “For big investigative protocols at an institution my size, for example, we don’t have researchers on staff who are initiating a lot of these prospective, blinded, investigational studies where we are doing something we consider experimental.”
Currently, the majority of protocols reviewed by the Emplify Health-Gundersen’s IRB are part of larger trials. “We are not the IRB of record. We are deferring to a centralized IRB that provides oversight, and we are reviewing it primarily to say: ‘We approve you to do this study locally with our patient population,’” Harter explains. Those studies already have been reviewed by the centralized or national IRB. The local IRB’s community members review informed consent documents to be sure that they are readable, especially regarding risks for participants. The IRB’s scientific members review the scientific aspects of the protocol. This is where there is a risk of inconsistency, since various IRBs at the local level might differ in terms of how closely they scrutinize the protocol. “When we send out a protocol to one of our scientific review members, they’re going to do a thorough job reading through the protocol. They will see if the research makes sense for what they are proposing and will give it a thorough read. I don’t know that all IRBs do that [when they are deferring to another IRB],” says Harter.
Some IRBs might take for granted that the research must be ethical, since it already has passed the scientific review through the centralized IRB. “There’s been a couple times when we had to pause and say, ‘We realize that the protocol has already been approved at the national level. But we are not sure that you have done enough to establish a research program that actually answers your research question. That’s rare, but it does happen,” says Harter.
IRB inconsistencies also happen because members lack the necessary expertise to review a particular study protocol. “Most inconsistencies are probably due to people thinking they can review a particular protocol when they really can’t because the scientific aspect is beyond the scope of the individual reviewer,” says Harter. The IRB’s operational procedures state that the IRB has an obligation to identify someone who can review a protocol if it is beyond the scope of the members’ expertise.
To ensure consistency, Harterʼs IRB invites the PI and their research team to attend the meeting whenever a protocol is going to a full committee. The IRB summarizes the protocol and asks the PI if there are any misunderstandings. “We ask the PI to explain anything that we did not understand: ‘This is our reading of it. What did we get wrong?’ It does slow down some of the pieces of the meeting, but we make a deep commitment to get it right,” says Harter.
Recently, the IRB was reviewing an investigator-initiated surgical study on pain medication used during surgery to determine if it reduces residual shoulder pain post-procedure. The IRB’s scientific reviewers noted that there was ample literature that already had addressed this question and those previous studies did not show the drug made a significant difference. The IRB raised this ethical concern with the PI and their co-investigator, asking, “Why would we allow you to do this with our patients to try to answer a research question that already seems pretty well-answered in the literature?” The PI asked for the opportunity to revise the protocol to explain how the clinical trial differed from previous studies. The IRB agreed to delay the vote and review the modified protocol.
Larger IRBs review much larger numbers of study protocols than smaller IRBs. This makes it challenging to invite PIs to explain the merits of their study in person. “We are a small enough IRB that [we] can take the time to do that with our PIs. But it is important, because everything that we are approving should be clear in the protocol itself. And if it’s not, then you need to go back and clarify,” says Harter.
Stacey Kusterbeck is an award-winning contributing author for Relias. She has more than 20 years of medical journalism experience and greatly enjoys keeping on top of constant changes in the healthcare field.
Reference
1. Shum C, Hey SP, Wilkes MS, et al. Flying blind: How thorough are IRBs when assessing scientific value? J Gen Intern Med. 2024; Dec 20. doi: 10.1007/s11606-024-09286-5. [Online ahead of print].
Institutional Review Boards inconsistently assess scientific merit, risking participant harm and ethical breaches. Lack of standardization, expertise gaps, and institutional conflicts undermine protections. Ethicists can advocate for rigorous review standards and transparent, participant-centered protocols.
Article Limit Reached
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content