In a move that has sparked fresh controversy, the California State Bar has admitted to using artificial intelligence (AI) to help develop multiple-choice questions for its February 2025 bar exam. This revelation comes after months of complaints from test takers who were already critical of technical issues, irregularities, and errors in the exam.

The news has caused a new furor, with critics claiming that using AI, along with recycled questions, represents a significant deviation from standard practices in bar exam preparation. Notably, AI-generated questions were crafted by non-lawyers, raising concerns about the integrity of the exam process. While the State Bar has attempted to downplay the significance of the issue, it has nonetheless triggered widespread debate and scrutiny.

AI’s Role in the February 2025 Bar Exam – A Staggering Admission

The State Bar of California’s acknowledgment that AI was used in the creation of some of the exam’s multiple-choice questions has left legal experts and prospective lawyers stunned. Mary Basick, Assistant Dean of Academic Skills at UC Irvine Law School, voiced her disbelief, calling the development “unbelievable” and “utterly botched.”

Basick, who had initially defended the integrity of the exam, had to concede that some questions did indeed have an “AI-like” quality. “I defended the bar,” Basick said. “No way! They wouldn’t do that!” But after taking a closer look at the questions, she recognized the telltale signs of AI involvement.

Critics argue that allowing non-legally trained psychometricians to use AI for exam question development presents an “obvious conflict of interest.” These same psychometricians, who developed the questions, were also responsible for assessing their validity and reliability—an arrangement that, according to experts, raises significant ethical concerns.

The State Bar’s Explanation: AI as a Cost-Saving Measure

The California State Bar has defended its decision, stating that the use of AI was meant to innovate and improve the exam’s cost-effectiveness. According to Leah Wilson, the State Bar’s Executive Director, the organization “has confidence in the validity of the multiple-choice questions to accurately and fairly assess the legal competence of test-takers.” Wilson emphasized that all questions, including those developed with AI assistance, were reviewed by content validation panels and legal experts to ensure accuracy and fairness.

Yet, the disclosure has not placated many in the academic community. Katie Moran, an associate professor at the University of San Francisco School of Law, expressed concern over the conflict of interest, stating, “The State Bar has admitted they employed a company to have a non-lawyer use AI to draft questions that were given on the actual bar exam.”

Moran’s worries align with those voiced by Basick, who argued that AI-created questions could not possibly measure the nuanced understanding needed for legal practice. “It’s not just, ‘Hey, do you know this rule?’ It’s ‘Do you know how to apply it in a situation where there’s ambiguity, and determine the correct course of action?’” she said.

The Aftermath: Fallout and Calls for Transparency

The response to the State Bar’s use of AI has been swift and severe. Affected test takers have expressed frustration over the perceived lack of transparency and accountability. Many are now calling for the full release of the exam questions to allow for proper scrutiny and to ensure that future bar candidates are better prepared.

While some have called for the return of the traditional, National Conference of Bar Examiners (NCBE)-designed exam, others have pointed out the irony of AI being employed to craft questions that were allegedly sourced from previous law school exams. This was highlighted by Basick, who emphasized that the recycled questions from a first-year law exam “represented a major change” in the bar exam’s preparation process. In her view, such changes should have required a two-year notice under California’s Business and Professions Code.

In a further blow to the State Bar’s credibility, Basick and Moran raised concerns about the speed with which the Kaplan Exam Services questions were drafted, noting that many of the released practice questions contained numerous errors. These errors further eroded confidence in the exam’s quality and fairness, particularly when coupled with the decision to introduce AI into the equation.

Legal and Political Fallout: A Call for Action

As the debate continues, the California legislature has taken notice. Senate Judiciary Chair Thomas J. Umberg (D-Santa Ana) has called for an audit of the State Bar, and the California Supreme Court has directed the agency to return to in-person administration of the bar exam in July. This move was prompted by the technical issues experienced by test takers during the remote exam and further amplified by the AI question revelation.

Despite these calls for reform, Alex Chan, the chair of the State Bar’s Committee of Bar Examiners, insists that the issue lies with the performance of the exam vendor, Meazure Learning, rather than the exam’s content. Chan noted that the State Bar had been directed to consider the use of AI by the California Supreme Court in October 2024, suggesting that this move was part of a broader strategy to innovate the testing process.

Still, many are skeptical of this rationale, pointing to the serious flaws in the February 2025 exam. “They have just shown that they cannot make a fair test,” said Moran, urging the State Bar to revert to the NCBE exam for the July sessions.

Can California’s Bar Exam Recover from the Fallout?

As the dust settles, the future of California’s bar exam remains uncertain. The controversy surrounding AI-generated questions, coupled with technical glitches and questions about fairness, have left many wondering if the State Bar will be able to regain trust.

With the California Supreme Court meeting soon to discuss potential remedies and adjustments, all eyes will be on whether the State Bar will take any meaningful steps to address the concerns raised by law professors, test takers, and lawmakers alike. One thing is clear: the February 2025 bar exam has brought unprecedented challenges to the forefront, and the legal community is demanding accountability and transparency.

The Impact of AI on the Legal Profession: A Turning Point for Bar Exams?

The use of artificial intelligence in bar exam question creation marks a turning point in the legal profession’s relationship with technology. While AI has proven its usefulness in many fields, the California State Bar’s experiment with AI-driven exam questions raises significant questions about the future of legal education and testing. As California’s bar exam continues to evolve, it will be interesting to see if other states follow suit or if this experiment proves to be a cautionary tale.