New research from the Institute for the Advancement of the American Legal System questions whether the current form of the bar exam satisfies what it claims to do — proving the minimum competency of attorneys to practice law.
Despite the exam’s century-plus existence, there’s never been an agreed-upon and evidence-based definition of minimum competence, according to the study. Without that definition, it’s impossible to know whether the bar is a valid measurement of competence, or “an artificial barrier to entry.” IAALS’s “Building a Better Bar: Capturing Minimum Competency” claims to take a step toward reshaping the lawyer licensing system using an empirically grounded definition of minimum competency.
Identifying Building Blocks
The study, a joint venture between IAALS and Ohio State University’s Moritz College of Law, used 50 research groups from around the country made of senior and junior attorneys to identify the skills necessary for practice and make recommendations for adapting the bar exam to better serve its purported purpose. The report suggests that minimum competency consists of 12 interlocking “building blocks” that allow new lawyers to practice effectively.
The study also recommends requiring law students to perform closely monitored externship work to help prepare them for the profession and claims that written exams aren’t well-suited to assessing minimum competence — and that multiple choice questions in particular should be used “sparingly.”
“This is a conversation that the entire legal profession needs to be engaging in,” said Logan Cornett, director of research at IAALS and researcher in the project.
While many in the courts and profession still believe the bar offers protection to the public from incompetent lawyers, “that belief has little concrete evidence to support it.”
Conventional views of minimum competence imagine a “bucket” of memorized legal rules and a few skills that new lawyers use to “scoop and serve.” This is a view the current bar reflects, according to the study. The current bar tests the ability to recall a large number of specific rules, their application, analysis of materials present in a case file and writing under tight time constraints.
However, the report suggests that minimum competency is more complex.
The “unequivocal” findings of the project showed that virtually no new lawyers rely on memory for anything, Cornett said. Competent lawyers take time and research everything before coming to a conclusion or deciding strategy.
“New lawyers in our focus groups did not base their first year of practice on a static set of rules and skills that they carried into the workplace,” the study states. Rarely did new lawyers rely on legal rules memorized from law school or for the bar — instead, they drew on more basic concepts and research skills to ID specific rules needed to represent clients effectively.
Instead, the report suggests professional competencies, like the ability to take individual responsibility and characteristics like trustworthiness, were most often considered immediately necessary when entering practice, Cornett said. However, legal skills were considered skills to be developed over time.
“This led us to question long-held assumptions about what minimum competence might entail,” she said. And while there’s long been skepticism about the efficacy and validity of the bar exam, “we were in desperate need of data to understand whether the skeptics were standing on firm ground.”
New lawyers also reported they lacked key knowledge surrounding the legal system, like the role of administrative agencies and alternative dispute resolution practices, according to the study.
“They also scrambled to acquire skills — such as interviewing, fact gathering, counseling, and negotiating — that were essential to competent practice,” the study states. “Preparing for the current bar exam gave them knowledge they did not need, while omitting knowledge and skills they did need.”
Challenging Assumptions
Despite the opinion of some law students, Cornett doesn’t believe the bar exam is purposely created as a hazing ritual. She said she thinks bar examiners want to protect the public and ensure examinees are minimally competent to be licensed.
“That said, we are overdue in examining our assumptions about the current licensing structure,” Cornett said. “And while it may not be a hazing ritual by intention, it has contributed to a serious lack of diversity in the profession because graduates of color perform disproportionately worse on the exam than white students.”
According to the bar examiners, themselves, the bar exam is designed to test competence.
“The current bar exam provides an independent, objective and consistent check on whether law graduates are competent to provide legal services,” said Valerie Hickman, a spokesperson for the National Conference of Bar Examiners.
The NCBE’s own Testing Task Force has been performing research of its own, offering suggestions such as increased emphasis on assessment of lawyering skills in the exam, and ensuring test content is aligned with what the NCBE has determined as minimum competency for entry-level practice.
Hickman added that IAALS’s findings complement the results of the task force’s practice analysis — particularly those listing the knowledge, skills and personal characteristics that serve as the foundation for minimum competence.
Cornett said that there is consistency between the task force’s findings and “Building a Better Bar” about what new lawyers need to practice competently. However, she said IAALS findings diverge because the NCBE does not appear to question whether an exam format is the best type of approach.
“We are questioning that long-held assumption,” she said.
Hickman said identifying the knowledge and skills relevant for legal practice is an important first step in defining minimal competence. However, the more complicated and critical step is establishing how much knowledge and skill is required in determining how to fairly and reliably test it.
The Better Bar Exam
The study notes that a uniform bar exam offers benefits for both lawyers and clients, such as moving easily through jurisdictions and across state lines — however, any uniform exam “must” account for new lawyers being more likely to apply the often individualistic laws of particular states.
Many new attorneys said local and state law played a more prominent role in their work than federal law, and nearly 50% of participants indicated they worked “exclusively or primarily” with local and state law.. However, contemporary bar exams focus on general principles, according to the study.
In addition, focus groups stressed that new lawyers should “never” rely on memorized principles from law school or the bar exam, and identified such an approach as “a bad way to practice law” or even as malpractice.
“No partner in a firm would ever say, ‘Just go off your memory, don’t consult any books, just say what you think you need to say and move on,’” the study quotes one participant. And supervisors concurred: “I often am reminding new lawyers that there’s a rule book that they should look at.”
Some aspects of minimum competency may not lend themselves to exam-based assessment, according to Cornett. For example, client interaction skills are likely best assessed using some format closely matching real-world interactions.
“Building a Better Bar” includes 12 recommendations surrounding the abilities and understandings necessary for new attorneys to practice law. The testing recommendations range from the ability to act professionally to interpreting legal material — as well as the ability to interact effectively with clients; identify legal issues; conduct research; communicate as a lawyer; manage workloads responsibly; cope with practice stress; pursue self-education and see the “big picture” of client matters.
Those who have these building blocks in their pocket are able to represent clients with little-to-no supervision, according to the study. Equally important, the lawyers possessing these blocks are able to continuously build on them, increasing their competency through their careers.
In order to assure that clients are adequately protected, the study argues lawyer licensing must ensure that new lawyers have these 12 blocks — and if they prove difficult to test in conventional exams, education requirements in workplaces, clinics, portfolios, simulations and other means could be used instead.
“A serious licensing system, one focused on protecting the public, cannot omit essential competencies simply because they are difficult to test,” the study states.
“We need to be open to experimenting — to trying new approaches to licensing — so that we can collect data to help us understand what really does work best,” Cornett said.
— Avery Martinez