U.S. News & World Report has established itself as a well known arbiter of measuring how colleges and universities stack up against each other. And now it’s in the process of developing a method for evaluating law schools based on their “scholarly impact” — how often faculty members publish scholarly articles and how many citations their work generates over a number of years.
Although U.S. News is jumping into this particular research field now, comparing law schools by how often their faculty publish and get cited isn’t new. Brian Leiter of the University of Chicago is known for launching this type of study of law professors. Gregory Sisk of the University of St. Thomas has carried on the study with a group of co-authors, and he estimated he’s been working on it for about 10 years.
Sisk and his colleagues use Westlaw to compile their research, which they use to update their rankings every three years.
“It’s appropriate to evaluate whether those law professors who are claiming to be doing scholarship and trying to make a difference in that regard are actually accomplishing something,” he said. “Is somebody really reading their work, or are they just writing articles that end up on a shelf somewhere that no one ever takes a look at?”
Sisk said the scholarly citations research isn’t trying to hold itself out as the single best way to measure the quality of law schools, or as the only way to measure scholarly impact.
“The idea was not to come up with a ranking that would be everything to everybody, which U.S. News tried to do, but rather speak to one aspect of legal education,” he said. “In contrast with U.S. News, which suggests their ranking is universal and pulls in everything that’s of value to evaluating a law school. … Our view is this is one part of the cathedral that should be taken a look at.”
According to initial reports, U.S. News will keep its scholarly impact ranking separate for its existing “Best Law Schools” ranking — at least at first. U.S. News announced it will partner with William S. Hein & Co. Inc to produce the research. It’s still unclear whether U.S. News will eventually integrate data on scholarly impact into the factors for its existing ranking.
Bruce Smith, dean of the University of Denver Sturm College of Law, spoke about the challenges of deciding who in a law school’s faculty should be included in the data collection on scholarly impact. Framing it by tenured and tenure-track faculty, he said as an example, risks under-reporting the impact of faculty members who publish often but aren’t tenured. It also can over-represent the impact on a law school’s overall score of tenured faculty, whose responsibilities are mostly teaching.
“Teaching faculty … really shouldn’t be measured along the lines of their scholarly productivity,” Smith said. “It’s not in their job description, it’s not what really they are assessed in light of. It’s going to be important, I think, for any tool to be fairly calibrated.”
Sisk said the traditional U.S. News ranking seems to reflect a school’s established reputation, while by comparison ranking based on scholarly impact seems to provide a current snapshot of schools’ work. Each set of scholarly impact data compiled by Sisk and his colleagues looks at the past five years.
“Most law professors rank another law school based on what they may have known about it 25 years ago and aren’t necessarily up to date on what’s happening with law schools,” he said of U.S. News’ peer-review structure. “As a consequence, as Brian puts it, reputation is very sticky.”
Sisk’s own University of St. Thomas School of Law seems to stand out as an example of how a school can stack up differently on U.S. News’ traditional list and in scholarly impact rankings. The school falls at 113 in U.S. News’ 2019 report, but it climbs by 90 spaces for scholarly impact to 23 in its 2018 ranking.
“I knew that we had a faculty that was producing and making an impact of top 25 caliber, but as a newer law school we weren’t getting that kind of attention,” Sisk said.
The University of Colorado Law School also has a significant difference between its U.S. News ranking and where it falls by scholarly impact. It falls at 46 on U.S. News’ 2019 report, while its 2018 scholarly impact ranking places the school at 29. The University of Denver’s law school rankings on the two lists are nearly identical, at 64 by scholarly impact and at 63 in the latest U.S. News ranking.
Sarah Krakoff, an associate dean for faculty affairs and research at CU Law, said while she’s confident in the quality of the school, she tends to be wary of school ranking systems in general because they measure what’s quantifiable, but that may not overlap totally with what the best measures of quality actually are. Job placement statistics are one important quantifiable measure, she said as an example. But by contrast, simply counting citations doesn’t reflect whether others are actually engaging with the work or just listing it in a string of citations. She noted her criticisms are her personal opinions rather than a reflection of CU Law’s views.
Krakoff said research on citations could also be expanded to look at the impact of published work outside academia, such as tracking how often it gets cited in judicial opinions or policy statements.
“If they were really serious about measuring scholarly impact instead of just counting things up and adding another ranking category for everyone to get all nervous about, they could do those other things,” she said.
— Julia Cardi