Bad Evidence

In a fluorescent-lit ballroom atop Baltimore’s downtown convention center, Lt. Gen. John F. Sattler stepped off the stage and began to pace. He’d had 14 cups of coffee, he boomed, along with a Hershey’s bar with almonds. He was not about to stay still.

It was a snowy morning in late February. Sattler, a motivational speaker and retired Marine who once directed strategic planning for the U.S. Joint Chiefs of Staff, was the final panelist at the opening plenary of the 71st Scientific Meeting of the American Academy of Forensic Sciences. The annual conference brings together lawyers, scientists, and forensic practitioners from around the world to discuss the latest research and pressing issues in the field. The plenary sets the tone for the event — past speakers have included U.S. Deputy Attorneys General Sally Yates and Rod Rosenstein — while providing occasional entertainment. At the academy’s 2010 meeting in Seattle, the plenary featured a performer called the Physics Chanteuse, who sang provocatively about scientific validity.

A leadership consultant often serving corporate clients, Sattler had no specialized background in forensics. But he was determined to rev up the crowd. He gamely hit upon the theme of the 2019 meeting, emblazoned somewhat awkwardly on its program: Diligence (to the Effort), Dedication (to the Handling of Details), Devotion (to the Field). “If you took those three Ds and you looked at them every morning, you conducted your daily activities with those as your standard, I don’t think you could do wrong,” he said.

Sattler showed a slide picturing a group of armed Marines. This was the eve of the battle of Fallujah, he said solemnly, where 10,000 warriors came “to drain the swamp and get rid of the terrorists.” A year earlier, those same young men (they all happened to be men) were playing high school football and delivering newspapers, he said. Now they were bestowed with “the moral authority to take a life.” Such profound responsibility meant they had to be instilled with a clear ethical code — “those moral principles that will guide that decision-making.” Likewise, forensics practitioners must ensure their own work was beyond reproach. “In your profession, you live and die — you live and die reputational-wise at a minimum — by getting it right.”

Rapidly veering from military scenarios to their application to forensics, Sattler was a bit hard to follow. It was unclear whether anyone was weirded out by his invoking of the Iraq War, a deadly misadventure based on false intelligence, as a vehicle for illustrating ethical decision-making. “Truth matters,” he said, only to pivot to a slide of Lance Armstrong.

Nevertheless, some key themes eventually emerged. For one, as forensic scientists have learned all too well, one bad actor can tarnish an entire discipline. Not only must individuals strive to earn the highest level of trust from their peers and subordinates — a measure Sattler calls one’s “T-factor” — they should be poised to stand up to bad behavior. It is part of having a “bias for action,” Sattler explained — a readiness to move and to keep moving; to be constantly seeking improvement. Settling for the status quo is dangerous, even deadly, he warned. Outside every base in Iraq, “there was a giant sign above the gate as you went out into enemy territory. And it said: ‘Complacency kills.’ Complacency kills.”

As an alternative theme for the 2019 AAFS meeting, “complacency kills” might not have been half bad, at least if anyone had wished to inject some urgency into things. The Baltimore event began amid an ongoing crisis within forensic science that remained woefully unresolved. When we first wrote about the AAFS for The Intercept following its 2016 conference in Las Vegas, we encountered an embattled field facing rising public scrutiny over some of its most cherished and longstanding disciplines. Wrongful convictions rooted in junk science, crime labs embroiled in scandal, and a devastating revelation about the FBI’s hair microscopy division in 2015 had turned the image of forensics popularized by shows like “CSI: Crime Scene Investigation” on its head. The implications of the FBI scandal were particularly alarming. Hair analysts testifying on the stand had made erroneous statements in at least 33 death penalty cases, according to the agency. “Nine of these defendants have already been executed and five died of other causes while on death row.”

But whereas there was some reason in Vegas to feel optimistic about the prospects for reform — the theme that year was “Transformation: Embracing Change” — things have seemed to go backward since then. A report critiquing the scientific validity of certain forensic techniques, released by the President’s Council of Advisors on Science and Technology in the fall of 2016, met with aggressive pushback by the FBI and Barack Obama’s Department of Justice. Then there was the election of Donald Trump and the elevation of Jeff Sessions, who put a halt on a number of federal initiatives that were just starting to get underway.

Much of the recent upheaval in the forensics world can be traced back to a landmark study released by the National Academy of Sciences in 2009. Titled “Strengthening Forensic Science in the United States: A Path Forward,” the report questioned the scientific basis for virtually every forensic discipline used to convict people and send them to prison. With the exception of DNA analysis, it found, “no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.”

The NAS report was particularly damning for the so-called pattern-matching disciplines, in which an analyst examines a piece of evidence — say a bloody fingerprint found at a crime scene — and tries to match it to a sample belonging to a suspect. At AAFS, where forensic areas are divided into 11 different sections, many members of such fields responded with a mix of denial and defiance. While some practitioners took up the call issued by the NAS report — the fingerprint community, for example, has worked to develop objective comparison methods and determine error rates — others insisted the old ways of doing things were just fine.

In the intervening years, high-profile forensics scandals and a rising tally of exonerations have made it hard for even the most stubborn forensic experts to ignore the problem of junk science. At the 2017 AAFS meeting in New Orleans, a Virginia exoneree named Keith Harward, who spent 33 years in prison for rape and murder based on faulty bite-mark evidence, confronted the forensic dentists of the odontology section. He vowed to show up outside any courthouse where bite-mark evidence is used in the future. “I will contact the media. I will stand on the street corner in a Statue of Liberty outfit with a big sign saying, ‘This Is Crap.’”

Harry Edwards, a senior judge of the Court of Appeals for the District of Columbia, at the U.S. Court of Appeals Courthouse in Washington, D.C., on Sept. 17, 2012.

Nothing to See Here

As the AAFS meeting was getting underway in Baltimore, an article was published on the Innocence Project website, authored by the Honorable Harry Edwards, U.S. circuit judge in Washington, D.C. Titled “10 Year Anniversary of the Landmark Report on Forensic Evidence,” it reflected on the progress made since the release of the NAS report — and the work left to do. Edwards is uniquely qualified to take stock of this legacy. He co-authored the report and released it to the world.

Edwards recalled the reaction to the NAS report in 2009. Some disliked that it “challenged established practices,” he wrote, a criticism he regards as absurd. “When I explained the situation to my 8-year-old grandson, he asked me, ‘Grandpa, does something bad become good just because it has been followed for a long time?’ The answer is obvious.”

Edwards never set out to be an authority on forensics. It was mainly the esteem of his colleagues that led him to co-chair the committee, formed in 2005, that would produce the NAS report. “I had a reputation of — at least this is what I was told — really fostering collegiality and bringing folks of different views together,” he told The Intercept. “That was how I came to be selected. It was certainly not based on any of my experience in science. That was not my world.”

As the committee did its research and heard testimony from leading forensic experts, Edwards became alarmed. “I was flabbergasted when I listened to the person that was testifying about bite marks,” he recalled. “There were no studies of any consequence on validation, reliability, and I didn’t have to be a scientist to understand that what he was saying was fragile, at best.” If there was reason to expect some pushback from the members of such disciplines upon the release of the NAS report, other reactions took Edwards aback.

“The group that surprised me the most were prosecutors,” he said. “Not just at Department of Justice, but prosecutors generally. Because I would’ve assumed, in my naïve way, that they would’ve welcomed a report saying we need more and better research to validate these practices, and to make them better. Because that serves both prosecutors and defendants well. And we got just the opposite. We got a lot of pushback from prosecutors.”

“I think a number of them were worried that if you took the report seriously and started doubting some of what they had been doing, this would open cases that they thought were long gone,” Edwards continued. For many in law enforcement who relied on longstanding forensic techniques to solve crimes, the NAS report was seen as a threat to their work.

If the 10-year anniversary of the NAS report was a clear opportunity to grapple with its ongoing legacy at AAFS, no one seemed to have told the conference organizers in Baltimore. Instead, they invited to the mainstage a plenary speaker who embodied the kind of skepticism and pushback that so troubled Edwards. They invited Ted Hunt.

A veteran prosecutor from Kansas City, Missouri, Hunt was tapped by Attorney General Sessions to head up the Department of Justice’s Forensic Science Working Group — a decidedly opaque pseudo-successor to the very public work of the Obama-created National Commission on Forensic Science, which the Trump administration essentially disbanded. The NCFS included a variety of stakeholders — lawyers, judges, scientists, forensic practitioners, and law enforcement officials — tasked with finding ways to “enhance the practice and improve the reliability of forensic science.” Hunt was among those who served on the NCFS, where he clashed with his colleagues on a number of issues — including those designed specifically to improve the reliability of forensic science. “Ted Hunt,” one veteran conference attendee concluded, “is the Mike Pence of forensics.”

Hunt is neatly-coiffed and trim, with the demeanor of an overcast day. He approached the lectern with an impassive expression and for roughly 30 minutes ambled through a series of platitudes — “we can’t let the perfect become the enemy of the good” — inaccurately quoted Carl Sagan and Winston Churchill, and gave lip service to the need for forensic reform while cautioning that getting too wedded to scientific truth could be a problem — “the aspirational may be inspirational, but it’s not necessarily operational.” And he offered an example to highlight the power of good, old-fashioned, pre-NAS forensics and to excoriate detractors.

Back in 1997, someone broke into a convent in Kansas City through a second-story bathroom window and strangled and sodomized a nun before fleeing the premises. A fingerprint lifted from the window ledge matched a suspect named Jerry Owens; head hairs found in the nun’s room were also consistent with Owens. Significantly, Hunt noted, a pubic hair later found on Owens’s sock was consistent with the nun’s pubic hair; the theory was that Owens picked up the hair after taking his shoes off in the bathroom in order to quiet his steps as he crept through the convent. There was no DNA testing available at the time, Hunt said, “but with the evidence we did have, it didn’t matter.” Based on the pattern-matching evidence, Owens was convicted and sentenced to two life terms plus 60 years.

But Owens maintained his innocence, persuading a big law firm to take up his cause. The hairs were sent off for DNA testing. “And on that day, science had the final say: The verdict came in and he was a perfect match to each hair,” Hunt said. “Post-conviction testing affirmed Owens’s guilt and also confirmed the relevance and reliability of the fingerprints and hair evidence offered at trial.”

This kind of vindicating result happens all the time in the criminal justice system, Hunt said. “What is uncommon is the fact that you just heard about it.” He pulled out an analogy that is a favorite among those who feel that the concerns raised by the NAS and other critical reports are overblown. “How many times do we hear about the planes that land safely at the airport each day? We only hear about the crashes. And then we hear about the same crashes, over and over again, as if the exception is the rule,” he said. “In this field, the frequent focus on isolated failures has led to a carefully crafted, constantly reinforced, and patently false narrative that the forensic sky is falling. That’s simply not true. Forensic science is not failing, it’s flourishing.”

Of course, we don’t hear about planes making routine, safe landings, because that’s what they’re supposed to do. And to the extent we hear about the same failures over and over, it’s because we need to know why they happened so that we can make improvements to prevent future disasters. The same is true in the criminal justice system, where forensic errors can lead an innocent man to the death chamber. It is impossible to know just how many cases have been negatively impacted by faulty forensics. Nearly half of DNA exonerations to date are in cases tainted by forensic errors. Focusing on mistakes — reviewing them and then applying lessons learned to the broader system — is the point, but not one that Hunt appears to have much patience for.

Instead he blamed critics — undisguised jabs at the Innocence Project, outspoken individuals within the forensics community, and journalists — for the crisis of confidence within the field. “Much of it is … strategic, dishonest, and destructive. Some of it is little more than agenda-driven advocacy in the guise of promoting scientific purity — a genre I call ‘forensic science fiction,’” he said. Others promote “what I call ‘junk journalism’ — media stories full of partisan misinformation, strawman arguments, and half-truths about forensic science.”

While Hunt’s speech might’ve provided him a few satisfying zinger moments, it did little to reflect the sobering reality on the ground: Many forensic practices still lack meaningful scientific underpinning even though they are regularly used to prosecute individuals charged with crimes. The federal government has thrown what appears to be an impressive amount of money toward funding foundational research in forensics — more than $200 million since the NAS report was released — but that’s hardly enough to cover the amount of ground necessary. In 2014, as one conference presenter noted, the feds funded forensic research at roughly $21 million; that same year the Department of Defense spent more than $41 million on Viagra. And there remain questions — and contention — over what should be researched and to what degree.

As the AAFS meetings over the past few years have made clear, in many ways, the current story of forensics is one of opposing factions: those who believe that science in service of the law needs to be just as strongly supported as pure sciences, and those — including many in law enforcement — who largely believe that things are just fine and that forensic sciences have proven themselves worthy over time regardless of whether their flaws can be drilled down to statistical truths. The question that remains is whether and how these two entrenched sides might find a workable middle ground.

Fighting Over Biting

Of all the forensic disciplines and practitioners thrown into turmoil since the release of the NAS report, perhaps no group has melted down so publicly as the forensic dentists involved with bite-mark matching — the process of deciding that a patterned injury left on a victim was made by human dentition and then attempting to match the impression of that injury to the teeth of a suspect. The practice rests on a two-pronged foundation: First, that human dentition, like DNA, is unique; second, that skin is a suitable medium for recording this uniqueness. The problem is that neither premise has been proven true; in fact, scientific research conducted to date has suggested the opposite — and that bite-mark matching is an entirely subjective affair.

The NAS report, and an even more stinging critique of pattern-matching practices released by the President’s Council of Advisors on Science and Technology in 2016, were particularly critical of bite marks. The “available scientific evidence strongly suggests that examiners cannot consistently agree on whether an injury is a human bite mark and cannot identify the source of [a] bite mark with reasonable accuracy,” reads the PCAST report — a problem the group did not think could be rectified. “PCAST considers the prospects of developing bite-mark analysis into a scientifically valid method to be low. We advise against devoting significant resources to such efforts.”

Despite the conclusions of the NAS, the PCAST, and academic researchers in the field — notably, Drs. Mary and Peter Bush, whose findings have been damning — the forensic odontologists have not gone quietly into the night. Instead, for the majority of the last 10 years they have spent considerable energy avoiding serious research into bite-mark matching and have instead focused on attacking their detractors, including skeptics among their ranks. Inevitably, this vitriol has found a public stage during the annual AAFS conference. Lions of the discipline, including David Senn, a professor at the University of Texas at San Antonio, have aggressively pursued dubious ethics complaints against detractors in an effort to get them tossed out of AAFS. Manhattan prosecutor Melissa Mourges, a chief bite-mark apologist, has repeatedly lashed out at Mary Bush, including by making childish public comments about her appearance.

People like Senn and Mourges have consistently come to AAFS conferences with presentations meant to bolster their insistence that bite-mark comparison is a righteous endeavor. In Las Vegas, Mourges pulled out the safely-landing-airplanes analogy to make the case that just because bite-mark analysis isn’t perfect — indeed, bite-mark evidence, which is nowhere near as ubiquitous as say, fingerprints, has nonetheless been implicated in 31 wrongful convictions and faulty criminal indictments to date — that doesn’t mean it’s all bad. In Seattle in 2018, one dentist tried to elevate bite-mark evidence above the need for scientific scrutiny by claiming it was merely an “observational science” and thus not suited to empirical testing — “It’s like astronomy, folks! You just can’t control it.”

That same year, another dentist, Dr. Robert Dorion, opined that the focus on wrongful convictions was “fake news,” and suggested that there was a “moral, ethical, and legal obligation” to report on “rightful convictions.” In fact, he said, wrongful convictions connected to bite marks “had ceased.” When Chris Fabricant, director of strategic litigation at the Innocence Project and a relentless critic of bite-mark evidence, stood with a question challenging Dorion’s assertion (Fabricant was then working on two wrongful conviction cases connected to bite marks), the dentist said that his presentation time was up, but that he’d answer the question in 2019 (he didn’t). He then tootled off stage — before immediately returning to give a second presentation.

Despite the general intransigence — an insistence that all is fine in the face of plain evidence to the contrary — some key members of the community have broken ranks to embrace an inevitable truth: Junk science has no place in the criminal justice system.

At the 2015 conference in Orlando, two members of the community — Dr. Adam Freeman and Dr. Iain Pretty — presented the results of a study that were undeniably disastrous. Thirty-nine bite-mark analysts certified by the American Board of Forensic Odontology were asked to review 100 case studies and render an opinion on a most basic question: Is this injury a bite mark? The veteran practitioners came to unanimous agreement in just four cases. (Notably, the study did not consider whether the decisions were correct.) The following year the Texas Forensic Science Commission concluded that “there is no scientific basis for stating that a particular patterned injury can be associated to an individual’s dentition,” and recommended a moratorium on the use of bite-mark evidence.

Prompted by back-to-back blows to the credibility of the practice, a number of previous defenders of bite-mark comparison have begun to change their tune — including Dr. Frank Wright, who stood before his colleagues in Baltimore and offered a mea culpa. Wright, who participated in the Texas commission’s bite-mark case review process, noted that for years he had stood before his fellow odontologists to say that bite-mark evidence was valuable. “I stand before you now to say I was wrong.”

He said that there were members of the community who had “very strongly requested, if not demanded” that the results of the 2015 study not be made public and revealed that a subsequent study involving a smaller group of the most veteran examiners had yielded even worse results. Looking at the outcome, he said, “the light went off in my head and I said, ‘Do we really know what we’re doing?’”

Ultimately, Wright said that the entire field of bite-mark comparison had been built on decades of folly: “We learned it because patterns were shown to us and we were told they were bite marks. They went into our memory bank and that’s what we compare” casework to, he said. “A 40-year history of confirmation bias as the sole support for bite-mark evidence. It cannot go on.”

Wright implored the directors of the American Board of Forensic Odontology to issue their own moratorium on bite-mark comparison, but that did not happen. And there were other signs that not everyone is ready to lay down their dental molds — chief among them, Senn, a veteran leader of the pro-bite-mark tribe that members of the community often refer to as “Sennites.” Senn was among a handful of odontologists who orchestrated a 2013 ethics complaint against Dr. Mike Bowers, who has spent the last two decades ringing the alarm over bite-mark evidence. The effort to bounce Bowers from the AAFS ultimately failed. Undeterred, Senn was behind a second set of complaints filed last year aimed at Freeman and Pretty, who have been increasingly outspoken about the practice since 2015. Those complaints were dismissed at the start of the Baltimore conference, which apparently did not please Senn. According to multiple witnesses, he and Freeman got into a heated discussion in the bar of the Baltimore Hilton that culminated in Senn trying to climb over a railing to attack Freeman. It didn’t work; Senn got his leg caught mid-climb and had to back down. (Senn did not respond to emailed requests for comment.)

Surprisingly Little Progress

If the crackup within the odontology section exposed a struggle for the future of forensic dentistry, elsewhere in Baltimore things were comparatively drama-free. Many presentations were geared toward improving reliability and guarding against the cognitive bias and bad incentives that can skew the work of forensic practitioners. In criminalistics, a fire marshal from Connecticut warned fire investigators not to veer too far into policing when determining how a fire started. In engineering, a presentation covered how to handle it “when a client’s ‘pet theory’ conflicts with good forensic science practice.” The jurisprudence section wrestled with all aspects of wrongful convictions, a regular theme. And members across sections got updates on the Organization of Scientific Area Committees, an ambitious effort underway by the National Institute of Standards and Technology with the ultimate goal of developing standards “to ensure that a sufficient scientific basis exists for each discipline.”

All of this made the opening plenary that much more vexing. Among AAFS regulars, the reactions ranged from weary indifference to indignation. Some were less bothered by Hunt’s haughty sermon — some version of which they have come to expect from him — than by the speech that preceded it. It was delivered by Bonnie Armstrong, founder and president of an organization called the Shaken Baby Alliance.

Although it remains widely enshrined in medical literature, the diagnosis once known as Shaken Baby Syndrome is a cautionary tale. First coined in 1971 by a pediatric neurosurgeon who identified a “triad” of symptoms as proof that an infant was subjected to violent shaking, the concept has since been thoroughly debunked. Today SBS is an emblem of the kind of junk science that sends innocent people to prison; the National Registry of Exonerations lists 17 cases involving an SBS diagnosis — in 16 of those, it was ultimately determined that there was no crime at all. The most recent exoneration took place last year, in the case of Zavion Johnson, a Sacramento man accused of violently shaking his 4-month-old baby girl to death. Johnson was only 18 years old when he called 911 to report that his child was unresponsive. He spent 10 years in prison before prosecutors finally dismissed the charges against him in January 2018.

A former kindergarten teacher from Texas, Armstrong did not address the controversy over SBS, except as a brief aside about the founding of her advocacy group (“Oh how I wish we’d never used that name”). She began with a harrowing description of being raped by a stranger at a rest stop when she was 13 years old, then moved to the story behind the Shaken Baby Alliance. The organization was inspired by her adoptive daughter, Tiffany, who was brutally attacked by her biological father when she was a baby in 1994. Tiffany had been “shaken violently and beaten with a baseball bat,” Armstrong said. Doctors gave a diagnosis of Shaken Baby Syndrome.

Armstrong was certainly sympathetic — and her account deeply disturbing. But her speech raised red flags. For one, there was the rather salient question of how a finding of SBS might relate to a baby being beaten with a baseball bat. More importantly, given the well-documented problems with SBS, why would AAFS place it center stage at an event ostensibly devoted to science?

One attendee, Michael Risinger, a law professor and expert on forensic evidence, walked out during the speech. He was dismayed that there was no one to provide the necessary scientific balance or context. “If the point of this was not to rally the troops to … evaluate the scientific basis of these things with a sort of rational and truth-conducive neutral eye, then I don’t know what the point of it was,” he said. The presentation reminded Risinger of a past AAFS plenary featuring an “awful film about abused babies,” courtesy of the odontology section. Such things might be powerful calls to action — as Armstrong put it, “your work matters” — but they do nothing to contribute to sound science. This was especially disconcerting given the recent backsliding in reform efforts.

To be fair, several academy members pointed out that the plenary session is shaped by whoever is in charge year after year and not necessarily indicative of a larger shift in mission. John Lentini, a top fire scientist and longtime leader within the criminalistics section, said that in fact, AAFS has come a long way. Like forensic techniques themselves, which grew mostly out of the needs of law enforcement rather than the work of scientific researchers, the organization evolved from a prosecutorial mindset. “If you look at the history of AAFS, they’ve always been a police laboratory organization,” Lentini said. In the early 1970s, he said, the organization refused to give fellowship status to Paul Kirk, “one of the great forensic scientists of the 20th century,” because he worked for famed criminal defense attorney Lee Bailey. “The academy has certainly reformed because now the highest award in the criminalistics section is the Paul L. Kirk Award,” Lentini said. “He didn’t get the honor in his lifetime, but they came around to it.”

This long slow evolution of what is today the most prestigious forensics organization is a good reminder of just how much time and effort it will take to get forensics as a whole on firmer scientific footing. After all, Risinger points out, some experts were raising the problem of scientific validity back in the 1980s. “The question is, have we made progress in 30 years in attempting to shift things? The answer to that is, yes, but surprisingly little.”

President Donald Trump shakes the hand of Jeff Sessions after Sessions was sworn in as attorney general in the Oval Office of the White House on Feb. 9, 2017.

Hostility to Reform

The slow progress is not just because of dissenting ranks within forensics. In many ways, the problem has been exacerbated by politics. While Vermont Sen. Patrick Leahy, a former prosecutor, has referred to the revelations of the 2009 NAS report as “rather chilling” and twice introduced a bill to codify some of its recommendations (the bills went nowhere), others have been openly hostile to the mere suggestion that forensic sciences need any meaningful reform. Chief among the naysayers when the NAS report was first released was then-Sen. Jeff Sessions, also a former prosecutor. “I don’t think we should suggest that those proven scientific principles that we’ve been using for decades are somehow uncertain,” he said at a hearing in September 2009. Given Sessions’s stance — during the hearing he lamented that forensic examinations take so long that they hold up quick prosecutions — it is probably not so surprising that as attorney general he quickly put the kibosh on the forensics commission in favor of the mysterious working group now led by Ted Hunt.

But hostility to reform hasn’t been a purely partisan issue. The release of the PCAST report in 2016 was met with even greater misgiving than the NAS report — perhaps in part because the language of the latter was far more judicious, while the PCAST report was blunt: “Without appropriate estimates of accuracy, an examiner’s statement that two samples are similar — or even distinguishable — is scientifically meaningless: It has no probative value and considerable potential for prejudicial impact,” reads the report. “Nothing — not training, personal experience nor professional practices — can substitute for adequate empirical demonstration of accuracy.”

The pushback was swift — including from Obama’s own attorney general, Loretta Lynch, who flatly rejected the report’s recommendations for shoring up forensic practices. In a statement to the Wall Street Journal, Lynch said that her agency remained “confident that, when used properly, forensic science evidence helps juries identify the guilty and clear the innocent,” and that the “current legal standards regarding the admissibility of forensic evidence are based on sound science and legal reasoning.” The FBI also balked, as did a number of other law enforcement groups, including the National District Attorneys Association, which was particularly harsh. In a statement, the group accused the PCAST of “pervasive bias” and wrote that the report “conveniently overlooks the ancient debate over precisely what constitutes ‘science.’”

Minnesota Judge Pamela King was working as a public defender when the NAS report came out during the 2009 AAFS conference in Denver. She had just become a member of the group and says that at the time she really didn’t appreciate how important it was, though its release caused quite a stir. The reaction was “very, very mixed,” she recalled. “I think there were some academy members who were upset, felt threatened — really felt like they had committed their careers to doing good work and held themselves to a high degree of professionalism and they really felt that the perspectives that were being offered” by the NAS report “were attacking that.” Still others, she said, “were delighted and were really excited” by what the report said and were “hopeful that that would improve the general way that forensic science is utilized and done.”

There’s been a fair amount of change since the report’s release, and the National Commission on Forensic Science, of which King was a member, made a number of recommendations that were adopted — including a ban on practitioners using the phrase “reasonable degree of scientific certainty” when testifying about their confidence in the matches they’ve made. The terminology has no meaning outside the courtroom and yet suggests a strong scientific foundation that cannot be said about most forensic practices. Yet a number of promised reforms have not materialized.

At the 2016 conference in Las Vegas, then-Deputy Attorney General and NCFS co-chair Sally Yates announced that the DOJ would be conducting a “stress test” on various disciplines performed in the FBI lab — not because there were any particular concerns, she was quick to say, but as a means of ensuring “the public’s ongoing confidence in the work we do.” The decision seemed a prudent one, given the alarming results of a joint review of thousands of FBI hair analysis cases, which revealed that FBI analysts had overstated their conclusions 95 percent of the time. “This doesn’t necessarily mean that there were problems with the underlying science,” Yates explained to the plenary audience. “It means that the probative value of the scientific evidence wasn’t always properly communicated to juries.”

Seven months later, Lynch announced that she was adopting the NCFS recommendations on testimonial language, but that wider stress test simply never happened. “We were doing great until the Trump administration came about and … the things that were happening at the federal level, the policy stuff and so on, that came to a screeching halt,” said Alicia Carriquiry, a professor of statistics at Iowa State University and director of the Center for Statistics and Applications in Forensic Evidence.

Carriquiry says that conversations happening among stakeholders and the DOJ just fizzled out after Trump was elected and Sessions was brought on board. And the NCFS wasn’t the only advisory panel to get the hook. While Trump continued the PCAST on paper, he hasn’t appointed any members to it. In December 2018, the DOJ disbanded its Science Advisory Board, which provided input to the department on what types of research — including in forensics — it should fund.

Even gains that had been made in curbing unsupportable testimony have since weakened, she said. “When the DOJ was under the Obama administration there were conversations. In fact, I participated in many of those where we would sit with the DOJ people and think about the type of language that should come out of crime labs and what type of reviews we should be doing of the disciplines,” she said. “The second Sessions came on board those initiatives were killed dead.”

It wasn’t just that things stopped. “Worse, you know. They’re trying to walk back many of the things we were making progress on,” she said. “For example, we had come to some sort of an agreement about the fact that we were going to do a broad review of the disciplines.” But when the Trump administration took over the attitude was, “‘What review?’ So that was completely done. Then we had come to an agreement on the type of language that should be used in reporting and testimony. That was squashed and the language went back to exactly what it used to be.”

Resisting Science

For all the high-stakes political shifts that have stymied reform efforts, the basic problem remains: Some in the forensic science community simply resist the NAS report and what it stands for.

At the 2015 academy meeting in Orlando, the opening plenary hadn’t even begun when a controversy broke out over the report. At issue was a question included in the Academy Cup, an annual contest in which members of each section form teams to answer forensics-related trivia. (Question: Who is the anthropology section member behind the hit TV show “Bones”? Answer: Kathy Reichs.) But one true-false question was tripping people up: Is the NAS report authoritative enough to be accepted as evidence in court?

The answer was — and is — yes, says retired New Mexico appeals court Judge Roderick Kennedy, who penned the question. But not everyone agreed, so the emcee polled the group: Who says it’s true? No hands went up among the roughly 75 people assembled. False? A sea of hands shot up — “False!” several called out in unison.

Kennedy is not surprised at their reaction — many forensic practitioners don’t necessarily understand the rules of court, he says. The NAS report has been admitted as evidence (though perhaps not as often as some defense attorneys would like) and cited by jurists. In December 2018 the notoriously rigid Texas Court of Criminal Appeals gave a firm nod to the report in an exhaustive opinion wherein the court concluded that Steven Mark Chaney had been wrongfully convicted based on bad bite-mark evidence.

The claim that the NAS report was not meant to be authoritative is as old as the study itself. Edwards, the federal judge and coauthor of the report, recalls some prosecutors authoring briefs stating that it “was not intended to have any impact on court proceedings. That’s silly, of course.” But some have gone even further. In one egregious case, Mourges, the New York prosecutor and defender of bite-mark analysis, brazenly manipulated the language of the NAS report in a 2014 brief seeking to admit bite-mark evidence in a murder trial — transforming its warnings over bite-mark analysis into an endorsement. After the journalist who exposed her linguistic sleight of hand was invited to give remarks at a luncheon held by the jurisprudence section in Baltimore — a speech in which he discussed the deceptive brief — Mourges came to confront him in a room full of lawyers and judges, demanding he “cut the shit.”

People like Mourges and Hunt may wish they could censor the media that tarnishes the image of forensic evidence. But just as pep talks from motivational speakers do nothing to address the pressing challenges in the forensics world, silencing journalists won’t make the problems go away. To Edwards, who remains proud of the NAS report and its influence, the real problem is that 10 years later, “we’re not where we ought to be.”

Edwards is dismayed at the shuttering of the NCFS, which “got killed just as it was beginning to get going.” But for all its promise, one fundamental problem with the commission was that it was housed at the DOJ, which he called a mistake. One of his “great disappointments,” he said, is the failure to fulfill one of the principal recommendations of the NAS report: a “national group that was independent, separate from law enforcement, that oversees forensic science.” This group would have a budget “to fund serious research,” assess validation and reliability studies, and help to set standards. “That hasn’t happened.”

Edwards has not given up hope that this might be possible in the future. “If we had one, I think it would work. And I think people would rally behind it. But you’ve gotta have the political will to set it up. … Do we have the political will? I don’t know.”