Potshots: “Blank Stares” and “No Data” on Latent Fingerprint Identification

According to as astute an observer as David Faigman,

In ... fields such as latent fingerprint identification, firearms, clinical psychology, and clinical psychiatry ... , if judges ask the question, “Where are the data?” they would be met with blank stares. If you ask a latent fingerprint examiner, “Where are your data?” the answer is likely to be, “Data. We have no data. In fact, we don't need data. We're specialists.” ... Many of these experts have been practicing their trade for twenty-five years; they know it when they see it. ... Under Daubert, however, even if your data happen to be experience, you have to be able to articulate how you came to know what you think you know. (Faigman 2013, 914).

A footnote explains that being “able to articulate how you came to know what you think you know” can be accomplished by “checking the basis for believing that the experience will produce reliable testimony.” (Ibid., 914 n. 64).

Now, I am no fan of claims of fingerprint examiners to be able to match latent prints to reference prints with absolute certainty (NIST 2012) and of lax and superficial court opinions allowing such testimony. (Kaye 2013; Kaye, Bernstein and Mnookin 2011). But the assertion that there are absolutely no data to show that latent print examiners can “produce reliable testimony” is too much for me to swallow. Indeed, in his treatise on scientific evidence, Professor Faigman does not insist that “no data” exist. The treatise correctly recognizes that “[a] few well-designed studies have now been conducted” (Faigman et al. 2012, § 33:56). To the list of six studies noted in the treatise (ibid., § 33.49 n. 10), one can add Tangen, Thompson, and McCarthy (2011). (As explained here last June, this Australian study showed a false negative rate of under 8% and a false positive rate of under 1% (Fingerprinting Error Rates Down Under, June 24, 2012)).

Perhaps Professor Faigman meant to say that even if data exist to support the judgments of fingerprint analysts—as they clearly do at a general level—a particular examiner’s judgments are not based on data, but on standardless, subjective impressions of the degree of similarity that warrants an identification or an exclusion. They just “know it when they see it.” That observation is closer to the mark (no pun intended). It is the gist of David Harris's contention that "most forensic science does not qualify as science in any true sense of that term." (Harris 2012, 36). Like Professor Faigman, Professor Harris complains that "[d]isciplines like fingerprint analysis, firearms tool-mark analysis, and bite-mark analysis have no basis in statistics, and do not originate in inquiry conducted according to scientific principles. Rather, they rely on human judgment grounded in experience ... without reference to rigorous and agreed-upon standards." (Ibid.) Identification experts who do not follow a protocol with quantitative or other external standards to achieve high inter-rater reliability should not insist that they are following the "scientific method." (Compare Kaye 2012, 123).

But "science" is not the only source of useful information, and experiments can measure the levels of accuracy for subjective as well as objective procedures. DNA laboratories have verified that DNA analysis performed in a specified way correctly distinguishes between samples taken from the same source and samples taken from different sources.  Indeed, this is the only sense in which it could be said that “DNA profiling [always] ... had known error rates” (Faigman 2013, 913). Even today, the error rates of DNA laboratories in actual case work is not really known. In the same manner, tests of fingerprint analyses performed by trained examiners show that they are capable of routinely distinguishing between marks taken from the same source and marks taken from different sources (with some errors). Again, however, we do not know the error rates of these examiners in actual case work. (Kaye 2012).

Consequently, appropriately documented latent print comparisons undertaken without unnecessary exposure to biasing information, presented with a recognition of the uncertainty in the largely subjective procedure and verified by an independent examiner blinded to the initial outcome as well as the output of an automated scoring system, should survive the “more rigorous test” (Faigman 2013) established in Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993), and embellished in later cases. Although there is ample room to improve fingerprint comparisons by human examiners and to implement automated systems for latent print work, “blank stares” and “no data” are no longer the only answers available to an objection under Daubert.

References

Faigman, David L. 2013. “The Daubert Revolution and the Birth of Modernity: Managing Scientific Evidence in the Age of Science.” University of California at Davis Law Review 46:893–930.

Harris, David A. 2012. Failed Evidence: Why Law Enforcement Resists Science. New York and London: New York University Press.

Kaye, David H. 2013. “Experimental and Scientific Evidence: Criminalistics.” In McCormick on Evidence, edited by Kenneth Broun, § 207. Eagan, MN: West Publishing Co.

Kaye, David H., ed. 2012. Expert Working Group on Human Factors in Latent Print Analysis, Latent Print Examination and Human Factors: Improving the Practice Through a Systems Approach. Gaithersburg: National Institute of Standards and Technology.

Kaye, David H., David E. Bernstein, and Jennifer L. Mnookin, 2011. The New Wigmore: A Treatise on Evidence: Expert Evidence. New York: Aspen Publishing Company, 2d ed.

Tangen, Jason M., Matthew B. Thompson, and Duncan J. McCarthy 2011. “Identifying Fingerprint Expertise.” Psychological Science 22:995 (available online).

Are Two Heads Better than One? Mathematics and the Amanda Knox Case

In yesterday’s New York Times, “Leila Schneps, a mathematician and mystery writer, and her daughter Coralie Colmez” wrote an op-ed entitled “Justice Flunks Math.” They gave this failing grade to an Italian judge who declined to order a second DNA test in the notorious murder case against the American student, Amanda Knox and her Italian boyfriend, Raffaele Sollecito.

If only the judge understood probability theory, they suggest, he might have ruled in favor of the prosecution's request for another DNA test. In their words:

Whatever concerns the judge might have had regarding the reliability of DNA tests, he demonstrated a clear mathematical fallacy: assuming that repeating the test could tell us nothing about the reliability of the original results. In fact, doing a test twice and obtaining the same result would tell us something about the likely accuracy of the first result. Getting the same result after a third test would give yet more credence to the original finding.

This claim of a mathematical fallacy in the judge’s reasoning, however, rests on an important assumption — that the test results are statistically independent (at least in substantial part). To illustrate their point, Schneps and Colmez explain:

Imagine, for example, that you toss a coin and it lands on heads 8 or 9 times out of 10. You might suspect that the coin is biased. Now, suppose you then toss it another 10 times and again get 8 or 9 heads. Wouldn’t that add a lot to your conviction that something’s wrong with the coin? It should.

But consider this example instead:

You test the surface of a gold coin and test it to make sure it is gold. The test is 90% certain to indicate gold when the metal is gold, so following the advice of Schneps and Colmez, you repeat the test three times and strike gold each time. When you try to sell the coin, however, a more astute buyer weighs it and finds that it does not have the density of gold. It is, in fact a thickly gold plated, lead coin.

Which example is more apt in the Knox case? A Nature blog explained that "[v]ery small amounts of Knox’s DNA were found on a knife located at the crime scene 46 days after Kercher’s murder," but this is not correct. The knife came from Sollecito's kitchen, and a more informed account in the New York Times states that "the court-appointed experts concluded that ... Ms. Knox's DNA was in fact on the handle" -- hardly a surprise given that she may have used it to cook dinner in her boyfriend's apartment. The only thing that made the knife incriminating was a police laboratory finding of DNA from the victim, Meredith Kercher, on the blade. But there was evidence that the knife could not have produced all the wounds, and the court-appointed experts were skeptical of the finding about the blade as well the police laboratory's analysis of "a bra clasp that belonged to the victim found on the floor at the scene 46 days after her murder" said to show Sollecito’s DNA.

Returning to Nature's account:

Speaking at the request of the defence, two forensic scientists, Stefano Conti and Carla Vecchiotti from Sapienza University in Rome, said police investigators failed to follow international protocols for collecting and handling evidence and conducting tests on small genetic samples known as low-copy-number (LCN) DNA analysis. For example, officers were not wearing protective masks or hair caps at the crime scene. ... In addition, Conti said police often used plastic bags, rather than paper, to wrap evidence, heightening the risk of contamination. ... “There are various circumstances do not adhere to protocols and procedures,” Conti told the court.

Consequently, the independent experts concluded that they could not rule out the possibility that the knife and bra had been contaminated by other sources of Knox’s and Sollecito’s DNA, such as other evidence at the crime lab where forensic testing was taking place.

If these experts’ concern — that the original DNA test was simply detecting traces of Kercher's and Sollecito’s DNA that investigators inadvertently transferred to the knife and bra clasp, respectively — then repeating the tests could well continue to detect that DNA — and prove nothing more than the original tests did. If the bra clasp sample showed a mixture of DNA from the victim and Sollecito (and nothing else), for example, then repeating it over and over would not reinforce the prosecution case in the slightest. It would be no different than retesting the surface of the lead coin with its gold plated contamination. The inability of the DNA evidence to demonstrate a convincing link to the defendants would remain after even an infinity of new tests.

Consequently, it is hard to judge Schneps' and Colmez's suggestion that "[t]he judge’s rejection of the retest — at least based on the notion that a confirming retest could tell us nothing about the likelihood that the DNA was a match — was a serious error, one that scuppered an opportunity to get at the truth of Ms. Kercher’s murder."

The judge’s decision may have been mathematically sound, or it may have been as naive and fallacious as Schneps and Colmez propose. They have a nice theory but it is fair to assume that there were uncontaminated samples for new testing? Without some specification of precisely what made the initial testing problematic and whether those problems could be reduced sufficiently with retesting, it seems precipitous to convict the judge who overturned the guilty verdict of "bad math."

Indeed, Schneps and Colmez seem to believe that the judge ignorantly opposed retesting of the small sample of DNA on the blade despite an improvement in the technology of testing low template DNA. They wrote that

Even though the identification of the DNA sample with Ms. Kercher seemed clear, there was too little genetic material to obtain a fully reliable result — at least back in 2007. By the time Ms. Knox’s appeal was decided in 2011, however, techniques had advanced sufficiently to make a retest of the knife possible, and the prosecution asked the judge to have one done. But he refused.

Yet, the judge clearly was open to new methods. He asked the two university experts to ascertain "whether it is possible, by means of a new technical analysis, to identify the DNA present on items 165b (bra clasp) and 36 (knife)." The Conti-Vecchiotti Report, Assignment. Finding "no evidence of cellular material in the samples analyzed," however, his experts concluded that "no DNA suitable for further laboratory investigations (amplification, electrophoresis) was present either on the swabs [tamponature] (A-B-C-D-E-F-G-H-I) taken from Exhibit 36 (knife) or on those (L-M) taken from Exhibit 165B (hooks of the bra)." Id. Conclusions (1). They based this conclusion on the absence of cellular material and the failure of "quantification of the extracts ... conducted via Real Time PCR [to] reveal the presence of DNA." Id. Conclusions (2).

The prosecution disagreed. It asked for still more testing. But Judge Claudio Pratillo Hellmann had had enough. Rather than prolong the appeal -- and the imprisonment of the defendants -- still longer to await tests that his experts told him would be useless, he and the jurors returned their not-guilty verdict. It remains to be seen why this verdict now has been overturned, but it is hardly obvious that "bad judicial math" is the reason.

References

• Natasha Gilbert, Knox Walks Free Due to Unreliable DNA Analysis, Nature News Blog,Oct. 4, 2011
• Elisabetta Povoledo, Italian Experts Question Evidence in Knox Case, N.Y. Times, June 30, 2011, at A6
• Leila Schneps & Coralie Colmez, Justice Flunks Math, N.Y. Times, March 27, 2013, at A23
• The Conti-Vecchiotti Report (translation), Welcome, July 4, 2011.

Later Postings

• http://for-sci-law-now.blogspot.com/2013/05/bad-math-or-passable-law-dna-testing-in.html

In Press: Confronting Science: Expert Evidence and the Confrontation Clause

The coming volume of the University of Chicago's Supreme Court Review will include an assessment of the Supreme Court's trilogy of opinions on the Confrontation Clause and laboratory reports: Jennifer L. Mnookin and David H. Kaye, Confronting Science: Expert Evidence and the Confrontation Clause, preprint available at http://ssrn.com/abstract=2238269.

Abstract: In Crawford v Washington, the Supreme Court substantially changed its understanding of how the Confrontation Clause applies to hearsay evidence. Since then, the Court has issued three bitterly contested expert-evidence-related Confrontation Clause decisions, and each one has generated at least as many questions as answers. This article analyzes this trilogy of cases, especially the most recent, Williams v Illinois.

In Williams, the Court issued a bewildering array of opinions in which majority support for admitting the opinion of a DNA analyst about tests that she did not perform was awkwardly knitted together out of several incompatible doctrinal bases. The most prominent and fully developed argument for admission was that the references to the work of the analysts who actually did the testing but who never testified were admitted for a purpose other than their truth. Although we maintain that this argument is, on the facts of Williams, implausible, we also recognize that in other, relatively limited instances, expert basis evidence might legitimately be introduced for a purpose other than its truth.

After striving for precision on this doctrinal point, we step back and suggest that the ongoing anxiety about how to think about expert evidence and the Confrontation Clause exists in large part because the Court has yet to face directly a set of larger, background concerns. There is significant uncertainty about how, and to what extent, scientific evidence should be treated as special or distinct from other kinds of evidence for confrontation purposes. We suggest that scientific and expert evidence might warrant some limited special treatment, based on what we see as one of the most critical dimensions of scientific knowledge production—that it is a collective, rather than an individual enterprise. Recognizing that scientists inevitably rely and build on facts, data, opinions, and test results of others, we suggest that courts should engage in a modest form of scientific exceptionalism within Confrontation Clause jurisprudence, through efforts to create procedures that respect the fundamental values of the Confrontation Clause, but also adapt when necessary, to the epistemic structures and processes of science.

In Press: The Database and the Dictionary

An improved (I hope!) version of an earlier post on the Ohio Supreme Court's opinion in State v. Emerson will be published in Jurimetrics: The Journal of Law, Science, and Technology (53:4). The current draft is available at http://ssrn.com/abstract=2238400.

Abstract: In State v. Emerson, 981 N.E.2d 787 (Ohio 2012), the Ohio Supreme Court discerned neither a statutory nor a constitutional barrier to retaining an acquitted defendant’s DNA profile in the state’s database when the profile was obtained with a search warrant and the defendant did not request expungement of the database record. The court wrote that the profile fell with the statutory category of “records from forensic casework or from crime scenes” and that “a person does not have standing to object to the retention of his or her DNA profile or to the profile's use in a subsequent criminal investigation.” This essay disputes both these conclusions. It argues that placing or retaining the profile of an acquitted individual, even one obtained with a warrant, in a database for convicted offenders violates the statute and that the affected individual has standing to complain. It concludes, however, that the entry and use of the profile in the database, although not statutorily authorized, does not contravene the Fourth Amendment.

Disgusting DNA

The Whooper Stopper

When my son worked at a pizza restaurant a long time ago, he told me that some of the kids preparing the food would spit into into their handiwork. I tried not to think about it, but the practice, at a Burger King in Vancouver, Washington, has prompted a major decision on the tort of intentional infliction of emotional distress.

The employee with the loose lips probably did not expect the Whopper with cheese to go directly to the police, but Clark County Deputy Sheriff Edward Bylsma evidently had a well honed sense for the suspicious. After receiving his burger from a drivethrough window, he stopped in a parking lot, removed the top bun, and uncovered a “slimy, clear and white phlegm glob” on the meat.

According to the Huffington Post, "Deputy Bylsma felt ill all day; he claims that he even vomited on account of his emotional distress. He sent the burger away for DNA testing to try and trace the spit back to an individual person. When the results linked the loogie to Burger King employee Gary Herb, Bylsma moved to sue Burger King. He said that he became unable to eat food from restaurants and lived in fear of contracting a foodborne illness."(How the deputy acquired a reference sample from Mr. Herb is not apparent from news accounts.)

A federal judge dismissed the 2009 case against Burger King for negligence, product liability and vicarious liability because the Washington Product Liability Act [WPLA] does not provide for damages for emotional distress in the absence of physical injury. Or so it thought. The Ninth Circuit asked the Washington Supreme Court to rule on the scope of its tort law, and last month, the state court wrote that "[t]he courts of this state recognize damages for ... emotional distress, and thus, such damages, if proved, are recoverable under the WPLA." It added that "[t]he WPLA permits relief for emotional distress damages, in the absence of physical injury, caused to the direct purchaser by being served and touching, but not consuming, a contaminated food product, if the emotional distress is a reasonable response and manifest by objective symptomatology."
.
Of course, whether the deputy will recover at trial (if it comes to that) remains to be seen. Will Burger King argue that the deputy, whose job may require him to view grisly scenes of human violence and carnage, is overstating his symptoms?

Moving Violations

In England, the national DNA database has been employed to curb spitting on transit workers. As part of Operation Gobstopper, bus drivers in northwest London received 2,500 evidence collection kits. Tube staff in central London and train wardens in Scotland were similarly equipped, the BBC reported in 2004.

The kits helped prosecute about 100 people on the Tube in 2007. Reportedly, 7 out of 10 samples yielded a match to DNA profiles in the national database of criminal offenders and arrestees.

Update of Mar. 25, 2013: The Whopper case, Bylsma v. Burger King Corp., 293 P.3d 1168 (Wash. 2013), did not involve a database search. Only two employees, an Herb and a McDonald, had been working in the restaurant.When police visited the restaurant that night and asked for saliva samples, the two refused. The state crime laboratory determined that the saliva on the burger was of human origin. The police returned with search warrants, and the laboratory named Herb as the source of the spit. Herb pleaded guilty to felony assault (although the spit was not initially visible and only touched Bylsma because he placed his finger in it to test its consistency) and was sentenced to 90 days in jail. (Thanks to Jeremy Gans, whose comment provided a link to the civil complaint setting forth these facts).

Hot Off the Presses: Beyond Uniqueness

The Birthday Problem, a classic puzzler of probability theory, has a counter-intuitive answer. Statisticians on the National Academy of Sciences' Committee on Forensic DNA Evidence: An Update, used the problem to distinguish two arguments that might support the conclusion that DNA types are unique in a population. Years later, law and business school professor Jay Koehler used the same problem to show how difficult it is to demonstrate total uniqueness -- and what other kind is there? -- of toolmarks on the basis of the collective experience of toolmark examiners. A report of a NIST (National Institute of Standards and Technology) expert working group on latent fingerprint analysis also invoked the problem to explain why even a tiny random match probability does not establish that fingerprint images are discernibly unique. Nevertheless the Birthday Problem cannot refute a claim that examiners making source attributions from almost unique features will almost never be wrong. (Of course, this observation does not mean that a claim of almost no errors in case work would be true.)

The relationship between a small number of duplicates and individualization in the sense of source attribution was the subject of a paper presented at the Eighth International Conference on Forensic Statistics in 2011. An updated version of the paper appeared this month as Beyond Uniqueness: The Birthday Paradox, Source Attribution, and Individualization in Forensic Science Testimony, Law, Probability and Risk, Vol. 12, March 2013, pp. 3-11. Regrettably, the publisher, Oxford University Press, forbids authors from posting the final versions of articles in the journal, but an earlier draft is available on SSRN.

Abstract: For many decades, forensic science identification experts have insisted that they can ‘individualize’ traces such as fingerprints and toolmarks to the one and only one object that produced them. They have relied on a theory of global uniqueness of patterns as the basis for such individualization. Although forensic practitioners and theorists are moving toward a more probabilistic understanding of pattern matching, textbooks and reference works continue to assert that uniqueness justifies individualization and that experience demonstrates discernible uniqueness. One response to the last claim applies a famous problem in probability theory — the Birthday Problem — to the forensic realm to show that even an extensive record of uniqueness does little to prove that all such patterns are unique. This essay describes the probabilistic reasoning and its limits. It argues that the logic of the Birthday Paradox does indeed undercut the theory of global, general uniqueness, but that the reasoning is logically compatible with opinion testimony that a specific object is nearly certain to be the source of a pattern or trace. It also notes some alternatives to categorical claims of individualization, whether those claims are based on the theory of global, general uniqueness or instead on some less sweeping and more defensible theory.

Hot Off the Presses: Chimeric Criminals

Nearly two years ago, I raised a question about the depth of the documentation and analysis in the book Genetic Justice by Sheldon Krimsky and Tania Simoncelli. A discussion of chimerism and the threat it supposedly poses to DNA exonerations prompted the following debunking essay: Chimeric Criminals, Minnesota Journal of Law, Science, and Technology, Vol. 14, No. 1, Winter 2013, pp. 1-9. It is available on the review's website or from SSRN.

Abstract: According to some commentators, an obscure genetic condition known as chimerism “could undermine the very basis of the forensic DNA system” and force a reconsideration of “the entire project of forensic DNA.” This conclusion is as unfounded as it is unnerving. Chimerism is a consideration in, but not a real obstacle to DNA identification. This essay explains why.