Ensuring the Reliability of Fire-Arm Identification Evidence

Notwithstanding the acceptance of firearm identification by courts, the scientific community has been reluctant to recognise firearm identification as a reliable method of conclusively establishing a connection between a particular bullet and a particular gun. The National Institute of Justice (NIJ) in the United States (US) has categorised firearm identification as a discipline under forensic science, and forensic science has been described as a "fractured and burdened discipline". In addition, in 2009 the National Academy of Sciences (NAS) concluded that forensic science is broken. With regard to firearm identification, the NAS Report emphasised the need for sufficient studies to be done because this report regarded this type of evidence as unreliable and lacking repeatability. The President's Council of Advisors on Science and Technology (PCAST) Report, released in September 2016, came to a conclusion similar to that of the 2009 NAS Report with regard to forensic science evidence. With regard to firearm identification, the report asserted that firearm identification evidence still "falls short of the scientific criteria for foundational validity". It is disturbing that courts across the globe are using different types of forensic science without subjecting them to scrutiny so as to determine their reliability. In the light of this, reliability and validity have become important factors which demand attention in Anglo-American litigation, even in jurisdictions that do not have a formal reliability standard (such as England and Wales, and South Africa). This article shows the role of cross-examination in establishing the reliability of firearm expert evidence. It also focusses on the role that South African forensic practitioners, prosecutors, defence counsels and presiding officers can play in ensuring the reliability of firearm identification evidence.


Introduction
The research which informs this article was prompted by the fact that the relevant scientific community has reservations regarding the way in which firearm identification has traditionally been accepted in courts, and is sceptical with regard to the assumption that current firearm identification methods can conclusively establish a connection between a specific bullet and a particular firearm. The National Institute of Justice (NIJ) in the United States (US) has categorised firearm identification as a discipline under forensic science, 1 and forensic science has been described as a "fractured and burdened discipline". 2 This scepticism is compounded by the fact that, in 2009 the National Academy of Sciences (NAS) concluded that forensic science is broken. 3 With regard to firearm identification, the NAS Report emphasised the need for more studies to be done, because this report regarded this type of evidence as unreliable and lacking repeatability. The President's Council of Advisors on Science and Technology (PCAST) Report, released in September 2016, came to a conclusion similar to that of the 2009 NAS Report with regard to forensic science evidence. 4 This report concluded that firearm identification evidence "still falls short of the scientific criteria for foundational validity". Despite this negative criticism of firearm identification evidence, courts across the globe still use different types of forensic science without subjecting them to scrutiny so as to determine their reliability. 5 In the light of this, reliability and validity have become important factors which demand attention in Anglo-American litigation, even in jurisdictions that do not have a formal reliability standard (such as England and Wales, and South Africa). 6 The aim of this article is to investigate the * Tanyarara Mutsavi. LLB (UFH) LLM (UFH). E-mail: mutsavitanyarara@gmail.com. ORCiD: https://orcid.org/0000-0002-3298-0360. LLM candidate University of Fort Hare, South Africa. The article is partially based on the first author's LLM thesis titled The Reliability of Firearm Identification in South Africa: A Comparative Perspective (UFS 2018). VAN DER WALT PER / PELJ 2020 (23) role of cross-examination in establishing the reliability of firearm expert evidence, and the discussion below focuses on the role that South African forensic practitioners, prosecutors, defence counsel and presiding officers can play in ensuring the reliability of firearm identification evidence.

Firearm identification
Firearm identification is "the forensic science discipline that identifies a bullet, cartridge case or other ammunition components as having been fired by a particular firearm". 7 If a firearm is fired, a number of the gun's features are conveyed to the cartridge casings and bullets, thereby making distinctive patterns called striae or scratch marks. 8 These marks, also called tool marks, are made each time the firearm is fired and they are formed as a result of the primer which detonates and the gunpowder which burns, resulting in the expansion of the casing in all directions, causing the imprinting of the casing and ammunition by the breech face of the gun. 9 The internal part of a gun is hard and that is why, when it gets into contact with the softer metal of the bullet and casings, it results in the making of marks on the casings and ammunition. 10 A firearm examiner purports to match bullets and cartridges to the weapon from which they originated by comparing bullets test-fired from a recovered gun with the spent bullets from the scene, using a comparison microscope to do so. 11

The role of forensic practitioners in ensuring reliability
Forensic practitioners, such as firearm examiners, should use mainstream scientific methods and norms. 12 According to Edmond et al, mainstream scientific methods will-13 4 T MUTSAVI & L MEINTJES VAN DER WALT PER / PELJ 2020 (23) Practitioners should furthermore ensure that there is "disclosure", "transparency", and "impartiality" with regard to the information provided to prosecutors, defence lawyers and judges when they give testimony and in their reports. 14 This would assist the court to comprehend how the evidence was collected, processed and analysed 15 in order to make it easier to determine the probative value of forensic science evidence and to determine the credibility of the practitioner. 16 The main function of the expert witness is to help judges to "administer justice" through the provision of impartial expert opinion and testimony. 17

The role of prosecutors in achieving the reliability of firearm identification
Edmond contends that prosecutors play an important role in the recognition and "social legitimating" of different types of expert evidence. 18 Should prosecutors be tempted to trivialise problems regarding the scientific reliability of firearm identification, trials and appeals might not be doing enough to regulate firearm identification and other types of forensic science. 19 This could lead to miscarriages of justice and the "pursuit of truth" could be threatened. 20 Gershman elucidates that "the prosecutor dominates the system, has exclusive control of the evidence, and decides how that evidence will be used." 21  Impartiality requires that forensic practitioners discharge their responsibilities to assist the court to reach an accurate conclusion, including by explaining uncertainties and limitationsrather than conceiving of their role as one of assisting the police or the prosecution to secure a conviction.  VAN DER WALT PER / PELJ 2020 (23) incriminating expert evidence. 23 They further argue that "unreliable, weak and speculative forensic science would be far less of a problem if trial mechanisms consistently identified and conveyed limitations with expert 24 evidence".
Nonetheless, it has emerged that prosecutors sometimes adduce forensic science and leave the defence to identify and explain its weaknesses and limitations through cross-examination. 25 However, prosecutors should on their own obtain information about the "limitations and oversights" with regard to expert opinion evidence. 26 Defence lawyers might lack resources and they might not be sufficiently technically literate to recognise and explain the weaknesses in forensic science. 27 According to Edmond et al, prosecutors and forensic practitioners need to refer to "validation studies", "limitations", "error rates" and "controversies" so that the defence and judges will find out about them. 28

The importance of cross-examination in achieving reliability in firearm identification
Black's Law Dictionary defines cross-examination as "the questioning of a witness upon a trial or hearing, or upon taking a deposition, by the party opposed to the one who produced him." 29 Cross-examination is not just a 23 Moriarty 2007 VAN DER WALT PER / PELJ 2020 (23) privilege but it is a right which a party is given to confront testimony from an opposing witness. 30 No statement should be used as testimony in the court system until it has been challenged by the opposing party. 31

Relevance
In South Africa expert evidence is accepted only if it is relevant. 32 The actual reliability of the expert evidence does not play an important role at the admissibility stage of the evidence but during trial, through crossexamination or when the opposing side adduces evidence, the reliability of the evidence can be attacked. 33 It is during the cross-examination stage that counsel is obliged to be conversant with and attend to the concerns of mainstream scientific organisations and the attentive community of scholars. 34 The Law Commission of England and Wales supports this argument in its report by saying it is through cross-examination that "the adduction of contrary expert evidence and judicial guidance at the end of the trial are currently assumed to provide sufficient safeguards in relation to expert evidence." 35 Acharya shares this view and asserts that "a keystone feature of the adversarial system is its ability, through properly resourced and informed cross-examination to best reveal and illuminate areas of scientific controversy." 36 He further believes that cross-examination is there to expose "inconsistencies" and "improprieties" in scientific evidence. 37 The questions asked in the course of cross-examination should be centred on establishing "experimental validation", "measures of reliability" and "proficiency", because these factors provide some information about "actual ability" and "accuracy" that makes it possible for expert evidence to be rationally evaluated by judges. This, as against focussing on things like qualifications, experience, common knowledge and previous admission. 38 30 Resurrection Gold Mining Co v Fortune Gold Mining Co 129 F 668 (8 th Cir 1904

Validation
When required to establish the weight to be attached to firearm identification, presiding officers in South Africa could derive significant assistance from the criteria for reliability and admissibility accepted in the landmark case of Daubert. The reliability criteria for "determining the admissibility of scientific evidence" established in this case could assist South African presiding officers in determining the weight which should be attached to firearm identification evidence. 39 This includes whether the theory or technique: 40 a) can be and has been tested; b) whether it has been subjected to peerreview and publication; c) whether the technique employed by the expert is generally accepted in the scientific community; d) whether the known or potential rate of error is known; and; e) whether the research was conducted independent of the particular litigation or dependent on an intention to provide the proposed testimony.
Presiding officers might find it helpful to apply the four Daubert criteria when they are required to evaluate the scientific value of firearm/tool mark examiners' conclusions regarding striated tool mark identity. 41 In the aftermath of Daubert, commentators like Koehler pointed out that forensic science is no longer regarded to be as infallible as it was in the past. 42 He accentuated several factors such as the fallibility of forensic science in "crime lab scandals, fraud, unsupported assumptions, high profile errors, and wrongful convictions", 43 which underlined the potential fallibility of some forensic disciplines.
Reliable scientific evidence is required to be based on a theory that is testable and falsifiable. 44 Meintjes-van der Walt likens falsifiability to refutability or testability. She goes on to say that "in order for a theory to be scientific, it must make predictions concrete enough to be proved wrong if the claim is not true." 45 VAN DER WALT PER / PELJ 2020 (23) When liberty hangs in the balance ... the standards should be higher than ... have been imposed across the country. The more courts admit this type of tool mark evidence without requiring documentation, proficiency testing, or evidence of reliability, the more sloppy practices will endure; we should require more.
South African courts, while possibly not consistently so, also require the testing of an expert's opinion when the weight of the evidence is decided. In the South African case of R v Jacobs, 48 many years before Daubert, Ramsbottom J held that: 49 ... it is of the greatest importance that the value of the opinion should be capable of being tested and unless the expert states the grounds upon which he bases his opinion, it is not possible to test its correctness so as to form a proper judgment upon it.
Legal decision-makers need to take note of the conclusion of the The Ballistic Imaging Report that "[t]he validity of the fundamental assumptions of uniqueness and reproducibility of firearms-related tool marks has not yet been fully demonstrated." 50 The adequacy of the empirical basis of firearm identification expertise, therefore, is still not conclusive and research in this regard is still in progress. 51 Presiding officers should decide whether experts testify on matters growing "naturally and directly" out of research they conducted "independent of the litigation", or whether they have developed their opinions expressly for the purposes of testifying. 52 The court in In Re: Paoli Railroad Yard PCB Litigation stated that testimony based on "legitimate, pre-existing research unrelated to the litigation" constitutes the "most persuasive" grounds for deciding on the scientific grounds for the testimony of a forensic expert. 53 Daubert indicates that peer-review is an important way means by which a court can determine the scientific validity and reliability of expert testimony. In terms of this paradigm, the scientific method of the particular identification process has to be peer-reviewed by other experts in the field. According to Grzybowski and Murdock, peer-review Grzybowski and Murdock 1998 AFTE Journal 9. 9 T MUTSAVI & L MEINTJES VAN DER WALT PER / PELJ 2020 (23) in order to comply with the requirements of scientific method, firearm identification should be published in a professional peer-reviewed journal: 55 The peer-review process will involve the assessment of the following:(1) the validity of the hypothesis; (2) how it was formulated and tested; (3) whether the scientific method was followed; and (4) whether proper conclusions were reached.
The Journal of Forensic Sciences, the official journal of the American Academy of Forensic Sciences (AAFS), together with others of "similar substance", utilise a peer-review process for firearm evidence. 56 Although the Association of Firearm and Tool Mark Examiners (AFTE) journal claims to be peer-reviewed, 57 the court in US v Diaz 58 did not recognise the AFT Journal as a peer-reviewed journal, citing the fact that it does not meet the basic requirements of a peer-reviewed journal.
It seems as if firearm and tool mark examiners often peer-review one another's work after identification has been reached. This has been recognised by some legal commentators as the source of confirmation bias. 59 Mahoney defines confirmation bias as "a phenomenon whereby scientists tend to settle on a theory at the outset and thereafter tend to look for data to confirm the theory, rather than trying to discredit or refute it." 60 Confirmation bias can give some value to evidence that is in favour of a person's opinion or version of events. 61 Commenting on the same issue, Dutton elucidates that "[i]f the expert doing the check only ever checks positive matches, then the perception will be that whenever he sits at the microscope to conduct a peer-review of casework, he will expect to see a positive match." 62 More criticism has emerged with regard to the peer-reviewing process in the disciplines of forensic science. According to Cooper, forensics analysts are often from the law enforcement field instead of a particular scientific field, 63 which has the effect that the forensic science disciplines tend to be rooted 55 Grzybowski  VAN DER WALT PER / PELJ 2020 (23) in research that has application which is only about criminal investigations and law enforcement and not about knowledge per se. As a result "these disciplines can be fragmented, poorly regulated and lack standardised procedures." 64 Research regarding scientific methods in these areas can be limited, unpublished and narrowly circulated, and there is often a lack of will to pursue the validation of the methods employed. 65 Experimental evidence to determine whether "a technique does what it purports to, and how well" is reached by means of validation. If the techniques are validated on the basis of empirical evidence, they will produce stable and consistent results. Validation also provides the appropriate framework to assess abilities and levels of performance. 66

Error rates
According to Edmond et al, validation studies provide information about "the circumstances in which a technique is known to work, how well it works as well as its limitations." 67 As a result, the testimony given in court must provide limitations and information about potential sources of error. 68 An expert opinion that does not reveal the basis on which it is grounded and which does not disclose its known limitations is incomplete. Furthermore, it "creates a serious risk of being misunderstood, and contravenes the expert's overriding duty impartially to assist the court." 69 It is difficult to 64 Although it should certainly be noted that the implementation and following of "standardized procedures" in forensic science identification methods do not automatically produce scientifically valid or reliable results. The NRC Report reached this very conclusion in relation to the ACE-V procedure used by fingerprint examiners:"[m]erely following the steps of ACE-V does not imply that one is proceeding in a scientific manner or producing reliable results." 2009 NAS Report 142. The NRC Report made a similar conclusion in relation to the AFTE protocol associated with tool-mark examination. "This AFTE document, which is the best guidance available for the field of toolmark identification, does not even consider, let alone address, questions regarding variability, reliability, repeatability, or the number of correlations needed to achieve a given degree of confidence. "The fact that any error rate will be somewhat artificial and might not capture the precise conditions of the analysis is not an excuse. Awareness of this issue did not T MUTSAVI & L MEINTJES VAN DER WALT PER / PELJ 2020 (23) evaluate the value of the practitioner's evidence when there is no indication of the error rate. 70 The third element to determine reliability as stipulated in Daubert is the determination of the error rate of the method which is being used. Error rate refers to the frequency with which one deviates or strays from a correct standard. 71 In cases involving firearms, experts are called to give testimony on whether the firearm in question can be identified as the source of a questioned tool mark or not. In these instances, presiding officers also need to know "how often such identifications are in error". 72 In other words, they want to know "how often the profession, using accepted techniques and controls, produces a mistaken identity." 73 In firearm identification the only international source of proficiency testing from which potential error rates can be inferred is the Collaborative Testing Service (CTS). 74 The 1978 Crime Laboratory Proficiency Testing Program in the USA registered "mixed results" regarding CTS results for firearm identification. 75 In one of the tests conducted, 5.3% of the laboratories that took part in the test misidentified firearms evidence. In another one, 13.6% of the laboratories erred. 76 The tests were done on the basis of bullet and cartridge case comparisons. The Project Advisory Committee held that these errors were "particularly grave in nature" and concluded that "they probably resulted from carelessness, inexperience, or inadequate supervision." 77 In the 1978-2005 period, less than 5% of responses were in error, but individual test results varied. 78 In some instances 30% to 40% of the replies were not conclusive, because the laboratories were not sure if the tool in question had been altered between the times when different markings were made.  (23) frequent for firearms testing. 79 Examiners in most cases stated that they were not able to come to a conclusion because of the unavailability of the actual weapon to test-fire ammunition. 80 In this context, questions and criticisms have arisen concerning the significance of these tests. One of the criticisms is that the sample for proficiency testing to determine error rate is "self-selecting and may not be representative of the complete universe of firearms examiners". 81 Moreover, the examinations are not blind. That is, examiners know when they are being tested. As a result, the examiner is likely to be more careful than in ordinary case work. 82 In relation to this, the 2008 Ballistic Imaging Report stated that most of these studies are limited in scale and have been conducted by firearms examiners (and examiners in training) in state and local law enforcement laboratories as adjuncts to their regular casework. 83 In regard to the factor of "known or potential error", the court in US v Diaz held that "it is not possible to calculate an absolute error rate for firearms identification." 84 This is partly because the standards and criteria for traditional pattern matching are subjective. 85 Furthermore, the court ended by stating that "No true error rate will ever be calculated so long as the firearm-examiner community continues to rely on the subjective traditional pattern matching method of identification." 86 This means that currently no true error rate for firearm identification is known.

Inadequacies of the AFTE Theory of Identification
In regard to the above, the Association of Firearm and Tool Mark Examiners (AFTE) in a review in its journal under the heading "Theory of Identification, Range of Striae Comparison Reports and Modified Glossary Definitions" discusses the discipline of forensic firearm and tool mark identification and also gives the "basic theory that allows opinions of common origin to be  VAN DER WALT PER / PELJ 2020 (23) made in tool mark comparisons." 87 The AFTE Theory of Identification adopted in 1992 states: 1.
The theory of identification as it pertains to the comparison of tool marks enables opinions of common origin to be made when the unique surface contours of two tool marks are in 'sufficient agreement'.

2.
This 'sufficient agreement' is related to the significant duplication of random tool marks as evidenced by a pattern or combination of patterns of surface contours. Significance is determined by the comparative examination of two or more sets of surface contour patterns comprised of individual peaks, ridges and furrows. Specifically, the relative height or depth, width, curvature and spatial relationship of the individual peaks, ridges and furrows within one set of surface contours are defined and compared to the corresponding features in the second set of surface contours. Agreement is significant when it exceeds the best agreement demonstrated between tool marks known to have been produced by different tools and is consistent with agreement demonstrated by tool marks known to have been produced by the same tool. The statement that 'sufficient agreement' exists between two tool marks means that the agreement is of a quantity and quality that the likelihood that another tool could have made the mark is so remote as to be considered a practical impossibility.

3.
Currently the interpretation of individualization/identification is subjective in nature, founded on scientific principles and based on the examiner's training and experience. 88 The NAS Report suggests that the AFTE theory of identification does not give a specific protocol, because the theory states that an examiner can give "an opinion that a specific firearm was the cause of a bullet striation pattern when sufficient agreement exists in the pattern of two sets of marks", 89 which is a subjective determination. In addition, the AFTE theory contends that "agreement is significant when it exceeds the best agreement demonstrated between tool marks known to have been produced by different tools and is consistent with the agreement demonstrated by tool marks known to have been produced by the same tool." 90 In this regard, the 2009 NAS Report comments that the meaning of "exceeds the best agreement" and "consistent with" are not clearly established, and the examiner is expected to "draw on his or her own experience". 91 The AFTE theory is also described as being circular, in the PCAST Report,  VAN DER WALT PER / PELJ 2020 (23) suggests that an examiner has the capacity to state that "two tool marks have a common origin when their features are [in] sufficient agreement." 92 The AFTE theory shows that "[c]urrently the interpretation of individualization/identification is subjective in nature". 93 It is crucial to note that under the subjective approach, examiners rely only on their mind and eye judgments in identifying resemblances instead of articulating the criteria which they used to arrive at conclusions with regard to the resemblances. 94 Nichols poses the question, "with subjective standards/criteria, how does one determine if either examiner has made an error?" 95 In US v Monteiro Judge Saris stated that the AFTE theory is "tautological: it requires each examiner to decide when there is 'sufficient agreement' of tool marks to constitute identification." 96 Judge Saris further criticised the AFTE theory for not providing examiners with any guidance on telling the difference between subclass and individual characteristics. 97 The judge held that: 98 Because an examiner's bottom line opinion as to identification is largely a subjective one, 99 there is no reliable statistical or scientific methodology which will currently permit the expert to testify that it is a 'match' to an absolute certainty, or to an arbitrary degree of statistical certainty.
In Ramirez v State of Florida 100 the Supreme Court criticised firearm and tool mark examiners who rely on "nothing more than their own subjective criteria for striae identification" and are unable to put forth "a convincing, logical, scientifically based explanation for the basis of their identifications." 101 92 2016 PCAST Report 60. In response to PCAST's concern about this circularity, the FBI Laboratory replied that: "'Practical impossibility' is the certitude that exists when there is sufficient agreement in the quality and quantity of individual characteristics." This answer did not address the issue helpfully. US v Monteiro 371. The Judge expressed concern that the examiner who made the identifications indicated that he does not even consider subclass characteristics when he examines breech face markings. The AFTE Theory is based on the assumption that interpretations based on forensic science are founded on an examiner's training and experience. 102 The PCAST Report, however, states that experience is not enough for one to draw judgments about whether two features have come from different sources or the same source. To show the fallacy of relying on "experience", the PCAST Report refers to the testimony of a former head of the FBI, in which he claimed that "the FBI had an error rate of one per every 11 million cases", based on the fact that the agency was aware of only one mistake. 103 This observation has been questioned in empirical studies by the FBI Laboratory which indicate a high error rate of roughly one in several hundred. 104 Speaking at the Ninth Circuit Judicial Conference, Mnookin contended that "[i]t must take scientific study to make a field scientifically reliable". 105 Mnookin further said that "[e]xperience, no matter how extensive, could not be a substitute for scientific study". 106 Moreover, Edmond believes that prosecutors, defence lawyers and judges should direct attention to formal evidence of reliability and not rely on evidence founded on the extent of the experience of the witness. 107 A theory of science has been defined by the NAS to mean "a comprehensive explanation of some aspect of nature that is supported by a vast body of evidence." 108 Accepting this definition, the PCAST Report contends that the AFTE theory of identification is clearly not a scientific theory. Instead, "it is a claim that examiners applying a subjective approach can accurately individualise the origin of a tool mark." 109 The report goes on to say that a "theory" is not what is needed at the moment but only empirical tests to figure out how reliable and valid the method is. 110 Moreover, in the late 1990s, the AFTE drafted a protocol (the AFTE Protocol) for experts to follow during their examinations. 111 105 Dinzeo 2017 https://www.courthousenews.com/skepticism-forensic-methods-urged-9th-circuit-conference. 106 Dinzeo 2017 https://www.courthousenews.com/skepticism-forensic-methods-urged-9th-circuit-conference.  (23) … an examiner may make one of the following four conclusions: (1) identification, (2) inconclusive, (3) elimination, or (4) unsuitable for comparison. To make an 'identification' (i.e., a 'match'), there must be 'sufficient agreement' between the tool-marks present on ammunition found at a crime scene and a test cartridge fired from a suspect weapon.
In spite of this protocol and the "routine admission of firearms identification evidence", the discipline has been criticised in the NAS and PCAST Reports. 113 The 2009 NAS Report maintains that the AFTE document which has been described as the most important source of guidance in the discipline of firearm identification does not touch on issues of "variability, reliability, repeatability, or the number of correlations needed to achieve a given degree of confidence." 114 The 2009 NAS Report concluded that the AFTE Protocol was not defined sufficiently for examiners to be able to follow it, particularly when an examiner can be said to have "matched" two samples. 115

Personal proficiency
Edmond et al warn that the court cannot assess the probative value of the evidence in instances where the professional proficiency and the mastery of particular techniques on the part of the expert witness have not been validated. 116

Expressions of opinion
Currently the examination of firearms involves human judgment. The expression of a forensic practitioner's opinion should be informed by experimental research based on a validated technique and based on the proficiency of forensic practitioners. Forensic practitioners should be in a position empirically to justify particular terminology or scales, and the justification should be clear and comprehensible.
In the NAS Report the National Research Council (NRC) Committee emphasised that there is a need to raise the standards for "reporting and testifying about the results" of investigations in most disciplines of forensic science. 117 A good example is the use of some terms by forensic examiners in reports and in their court testimony in describing their findings and conclusions. These terms include "match", "consistent with", "identical", "similar in all respects tested", and "cannot be excluded as the source of". 118 The use of these terms can have some influence on how the presiding officer in court proceedings assesses and evaluates the evidence. Nonetheless, there has not been agreement or consensus on the exact meaning of the terms in forensic science. 119 They have not been standardised, even if some fields of forensic science "have developed vocabulary and scales" which they use in reporting their results. 120 Laboratory reports generated by scientific analysis should be thorough and should describe their-121 methods and materials, procedures, results, and conclusions, and they should identify, as appropriate, the sources of uncertainty in the procedures and conclusions which indicate the level of confidence in the results.
According to the NRC Committee, most forensic laboratory reports do not meet this standard of reporting, although some do. 122 In addition to the above, the forensic science reports and courtroom testimony must present the limitations of the analyses, including the associated probabilities, where this is possible. The courtroom testimony should be given in clear terms that everyone engaged in the trial can understand, so as to be able to interpret the testimony. The NAS Report contends that this is achievable and that "research must be undertaken to evaluate the reliability of the steps of the various identification methods and the confidence intervals associated with the overall conclusions." 123 Since the first assumption in firearm identification is that a mark can be individually related to the specific gun from which it was fired, some firearm analysts, when giving opinions of mark identity, have concluded that the firearm responsible for making the mark can be individualised "to the exclusion of all other tools". 124  Notwithstanding this criticism, firearm identification evidence has been admitted into American courtrooms and is still being admitted. 126 Murdock states that: 127 Absolute certainty opinions may have been adopted in the past, but this type of position has been retired for some time and no longer represents the consensus thinking of the firearm and tool mark community. … [O]ur everyday lives are predicated upon practical certainty. 128 Firearm identification became well known in the early 20 th century and by the middle of the twentieth century courts were admitting firearm testimony by firearm experts. 129 Cases decided after the middle of the twentieth century relied on these previous cases for precedence in admitting evidence of bullet, 130 cartridge case, 131 and shot shell 132 identifications. A number of courts have also permitted an expert to testify that "a bullet could have been reliability and repeatability of the methods. The committee agrees that class characteristics are helpful in narrowing the pool of tools that may have left a distinctive mark. Individual patterns from manufacture or from wear might, in some cases, be distinctive enough to suggest one particular source, but additional studies should be performed to make the process of individualization more precise and repeatable.  (23) The more courts admit this type of tool mark evidence without requiring documentation, proficiency testing, or evidence of reliability, the more sloppy practices will endure; we should require more.
The court in United States v Glynn 142 was of the view that the expert should not use the term "reasonable scientific certainty" in testifying, but rather that the expert should be permitted to testify only that it was "more likely than not" that recovered bullets and cartridge cases came from a particular 143 weapon.
The district court in United States v Ashburn, 144 while declining to go as far as Green and Glynn in circumscribing source opinions, relied on the 2009 NAS Report and the criticisms of the AFTE sufficiency theory in the opinions discussed above to preclude "this expert witness from testifying that he is 'certain' or '100%' sure [or] that a match he identified is to the exclusion of all other firearms in the world' or that there is a 'practical impossibility' that any other gun could have fired the recovered materials." 145 Another pertinent case which involved the reliability of firearm identification is the Ramirez case. 146 Firearm experts had to be able to put forth "a convincing, logical, scientifically based explanation for the basis of their identifications." 147 The court in this case rejected the argument by the expert that "I know it is a match because I have sufficient background, training and experience." The judge also rejected the argument by the firearm examiner that "there is absolute certainty of his identification and that there are no : "According to his testimony, these tool marks were sufficiently similar to allow him to identify Defendant's gun as the gun that fired the cartridge found at the crime scene. He opined that he held this opinion to a 100% degree of certainty. The Court also finds [the examiner's] opinions reliable and based upon a scientifically valid methodology. Evidence was presented at the hearing that the tool mark testing methodology he employed has been tested, has been subjected to peer-review, has an ascertainable error rate, and is generally accepted in the scientific community". Commonwealth v Meeks Nos 2002-10961, 2003-10575, 2006WL 2819423 50 (Mass Super Ct Sept 28, 2006: "The theory and process of firearms identification are generally accepted and reliable, and the process has been reliably applied in these cases. Accordingly, the firearms identification evidence, including opinions as to matches, may be presented to the juries for their consideration, but only if that evidence includes a detailed statement of the reasons for those opinions together with appropriate documentation.  VAN DER WALT PER / PELJ 2020 (23) objective criteria that must be met." 148 Finally, in the Ramirez case the court found the examiner's scientific methodology had not gone through "meaningful peer-review or publication". 149 In United States v Monteiro, 150 after reviewing the Daubert requirements at length, the court found that firearm evidence was generally admissible but that the prosecution's witnesses were not qualified. Moreover, the judge held that: 151 even a qualified government expert may testify that the cartridge cases were fired from a particular firearm to a reasonable degree of ballistic certainty. However, the expert may not testify that there is a match to an exact certainty.
In this case the expert had not taken photographs of the evidence to show the comparisons which had been made. In this regard the court ruled that: 152 Until the basis for the identification is described in such a way that the procedure performed by [the examiner] is reproducible and verifiable, it is inadmissible under Rule 702.
When presenting firearm identification evidence to the court, the expert must satisfy the court according to Rule 702 with regard to the following four factors: 153 scientific, technical, or other specialized knowledge that will help the trier of fact to understand the evidence or to determine a fact in issue; the testimony is based on sufficient facts or data; the testimony is the product of reliable principles and methods; and the expert has reliably applied the principles and methods to the facts of the case.
These rules would be equally applicable in South Africa even although South Africa does not apply the Daubert rule for the purposes of admissibility. 154 The court in United States v Diaz, 155 after finding that the record did not support the conclusion that identifications could be made to the exclusion of all other firearms in the world, held that "the examiners who testify in this 148 Ramirez case may only testify that a match has been made to a 'reasonable degree of certainty in the ballistics field. '" 156 In United States v Taylor 157 the court ruled that the government expert "will not be allowed to testify that he can conclude that there is a match to the exclusion, either practical or absolute, of all other guns." Furthermore, in US v Anderson 158 the court was of the view that a firearm examiner may testify to "a reasonable degree of certainty" in the field of firearms and tool mark identification or "to a practical certainty" but not to "a reasonable degree" of scientific certainty or a "practical impossibility". 159 US v Willock 160 joined the ranks of cases restricting the conclusions that may be offered in testimony by a firearms and tool mark examiner. Adopting a magistrate judge's Report and Recommendations (R & R), 161 the court ordered that the witness shall not express the opinion that it is a "practical impossibility for any other firearm to have fired the cartridges" recovered at the crime scene and that the witness "shall state his opinions and conclusions without characterisation as to the degree of certainty with which he holds them." 162 The court based these restrictions on the 2009 NAS Report andthe 2008 Ballistic Imaging Report. 163 In Williams v United States 164 the appellant was arrested and prosecuted for murder with the use of a firearm. After searching the appellant's apartment, the police recovered a gun that, when test-fired, left markings on the bullets that appeared to match the markings on bullets recovered at the crime scene. After considering this evidence, a jury convicted the appellant. On appeal, the appellant argued that the firearm and tool mark examiner should not have been able to testify that the markings on the bullets recovered from the crime scene were unique or that he was without "any doubt" that these bullets were fired from the gun found with the appellant. With regard to this issue the appeal court quoted the case of Jones v United States. 165 In that case the court argued inter alia that tool mark and firearms examiners could not "[s]tate their conclusions with 'absolute certainty excluding all other possible firearms.'" 166 In the light of this, the appeal court held that "experts should not be permitted to testily that they are 100 per cent certain of a match, to the exclusion of all other firearms." 167 In 2017 Jessica Brand 168 referred to a murder case decided in 1993. The accused was convicted by the jury based on firearm testimony by the firearm experts. They declared that cartridges and bullets recovered from the crime scene matched the gun of the accused "to the exclusion of all other firearms". In 2017, the state agreed that the examiner should never have made such a statement and that the examiner should have said the convicted person's gun "could not be eliminated". The court granted the convicted person a new trial. 169 Presiding officers should "place constraints" on what the forensic experts say in court. 170

Verification and peer-review
Verification and peer-review are crucially important in order to confirm the value of the result and the conclusion reached by using the particular technique. Verification and peer-review are uncertain and of no importance if the techniques are not validated. Forensic practitioners should not suggest that peer-review or other verification procedures in themselves can somehow overcome or repair the absence of validation. Furthermore, for peer-review to be effective, the reviewer must be unaware of the original results. In this regard, peer-review and verification are most likely to help to reduce errors when they are conducted "where procedures are known to be 171 valid and review is blind".

Cognitive and contextual effects
Forensic scientists could be exposed to some information that might not be relevant to their processing and interpretation of evidence, and this is problematic in that their interpretation and the value of their opinion evidence could be threatened. 172 The problem which arises as a result of contextual and cognitive biases is also discussed in a report jointly made by the National Institute of Standards and Technology (NIST) and the NIJ. 173 According to Edmond et al, where there is vulnerability, the only way to avoid being influenced inappropriately is "to restrict access to domainirrelevant information (or, more precisely, information with the potential to mislead)." 174 Edmond further elucidates that the separation of roles: 175 facilitates blind analysis while allowing analysts to have access to appropriate information thereby ensuring the case manager (and the institution) is informed about the overall case.
Another proposal in this regard involves sequential unmasking, where information is "gradually revealed to the analyst". 176 Under this proposal, for example, an analyst might carry out an "initial examination of the trace evidence and limit their interpretation to the legible or salient parts of the sample before comparing it to the suspect sample." 177 Of course, the process will depend on the type of trace evidence, but the idea behind this is to make the analyst blind to the information that has the potential to cause bias. 178 Foster and Huber 179 contend that "an emphasis on falsifying theories has an additional advantage in that it helps to overcome the effects of confirmation bias. Mahoney Scientist as Subject 155. "Confirmation bias is a phenomenon whereby scientists tend to settle on a theory at the outset and thereafter tend to look for data to confirm the theory, rather than trying to discredit or refute it." This phenomenon is illustrated in a paper by Garry et al 1994 Consciousness and Cognition 438, "where it is shown that mental health professionals investigating child abuse may too readily albeit unwittingly collaborate with the presumed victim to conjure up memories of abuse that never happened", as cited by Meintjes-van der Walt 2003 J Afr L 101. T MUTSAVI & L MEINTJES VAN DER WALT PER / PELJ 2020 (23) in most cases left unchecked because if a firearm has been considered to be an unique source of marks, the firearm examiners in that case are no longer examining "any other gun" to determine if it might produce tool marks "that do at least as good a job at matching the evidence tool marks." 181 Related to confirmation bias are observer effects. It is important to falsify and test theories so as to avoid observer bias and other sources of human error in forensic examinations. 182 The psychological theory of observer effects states that "external information provided to persons conducting analyses may taint their conclusions." 183 This a serious problem in techniques with a subjective component. 184 This type of information has serious effects if the expert is exposed to it. 185 With regard to the issue of bias, the NRC committee proposed that a body of research to reform institutional procedures and workflows is needed to address the impact of bias. 186

How presiding officers can determine whether a bullet/cartridge comes from a specific firearm
Firearm identification experts rely on markings on cartridges and projectiles to determine the link with a particular weapon, and in cases where firearm identification evidence is involved the court is concerned with how the marks associated with the suspect weapon were matched with other evidence. 191 This is based on the assumption that "all firearms possess distinctive features that in turn impart distinctive markings onto bullets and cartridge casings when the weapon is fired." 192 In order to determine whether the marks were fired from a particular weapon, firearm examiners use a comparison microscope. Under this comparison microscope bullets and cartridge casings recovered from the crime scene are compared with those test-fired from a seized weapon. 193 The similarities and differences between the marks must be observed and recorded during this process. An examiner uses the observed information for the purposes of identification in order to reach a decision regarding a common source of the bullet or cartridge in question. 194 After the matching has been completed, enlarged photographs of the marks should be made for the purpose of illustrating the correctness of the examiner's conclusions. 195 The significant factors the presiding officer should consider in assessing whether a particular firearm is the source of a mark on a questioned bullet or cartridge case are: (i) similarities between impressions left by different firearms and (ii) differences observed during the comparison between impressions left by the same firearm. 196 In addition, presiding officers need to make sure that statements on matches made "should be supported by the work that was done in the laboratory by the notes and documentation made by examiners and by proficiency testing or established error rates for individual examiners in the field and in that particular laboratory, but should not overreach to make extreme probability statements.  (23) jurors must not engage in speculation about "validity and reliability or trace the probative value of evidence from the subjective beliefs of practitioners based on impressions of the apparent value of experience and independence, or their demeanour, confidence and resilience during crossexamination." 199 According to the court "there should be information supporting reliability and facilitating evaluation." 200

Conclusion
Although firearm identification has been described as unreliable, it has played an important role in identifying the perpetrators of crime. 201 If efforts are made to validate this type of evidence, it could generate "legitimate convictions" that do not change even when there is a "post-conviction relief appeals process" in the criminal justice system. 202 The 2009 NAS Report is particularly critical of weaknesses in the scientific underpinnings of a number of the forensic disciplines routinely used in the criminal justice system, including firearm identification. That report shows that firearm identification has not yet reached a stage where it can be regarded as reliable evidence. 203 Science is developing exponentially, and investigating officers, lawyers, magistrates, judges and even forensic technicians cannot reasonably be expected to be au fait with cutting-edge developments in the field of forensic science and technology. For this reason, continuing education, as is a requirement in the medical profession, is urgently necessary in the field of forensic science. The introduction in South Africa of the NRC recommendation regarding "legal education programmes for law students, practitioners and judges" 204 would be logistically uncomplicated and relatively inexpensive and would have the potential significantly to relieve part of the enormous burden on prosecutors, defence lawyers, magistrates and judges, and above all would have the potential to go some way towards improving the scientific validity of firearm identification.  (23) above, this would engender a courtroom culture which would be even more circumspect in dealing with firearm identification than before, by incisively interrogating the scientific foundations on which the expert evidence is based in order to decide what weight should be attached to the evidence.
The Daubert rules and other precautionary measures in dealing with firearm identification evidence in court have become the gold standard for the scientific evaluation of evidence based on forensic science in some jurisdictions, and even although the application of the Daubert rules is not yet mandatory in South Africa, 205 magistrates and judges could serve justice by using those criteria to determine the weight which should be attached to forensic evidence in general and to firearm identification in particular.