Continuing the discussion of certification in general, the NAS Report notes the following about certification programs, "The American Bar Association has recommended that certification standards be required of examiners, including “demanding written examinations, proficiency testing, continuing education, recertification procedures, an ethical code, and effective disciplinary procedures.” Let's break these down.
Demanding written exams. What does this mean? Are "written exams" composed of essay questions where one explains topics in depth? Are "written exams" objective tests, like multiple choice exams? What makes an exam "demanding?"
Could an examination body survey the entirety of my written work, books, papers, speeches, this blog, and so on? Could they judge my proficiency based on my written work? I'm not sure. But, pulling from my academic life, it is entirely possible for someone to be proficient in their craft, but be non-verbal. How do you conduct a "demanding written exam" on a non-verbal person? What about differences of language? Do we insist upon all tests being in US English? How do we handle the issue of testing vulnerable populations? Given my extensive work in the autistic community, both academically and professionally, these are relevant questions.
Proficiency testing. Does this belong to an independent certification program or within a vendor's training program? For example, who is a better judge of my proficiency with my chosen tools, Amped Software, Inc. (vendor) or the IAI (third party)? What does the IAI know about FIVE and Authenticate? Nothing, actually. They're still focussed on Photoshop. How do I know this? Their reading list has George Reis' Photoshop book as one of the sources for the test. Whilst still relevant, and quite good, it's a 10 year old treatise on George's favorite PS techniques in no particular order. Given that I no longer use Photoshop in my analytical workflow, how is a test on Photoshop helpful in assessing my proficiency as an examiner?
Perhaps the "requirement" could be met with a review of an analyst's case documentation. Submit a few reports, have the panel examine them in light of the ASTM and SWGDE standards / guidelines.
Continuing education. There comes a point in time, when you've been doing this for a while, that you run out of classes to take. Some, that are available, are cost prohibitive. That's a large part of why I've moved a lot of the classes that I teach to an LMS as micro-learning. As an example, if you can't get in to your local college to take a statistics course, you can take Statistics for Forensic Analysts on-line at an affordable price. Same for redaction, Redaction for Standards Compliance is offered on-line featuring a variety of software platforms.
I've been around the US and Canada quite a bit. I realize that traveling to training is not always an option. I think the internet offers some amazing options to level the playing field as regards continuing education. The question becomes, will your chosen certification program accept your completion certificate from an on-line course or provider?
Recertification procedures. I think the important part here is not only that the individual circle back and re-certify, but that the certification program continually monitors the industry and keeps the certification program up to date. Four years from now, when my AVFA certification is due for renewal, I should not be presented with the same test instrument. Laws and technology change. So too should the certification program. Again, I love George Reis to death, but is his book still relevant in a modern multimedia workflow? Perhaps. But are there other, more appropriate texts? Yes. Absolutely. What about listing Dr. John Russ' Image Processing Cookbook as well, given the preference for Photoshop? Or his Image Processing Handbook, for a more academic look at image processing? If price is an issue, they could choose Fundamentals of Digital Image Processing by Dr. Anil Jain. It's a text book that can be bought used for under $10.
Ethical code. Here's LEVA's Code of Ethics. This post is an example of article 5 in LEVA's COE, offering constructive criticism. Interestingly, article 9 is what prevented me from ever being LEVA certified. Article 9 states, "An Analyst shall maintain the confidentiality of evidence received and work undertaken in accordance with any such request by the submitting agency." It was the policy of the Los Angeles Police Dept. that case work remain confidential and no evidence leave the secure facility without permission. When I sought permission to begin the process of certification, it was flatly denied. Given the LEVA certification methodology, if you can't present one of your cases, you can't get certified. Thus, in honoring their COE, I was prevented from ever being certified whilst in police service. Now, in retirement and in private practice, LEVA discriminates against me for not being a current public service employee by charging confiscatory prices for their offerings.
For the IAI's COE, click here. It begins on pg 79 of the linked document.
Effective disciplinary procedures. One of the things that we do is review opposing counsel's analysis. The examiner on the other side of any case should expect a thorough review of not only their work but their credentials as submitted through the discovery process. I have lost track of the amount of times I have encountered embellishments and fabrication of credentials by analysts, some of which have been certified by LEVA or the IAI. Everything you write in your report should be factual. Everything you write about yourself should be verifiable. As an example of this, I note as a matter of fact that I am retired from police service, having served just shy of 15 years with the Los Angeles Police Dept. under civil service classification 3687. This is a verifiable statement. I can provide the job description for civil service classification 3687, or you can click on the link in the previous sentence. I can provide my retiree ID. I could even print out an account statement from the City's retirement plan noting how many years/months I have of retirement credits (an indication of years/months worked). But, if I simply quit my job, it is a slightly different process than retiring. Thus, I wouldn't say that I had retired, but that I had quit / separated / moved on - or some other term that would accurately describe the end of my time in police service. Same goes for my education. No embellishment, just a simple presentation of the facts.
Back to ethics, there have been a few times that I reported my findings to a certification body. To the best of my knowledge, nothing was ever done with the information that I provided. This is the problem with most professional organizations. We're all friends. No one wants to judge their friends. No waves. No trouble. Let it slide. If there are no consequences, there's no effective discipline.
As these all relate to the certification experience that I just undertook, the AVFA includes all elements except the "proficiency test." I was not given an item of evidence and asked to examine it vs. a question under enquiry. Could my body of work stand in a proficiency test's stead? How much have I written? How many courses of instruction have I designed? How many have I trained to competency? How many cases have I worked? After all, it's this body of work that prepared me to take and pass the exam.
Getting back to the NAS Report, "In essence, ‘certification’ usually means that a particular individual has completed a defined course of education, training, and experience, and has passed an examination prepared by peers which demonstrates that the individual has obtained at least the minimum level of competence required to practice the specific discipline." My education, training, and experience prepared me to take and pass the examination that was prepared by peers, thus demonstrating to the certificate program a minimum level of competence. As to the peers who prepared the exam, I may not know a single one of them, but having a list of names with contact information is a great start.
Friday, May 31, 2019
Thursday, May 30, 2019
Taking the AVFA Certification Exam
I have to say that signing up and taking the AVFA certification exam was a rather pleasant experience. Yes, that seems a bit odd to say. But, it's true. It should be true that an organization treats you like an adult, respectfully walking you through the process of engaging their services.
I started on the ETA-i web site. This is where one pays the exam fees, the first stop in the process. Simply select the "Pay Exam Fees" option in their store, select the exam, and pay the fee.
You can do a bit of scouting as to a testing location ahead of your purchase, and include that information in the purchasing process. It will speed things along a bit. I selected a Prometric testing close by. After I paid the exam fees, ETA-i sent my information and the exam to the testing center. I then paid the proctoring fees to the testing center. In total, I paid $120 in fees.
I can't emphasize this enough, I paid $120 in fees to take this certification exam.
If you're sitting for the LEVA CFVA board, you will have to get there (transportation costs) and stay there at least a few days (hotel and food). Likely, you'll want to stay the whole week and, assuming you pass, get your certificate at the Friday dinner. Assuming average travel costs, you or your agency will spend about $2000 for this experience. This $2000 comes after spending thousands upon thousands in the Levels courses and the electives.
For the IAI's CFVE, the cost of testing is significantly lower than LEVA. The IAI charges $400 for the initial test. Many complain about the process of registering for the exam - the letters of recommendation, the documentation of training, and so forth. Plus, finding a proctor location can also be an issue. You can look at their list of certified examiners here. There are only 22 people on the list. Considering the previous post, and the NAS report's admonition about the need for certified analysts, 22 examiners does not even come close to addressing the need. Then, you can see that there are only 9 new examiners since the original group passed through the gates. That's not sustainable.
At $100, the ETA-i's AVFA is affordable. Many agencies do not support their employees in certification. Mine didn't. Everything I did for LEVA, NaTIA, the IRIT, FISWG, and the OSAC, I did on my own time (vacation / comp time). My decision to not get LEVA certification was as much financial as it was practical - I couldn't afford to go out of pocket on certification given the fact that my employer didn't care if I was certified or not. It wasn't a condition of employment and California does not require it for testimony.
Back to the experience.
I paid my fees to the ETA-i and to the testing center. I made an appointment to take the test and showed up a bit early to fill out the paperwork and sign all the forms. The testing center was just that, a quiet place full of cubicles and people taking a variety of tests. The proctor verified my identity. But the proctor, and the ETA-i, did not ask me about my employer or if I work primarily for "the defense." It shouldn't matter for whom I work or what types of cases I typically process. For the AVFA, the price is the price. No membership fees. No tiered pricing for government / non-government. No defense vs prosecution bias (in truth, I probably spent more time defending my self-insured public agency employer than working as a part of prosecuting criminal cases). That bias has always bugged me.
The test instrument was delivered on time and I had 2 hours to answer 100 questions. A comfortable amount of time for me, given my almost 2 decades of experience in the discipline. Upon completion, I notified the proctor who collected all of the materials and wished me well. I returned to my car and to my day. I wasn't stressed out at all. I was prepared to take the assessment and I was treated respectfully by all parties involved. In all, it was probably one of the more pleasant testing experiences I've experienced.
The testing center does not score the test. That's done back at the ETA-i offices. Thus, there's a bit of a delay in finding out if you've passed. For me, I had the results back in less than a day. I passed the test. In the next week or so, I'll receive the official letter notifying me of the results as well as the actual certificate. They also include a wallet card - perhaps necessary if confronted about one's status.
In thinking back over this whole experience, the ETA-i is separate from all of the training that I've received over the years, as is the testing center. Thus, it seems to comply with the ASTM's E2659 – 09. Standard Practice for Certiļ¬cate Programs. Their accreditation provider follows ISO/IEC 17024 in providing oversight of the ETA-i's programs. This for me is a big deal. I want to know that the certificate that I just earned is legitimate. Given my experiences with NaTIA and LEVA on retirement, I also want to know that in the transition from police service to private practice I'll be treated the same. I was treated with respect throughout the entire process with the ETA-i.
Given the reach of the ETA-i, as well as the availability of proctoring sites from companies like Prometric, the AVFA seems poised to help the NAS Report's recommendations come to fruition - if there's a certification in your discipline, you should have it. The AVFA's certification is well within the reach (financially) of the average analyst / examiner. It's certainly worth a look.
I started on the ETA-i web site. This is where one pays the exam fees, the first stop in the process. Simply select the "Pay Exam Fees" option in their store, select the exam, and pay the fee.
You can do a bit of scouting as to a testing location ahead of your purchase, and include that information in the purchasing process. It will speed things along a bit. I selected a Prometric testing close by. After I paid the exam fees, ETA-i sent my information and the exam to the testing center. I then paid the proctoring fees to the testing center. In total, I paid $120 in fees.
I can't emphasize this enough, I paid $120 in fees to take this certification exam.
If you're sitting for the LEVA CFVA board, you will have to get there (transportation costs) and stay there at least a few days (hotel and food). Likely, you'll want to stay the whole week and, assuming you pass, get your certificate at the Friday dinner. Assuming average travel costs, you or your agency will spend about $2000 for this experience. This $2000 comes after spending thousands upon thousands in the Levels courses and the electives.
For the IAI's CFVE, the cost of testing is significantly lower than LEVA. The IAI charges $400 for the initial test. Many complain about the process of registering for the exam - the letters of recommendation, the documentation of training, and so forth. Plus, finding a proctor location can also be an issue. You can look at their list of certified examiners here. There are only 22 people on the list. Considering the previous post, and the NAS report's admonition about the need for certified analysts, 22 examiners does not even come close to addressing the need. Then, you can see that there are only 9 new examiners since the original group passed through the gates. That's not sustainable.
At $100, the ETA-i's AVFA is affordable. Many agencies do not support their employees in certification. Mine didn't. Everything I did for LEVA, NaTIA, the IRIT, FISWG, and the OSAC, I did on my own time (vacation / comp time). My decision to not get LEVA certification was as much financial as it was practical - I couldn't afford to go out of pocket on certification given the fact that my employer didn't care if I was certified or not. It wasn't a condition of employment and California does not require it for testimony.
Back to the experience.
I paid my fees to the ETA-i and to the testing center. I made an appointment to take the test and showed up a bit early to fill out the paperwork and sign all the forms. The testing center was just that, a quiet place full of cubicles and people taking a variety of tests. The proctor verified my identity. But the proctor, and the ETA-i, did not ask me about my employer or if I work primarily for "the defense." It shouldn't matter for whom I work or what types of cases I typically process. For the AVFA, the price is the price. No membership fees. No tiered pricing for government / non-government. No defense vs prosecution bias (in truth, I probably spent more time defending my self-insured public agency employer than working as a part of prosecuting criminal cases). That bias has always bugged me.
The test instrument was delivered on time and I had 2 hours to answer 100 questions. A comfortable amount of time for me, given my almost 2 decades of experience in the discipline. Upon completion, I notified the proctor who collected all of the materials and wished me well. I returned to my car and to my day. I wasn't stressed out at all. I was prepared to take the assessment and I was treated respectfully by all parties involved. In all, it was probably one of the more pleasant testing experiences I've experienced.
The testing center does not score the test. That's done back at the ETA-i offices. Thus, there's a bit of a delay in finding out if you've passed. For me, I had the results back in less than a day. I passed the test. In the next week or so, I'll receive the official letter notifying me of the results as well as the actual certificate. They also include a wallet card - perhaps necessary if confronted about one's status.
In thinking back over this whole experience, the ETA-i is separate from all of the training that I've received over the years, as is the testing center. Thus, it seems to comply with the ASTM's E2659 – 09. Standard Practice for Certiļ¬cate Programs. Their accreditation provider follows ISO/IEC 17024 in providing oversight of the ETA-i's programs. This for me is a big deal. I want to know that the certificate that I just earned is legitimate. Given my experiences with NaTIA and LEVA on retirement, I also want to know that in the transition from police service to private practice I'll be treated the same. I was treated with respect throughout the entire process with the ETA-i.
Given the reach of the ETA-i, as well as the availability of proctoring sites from companies like Prometric, the AVFA seems poised to help the NAS Report's recommendations come to fruition - if there's a certification in your discipline, you should have it. The AVFA's certification is well within the reach (financially) of the average analyst / examiner. It's certainly worth a look.
Wednesday, May 29, 2019
The AVFA Certification
On November 22, 2005, the Science, State, Justice, Commerce, and Related Agencies Appropriations Act of 2006 became law. Under the terms of the statute, Congress authorized “the National Academy of Sciences to conduct a study on forensic science, as described in the Senate report."
In 2009, the National Academies Press published Strengthening Forensic Science in the United States: A Path Forward (aka, the NAS Report). The report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
On page 208 of the NAS Report, the authors address the issue of certification.
"The certification of individuals complements the accreditation of laboratories for a total quality assurance program. In other realms of science and technology, professionals, including nurses, physicians, professional engineers, and some laboratorians, typically must be certified before they can practice. The same should be true for forensic scientists who practice and testify. Although the accreditation process primarily addresses the management system, technical methods, and quality of the work of a laboratory (which includes the education and training of staff), certification is a process specifically designed to ensure the competency of the individual examiner."
The NAS Report goes on to note, "In addition to improving quality, certification programs can enhance the credibility of certificate holders."
As I noted last week, there are three certifications available in the discipline of digital / multimedia analysis. I would like to correct that here first, leaving the original post unedited.
Of the domains of what is traditionally considered Image Analysis / Video Analysis, there is no specific component within the LEVA / IAI that addresses the testing of a practitioner's competency in authentication. Photogrammetry is sometimes covered at their conventions, but there again, it's not a specified component of the certification programs.
I can say this with certainty about the LEVA certification. One may stand for certification upon successfully completing the LEVA Levels courses, the Courtroom Prep class, and some electives. For the exam, the analyst chooses a case that they've worked and presents it to the certification board. Thus, what is covered in that "exam" is unique to the analyst - varying from analyst to analyst. As a certificate program, it seems quite out of alignment with ASTM E2659 – 09, Standard Practice for Certificate Programs.
As to the domains under examination, there is no list available from either group. The IAI has a reading list that consists of a few older texts as well as some SWGDE guidelines. It's assumed that the practitioner will gain proficiency in these areas via vendor training, attending workshops at conferences, and other training opportunities.
What's missing: audio.
In Tx vs Oliver, the LEVA CFVA's testimony notes the importance of the audio track of the body-worn video recording, and how it helped in the analysts work. But, the CFVA exam does not specifically test for audio competency. I'm certainly not knocking the analyst. Just noting that the CFVA exam doesn't specifically test for competency in audio analysis and the analyst noted how important the audio track was in guiding his work and his conclusions.
Yes, forensic multimedia analysis includes everything in the multimedia container - audio, video, images, metadata, error, and junk. The CFVA - video analyst - seems to just cover the video portion of that list. Not necessarily a bad thing, but it's important to know what it does and doesn't cover.
Let's say you have an "image" from an iPhone - a Live Photo. Live Photos contain sound (link). Does your chosen software platform recognize that a Live Photo is essentially a video file? Does your chosen software platform have the ability to play the audio? What about "enhancing" the audio? Do you know how to enhance audio?
Here's where the AVFA certification has a leg up. Whilst the "competency requirements" aren't all that well written, and the format of the test (as noted on their site) does not match the language in the comps document, the analyst will definitely have an idea as to what will be on the test.
Yes, the AVFA exam includes audio. It's the first A in the acronym. The AVFA exam includes Photogrammetry and Authentication. There doesn't appear to be anything related to comparisons or content analysis, but those domains can sometimes be blended into the others, or generalized to "analysis."
Back to the NAS Report, "In essence, ‘certification’ usually means that a particular individual has completed a defined course of education, training, and experience, and has passed an examination prepared by peers which demonstrates that the individual has obtained at least the minimum level of competence required to practice the specific discipline." It seems that the AVFA meets that definition.
I'm going to take the test soon. I'll let you know how that goes as well as comment on the whole process - from sign-up to notification of results and beyond.
In 2009, the National Academies Press published Strengthening Forensic Science in the United States: A Path Forward (aka, the NAS Report). The report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
On page 208 of the NAS Report, the authors address the issue of certification.
"The certification of individuals complements the accreditation of laboratories for a total quality assurance program. In other realms of science and technology, professionals, including nurses, physicians, professional engineers, and some laboratorians, typically must be certified before they can practice. The same should be true for forensic scientists who practice and testify. Although the accreditation process primarily addresses the management system, technical methods, and quality of the work of a laboratory (which includes the education and training of staff), certification is a process specifically designed to ensure the competency of the individual examiner."
The NAS Report goes on to note, "In addition to improving quality, certification programs can enhance the credibility of certificate holders."
As I noted last week, there are three certifications available in the discipline of digital / multimedia analysis. I would like to correct that here first, leaving the original post unedited.
Of the domains of what is traditionally considered Image Analysis / Video Analysis, there is no specific component within the LEVA / IAI that addresses the testing of a practitioner's competency in authentication. Photogrammetry is sometimes covered at their conventions, but there again, it's not a specified component of the certification programs.
I can say this with certainty about the LEVA certification. One may stand for certification upon successfully completing the LEVA Levels courses, the Courtroom Prep class, and some electives. For the exam, the analyst chooses a case that they've worked and presents it to the certification board. Thus, what is covered in that "exam" is unique to the analyst - varying from analyst to analyst. As a certificate program, it seems quite out of alignment with ASTM E2659 – 09, Standard Practice for Certificate Programs.
As to the domains under examination, there is no list available from either group. The IAI has a reading list that consists of a few older texts as well as some SWGDE guidelines. It's assumed that the practitioner will gain proficiency in these areas via vendor training, attending workshops at conferences, and other training opportunities.
What's missing: audio.
In Tx vs Oliver, the LEVA CFVA's testimony notes the importance of the audio track of the body-worn video recording, and how it helped in the analysts work. But, the CFVA exam does not specifically test for audio competency. I'm certainly not knocking the analyst. Just noting that the CFVA exam doesn't specifically test for competency in audio analysis and the analyst noted how important the audio track was in guiding his work and his conclusions.
Yes, forensic multimedia analysis includes everything in the multimedia container - audio, video, images, metadata, error, and junk. The CFVA - video analyst - seems to just cover the video portion of that list. Not necessarily a bad thing, but it's important to know what it does and doesn't cover.
Let's say you have an "image" from an iPhone - a Live Photo. Live Photos contain sound (link). Does your chosen software platform recognize that a Live Photo is essentially a video file? Does your chosen software platform have the ability to play the audio? What about "enhancing" the audio? Do you know how to enhance audio?
Here's where the AVFA certification has a leg up. Whilst the "competency requirements" aren't all that well written, and the format of the test (as noted on their site) does not match the language in the comps document, the analyst will definitely have an idea as to what will be on the test.
Yes, the AVFA exam includes audio. It's the first A in the acronym. The AVFA exam includes Photogrammetry and Authentication. There doesn't appear to be anything related to comparisons or content analysis, but those domains can sometimes be blended into the others, or generalized to "analysis."
Back to the NAS Report, "In essence, ‘certification’ usually means that a particular individual has completed a defined course of education, training, and experience, and has passed an examination prepared by peers which demonstrates that the individual has obtained at least the minimum level of competence required to practice the specific discipline." It seems that the AVFA meets that definition.
I'm going to take the test soon. I'll let you know how that goes as well as comment on the whole process - from sign-up to notification of results and beyond.
Monday, May 27, 2019
When the data supports no conclusion, just say so
I can't quantify the amount of requests I've received over the years where investigators have asked me to resolve a few pixels or blocks into a license plate or other identifying item. If, at the Content Triage step, nominal resolution isn't sufficient to answer the question, I say so.
When the data supports no conclusion, I try to quantify why. I usually note the nominal resolution and / or the particular defect that may be getting in the way - blur, obstruction, etc. What I don't do is equivocate. Quite the opposite, I try to be very specific as to why the question can't be answered. In this way, there is no ambiguity in my conclusion(s).
Additionally, whomever is responsible for the source of the evidence will have insight as to potential improvements to the situation. For example, if the question is "what is license plate," and the camera is positioned to monitor a parking lot, then the person / company responsible for the security infrastructure can be alerted to the potential need for additional coverage in the area of interest. This could mean additional cameras, or a change in lensing, or ...
Why this topic today?
I received a link to this article in my inbox. "Expert says Merritt truck ‘cannot be excluded’ as vehicle on McStay neighborhood video." Really? "Cannot be excluded?" What does, "cannot be excluded" mean?
At issue is a 2000 Chevy work truck, similar to the one below. Chevy makes some of the most popular trucks in the US, second only to Ford. This case takes place in Southern California, home to more than 20m people from Kern to San Diego counties.
"Cannot be excluded" is not a conclusion, it's an equivocation. Search CarFax.com for used Chevy Silverado 3500 HD trucks for sale within a 100 mile radius of Fallbrook, CA (92028). I did. I found 49 trucks for sale on that site. 49 trucks that "cannot be excluded" ... AutoTrader listed 277 trucks for sale. 277 more trucks that "cannot be excluded" ...
All of this requires us to ask a question, is the goal of a comparative analysis to "exclude" or to "include?" How would you know if you've never taken a course in comparative analysis? Perhaps we can start with the SWGDE Best Practices for Photographic Comparison for All Disciplines. Version: 1.1 (July 18, 2017) (link):
Class Characteristic – A feature of an object that is common to a group of objects.
Individualizing Characteristic – A feature of an object that contributes to differentiating that object from others of its class.
5.2 Examine the photographs to determine if they are sufficient quality to complete an examination, and if the quality will have an effect on the degree to which an examination can be completed. Specific disciplines should define quality criteria, when possible, and how a failure to meet the specified quality criterion will impact results. (This may apply to a portion of the image, or the image as a whole.)
5.2.1 If the specified quality criteria are not met, determine if it is possible to obtain additional images. If the specified quality criteria are not met, and additional images cannot be obtained, this may preclude the examiner from conducting an examination, or the results of the examination may be limited.
5.3 Enhance images as necessary. Refer to ASTM Guide E2825 for Forensic Digital Image Processing.
These steps are the essence of the Content Triage step in the workflow - do I have enough nominal resolution to continue processing and reach a conclusion?
But, there is more to this process than just a comparison of a "known" and an "unknown." How does one go from "unknown" to a "known" for a comparison? How do you "know" what is "known?" First, you must attempt a Vehicle Make / Model Determination of the vehicle in the CCTV footage.
For a Vehicle Make / Model Determination, the SWGDE Vehicle Make/Model Comparison Form. Version: 1.0 (July 11, 2018) (source) is quite helpful.
How many features are shared between model years in a specific manufacturer's product line? Class Characteristics can help get you to "truck," then to "work truck" (presence of exterior cargo containers not typically present in a basic pickup truck), then to "make" based on shapes and positions of features of the items found in the Comparison form. The form can be used to document your findings.
You may get to Make, but getting to Model in low resolution images and video can be frustrating. What's the difference between a Chevy Silverado 1500, 1500LD, 2500HD, 3500HD? There are more than 10 trim variations of the 1500 series alone. What's the difference between a 2500HD and a 3500HD?
After you've documented your process of going from "object" to "work truck" to a specific model of work truck, how do you move beyond class, to make, to model, to year, to a specific truck? Remember, an Individualizing Characteristic is a feature of an object that contributes to differentiating that object from others of its class. Before you say "headlight spread pattern," please know that there is no valid research supporting "headlight spread pattern" as an individualizing characteristic - NONE. I know that there are cases where this technique has been used, but rhetoric is not science. Many jurisdictions, such as California and Georgia, will allow just about everything in at trial, so not having one's testimony excluded at trial is not proof of anything scientific.
Taking your CCTV footage, you've made your make / model / year determination using the SWGDE's form. Now, how do you move to an individual truck?
This is where basic statistics and inferential reasoning are quite necessary. Do you have sufficient nominal resolution to pick out identifying characteristics in the footage? If not, you're done. The data supports no conclusion as to individualization.
But assuming that you do, how do you work scientifically and as bias free as possible? Unpack the biasing information that you received from your "client" and design an experiment. In the US, given our Constitutional provisions that the accused are innocent until proven guilty, it is for the prosecution to prove guilt. Thus, the staring point for your experiment is that the truck in question is not a match. With sufficient nominal resolution, you set about to prove that there is a match. If you can't, there is no match as far as you're concerned. Remember, the comparative analysis should not be influenced by any other factors or items of evidence.
In designing the experiment, you'll need a sample set of images. You see, a simple "match / no match" comparison needs an adequate sample. It perverts the course of justice to simply attempt the comparison on the accused's vehicle. We don't do witness ID line-ups with just the suspect. Neither should anyone attempt a comparison with just a single "unknown" image - the accused's. Yes, I do use this specific provision of English Common Law to explain the problem here. Perverting the Course of Justice can be any of three acts, fabricating or disposing of evidence, intimidating or threatening a witness or juror, intimidating or threatening a judge. In this case, one Perverts the Course of Justice when one fabricates a conclusion (scientific evidence) where none is possible.
Back to the experiment. How many "unknown" images would you need to approach 99% confidence in your results, thus assisting the course of justice? Answer = 52. How did I come up with 52?
Exact - • Generic binomial test
Analysis: A priori: Compute required sample size
Input: Tail(s) = One
Proportion p2 = 0.8
Ī± err prob = 0.01
Power (1-Ī² err prob) = .99
Proportion p1 = 0.5
Output: Lower critical N = 35.0000000
Upper critical N = 35.0000000
Total sample size = 52
Actual power = 0.9901396
Actual Ī± = 0.008766618
A generic binomial test is similar to the flip of a coin - only two possible outcomes, heads / tails or match / no match. It's the simplest test to perform.
The error probability is your chance of being wrong. At 52 test images, you've got a 1 in 100 chance of being wrong (.99). As you move below 15 test images, you have a greater chance of being wrong than being right. With a sample size of 1, you're likely more accurate tossing a coin.
The 52 samples help us to get to make / model / year. You may chose to refresh those samples with new ones to perform a "blind comparison," and attempt to "include" the suspect's vehicle in your findings. To do this, you'd need the specific description of the "known" vehicle that makes it unique vs the others in the sample.
If I were performing a make / model / year determination, and then a comparison, I would note any errors or limitations in my report. If the data supported no conclusion, or if the limitations in the data prevented me from arriving at a determination, I would note that the data supported no conclusion. If I was able to make a determination, I would have noted my process and how I arrived at the conclusion (in a reliable, valid, and reproducible fashion).
The problem with the reporting of the case is the "cannot be excluded" portion is in the headline. One has to read deeper into the article to find, "... Liscio denied (that his conclusions may have been formed to fit the bias of the prosecution, who was paying him...), and reminded McGee more than once that he had not identified the truck specifically as Merritt’s..."
Which requires another question be asked, if the analyst had not identified the vehicle, what was he doing there in testimony?
"Among the items that helped to reach the conclusion that the vehicle was “consistent” with Merritt’s truck was a glint caught by the video that matched the position of a latch on a passenger-side storage box toward the rear of the truck, said Liscio,who uses 3D imagery."
Here we move from "cannot be excluded" to "consistent with," another equivocation. How does one not identify a vehicle, but find that said unknown vehicle is "consistent with" the "known" vehicle? This is the problem with Demonstrative Comparisons. When you place a single "known" against a single "unknown" in a demonstrative exhibit, you are making a choice as to what to include in your exhibit - thus you have concluded.
Back to the demonstrative. What is it about the latch on the side of the truck that is unique? Won't all work trucks of this type have latches on their cargo containers? Why is this one so special that it can only be found on the accused's truck? Of these questions, the article does not give an answer.
"I’m not saying that this your client’s vehicle,” Liscio repeated. “All I am saying is that the vehicle in question is consistent with my report, and if there is another vehicle that looks similar, that is possible.” How about at least 326 vehicles found on just two used car web sites?
If you'd like to explore these topics in depth, I'd invite you to sign up for any one (or all) of our upcoming training sessions. Our Statistics for Forensic Analysts course is offered on-line as micro learning and thus enrollment can happen at your convenience. Our other courses can be facilitated at your location or at ours, in Henderson, NV.
When the data supports no conclusion, I try to quantify why. I usually note the nominal resolution and / or the particular defect that may be getting in the way - blur, obstruction, etc. What I don't do is equivocate. Quite the opposite, I try to be very specific as to why the question can't be answered. In this way, there is no ambiguity in my conclusion(s).
Additionally, whomever is responsible for the source of the evidence will have insight as to potential improvements to the situation. For example, if the question is "what is license plate," and the camera is positioned to monitor a parking lot, then the person / company responsible for the security infrastructure can be alerted to the potential need for additional coverage in the area of interest. This could mean additional cameras, or a change in lensing, or ...
Why this topic today?
I received a link to this article in my inbox. "Expert says Merritt truck ‘cannot be excluded’ as vehicle on McStay neighborhood video." Really? "Cannot be excluded?" What does, "cannot be excluded" mean?
At issue is a 2000 Chevy work truck, similar to the one below. Chevy makes some of the most popular trucks in the US, second only to Ford. This case takes place in Southern California, home to more than 20m people from Kern to San Diego counties.
"Cannot be excluded" is not a conclusion, it's an equivocation. Search CarFax.com for used Chevy Silverado 3500 HD trucks for sale within a 100 mile radius of Fallbrook, CA (92028). I did. I found 49 trucks for sale on that site. 49 trucks that "cannot be excluded" ... AutoTrader listed 277 trucks for sale. 277 more trucks that "cannot be excluded" ...
All of this requires us to ask a question, is the goal of a comparative analysis to "exclude" or to "include?" How would you know if you've never taken a course in comparative analysis? Perhaps we can start with the SWGDE Best Practices for Photographic Comparison for All Disciplines. Version: 1.1 (July 18, 2017) (link):
Class Characteristic – A feature of an object that is common to a group of objects.
Individualizing Characteristic – A feature of an object that contributes to differentiating that object from others of its class.
5.2 Examine the photographs to determine if they are sufficient quality to complete an examination, and if the quality will have an effect on the degree to which an examination can be completed. Specific disciplines should define quality criteria, when possible, and how a failure to meet the specified quality criterion will impact results. (This may apply to a portion of the image, or the image as a whole.)
5.2.1 If the specified quality criteria are not met, determine if it is possible to obtain additional images. If the specified quality criteria are not met, and additional images cannot be obtained, this may preclude the examiner from conducting an examination, or the results of the examination may be limited.
5.3 Enhance images as necessary. Refer to ASTM Guide E2825 for Forensic Digital Image Processing.
These steps are the essence of the Content Triage step in the workflow - do I have enough nominal resolution to continue processing and reach a conclusion?
But, there is more to this process than just a comparison of a "known" and an "unknown." How does one go from "unknown" to a "known" for a comparison? How do you "know" what is "known?" First, you must attempt a Vehicle Make / Model Determination of the vehicle in the CCTV footage.
For a Vehicle Make / Model Determination, the SWGDE Vehicle Make/Model Comparison Form. Version: 1.0 (July 11, 2018) (source) is quite helpful.
How many features are shared between model years in a specific manufacturer's product line? Class Characteristics can help get you to "truck," then to "work truck" (presence of exterior cargo containers not typically present in a basic pickup truck), then to "make" based on shapes and positions of features of the items found in the Comparison form. The form can be used to document your findings.
You may get to Make, but getting to Model in low resolution images and video can be frustrating. What's the difference between a Chevy Silverado 1500, 1500LD, 2500HD, 3500HD? There are more than 10 trim variations of the 1500 series alone. What's the difference between a 2500HD and a 3500HD?
After you've documented your process of going from "object" to "work truck" to a specific model of work truck, how do you move beyond class, to make, to model, to year, to a specific truck? Remember, an Individualizing Characteristic is a feature of an object that contributes to differentiating that object from others of its class. Before you say "headlight spread pattern," please know that there is no valid research supporting "headlight spread pattern" as an individualizing characteristic - NONE. I know that there are cases where this technique has been used, but rhetoric is not science. Many jurisdictions, such as California and Georgia, will allow just about everything in at trial, so not having one's testimony excluded at trial is not proof of anything scientific.
Taking your CCTV footage, you've made your make / model / year determination using the SWGDE's form. Now, how do you move to an individual truck?
This is where basic statistics and inferential reasoning are quite necessary. Do you have sufficient nominal resolution to pick out identifying characteristics in the footage? If not, you're done. The data supports no conclusion as to individualization.
But assuming that you do, how do you work scientifically and as bias free as possible? Unpack the biasing information that you received from your "client" and design an experiment. In the US, given our Constitutional provisions that the accused are innocent until proven guilty, it is for the prosecution to prove guilt. Thus, the staring point for your experiment is that the truck in question is not a match. With sufficient nominal resolution, you set about to prove that there is a match. If you can't, there is no match as far as you're concerned. Remember, the comparative analysis should not be influenced by any other factors or items of evidence.
In designing the experiment, you'll need a sample set of images. You see, a simple "match / no match" comparison needs an adequate sample. It perverts the course of justice to simply attempt the comparison on the accused's vehicle. We don't do witness ID line-ups with just the suspect. Neither should anyone attempt a comparison with just a single "unknown" image - the accused's. Yes, I do use this specific provision of English Common Law to explain the problem here. Perverting the Course of Justice can be any of three acts, fabricating or disposing of evidence, intimidating or threatening a witness or juror, intimidating or threatening a judge. In this case, one Perverts the Course of Justice when one fabricates a conclusion (scientific evidence) where none is possible.
Back to the experiment. How many "unknown" images would you need to approach 99% confidence in your results, thus assisting the course of justice? Answer = 52. How did I come up with 52?
Exact - • Generic binomial test
Analysis: A priori: Compute required sample size
Input: Tail(s) = One
Proportion p2 = 0.8
Ī± err prob = 0.01
Power (1-Ī² err prob) = .99
Proportion p1 = 0.5
Output: Lower critical N = 35.0000000
Upper critical N = 35.0000000
Total sample size = 52
Actual power = 0.9901396
Actual Ī± = 0.008766618
A generic binomial test is similar to the flip of a coin - only two possible outcomes, heads / tails or match / no match. It's the simplest test to perform.
The error probability is your chance of being wrong. At 52 test images, you've got a 1 in 100 chance of being wrong (.99). As you move below 15 test images, you have a greater chance of being wrong than being right. With a sample size of 1, you're likely more accurate tossing a coin.
The 52 samples help us to get to make / model / year. You may chose to refresh those samples with new ones to perform a "blind comparison," and attempt to "include" the suspect's vehicle in your findings. To do this, you'd need the specific description of the "known" vehicle that makes it unique vs the others in the sample.
If I were performing a make / model / year determination, and then a comparison, I would note any errors or limitations in my report. If the data supported no conclusion, or if the limitations in the data prevented me from arriving at a determination, I would note that the data supported no conclusion. If I was able to make a determination, I would have noted my process and how I arrived at the conclusion (in a reliable, valid, and reproducible fashion).
The problem with the reporting of the case is the "cannot be excluded" portion is in the headline. One has to read deeper into the article to find, "... Liscio denied (that his conclusions may have been formed to fit the bias of the prosecution, who was paying him...), and reminded McGee more than once that he had not identified the truck specifically as Merritt’s..."
Which requires another question be asked, if the analyst had not identified the vehicle, what was he doing there in testimony?
"Among the items that helped to reach the conclusion that the vehicle was “consistent” with Merritt’s truck was a glint caught by the video that matched the position of a latch on a passenger-side storage box toward the rear of the truck, said Liscio,who uses 3D imagery."
Here we move from "cannot be excluded" to "consistent with," another equivocation. How does one not identify a vehicle, but find that said unknown vehicle is "consistent with" the "known" vehicle? This is the problem with Demonstrative Comparisons. When you place a single "known" against a single "unknown" in a demonstrative exhibit, you are making a choice as to what to include in your exhibit - thus you have concluded.
Back to the demonstrative. What is it about the latch on the side of the truck that is unique? Won't all work trucks of this type have latches on their cargo containers? Why is this one so special that it can only be found on the accused's truck? Of these questions, the article does not give an answer.
"I’m not saying that this your client’s vehicle,” Liscio repeated. “All I am saying is that the vehicle in question is consistent with my report, and if there is another vehicle that looks similar, that is possible.” How about at least 326 vehicles found on just two used car web sites?
If you'd like to explore these topics in depth, I'd invite you to sign up for any one (or all) of our upcoming training sessions. Our Statistics for Forensic Analysts course is offered on-line as micro learning and thus enrollment can happen at your convenience. Our other courses can be facilitated at your location or at ours, in Henderson, NV.
Monday, May 20, 2019
What new certification program?
We had a lively discussion in last week's Advanced Processing Techniques class. This class seems to bring out these types of conversations.
We were discussing certifications in our discipline: digital / multimedia analysis. When I said that I know of three, the students seemed puzzled. Three? They were aware of LEVA's CFVT and CFVA programs and the IAI's CFVE program. LEVA counts as one, the IAI, two. Where's three?
Whilst you were not paying attention, a group of analysts got together with the Electronics Technicians Association, International, to create the Audio Video Forensic Analyst (AVFA) certification. Hat's off to them for pulling that one off.
Who is the ETA-i? What is the AVFA? Let's take a look.
The ETA-i is a legit organization that certifies electricians, cable splicers, and other comms techs. It's been around since 1978 and has over 90 certification programs. I spoke with the office manager there about the AVFA certification and received an interesting answer about it's genesis. I was told that a few years ago, they began an expansion into other related industries. They added a Digital Video (DVE) Editor certification. From there, they added the AVFA certification. Some of those involved in the DVE certification were involved in the creation of the AVFA program.
Before you grab your pitch fork and start marching towards the town square, let's take a quick look at the program and compare it to what we know about the existing market for certifications.
Validity is defined as the extent to which an assessment accurately measures what it is intended to measure. Validity refers to the accuracy of measurement. In this case, the assessment is the certification exam. Determining validity can be viewed as constructing an evidence-based argument regarding how well a tool measures what it is supposed to do. Evidence can be found in content, response process, relationships to other variables, and consequences (source).
Reliability refers to whether an assessment instrument gives the same results each time it is used in the same setting with the same type of subjects. For test/retest reliability, the test should give the same results each time, assuming there are no interval changes in what you are measuring, and they are often measured as correlation.
Face Validity - the extent to which a test is subjectively viewed as covering the concept it purports to measure.
Is the AVFA reliable? As a multiple choice exam, it is certainly possible. One would hope that it's written in such a way that you'd answer the same questions in the same way on successive administrations of the test.
Does the AVFA certification test have face validity? You can find out what it's supposed to measure on it's web site (link). You might look at the competencies sheet and note that there are programs listed that you're likely not familiar with. But, every analysis requires a platform. The creators of the AVFA had to choose tools and terminology. They seem to have chosen freeware and sources of information not hidden behind paywalls. Good for them. The ASTM's glossary is great, but how much do you have to pay to use it?
You can sign up and take the test yourself. No membership required. The test is affordable ($100 + proctor fees). The test is entirely multiple choice. Pass it, you're certified. Fail it, re-take it for free. But, on the face of it (pun intended), it seems to have face validity.
Accredited? Yes. The program at ETA-i is accredited by International Certification Accreditation Council (ICAC). Who are they? I didn't know either, so I called them as well. I had a nice and long conversation with a representative who explained their program and their long relationship with the ETA-i. According to them, they follow ISO 17011 / 17024 in the administration of their accreditation program. So far so good.
What's the deal with accreditation? One of the things that ASTM E2659-09 as well as ISO 17011 / 17024 emphasize is the separation of the training event / training provider and the certifying body. There is a risk of bias if the training provider also runs the certification process - as in teaching the test. This is a large part of why you take Photoshop classes from me, George, Dorothy, Scott Kelby, and etc. ... then trek on over to ProMetric to take the Adobe Certified Expert exam.
This contrasts with LEVA, as an example. LEVA is both training provider and certifying body. As such, it does not follow or comply with ASTM E2659-09 as well as ISO 17011 / 17024 in the design of its program. Thus, it's program is not accredited. There's no way to simply sit for an objective certification exam with LEVA. You must take their classes, all of them, in order to sit for the test. Additionally, it's impossible to judge the reliability or validity of the certification exam as all LEVA certification exams are unique experiences - and are closed to the public. I'm not saying that there's no value to the experience, just that it's not valid from a design / standards standpoint.
The IAI is in the middle. I'm aware that they are seeking accreditation for their programs. They are not a training provider, per se. They do offer training at their conferences, but you can't call that "training to competency." It's more like information sessions, as the ASTM defines training events. Thus, it is possible for the IAI to get their programs accredited. They should, however, remove the discriminatory membership requirement in order to sit for their test.
Thus, as someone who is trained / educated in curriculum and instructional design, who has created many assessments, and has reviewed countless others, I'm excited to see how this new program shakes out. I'll likely sit for the exam soon, just to see what it's like.
If you decide to take the test, let me know what you think of it. Try to go into the process with an open mind. I'm a big fan of democratizing science. If this is a legit certification, I'm all for it.
We were discussing certifications in our discipline: digital / multimedia analysis. When I said that I know of three, the students seemed puzzled. Three? They were aware of LEVA's CFVT and CFVA programs and the IAI's CFVE program. LEVA counts as one, the IAI, two. Where's three?
Whilst you were not paying attention, a group of analysts got together with the Electronics Technicians Association, International, to create the Audio Video Forensic Analyst (AVFA) certification. Hat's off to them for pulling that one off.
Who is the ETA-i? What is the AVFA? Let's take a look.
The ETA-i is a legit organization that certifies electricians, cable splicers, and other comms techs. It's been around since 1978 and has over 90 certification programs. I spoke with the office manager there about the AVFA certification and received an interesting answer about it's genesis. I was told that a few years ago, they began an expansion into other related industries. They added a Digital Video (DVE) Editor certification. From there, they added the AVFA certification. Some of those involved in the DVE certification were involved in the creation of the AVFA program.
Before you grab your pitch fork and start marching towards the town square, let's take a quick look at the program and compare it to what we know about the existing market for certifications.
Validity is defined as the extent to which an assessment accurately measures what it is intended to measure. Validity refers to the accuracy of measurement. In this case, the assessment is the certification exam. Determining validity can be viewed as constructing an evidence-based argument regarding how well a tool measures what it is supposed to do. Evidence can be found in content, response process, relationships to other variables, and consequences (source).
Reliability refers to whether an assessment instrument gives the same results each time it is used in the same setting with the same type of subjects. For test/retest reliability, the test should give the same results each time, assuming there are no interval changes in what you are measuring, and they are often measured as correlation.
Face Validity - the extent to which a test is subjectively viewed as covering the concept it purports to measure.
Is the AVFA reliable? As a multiple choice exam, it is certainly possible. One would hope that it's written in such a way that you'd answer the same questions in the same way on successive administrations of the test.
Does the AVFA certification test have face validity? You can find out what it's supposed to measure on it's web site (link). You might look at the competencies sheet and note that there are programs listed that you're likely not familiar with. But, every analysis requires a platform. The creators of the AVFA had to choose tools and terminology. They seem to have chosen freeware and sources of information not hidden behind paywalls. Good for them. The ASTM's glossary is great, but how much do you have to pay to use it?
You can sign up and take the test yourself. No membership required. The test is affordable ($100 + proctor fees). The test is entirely multiple choice. Pass it, you're certified. Fail it, re-take it for free. But, on the face of it (pun intended), it seems to have face validity.
Accredited? Yes. The program at ETA-i is accredited by International Certification Accreditation Council (ICAC). Who are they? I didn't know either, so I called them as well. I had a nice and long conversation with a representative who explained their program and their long relationship with the ETA-i. According to them, they follow ISO 17011 / 17024 in the administration of their accreditation program. So far so good.
What's the deal with accreditation? One of the things that ASTM E2659-09 as well as ISO 17011 / 17024 emphasize is the separation of the training event / training provider and the certifying body. There is a risk of bias if the training provider also runs the certification process - as in teaching the test. This is a large part of why you take Photoshop classes from me, George, Dorothy, Scott Kelby, and etc. ... then trek on over to ProMetric to take the Adobe Certified Expert exam.
This contrasts with LEVA, as an example. LEVA is both training provider and certifying body. As such, it does not follow or comply with ASTM E2659-09 as well as ISO 17011 / 17024 in the design of its program. Thus, it's program is not accredited. There's no way to simply sit for an objective certification exam with LEVA. You must take their classes, all of them, in order to sit for the test. Additionally, it's impossible to judge the reliability or validity of the certification exam as all LEVA certification exams are unique experiences - and are closed to the public. I'm not saying that there's no value to the experience, just that it's not valid from a design / standards standpoint.
The IAI is in the middle. I'm aware that they are seeking accreditation for their programs. They are not a training provider, per se. They do offer training at their conferences, but you can't call that "training to competency." It's more like information sessions, as the ASTM defines training events. Thus, it is possible for the IAI to get their programs accredited. They should, however, remove the discriminatory membership requirement in order to sit for their test.
Thus, as someone who is trained / educated in curriculum and instructional design, who has created many assessments, and has reviewed countless others, I'm excited to see how this new program shakes out. I'll likely sit for the exam soon, just to see what it's like.
If you decide to take the test, let me know what you think of it. Try to go into the process with an open mind. I'm a big fan of democratizing science. If this is a legit certification, I'm all for it.
Thursday, May 16, 2019
The need for objective measures in forensic evidence
The latest edition of Significance, the journal of the Royal Statistical Society, features an article on the importance of objective measures and standards in evaluating evidence. Whilst many of the points made by the author are valid and applicable, practitioners within the digital / multimedia forensic sciences already have standards and guidelines for basing their work on an objective footing.
Investigators want to use video evidence to "identify" an object in a video - gun, knife, etc. But this "identification" is often complicated by a lack of nominal resolution in the region of interest. Many will simply state that the object has been identified, without actually taking the steps to formally conclude (prove) their assertions. They go with what they "know" (abductive reasoning), as opposed to what they can prove. There's a better way.
First off, here are some relevant definitions from the SWGDE Digital & Multimedia Evidence Glossary. Version: 3.0 (June 23, 2016)
Artifact: A visual/aural aberration in an image, video, or audio recording resulting from a technical or operational limitation. Examples include speckles in a scanned picture or “blocking” in images compressed using the JPEG standard.
Image Analysis: The application of image science and domain expertise to examine and interpret the content of an image, the image itself, or both in legal matters.
Image Comparison (Photographic Comparison): The process of comparing images of questioned objects or persons to known objects or persons or images thereof, and making an assessment
of the correspondence between features in these images for rendering an opinion regarding identification or elimination.
Image Content Analysis: The drawing of conclusions about an image. Targets for content analysis include, but are not limited to: the subjects/objects within an image; the conditions under which, or the process by which, the image was captured or created; the physical aspects of the scene (e.g., lighting or composition); and/or the provenance of the image.
Nominal resolution: The numerical value of pixels per inch as opposed to the achievable resolution of the imaging device. In the case of flatbed scanners, it is based on the resolution setting in the software controlling the scanner. In the case of digital cameras, this refers to the number of pixels of the camera sensor divided by the corresponding vertical and horizontal dimension of the area photographed.
Video Analysis: The scientific examination, comparison, and/or evaluation of video in legal matters.
It's also important to define "forensic science." For this, I'll refer to "A Framework to Harmonize Forensic Science Practices and Digital/Multimedia Evidence." OSAC Task Group on Digital/Multimedia Science. 2017: "Forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context."
What is a trace? "A trace is any modification, subsequently observable, resulting from an event. You walk within the view of a CCTV system, you leave a trace of your presence within that system."
SWGDE Best Practices for Photographic Comparison for All Disciplines. Version: 1.1 (July 18, 2017) provides a few more definitions.
Class Characteristic: A feature of an object that is common to a group of objects.
Individualizing Characteristic: A feature of an object that contributes to differentiating that object from others of its class.
With the definitions in mind, SWGDE Best Practices for Image Content Analysis. Version: 1.0 (February 21, 2017) provides the framework to begin the Content Triage step in the workflow - can I answer the question with the evidence file?
5. Evidence Preparation
5.2 Based on the request for examination, determine if submitted imagery is available to complete requested analysis. Determine whether submitted imagery is of sufficient quality to complete the requested examination, or if the image quality will have an effect on the degree to which an examination can be completed.
5.2.1 If the specified quality criteria are not met, determine if it is possible to obtain additional images. If the specified quality criteria are not met, and additional images cannot be obtained, this may preclude the examiner from conducting an examination, or the results of the examination may be limited.
(do I have sufficient nominal resolution within the target area to fulfill the request - answer the question? The nominal resolution, measured in pixels per inch or inches per pixel - or it's metric equivalent, is the objective measure.)
6. Examinations Method
There is no one specific methodology for content analysis. The methodology for analysis will primarily be derived to answer the requested examination. However, any methodology applied to content analysis should incorporate an analysis of the imagery, the cataloguing of relevant features, an evaluation of the significance of the detected features, an evaluation of the limiting factors of the imagery, the formation of a conclusion, and a verification of the analysis. The repeatability (and / or reproducibility) of the procedure and documentation of the workflow is of paramount importance. Documentation should be performed contemporaneously.
6.1 Assess the contents of the image, to determine whether factors are present that can answer the examination request. The examination request generally will fall into one of the following categories:
6.1.1 Analysis to determine the conditions under which, or the process by which, the image was captured or created. Examples include, but are not limited to, the limitations of the recording device, and the inclusion of artifacts based on the file format or compression. This can help to answer the question “How does the recording system affect what is visible in the scene?”
6.1.2 Analysis to determine the physical aspects of the scene, including events captured. Examples include, but are not limited to, the lighting and composition of the scene, the presence of specific objects within the scene, a determination of the interaction between objects in the scene, and a description of events within a scene. This can help to answer the questions “Is a specific object visible in the scene?” or “What happened in the scene?”
6.1.3 Analysis to determine the classification of an object within an image. Examples include, but are not limited to, the make, model, and year of a vehicle, the determination of a manufacturing logo, and the determination of the brand and model of a weapon. This can help to answer the question “What is the object visible in the scene?”
6.1.4 Analysis to determine the location or setting of the image content. Examples may include either a general setting (e.g. Portland, Oregon) or a specific setting (e.g. Conference Room 23, the Northwest Corner). This can help answer the question “Where is the scene?”
6.3 Assess the image for features that contribute to the ability to form a conclusion, and record observed features. Consider the weight or importance of identified features, in order to determine the focus of the examination. Examples of features may include logos, shapes, reflections, or specific items.
How does one move from "artifact" (6.1.1) to "object" (6.1.3)?
As regards “artifact:"
When nominal resolution is low, the analyst must be certain of the provenance of the features within the region of interest in the evidence item. Are the “items of interest” actual representations of things, or are they an “aberration” in the image, resulting from problems in compression and etc. In order to move from “a few dark blocks of pixels” is not an “artifact” to the "few dark blocks of pixels” is an “object” - the rules for classifying an object apply. Additionally, one must rule out the possibility that what one sees is not an artifact. This must be done in a systematic / coherent fashion in order to be a forensic science exercise.
As regards “object:"
In order to move from “a few dark blocks of pixels” that are not a result of an “aberration" to firearm, there needs to be Class Characteristics present in the view that would lead to the conclusion of "object." In order to form a conclusion, those Class Characteristics should be identified by the analyst - overall shape: the presence of features of an object that contributes to differentiating that object from others of its class Is the nominal resolution in the region of interest sufficient to conclude as to class and individualizing characteristics? This is the objective measure.
The standards and guidelines are there. It is left to the investigator, the analyst, the attorney, and the Trier of Fact to follow the rules and accept the facts as they present themselves. What does the investigator do when told by the analyst that the nominal resolution does not support a conclusion? Does he / she shop around for another analyst? The investigator must not fall victim to cognitive perseverance (bias) when faced with this new information.
Additionally, these stakeholders must understand when they're on sound scientific footing or when they're just taking their best shot.
Abductive reasoning: taking your best shot
Deductive reasoning: conclusion guaranteed
Inductive reasoning: conclusion merely likely
What most in the forensic sciences don't quite acknowledge is that they operate in the world of abductive reasoning most of the time.
From a US Constitution standpoint, from the standpoint of Constitutional Policing, it is for the side offering an item as “proof” of some condition (e.g., object = firearm) to actually prove class = firearm. It perverts the cause of justice to require the defense to prove that it isn’t.
This is the level of detail that we get into when I present training in the digital / multimedia forensic sciences, as well as my other educational offerings. If you need help on a case, or want a second opinion, feel free to contact me today.
Investigators want to use video evidence to "identify" an object in a video - gun, knife, etc. But this "identification" is often complicated by a lack of nominal resolution in the region of interest. Many will simply state that the object has been identified, without actually taking the steps to formally conclude (prove) their assertions. They go with what they "know" (abductive reasoning), as opposed to what they can prove. There's a better way.
First off, here are some relevant definitions from the SWGDE Digital & Multimedia Evidence Glossary. Version: 3.0 (June 23, 2016)
Artifact: A visual/aural aberration in an image, video, or audio recording resulting from a technical or operational limitation. Examples include speckles in a scanned picture or “blocking” in images compressed using the JPEG standard.
Image Analysis: The application of image science and domain expertise to examine and interpret the content of an image, the image itself, or both in legal matters.
Image Comparison (Photographic Comparison): The process of comparing images of questioned objects or persons to known objects or persons or images thereof, and making an assessment
of the correspondence between features in these images for rendering an opinion regarding identification or elimination.
Image Content Analysis: The drawing of conclusions about an image. Targets for content analysis include, but are not limited to: the subjects/objects within an image; the conditions under which, or the process by which, the image was captured or created; the physical aspects of the scene (e.g., lighting or composition); and/or the provenance of the image.
Nominal resolution: The numerical value of pixels per inch as opposed to the achievable resolution of the imaging device. In the case of flatbed scanners, it is based on the resolution setting in the software controlling the scanner. In the case of digital cameras, this refers to the number of pixels of the camera sensor divided by the corresponding vertical and horizontal dimension of the area photographed.
Video Analysis: The scientific examination, comparison, and/or evaluation of video in legal matters.
It's also important to define "forensic science." For this, I'll refer to "A Framework to Harmonize Forensic Science Practices and Digital/Multimedia Evidence." OSAC Task Group on Digital/Multimedia Science. 2017: "Forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context."
What is a trace? "A trace is any modification, subsequently observable, resulting from an event. You walk within the view of a CCTV system, you leave a trace of your presence within that system."
SWGDE Best Practices for Photographic Comparison for All Disciplines. Version: 1.1 (July 18, 2017) provides a few more definitions.
Class Characteristic: A feature of an object that is common to a group of objects.
Individualizing Characteristic: A feature of an object that contributes to differentiating that object from others of its class.
With the definitions in mind, SWGDE Best Practices for Image Content Analysis. Version: 1.0 (February 21, 2017) provides the framework to begin the Content Triage step in the workflow - can I answer the question with the evidence file?
5. Evidence Preparation
5.2 Based on the request for examination, determine if submitted imagery is available to complete requested analysis. Determine whether submitted imagery is of sufficient quality to complete the requested examination, or if the image quality will have an effect on the degree to which an examination can be completed.
5.2.1 If the specified quality criteria are not met, determine if it is possible to obtain additional images. If the specified quality criteria are not met, and additional images cannot be obtained, this may preclude the examiner from conducting an examination, or the results of the examination may be limited.
(do I have sufficient nominal resolution within the target area to fulfill the request - answer the question? The nominal resolution, measured in pixels per inch or inches per pixel - or it's metric equivalent, is the objective measure.)
6. Examinations Method
There is no one specific methodology for content analysis. The methodology for analysis will primarily be derived to answer the requested examination. However, any methodology applied to content analysis should incorporate an analysis of the imagery, the cataloguing of relevant features, an evaluation of the significance of the detected features, an evaluation of the limiting factors of the imagery, the formation of a conclusion, and a verification of the analysis. The repeatability (and / or reproducibility) of the procedure and documentation of the workflow is of paramount importance. Documentation should be performed contemporaneously.
6.1 Assess the contents of the image, to determine whether factors are present that can answer the examination request. The examination request generally will fall into one of the following categories:
6.1.1 Analysis to determine the conditions under which, or the process by which, the image was captured or created. Examples include, but are not limited to, the limitations of the recording device, and the inclusion of artifacts based on the file format or compression. This can help to answer the question “How does the recording system affect what is visible in the scene?”
6.1.2 Analysis to determine the physical aspects of the scene, including events captured. Examples include, but are not limited to, the lighting and composition of the scene, the presence of specific objects within the scene, a determination of the interaction between objects in the scene, and a description of events within a scene. This can help to answer the questions “Is a specific object visible in the scene?” or “What happened in the scene?”
6.1.3 Analysis to determine the classification of an object within an image. Examples include, but are not limited to, the make, model, and year of a vehicle, the determination of a manufacturing logo, and the determination of the brand and model of a weapon. This can help to answer the question “What is the object visible in the scene?”
6.1.4 Analysis to determine the location or setting of the image content. Examples may include either a general setting (e.g. Portland, Oregon) or a specific setting (e.g. Conference Room 23, the Northwest Corner). This can help answer the question “Where is the scene?”
6.3 Assess the image for features that contribute to the ability to form a conclusion, and record observed features. Consider the weight or importance of identified features, in order to determine the focus of the examination. Examples of features may include logos, shapes, reflections, or specific items.
How does one move from "artifact" (6.1.1) to "object" (6.1.3)?
As regards “artifact:"
When nominal resolution is low, the analyst must be certain of the provenance of the features within the region of interest in the evidence item. Are the “items of interest” actual representations of things, or are they an “aberration” in the image, resulting from problems in compression and etc. In order to move from “a few dark blocks of pixels” is not an “artifact” to the "few dark blocks of pixels” is an “object” - the rules for classifying an object apply. Additionally, one must rule out the possibility that what one sees is not an artifact. This must be done in a systematic / coherent fashion in order to be a forensic science exercise.
As regards “object:"
In order to move from “a few dark blocks of pixels” that are not a result of an “aberration" to firearm, there needs to be Class Characteristics present in the view that would lead to the conclusion of "object." In order to form a conclusion, those Class Characteristics should be identified by the analyst - overall shape: the presence of features of an object that contributes to differentiating that object from others of its class Is the nominal resolution in the region of interest sufficient to conclude as to class and individualizing characteristics? This is the objective measure.
The standards and guidelines are there. It is left to the investigator, the analyst, the attorney, and the Trier of Fact to follow the rules and accept the facts as they present themselves. What does the investigator do when told by the analyst that the nominal resolution does not support a conclusion? Does he / she shop around for another analyst? The investigator must not fall victim to cognitive perseverance (bias) when faced with this new information.
Additionally, these stakeholders must understand when they're on sound scientific footing or when they're just taking their best shot.
 |
| Types of Resoning |
Deductive reasoning: conclusion guaranteed
Inductive reasoning: conclusion merely likely
What most in the forensic sciences don't quite acknowledge is that they operate in the world of abductive reasoning most of the time.
From a US Constitution standpoint, from the standpoint of Constitutional Policing, it is for the side offering an item as “proof” of some condition (e.g., object = firearm) to actually prove class = firearm. It perverts the cause of justice to require the defense to prove that it isn’t.
This is the level of detail that we get into when I present training in the digital / multimedia forensic sciences, as well as my other educational offerings. If you need help on a case, or want a second opinion, feel free to contact me today.
Monday, May 13, 2019
Adobe Tells Users They Can Get Sued for Using Old Versions of Photoshop
It was just a month ago that I warned you to keep your installer files. Now this from Adobe:
"Users of older versions of Adobe Creative Cloud apps including Photoshop have been told to stop using them or face potential "infringement claims" from third-party companies who are unnamed but suspected to be Dolby. Adobe cites only "ongoing litigation" as the reason for the abrupt announcement."
"Adobe recently discontinued certain older versions of Creative Cloud applications. Customers using those versions have been notified that they are no longer licensed to use them and were provided guidance on how to upgrade to the latest authorized versions," said Adobe in a statement to AppleInsider."
"Unfortunately, customers who continue to use or deploy older, unauthorized versions of Creative Cloud may face potential claims of infringement by third parties. We cannot comment on claims of third-party infringement, as it concerns ongoing litigation."
"Instead, users are receiving the equivalent of a cease and desist email, informing them that the apps that they are using are discontinued."
If you're an individual user in a private lab, you may not become the focus of Adobe's attention. But, if you're a part of a large organization or government lab, I'll bet that Adobe will see the potential profit in coming after your agency.
Stay tuned ...
"Users of older versions of Adobe Creative Cloud apps including Photoshop have been told to stop using them or face potential "infringement claims" from third-party companies who are unnamed but suspected to be Dolby. Adobe cites only "ongoing litigation" as the reason for the abrupt announcement."
"Adobe recently discontinued certain older versions of Creative Cloud applications. Customers using those versions have been notified that they are no longer licensed to use them and were provided guidance on how to upgrade to the latest authorized versions," said Adobe in a statement to AppleInsider."
"Unfortunately, customers who continue to use or deploy older, unauthorized versions of Creative Cloud may face potential claims of infringement by third parties. We cannot comment on claims of third-party infringement, as it concerns ongoing litigation."
"Instead, users are receiving the equivalent of a cease and desist email, informing them that the apps that they are using are discontinued."
If you're an individual user in a private lab, you may not become the focus of Adobe's attention. But, if you're a part of a large organization or government lab, I'll bet that Adobe will see the potential profit in coming after your agency.
Stay tuned ...
Subscribe to:
Posts (Atom)