Accreditation and Certification Standards - the Holy Grail of Digital Forensics

They say that those who do not study history are doomed to repeat it. That sounds like "deja vu all over again." Despite the calls from certain factions that digital forensics develop, appoint, and/or anoint a body to oversee its development, we are no closer to having a solution than we were 5 years ago. I would like to say that this was due to some complex technical or impassable philosophical schism, but the sad truth is that we are mired in political turf wars, greedy entrepreneurs, and resume padders.

We as a collective discipline do not seem to understand that if we do not address this fundamental issue, we will have a solution thrust upon us. Historically the "thrusting" of regulations or administrative processes has not been beneficial to most groups. We can also look to other professions like the CPA's to find models that appear to work and have somehow overcome all of the so called challenges we are facing.

To date we have become dysfunctional and petty in our attempts to address how we are going to mature and police our own discipline. While the penultimate solution seems to be hidden from us, the stop gap solutions currently being attempted (numerous organizations and companies all claiming they alone speak for the science) are not solutions at all and only further complicate and alienate those of us in the community.

Maybe it is time we call our own detente, collectively sit down, look at the historical record of other forensic sciences and actually come up with a solution! Alas, I fear this will not happen soon as there is far too much money to be made in this area and it seems greed and ego stroking trumps all.

 

One Big Happy Family?

I have spent most of my academic career dealing with the problem of how do we combine the different digital forensic communities under one umbrella. I have observed, written, and advocated that we become a homogeneous group. The arguments followed that the different communities/constitutents (i.e., law enforcement, military, private sector/business and academia) shared a common goal and since the evidence that is digital in nature was our purview it was logical that we could identify our common ground and be able to develop standards, certifications, and professional ethics that were universal.

I now think I got it totally wrong. While the communities share some commonalities, the goals, motives, requirements are so vastly different and I will argue, diverging more each year, that to think we can have a community agnostic anything is false. I will use the private sector and e-discovery as a quick example. E-discovery is maturing and developing standards and processes that are unique to civil proceedings and the requirements of civil litigation. There is no necessity to make whole drive forensic images (and in some cases it is strictly forbidden by the discovery order). This is foreign to law enforcement and at odds with the basic tenet of imaging everything – in case we need it at a later date.

Futile efforts to develop a universal code of professional ethics further illiustrates the heterogeneity of the communities. In almost every case, articulated codes of ethics have run imto issues were it would be unethical for private sector practitiioner to follow, or practitiioner who worked for the defense as opposed to the prosecution. An example of such a clause is full disclosure of all findings ( I will leave it to the reader to think up examples that would run contrary to expected conduct of the practitioner examples I indicated).

We may be better served by developing certifications, standards, and codes of ethics that are community specific. I would never thought I would even be considering this, and many of my students would be very shocked by my makiing such an assertion. However, one cannot ignore the realities that seem to continually jump up and slap one in the face.

While this is just a thought experiment for me at the moment, it has some profound implications for the future of our scientific discipline and therefore needs to be seriously  considered and discussed.

Does Digital Forensics Suffer from Physics Envy?

For over a decade we have been fighting to have digitial forensics recognized as a science. Committees have been struck, organizations created and a great deal of blood sweat and tears has gone into having the American Academy of Forensic Sciences recognize us as a new section - Digital and Multimedia Sciences. We have also written numerous papers, conducted workshops and heavily marketed for the covetted recognition as a "SCIENCE".

What if we got it wrong? Can we really make a solid case for digital forensics being a science? The goal of science is the pursuit of knowledge. This is accomplished by using the scientific method or process. Theories are derived, hypotheses created and experiments designed to test these educated guesses. The interpretation of the findings are supposed to be value free and the results reproducable.

The goal of technology on the other hand is to meet the needs of some applied problem, focusing on some short term solution. The major processes include design, implementation, and testing. The deteremination of the success or failure is value ladden and reproducability of findings not necessarily criticial.

If we limit our discussion to the current state of digital forensics which category do we more easily fit into? It is really a no brainer - we are a technology that may at some point in the future move to a science, but we are not their yet.

The next important issue to contemplate is whether we actually have to become a science. Can we still serve our purpose and mandate (as well as the courts') by remaining a technology? Maybe we just have a case of Physics envy.

Who watches the watchers?

We are reaching what could be termed a tipping point in the development/ maturation of digital forensics. We are starting to see an increasing demand for certifications specific to digital forensics. This parallels the historical development of information security and assurance. As the demand for more secure networks and personnel trained to test and defend these systems increased, so to did the need for industry to gauge who was at least minimally qualified to claim they were professionals in this field. This demand spurred on a industry dedicated to certify professional information security professionals. Unfortunately a conflict of interest soon arose. The same companies that were certifying the professionals through some type of examination, were also selling the training and study guides for their tests. One of the cardinal rules of accreditation and certification was broken – no independent accreditation body was setup to provide oversight for the training and testing companies. The fox was and is guarding the hen house.

Digital forensics now has numerous certifying bodies which in fact are thinly disguised training and testing companies. Come take our training, read our guide, take our test and low and behold you are certified in digital forensics. The arguments put forward for using this model yet again is that it if it is good enough for information security then its good enough for us. But wait, we can claim a direct lineage to the forensic sciences. Given this context we can look to the world of forensics and determine what our sister sciences have done. The forensic sciences seem to be very sensitive to the issues of conflicts of interest, whether real or perceived. As such the Forensic Specialities Accreditation Board (FSAB) was developed. The mission of the FSAB is as follows:

"The goal of this program is to establish a mechanism whereby the forensic community can assess, recognize and monitor organizations or professional boards that certify individual forensic scientists or other forensic specialists. This program has been established with the support and grant assistance of the American Academy of Forensic Sciences (AAFS), the National Forensic Science Technology Center (NFSTC) and theNational Institute of Justice (NIJ)."

Currently there about 16 accrediting boards recognized by the FSAB. Unfortunately there are no recognized boards or bodies related to digital forensics. This presents us with a very real problem. The FSAB standards and criteria cover the management structure of the body, examination and certification standards, competency of evaluation personnel, recertification etc. The standards also states that no certified members can be "grand fathered":

"Grandfathering is not an acceptable method of certification. 5.1.4.1 Certification bodies that used grand fathering and which were established prior to February 17, 2001, may apply for accreditation if not more than 50% of its certificants were grandfathered. Note: An individual is considered “grandfathered” if the person was issued a certificate without having taken and attained a satisfactory score in an examination designed to assess the knowledge, skills and abilities in the stated field of certification. 5.1.4.1.1 Any grandfathered certificants must be subjected to the same examination and competency assessment as new applicants (as defined in 5.3 of these standards) no later than the regularly scheduled recertification for that individual, not to exceed a period of five years.5.1.4.2 No certification body established after Feb 17, 2001, may apply for accreditation until all its certificates have been issued according to the standards as defined section 5.1.4 of these standards."

Grand fathering has been a popular method in the Information Security field in order to instantly populate a new credential with a critical mass of members. It would seem that this practice is off the table with forensics.

The lack of any recognized accreditation board for digital forensics (at least by the FSAB, AAFS, NIJ) translates into certifications that may have little if any value. This is not to say that there aren't some first rate digital forensics professional certifications out there, it just means that none are technically recognized by the same standard as other forensic sciences. This fact cannot be lost on opposing counsels and will certainly find its way into the court room in the not so distant future.

More upcoming Pubs

I am proud to announce that I have been working with the folks at Chelsea House (James Chambers in particular) on a series of edited books for young people ages 12-18 yrs old. The objective of the series is to provide young people with a frank discussion related to "Cybersafety". As the Chief Technical Editor I had the pleasure of working with some of the leading authors and researchers in the areas of cyber stalking, cyber bullying, online addictions, cyber predators, identity theft etc.

The series should make an excellent resource for middle and secondary schools. If anyone is interested in being involved in the second edition of this series, please just let me know.

http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=cybersafety+Rogers&rh=i%3Aaps%2Ck%3Acybersafety+Rogers&ajr=3

Response to Where's the Science

I appreciate the opportunity to discuss this topic on your blog.  I find that my opinion would vary from yours to some degree.

While science and experiments are a vital and necessary part of digital forensics a large part of the evidence uncovered during an exam does not necessarily require an application of science or an experiment.  The people who originated the term “Computer Forensics” could have picked a better phrase to define the discipline.  I prefer to think of most of the work I do as a “forensic search” of a piece of evidence.  I preserve the evidence, the image, in such a way that it is not altered and anyone can duplicate my work from that image and then I search it just as a detective would search a house for a gun, narcotics etc. 

The vast majority of evidence that I have located in exams has come from allocated space.  In my report I document the file, its dates and times and its location on the storage medium.  As an example I had a case where I was requested to examine a Blackberry for evidence which might link a given suspect to a bank robbery.  On the micro SD card in the phone I recovered a photograph of the suspect holding a version of the MAC 10 machine gun.  Statements from the witnesses and the video show one of the suspects holding a MAC 10.  The prosecutor and the jury found the photograph amusing.  My testimony consisted of stating that I found the graphic on the micro SD card.  I did not testify that it was a real gun or to any other fact other then I found the picture and where.

Using the image of the SD card the graphic could be located by any forensic tool available.  Since the report provides the location of the file I could even clone the image to another SD card.  The card could be put in a media reader and anyone computer literate could navigate to the same evidence I found.  It was nothing special and anyone could have done the same.

I read the report from the National Academy of Sciences and came out with a slightly different idea of what they found distressing.  I believe the central complaint in that report is not that the science in the disciplines is lacking but that the opinions expressed by the experts in court tend to go beyond their literal/explicit findings assuming they bother to do an exam at all.

In a recent discussion in a college class I asked the question if a computer examiner could state that passwords held any evidentiary value.  My position was that they hold none except in very limited circumstances.  As an examiner I can determine that a password exists or does not exist and I may even be able to determine what it is but that is it.  From an exam I cannot tell who in the home or office that the device was recovered knew the password.  I cannot tell if someone walked away from their computer etc.  In short I cannot testify as a computer examiner to what other people know.  It was interesting that many of the students insisted that you might be able to associate a password with a user if they used something personal like a date of birth or used the same password for multiple things.  They completely over looked the fact that they would be testifying to information not in their forensic exam but their personal opinion of someone’s computer habits.

Ego can be a problem when testifying.  As an “expert” there is a temptation to be willing to provide an opinion to whatever question you are asked on the stand.  Judges and attorneys often do not help as they frequently regard anyone with computer knowledge as an all knowing expert on all subjects involving computers.  It is up to the examiner to let the attorney/judge know that he or she does not have that knowledge or expertise.  It is very difficult to tell people looking up to you as the source of all knowledge that you don’t know.  I would say that the science is not lacking in forensics but the willingness of experts to provide opinions outside of the literal results of their findings is the actual problem.

Please don’t get the impression that I believe that science and research have limited value in digital forensics.  I do believe that experiments and research are of use and necessary in digital forensics.  I have engaged in them from time to time in particular when dealing with files recovered from unallocated space or fragments of files.  Determining what program generated the file or fragment, determining the evidentiary value or lack there of, has most of the time required experiments and research.  I do believe that much of my work is simply a search which identifies information of value.

Posted by:

Sgt. Kevin Stenger
Orange County Sheriffs Office
Orlando Florida 

Request for Authors

I am proud to announce that the "Encyclopedia of Information Assurance": SBN-10: 142006620X ISBN-13: 978-1420066203 is now out.

Encyclopedia of Information Assurance

We are currently working on the second edition and are actively soliciting authors for this edition. If you are interested in working on the second edition please contact me directly at: rogersmk@purdue.edu or Rich O'Hanley <rich.ohanley@taylorandfrancis.com>.

Where's the science?

After a long delay, I have finally found time to update the blog (probably due to the knee replacement surgery I had done and I am getting bored laying around). The topic of this posting has its origins from multiple sources. The first being my attending the AAFS conference and sitting through several presentations in the Digital & Multimedia Sciences Section [full disclosure – myself and a student presented 2 papers]. The second source of motivation was the excellent book by Dr. Ben Goldacre "Bad Science". Both of these got me thinking about where the science is in digital forensic science? We seem to have plenty of case study presentations, tools being developed, and novel investigative protocols being proffered. What appears to be missing is any real empirical research!

Very few of the manuscripts I review report any type of hypothesis testing, statistical analysis, or at the very least error rates or reliability estimates. When these oversights get brought up, the typical refrain is that we are an applied science, not basic research. This rings hollow with me. The term applied science should and is not synonymous with a lack of proper scientific analysis, data reporting, validation or replication of findings. It is almost as if we in the community have an inferiority complex and some believe that our field is not worthy of scientific rigor.

In the context of the National Academy of Sciences report to congress on forensic sciences and the pending bills being floated around the Whitehouse (e.g. Senator Leahy's), we need to step up and step back to cast a critical eye on the science of forensic science across all of the fields, ours being no exception. I have commented before how there seems to be a lack of scientists actually involved in charting the direction of digital forensic science, a fatal mistake in my opinion.

It should be very interesting to see if external bodies such as the proposed Office of Forensic Science and the Forensic Science Board will push us in the direction of being more scientific or if they will be the typical political lame ducks and produce only the illusion of science. Unfortunately based on the historical record I predict the latter will happen. Therefore it is up to we in the community to push for better accountability and research based on proper scientific methods (even a focus on reproducibility would be a giant leap in the right direction).

Here is an interesting interview with Ben Goldacre on the booming age of pseudo-science:


Pseudo Science

Ben Goldacre