I had already started considering the topic of data forensics and ethics for this blog when I was asked about fairness at a recent conference. The ethical interpretation of data forensics is clearly a topic on people’s minds in the testing industry.
Concerns are understandable when the pace of change is so rapid. The options available to increase test security are constantly evolving and improving, and the field of data forensics is an important part of this. We are developing new tools and techniques all the time – moving closer to real time data forensics and real time decision making. But as with other changes and developments in the testing world, we must be mindful of the impact data forensics might have on our test takers.
Importance of ethical data forensics
When it comes to online testing, a data forensics program uses statistical detection methods to find anomalies in the testing data that might indicate malpractice. When an anomaly is found, a flag is raised. Identifying suspicious patterns supports early detection and investigation of potential misconduct.
So, why is ethical interpretation of these suspicious data patterns important? I often recall a central principle of medicine – do no harm – when I think about data forensics in testing. Our aim is to protect the test without harming the test taker, or their ability to improve their situation in life through the test. We want to pass the test takers who deserve to pass and stop those who don’t have the required knowledge and skills yet to hold the license or certification.
When someone chooses to take a certification or licensure exam, they are doing it to improve their life. It is a courageous step and we do not want to create unnecessary barriers. If we make a mistake and take punitive action against someone who does not deserve it – such as invalidating a test result without sufficient evidence– we are doing harm.
Principles for ethical data forensics
By creating equitable and fair test security policies and processes, we can minimize the risk of doing harm. An example is generating data forensics results that are clear and highly unlikely to be due to chance, based on multiple data points. At PSI, our data forensics programs use multiple indices like response similarity, errors in common, abnormal item and test time, incongruent score patterns, and pass rate elevation.
Equally, data forensics is not necessarily the only element to consider when making a decision. It is best practice to draw from multiple sources, including proctoring irregularity flags, tip offs, trend data, or audio and video recordings, for example. When data forensics raises a flag, it’s important to investigate and validate, leveraging as many data points or evidence available to piece the information together and draw a conclusion.
An essential first step is to develop and communicate a policy stating how decisions will be made as part of a data forensics program. Any decision should then be policy based, to ensure outcomes are equitable and fair. Clear processes and a means to appeal should be contained within the policy and communicated to test takers.
Key stages for ethical interpretation
When supported by these principles, data forensic flags don’t exist in isolation. They are part of a bigger test security picture and process that is followed before any punitive action is taken:
- When something is flagged by data forensics but there is insufficient evidence to act, monitor the situation and make all parties aware.
- When further evidence becomes available, launch an investigation that brings together all the available information.
- Collate the findings from the investigation, including data forensics results, to draw a conclusion.
Read our blog: Data forensics case studies from our team of experts.
Human involvement for ethical interpretation
As well as involving multiple pieces of information, data forensics should not be a plug and play model. Any data point, including flags from online proctoring or identity checks, is just a tool to help us gather information. A human being should be behind any decision made based on data forensics outcomes.
At PSI this initially involves the individual analyst who identifies a data anomaly that might indicate malpractice. This is followed by internal PSI peer review for a second opinion and then a full team review. We challenge each other and enjoy these robust discussions, which contribute to our continuous learning about the changing test security landscape.
Advice for testing organizations
As well as human involvement in decision making, you should always receive your data forensics results in a written report for further consideration. My strong recommendation is to take the opportunity to go through the reports with your data forensics team. Discussing your data forensics report with experts means findings are less open to misinterpretation. Not all organizations take this opportunity, but in my experience these discussions help both the testing organization and PSI to understand what is happening. You have the context and PSI has the analysis – the strength lies in the combination.
Don’t wait until you’re sick
My final piece of advice returns to a medical analogy. Don’t wait until you’re sick to go to the doctor. Just because you visit the doctor for a health check, it doesn’t mean there’s a problem. Much like data forensics, a check-up might find a concern that you can then take steps to address. Not knowing about it won’t make it go away. And if your results are clear, it means you can build the right program for your test takers based on that knowledge.