Since 2000, television shows like CSI: Crime Scene Investigation have been extremely popular with multiple spin offs for different cities. In the show, dedicated forensic investigators meticulously analyze crime scenes to unlock hidden stories related to the crime that was committed.
At PSI, our Data Forensics team engages in similar efforts to detect and analyze fraudulent testing activity through the use of applied statistics. Fraudulent test activity or cheating can occur in multiple forms, including proxy testing, which is when another person takes the test for a test taker, or item harvesting, which is an attempt to steal active test questions. When people interact with tests and test questions in abnormal and fraudulent ways, they are likely to leave traces of their acts.
Identifying suspected test fraud and taking action as early as possible are crucial components of test security to maintain the integrity of a certification or licensure program. Using data forensics, we aim to quickly detect activity that might challenge the validity and fairness of these tests and to provide guidance on what steps can be taken to minimize any impact.
Read more: Authenticate and Protect with Online Testing
A successful data forensics program utilizes a suite of multiple statistical indices to assess testing irregularities. The analysis begins with ongoing surveillance of test taker performance and identifying any clustering of abnormal trends by test sites, including remote or online testing, test forms or versions of the test, and geographical areas (hot spots). Specific attention is given to unusually high pass rates, abnormal item response patterns, and irregular test times. Data visualization techniques are also incorporated to graphically and clearly show any anomalies that are consistent with cheating and are used to assist with early detection.
What next?
When data forensics detects trends consistent with cheating behavior, a range of investigations and in-depth analyses are used to confirm whether misconduct has indeed taken place. A test taker, test site, or test form associated with anomalous findings may be the subject of further in-depth investigation. This may involve a full review of electronic records, including video or photos, could encompass “drop-in” inspections of a test center – and for remote or online test takers may include a review of any background computer applications running at the time of the test that could facilitate proxy testing or the theft of test content like screen-sharing software.
Upon confirming fraudulent activity, we work with the testing organization to identify and implement corrective actions. This might involve deactivating a test form or removing items from the bank that have been compromised. Some cases may require a test center to be de-commissioned or test takers may be banned from future tests or have their credentials revoked.
Why data forensics?
Educational and occupational certifications and licenses are extremely desirable as test takers attempt to advance their careers. Unfortunately, this desirability can increase the likelihood of these tests being targeted for fraudulent test taking. Successful test security, therefore, utilizes a multi-method strategy to ensure the validity and fairness of the test. Data forensics adds an additional layer to the traditionally used test security protocols.
The Covid-19 pandemic has resulted in the need for flexible testing options like remote proctoring, also known as online testing. While these options have been extremely helpful in allowing test takers to continue to pursue certifications to advance their careers, it also produces challenges for high stakes tests as tests are taken outside of the traditional test site environment. Introducing data forensics early as part of an overall test security strategy allows time to gather baseline data that will help to establish expected norms, thus allowing easier detection if and when any cheating activity starts to occur.