When it comes to test fraud, the consequences of pre-knowledge can be serious. An unreliable or invalid test outcome might mean a nurse is able to practice without the skills needed to keep patients safe. Or a pilot could be granted a license without the knowledge required to safely fly a plane. That’s why you need to be able to detect test content theft as early as possible.
But how do you know when your test content has been stolen and shared with other test takers? And what actions can you then take to ensure every test taker gets the outcome they deserve?
How can test content be stolen?
Test content theft can happen at multiple stages of the assessment lifecycle. And the most effective approach to test security is holistic, with safeguards in place at every stage. Examples of test content theft and pertinent prevention measures include:
- Sharing or leaking items during test content development.
- Non-disclosure agreements with Subject Matter Experts writing test items, alongside secure item authoring and banking software, will protect test content.
- Recording or photographing a testing session or test paper.
- Proctored exams and tests, either in a secure test center or with online proctoring, will deter and prevent test takers from using hidden cameras e.g. camera-equipped eye glasses.
- Memorizing test items during a testing session.
- Testing technologies, such as Linear on the Fly Testing (LOFT) and Automatic Item Generation (AIG), will minimize item exposure and make this more difficult.
All these methods for stealing test content can lead to your items being shared with test takers. Either on a small scale or, in a worst-case scenario, more widely for sale on the internet or social media.
Data Forensics to detect test content theft
Data Forensics is the application of statistical detection methods to help recognize anomalies in data patterns that might indicate cheating. And one of the areas Data Forensics can help us with is detecting test content theft.
We use a variety of different indices in our Data Forensics programs. And one of the indices we use for the detection of test content theft is an Errors in Common index. This measures the degree of similar errors between pairs of test takers in their response choices. And it’s sensitive to abnormally high levels of matching responses, which is often found with test takers who study from the same pirated test content.
The Errors in Common Index recently proved valuable for a client when we were able identify test takers using pirated test content – and take action:
- Our client received a tip off and a document containing compromised test content that was being circulated. This document included some incorrect answers.
- A standard Data Forensics analysis flagged a test form that was identical to the document.
- This form was confirmed to be compromised with a separate review of the items within the document.
- Data Forensics also flagged test takers who selected the incorrect answers in the document, even for relatively easy items.
- Scores for the flagged test takers were invalidated and the test taker were required to re-test.
Web crawling to find stolen test content
While Data Forensics is a valuable tool in the box to help us detect test content theft when items or forms have already been compromised, we also want to prevent further exposure. Particularly if this is happening on a large scale. That’s when we go back to the toolbox and put our web crawling services to use.
Historically it has been both time consuming and expensive for testing organizations to manually monitor the internet for proprietary test content. Trawling through known brain dump websites, social media and the new websites that are appearing all the time is a lot!
Thankfully, we can now use web crawlers to search automatically and systematically for test content. Much like Google does to answer your questions about what to watch, what to listen to, or what to eat for dinner. These web crawlers show us when and where compromised test content has been shared. And we then match these test items or forms with actual content to confirm.
What next if you detect test content theft?
When Data Forensics provides evidence of test content theft, the next step is to conduct a further investigation. This often involves targeted web crawling to find the items or forms that have been compromised.
Additional investigative measures for remote online proctored tests and exams include a review of testing session recordings, registration data and computer logs. For in-person testing, we might trigger a series of audits, drop-in inspections or secret shopper visits at flagged test centers.
When the evidence is collated, we work with testing organizations to decide what next. Measures might include:
- Removal of a test form or items from the item bank.
- Commencement of take down proceedings on a brain dump website.
- Invalidation of one or more test taker’s results.
- Legal action against a test taker or other individuals involved in stealing and sharing test content.
The stakes are high
Test content theft brings multiple risks and costs. A risk to the integrity of your tests and the reputation of your organization. A loss of the investment you made to develop compromised test content – and the need to re-invest in developing new test items or forms. And a potential risk to the public if pre-knowledge leads to an inaccurate decision to grant a certificate or license. That’s why we take a belt and braces approach, with rigorous test security measures to prevent content theft and further steps to detect it, should it happen.