In this three-part blog series we are looking at emerging testing trends for 2023 and beyond – through the whole assessment lifecycle. In part one, my colleague Isabelle Gonthier explored test development trends. In this blog I’ll be looking at my area of expertise and passion, data forensics. And next time PSI will share trends in test delivery.
A growing trend we are seeing across the whole assessment lifecycle is a positive move towards a more holistic approach to test security. This starts during test development, with technologies that ensure the security of test content and help to meet the increasing demand for new test content. And when it comes to data forensics, it involves structuring data forensics to ensure it is fully integrated into test security plans.
Holistic test security plans
Before you can align your data forensics results with your test security plan, you need a test security plan! One that is fully documented and involves all the right people, both internal and external to your organization.
We can’t sell certainties but used in this way, as part of a holistic approach, data forensics can certainly help reduce ambiguity. By highlighting potential issues and providing data and information to support further investigations.
Read how data forensics in the assessment lifecycle improves security.
Enhanced statistical indices
Another rapidly advancing area is the continued enhancement of the suite of statistical indices we use in our data forensics programs. One example is a time-based statistical index where we look at item response times in several different ways. For instance, we look at the extent to which a test-taker tends to respond more quickly or slowly than normal averaged across the set of items on a test to assess their overall speed, and we also look at the degree of correspondence between their item-to-item pattern of response times throughout a test and the expected time demands of the set of items.
These statistical indices help us to flag more accurately:
- Test takers who may have pre-knowledge – because they already know the answers, their response times are unusually fast and do not fit the expected pattern for the items.
- Test takers who are item harvesting – because the test taker isn’t really trying to answer an item, and instead may be harvesting items to share and/or sell on the internet, so the amount and pattern of time they spend memorizing the items may be very different from the time people spend actually trying to answer them.
Where test takers are flagged using these indices, we also look at unrecognized applications they have running in the background and then add these applications to our blocked list. What’s more, we find that when we examine flags raised by these new indices, they often correlate with test takers who have rogue applications open during their test, and also with other measures of response similarity that help determine if their behavior is part of a widespread pattern.
Moving more indices from the research stage into our standard suite of statistical indices for data forensics programs continues to be a big win for us, successfully increasing test security as we head into 2023.
Data forensics with web crawling
What started as a very niche service just a few years ago has now become a fully matured program. Web Crawling is utilized by a rapidly growing number of clients in combination with their data forensics program as part of a holistic test security plan.
One recent example was the identification of leaked exam content on a large website of user-submitted test-prep materials. Thanks to a tip off, we knew some test items were located on this specific dump site. However, we did not know where on the site these items were housed. Due to the extensive variety of content on the site, a manual search wasn’t feasible, and further, simple text searches may have led to further items being exposed.
The solution? We used safeguarding web crawling techniques and only searched the site for items that data forensics had flagged as potentially compromised. In this scenario, web crawling identified around 90% of the items that were flagged by the data forensics. We were then able to take steps to get the test content removed from the website.
Partnership working
Across all these emerging trends, we are working closely with our clients to develop new tools and techniques that raise the bar for the whole industry. In collaboration with testing organizations, we are conducting research, presenting at industry events and sharing best practice. Whether it’s developing new indices or working with new technologies, it’s an exciting time to be working in data forensics.