How data forensics in the assessment lifecycle increases test security

How to use data forensics in the assessment lifecycle to increase test security

Test security is essential for the integrity of certification testing programs. But the testing landscape has shifted during the past few years, in part due to the coronavirus pandemic and the increased popularity of hybrid working. New ways of enhancing test security have emerged alongside this shift. Including advances in data forensics.

Data forensics uses statistical methods to study or investigate digital data to detect anomalies. Identifying suspicious patterns in testing data helps us to detect and deal with potential misconduct early.

Although data forensics is most well-known for its role in the post-test phase, it can be usefully leveraged across the whole assessment lifecycle. From item writing and test assembly to test delivery and beyond, data forensics is highly valuable for informing the secure, robust evolution of your testing programs.

Data forensics in test development

Rigorous test development will ensure your test content matches what needs to be assessed. Data forensics aids multiple stages of the test development process including item writing, test assembly and test review.

Item writing

Usually written by Subject Matter Experts (SMEs) in the field, test items or questions should be relevant and encourage test takers to demonstrate their knowledge and skills.

However, a highly specialized topic limits the number of potential test items. It can be difficult to navigate this terrain, but data forensics and analytics will show if an item has been compromised or isn’t properly testing the content. If so, these items can be prioritized for updating.

Test assembly

Test forms should be equivalent and include a diversity of items and levels of difficulty. Technology supports this with Linear on the Fly Test (LOFT) assembly that limits the exposure of test items, as test forms are different for each test taker.

Read more about LOFT here.

As well as LOFT, developing a plan for limiting item exposure is essential to protecting test integrity. This approach is useful in geographic areas where proxy testing and item harvesting are more prevalent. When data analytics shows an increase in suspicious behaviors, specific exam forms can be used to limit the risk to the entire item bank. The National Association of Healthcare Quality is one organization to take this approach.

“Our test security program exists to protect the public, the individuals we certify, and the reputation of our organization. Data forensics is particularly important as we expand internationally into areas where we don’t understand the local norms.”

— Frank Perna, Director of Certification, NAHQ

Test review

Tests should also remain valid and current. Regular test reviews, aided by information from data analytics, will inform important decisions about test content development.

For example, data analysis shows the impact of stolen content – found by web crawling – on test taker performance. Organizations can then decide whether to remove or edit an item. This is particularly helpful with specialized topics that have a small number of items.

Data forensics in test delivery

Modern technology means that a test can be taken almost anywhere in the world. Whether in-person at a test center, online with a remote proctor, or even off-grid, tests should be delivered securely, ensuring fairness and an equitable experience for all.

There are several examples of best practice for using data forensics in test delivery. For instance, using analytics to:

• Determine if test content has become exposed, including whether exposure is concentrated on or limited to a specific test form.
• Inform delivery policy, such as considering online proctoring or multi-modal testing.
• Flag potential issues at a test site or school, particularly where a client looks after their own test admin.

In addition, the PSI lockdown browser is used to detect as well as prevent malpractice. For example, ISACA uses the browser to record any applications open during a test – not just those that are currently recognized. When data analytics identifies a test taker as suspicious, any unusual applications that were open during their session are added to the list of unauthorized applications to prevent future use.

Tell me more about… Data Forensics for Online Testing.

Statistical analysis and reporting

Data forensics makes it easier than ever to detect traces of fraudulent behavior that may indicate cheating. Statistical trends and analytic techniques are leveraged at test taker, test and group level to uncover unethical behavior. As well as supporting further investigations, including test recordings and registration data, and drop-in inspections or secret shopper visits at test sites.

Data analysis at…

• Test taker level
  • Identifies similar responses across different test takers, unusual response times and other patterns.
• Test level
  • Identifies shifts in an item’s average score or response time.
• Group level
  • Identifies regions and test sites with high rates of flagged test takers.

“It is hard to prove cheating, but data forensics enables us to identify individuals who have pre-knowledge and, in compliance with our candidate agreement, we enforce nullifications of their exam scores. My advice to any organization considering a data forensics program with PSI is to do it.”

— Kim Cohen, Senior Director – Credentialing, ISACA

Data forensics finds malpractice. What next?

PSI works with testing organizations to appropriately respond to any malpractice uncovered by data forensics, depending on the individual circumstances. Whether it’s sending cease and desist letters to those sharing test content, decommissioning third-party test centers or revoking test taker results, data analysis helps organizations to respond effectively.

Your test takers deserve rigorous, robust testing to demonstrate their excellence. And your organization needs secure testing to protect the reputation and future of your programs. Data forensics is valuable for all stages of your testing programs. By leveraging the power of data forensics from the outset in test specification and development, you can be sure your test content and delivery is as secure and resilient as possible.