Strategies for using automatic item generation (AIG) in test design

There is a never–ending, urgent need for high-quality test items. However, typical item development is both time consuming and cost prohibitive. The needs have increasingly surpassed the ability to generate large quantities of test items with minimal human involvement, especially under the circumstances we’ve seen in 2020 thus far – a travel ban and social distancing due to a pandemic.

Nonetheless, new challenges bring new opportunities to innovate. One of the ways to address the issue is to use Automatic Item Generation (AIG). AIG offers an intuitive way to generate new items efficiently and effectively to produce hundreds, and even thousands, of items from a single item model.

AIG is an evolving research area where cognitive and psychometric theories are being used to create items via computer technology. This allows the implementation of item templates/models by controlling for cognitive levels and performance characteristics quickly and effectively, thereby minimizing the efforts of human intervention across the development process.

Why Use Automatic Item Generation (AIG)?

AIG has several critical benefits throughout the test development process.

• Test Security – A large number of items drastically decreases item exposure rate. Item models are constantly manipulated and varied, making it challenging to memorize items or cheat. Read more about methods to improve test security.
• Cost Effectiveness – Because the model is continuously reused automatically, as opposed to developing each item manually, it presents a cost savings on development that would not be attained if created from scratch.
• Training and Instruction – Subject-matter experts can be trained to develop item templates or models and come up with lists of variables that can be manipulated. For examinees, personalized and self-directed learning activities can be supported by preparing appropriate and individualized test items easily, automatically, and with little effort.
• Multi-Modality – Automatically generated items can be exported and imported into web-based, multi-modal test delivery systems to support administrations where online, remote proctoring, and testing centers are being used.

How to Create AIG

So where do you start? An AIG model can be created based on three approaches:

1. Based on a Theoretical Framework, where an item template is created from cognitive and psychometric theory.
2. Based on Successful Items, where an item template is created from an existing item.
3. Based on a Content Outline from Job Analysis, where a template is created from a consensus skill description, objective, or competency standard.

There are several key elements of AIG. First, the item templates of models are created to highlight the key features in the task that can be manipulated. These features can then be converted into variables with certain parameters as defined. These then can be manipulated with logic or formulas to capture responses and common errors.

Data sources can be generated using manual entry, import, or even within a database with a formula, text, images, and media (including audio and video) format. All item types are supported with AIG as well, including multiple choice, matching, drag-and-drop, fill-in-the-blank, true/false, and hotspot. These variations can be reviewed as a batch or separately and documented with corresponding references and rationale.

Takeaways and Recommendations

AIG can be used in all the ways you’re traditionally using items. It can be used with a variety of item types and modeling, and items can be exported for usage via text or QTI format in any banking or delivery system. Moreover, AIG data sources and templates can be shared, reused, and uploaded. AIG templates can be used widely for calculation or numerical items and can be extended to graphic, audio, and video items. And lastly, AIG item development and review processes are very similar to those of traditional items.

When using AIG, you’ll also be working closely with subject-matter experts who have knowledge about the item content to help you draft templates. Some places you can start are previously developed items, and then you can discuss challenges and shortcomings of those items in order to improve them. It’s important to collect relevant data beforehand, such as tables, figures, images, links, etc., and then discuss plausible distractors and the best method to generate them. Once you’ve collected and discussed those areas, the subject-matter expert will be able to help you finalize the template before generating variations of the item to review.

Overall, AIG is an exciting, accessible, and expanding area with numerous research studies in progress regarding item model calibration, item property prediction, and item survival rate.