7 key steps in the licensure exam development process
Licensure programs need to be able to rely on the validity of their exam results. It is these exams that maintain standards and protect the public by restricting practice to qualified individuals. And to do this you need high-quality exam content, which can only be achieved through a rigorous exam development process.
Taking the following steps during your licensure examination content development process ensures your exams consistently assess what they are supposed to measure – without unfairly disadvantaging any individual or group of test takers. For a valid, reliable and fair outcome, every time.
1. Job analysis and exam specification
The exam development process should always start by defining what needs to be assessed and how – the content outline or exam specification. Best practice begins with a thorough job analysis and in-depth consultation with subject matter experts (SMEs). This identifies the tasks an individual must perform in their job, and what they need to know to perform these tasks.
Involving a broad and representative group of SMEs throughout the licensure examination content development process means the right perspectives are counted, which is critical to the validity of your exam content.
Read more about SME assembly here
2. Item writing
Item writing is the process of constructing the individual items or common questions that are needed for an exam. In this next stage, new exam items are created that map back to the exam specification. And it’s important to involve your SMEs at this stage as well.
SMEs will likely need training in writing exam items. Some will be taking part in item writing for the first time, others may not have been involved in item writing for several years and will need a refresher. This training usually takes place in a workshop, with a full guide to item writing circulated afterwards for your SMEs to refer to.
3. Item review
The next step is validating your content, an essential part of the licensure process before exam items are finalized. Items should always be reviewed by either a second SME or a different group of SMEs. Depending on your preference or need, group review can take place either in-person or remotely, where a group of SMEs go through every draft item and supply feedback.
During the pandemic, many licensure programs were forced to switch to remote SME meetings. While there were initial concerns about SME dropout and exam content security, our experience shows that remote reviewing is highly effective. Many of our clients plan to use a hybrid approach or even stay fully remote because it works for them and their SMEs.
Find out how we’ve improved the online proctoring experience.
4. Exam assembly
Exam assembly involves putting the items your SMEs have written, reviewed and signed-off into exam forms. This is another rigorous process with checks to ensure your exam form reflects your exam specification, is equivalent to any alternate exam forms, and does not unfairly disadvantage any test takers.
Licensure programs tend to use two options for exam form assembly, depending on their specific needs:
- Fixed forms – where all test takers see one of a small number of alternate and equivalent exam forms.
- Linear On The Fly Testing (LOFT) – where every test taker sees a unique and equivalent exam form with slightly different items.
Read more about Linear On The Fly Testing here
5. Exam review
Finally, exams should be reviewed by a committee of SMEs for final validation of your item pools and exam forms. With fixed forms, your SME committee will go through each exam form to check they are all valid, reliable and fair. With LOFT, your committee reviews only the item pool, not each individual form. This saves the SMEs time, because they don’t need to review the same item multiple times if it appears on multiple forms.
6. Standard setting
A cut score is the carefully chosen score on the scale of an exam that determines whether a test taker has passed or failed. It is the minimum score a test taker must achieve to pass the exam.
A facilitated group of SMEs is usually involved to set the cut score of a licensing exam. Their decision is based on the definition of a minimally competent test taker and what they need to know. There are a variety of different standard setting methods to decide a cut score.
7. Implementation and maintenance
After all the hard work you and your SMEs have put in, it’s time to go live with your exam. It doesn’t finish there though. Continued statistical analysis of your items and exams will give you the information needed to make recommendations about items that must be updated, edited, put on hold, or removed from your item bank.
While new laws, technologies or techniques in your industry may trigger a review of the relevant area or subtopic in your item pool, it is good practice to undertake a full review every 5-7 years. Starting from the beginning of the exam content development process with a full job analysis and exam specification.
Exam development expertise
Our team of psychometricians and testing experts have a wide variety of techniques and technologies available to help you develop high-quality licensure exam content. Including an item authoring and banking platform that makes the process fully secure – and easier for your SMEs.
By using these tools and following this comprehensive process, you can be assured that your exam content is fair, measures what’s important, protects the public, and is legally defensible.
