Whether you are reviewing the content of certification tests or licensure exams, a rigorous process is essential. Regular test content reviews aren’t just good practice. They ensure that the right skills and knowledge are still being assessed. And that your tests reflect the current and most up-to-date requirements of the target job role or specialty.
So, comprehensive test reviews should be carried out at regular intervals. For certification testing, such as IT or healthcare, as well as for licensure exams, including real estate, construction and cosmetology. But what does an effective test review process look like? And how can testing organizations be sure they are taking the necessary steps?
This blog covers the key stages during exam or test content review. With best practice examples where PSI has supported our certification and licensure clients throughout the process.
Involve Subject Matter Experts
The involvement of Subject Matter Experts (SMEs) who work in your industry is fundamental to the effectiveness of your test review process. A job analysis, usually achieved through survey distribution to as many SMEs as possible, is the first step. The results of this survey inform any necessary updates to the topics and subtopics in your content outline.
A smaller group of SMEs are then recruited to review your existing item bank and reach a consensus about next steps for every item – all based on the up-to-date content outline. Specifically, whether items should be retained, edited, reworked or deleted.
The group of SMEs you work with should all have experience and insights to offer in the target certification or license. In addition, they should be representative of the target population’s demographic. For example, in terms of years of experience, gender, race and ethnicity, and area of expertise.
The benefits of involving SMEs in your test review process are numerous. Not only will it ensure you are accurately defining what needs to be assessed – and how. It is also a positive opportunity to build relationships with some of your key stakeholders and offer them a valuable development opportunity.
Best practice SME engagement
High standards of participation and engagement are required from SMEs throughout the test review process. The time and effort required to manage your SMEs should not be underestimated, but when it’s done well the rewards are evident for all involved.
One example is the PSI partnership with the American Association of Critical Care Nurses (AACN) for their annual examination review, in addition to exam delivery and security. The ability of the PSI team to maximize the impact of the certified nurses who serve as SMEs during the process is vital for the success of the review process and testing programs.
“The respect SMEs come away from the experience with is paramount. The incredible work that goes into a certification exam development process, and the rigor of that process, makes the SMEs, who are certificants themselves, even more proud.” — Kristen Thomas, Certification Manager, AACN
Similarly, the National Board of Surgical Technology and Surgical Assisting (NBSTSA) work with PSI on their annual test development programs to guarantee only the most relevant, up-to-date content is reaching test takers. Ongoing evaluations include meeting with the NBSTSA Examination Review Committee to develop and trial updated test content.
“Access to PSI psychometricians and their knowledge of test development is really everything for us. They hold our hand through the whole process.” — Ben Price, CEO, NBTSA
Train Subject Matter Experts
SME meetings can be either in-person or online depending on individual program needs. Through the pandemic we worked effectively with SMEs on a purely virtual basis throughout test development and review processes. And this applies equally to SME training, which takes place in a workshop with supporting material circulated after the event.
Whether an SME is coming to the test review process for the first time or has some previous experience, training is essential. This will support your SMEs to make the best possible decisions about individual items, as well as to develop the necessary skills for item edits or reworking.
Review item bank
When you have trained your SMEs, it’s time for them to review your item bank. As a group, SMEs review each item, verify the answer is correct and that any distractors are incorrect (alternative answers if questions are multiple choice). They will then decide whether to:
- Retain – keep the current item in the item bank with no changes.
- Edit – modify the item to ensure it is easy to understand and reflects current industry practice.
- Rework – make more comprehensive changes to the item prior to editing.
- Delete – remove the item from the bank if it is not appropriate or relevant.
Gap analysis
Following the initial review of all items, a comprehensive gap analysis will identify areas where new items are needed. The number of items on a range of existing topics can be increased and/or updated to reflect industry practice.
For example, in early 2021 the California Department of Real Estate (DRE) contracted with PSI to provide item review, gap analysis and item writing for Broker and Salesperson licensure exams, as well as the Professional Responsibility exam. As a result of a gap analysis, new topics were added such as marketing & advertising, fair housing, and adaptive reuse. The new and expanded items were identified by DRE staff and validated by SMEs.
“This project was a full team effort, which involved multiple PSI facilitators and up to three virtual meetings per week. We reviewed thousands of items and developed new item content in less than a year.” — Jeff Oboyski Assistant Commissioner – Licensing, California Department of Real Estate
Seek innovations
While we have a well-established and proven process for effective and rigorous test reviews, it’s also important to seek potential improvements and enhancements. A good example of innovation with the California DRE during the pandemic was a change to the method for data transfer. Data previously shared on password protected CDs is now delivered easily and more quickly through secure online transfer.
Another area where the DRE embraced innovation is that license applicants are now able to apply online – for an exam, or both the exam and license. Users are also able to upload any necessary documents, pay the required fees, and check the status of their application online.
Efficient and effective
With a structured process, and ongoing commitment to maintain SME engagement, it’s possible to review a large item bank both efficiently and effectively. In collaboration with the California DRE and SMEs, the PSI team reviewed 4,367 items. Of these items, 1,261 were retained, 1,862 were edited, and 546 were reworked for editing – in less than a year and during the most challenging of times.
Through decades of working in partnership with our clients, we’ve learned that a rigorous test review process is just as important as ever. And more recently we have seen that innovation, flexibility and agility around the process will ensure the continued engagement of SMEs and overcome unforeseen challenges for a timely and successful test review.
Dr. Isabelle Gonthier is the Chief Assessment Officer at PSI Services where she leads psychometric and consulting services, test and content development, content management, data forensic and analytics, and research across certification, licensure, and qualification markets globally.