One of the first steps in any content analysis or form development process is to recruit subject matter experts (SMEs). Ensuring that the right perspectives are counted is critical to the content validity of a knowledge-referenced assessment. But who are your SMEs? What characteristics lend themselves to a quality SME? How many SMEs should there be? Come with me if you want to learn.
Step 1: Define the Target Population
The initial step of determining a SME is determining the target population. The target population is the “pool” of individuals that SMEs should be pulled from. The target population should consist of individuals who currently hold the target credential, currently hold a related credential with similar eligibility requirements, or are currently eligible to hold the credential. Furthermore, these individuals should have recent (e.g., within the past 6 to 12 months) work experience in the profession. These individuals are best suited to providing insights into the target job role or areas of knowledge related to a skill set.
Every target population can have different demographic and professional characteristics of interest, such as: years of experience, level of education, geographic location, specialty. Every collection of SMEs (whether that is committee, panel, task force, or even just a handful of individual contributors) should be a reflective sample of the target population’s demographic characteristics. For example, if 25 percent of the target population live and work in Canada, roughly 25% of the SME committee should be comprised of individuals who live and work in Canada. It is fine if the committee’s demographics are not a perfect match compared to the target population as long as it is similar to or reflective of the target population.
You might also like: The Value Of Assessing Soft Skills In Credentialing or Soft Skills Assessments in Credentialing
Step 2: Recruitment
Research suggests that 12 to 15 individuals or 10 to 20 percent of the target population, whichever is most practical, allows for sufficient diversity regarding to the demographic characteristics described previously. In any case, it is preferable to target more potential SMEs than necessary due to the likelihood of attrition. While potential SMEs may be eager to be part of the process, some may not realize the amount of time that is truly required of them and may resign midway through – or worse, do nothing. Attrition with an already small SME committee could be detrimental to the progression of the project and possibly push back the implementation of a new form. A common tactic to entice SMEs to serve on these committees is to present an incentive, such as a few continuing education credits towards certification renewal or nominal honorarium, in return for their service.
A question that may present itself is: Who should I look for? Treat the recruitment of SMEs quite similarly to the recruitment of a normal paid position. Request cover letters, resumes, and/or curriculum vitae, and then interview those who appear promising. It is best to get a mix of familiar faces and new perspectives too. Recruiting SMEs who have previous experience with exam development can be a boon, but representativeness of the group can be skewed if first-time contributors are not also included.
Step 3: Get to Work
Once you have defined the target population and assembled your committee, it’s time to put them to work and create your credentialing examination!