CLEAR Exam Review Fall 2023

CLEAR Exam Review Fall 2023, Volume 33, No. 2

In "Abstracts and Updates," George Gray features a review of a recently published book on standard setting that goes beyond methodology to cover the complete process of setting a defensible passing score. Dr. Gray also introduces an updated text for understanding and applying measurement concepts. He also reviews articles on the social values underlying psychometric practice and on interpreting effect sizes. He goes on to present abstracts from several articles describing research on specific certification programs along with an investigation of study methods to improve performance on certification exams. The column wraps up with an introduction to a new NCME ITEMS module on a topic vital to defensible credentialing programs: test validity.

"Legal Beat" addresses the reasons why, for licensed professions, competency exams are required in addition to graduation from a specified educational program. The column outlines differences in the nature and purpose of these two prerequisites to licensure and goes on to describe a case in which the plaintiff completed the required academic program but failed to pass the bar examination in six attempts. The court ultimately ruled against the plaintiff due to statutes of limitations. However, regulators and licensure programs should be aware that the court expressed concerns regarding Puerto Rico’s bar exam, its relatively low pass rate, and the bar examiners’ failure to provide information regarding the development of the examination and procedures used to set the passing score.

CLEAR Quick Polls over the past few months have collected snapshot data on several topics related to licensure and certification examinations. Carla Caro summarizes results of recent CLEAR membership surveys in the Quick Poll column. Three surveys administered in the past year are described. The first focuses on technological changes in regulated professions and how regulatory organizations react to such changes. The second explores the number and types of testing accommodations requested by candidates. Finally, results are presented from a survey seeking feedback regarding familiarity with CLEAR Exam Review (CER) and the topics and types of articles that are of interest to CLEAR members.

"Licensing Exam Construction, Validation, and Differential Pass Rates: What Regulators Should Know," by Matt Turner and Tara Raines- Our first feature article, is geared toward helping regulators understand how test development and validation can follow best practices to promote fairness; deter bias; address diversity, equity, and inclusion (DEI) concerns; and ultimately maintain the validity and defensibility of their exams. The purpose for requiring licensing exams in addition to educational programs is described. Current criticisms of licensing exams are addressed. Concerns regarding differential pass rates (by race, gender, age, etc.) are discussed. Steps are outlined for creating examinations that are fair, defensible, and aligned with the Standards for Educational and Psychological Testing (AERA et al., 2014).

"The Absence of a Live Remote Proctoring Effect on Scores from Two Examinations," by Robert C. Shaw, Jr.- Dr. Robert Shaw of the National Board for Respiratory Care (NBRC) presents the results of a study comparing candidate performance on exams administered via live remote proctoring (LRP) and in test centers. Respiratory therapy is a profession that was spotlighted during the COVID-19 pandemic, with an enormous spike in demand for licensed practitioners. With test centers unavailable, NBRC opted to employ LRP for their multiple-choice and simulation exams to help meet the demand. As test centers reopened, candidates were able to choose whether to take the exams in test centers or at home with LRP. Though the study is observational rather than randomized and controlled, no significant differences were found in the mean performance of candidates testing under the two modalities, offering assurance that testing via LRP likely did not advantage or disadvantage the candidates and the method can continue to be used with these exams in the future.