Reference Guide for Auditing a 
Credentialing Examination Program

by Barbara Showers

 



"Audit" means "to examine with intent to verify," according to Webster�s. It also means, "a methodical examination and review."

In the context of a credentialing examination program, there are four general purposes of an audit:

This guide can be used to conduct a self-evaluation by examination program staff, or to guide an evaluation by an external audit team that is neutral to the interests of the examination provider. The external audit team can be hired by the test provider or an external body which may contract for examination services, such as an association of regulatory boards.

Verification is best established by developing audit questions, collecting and evaluating specific written documentation which responds to the audit questions, and summarizing the findings in a written report. An office site visit may be needed to verify resources and operations, and a meeting with the provider after initial review of documents may be helpful to clarify questions that arise. In addition a visit to a test administration site might be appropriate.

Persons conducting the audit should be objective and knowledgeable of psychometrics and credentialing testing, and should be prepared to spend significant time evaluating the information provided, and writing the report. The scope of the audit and the timeline of activities should be agreed upon in advance. The audit may take weeks, months, or even a year, depending on the size and complexity of the testing program.

The most commonly accepted professional testing standards against which a high-stakes testing program can be measured are the Standards for Educational and Psychological Testing of the American Psychological Association, the American Educational Research Association, and the National Council on Measurement in Education. (See Appendix A, Bibliography, for additional sources of information regarding professional testing standards.)

The sponsor of the audit should consider in advance how the audit report will be distributed and used. A procedure for resolution of any negative findings should be determined.

The following pages detail questions that should be asked by organizations wishing to conduct an audit of a credentialing examination program. The purpose of this publication is to provide a framework and suggestions for planning and conducting the audit process.

The questions were compiled by a subcommittee of the Examination Resources and Advisory Committee of the Council on Licensure, Enforcement and Regulation (CLEAR). Committee participants were:

Barbara Showers, Ph.D., Chair
Director, Office of Examinations, Wisconsin Department of Regulation and Licensing

James D. Blum, CPA, Ph.D.
Director, Examinations Division, American Institute of Certified Public Accountants

F. Jay Breyer, Ph.D.
Managing Principal, The Chauncey Group International

Tadas Dabsys, B.A.
Manager of Operations, PSI Examination Services

Chuck Friedman, Ph.D.
Assistant Vice President, ACT, Inc.

Sandra Greenberg, Ph.D.
Vice President, Public Service Activities, Professional Examination Service

Steven Nettles, Ed.D.
Vice President, Research and Development, Applied Measurement Professionals, Inc.

Rae Ramsdell, M.A.
Examination Specialist, Michigan Department of Consumer and Industry Services

Kara Schmitt, Ph.D.
Director of Testing Services, Michigan Department of Consumer and Industry Services

Lee Schroeder, Ph.D.
President, Schroeder Measurement Technologies

Rina Sjoland, M.A.
Assistant Vice President, ACT, Inc.

Anthony Zara, Ph.D.
Senior Director of Testing and Research Services, National Council of State Boards of Nursing

Sherry Young, M.A.
Senior Manager, Cooperative Personnel Services

Jim Zukowski, Ed.D.
Director, Professional Licensing and Certification Division, Texas Department of Health

 

HOW TO STRUCTURE THE AUDIT PROCESS

The following issues and decisions should be considered prior to the beginning of the audit. For an audit with external auditors, an engagement letter, or letter of understanding, is helpful to document in advance the scope and timeline of the audit, and the responsibilities and expectations of the parties. Elements of an engagement letter are suggested in Appendix B.

A statement of compliance should be signed by the auditors. This statement acknowledges what types of materials the auditor is obligated not to disclose (such as secure examination questions, for example), and it verifies that the auditor does not have conflicts of interest, such as working for a review course provider.

A letter of representation may also be obtained from the organization being audited, in which the organization acknowledges its responsibility for the testing program, and agrees to make available the necessary documents for the audit, to accurately disclose all relevant information known to it, and not to interfere with the audit.

  1. What is the purpose of the audit?

  1. Who should the auditors be? 
    Important characteristics of the audit team include:

  1. What should be expected of the auditors?

  1. What types of information should be made available to the auditors?

The following information should be available in written documentation. Verbal statements without written verification indicate a weakness in the program:

  1. What are the audit procedure considerations?

  1. How will the findings be distributed and used?


BACKGROUND INFORMATION ABOUT THE TEST PROVIDER

Credentialing examinations are often provided through the efforts of more than one organization. For example, a credentialing body with a governing board might contract with a testing company to develop a credentialing examination. The credentialing body might establish certain purposes and standards for the credentialing examination. The testing company might consult with the credentialing body regarding technical testing issues, and might also be responsible for carrying out the test development activities requested in accord with its own mission. The testing company or the credentialing body might contract with another company for the administration of the examination.

A comprehensive audit would include an analysis of all parties which contribute to provision of the final product of the credentialing examination, or, an audit might be conducted of one or more of the parties.

  1. What is the mission of the test provider organization(s) being audited?

  1. What resources are available to the test provider organization(s)?

  1. What is the purpose of the test? Is there a difference between the stated goals and the documentation?

  1. Who are the users of the test? For what purpose do they use it? Is there an opportunity for user involvement or input into the testing process?

  1. What level of performance is the examination designed to measure (e.g., minimum competency or specialized skill mastery)? What is the expected education and experience preparation of eligible candidates? Are eligibility criteria consistently applied?

  1. Who has decision-making authority for the examination process?

  1. What are the name(s) and qualifications of the people who are responsible for:

  1. What is the financial status of the organization?

  1. What information is provided to the public (candidates and users) about the testing program?


AUDIT OF TEST DEVELOPMENT

The following questions are designed to investigate elements of test development procedures which affect the validity and quality of the resulting examination. The Standards for Educational and Psychological Testing are a particularly relevant resource for audit of test development.

  1. Is the test or evaluation instrument based on a job analysis?

  1. Do test specifications exist?

  1. Are the test items consistent with test specifications and cognitive level?

  1. How are the items developed?

  1. How are items introduced into production?

  1. How are test forms assembled?

  1. Is the bank of items large enough?

  1. How is the passing score established?

  1. How often is test content updated? What procedures are in place to assure currency of test content?

  1. What procedures assure that test content is secure during the test development process? Consider office security, electronic security, and security measures for at-home writing of questions, if used.

  1. If there are essay, practical or oral questions:

  1. If the test is now administered in a computer format, but is intended to be comparable to a test previously administered in paper and pencil format, is there data to demonstrate that the results are comparable?

  1. If simulations are used, what support is there for the validity and accuracy of the simulation?

 

AUDIT OF TEST ADMINISTRATION

The following questions are designed to investigate elements of test administration which affect fairness, due process, security, and ultimately score validity.

  1. What is included in the candidate information bulletin?

  1. How is the test administered: paper and pencil, computer, interview, take-home, etc.?

  1. What procedures are established to maintain security of the test materials?

  1. Are there standardized written procedures and instructions for test administration? Does administration staff receive training? Are the procedures followed? Procedures include:

  1. Is an emergency plan established in case there are problems at the test site such as noise, fire, flood, or other interruptions?

  1. What are the procedures if someone is suspected of cheating at the test site?

  1. What is the procedure for providing modifications to administration for people with disabilities?

  1. What is the procedure for providing feedback to the credentialing authority regarding irregularities in administration, such as cheating?


AUDIT OF STATISTICAL ANALYSES AND SCORING

The following questions are designed to investigate elements of statistical analysis and scoring which affect the validity and quality of the resulting examination. The Standards for Educational and Psychological Testing are a relevant resource of requirements in this area.

  1. Are item statistics calculated?

  1. Are reliability statistics available?

  1. Is the test equated so that reported scores on different forms will be a result of candidate ability and not item difficulty?

  1. What are the pass/fail rates, and the sample groups on which they were calculated? Are the pass/fail rates reviewed for expected outcomes, given the sample group?

  1. If the test is comprised of more than one scored part, are the scores combined into one total score, or must the parts be passed separately?

  1. How are tests scored? What quality control procedures assure scoring accuracy?

  1. How are individual candidate scores reported?

  1. What types of reports are generated: school results, committee reports, etc.?

  1. Are examination reviews or challenges allowed? If candidates can contest individual items on the test, what is the procedure for the protest? If not, what is the reason for not allowing appeals?

  1. What complaints have been expressed about the current test? What responses have been generated by the testing agency?

 

AUDIT REPORTS AND MANAGEMENT LETTER

In its briefest form, an audit report can be a short attestation by the auditors describing the audit scope, the audit method, and whether, in the auditors’ opinion, the scores of the examination can be relied upon. The auditor’s report should include:

A management letter is a more informative document, detailing the scope and method, the findings of the auditors, and their recommendations to management regarding areas of improvement. The auditors must report any conditions that they consider to be significant deficiencies in meeting recognized technical standards, such as the Standards for Educational and Psychological Testing, or deficiencies in meeting user goals or contract requirements, if these are part of the purpose of the audit. This letter may be presented to the board of directors of the test provider organization. In accounting audit practice, this letter may be designated as solely for the information and use of the board of directors, management and others within the entity.

Credentialing examinations are often used by state governments to screen candidates for state-regulated credentials. Unlike the private sector, there may be a presumption that results of audits of government activities must be available to the public as part of government’s accountability to the public. In the broadest sense, the public is the "board of directors." Therefore, a report of audit results for credentialing examinations used by states may be expected to be made public, as are results of audits of other government functions.

If the management letter is to be available to the public, the public report can incorporate management’s response to the audit findings and recommendations. The management response can be separate from the management letter, or can be incorporated into the public report after each major section of findings and recommendations. The management response may include a plan and timetable for resolution of any identified problems.

For example, a public report of audit could be structured as follows:

[And so forth]


APPENDIX A

BIBLIOGRAPHY AND RESOURCES

American College Testing Program. (1990). ACT Elements of a Request for Proposal. Iowa City, IA.

American College Testing Program. (1995). ACT Elements of a Sound Certification Testing Program. Iowa City, IA.

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1985). Standards for Educational and Psychological Testing. Washington, D.C.: American Psychological Association, Inc.

AICPA Auditing and Accounting Manual, Section 3175, Sample Engagement Letters, June, 1997.

AICPA Auditing and Accounting Manual, Section 7400, Representation Letters, June, 1997.

AICPA Auditing and Accounting Manual, Section 7500, Communication with Audit Committees, June, 1997.

Downing, S.M., & Haladyna, T.M. (Spring, 1996). A Model for Evaluating High-Stakes Testing Programs: Why the Fox Should Not Guard the Chicken Coop. Educational Measurement: Issues and Practice, 5-12.

Educational Testing Service. (1987). ETS Standards for Quality and Fairness, Princeton, NJ.

National Association of State Boards of Accountancy. (April, 1998). Report of the CPA Examination Review Board on the 1997 Uniform CPA Examinations and IQEX. Nashville, TN.

National Commission for Certifying Agencies. (1995). Standards for Accreditation of National Certification Organizations. Washington, D.C.

National Council of State Boards of Nursing. (February, 1998). NCLEX® Comprehensive Evaluation Guide Report By Evaluator. Chicago, IL.

Office of the State Auditor. (1996). An Audit Report on Management Controls at the Texas Department of Health. The State of Texas, Austin, TX.

Office of the State Auditor. (1997). An Audit Report on the Compliance and Effectiveness of the Texas Board of Nursing Facility Administrators. The State of Texas, Austin, TX.

Office of the State Auditor. (1996). An Audit Report on Management Controls at the Department of Health’s Licensing and Certification Division. The State of Texas, Austin, TX.

Professional Examination Service. PES Guidelines for the Development, Use, and Evaluation of Licensure and Certification Programs, adopted by the Board of Directors in 1995. New York, New York.

Professional Examination Service. (Winter 1998-1999). Methods for Evaluating and Auditing Credentialing Programs. PES News, Volume XIX, Number 1. New York, NY.

State of Florida, Department of Business and Professional Regulation, Bureau of Testing. (1995). Examination Evaluation Questionnaire. Unpublished.

State of Michigan, Department of Consumer and Industry Services, Office of Testing Services. Survey of Counseling Examinations. Unpublished.

State of Wisconsin. Department of Regulation and Licensing, Office of Examinations. (1997) Essential Elements of a Credentialing Examination. Unpublished.

The Council on Licensure, Enforcement and Regulation. (1993). Development, Administration, Scoring and Reporting of Credentialing Examinations: Recommendations for Board Members. Lexington, KY.

The Council on Licensure, Enforcement and Regulation. (1993). Principles of Fairness: An Examining Guide for Credentialing Boards. Lexington, KY.

Contacts for Documents Referenced Above:

American College Testing Program
PO Box 168
Iowa City, IA 52243

American Psychological Association, Inc. (publishers of Standards for Educational and Psychological Testing)
1200 Seventeenth Street, NW
Washington, DC 20036

American Institute of Certified Public Accountants
Harborside Financial Center
201 Plaza Three
Jersey City, NJ 07311-3881

Educational Testing Service
Princeton, NJ 08541

National Association of State Boards of Accountancy
150 Fourth Avenue North Suite 700
Nashville, TN 37219-2417

National Commission for Certifying Agencies
1200 19th Street, NW, Suite 300
Washington, DC 20036-2422

National Council of State Boards of Nursing
676 N. St. Clair, Suite 550
Chicago, IL 60611-2921

Office of the State Auditor
Two Commodore Plaza
206 East Ninth Street, Suite 1900
Austin, TX 78701

Professional Examination Service
475 Riverside Drive
New York, NY 10115-0089

State of Florida
Department of Business and Professional Regulation

Bureau of Testing
1940 North Monroe Street
Tallahassee, FL 32399-0791

State of Michigan
Department of Consumer and Industry Services
Office of Testing Services
PO Box 30018
Lansing, MI 48909

State of Wisconsin
Department of Regulation and Licensing
Office of Examinations
PO Box 8935
Madison, WI 53708

The Council on Licensure, Enforcement and Regulation (CLEAR)
403 Marquis Avenue, Suite 100
Lexington, KY 40502


APPENDIX B

ELEMENTS OF AN ENGAGEMENT LETTER FOR AUDIT OF A CREDENTIALING EXAMINATION PROGRAM

 

Note: Ideas for this appendix were taken from sample engagement letters found in the American Institute of Certified Public Accountants Audit and Accounting Manual, Section 3175. However, the information contained here has been adapted to audits of testing programs rather than finances, and is the product of the authors of this reference guide.




COPYRIGHT 2000. Rights to copy and distribute this publication are hereby granted to members of the Council on Licensure, Enforcement and Regulation (CLEAR), providing credit is given to CLEAR and copies are not distributed for profit.