October 2017—To maintain the strength, fairness, and ongoing validity of CCO certification, NCCCO audits all aspects of its programs regularly—from the exams themselves to those administering the tests, and even the program and test auditors themselves. This is particularly important for the practical exams, since there are so many more “moving parts.”
ANSI assessors have singled out for praise the high inter-rater reliability NCCCO has achieved among practical examiners on its practical exams, which they have called “exemplary of the best practices in performance testing.”
The Practical Exam Audit Program, which is overseen by the Practical Exam Audit Committee (PEAC), provides a systematic means of ensuring that CCO practical tests are delivered in a consistent and fair manner, wherever and whenever they are administered. During an audit, an auditor checks all elements required for the practical exam, including the test site facilities and test course preparation. The auditor also ensures that candidates are briefed with the required materials, and that the test course meets the precise requirements of the test site layout (CAD) issued for that test.
To verify that practical examiners are conducting tests in conformance with NCCCO testing protocols, their performance is compared with a “shadow exam” conducted by the auditor at the same time that the candidate is examined by the practical examiner. Any discrepancies between the auditor’s shadow exam score and the candidate’s score sheet are noted. Auditors are authorized to stop tests and provide corrective action if they believe that a test’s integrity is being compromised.
During their mandatory annual refresher training, NCCCO Practical Exam Auditors (shown discussing elements of test site layout verification at the 2017 Auditor Refresher Training Workshop hosted in September by Crawford Custom Consulting, Meadville, PA) receive updates on program enhancements and modifications. Their experiences and expert observations allow NCCCO’s audit process to be calibrated to ensure consistency and fairness for all Practical Examiners as well as the candidates to whom they administer tests.
To date, NCCCO’s team of more than two dozen trained and experienced auditors have conducted thousands of audits at hundreds of practical test sites across the country, all of them aimed at providing effective and transparent oversight of NCCCO’s practical exam process to ensure that all candidates are examined in a consistently fair, valid, and reliable manner. An example of the layered levels of checks and balances employed by NCCCO is the way in which auditors are selected from among NCCCO’s practical examiners in good standing based on such factors as the quality of their work and the professionalism they bring to test administrations. Those who qualify receive thorough training in auditing techniques including examiner reliability calibration, shadow exam practicing, recognition techniques, inter-examiner reliability, and all pertinent NCCCO policies.
Auditors also receive comprehensive practical, hands-on training on an approved crane and course layout at an official NCCCO test site. Only then are they deemed qualified to serve in this vitally important oversight role.