October 2014—NCCCO Practical Exam Auditors recently completed their annual training as mandated by NCCCO’s quality assurance program.
NCCCO’s Practical Exam Auditor Program is designed to ensure that all certification examinations are delivered in a standardized manner that is fair to all candidates, while remaining psychometrically valid and reliable.
During three mandatory annual training sessions held this summer in Meadville, PA; La Porte, TX; and Salt Lake City, UT, a total of 24 carefully selected and trained Auditors reviewed the latest updates on auditing procedures for all NCCCO programs. Topics covered included correct test site setup, refining observational skills, identification of discrepancies, and review of typical issues found during audits.
“In line with NCCCO’s commitment to quality, fairness, and integrity in testing, annual Auditor training provides a systematic means of ensuring that CCO practical tests are delivered in a consistent, standardized manner, wherever and whenever they are administered,” said Erin Jones, NCCCO Program Manager for the Practical Examiner Accreditation and Audit Program.
“By keeping everyone on the same page regarding program improvements, we can be sure that we are fulfilling NCCCO’s mission to protect all stakeholders," said NCCCO Director of Operations, Joel Oliva. “The Audit Program brings our accountability full circle and improves the ability of the CCO certification process to reduce accidents, save lives, and generally make the construction industry and lifting operations a safer environment for all to work in.”
NCCCO Auditors receive mandatory annual continuing training to ensure all are up to speed with the latest refinements to the testing process and to ensure all Examiners receive fair and consistent audits.
Auditors are carefully selected from among NCCCO’s Practical Examiners in good standing based on such factors as the quality of their work and the professionalism they bring to test administrations. Those who qualify then receive thorough training in auditing techniques including examiner reliability calibration, shadow exam practicing, recognition techniques, inter-examiner reliability, and all pertinent NCCCO policies regarding the Practical Exam process. Auditors receive comprehensive practical, hands-on training on an approved crane and course layout at an official NCCCO test site.
Most CCO programs require candidates to pass rigorous practical exams, and Practical Examiners undergo thorough training and auditing to understand the intent of psychometric principles such as validity, reliability, bias avoidance, and Behaviorally Anchored Rating Scales (BARS). The auditing process provides feedback to the governing program committees and accrediting bodies that exam procedures and policies are being followed correctly.
To date, NCCCO’s team of more than two dozen trained and experienced Auditors have conducted hundreds of audits at practical test sites across the country.
During an audit, Auditors first ensure that the practical test sites are set up correctly according to the test site layout CAD. To ensure maximum inter-rater reliability, the Auditor also records candidates’ performance on the exams alongside the Practical Examiner; both score sheets are then compared to ensure that the Practical Examiner is scoring the exams correctly. Any discrepancies noted are based on the Auditor’s score sheet, so it is vital that Auditors understand all the details and nuances of setting up the courses and recording candidates’ performance of the test properly. Although the auditing process seeks to avoid disruption of the Practical Exam process, an Auditor can stop the test and provide corrective action if he or she believes the integrity of the test is being compromised.
After review of the Auditor’s report by NCCCO staff, the Practical Examiner is notified in writing of any discrepancies noted either in the test site set-up or any aspect of the practical test. This includes any inconsistencies between the Auditor’s score sheet and that completed by the Examiner. Corrective actions, along with references to all pertinent NCCCO policies regarding the Practical Exam process, are also outlined.