On February 28, 2021, (ISC)² concluded a pilot test that assessed the feasibility of online proctoring for exams that are an essential part of our nine certification programs. Online examinations for the CAP, CCSP, CSSLP, HCISPP, CISSP-ISSAP, CISSP-ISSEP, CISSP-ISSMP and SSCP certifications were administered February 15, 2021 – February 21, 2021. Online CISSP examinations were administered February 22, 2021 – February 28, 2021. For quality control purposes and anticipated candidate support concerns, we elected to offer all exams via online proctor only to candidates physically present within the United States.
Since the conclusion of the pilot test, we have been collaborating with our development, delivery and security partners to assess and validate the results of the pilot program. While some outcomes were expected and validated, we have also learned a great deal about online proctoring, candidate behavior and expectations, as well as our own processes during the pilot. This blog shares what we learned.
Pilot Objectives
The pilot test was intended to accomplish several objectives:
- First, the pilot would provide a technical “proof of concept” that the online delivery system would deliver (ISC)² examinations reliably and as intended.
- Second, the pilot would identify and document similarities and differences between online and center-based deliveries that impact the standardization of the environment and potentially impact the integrity of exam results.
- Lastly, the pilot was established to identify security risks not prevalent in center-based delivery.
Favorable Conditions for the Pilot
Circumstances for the pilot test exceeded our expectations, with exam administration in the U.S. nearly evenly split between candidates choosing traditional in-person exams and those choosing the online proctor option over the same time period. Approximately 1,500 candidates, across all nine examinations, received their examination, in each the online and test center environment. This provided an acceptable degree of confidence during our statistical and psychometric assessment of results. We were able to compare results and candidate experience in the pilot program alongside a comparable pool of in-person candidates.
The Online Test Experience
Overall, pilot participants were able to log in, conduct the required identification verification and security protocols, and take the exam without any widespread issue. Technical issues were generally localized to individual systems or bandwidth challenges.
Online exam proctors were responsible for enforcing behavior restrictions during each exam to help prevent cheating and fraud. These are standard controls inherent in any online exam. We have received feedback about interactions with the online proctors, and we are reviewing that alongside all our results in their entirety.
The pilot test did successfully affirm that the online systems can reliably deliver (ISC)² exams via remote proctoring. This helped satisfied the first two of our pilot objectives.
Extended Analysis Required
The integrity of the (ISC)² exam process is of paramount importance for our entire association. Just as we do for all exams, online proctor pilot participants received unofficial results at the conclusion of their examination. All exams, however, are subject to thorough statistical and psychometric analysis of the score data to establish the pass/fail score before releasing official results. Depending on the volume of test takers for a given test, there may be times when scores are delayed for up to eight weeks to complete this critical process. For some online proctor participants, the wait was slightly longer than we had originally anticipated as we worked with our partners to assess and verify all our online exam results.
We undertook extraordinary care to carefully evaluate all potential irregularities in online exam results and candidate behavior during exam administration.
Assessments and Results
In April, we began communicating final official results to pilot participants. While we will not disclose exact figures, generally official results included candidates who passed or failed the exam without any outstanding issues, and a proportion of candidates whose exam scores were revoked due to statistical irregularities and/or candidates who violated the terms of the exam administration protocols.
This extended analysis was necessary. It enabled us to complete the third objective of our pilot program and assess the risks associated with online proctoring. These risks included a higher degree of exam irregularities when compared to the performance of our in-person exams prior to, during and after the online proctor pilot.
Participant Expectations and Communications
This pilot test was an excellent overall test of capabilities and provided ample evidence of how candidates interact – intended and unintended – in an online exam administration context. As stated above, evaluating the results of this pilot test unfortunately required more analysis and investigation than anticipated. This was a learning process for (ISC)², and we now better understand how we can improve policies, procedures and communications for both online and test center exam administrations in the future.
Next Steps
At this time, the online proctoring pilot has concluded, and (ISC)² is offering examinations exclusively via in-person test center administration.
Due to the scope and degree of irregular exam results, we will not be offering online proctored exams at this time. The results of our pilot program, however, will not prevent (ISC)² from continuing to evaluate online exam delivery through additional future pilot tests. We believe that increasing the accessibility of our exams is extremely important, especially for candidates located in geographies where travel to a test center is difficult, time-consuming or costly. To ensure the integrity and security of our exams, we continue to explore options for exam delivery that meet the standards of our certification programs.
Any future pilot programs will be communicated to candidates via www.isc2.org and the Pearson VUE exam registration process.