Adaptive Testing

Can a few target questions determine who's certifiable?

Microsoft has changed its certification test-delivery format several times. Examinees who have recently taken the Windows NT Core exams might have faced the newest format—adaptive exams. This new format scares many people who fear change, especially when the change affects certification. People have canceled their exams because they find out the exams are adaptive. However, Computer Adaptive Testing (CAT—in which a computer administers the adaptive exam) isn't new: Intelligence tests have used CAT for decades, higher education entrance exams currently use CAT, and Novell has been using the adaptive format to administer tests since 1993. In addition, adaptive testing provides benefits to both the testing center and the examinee. Let's take a close look at the adaptive format: the definition of an adaptive exam, how it differs from the standard fixed-length exam format, the adaptive format's benefits to the tester and the examinee, tips to prepare for an adaptive exam, and Microsoft's certification testing strategy.

What Is an Adaptive Exam?
A standard multiple-choice exam uses a set number of questions, but an adaptive test uses as few target questions as possible to assess a candidate's ability level. The number of questions a candidate receives depends on how well the candidate performs throughout the exam. The adaptive exam bases each new question on the candidate's response to the previous question.

The adaptive exam's first question is of medium difficulty. If the candidate answers this question correctly, the next question is more difficult; if the candidate answers the first question incorrectly, the next question is less difficult. The same formula applies to the rest of the exam questions, and eventually candidates either fail the exam because they answer too many questions incorrectly or pass the exam because they answer enough questions correctly. The exam's length and the number of questions each candidate receives vary for each exam, but candidates usually answer a minimum of about 15 questions correctly before they pass and a maximum of about 25 questions before they fail.

Figure 1 shows three adaptive-exam scenarios. Scenario 1 illustrates a candidate who passes the test quickly because the candidate consistently answers difficult questions correctly. In Scenario 2, the candidate fails the exam quickly because he or she incorrectly answers the first questions and continues to waiver at the medium-level questions. Scenario 3 illustrates the worst case because the test program can't quickly ascertain the candidate's ability level, so the program must ask the maximum number of questions to determine whether the candidate passes or fails the exam. In this case, the minimum score determines the candidate's success.

This format appears to show mercy to candidates who get questions wrong, but candidates who answer incorrectly receive a lower score and endure a longer, more stressful exam. Another perception about Microsoft and Novell adaptive tests is that the tests attempt to exploit a person's weakness on a given topic. Microsoft and Novell reject such assertions and claim their tests deliver questions of varying difficulty in the most efficient way to determine a candidate's ability. (For preparation resources and more information about adaptive exams, see "Adaptive Exam Online Resources.")

Adaptive vs. Fixed-Length Standard Exams
An important difference between adaptive and fixed-length standard exams is that you can't go back and review your answers on an adaptive exam. The adaptive exam bases each question on your answer to the previous question, so going back to change or review an answer is impossible. After you answer a question and move on, the adaptive exam locks in your previous answer.

Another difference is that adaptive exams don't break down your final score by subject, whereas standard exams provide a score for each category on the exam. For example, a standard exam tells you that you performed well on the installation and configuration questions and that you didn't answer many troubleshooting questions correctly. Microsoft claims it eliminated individual scoring reports from adaptive tests because the tests are too short to provide an accurate breakdown of a candidate's performance by category.

In addition, adaptive exams are over quicker than standard exams because the testing program selects questions based on your previous responses. Thus, an adaptive program requires fewer questions to determine your ability level than a standard program requires.

Adaptive Exams' Benefits
A major reason why Microsoft and Novell administer adaptive exams is to keep their certification tests secure. Although examinees must sign testing agreements that strictly prohibit sharing information, some people who take certification exams share questions with their colleagues through email or post questions on the Internet. Adaptive exams make it more difficult for people to compromise exam questions because examinees see fewer questions during the exam. In addition, examinees that have different ability levels will see different questions, so one candidate's report of exam questions is probably useless to other candidates.

In addition to increased security, Microsoft cites the following benefits of administering adaptive tests:

Difficult Questions
Each candidate has different abilities, so determining which questions to rank with a higher difficulty rating can be tricky. Test developers currently use two methods to determine which questions to rate as difficult.

One method is to use a beta exam and record which questions most people answer correctly and which questions most people answer incorrectly. The test developers consider the questions that more people answer incorrectly as difficult and the questions that most people answer correctly as easy. Novell currently uses this method to determine which questions are difficult.

Alternatively, the test developers release fixed-length exams to the public and collect statistics about each question until the developers have enough information to convert the fixed-length exams to adaptive exams. Microsoft currently uses this method to implement its adaptive exams.

Tips for Taking Adaptive Exams
Examinees don't usually get to choose whether they take an adaptive or fixed-length standard certification exam. The testing vendor, not the examinee, determines the exam format.

Although you can tell how well you're doing on an adaptive exam by the difficulty of the questions, you'll just waste time and energy if you try to determine whether the previous question was more difficult than the current question. Keep in mind that questions that seem easy to you might seem more difficult to others and thus bear a rating of difficult. Don't worry about the difficulty level of each question; concentrate on correctly answering the present question.

People might advise you to miss the first question so the following questions will be easier. Although the exam will present easier questions if you answer the previous questions incorrectly, you'll increase your chances of failing the exam. Answering any question incorrectly, even if you do so purposely, only makes your exam longer and enables you to fail more quickly. This method is dangerous and lowers your final score.

Microsoft claims that adaptive exams are easier than standard fixed-length exams, and some examinees think that adaptive exams are more difficult. However, the consensus is that adaptive testing is no easier or harder than fixed-length standard exams. When you're facing a certification exam, you need to know the material no matter which form of exam you receive.

Microsoft's Testing Strategy
In mid-March 1999, Microsoft released the Networking Essentials and TCP/IP exams in another new format—a seeded exam. A seeded exam contains trial questions that Microsoft might include in future exams, but these trial questions don't count toward your final score. You can't tell which questions are trial questions, so you must attempt to answer all questions correctly.

Some examinees interpreted the release of the seeded exam to mean that Microsoft is backing away from the adaptive-exam format. I called the Microsoft Certified Professional (MCP) program to find out why Microsoft didn't use the adaptive format for the Networking Essentials and TCP/IP exams. The customer service representative told me that rather than give up on the adaptive format, Microsoft has developed a new certification-testing strategy: The company plans to rotate the testing format so examinees can't develop test-taking techniques based on one certification exam format. Thus, when you take a Microsoft certification exam, you might get any of the following exam formats:

Microsoft reserves the right to modify the test format to prevent you from preparing for the format of the exam. The company wants to force certification candidates to focus on the test's concepts, not the format in which the test presents the questions.

Can You Adapt?
You can view adaptive testing as a positive development. Adaptive testing benefits testing centers because examinees can't easily share questions and the testing centers can test more people in less time. In addition, examinees benefit because the adaptive tests are over quicker than fixed-length standard exams.

Most important, be prepared for any exam format. If you know the test concepts, you'll do fine.