Microsoft Certified Professional Exam Development

Microsoft Certified Professional (MCP) exams are available forseveral Microsoft OSs, development tools, and desktop and server applications.Microsoft's Certification Strategy and Development group develops the exams.Exam development professionals, not experts in the exam topic areas, staff thisgroup. To add topic expertise for the exams, Microsoft relies on internalMicrosoft groups (the product group that markets and develops the product andthe product support specialists who support the product), independentcontractors (such as I), and users outside Microsoft. (For information aboutbecoming a technical contributor, see Table 1 on page 74.)

For several years, I've helped develop a variety of MCP exams. My area ofexpertise is Visual Basic (VB). I wrote about half the questions for the VB 3.0exam and modified other questions. I also constructed part of the VB 4.0 examthat Microsoft released on April 8, 1996.

If you've taken or are planning to take an exam, if you rely on examresults for hiring and personnel-related decisions, or if you want to helpdevelop an exam, you'll be interested in what goes into making an MCP exam.Here's a brief overview.

Exam Development
Although Microsoft constantly refines the process for developing MCP exams,all exams go through several major steps. These steps are job analysis,objective domain development, exam blueprint survey, item pool development, examalpha (ensuring technical accuracy), exam beta, cut score, certification exampublication, localization, and performance monitoring.

Some steps require explanation. After the exam developers create the examobjectives, the developers survey about 30 internal and external reviewers. Thisexam blueprint survey helps refine the objectives. Following this review, thedevelopers solidify the exam topics.

Adding an item for a topic not on the list of objectives is almostimpossible. I can't tell you how often developers submit a great question onlyto hear, "What objective does this item map to?" If an item doesn'tmap to an objective, it doesn't get on the exam.

The cut score is a number that determines the pass or fail cutoff for eachexam. Each developer decides what percentage of minimally competent candidatesmust answer each item correctly. The developers then sum all these percentagesand divide by 100 to calculate a suggested cut score. They then average the cutscores from everyone contributing in this phase to determine a final cut scorefor the entire exam.

Bloom's Taxonomy
Microsoft gives test developers a lot of guidance, including instruction onBenjamin Bloom's Taxonomy of Educational Objectives for theCognitive Domain. Bloom's taxonomy has six levels:

The idea behind this taxonomy is to test the student's ability to useknowledge. Microsoft instructs developers to prepare items in high categoriessuch as evaluation, synthesis, and analysis, instead of testing raw knowledge.

The 7 Cs
The internal developers thoroughly evaluate each item before it appears onan alpha and then beta exam. Microsoft internally identifies this evaluationprocess as the 7 Cs, for correct, clear, consistent, complete, concise, current,and culturally unbiased. The developers use several questions to evaluate the 7C's, including Is the topic relevant? Is the wording straightforward? Is theitem free of contradictions? Are there at least three distractors that areplausible to an uninformed person? Are the answers short and simple? Is theinformation current? Is the item fair? Is the item free of ethnic, cultural, andgender bias?

Microsoft gives external developers a shorter list. For example, theytypically don't need to worry whether an item is culturally unbiased; theeditorial staff addresses such concerns later in the development process.

Types of Items
Exam items can take several forms. Multiple-choice questions can have onecorrect answer, or multiple correct answers (and the student knows how many arecorrect), multiple correct answers (and the student doesn't know how many arecorrect), multiple-response questions with one scenario and several questions(the student must select the best answer), free-form text (for example, commandlines where several possible syntax alternatives can work; one such command isDIR), and graphics with hot-spots (the student points and clicks on a correctanswer).

I make the exams as difficult as possible. I favor items with multiplecorrect answers where the student doesn't know how many are correct. Thesequestions are like a string of several true/false questions, and the studentknows only that at least one question is true. Unfortunately, these items givemore unpredictable test results than other types of items. They would probablybe more useful if a mechanism were available for calculating partial credit.

Although the exams emphasize questions with one correct answer, some items(especially the multiple-response questions) require students to select the bestanswer. Microsoft uses this approach for questions on troubleshooting andplanning, where professional judgment is important.

Item (Question) Development
The taxonomy and 7 Cs are somewhat abstract, but Microsoft gives developersspecific instructions. For example, no item can be a verbatim copy frominstructional or reference materials. Also, each item must reflect facts, notopinions; use straightforward language; not be tricky; be relevant, not obscureor trivial; contain specific determiners (all, none, always, only, never, best,most) only when there are no exceptions; contain no negative words (no, not);and contain no indefinite qualifiers (some, many, several, few, usually). Inaddition, the answers must present one unquestionably correct answer; bemutually exclusive (i.e., not overlap); contain no phrasing such as "all ofthe above," "none of the above," or "both A and B."Distractors (incorrect alternatives in multiple-choice questions) must beplausible to an uninformed person and unquestionably incorrect to an informedperson.

Sample Question
The sidebar, "How an MCSE Exam Question Is Created," on page 72shows the development of a typical exam item. Embedded comments are visible tohelp you understand the gradual refinement process.

Several things are wrong with the original version of the item: Genericreferences to a network administrator make the item hard to follow; the itemdoesn't relate to a scenario, and answers don't follow from the question; theitem contains a hidden negative and sentences in passive voice; and answer C isthe only one that contains two conditions.

Each MCP exam involves a lot of work and a lot of people at various steps.Microsoft evaluates exam statistics to continually refine the process. To obtainmore information about individual tests, visit the Microsoft Training andCertification Web page at http://www.microsoft.com/train_cert.

To become a technical contributor for Microsoft certification exams, send thefollowing information by mail, email, or fax to the Certification DevelopmentTeam:

Developing an exam doesn't give you an edge in taking the exam. Yougenerally see only some items, which aren't in final form, and several monthscan pass between helping develop the items and taking the exam. That's justenough time for everything--including the distractors--to look vaguely familiar.