Items Types: Overview and Best Practices for Using

Multiple Choice is the most fundamental and familiar item type in this family. Students are presented with several answer choices, and must pick one of them. 

• It is usually best to offer 4 or 5 answer choices, although it is also possible to use Multiple Choice items for true/false or similar questions. 
• Incorrect answers known as “distractors” are ideally chosen to represent common misconceptions or errors, so that students’ answer choices will provide insight into their knowledge and understanding. 

The following information applies to the answer choice order of Multiple Choice, Multiple Select, EBSR, and Inline Dropdown items, but not Matching Table or Multiple Binary:

• By default, answer choices can potentially be shuffled when a question is delivered. This won’t always happen when an assessment is assigned. The administrator decides whether or not answer choices should be shuffled. If, however, a particular question has had “Lock Choice Order” set to true, then answer choices will always be fixed, even if shuffling is otherwise being used on the assessment.
• One reason to lock answer choices is if there is a “none of the above” or “all of the above” answer choice. It wouldn’t make sense for such a choice to appear anywhere but last. (There is not currently a mechanism to lock only the last answer choice.)
• Another reason to lock answer choices would be if they are in a particular order that should always be used. For instance, when the answer choices for a math question are numbers, it is a common practice to list them in numerical order. 

Multiple Select is like Multiple Choice, except that students can select more than one choice. In the language of computer interfaces, these items use square “check boxes” (which are “multi-select”) rather than circular “radio buttons” (which are “single-select”). 

• It is often a good idea for a Multiple Select question to tell the student the number of answers to select. E.g., “Choose TWO”. This is especially true if partial credit is not going to be awarded. Questions that don’t indicate how many choices should be selected may be quite difficult to answer correctly, and this is particularly true of reading comprehension questions where there is some amount of judgment involved. When answers are completely objective and unambiguous, as is common in math, it may be more reasonable to omit this information. (e.g., “Choose all expressions that represent an even number.”)
• It is usually best to avoid having a “Multiple Select” question where only one answer choice is actually correct, which would likely be viewed as a “trick question”.
• We generally recommend offering 4 to 6 answer choices, of which no more than n/2 + 1 are correct—i.e. no more than 3 of 4 or 5, and no more than 4 of 6.
• When partial credit scoring is used, scoring is based on the total number of correct answer choices. For instance, if there are two correct answer choices, then ½ point will be awarded for each that is selected; if three, then ⅓ point will be awarded for each that is selected. Deductions for incorrect answers are only taken if the student marked more answer choices as correct than was in fact the case. For example, if the correct answer is ABC, then both AB and ABD would be scored as ⅔, whereas ABDE would be scored as ⅓. 

EBSR (Evidence-Based Selected Response). This item type essentially consists of two linked multiple choice questions, but scoring is different from stand-alone multiple choice items. Generally the second question should ask the student to identify the evidence for the first question; usually these questions are based on a passage. If a pair of questions do not adhere to this structure, EBSR may not be the right item type to use. For instance, the first question might ask the student to choose a character’s emotional state, and the second question might ask the student to choose the textual evidence from which this can be inferred.

• If all-or-nothing scoring is in effect, then both parts of an EBSR item must be answered correctly for credit to be awarded.
• If partial credit scoring is in effect, then a student can be awarded half credit if only the first part is correct, but will receive no credit if only the second part is correct. 
• It is also possible to make the second part a Multiple Select question rather than a Multiple Choice question.

Inline Dropdown. In an inline dropdown item answer choices appear in-line with text, in a dropdown menu (as the name suggests). An inline dropdown item can have a single dropdown menu (“response area”), or several. In each case, just one response will be correct (as with Multiple Choice items). 

• It is usually best to keep Inline Dropdown answer choices somewhat succinct. A word, or a relatively short phrase, is ideal. For longer answer choices, especially those that might require more than one line,  Multiple Choice may be a more appropriate option.
• Compared to Multiple Choice items, it is more common to find situations where it makes sense to offer only two or three answer choices, at least for Inline Dropdown items with several response areas.

Multiple Binary consists, in effect, of several multiple choice “questions”, all of which share a common set of “choices”. These items are set up as a table, with the “questions” appearing as row headers, the “choices” appearing as column headers, with a set of radio buttons for each row / “question”. So for each “question”, the student needs to select a single “choice”. (“Question” is not necessarily the best word to describe these things; they are more often things that need to be classified than questions to be answered.)

• Although it is acceptable for “questions” or even “choices” to require more than one line, Matching Table typically works best when both are kept brief. 
• The sorts of sets of choices that work well include things like:
  • “True” and “False”, “Yes” and “No”, or “Correct” and “Incorrect” 
  • “Negative”, “Zero”, and “Positive”
  • “Article 1”, “Article 2”, “Both Articles”, and “Neither Article”.
  • “Choices” can also be highly item-specific; the above are just examples.       
• Although this item type can be used for matching type questions, where there are n “questions” to be matched to n choices, Match List may be a better fit to questions with this structure. 

• This is essentially a “multiple select” version of Matching Table, which allows a student to select more than one “choice” per “question”. 
• It is generally best to ensure that at least one “choice” should be selected for each “question”; including a “question” for which no “choice” is correct may confuse students. If that seems problematic, consider adding a new “choice” that is functionally “none of the above”. (You will probably want to label it more specifically; e.g. if the other choices are “Factor of 2” and “Multiple of 2”, then the “none” choice might be labeled,  “Neither a Factor Nor a Multiple of 2”). 
• In contrast with a stand-alone Multiple Select item, it is fine to have individual “questions” for which only one “choice” should be selected. But if that is the case for every “question”, then Multiple Binary is likely a better fit. 

Explicit Constructed Response (ECR) is appropriate for questions that have very well-defined answers, such that a student can reasonably be expected to type the correct answer so that it exactly matches the “official” correct answer specified by the item author. 

• For an answer to be recognized as correct, it must EXACTLY match the answer specified by the author. The ability to make validation indifferent to capitalization is planned, but is not currently available, so if the primary correct answer is  “minute”, and you want “Minute” and “MINUTE” to also be recognized as correct, they would each have to be added as alternate answers.
• ECR can be used with numerical answers, although in some cases Math Equation Response may be a better fit.
• ECR is also appropriate for names, and single-word answers where there is no ambiguity. If the correct answer is a multi-word phrase, consider whether this is really the best item type for that question.
• ECR response areas are usually embedded inside of text. E.g. “The capital of New York is [      ].” An ECR item can have one or more response areas.

Math Equation Response (MER) is similar to Explicit Constructed Response (ECR), but designed specifically for answers that are mathematical expressions. The biggest differences relative to ECR are as follows:

• Students are provided with an on-screen math editor that allows them to enter mathematical expressions like fractions and radicals, as well as mathematical characters like the multiplication and division symbols. (Students can also enter their responses via the keyboard.) Various editors are available, with buttons to support symbols and notation commonly used in different grade levels (e.g. “Grade 6-7”) and topic areas (e.g. “Statistics”). The student cannot change the editor, so choose one that is a good fit to the question. 
• Answers are validated as correct or incorrect by a validation engine, which provides more flexibility than simple string matching. An overview of MER validation options follows. 
  • There are two validation options, Literal and Symbolic. 
  • Literal Validation is similar to the basic string validation used by ECR, but with several features that make it more flexible.
    • Leading zeroes are always ignored. So 0.5 ⬌ .5. (Read ⬌ as “is equivalent to”. In other words, if the correct answer is specified as either 0.5 or .5, the other will automatically validate as correct.)
    • Spaces are generally ignored. So “a+b” ⬌ “a + b”.
    • Thousands-separator commas are ignored. So “1,000” ⬌ “1000”
    • The multiplication sign and dot are interchangeable. So “5×a” ⬌ “5⋅a”
  • Symbolic validation, by contrast, assesses whether the expression entered by the student is mathematically equivalent to the correct answer. So if the correct answer specified by the author is 1 1/4, symbolic validation will also accept 5/4,  1 2/8, or 1.25, all of which would be rejected by literal validation. 
  • Literal validation also offers two optional settings that tweak its behavior. 
    • If Allow Trailing Zeros is set to true, then trailing zeros will be ignored (as leading zeros always are) . So 5 ⬌ 5.0 ⬌ 5.00 ⬌ 5.000, etc., 
    • If Ignore Order is set to true, then 
      • ab  ⬌  ba,  a+b  ⬌  b+a,    ab+c  ⬌   c+ba 
      • a=5  ⬌   5=a 
      • 0<x≤5  ⬌ 5≥x>0
• MER items authored in New Item Bank have a single stand-alone response area. It is not currently possible to create an MER item with multiple response areas, or with a response area that is inline with text.
• For numerical answers, when is it better to use MER vs. ECR? 
  • For simple whole-number answers under 1,000, ECR is simpler.
  • For decimals, MER has some advantages (e.g. treatment of leading zeros)
  • For fractions MER is the preferred choice, since students can format them properly, rather than just seeing a typed expression like “1/16", which is what you would get with ECR.
  • For numbers over 999, MER will automatically recognize commas used as thousands separators, without needing alternate correct answers.
  • For dollar values and the like, MER can recognize responses with trailing zeros (e.g. “4.00”) without needing an alternate correct answer (depending on how the validation is configured). 
  • ECR allows response areas to be embedded within other text, which is not currently possible with MER, although that is under development.

• CR items must be manually scored using a rubric. For that reason, it is recommended that they be used only in situations where other item types cannot adequately assess the skill in question. 
• For math CR questions where a student is asked to explain their work, consider turning on the “Equation Editor” setting, which will allow the student to access a math editor when writing their answer. If you do this, choose an editor consistent with the demands of the question. 

• These items must be scored by hand using a rubric. For that reason, even though this is a very flexible item type, we suggest only using it in situations where other item types cannot adequately assess the learning objective. 
• It often makes sense to include a background image, which students will draw their answers on top of. For instance: 
  • The background image could include shapes that students are asked to divide into thirds, or to draw lines of symmetry on.
  • The background image could include a diagram, where students are asked to label certain parts. (The drawing tool supports basic text entry.)
• Bear in mind that the drawing tools provided are rudimentary, so item demands should be consistent with their limitations.   

• This item type takes as its starting point a block of text. The author designates portions of that text (words, sentences, etc.) as “tokens”, which students can select, and then designates one or more of the tokens as the correct answer. 
• Sometimes, an item will have a small number of tokens (perhaps 4 to 6), only 1 or 2 of which are correct, making for a student experience quite similar to a Multiple Choice or Multiple Select item. In other cases, every word, or every sentence, will be “tokenized”, potentially making for a quite different student experience. For instance, a student might be asked to identify all the misspelled words, or all the adjectives. 
• Students can select a token by clicking on it, which highlights it. Subsequent clicks toggle the selection state. Students can always select at least as many tokens as the number of selected tokens in the correct answer, but may be restricted from making additional selections beyond that number. For full credit, the student’s selections must fully match the selections in the correct answer.  
• Hot Text has a setting called Highlight Choices. 
  • When Highlight Choices is on (as it is by default), each token will be surrounded by a dashed box, making it obvious on sight what the tokens are. This may be a good option for items where just a few words have been tokenized, since it will otherwise take the student a few seconds just to identify the available answer choices.   
  • When Highlight Choices is off, tokens are not called out, but mousing over any part of a token will highlight the entire token. This is generally preferred for Hot Text items where every or almost every word or sentence is a token, since drawing a box around every word can make the text difficult to read.

• This item type takes as its starting point an image. The author designates “hot spots” by click-and-dragging to create rectangular regions on the image, and then select to designate one or more of them as the correct answer. 
• Hot spots appear as semi-transparent colored rectangles; selected hot spots are indicated by a heavy border. The author can choose from three colors (red, yellow, and blue) for both the semi-transparent regions, and the borders. For black and white or gray-scale images, any color choice is fine, but for color images it is recommended that you try the various options to find one that makes the hot spots clearly visible without obscuring important features of the image itself.  
• Similar to Hot Text, there can be just a few hot spots, or a great many. For instance, if the image consists of an unlabeled map, the author might designate every region as a hot spot.  
• The default authoring behavior is to limit the item to just one correct answer, in which case clicking on a hot spot will select it (designating the correct answer for an item author, or the student’s answer choice for a student). To change this, the Multiple Correct Responses setting must be turned on; in this case clicking a hot spot will toggle its selection status. When this setting is on, there is no mechanism to limit how many answer choices a student can select. To receive full credit, a student must select all the correct answers, and no incorrect answers.

• In this item type, the author creates unique “choices”, draggable chunks of text, and “response areas”, containers, into which choices can be dragged (one choice per response area). Initially, students see all the answer choices grouped together; to answer the question, choices are dragged to response areas. 
• Response areas are embedded inside of a text-based template. Although the template can include images and math markup, neither of these can itself hold a response area. To answer an item, students must drag the correct choice(s) to the matching response area(s).  
• Choices can include mathematical markup, but cannot currently include images. 
• Each response area needs to have a unique correct choice. For instance, it would not be a good idea to have an item that says “To create purple, mix [red] and [blue]” since if a student dragged “blue” to the first response area and “red” to the second, that would be marked as incorrect.
• By default, each choice can only be used once; once a choice has been placed into a response area, it will no longer be present in the “pool” of available choices. To change this behavior, turn on the “Allow Duplicates” setting. Then, answer choices will remain in the “pool” even after they have been used, such that the same answer choice can be dragged to multiple response areas.
• It is fine to have more choices than response areas, such that there will be “left over” choices.
• Although it is currently possible to create an item with unused response areas, response areas into which no choice should be placed, this is generally not a recommended practice. 
• By default, choices appear below the template; to change this, select a different Choices Position setting. 
• By default, the order in which choices appear to students will be randomized. To change this, turn on the Lock Choice Order setting, and then answer choices will be presented in the order in which they were created. For the most part, randomizing the order is recommended. If you choose not to randomize, pay attention to the order in which you create answer choices, to make sure that they aren’t “pre-sorted” into the correct order. 
• If an item has multiple response areas, the student must place the appropriate choice into each of them to receive full credit. 

• This item type requires students to sort a list of choices. Uses could include arranging choices in chronological order (e.g., incidents from a story, or historical events), numerical order (e.g. ordering numbers or expressions), or alphabetical order.  
• While this results in an item type that looks a little like Match List, partial credit scoring is handled differently. For instance, if the desired sort order is 1,2,3,4,5, and a student placed the tokens in the order 5,1,2,3,4, this would qualify for a fair amount of partial credit, because although every choice is in the “wrong” absolute position, most of the choices are correctly positioned relative to one another. 
• The item type supports the ability to either have students sort the answer choices “in place”, or to offer an interface in which students drag them from an unsorted list on the left to a “placement area” on the right where the student builds a sorted list. In general, the use of a placement area is recommended, since it provides a clear visual indication of the student’s progress towards fully answering the question. It does require twice as much horizontal space, however, so with longer choices sorting in place may be preferable.
• When using a placement area, there is also an option to provide numbered guides. This means that the response areas on the right will be numerically labeled, 1, 2, 3,  When a choice is dragged to the placement area, it will “cover up” the numeric label. Typically this just comes down to an aesthetic preference. 

• The author configures a number line, defining the range of values to be displayed, and the intervals at which tick marks and labels will appear. For instance, a number line might span the range from −1 to 1 with tick marks every 0.1 units and numeric labels every 0.2 units. 
• The author also decides what plotting tools the student will have access to, and uses them to define the correct answer. Available plotting tools include points and segments and rays with various combinations of open and closed endpoints. Points and endpoints must align to tick marks on the number line. 
• Common use cases include plotting the solution to an inequality or equation, questions designed to assess conceptual understanding of fractions, and of course questions about the number line itself.

• Graphing is basically a two-dimensional version of Number Line. The author configures a two-dimensional coordinate grid system, defining the range of values to be displayed on each axis and the intervals at which grid lines and labels will appear.
• The author can also include various sorts of labels on the graph, and add background shapes to it.
• The author also decides what plotting tools the student will have access to, and uses them to define the correct answer. Available plotting tools include points, lines, rays, parabolas, circles, and more. There is also a tool for labeling graph objects. The points used to define a graph object must align to the grid lines defined by the author. 
• There are three main use cases for Graphing items:
  • Coordinate plane questions. E.g., “Graph the point (3, -4) and label it as A”, or “Graph the solution to the equation y = 3x+2”.
  • Data graph questions. E.g. “Sue walks in a straight line for 30 minutes at 3 miles an hour. The x-axis represents minutes since she started walking, and the y-axis represents distance from her starting point (in miles). Graph a line segment to represent her position over time.” 
  • Geometry questions. E.g. “draw all lines of symmetry for the shape on the graph”