Writing Items

Learning Objectives

  • Create effective multiple-choice items
  • Describe the difference among Thurstone, Likert, and Guttman scaling
  • Follow suggestions on writing Likert items
  • Describe the impact of response distortion on noncognitive scales

Cognitive Items

  • Objective items and performance assessments
    • Focus on maximum performance
  • selection-/regconition-type vs. supply-type
  • Multiple-choice, true-false, matching, short-answer, etc

Writing Multiple-Choice Items

  • Be clear in item stem
  • Use positive form (avoid not or no, or bold/underline them)
  • Make all options the same length, grammatically consistent, and plausible
  • Avoid verbal clues (e.g., “always”/“never”)
  • Alphabetize/randomize the options

Noncognitive Items

  • Thurstone scaling

  • Likert scaling1

  • Guttman scaling

Exercise

Example of Guttman Scale

Writing Likert Items

See all 16 suggestions in the Chapter

  • Be clear, concise, and avoid ambiguity
  • Avoid statements that everyone would agree or disagree
  • Include negatively oriented statements, but avoid negatively stated statements

Theories of Item Responding

  • Interpreting the item
  • Generating a response
  • Formatting and reporting the response
  • Editing the response

We want respondents to optimize, not satisfice

Interpreting the Item

  • Effect of response options

    • E.g., How often do you use your smartphone?

      • up to 30 min per day
      • 30 min to 1 hour
      • 1 to 1.5 hours
      • 1.5 to 2 hours
      • . . .
      • more than 4.5 hours

Generating a Response

  • If something is not important for someone, they may
    • draw on accessible information
    • general impressions or stereotypes
    • just choose neutral/no opinion
  • Response consistency is generally higher for those with a strong opinion on the subject of the item
  • Be aware of “priming” and order effects

Implications for creating scales

  • Make wording as clear as possible
  • Do not use vague terms or ambiguous phrases
    • E.g., often, a few, military machine
  • Consider possible effects of context
    • E.g., Asking How many children do you have? to teachers

Implications for creating scales (cont’d)

  • Encourage responders to take their time in answering
  • Order items within a scale randomly
    • Could be randomized across participants

Formatting and Reporting the Response

  • Check response options match the item
    • E.g., sometimes vs. agree
  • Label all response options when feasible

Editing the Response

  • Social desirability
    • To reduce, put respondents at ease, and ensure confidentiality
    • Or include a measure of it (e.g., BIDR; SDRS)1

Problems in Measuring Noncognitive Outcomes

  • Response distortion: systematic tendency to respond to a range of items on some basis other than the intended content

    • Response style: habitual distortion

    • Response set: only on certain occasions (e.g., when fatigued)

  • Examples
    • Social desirability
    • Acquiescence
    • Extremity and moderacy
    • Malingering

Managing Response Distortion

Practical Issues in Noncognitive Scale Construction

  • Number of scale points
    • Respondents can reasonably be expected to discriminate ___ categories
    • Visual analogue scales
      • Can potentially detect small differences, but support is mixed and depends on context
    • Depends on age: 3–4 options may be what children in elementary school can differentiate
  • Negatively oriented items
    • Does being sad = not happy? Positive vs. negative affects?
    • Is method variance artificial?
      • Some research supports it is from response style
  • Neutral option
    • Include it if many respondents may be neutral or unaware of the issue