Modules

Week 1

Week Learning Objectives

By the end of this module, you will be able to

• Navigate the course website and Brightspace site
• Understand the structure of the class and components of assessment
• Define measurement, constructs, and test
• Discuss problems inherent in psychological science measurement
• Describe some early history of testing in China and the major tests developed in the U.S.
• Render a simple Quarto document (.qmd) using RStudio

Task List

1. Review the syllabus
2. Complete the assigned readings
   • Bandalos ch 1
   • Quarto Tutorial
   • Markdown Basics
3. Review the slides
4. Install/Update R and RStudio on your computer
5. Attend the Tuesday and Thursday class meetings

Slides

View slides in full screen

Week 2

Week Learning Objectives

By the end of this module, you will be able to

• Provide examples of norm and criterion referencing
• Compute different types of norm-referenced scores from raw scores
• Perform conversions of item-level data using R

Task List

1. Complete the assigned readings
   • Bandalos ch 2
   • Standards ch 5 (pp. 95–97; 102–105)
2. Review the videos and slides
3. Attend the Thursday class meeting
4. Complete Homework 1 (see instruction on Brightspace)

Slides

View slides in full screen

Lecture Videos

Norm Referencing

Percentile Ranks and Standardized Scores

Normalized Developmental Level Scores

Correction

There was a typo around 4:35, where “developmental-level grades” should be “developmental-level scores.”

Criterion Referencing

Week 3

Week Learning Objectives

By the end of this module, you will be able to

• Summarize and apply the standards for test development
• Describe the components in test specification
• Explain the restriction of range problem and its implications for selecting participants for test tryout.
• Create effective multiple-choice items
• Describe the difference among Thurstone, Likert, and Guttman scaling
• Follow suggestions on writing Likert items
• Describe the impact of response distortion on noncognitive scales

Task List

1. Complete the assigned readings
   • Bandalos ch 3, 4 (pp. 63–70), 5
   • Standards ch 4
2. Review the slides
3. Attend the Tuesday and Thursday class meetings
4. Complete Homework 2 (see instruction on Brightspace)

Slides

View slides in full screen

View slides in full screen

Week 4

Week Learning Objectives

By the end of this module, you will be able to

• Compute item difficulty and discrimination for cognitive items
• Compute interitem correlations and corrected item-total correlations for noncognitive items
• Articulate the importance of reliability in measurement
• Explain what true score and error score are in classical test theory (CTT)
• Define and derive reliability in CTT
• Explain what parallel, tau-equivalent, and congeneric tests are

Task List

1. Complete the assigned readings
   • Bandalos ch 6, 7
   • Standards ch 7
2. Review the slides and note
3. Attend the Tuesday and Thursday class meetings

Slides

View slides in full screen

Week 5

Week Learning Objectives

By the end of this module, you will be able to

• Identify the sources of error associated with internal consistency, test-retest, and alternate forms reliability
• Derive the reliability of a composite made up of parallel and tau-equivalent components (i.e., \(\alpha\) coefficient)
• Describe the factors affecting \(\alpha\)
• Explain why high reliability does not imply unidimensionality
• Explain why reliability is a property of test scores, not of the test itself
• Compute reliability for multilevel (longitudinal) test scores

Task List

1. Complete the assigned readings
   • Bandalos ch 8
   • Standards ch 2
2. Review the slides and note
3. Attend the Tuesday and Thursday class meetings
4. Complete Homework 3 (see instruction on Brightspace)

Slides

View slides in full screen

Week 8

Week Learning Objectives

By the end of this module, you will be able to

• Distinguish between reliability and agreement
• Articulate the limitations of Cohen’s \(\kappa\) for interrater agreement
• Choose the right intraclass correlation (ICC) formula for different study designs and decisions
• Obtain confidence intervals for ICCs

Task List

1. Complete the assigned readings
   • Bandalos ch 9
2. Review the slides and note
3. Attend the Tuesday and Thursday class meetings
4. Complete Project Prospectus (see instruction on Brightspace)

Slides

View slides in full screen

Week 9

Week Learning Objectives

By the end of this module, you will be able to

• Provide some example applications of the generalizability theory (G theory)
• Contrast G theory with CTT (Table 10.1)
• Explain the differences between crossed and nested facets, and between random and fixed facets, and between G studies and D studies
• Estimate variance components and the G and \(\phi\) coefficients for two facets designs

Task List

1. Complete the assigned readings
   • Bandalos ch 10
2. Review the slides and note 1 and note 2
3. Attend the Tuesday and Thursday class meetings
4. Complete Homework 4 (see instruction on Brightspace)

Slides

View slides in full screen

Week 10

Week Learning Objectives

By the end of this module, you will be able to

• Contrast historical and current view of validity
• Explain the argument-based approach to validity
• Describe types of evidence that should be obtained, and the common approaches to obtain them
• Articulate how one can obtain convergent and discriminant evidence in an MTMM matrix

Task List

1. Complete the assigned readings
   • Bandalos ch 11
   • Standards ch 1
2. Review the slides and note
3. Attend the Tuesday and Thursday class meetings

Slides

View slides in full screen

Week 11

Week Learning Objectives

By the end of this module, you will be able to

• Write down the mathematical expression of the common factor model
• Define factors, loadings, and uniqueness
• Describe the differences between exploratory and confirmatory factor analysis
• Be familiar with the major considerations in EFA, including factor extraction and rotation
• Conduct EFA in R, and interpret and report the results

Task List

1. Complete the assigned readings
   • Bandalos ch 12
2. Review the videos, slides and note
3. Attend the Tuesday class meeting, and review posted lectures on Thursday

Slides

View slides in full screen

Lecture Videos

Factor Analysis: Overview

Common Factor Model

Assumptions

Check your learning

Local independence is a key assumption in the factor model. Which of the following best illustrates local independence?

Items are independent of each other given the latent trait.
Items from different factors are uncorrelated.
Items are independent to factors they do not load on.
Multiple items share the same stimulus.

EFA vs. CFA

Check your learning

A researcher is translating a well-established measure of self-esteem for a new population. They want to examine the factor structure of the measure in the new population. Should they use EFA or CFA?

EFA, because the factor structure is not known.
CFA, because the factor structure is known.
EFA, because the factor structure is known.
CFA, because the factor structure is not known.

Factor Extraction

Check your learning

What is the main difference between component analysis and factor analysis?

Component analysis uses the correlation matrix, while factor analysis uses the covariance matrix.
Component analysis uses eigen decomposition, while factor analysis uses Cholesky decomposition.
Component analysis uses the reduced correlation matrix, while factor analysis uses the full correlation matrix.
Component analysis does not assume unique factors, while factor analysis accounts for uniqueness of the items.

Determining Number of Factors

Note: The K1 rule is also known as the eigenvalue greater than 1 rule, and the Kaiser criterion.

Check your learning

What is the main problem with the K1 rule?

It usually underestimates the number of factors.
It does not account for the the interfactor correlations.
It ignores the fact that the largest eigenvalues in a random matrix are usually greater than 1.

Parallel Analysis

Check your learning

The problem with the K1 rule is related to ordered statistics, which can be illustrated in this analogy. Suppose three people plays a game, where the winner gets $2, the second place gets $1, and the last place gets $0. Assume that the three people are equally skilled. What is the expected amount each person will get in each round? What is the expected amount the winner in each round will get?

Each person is expected to get $1, and the winner is expected to get $1.
Each person is expected to get $1, and the winner is expected to get $2.
Each person is expected to get $1.5, and the winner is expected to get $1.5.

Week 12

Week Learning Objectives

By the end of this module, you will be able to

• Explain the differences between EFA and CFA
• Compute the degrees of freedom for a CFA model with covariance matrix as input
• Explain the need for identification constraints to set the metrics of latent factors
• Explain the pros and cons of the global \(\chi^2\) test and the goodness-of-fit indices
• Run a CFA in R and interpret the model fit and the parameter estimates
• Write up the result section reporting CFA results

Task List

1. Complete the assigned readings
   • Bandalos ch 13
2. Review the slides and note
3. Attend the Tuesday and Thursday class meetings
4. Complete Homework 5 (see instruction on Brightspace)

Slides

View slides in full screen

Week 14

Week Learning Objectives

By the end of this module, you will be able to

• Explain the meaning of item difficulty and item discrimination in one- and two-parameter IRT models
• Estimate item and person parameters from a Rasch model and a 2PL model in R
• Assess model fit

Task List

1. Complete the assigned readings
   • Bandalos ch 14
2. Review the slides and note
3. Attend the Tuesday and Thursday class meetings
4. Complete Homework 6 (on CFA; see instruction on Brightspace)
5. Prepare for the project presentation (see rubric on Brightspace)

Slides

View slides in full screen

P.S.: If you’d like to print the slides to PDF, follow the instruction here.