Parallel, Tau-Equivalent, and Congeneric Tests

# install.packages("here")
library(tidyverse)
library(modelsummary)  # for summarizing data

Parallel Tests

Here we simulate scores for two tests that are parallel:

set.seed(2237) # setting the seed ensure reproducibility
num_person <- 1000 # number of respondents
# True scores for Test 1
t1 <- rnorm(num_person, mean = 20, sd = 5)
# Error scores for Test 1
e1 <- rnorm(num_person, mean = 0, sd = 2)
# Observed scores for Test 1
x1 <- t1 + e1
# True scores for Test 2
t2 <- t1 # parallel tests have equal true scores
# Error scores for Test 2
e2 <- rnorm(num_person, mean = 0, sd = 2)
# Observed scores for Test 2
x2 <- t2 + e2

# Merge into a data frame
test_df <- data.frame(x1, x2)
# Get means and variances
mv <- datasummary(x1 + x2 ~ Mean + Var, data = test_df,
                  output = "data.frame")
# Correlation, adding means and variances
datasummary_correlation(test_df, add_columns = mv[-1])

Warning in attr(x, "align"): 'xfun::attr()' is deprecated.
Use 'xfun::attr2()' instead.
See help("Deprecated")

	x1	x2	Mean	Var
x1	1	.	20.41	29.20
x2	.87	1	20.31	30.27

If we know the true scores, the reliability estimate for X3 is

# Reliability for x3
var(t2) / var(x2)

[1] 0.8473518

Parallel Tests

Means and Variances of observed scores are equal, theoretically
Correlation = reliability

Tau-Equivalent Tests

Here we simulate scores for two tests that are tau-equivalent:

set.seed(2237) # setting the seed ensure reproducibility
num_person <- 1000 # number of respondents
# True scores for Test 1
t1 <- rnorm(num_person, mean = 20, sd = 5)
# Error scores for Test 1
e1 <- rnorm(num_person, mean = 0, sd = 2)
# Observed scores for Test 1
x1 <- t1 + e1
# True scores for Test 3
t3 <- t1 # parallel tests have equal true scores
# Error scores for Test 3
e3 <- rnorm(num_person, mean = 0, sd = 4)
# Observed scores for Test 3
x3 <- t3 + e3

If we know the true scores, the reliability estimate for X3 is

# Reliability for x3
var(t3) / var(x3)

[1] 0.5957004

# Merge into a data frame
test_df <- data.frame(x1, x3)
# Get means and variances
mv <- datasummary(x1 + x3 ~ Mean + Var, data = test_df,
                  output = "data.frame")
# Correlation, adding means and variances
datasummary_correlation(test_df, add_columns = mv[-1])

Warning in attr(x, "align"): 'xfun::attr()' is deprecated.
Use 'xfun::attr2()' instead.
See help("Deprecated")

	x1	x3	Mean	Var
x1	1	.	20.41	29.20
x3	.74	1	20.27	43.06

Tau-Equivalent Tests

Means of observed scores are equal, theoretically, but variances are different
Correlation $\neq$ reliability

Essentially Tau-Equivalent Tests

# True scores for Test 4
t4 <- 5 + t1 # essentially tau-equivalent tests
# Error scores for Test 4 (larger error SDs)
e4 <- rnorm(num_person, mean = 0, sd = 4)
# Observed scores for Test 4
x4 <- t4 + e4

# Merge into a data frame
test_df2 <- data.frame(x1, x4)
# Get means and variances
mv <- datasummary(x1 + x4 ~ Mean + Var, data = test_df2,
                  output = "data.frame")
# Correlation, adding means and variances
datasummary_correlation(test_df2, add_columns = mv[-1])

Warning in attr(x, "align"): 'xfun::attr()' is deprecated.
Use 'xfun::attr2()' instead.
See help("Deprecated")

	x1	x4	Mean	Var
x1	1	.	20.41	29.20
x4	.72	1	25.41	41.50

If we know the true scores, the reliability estimate for X4 is

# Reliability for x4
var(t4) / var(x4)

[1] 0.6180122

Essentially Tau-Equivalent Tests

Means and Variances of observed scores are different
Correlation $\neq$ reliability

Congeneric Tests

# True scores for Test 5
t5 <- 2 + 0.8 * t1
# Error scores for Test 5 (larger error SDs)
e5 <- rnorm(num_person, mean = 0, sd = 3)
# Observed scores for Test 5
x5 <- t5 + e5

# Merge into a data frame
test_df3 <- data.frame(x1, x5)
# Get means and variances
mv <- datasummary(x1 + x5 ~ Mean + Var, data = test_df3,
                  output = "data.frame")
# Correlation, adding means and variances
datasummary_correlation(test_df3, add_columns = mv[-1])

Warning in attr(x, "align"): 'xfun::attr()' is deprecated.
Use 'xfun::attr2()' instead.
See help("Deprecated")

	x1	x5	Mean	Var
x1	1	.	20.41	29.20
x5	.73	1	18.27	24.23

If we know the true scores, the reliability estimate for X5 is

# Reliability for x5
var(t5) / var(x5)

[1] 0.6773983

Congeneric Tests

Means and Variances of observed scores are different
Correlation $\neq$ reliability
Need more than two tests to be disinguishable from essentially tau-equivalent tests

--- title: "Parallel, Tau-Equivalent, and Congeneric Tests" format: html --- ```{r} #| message: false # install.packages("here") library(tidyverse) library(modelsummary) # for summarizing data ``` ## Parallel Tests Here we simulate scores for two tests that are parallel: ```{r} set.seed(2237) # setting the seed ensure reproducibility num_person <- 1000 # number of respondents # True scores for Test 1 t1 <- rnorm(num_person, mean = 20, sd = 5) # Error scores for Test 1 e1 <- rnorm(num_person, mean = 0, sd = 2) # Observed scores for Test 1 x1 <- t1 + e1 # True scores for Test 2 t2 <- t1 # parallel tests have equal true scores # Error scores for Test 2 e2 <- rnorm(num_person, mean = 0, sd = 2) # Observed scores for Test 2 x2 <- t2 + e2 ``` ```{r} # Merge into a data frame test_df <- data.frame(x1, x2) # Get means and variances mv <- datasummary(x1 + x2 ~ Mean + Var, data = test_df, output = "data.frame") # Correlation, adding means and variances datasummary_correlation(test_df, add_columns = mv[-1]) ``` If we know the true scores, the reliability estimate for X3 is ```{r} # Reliability for x3 var(t2) / var(x2) ``` ::: {.callout-important} ## Parallel Tests - Means and Variances of observed scores are equal, theoretically - Correlation = reliability ::: ## Tau-Equivalent Tests Here we simulate scores for two tests that are tau-equivalent: ```{r} set.seed(2237) # setting the seed ensure reproducibility num_person <- 1000 # number of respondents # True scores for Test 1 t1 <- rnorm(num_person, mean = 20, sd = 5) # Error scores for Test 1 e1 <- rnorm(num_person, mean = 0, sd = 2) # Observed scores for Test 1 x1 <- t1 + e1 # True scores for Test 3 t3 <- t1 # parallel tests have equal true scores # Error scores for Test 3 e3 <- rnorm(num_person, mean = 0, sd = 4) # Observed scores for Test 3 x3 <- t3 + e3 ``` If we know the true scores, the reliability estimate for X3 is ```{r} # Reliability for x3 var(t3) / var(x3) ``` ```{r} # Merge into a data frame test_df <- data.frame(x1, x3) # Get means and variances mv <- datasummary(x1 + x3 ~ Mean + Var, data = test_df, output = "data.frame") # Correlation, adding means and variances datasummary_correlation(test_df, add_columns = mv[-1]) ``` ::: {.callout-important} ## Tau-Equivalent Tests - Means of observed scores are equal, theoretically, but variances are different - Correlation $\neq$ reliability ::: ## Essentially Tau-Equivalent Tests ```{r} # True scores for Test 4 t4 <- 5 + t1 # essentially tau-equivalent tests # Error scores for Test 4 (larger error SDs) e4 <- rnorm(num_person, mean = 0, sd = 4) # Observed scores for Test 4 x4 <- t4 + e4 ``` ```{r} # Merge into a data frame test_df2 <- data.frame(x1, x4) # Get means and variances mv <- datasummary(x1 + x4 ~ Mean + Var, data = test_df2, output = "data.frame") # Correlation, adding means and variances datasummary_correlation(test_df2, add_columns = mv[-1]) ``` If we know the true scores, the reliability estimate for X4 is ```{r} # Reliability for x4 var(t4) / var(x4) ``` ::: {.callout-important} ## Essentially Tau-Equivalent Tests - Means and Variances of observed scores are different - Correlation $\neq$ reliability ::: ## Congeneric Tests ```{r} # True scores for Test 5 t5 <- 2 + 0.8 * t1 # Error scores for Test 5 (larger error SDs) e5 <- rnorm(num_person, mean = 0, sd = 3) # Observed scores for Test 5 x5 <- t5 + e5 ``` ```{r} # Merge into a data frame test_df3 <- data.frame(x1, x5) # Get means and variances mv <- datasummary(x1 + x5 ~ Mean + Var, data = test_df3, output = "data.frame") # Correlation, adding means and variances datasummary_correlation(test_df3, add_columns = mv[-1]) ``` If we know the true scores, the reliability estimate for X5 is ```{r} # Reliability for x5 var(t5) / var(x5) ``` ::: {.callout-important} ## Congeneric Tests - Means and Variances of observed scores are different - Correlation $\neq$ reliability - Need more than two tests to be disinguishable from essentially tau-equivalent tests :::