Mixed vs Random Effects ICC

I am aware this is an old question, but I guess it would be helpful to give a more conclusive answer because I had the same question last week.

The paper Shrout and Fleiss (1979) defines clearly each intraclass correlation.

Let x_{ij} denote the i-th rating on the j-th target for i = 1, \ldots, k and j = 1, \ldots, n. For one-way random effects, they consider the following model:

(1) x_{ij} = \mu + b_{j} + w_{ij}

where \mu is the overall mean, b_j \sim N(0, \sigma^2_B) is the difference between the overall mean and target j, and w_{ij} \sim N(0, \sigma^2_W) are the residual components containing the inseparable effects of the rater, the interaction effect rater \times target and the random error \epsilon_{ij}.

For the two-way random effects and mixed effects, they consider the following model:

(2, 3) x_{ij} = \mu + a_{i} + b_{j} + ab_{ij} + \epsilon_{ij}

where \mu and b_j are defined as in the previous model, a_i is the difference between the overall mean and rater i, and \epsilon_{ij} \sim N(0, \sigma^2) are the random errors. The difference between the models are:

(2) In the two-way random effects model, there is the additional assumption that a_i \sim N(0, \sigma^2_A).
(3) In the two-way mixed effects model, a_i is a fixed effect with the constraint \sum_{i = 1}^ka_i = 0. The parameter corresponding to \sigma^2_A is \theta = \sum_{i = 1}^k a_i^2/(k - 1).

Furthermore, in the absence of repeated ratings by each rater on each target, the components ab_{ij} and \epsilon_{ij} cannot be estimated separately. Nonetheless, the interaction needs to be considered when calculating the ICC formulas,

(2) In the two-way random model, (ab)_{ij} \sim N(0, \sigma^2_I).
(3) In the two-way mixed model, we need the constraint \sum_{i = 1}^k(ab)_{ij} = 0 to have an identifiable model. A consequence of this constraint is that Cov((ab)_{ij}, (ab)_{ij'}) = -\frac{\sigma^2_I}{k - 1}.

Then, the following table with Mean Squares (MS) and Expected Mean Squares (EMS) is presented:

Source of Variation MS EMS: One way EMS: Two-way random EMS Two way mixed
Between targets MS_R k\sigma^2_b + \sigma^2_W k\sigma^2_b + \sigma^2_I + \sigma^2 k\sigma^2_b + \sigma^2
Within targets MS_W \sigma^2_W \sigma^2_a + \sigma^2_I + \sigma^2 \theta^2 + f\sigma^2_I + \sigma^2
Between raters MS_C - n\sigma^2_a + \sigma^2_I + \sigma^2 n\theta^2+ f\sigma^2_I + \sigma^2
Residual MS_E - \sigma^2_I + \sigma^2 f\sigma^2_I + \sigma^2
where f = k/(k - 1).

Intraclass definitions are given in Koo and Li (2016).

References:
Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin. 1979 Mar;86(2):420

Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine. 2016 Jun 1;15(2):155-63.

In R,

library(lme4)

n.subject <- 100
n.rater <- 100

random.subject <- rep(rnorm(n = n.subject, mean = 0, sd = 1), each = n.rater)
random.rater <- rep(rnorm(n = n.rater, mean = 0, sd = 1), n.subject)
random.error <- rnorm(n = n.subject*n.rater, mean = 0, sd = 1)

id <- as.factor(rep(1:n.subject, each = n.rater))
rater <- as.factor(rep(1:n.rater, n.subject))


### one-way random effect
design.matrix <- model.matrix(~ id)
response <-random.subject + random.error
dt <- data.frame(id, rater, response)

## fiting a model
fit <- lmer(response ~ 1 + (1|id), data = dt)
sm <- summary(fit)

# ICC (1, k)
msr <- as.numeric(n.rater*sm$varcor$id + sm$sigma)
msw <- as.numeric(sm$sigma)
icc <- (msr - msw)/(msr + (n.rater + 1)*msw)

### two-way random effects

## data
response <- design.matrix%*%unique(random.rater) + random.subject + random.error
dt <- data.frame(id, rater, response)

## fiting a model
fit <- lmer(response ~ 1 + (1|id) + (1|rater), data = dt)
sm <- summary(fit)

# ICC (2, k)
msr <- as.numeric(n.rater*sm$varcor$id + sm$sigma)
mse <- as.numeric(sm$sigma)
msc <- as.numeric(n.subject*sm$varcor$rater + sm$sigma)
icc <- (msr - mse)/(msr + (msc - msr)/n.subject)

### two-way mixed-effects

## data
design.matrix <- model.matrix(~ rater)
response <- design.matrix%*%unique(random.rater) + random.subject + random.error
dt <- data.frame(id, rater, response)

## fiting a model
fit <- lmer(response ~ rater + (1|id), data = dt)
sm <- summary(fit)

# ICC(3, k)
msr <-  as.numeric(n.rater*sm$varcor$id + sm$sigma)
mse <- as.numeric(sm$sigma)
icc <- (msr - mse)/msr

3 Likes