Inter-Rating Reliability / Intra-Class Correlation in SPSS

Moderators: statman, Analyst Techy, andris, Fierce, GerineL, Smash

GarethLock
Posts: 1
Joined: Thu Oct 24, 2013 7:50 pm

Inter-Rating Reliability / Intra-Class Correlation in SPSS

Postby GarethLock » Mon Nov 04, 2013 12:50 pm

Hi,

To start with I have almost zero stats experience so please bear with me :) I have posted this request somewhere else on 24 Oct but haven't had any responses yet but hoping I get one here...

I have recently conducted a survey with 14 raters assessing the influence of human factors from a modified HFACS (Human Factors Analysis and Classification System) model with 96 factors. The assessment was conducted against 10 scenarios with scores of 0-3 allocated to each of the factors (0 - no factor/no obvious factor, 1 - minor factor, 2 - major factor and 3 - key factor)

The results from each scenario were collected into a single Excel sheet (14 columns, 96 rows, scores 0-3) and the data imported into SPSS per scenario.

Following some advice I found here http://neoacademic.com/2011/11/16/compu ... y-in-spss/ which recommended I used Two Way Mixed ICC in the Scale>Analysis of SPSS because the raters are part of a wider population and are rating all 10 scenarios against 96 factors. I also ran Inter-item correlation against the 14 raters.

The output from SPSS is here

Image

The figures from which the results are derived are here https://dl.dropboxusercontent.com/u/221 ... mple1.xlsx

What I would like to know is:

1. From what I have read, the ICC scores (between .877 and .927) I am getting mean that I have good ICC across the 96 x 14 matrix. Is this correct?

2. When looking at the Inter-item Correlation matrix, the scores aren't so good and would be unlikely to pass scrutiny. I think I understand that this is because of the 1:1 variance but overall this is much better reliability. Is this correct?

Any other information relevant to this will be gratefully received. Thanks in advance.

Regards

Gareth

Who is online

Users browsing this forum: No registered users and 1 guest

cron