To start with I have almost zero stats experience so please bear with me
I have posted this request somewhere else on 24 Oct but haven't had any responses yet but hoping I get one here...
I have recently conducted a survey with 14 raters assessing the influence of human factors from a modified HFACS (Human Factors Analysis and Classification System) model with 96 factors. The assessment was conducted against 10 scenarios with scores of 0-3 allocated to each of the factors (0 - no factor/no obvious factor, 1 - minor factor, 2 - major factor and 3 - key factor)
The results from each scenario were collected into a single Excel sheet (14 columns, 96 rows, scores 0-3) and the data imported into SPSS per scenario.
Following some advice I found here http://neoacademic.com/2011/11/16/compu ... y-in-spss/
which recommended I used Two Way Mixed ICC in the Scale>Analysis of SPSS because the raters are part of a wider population and are rating all 10 scenarios against 96 factors. I also ran Inter-item correlation against the 14 raters.
The output from SPSS is here
The figures from which the results are derived are here https://dl.dropboxusercontent.com/u/221 ... mple1.xlsx
What I would like to know is:
1. From what I have read, the ICC scores (between .877 and .927) I am getting mean that I have good ICC across the 96 x 14 matrix. Is this correct?
2. When looking at the Inter-item Correlation matrix, the scores aren't so good and would be unlikely to pass scrutiny. I think I understand that this is because of the 1:1 variance but overall this is much better reliability. Is this correct?
Any other information relevant to this will be gratefully received. Thanks in advance.