Kappa for nominal data was first described by Fleiss in 1969. It is a measurement of concordance or agreement between two or more judges, in the way they classify or categorise subjects into different groups or categories. The following terms are often used

• Nominal data These are data sets where the numbers are names and not scales, and they have no mathematical relationship with each other. For example, in the classification of religion, 1 may be Buddhism, 2 may be Ancestor Worship, and 3 may be Hinduism. The difference between 1 and 2, 2 and 3 or 1 and 3 have no particular order, and the only thing that can be said is that they represent things that are different.
• Judges are persons or instruments that classify a subject into a particular group or category. For example, a questionnaire that solicit beliefs and activities may be used to classify someone into a particular religion. Similarly, a person, after conversing with a subject, may also perform a classification.
• Subjects are the subjects that are classified. They are patients, school children, members of the public, monkeys, rats, and so on.
• Groups or categories are the classification that the judges place the subjects in.
• Concordance is a measure of how much classifications produced by different judges agree. Commonly, concordance is expressed as a number between 0 and 1, where 0 represents no agreements at all, and 1 represent complete agreement.

Data Entry

Data entry and interpretation are best described by using the default example provided in the Javascript program panel. In this example, A school provides 5 councillors which assesses then advises students on future careers, and we wish to evaluate how much the assessments of the students from these 5 councillor agree with each other.

We used a class of 10 students in their final school year, and each of the 5 councillors interviews every student, and classify them into the following categories.

• 1 = suitable for the caring profession
• 2 = suitable for engineering
• 3 = suitable for business

Raw data entry for Kappa Data entry is as shown in the program. It is a table where there are 10 rows, each represents data for a student, and 5 columns, each represents a councillor. The cells contains 1, 2, or 3, according to how each councillor classify that student. For example

• The first subject was categorised 1,2,2,2,2 by the 5 judges, suitable for the caring profession by the first judge, and engineering by the other 4.
• The second subject categorised 1,1,3,3,3, caring profession by the first two judges, and business by the other 3, and so on.

The program collates the raw data, and produces a table of counts. The table of counts has 10 rows, representing the 10 students, and 3 columns representing the 3 classifications. The cell contains the number of time each students is classified in that category. In this example, the first column is for caring profession, the second engineering, and third business, so that

• The first student (1,4,0), is classified for caring profession once, and engineering 4 times
• The second (2,0 3), caring profession twice, and business 3 times,
• and so on

The table of counts is then analysed to obtain Kappa, and whether the Kappa value significantly deviates from 0 (no agreement).

If the raw data is already collated into a table of counts, this can also be used as entry data, using the buttons on the right side of the program panel.

Results of Analysis

Kappa in this example is 0.41, with a Standard Error of 0.08, and the 95% confidence interval od 0.27 to 0.57. This is the level of agreements between the 5 counsellors. Conventionally, a Kappa of <0.2 is considered poor agreement, 0.21-0.4 fair, 0.41-0.6 moderate, 0.61-0.8 strong, and more than 0.8 near complete agreement.

Given Kappa is an estimate from a sample, its Standard Error (se) and 95% confidence interval can be estimated. In this example, the 95% confidence interval did nor traverse the null (0) value, allowing a conclusion to be made that there are significant agreements between the councillors.

References

Fleiss J L (1971) Measuring nominal scale agreement amongst many raters. Psychological Bulletin 76:378-382

Siegel S and Castellan Jr. N.J. Nonparametric Statistics for the Behavioral Sciences (1988) International Edition. McGraw-Hill Book Company New York. ISBN 0-07-057357-3 p. 284-291

Data Entry using Table of Raw Scores The data is a matrix of numbers with multiple columns   - Each row a subject   - Each column a judge, rater, or scale   - Each cell contains the classification number   - Data is converted to a table of counts for analysis.	Data Entry using Table of Counts The data is a matrix of numbers with multiple columns   - Each row a subject   - Each column a classification   - Each cell contains the number of times the subject (row)     is so classified (col)

# Kappa.R # Kappa for nominal data

# Kappa Algorithm
CalKappa <- function(dfCount)
{
  print("Matrix of counts")
  print(dfCount)
  n = nrow(dfCount)   # n rows
  g = ncol(dfCount)   # catagories cols
  r = sum(dfCount[1,])
  print(c(n, g, r))        

  Nk = n*r
  Pe = 0;
  EP3 = 0;
  Cj = rep(0,g)
  Pj = rep(0,g)
  for(j in 1 : g)
  {
    for(i in 1:n)Cj[j] = Cj[j] + dfCount[i,j] * 1
    Pj[j] = Cj[j] / Nk
    Pe = Pe + Pj[j] * Pj[j]
    EP3 = EP3 + Pj[j] * Pj[j] * Pj[j]
  }
  Nk = r * (r - 1)
  Pa = 0;
  Si = rep(0, n)
  for(i in 1 : n)
  {
    for(j in 1 : g)
    {
      k = dfCount[i,j]
      Si[i] = Si[i] + k * (k - 1)
    }
    Si[i] = Si[i] / Nk
    Pa = Pa + Si[i]
  }
  Pa = Pa / n
  Kappa = (Pa-Pe)/(1.0-Pe)
  f1 = 2.0 / (1.0 * n * r * (r-1))
  f2 = Pe - (2*r-3)*Pe*Pe + 2*(r-2)*EP3
  f3 = (1-Pe) * (1-Pe)
  se = sqrt(f1 * f2 / f3);
  z = Kappa / se;
  p = 1 - pnorm(z)
  print(paste("Kappa=", Kappa," SE=", se ))
  print(paste("z=", z, " p=", p))
  print(paste0("95% CI = ", (Kappa - 1.96 * se), " to ", (Kappa + 1.96 * se)))
}

#Program 1: Converting raw data to table of counts and calculate Kappa
datRaw = ("
1  2  2  2  2
1  1  3  3  3
3  3  3  3  3
1  1  1  1  3
1  1  1  3  3
1  2  2  2  2
1  1  1  1  1
2  2  2  2  3
1  3  3  3  3
1  1  1  3  3
          ")
dfRaw <- read.table(textConnection(datRaw),header=FALSE)  # conversion to data frame    
#dfRaw                                           # check means and SDs
minv = min(dfRaw)
maxv = max(dfRaw)
nc = maxv - minv + 1
c(minv, maxv, nc)

dfCount <- matrix(0, ncol = nc, nrow = nrow(dfRaw))
for(i in 1:nrow(dfRaw)) for(j in 1:ncol(dfRaw))
{
  v = dfRaw[i,j]
  dfCount[i,v] = dfCount[i,v] + 1
}
CalKappa(dfCount)

[1] "Matrix of counts"
   V1 V2 V3
1   1  4  0
2   2  0  3
3   0  0  5
4   4  0  1
5   3  0  2
6   1  4  0
7   5  0  0
8   0  4  1
9   1  0  4
10  3  0  2
[1] 10  3  5
[1] "Kappa= 0.417892156862745  SE= 0.0766306770750035"
[1] "z= 5.45332721585803  p= 2.47179898771321e-08"
[1] "95% CI = 0.267696029795738 to 0.568088283929752"
> 

# Program 2: Input table of counts and calculate Kappa
datCount = ("
1  4  0
2  0  3
0  0  5
4  0  1
3  0  2
1  4  0
5  0  0
0  4  1
1  0  4
3  0  2
            ")
dfCount <- read.table(textConnection(datCount),header=FALSE)  # conversion to data frame 
CalKappa(dfCount)

[1] "Matrix of counts"
   V1 V2 V3
1   1  4  0
2   2  0  3
3   0  0  5
4   4  0  1
5   3  0  2
6   1  4  0
7   5  0  0
8   0  4  1
9   1  0  4
10  3  0  2
[1] 10  3  5
[1] "Kappa= 0.417892156862745  SE= 0.0766306770750035"
[1] "z= 5.45332721585803  p= 2.47179898771321e-08"
[1] "95% CI = 0.267696029795738 to 0.568088283929752"
>