Table 3 Inter–rater reliability
From: Chest radiograph reading and recording system: evaluation in frontline clinicians in Zambia
Agreement Index | Parenchymal abnormalities | Large opacifications | Small opacifications | Cavitation | Pleural abnormalities | Central abnormalities | Any abnormality | Abnormalities consistent with TB |
|---|---|---|---|---|---|---|---|---|
CRRS–certified medical officers | ||||||||
HIV Positive patients (N = 231)a | ||||||||
Both readers agree abnormality present | 38 | 8 | 24 | 2 | 32 | 17 | 78 | 69 |
Only Reader 1 says abnormality present | 56 | 38 | 36 | 30 | 13 | 21 | 41 | 28 |
Only Reader 2 says abnormality present | 7 | 5 | 10 | 0 | 16 | 17 | 22 | 29 |
Both readers agree abnormality not present | 130 | 180 | 161 | 199 | 170 | 176 | 90 | 105 |
Percent Agreement (95 % CI) | 0.73 | 0.81 | 0.80 | 0.87 | 0.87 | 0.84 | 0.73 | 0.75 |
(0.67–0.78) | (0.76–0.86) | (0.75–0.85) | (0.83–0.91) | (0.83–0.92) | (0.79–0.88) | (0.67–0.78) | (0.70–0.81) | |
Kappa (95 % CI) | 0.38 | 0.20 | 0.40 | 0.10 | 0.61 | 0.38 | 0.46 | 0.49 |
(0.27–0.50) | (0.06–0.35) | (0.26–0.53) | (–0.03–0.24) | (0.48–0.74) | (0.22–0.54) | (0.34–0.57) | (0.38–0.61) | |
Strength of agreement (based on kappa) | Fair | Poor | Fair | Poor | Good | Fair | Moderate | Moderate |
HIV Negative patients (N = 335)a | ||||||||
Both readers agree abnormality present | 58 | 10 | 41 | 6 | 60 | 18 | 136 | 122 |
Only Reader 1 says abnormality present | 100 | 74 | 64 | 39 | 24 | 32 | 55 | 31 |
Only Reader 2 says abnormality present | 5 | 1 | 12 | 0 | 17 | 15 | 9 | 19 |
Both readers agree abnormality not present | 172 | 250 | 218 | 290 | 234 | 270 | 135 | 163 |
Percent Agreement (95 % CI) | 0.96 | 0.78 | 0.77 | 0.88 | 0.88 | 0.86 | 0.76 | 0.85 |
(0.64–0.74) | (0.73–0.82) | (0.73–0.82) | (0.85–0.92) | (0.84–0.91) | (0.82–0.90) | (0.72–0.81) | (0.81–0.89) | |
Kappa (95 % CI) | 0.35 | 0.16 | 0.39 | 0.21 | 0.66 | 0.36 | 0.62 | 0.70 |
(0.27–0.43) | (0.07–0.26) | (0.29–0.50) | (0.07–0.35) | (0.57–0.76) | (0.21–0.50) | (0.54–0.70) | (0.62–0.77) | |
Strength of agreement (based on kappa) | Fair | Poor | Fair | Fair | Good | Fair | Good | Good |
CRRS-oriented clinical officers | ||||||||
HIV Positive patients (N = 231)a | ||||||||
Both readers agree abnormality present | 46 | 33 | 5 | 1 | 55 | 14 | 107 | 102 |
Only Reader 3 says abnormality present | 33 | 28 | 22 | 5 | 16 | 41 | 40 | 35 |
Only Reader 4 says abnormality present | 44 | 37 | 11 | 11 | 63 | 21 | 41 | 43 |
Both readers agree abnormality not present | 108 | 133 | 193 | 214 | 97 | 155 | 43 | 51 |
Percent Agreement | 0.67 | 0.72 | 0.86 | 0.93 | 0.66 | 0.73 | 0.65 | 0.66 |
(0.61–0.73) | (0.66–0.78) | (0.81–0.90) | (0.90–0.96) | (0.60–0.72) | (0.67–0.79) | (0.59–0.71) | (0.60–0.72) | |
Kappa (95 % CI) | 0.28 | 0.31 | 0.16 | 0.08 | 0.32 | 0.15 | 0.24 | 0.29 |
(0.16–0.41) | (0.18–0.44) | (–0.02–0.34) | (–0.12–0.28) | (0.21–0.43) | (0.01–0.29) | (0.11–0.37) | (0.17–0.42) | |
Strength of agreement (based on kappa) | Fair | Fair | Poor | Poor | Fair | Poor | Fair | Fair |
HIV Negative patients (N = 335)a | ||||||||
Both readers agree abnormality present | 88 | 58 | 9 | 1 | 107 | 36 | 177 | 165 |
Only Reader 3 says abnormality present | 42 | 38 | 43 | 10 | 18 | 70 | 52 | 51 |
Only Reader 4 says abnormality present | 36 | 48 | 11 | 15 | 86 | 22 | 48 | 48 |
Both readers agree abnormality not present | 169 | 191 | 272 | 309 | 124 | 207 | 58 | 71 |
Percent Agreement (95 % CI) | 0.77 | 0.74 | 0.84 | 0.92 | 0.69 | 0.73 | 0.70 | 0.70 |
(0.72–0.81) | (0.70–0.79) | (0.80–0.88) | (0.90–0.95) | (0.64–0.74) | (0.68–0.77) | (0.65–0.75) | (0.66–0.75) | |
Kappa (95 % CI) | 0.51 | 0.39 | 0.18 | 0.04 | 0.40 | 0.28 | 0.32 | 0.36 |
(0.41–0.60) | (0.28–0.50) | (0.05–0.31) | (–0.10–0.18) | (0.31–0.49) | (0.17–0.38) | (.21–0.42) | (0.25–0.46) | |
Strength of agreement (based on kappa) | Moderate | Fair | Poor | Poor | Fair | Fair | Fair | Fair |