Measuring corroboration of contradiction flags in neutrosophic expert ratings: agreement level, observer propensity and excess co-location

Resumo

Neutrosophic ratings record truth and falsity as independent degrees, so an expert can report evidence that both supports and undermines a claim. Whether such contradiction flags are corroborated across experts can be measured once two quantities are separated: the level of agreement, and the excess co-location of flags beyond what each expert's propensity to flag predicts. We re-analyse a published elicitation in which 22 experts rated 30 hypotheses for truth, indeterminacy and falsity. An earlier reading of a 17-expert subset reported that 5.6 % of flags were corroborated; that figure was a majority share, and the probability that a second expert confirms a flag is 0.376 with all 22 experts. We prove that pairwise agreement equals the base rate plus a dispersion term and that the observer-marginal permutation distribution is centred on the propensity prediction. When a flag requires both components to be substantial (T > 0.3 and F > 0.3), the experts' flags fall on the same hypotheses more often than their propensities predict (E = 0.010, BCa interval 0.003 to 0.019; Holm-adjusted p = 0.023); this positive excess is exploratory and conditional on these 22 experts being fixed: resampling both experts and items gives an interval of −0.014 to 0.048. The sum rule T + F > 1 shows no detectable excess (E = 0.001), so the flag rule is part of the measurement. Classifier outputs on 13 audited data sets give E of 0.020 to 0.021, reported descriptively because classifiers share inputs. A simulation over 88 designs shows that a pooled-rate null can lose substantial power when observers differ, which hid the expert excess in an earlier audit. A reporting protocol follows.

Received: 3 August 2026 | Accepted: 12 September 2026