When to refer instead of classify: selective prediction with typed abstention in optometric screening
DOI:
https://doi.org/10.5281/zenodo.22755549Palavras-chave:
selective prediction, reject option, conformal prediction, missing data, vision screening, risk-coverageResumo
Recording why a screening classifier abstains, because an examination is incomplete or because models built on different parts of the examination disagree, changed where residual risk fell but not measurably the risk-coverage curve in the data examined. In eye-care screening, where a record is decided automatically or referred, we compare typed abstention with a calibrated logistic threshold, Chow's rule, selection with guaranteed risk (SGR) and label-conditional conformal prediction on the same information. For a calibrated predictor, typing can lower the risk-coverage frontier only where the predictor is miscalibrated conditionally on the type, and per-type risk guarantees survive changes in the mix of types while pooled guarantees need not (proved). In public-use NHANES 2005-2006 records (2410 adults aged 40 or older, reference standard from graded retinal photographs, self-reported eye diagnoses excluded), a typed error model changed the area under the risk-coverage curve by -0.005 (95% CI -0.010 to 0.001) relative to Chow's rule, and an untyped error model did slightly better. At a nominal risk of 0.20, SGR achieved 0.145 overall but 0.232 (participant-bootstrap 95% CI 0.184 to 0.280) among complete records with discordant source models, an interval that includes the target, and 0.290 (0.242 to 0.340) in the 2007-2008 cycle; a per-type rule decided only 1.3% of discordant records, so their risk was estimable in 1 of 10 repeats, and overall coverage fell from 0.68 to 0.43. A declared simulation shows typing lowering the curve only when unreliable instrument results go unrecognised by the base model. Typing helped to audit where risk lands, not to rank records.
Referências
[1] Y. González Vargas and D. Padrón Carrasco, "Neutrosophic-Logistic Predictive Model for the Identification of Optometric Visual Alterations in Contexts of High Clinical Uncertainty," Neutrosophic Computing and Machine Learning, vol. 42, pp. 205-217, 2026, doi: 10.5281/zenodo.18331786.
[2] C. Chow, "On optimum recognition error and reject tradeoff," IEEE Transactions on Information Theory, vol. 16, no. 1, pp. 41-46, 1970, doi: 10.1109/TIT.1970.1054406.
[3] R. El-Yaniv and Y. Wiener, "On the Foundations of Noise-free Selective Classification," Journal of Machine Learning Research, vol. 11, pp. 1605-1641, 2010.
[4] Y. Geifman and R. El-Yaniv, "Selective Classification for Deep Neural Networks," in Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017, arXiv version, doi: 10.48550/arXiv.1705.08500.
[5] V. Vovk, A. Gammerman, and G. Shafer, Algorithmic Learning in a Random World. Springer, New York, 2005, doi: 10.1007/b106715.
[6] A. N. Angelopoulos and S. Bates, "Conformal Prediction: A Gentle Introduction," Foundations and Trends in Machine Learning, vol. 16, no. 4, pp. 494-591, 2023, doi: 10.1561/2200000101.
[7] V. Franc, D. Prusa, and V. Voracek, "Optimal strategies for reject option classifiers," Journal of Machine Learning Research, vol. 24, no. 11, pp. 1-49, 2023, arXiv version, doi: 10.48550/arXiv.2101.12523.
[8] Ú. Hébert-Johnson, M. P. Kim, O. Reingold, and G. N. Rothblum, "Multicalibration: Calibration for the (computationally-identifiable) masses," in Proc. 35th International Conference on Machine Learning, PMLR, vol. 80, pp. 1939-1948, 2018, arXiv version, doi: 10.48550/arXiv.1711.08513.
[9] V. Vovk, "Conditional validity of inductive conformal predictors," Machine Learning, vol. 92, no. 2-3, pp. 349-376, 2013, doi: 10.1007/s10994-013-5355-6.
[10] A. Podkopaev and A. Ramdas, "Distribution-free uncertainty quantification for classification under label shift," arXiv preprint, 2021, doi: 10.48550/arXiv.2103.03323.
[11] E. Jones, S. Sagawa, P. W. Koh, A. Kumar, and P. Liang, "Selective Classification Can Magnify Disparities Across Groups," in International Conference on Learning Representations (ICLR 2021), 2021, arXiv version, doi: 10.48550/arXiv.2010.14134.
[12] T. P. Morris, I. R. White, and M. J. Crowther, "Using simulation studies to evaluate statistical methods," Statistics in Medicine, vol. 38, no. 11, pp. 2074-2102, 2019, doi: 10.1002/sim.8086.
[13] R. Herbei and M. H. Wegkamp, "Classification with reject option," Canadian Journal of Statistics, vol. 34, no. 4, pp. 709-721, 2006, doi: 10.1002/cjs.5550340410.
[14] P. L. Bartlett and M. Wegkamp, "Classification with a Reject Option using a Hinge Loss," Journal of Machine Learning Research, vol. 9, pp. 1823-1840, 2008.
[15] C. Cortes, G. DeSalvo, and M. Mohri, "Learning with Rejection," in Algorithmic Learning Theory (ALT 2016), Lecture Notes in Computer Science, vol. 9925, pp. 67-82, 2016, doi: 10.1007/978-3-319-46379-7_5.
[16] J. Traub, T. J. Bungert, C. T. Lüth, M. Baumgartner, K. H. Maier-Hein, L. Maier-Hein, et al., "Overcoming Common Flaws in the Evaluation of Selective Classification Systems," arXiv preprint, 2024, doi: 10.48550/arXiv.2407.01032.
[17] K. Hendrickx, L. Perini, D. Van der Plas, W. Meert, and J. Davis, "Machine learning with a reject option: a survey," Machine Learning, vol. 113, no. 5, pp. 3073-3110, 2024, doi: 10.1007/s10994-024-06534-x.
[18] H. Mozannar and D. Sontag, "Consistent Estimators for Learning to Defer to an Expert," arXiv preprint, 2020, doi: 10.48550/arXiv.2006.01862.
[19] J. Lei, M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman, "Distribution-Free Predictive Inference for Regression," Journal of the American Statistical Association, vol. 113, no. 523, pp. 1094-1111, 2018, doi: 10.1080/01621459.2017.1307116.
[20] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, "Conformal Risk Control," in International Conference on Learning Representations (ICLR 2024), 2024, arXiv version, doi: 10.48550/arXiv.2208.02814.
[21] A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei, "Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control," arXiv preprint, 2021, doi: 10.48550/arXiv.2110.01052.
[22] R. F. Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani, "Conformal prediction beyond exchangeability," The Annals of Statistics, vol. 51, no. 2, 2023, doi: 10.1214/23-AOS2276.
[23] B. Dubuisson and M. Masson, "A statistical decision rule with incomplete knowledge about classes," Pattern Recognition, vol. 26, no. 1, pp. 155-165, 1993, doi: 10.1016/0031-3203(93)90097-G.
[24] E. Hüllermeier and W. Waegeman, "Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods," Machine Learning, vol. 110, no. 3, pp. 457-506, 2021, doi: 10.1007/s10994-021-05946-3.
[25] N. D. Belnap, "A Useful Four-Valued Logic," in Modern Uses of Multiple-Valued Logic, D. Reidel, Dordrecht, pp. 5-37, 1977, doi: 10.1007/978-94-010-1161-7_2.
[26] D. B. Rubin, "Inference and missing data," Biometrika, vol. 63, no. 3, pp. 581-592, 1976, doi: 10.1093/biomet/63.3.581.
[27] M. Sperrin, G. P. Martin, R. Sisk, and N. Peek, "Missing data should be handled differently for prediction than for description or causal explanation," Journal of Clinical Epidemiology, vol. 125, pp. 183-187, 2020, doi: 10.1016/j.jclinepi.2020.03.028.
[28] R. H. H. Groenwold, "Informative missingness in electronic health record systems: the curse of knowing," Diagnostic and Prognostic Research, vol. 4, no. 1, Art. no. 8, 2020, doi: 10.1186/s41512-020-00077-0.
[29] M. Le Morvan, J. Josse, E. Scornet, and G. Varoquaux, "What's a good imputation to predict with missing values?," arXiv preprint, 2021, doi: 10.48550/arXiv.2106.00311.
[30] V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, et al., "Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs," JAMA, vol. 316, no. 22, p. 2402, 2016, doi: 10.1001/jama.2016.17216.
[31] M. D. Abràmoff, P. T. Lavin, M. Birch, N. Shah, and J. C. Folk, "Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices," npj Digital Medicine, vol. 1, no. 1, Art. no. 39, 2018, doi: 10.1038/s41746-018-0040-6.
[32] C. Leibig, V. Allken, M. S. Ayhan, P. Berens, and S. Wahl, "Leveraging uncertainty information from deep neural networks for disease detection," Scientific Reports, vol. 7, no. 1, Art. no. 17816, 2017, doi: 10.1038/s41598-017-17876-z.
[33] K. Dvijotham, J. Winkens, M. Barsbey, S. Ghaisas, R. Stanforth, N. Pawlowski, et al., "Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians," Nature Medicine, vol. 29, no. 7, pp. 1814-1820, 2023, doi: 10.1038/s41591-023-02437-x.
[34] B. Kompa, J. Snoek, and A. L. Beam, "Second opinion needed: communicating uncertainty in medical machine learning," npj Digital Medicine, vol. 4, no. 1, Art. no. 4, 2021, doi: 10.1038/s41746-020-00367-3.
[35] A. Niculescu-Mizil and R. Caruana, "Predicting good probabilities with supervised learning," Proc. 22nd International Conference on Machine Learning (ICML), pp. 625-632, 2005, doi: 10.1145/1102351.1102430.
[36] B. Van Calster, D. J. McLernon, M. van Smeden, L. Wynants, and E. W. Steyerberg, "Calibration: the Achilles heel of predictive analytics," BMC Medicine, vol. 17, no. 1, Art. no. 230, 2019, doi: 10.1186/s12916-019-1466-7.
[37] A. J. Vickers and E. B. Elkin, "Decision Curve Analysis: A Novel Method for Evaluating Prediction Models," Medical Decision Making, vol. 26, no. 6, pp. 565-574, 2006, doi: 10.1177/0272989X06295361.
[38] G. S. Collins, K. G. M. Moons, P. Dhiman, R. D. Riley, A. L. Beam, B. Van Calster, et al., "TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods," BMJ, vol. 385, Art. no. e078378, 2024, doi: 10.1136/bmj-2023-078378.
[39] R. Howard, "Information Value Theory," IEEE Transactions on Systems Science and Cybernetics, vol. 2, no. 1, pp. 22-26, 1966, doi: 10.1109/TSSC.1966.300074.
[40] R. R. A. Bourne, S. R. Flaxman, T. Braithwaite, M. V. Cicinelli, A. Das, J. B. Jonas, et al., "Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis," The Lancet Global Health, vol. 5, no. 9, pp. e888-e897, 2017, doi: 10.1016/S2214-109X(17)30293-0.
[41] R. Chou, T. Dana, C. Bougatsos, S. Grusing, and I. Blazina, "Screening for Impaired Visual Acuity in Older Adults," JAMA, vol. 315, no. 9, p. 915, 2016, doi: 10.1001/jama.2016.0783.
[42] X. Zhang, J. B. Saaddine, C.-F. Chou, M. F. Cotch, Y. J. Cheng, L. S. Geiss, et al., "Prevalence of Diabetic Retinopathy in the United States, 2005-2008," JAMA, vol. 304, no. 6, p. 649, 2010, doi: 10.1001/jama.2010.1111.
[43] R. Klein, C.-F. Chou, B. E. K. Klein, X. Zhang, S. M. Meuer, and J. B. Saaddine, "Prevalence of Age-Related Macular Degeneration in the US Population," Archives of Ophthalmology, vol. 129, no. 1, p. 75, 2011, doi: 10.1001/archophthalmol.2010.318.
[44] P. Gupta, D. Zhao, E. Guallar, F. Ko, M. V. Boland, and D. S. Friedman, "Prevalence of Glaucoma in the United States: The 2005–2008 National Health and Nutrition Examination Survey," Investigative Ophthalmology & Visual Science, vol. 57, no. 6, p. 2905, 2016, doi: 10.1167/iovs.15-18469.
[45] K. E. Cello, J. M. Nelson-Quigg, and C. A. Johnson, "Frequency doubling technology perimetry for detection of glaucomatous visual field loss," American Journal of Ophthalmology, vol. 129, no. 3, pp. 314-322, 2000, doi: 10.1016/S0002-9394(99)00414-6.
[46] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, et al., "Scikit-learn: Machine Learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825-2830, 2011.
[47] S. Holm, "A Simple Sequentially Rejective Multiple Test Procedure," Scandinavian Journal of Statistics, vol. 6, pp. 65-70, 1979.
[48] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. Springer US, Boston, MA, 1993, doi: 10.1007/978-1-4899-4541-9.
[49] C. J. Clopper and E. S. Pearson, "The use of confidence or fiducial limits illustrated in the case of the binomial," Biometrika, vol. 26, no. 4, pp. 404-413, 1934, doi: 10.1093/biomet/26.4.404.
Downloads
Publicado
Edição
Secção
Licença
Este trabalho encontra-se publicado com a Licença Internacional Creative Commons Atribuição 4.0.
