When indicators disagree: separating missing from conflicting evidence in cross-country rankings of student mobility
DOI:
https://doi.org/10.5281/zenodo.22755541Palabras clave:
composite indicators, source disagreement, multiple imputation, international student mobility, tertiary education finance, rank uncertaintyResumen
Missing evidence, disagreement between statistical producers and divergence between indicators can be told apart in cross-country education statistics; the distinction exposes rank uncertainty that multiple imputation on one producer's values does not represent, but did not improve a ranking-based decision. We downloaded from OECD, UNESCO Institute for Statistics and World Bank interfaces the inbound mobility rate, government expenditure per tertiary student and government tertiary expenditure as a share of GDP for 42 countries. A rule whose threshold was chosen after exploring 2013-2016 classified 252 country-years in 2017-2022 as concordant (42.9%; 95% CI 30.6-55.2), producers disagreeing (25.8%; 16.3-36.1), indicators diverging (15.1%) or missing (16.3%). Producer disagreement mixes definitional differences with numerical conflict: removing the systematic OECD-UNESCO level difference lowers its share to 15.9%. Mobility rates coincided across producers. Putting UNESCO values first moved disagreeing country-years by 10.4 rank positions (95% CI 5.8-15.9) and concordant ones by 2.9. That rank fell outside the 95% multiple-imputation interval for 66.1% of disagreeing and 35.2% of concordant country-years (difference 31.0 points, 7.3-53.5, country bootstrap within this convenience panel); without the check that removed one impossible OECD value the difference was 16.0 points (-4.4 to 34.1). Abstaining on non-concordant evidence did not reduce next-year reversals of a top-10 flag relative to abstaining near the cut-off (3.3% versus 0.0%; Holm-adjusted p = 0.27).
Referencias
[1] A. Matheu Pérez, R. Cortés, A. Carrier, and C. Sebrango, "Contributions of educational investment to international student mobility: A multivariate and neutrosophic analysis of educational service quality in OECD data," Neutrosophic Sets and Systems, vol. 89, pp. 357-374, 2025, doi: 10.5281/zenodo.16636867.
[2] D. B. Rubin, Multiple Imputation for Nonresponse in Surveys. New York, NY, USA: Wiley, 1987, doi: 10.1002/9780470316696.
[3] C. F. Manski, Partial Identification of Probability Distributions. New York, NY, USA: Springer, 2003, doi: 10.1007/b97478.
[4] A. B. Atkinson and A. Brandolini, "Promise and pitfalls in the use of “secondary” data-sets: Income inequality in OECD countries as a case study," Journal of Economic Literature, vol. 39, no. 3, pp. 771-799, 2001, doi: 10.1257/jel.39.3.771.
[5] S. Johnson, W. Larson, C. Papageorgiou, and A. Subramanian, "Is newer better? Penn World Table revisions and their impact on growth estimates," Journal of Monetary Economics, vol. 60, no. 2, pp. 255-274, 2013, doi: 10.1016/j.jmoneco.2012.10.022.
[6] OECD and European Commission Joint Research Centre, Handbook on Constructing Composite Indicators: Methodology and User Guide. Paris, France: OECD Publishing, 2008, doi: 10.1787/9789264043466-en.
[7] M. Saisana, A. Saltelli, and S. Tarantola, "Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators," Journal of the Royal Statistical Society: Series A (Statistics in Society), vol. 168, no. 2, pp. 307-323, 2005, doi: 10.1111/j.1467-985X.2005.00350.x.
[8] P. Paruolo, M. Saisana, and A. Saltelli, "Ratings and rankings: Voodoo or science?," Journal of the Royal Statistical Society: Series A (Statistics in Society), vol. 176, no. 3, pp. 609-634, 2013, doi: 10.1111/j.1467-985X.2012.01059.x.
[9] M. Mogstad, J. P. Romano, A. M. Shaikh, and D. Wilhelm, "Inference for ranks with applications to mobility across neighbourhoods and academic achievement across countries," Review of Economic Studies, vol. 91, no. 1, pp. 476-518, 2024, doi: 10.1093/restud/rdad006.
[10] S. P. Jenkins, "World income inequality databases: An assessment of WIID and SWIID," The Journal of Economic Inequality, vol. 13, no. 4, pp. 629-671, 2015, doi: 10.1007/s10888-015-9305-3.
[11] S. van Buuren and K. Groothuis-Oudshoorn, "mice: Multivariate imputation by chained equations in R," Journal of Statistical Software, vol. 45, no. 3, 2011, doi: 10.18637/jss.v045.i03.
[12] M. Beine, R. Noël, and L. Ragot, "Determinants of the international mobility of students," Economics of Education Review, vol. 41, pp. 40-54, 2014, doi: 10.1016/j.econedurev.2014.03.003.
[13] A. Abbott and M. Silles, "Determinants of international student migration," The World Economy, vol. 39, no. 5, pp. 621-635, 2016, doi: 10.1111/twec.12319.
[14] E. Ovchinnikova, C. Van Mol, and E. Jones, "The role of language proximity in shaping international student mobility flows," Globalisation, Societies and Education, vol. 21, no. 4, pp. 563-574, 2023, doi: 10.1080/14767724.2022.2070132.
[15] J. Bound, B. Braga, G. Khanna, and S. Turner, "A passage to America: University funding and international students," American Economic Journal: Economic Policy, vol. 12, no. 1, pp. 97-126, 2020, doi: 10.1257/pol.20170620.
[16] J. Bound, B. Braga, G. Khanna, and S. Turner, "The globalization of postsecondary education: The role of international students in the US higher education system," Journal of Economic Perspectives, vol. 35, no. 1, pp. 163-184, 2021, doi: 10.1257/jep.35.1.163.
[17] K. Shih, "Do international students crowd-out or cross-subsidize Americans in higher education?," Journal of Public Economics, vol. 156, pp. 170-184, 2017, doi: 10.1016/j.jpubeco.2017.10.003.
[18] S. Machin and R. Murphy, "Paying out and crowding out? The globalization of higher education," Journal of Economic Geography, vol. 17, no. 5, pp. 1075-1110, 2017, doi: 10.1093/jeg/lbx006.
[19] M. Beine, G. Peri, and M. Raux, "International college students’ impact on the US skilled labor supply," Journal of Public Economics, vol. 223, Art. no. 104917, 2023, doi: 10.1016/j.jpubeco.2023.104917.
[20] OECD, "Tuition fee reforms and international mobility," Education Indicators in Focus, Paris, France: OECD Publishing, 2017, doi: 10.1787/2dbe470a-en.
[21] M. Parey and F. Waldinger, "Studying abroad and the effect on international labour market mobility: Evidence from the introduction of ERASMUS," The Economic Journal, vol. 121, no. 551, pp. 194-222, 2011, doi: 10.1111/j.1468-0297.2010.02369.x.
[22] M. Lörz, N. Netz, and H. Quast, "Why do students from underprivileged families less often intend to study abroad?," Higher Education, vol. 72, no. 2, pp. 153-174, 2016, doi: 10.1007/s10734-015-9943-1.
[23] H. Cebolla-Boado, Y. Hu, and Y. N. Soysal, "Why study abroad? Sorting of Chinese students across British universities," British Journal of Sociology of Education, vol. 39, no. 3, pp. 365-380, 2018, doi: 10.1080/01425692.2017.1349649.
[24] R. Choudaha, "Three waves of international student mobility (1999–2020)," Studies in Higher Education, vol. 42, no. 5, pp. 825-832, 2017, doi: 10.1080/03075079.2017.1293872.
[25] C. R. Glass and N. I. Cruz, "Moving towards multipolarity: Shifts in the core-periphery structure of international student mobility and world rankings (2000–2019)," Higher Education, vol. 85, no. 2, pp. 415-435, 2023, doi: 10.1007/s10734-022-00841-9.
[26] K. R. Gabriel, "The biplot graphic display of matrices with application to principal component analysis," Biometrika, vol. 58, no. 3, pp. 453-467, 1971, doi: 10.1093/biomet/58.3.453.
[27] I. T. Jolliffe and J. Cadima, "Principal component analysis: A review and recent developments," Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 374, no. 2065, Art. no. 20150202, 2016, doi: 10.1098/rsta.2015.0202.
[28] W. Becker, M. Saisana, P. Paruolo, and I. Vandecasteele, "Weights and importance in composite indicators: Closing the gap," Ecological Indicators, vol. 80, pp. 12-22, 2017, doi: 10.1016/j.ecolind.2017.03.056.
[29] S. Greco, A. Ishizaka, M. Tasiou, and G. Torrisi, "On the methodological framework of composite indices: A review of the issues of weighting, aggregation, and robustness," Social Indicators Research, vol. 141, no. 1, pp. 61-94, 2019, doi: 10.1007/s11205-017-1832-9.
[30] A. Saltelli, "Composite indicators between analysis and advocacy," Social Indicators Research, vol. 81, no. 1, pp. 65-77, 2007, doi: 10.1007/s11205-006-0024-9.
[31] M. Saisana, B. d'Hombres, and A. Saltelli, "Rickety numbers: Volatility of university rankings and policy implications," Research Policy, vol. 40, no. 1, pp. 165-177, 2011, doi: 10.1016/j.respol.2010.09.003.
[32] J. G. Kelley and B. A. Simmons, "Politics by number: Indicators as social pressure in international relations," American Journal of Political Science, vol. 59, no. 1, pp. 55-70, 2015, doi: 10.1111/ajps.12119.
[33] H. Goldstein and D. J. Spiegelhalter, "League tables and their limitations: Statistical issues in comparisons of institutional performance," Journal of the Royal Statistical Society. Series A (Statistics in Society), vol. 159, no. 3, p. 385, 1996, doi: 10.2307/2983325.
[34] P. Hall and H. Miller, "Using the bootstrap to quantify the authority of an empirical ranking," The Annals of Statistics, vol. 37, no. 6B, 2009, doi: 10.1214/09-AOS699.
[35] D. B. Rubin, "Inference and missing data," Biometrika, vol. 63, no. 3, pp. 581-592, 1976, doi: 10.1093/biomet/63.3.581.
[36] R. J. A. Little and D. B. Rubin, Statistical Analysis with Missing Data, 3rd ed. Hoboken, NJ, USA: Wiley, 2019, doi: 10.1002/9781119482260.
[37] J. Honaker and G. King, "What to do about missing values in time-series cross-section data," American Journal of Political Science, vol. 54, no. 2, pp. 561-581, 2010, doi: 10.1111/j.1540-5907.2010.00447.x.
[38] R. Lall, "How multiple imputation makes a difference," Political Analysis, vol. 24, no. 4, pp. 414-433, 2016, doi: 10.1093/pan/mpw020.
[39] J. Josse and F. Husson, "missMDA: A package for handling missing values in multivariate data analysis," Journal of Statistical Software, vol. 70, no. 1, 2016, doi: 10.18637/jss.v070.i01.
[40] F. Solt, "The Standardized World Income Inequality Database," Social Science Quarterly, vol. 97, no. 5, pp. 1267-1281, 2016, doi: 10.1111/ssqu.12295.
[41] R. J. Barro and J. W. Lee, "A new data set of educational attainment in the world, 1950–2010," Journal of Development Economics, vol. 104, pp. 184-198, 2013, doi: 10.1016/j.jdeveco.2012.10.001.
[42] D. Cohen and M. Soto, "Growth and human capital: Good data, good results," Journal of Economic Growth, vol. 12, no. 1, pp. 51-76, 2007, doi: 10.1007/s10887-007-9011-5.
[43] A. de la Fuente and R. Doménech, "Human capital in growth regressions: How much difference does data quality make?," Journal of the European Economic Association, vol. 4, no. 1, pp. 1-36, 2006, doi: 10.1162/jeea.2006.4.1.1.
[44] N. D. Belnap, "A useful four-valued logic," in Modern Uses of Multiple-Valued Logic, J. M. Dunn and G. Epstein, Eds. Dordrecht, The Netherlands: Reidel, 1977, pp. 5-37, doi: 10.1007/978-94-010-1161-7_2.
[45] D. Dubois, "On ignorance and contradiction considered as truth-values," Logic Journal of the IGPL, vol. 16, no. 2, pp. 195-216, 2008, doi: 10.1093/jigpal/jzn003.
[46] D. Ciucci, D. Dubois, and J. Lawry, "Borderline vs. unknown: Comparing three-valued representations of imperfect information," International Journal of Approximate Reasoning, vol. 55, no. 9, pp. 1866-1889, 2014, doi: 10.1016/j.ijar.2014.07.004.
[47] K. J. Kaplan, "On the ambivalence-indifference problem in attitude theory and measurement: A suggested modification of the semantic differential technique," Psychological Bulletin, vol. 77, no. 5, pp. 361-372, 1972, doi: 10.1037/h0032590.
[48] A. N. Uma, T. Fornaciari, D. Hovy, S. Paun, B. Plank, and M. Poesio, "Learning from disagreement: A survey," Journal of Artificial Intelligence Research, vol. 72, pp. 1385-1470, 2021, doi: 10.1613/jair.1.12752.
[49] C. Chow, "On optimum recognition error and reject tradeoff," IEEE Transactions on Information Theory, vol. 16, no. 1, pp. 41-46, 1970, doi: 10.1109/TIT.1970.1054406.
[50] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York, NY, USA: Chapman and Hall/CRC, 1994, doi: 10.1201/9780429246593.
