Keywords
contest scoring
reliability weighting
Monte Carlo simulation
inter-rater agreement
Abstract
Three strands anchor this examination of reliability and rater agreement: Evaluation of the departmental inter-rater reliability when scoring thyroid nodules according…; Inter-rater agreement in Korean constructed-response writing assessment: a large-scale empirical baseline…; and Reliability of OMERACT Scoring System in Ultra-High Frequency Ultrasonography of Minor…. The synthesis centers on human–AI calibration, agreement statistics, and competency gaps, separating recurring design principles from application-specific choices. It also considers what information is required for an independent reader to reproduce or audit the findings. Taken together, the references support a conditional design framework rather than a universal method ranking. The proposed agenda calls for explicit failure analysis, comparable baselines, and reporting that makes scope and uncertainty visible.
References
Zhao, R., Guo, Y., Ma, X., Yin, X., & Tang, J. (2022). Research on evaluation system of the judge quality in students' contest based on data driven. In 2022 3rd International Conference on Education, Knowledge and Information Management (ICEKIM) (pp. 1075-1079). IEEE. https://doi.org/10.1109/ICEKIM55072.2022.00233
Rtam, N. (2024). Evaluation of the departmental inter-rater reliability when scoring thyroid nodules according to the British Thyroid Association Ultrasound-classification model: Is there significant disagreement?. Ultrasound, 32(2), 76-84. https://doi.org/10.1177/1742271x231215500
Gazi, M., Fadairo, A., Minor, M., Acker, J., & Gropen, T.-I. (2019). Abstract WP298: Communication Center Guided Prehospital Stroke Assessment Scoring has High Inter-rater Agreement. Stroke, 50(Suppl_1). https://doi.org/10.1161/str.50.suppl_1.wp298
Söğüt, S., & Büyükkıdık, S. (2026). GenAI for automated essay scoring: A turing test of rubrics, rater agreement, and authorship in L2 writing assessment. Educational Assessment, Evaluation and Accountability. https://doi.org/10.1007/s11092-026-09495-y
Kim, T. (2026). Inter-rater agreement in Korean constructed-response writing assessment: a large-scale empirical baseline from the AI Hub 2024 writing evaluation dataset. Frontiers in Education, 11. https://doi.org/10.3389/feduc.2026.1887497
Gonzales, F. (2025). Inter-Rater Agreement and Competency Gaps in QSEN: A Comparative Study of Student Nurse Self-Assessment and Nurse Educator Evaluation. NURSE EDUCATORS AND PRACTITIONERS JOURNAL, 1(2), 9. https://doi.org/10.64397/nepj.v01i02.2025.a13
Chen, X., Sima, S., Sandhu, H., Kuan, J., & Diwan, A. (2022). Radiographic Evaluation of Lumbar Intervertebral Disc Height Index: An Intra and Inter-Rater Agreement and Reliability Study. . https://doi.org/10.2139/ssrn.4137633
Izzetti, R., Fulvio, G., Nisi, M., Gennai, S., & Graziani, F. (2022). Reliability of OMERACT Scoring System in Ultra-High Frequency Ultrasonography of Minor Salivary Glands: Inter-Rater Agreement Study. Journal of Imaging, 8(4), 111. https://doi.org/10.3390/jimaging8040111
Yun, J. (2023). Relationships among Different Effect-Size Indexes for Inter-Rater Agreement between Human and Automated Essay Scoring. Korean Association For Learner-Centered Curriculum And Instruction, 23(18), 901-919. https://doi.org/10.22251/jlcci.2023.23.18.901
Liao, S.-C., Hunt, E.-A., & Chen, W. (2010). Comparison between Inter-rater Reliability and Inter-rater Agreement in Performance Assessment. Annals of the Academy of Medicine, Singapore, 39(8), 613-618. https://doi.org/10.47102/annals-acadmedsg.v39n8p613