Comparación reproducible de modelos de lenguaje en problemas textuales de mecánica clásica mediante un benchmark automatizado

Omayra Janeth Pérez Castro; Bernardo Fernández García; Luis Antonio Marín Díaz; Juan Manuel Rodríguez Cisneros

doi:10.48204/j.tecno.v8n2.a10864

Submitted July 22, 2026

Published 2026-07-30

Artículos

Vol. 8 No. 2 (2026): Tecnociencia

Reproducible comparison of language models in textual problems of classical mechanics using an automated benchmark

Omayra Janeth Pérez Castro⁺⁻
Bernardo Fernández García⁺⁻
Luis Antonio Marín Díaz⁺⁻
Juan Manuel Rodríguez Cisneros ⁺⁻

DOI https://doi.org/10.48204/j.tecno.v8n2.a10864

PDF (Español (España))

References

DOI: 10.48204/j.tecno.v8n2.a10864

Published: 2026-07-30

How to Cite

Pérez Castro, O. J., Fernández García, B., Marín Díaz, L. A. and Rodríguez Cisneros , J. M. (2026) “Reproducible comparison of language models in textual problems of classical mechanics using an automated benchmark”, Tecnociencia, 8(2), pp. 157–172. doi: 10.48204/j.tecno.v8n2.a10864.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Abstract

This study presents a textual benchmark to compare nine large-scale commercial language models for solving Classical Mechanics problems. A total of 42 self-contained items, without explicit dependence on diagrams, were extracted from MIT OpenCourseWare open-source materials from 2005 and 2016. The evaluated models came from three vendors: OpenAI, Anthropic, and Google Gemini. Each problem was presented using the same structured prompt, including a final answer, a summary of the reasoning, equations, units, and confidence level. The complete run yielded 378 usable responses. The correction reference was built in two stages: numerical consensus for 22 items and manual curation for 19 cases lacking consensus; one item remained ambiguous due to residual dependence on missing figures. The pipeline achieved 100% final success, although 29 truncated outputs required parser-level recovery. In heuristic agreement with the curated reference, Gemini achieved the best aggregate performance by vendor (43.7%; 55/126), while gemini-3.1-pro-preview achieved the best result at the model level (61.9%; 26/42). Anthropic produced the longest and slowest responses, and gemini-3-flash-preview was the fastest model. The findings show that LLMs already support reproducible assessments in physics without replacing experts.

Downloads

Download data is not yet available.

References

Beguš, G., D?bkowski, M., & Rhodes, R. (2025). Large linguistic models: Investigating LLMs’ metalinguistic abilities. IEEE Transactions on Artificial Intelligence, 6(12), 3453–3467. https://doi.org/10.1109/TAI.2025.3575745
Dao, X.-Q., Le, N.-B., Phan, X.-D., Ngo, B.-B., & Vo, T.-D. (2023). Evaluation of ChatGPT and Microsoft Bing AI chat performances on physics exams of Vietnamese National High School Graduation Examination. arXiv. https://doi.org/10.48550/arXiv.2306.04538
Ding, J., Cen, Y., & Wei, X. (2023). Using large language model to solve and explain physics word problems approaching human level. arXiv. https://doi.org/10.48550/arXiv.2309.08182
Gil Pérez, D., & Valdés Castro, P. (1994). La resolución de problemas de física: De los ejercicios de aplicación al tratamiento de situaciones problemáticas. Revista Española de Física, 8(3), 21–26.
Guisasola, J., Zuza, K., Garmendia, M., & Barragués, J. I. (2015). Resolver ejercicios no es fácil: El papel de la metodología científica en la resolución de problemas de física. Revista Brasileira de Ensino de Física, 37(3), 3508. https://doi.org/10.1590/S1806-11173731969
Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force Concept Inventory. The Physics Teacher, 30(3), 141–158. https://doi.org/10.1119/1.2343497
Kortemeyer, G. (2023). Could an artificial-intelligence agent pass an introductory physics course? Physical Review Physics Education Research, 19(1), 010132. https://doi.org/10.1103/PhysRevPhysEducRes.19.010132
Massachusetts Institute of Technology. (2005). Physics I: Classical Mechanics. MIT OpenCourseWare. https://ocw.mit.edu/courses/8-01-physics-i-classical-mechanics-fall-2005/
Massachusetts Institute of Technology. (2016). Classical Mechanics. MIT OpenCourseWare. https://ocw.mit.edu/courses/8-01sc-classical-mechanics-fall-2016/
Navarro Altamar, S. L. (2006). La resolución de problemas planteada como un proceso de investigación hacia la enseñanza de la física. Prospectiva, 4(2), 60–65. https://www.redalyc.org/articulo.oa?id=496251108010
Perales Palacios, F. J. (1993). La resolución de problemas: Una revisión estructurada. Enseñanza de las Ciencias. Revista de Investigación y Experiencias Didácticas, 11(2), 170–178. https://doi.org/10.5565/rev/ensciencias.4533
Rodríguez Fernández, L., Fernández Mena, A. L., Torres Magaña, M. P., Rodríguez Magaña, M. A., & Rodríguez Fernández, M. A. (2024). Inteligencia artificial en la educación: Modelo de lenguaje de gran tamaño (LLM) como recurso educativo. Revista Ipsumtec, 7(2), 157–164. https://doi.org/10.61117/ipsumtec.v7i2.321
Sato, K. (2024). Exploring the educational landscape of AI: Large language models’ approaches to explaining conservation of momentum in physics. arXiv. https://doi.org/10.48550/arXiv.2407.05308
Xu, X., Xu, Q., Xiao, T., Chen, T., Yan, Y., Zhang, J., Diao, S., Yang, C., & Wang, Y. (2025). UGPhysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. Proceedings of Machine Learning Research, 267, 69849–69877.
Zhang, X., Dong, Y., Wu, Y., Huang, J., Jia, C., Fernando, B., Shou, M. Z., Zhang, L., & Liu, J. (2025). PhysReason: A comprehensive benchmark towards physics-based reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 16593–16615). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.acl-long.811
Zhao, J., Tong, J., Mou, Y., Zhang, M., Zhang, Q., & Huang, X. (2024). Exploring the compositional deficiency of large language models in mathematical reasoning through trap problems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 16361–16376). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.915