CAPÍTULO 8. CONCLUSIONES Y LÍNEAS DE ACTUACIÓN

Como se ha descrito en esta tesis, el aprendizaje y evaluación de habilidades que contribuyen al desarrollo del PC en la educación reglada está cobrando una gran importancia a nivel internacional, habiendo consenso en que este aprendizaje debe comenzar en edades tempranas. Debido a su carácter reciente, aún no existen instrumentos validados y consensuados para el aprendizaje y la evaluación del PC y, además, es necesario establecer cuáles son las habilidades alcanzables y si hay diferencias en la adquisición de estas habilidades según la edad y género de los niños.

A través de la revisión sistemática de la literatura en cuanto al PC teórico y operacional, se extrajo una propuesta de definición y descomposición del PC fundamentada, así como varias hipótesis sobre los sistemas de aprendizaje y de evaluación que se comprobaron más adelante. Una de las contribuciones más importantes existentes en la literatura era la necesidad de un sistema de evaluación [148], [161] constituido por varios instrumentos de medida diferentes. Una de las hipótesis de la tesis planteaba el posible impacto de cada uno de estos instrumentos de evaluación en las dimensiones del marco 3D [46] (siendo éste el marco de referencia más utilizado para el PC) y, a través de las experiencias realizadas, se comprobó el impacto esperado (véase la Figura 11). En esta tesis se propone no sólo la creación de un sistema de evaluación, sino también la creación de un sistema de aprendizaje del PC, en donde se utilicen diferentes estrategias pedagógicas de aprendizaje combinadas. Mediante el desarrollo exprofeso de un entorno de aprendizaje, se evalúa también el impacto en el aprendizaje del PC de cada una de las estrategias en las dimensiones del marco 3D (véase la Figura 10).

Mediante los instrumentos desarrollados en esta tesis, tanto para el aprendizaje como para la evaluación del PC y los hallazgos en cuanto a las habilidades relacionadas, en etapas tempranas, extraídos de las sucesivos desarrollos y experiencias llevadas a cabo en centros escolares y no escolares, se ha cumplido el objetivo principal (OP) de la tesis, así como todos los objetivos específicos indicados en el apartado 1.2, avanzando en el conocimiento del PC e incidiendo en su componente operacional. También se han confirmado las hipótesis propuestas (apartado 3.2) a lo largo de los desarrollos instrumentales y experiencias realizadas (apartado 3.3). En cuanto al objetivo común OC, dado que corresponde a una colaboración internacional, consideramos que se ha realizado una gran contribución a la investigación global, y se tiene la intención de seguir avanzando en este sentido manteniendo las colaboraciones y trabajos en marcha con otros países (apartado 8.2). En la Tabla 47 puede verse un resumen de la estructura de la tesis en cuanto a objetivos, hipótesis y experiencias realizadas (que se enumeran a continuación):

- E1: Desarrollo y validación de prueba independiente para la evaluación del PC en Educación Primaria: Beginners Computational Thinking test (BCTt).

- E2: Estimación de la validez del BCTt en alumnos de Educación Infantil.

- E3: Desarrollo y validación de videojuego para la enseñanza y evaluación del PC: Blue Ant Code (BAC) en Educación Primaria.

- E4: Análisis de motivación intrínseca, intereses, persistencia y comportamiento frente a recompensas, en etapas tempranas, utilizando el entorno BAC para el aprendizaje de la programación de forma voluntaria.

En los siguientes apartados, se resumen las principales contribuciones, conclusiones y hallazgos en el contexto de la tesis.

8.1. Beginners Computational Thinking Test (BCTt)

El principal obstáculo encontrado al inicio de la investigación fue la inexistencia, en ese momento, de un instrumento autónomo (independiente de un entorno de aprendizaje) para la evaluación del PC en Educación Primaria, lo que dificultaba enormemente evaluar o validar nuevos instrumentos tanto de aprendizaje como de evaluación no-autónomos, sin posibilidad de poder realizar experiencias de tipo pre y post-test para evaluar la idoneidad de nuevos entornos y herramientas para el desarrollo del PC.

Esta carencia de instrumentos hacía muy difícil reconocer de forma aséptica las habilidades a cada edad, ya que podía haber sesgos debido a que ciertas habilidades pudieran ser más o menos alcanzables debido a las características del entorno particular utilizado. Por este motivo se llevó a cabo las experiencias enmarcadas en E1 (capítulo 4), en las que se desarrolló un instrumento autónomo, el Beginners Computational Thinking Test (BCTt) [7], [8], y se validó en la etapa de Educación Primaria, en tres centros educativos. Los datos obtenidos mostraron una alta fiabilidad con un Alfa de Cronbach=0,824 en toda la etapa de Educación Primaria, aunque la recomendación final es que el BCTt es idóneo para 1º y 2º de primaria principalmente. Al constatar una reducción de la fiabilidad del test en los alumnos más mayores, se desarrolló, a través la colaboración con un equipo de la Ecole Polytechnique Fédérale de Lausanne (EPFL) en Suiza, el competence Computational Thinking test (cCTt) [165], y se validó en centros educativos de Suiza. Además, se establecen los límites de edad para la administración del BCTt y el cCTt a través de una experiencia administrando ambos test a estudiantes de Educación Primaria en centros educativos de Portugal. Los resultados del análisis psicométrico indican que se debería preferir el cCTt para los alumnos de 3º y 4º curso. No obstante, el BCTt puede utilizarse si el objetivo es discriminar entre alumnos con capacidades bajas de estos dos cursos.

Por otro lado, para establecer el límite inferior de edad en el que se puede administrar el BCTt y a través de la colaboración internacional con la Fontys University of Applied Science y la Zuyd University of Applied Science, ambas en Países Bajos, se llevó a cabo la experiencia E2 (capítulo 5) utilizando entornos de aprendizaje robóticos en Educación Infantil, con niños de 4 y 5 años, en un centro escolar de los Países Bajos [6]. Mediante esta experiencia, se determina la idoneidad de estos entornos de aprendizaje para el desarrollo del PC. Al mismo tiempo, se confirma la fiabilidad del BCTt en niños de 4 años (Alfa de Cronbach = 0,802). Sin embargo, cuatro de los seis conceptos computacionales que se abordan en el test parecen no ser alcanzables para los alumnos de 4 años, ya que no fueron capaces de responder a las preguntas relacionadas con ellos ni mostraron ninguna mejora en el post-test respecto a dichos conceptos. Por ello, es necesario el diseño de otra versión del BCTt dirigida a los alumnos de 4 años o menos. Esta versión se está actualmente realizando en colaboración con un equipo de la Open University Netherlands (OU), en Países Bajos.

El BCTt y el cCTt son un instrumentos autónomos, independientes de cualquier entorno, que se han revelado fiables para la evaluación del PC en Educación Primaria y pueden ser administrados como pre-test y post-test en las investigaciones que lo requieran. Estos test pueden considerarse una extensión fiable del CTt de Román et al. para estudiantes más jóvenes, y se enmarcan en el objetivo internacional común OC de creación de instrumentos independientes de evaluación del PC para todos los rangos de edad y poblaciones. En la Tabla 48 se muestran los instrumentos aplicables a cada edad en el contexto del objetivo OC.

En la Tabla 49 se muestran las habilidades de PC alcanzables y/o nivel de adquisición posible, que se desprende de forma preliminar de la administración simple del test, en las experiencias E1 y E2.

Además, el BCTt se ha traducido a numerosos idiomas, se han realizado y actualmente se están llevando a cabo varias experiencias en diferentes poblaciones y países (como Portugal, Bélgica o Singapur) y planteado líneas de actuación futuras.

8.2. Evaluación del PC: líneas actuales y futuras

Como se ha visto, el BCTt [7], [8] ha tenido gran impacto internacional. Las primeras colaboraciones con Suiza, Países Bajos y Portugal, han resultado en el desarrollo y validación del competence computational Thinking test (cCTt) junto a la Ecole Polytechnique Fédérale de Lausanne (EPFL): MOBOTS Group of the BIOROB Laboratory; y LEARN – Center for Learning Sciences; y Computer Human Interaction in Learning and Instruction (CHILI) [165].

Además, se estableció una colaboración entre la Ecole Polytechnique Fédérale de Lausanne (EPFL) y la ONG Tree Tree 2⁷, para la realización de un estudio comparativo del BCTt y el cCTt en centros escolares de Portugal. Este estudio está actualmente en fase de revisión para su publicación en la ITiCSE 2022 (27th Annual Conference on Innovation and Technology in Computer Science Education) (apartado 4.6.2).

Por otro lado, mediante la colaboración con la Fontys University of Applied Science y la Zuyd University of Applied Science, ambas en Países Bajos, se pudo establecer el límite de edad inferior para la administración del BCTt [6].

En la actualidad, se están llevando a cabo otras colaboraciones con otros países y poblaciones con respecto al OC: Diseño y validación de instrumentos de evaluación del PC que cubran un amplio rango de edades y poblaciones, a continuación, se enumeran las más relevantes:

- Realización de un estudio sobre la posibilidad de desarrollar el PC en Educación Primaria a través de actividades Bebras [161], mediante otra experiencia en centros escolares de Portugal, utilizando el BCTt como herramienta de pre y post-test. Esta experiencia está en fase de redacción del artículo científico.

- Desarrollo de una versión del test para niños menores de 4 años en colaboración con la Open University (OU), en Países Bajos, incluyendo aspectos sensoriales diferentes al visual. Este proyecto se encuentra en su fase inicial y contempla incluir otros elementos sensoriales como el tacto o el oído en el instrumento de evaluación.

- Validación de límites del BCTt y estudio de habilidades del PC en Educación Primaria: colaboración con la UNED y el INTEF (Ministerio de Educación) en la administración del BCTt en 2.500 estudiantes de entre alumnos de infantil (5 años) y 3º de primaria. Los estudios preliminares indican una una fiabilidad muy buena (Alfa de Cronbach = 0.87). Este estudio está en fase de redacción.

- Validación del BCTt en otras poblaciones, en colaboración con el Instituto Universitario de Lisboa (ISCTE), en la que se administró el BCTt en colegios de Portugal a +600 alumnos. Estos resultados están en fase de análisis.

- Integrar los conceptos e ideas de la educación del PC en las prácticas de los profesores de Educación Primaria es un reto difícil. Los profesores deben adquirir nuevos conocimientos y habilidades relacionados con la forma en que los contenidos, las estrategias pedagógicas y las herramientas TIC, deben combinarse para introducir el PC en sus prácticas educativas de manera significativa [38]. En este contexto, se plantea un estudio de percepciones del profesorado sobre el PC en el currículo escolar. Esta investigación resulta de una colaboración entre 4 países: España, Países Bajos, Portugal y Singapur, y está en estos momentos en la fase de recogida de datos. Se prevee la presentación del estudio en el congreso CTE-STEM 2022 (Sixth APSCE International Conference on Computational Thinking and STEM Education) con el título: “Understanding Teachers’ Attitudes and Self-Assessment Towards Computational Thinking”.

- Creación de un equipo internacional (España, Países Bajos, Bélgica, Portugal, Suiza), a través de una convocatoria Erasmus + para la inclusión del PC en el currículo de forma estandarizada, desarrollo y validación de instrumentos para su evaluación y atención especial a la diversidad, ya que la integración del PC en la educación formal puede ser una oportunidad para redefinir el currículo escolar en la educación informática, de forma hacia una mayor inclusión y diversidad [34]. Este proyecto se encuentra en fase de propuesta y está enmarcado especialmente en el objetivo común de la tesis OC.

Además, está habiendo algunas iniciativas independientes y se están realizando experiencias utilizando el BCTt en multitud de países como, por ejemplo, en Bélgica, el Service d'Ingénierie Pédagogique et du Numérique éducatif, de la Universidad Umons-FPSE ha desarrollado una versión desenchufada del BCTt (véase la Figura 73) para niños de 3 y 4 años: Beginners Computational Thinking Test: Adaptation pour un public de moins de 5 ans [234], que ha resultado muy fiable para la evaluación del PC.

También se están desarrollando investigaciones a gran escala, por ejemplo, en colegios de Italia utilizando el BCTt, a través de un proyecto financiado por el Ministerio Italiano de Educación, Universidades e Investigación. En Suiza, la Ecole Polytechnique Fédérale de Lausanne (EPFL) incluyó el BCTt en el sitio Web de Roteco⁸, que aúna las siguientes instituciones: Academias Suizas de Ciencias, SUPSI, la EPFL, la ETHZ y escuelas St. Gallen, Lucerna, Schwyz y Valais. También hay un inicio de colaboración con profesores e investigadores de China y el Ministerio de Educación de Singapur, así como comienzos de colaboraciones con Cornell University (EEUU), University of Hradec Kralove (República Checa), University of Tübingen (Alemania), Vilnius University (Lituania), y la Université Lumière Lyon 2 (ISPEF) (Francia), entre otras.

Asimismo, para estas investigaciones en múltiples poblaciones, el BCTt se ha traducido a numerosos idiomas, por ejemplo, italiano, neerlandés, portugués, alemán, francés, chino o farsi.

8.3. Blue Ant Code (BAC)

En el contexto de la tesis, se ha desarrollado un entorno de aprendizaje y evaluación a través de analíticas de aprendizaje. El desarrollo del entorno BAC utilizado incluye el videojuego compatible con múltiples plataformas, una api REST para la conexión con la base de datos relacional y módulos complementarios para el filtrado, la visualización y análisis de datos con GLA. Este entorno se utilizó en la experiencia E3: Validación y análisis del entorno educativo basado en el juego BAC, en modos individual y colaborativo y análisis de habilidades de PC en Educación Primaria (véase capítulo 6). Además, se desarrolló una segunda versión de BAC para su adaptación al entorno masivo no escolar, incluyendo nuevas funcionalidades como la inclusión de un sistema de recompensas, que se utilizó en la E4: Análisis de competencias: motivación intrínseca, persistencia y recompensas mediante analíticas de aprendizaje con BAC (véase capítulo 7).

La investigación sobre cómo enseñar PC y especialmente en las dimensiones de prácticas y perspectivas computacionales del marco 3D es escasa, por lo que en esta tesis se propone un sistema de aprendizaje combinando tres de las estrategias de aprendizaje más utilizadas en el entorno BAC, siendo éste un entorno colaborativo basado en el juego, construido sobre una estrategia constructivista basada en problemas, que ha sido validado en la etapa de Educación Primaria en la experiencia E3. Los resultados de esta experiencia muestran que este enfoque es adecuado para el aprendizaje del PC, especialmente en edades tempranas, cubriendo las tres dimensiones clave del marco 3D (véase la Figura 49), como se planteó en la hipótesis HE2, y se ha confirmado mediante el caso de estudio. La combinación de diferentes metodologías de evaluación, como un sistema de evaluación [148]: BCTt, GLA a través de GBL, y datos cualitativos (observación directa y cuestionarios), permite una evaluación tridimensional, como se planteó en la HE4 y se ha confirmado también mediante este caso de estudio (véase la Figura 50).

En cuanto a la validación de BAC en Educación Primaria, los resultados de la experiencia E3 confirmaron que es un entorno de aprendizaje apropiado, con mejoras muy significativas en el desarrollo del PC incluso con poco tiempo de interacción con el entorno. Además, también resultó adecuado como herramienta de evaluación con resultados coherentes con el análisis de los datos recogidos por BAC y, además, puede ofrecer datos en tiempo real que pueden ser útiles para una evaluación personal detallada, un fuerte análisis de datos de bajo nivel, y la identificación de conceptos computacionales inalcanzables para cada edad específica no solamente en un instante determinado, sino a lo largo y al final de un proceso de aprendizaje específico. No obstante, tal y como ocurrió con el BCTt, BAC parece estar mejor adaptado a los estudiantes más jóvenes e incluso ser adecuado para cursos inferiores (Educación Infantil), sobre todo en el modo colaborativo. Además, BAC es especialmente adecuado y motivador para alumnos con necesidades especiales o de bajo percentil particularmente en la modalidad de juego colaborativo.

También se obtuvieron resultados positivos en la prueba de retención del aprendizaje, lo que sugiere que esta metodología basada en el juego fomenta el aprendizaje duradero. Los resultados de retención del aprendizaje fueron mejores en la modalidad colaborativa que en la individual, sobre todo en los cursos inferiores, lo que nos lleva a la conclusión de que la modalidad de juego colaborativo fomenta un aprendizaje más eficaz y duradero. Sin embargo, es posible que la estrategia colaborativa no sea tan eficaz para el último ciclo de Educación Primaria, ya que estos alumnos parecen ser capaces de una reflexión individual más eficaz. Se necesitan más investigaciones para comprobar esta hipótesis.

Uno de los hallazgos más relevantes es que los alumnos de los primeros cursos de Educación Primaria fueron capaces de superar las habilidades de PC los alumnos de cursos superiores al final de la experiencia, con lo que en esta tesis se resalta la importancia de desarrollar el PC desde edades muy tempranas y la recomendación de incorporar ciertos conceptos, como los bucles anidados o los condicionales, en el currículo ya en primer curso o incluso antes, en lugar de esperar hasta el último ciclo de Educación Primaria (o incluso secundaria), como ocurre actualmente en la mayoría de los centros educativos en España.

La experiencia E4, de despliegue masivo de BAC en un entorno no escolar, se realizó con el objetivo general de analizar estrategias para el aprendizaje y evaluación del PC y las habilidades de PC de alumnos de Educación Primaria, con especial atención en las disposiciones y predisposiciones para el PC basadas en la motivación intrínseca y el juego voluntario. Para esta experiencia, se utilizó el entorno adaptado de BAC para el despliegue masivo. A continuación, se resumen los principales hallazgos de esta experiencia:

- Interés inicial: A priori parece haber un mayor interés por aprender a programar en los niños que en las niñas, los niños juegan más a menudo y pasan más tiempo jugando hasta los 5 años; sin embargo, a partir de los 6 años, el tiempo de juego y el número de partidas es mucho mayor en las niñas que en los niños.

- Rendimiento: En todas las edades es significativamente mayor en las niñas que en los niños, especialmente de los 3 a los 5 años. Sin embargo, de los 6 a los 8 años, aunque su rendimiento sigue siendo mejor, las niñas pasan mucho más tiempo jugando y, según los resultados de la investigación realizada, este comportamiento no está relacionado con el dominio de los conceptos abordados, sino que se debe a un comportamiento de persistencia-recompensa.

- Resolución de problemas computacionales específicos: Las chicas tienen un mayor interés y rendimiento en los niveles más difíciles (bucles anidados y secuencias largas), en comparación con los chicos, que tienen un interés similar por cada concepto, pero un mayor rendimiento sólo en bucles simples, de dificultad media. Las secuencias simples parecen dominarse a partir de los 3 años de edad, independientemente del género. Sin embargo, debido a la memoria de trabajo las secuencias largas no se dominan hasta la edad de 5 años, independientemente del género. Los bucles simples se dominan en torno a los 3 años para los niños y a los 4

- años para las niñas. En cambio, los bucles anidados parecen superarse a partir de los 5 años para las niñas y de los 6 años para los niños. Por último, en lo que respecta a los condicionales, el rendimiento aumenta a los 4 años, independientemente del sexo. Todos los conceptos abordados pueden ser superados entre los 3 y los 6 años, lo que podría ser un aspecto a tener en cuenta para la elaboración de los programas escolares.

- Progresión en el aprendizaje: Tanto las niñas como los niños progresan, pero la mejora de los niños es más significativa tanto en tiempo como en rendimiento. Sin embargo, las niñas obtienen un logro significativamente mayor que los niños y alcanzan el dominio completo en todos los niveles en los primeros 10 intentos entre los 3 y 5 años, mientras que este comportamiento no se observa en el caso de los niños. A partir de los 6 años, las niñas tardan más tiempo e intentos en resolver los retos debido a un comportamiento de persistencia-recompensa. Es destacable que, aunque los bucles anidados parecen ser difíciles para los alumnos de primaria, es posible dominar completamente este concepto a los 4 años, independientemente del género, si se realizan suficientes intentos y, además, los niños son capaces de persistir en el reto voluntariamente, intrínsecamente motivados, hasta dominarlo. Este resultado contrasta con el obtenido en la experiencia E2, en el que muchos de estos conceptos no fueron alcanzables a los 4 años, con lo que se confirma la importancia de la motivación intrínseca en los procesos de aprendizaje. Este podría ser también un aspecto a tener en cuenta para el desarrollo de los programas escolares, ya que todos los conceptos abordados (secuencias, bucles simples y anidados, y condicionales) podrían dominarse a los 4 años si existe una persistencia intrínsecamente motivada.

- Comportamientos de persistencia ante el reto y frente a las recompensas: La persistencia-reto (repetición a pesar de perder) se ha observado claramente en ambos géneros, pero es más pronunciado en los niños, lo que podría explicar por qué su progreso está altamente correlacionado con la persistencia. En cambio, el comportamiento de persistencia-recompensa (repetición después de ganar) se observa significativamente más en las niñas, especialmente entre los 5 y los 7 años, cuando ya dominan todos los conceptos abordados. Este es un hallazgo relevante de nuestra investigación ya que, aunque el aprendizaje derivado de este comportamiento es posible, este aprendizaje podría ser reducido ya que el objetivo no es superar el reto sino obtener recompensas. Dado que los niños son capaces de dominar los conceptos abordados antes de los 5 años mediante la práctica continuada, nos preguntamos si no es contraproducente retrasar la adquisición de estos conceptos hasta después de los 5 años, cuando la conducta de persistencia- reto es menor y surge la de persistencia-recompensa, especialmente en las niñas. Por otro lado, a partir de los 5 años, también se podría sugerir la motivación a través de recompensas, especialmente en las niñas, pero siempre teniendo en cuenta no sobrepasar el límite en el que las recompensas pueden ser perjudiciales para el aprendizaje, disminuyendo la motivación intrínseca.

Las posibles aplicaciones derivadas de los resultados de la investigación se basan en la recomendación de analizar los aspectos de persistencia y recompensas en los juegos educativos. El entorno BAC podría utilizarse en contextos escolares, además de para el aprendizaje de la programación y su evaluación, para analizar aspectos relacionados con la motivación intrínseca, pero, en este caso, siempre que su uso sea totalmente voluntario. En este mismo sentido, se recomienda incorporar juegos educativos basados en la motivación intrínseca en los entornos escolares. Los currículos escolares podrían diseñarse teniendo en cuenta que, a través de esta motivación, los niños pueden ser capaces de superar los conceptos computacionales abordados (incluidos los bucles anidados) antes de los 5 años, existiendo diferencias de género en la adquisición de estos conceptos. También se recomienda diseñar juegos educativos teniendo en cuenta estos aspectos, y de esta forma potenciar la conducta de persistencia-reto en los niños y, a partir de los 5 años, controlar o potenciar la conducta de persistencia- recompensa respecto a las niñas. Creemos que los hallazgos podrían guiar el diseño de mejores herramientas para el aprendizaje de la programación y el desarrollo del PC. Dado que la edad y el género influyen en el rendimiento en estos retos, es aconsejable adaptar las herramientas de aprendizaje en consecuencia, así como el currículo escolar. Además, los entornos de aprendizaje podrían adaptar la forma de proporcionar refuerzos y recompensas, especialmente para los niños en los retos más complejos y para las niñas a partir de los 5 años. Sin embargo, se necesitan nuevos estudios que tengan en cuenta estas complejidades.

8.4. Aprendizaje del PC: líneas actuales y futuras

Actualmente se están realizando nuevos módulos de BAC para explorar el aprendizaje y evaluación de otros conceptos y habilidades de PC, además de módulos de análisis y visualización de datos con GLA, aparte de los existentes (apartado 6.4.3). Algunos de los proyectos actualmente en desarrollo son:

- Módulo de juego de sudokus adaptativos: se plantea el desarrollo de la habilidad de resolución de problemas mediante la descomposición a través de la resolución de sudokus. Para ello, se han desarrollado a nivel de prototipo jugable y validado, varios niveles de juego con sudokus de creciente dificultad (véase la Figura 74, parte izquierda) [235].

- Módulo de juego de cifrado y descifrado de códigos: también se ha realizado un prototipo jugable con dos niveles de dificultad para su futura implementación (véase la Figura 74, parte derecha) [236].

- Módulo para el análisis de datos automático en cuanto a caminos óptimos en BAC. En este caso, el módulo analizará los datos de la base de datos y se crea un informe estadístico periódico sobre la progresión de los jugadores en cuanto a soluciones óptimas. Este módulo se está desarrollando en C# con el IDE Visual Studio Code.

Algunas de las líneas futuras de investigación son:

- Ampliación del entorno de juego BAC con más conceptos computacionales.

- Inclusión y evaluación mediante experiencias de entornos de aprendizaje basados en la motivación intrínseca en contextos escolares.

- Análisis del código propuesto por los estudiantes en términos de optimización y estilos de programación utilizando GLA.

- Ampliación de BAC incluyendo los hallazgos sobre comportamientos de juego.

- Ampliación de BAC para hacer uso de GLA adaptativo sin reducir la MI.

Además, para la evaluación de nuevas metodologías de aprendizaje, se ha iniciado una colaboración con la Universidad de Harvard para guiar la creación de un videojuego para el aprendizaje y evaluación del CT en alumnos de educación secundaria.

En general, se pretende seguir investigando y colaborando internacionalmente para la creación de instrumentos y herramientas de aprendizaje y evaluación para el desarrollo del PC en etapas tempranas, con especial atención a la diversidad.

8.5. Publicaciones

Esta tesis ha dado lugar a publicaciones en artículos en revistas, artículos en congresos y un capítulo de libro. En este apartado, se detallan las publicaciones y acciones más relevantes.

8.5.1. Revistas JCR (Q1 y Q2)

- M. Zapata-Caceres, E. Martin, and M. Roman-Gonzalez, "Collaborative Game-Based Environment and Assessment Tool for Learning Computational Thinking in Primary School: A Case Study," in IEEE Transactions on Learning Technologies, vol. 14, no. 5, pp. 576-589, 1 Oct. 2021, doi: 10.1109/TLT.2021.3111108.

- M. Zapata-Cáceres and E. Martín-Barroso, "Applying Game Learning Analytics to a Voluntary Video Game: Intrinsic Motivation, Persistence, and Rewards in Learning to Program at an Early Age," in IEEE Access, vol. 9, pp. 123588-123602, 2021, doi: 10.1109/ACCESS.2021.3110475.

- L. El-Hamamsy, M. Zapata-Caceres, E. Martín, F. Mondada, J. Dehler Zufferey, and B. Bruno, "The competent Computational Thinking test (cCTt): Development and validation of an unplugged Computational Thinking test for upper primary school," Journal of Educational Computing Research, 2022. In press.

8.5.2. Congresos internacionales

- M. Zapata-Cáceres, E. Martín-Barroso, and M. Román-González, “Computational thinking test for beginners: Design and content validation,” in IEEE Global Engineering Education Conference (EDUCON’20), Porto, Portugal, pp. 1905–1914, 2020, doi: 10.1109/EDUCON45650.2020.9125368.

- M. Zapata-Cáceres and N. Fanchamps, "Using the beginners computational thinking test to measure development on computational concepts among preschoolers," in Proceedings of the 5th APSCE International Computational Thinking and STEM in Education Conference 2021, Singapore, National Institute of Education, 2021, pp. 3237. Premiado como “Best full paper”.

- L. El-Hamamsy, M. Zapata-Caceres, P. Marcelino, J. Dehler Zufferey, B. Bruno, and E. Martin, “Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: the Beginners' CT test (BCTt) and the competent CT test (cCTt)” in ITiCSE 2022, 27th Annual Conference on Innovation and Technology in Computer Science Education. Under peer review.

- M. Zapata-Caceres, N. Fanchamps, I. H. Yeter, P. Marcelino, and E. Martín-Barroso, “Understanding Teachers’ Attitudes and Self-Assessment Towards Computational Thinking” in CTE-STEM 2022, Sixth APSCE International Conference on Computational Thinking and STEM Education. In submission process.

8.5.3. Congresos nacionales

- Presentación de póster “Computational Thinking at early stages: learning assessment tolos and school curricula approach”, en II Congreso de la Escuela Internacional de Doctorado de la Universidad Rey Juan Carlos, Noviembre 2019.

- Presentación “Videojuegos para enseñar a programar: analíticas y motivación” en Congreso CE + Ed Educare + Educere de Formación del Profesorado, 2021.

8.5.4. Capítulos de libro

- Zapata-Cáceres María, E. Martín-Barroso and M. Román-González, "BCTt: Beginners computational thinking test," in Raspberry Pi Foundation Research Seminars, Apr 2021, pp. 46-56. ISSN 2514-586X (16). Available: https://www.raspberrypi.org/app/uploads/2021/05/Understanding-computing-education-Volume-1-%E2%80%93-Raspberry-Pi-Foundation-Research-Seminars.pdf.

8.5.5. Divulgación

- Presentación de la experiencia “Experiencias de enseñanza de conceptos computacionales en Educación Primaria a través de un videojuego en modos colaborativo e individual” en las IV Jornadas de Innovación Educativa iEDU - CRIF Las Acacias, mayo 2019.

- Presentación “Experiencia de enseñanza y evaluación del pensamiento computacional en Educación Primaria a través de un videojuego colaborativo”, en SIMO Educación, noviembre 2019.

- Presentación y panel de discusión Q&A “Computational thinking test for beginners” en the online seminars on computing education research from the Raspberry Pi Foundation, Cambridge UK, julio 2020.

- Taller “Recursos Tecnológicos en Educación Infantil, Primaria y NEE”, en XIX Semana de la Ciencia y de la Innovación de Madrid, Universidad Rey Juan Carlos, noviembre 2019.

- Taller “Aprende a programar jugando a videojuegos con amigos”, en XIX Semana de la Ciencia y de la Innovación de Madrid, Universidad Rey Juan Carlos, noviembre 2019.

- Taller “Cómo aprender a programar jugando”, en XI Noche Europea de los Investigadores de Madrid, Universidad Rey Juan Carlos, noviembre 2020.

- Taller “Aprende a programar jugando a videojuegos”, en XI Noche Europea de los Investigadores de Madrid, Universidad Rey Juan Carlos, noviembre 2020.

CHAPTER 8. CONCLUSIONS AND LINES OF ACTION

As described in this thesis, the learning and assessment of skills that contribute to the development of CT in formal education is gaining importance at an international scale, and there is a consensus that this learning should begin at an early age. Due to its recent nature, there are still no validated and agreed instruments for learning and assessing CT, and it is also necessary to establish which skills are achievable and whether there are differences in the acquisition of these skills according to the age and gender of the children.

Through the systematic literature review on theoretical and operational CT, a proposal for a grounded definition and decomposition of CT was provided, as well as several hypotheses on learning and assessment systems that were further tested. One of the most important contributions in the literature was the need for a system of assessments [148], [161] consisting of several different measuring instruments. One of the hypotheses of the thesis posed the possible impact of each of these assessment instruments on the dimensions of the 3D framework [46] (this being the most widely used reference framework for CT) and, through the studies carried out, the expected impact was verified (Figure 11). Moreover, this thesis proposes not only the creation of a system of assessments, but also the creation of a CT learning system, where different pedagogical learning strategies are used in combination. By developing a specific learning environment, the impact on the CT learning of each of the strategies in the dimensions of the 3D framework is also evaluated (Figure 10).

By means of the instruments developed in this thesis, both for the learning and assessment of CT, and the findings regarding the CT skills in early stages, extracted from the successive developments and studies carried out in schools and non-school settings, the main aim (OP) of the thesis has been fulfilled, as well as all the specific aims indicated in section 1.2, advancing in the knowledge of CT and impacting on its operational component. The hypotheses proposed (section 3.2) have also been confirmed throughout the instrumental developments and studies carried out (section 3.3). As for the common aim OC, given that it corresponds to an international collaboration, we consider that a great contribution has been made to global research, and we intend to continue advancing in this direction by maintaining the collaborations and work in progress with other countries (section 8.2). Table 47 shows a summary of the structure of the thesis in terms of aims, hypotheses and studies carried out (listed below).

- E1: Development and validation of a stand-alone test for the assessment of CT in Primary Education: Beginners Computational Thinking test (BCTt).

- E2: Estimation of the validity of the BCTt in Infant Education students.

- E3: Validation and analysis of the educational environment based on the BAC game, in individual and collaborative modes and analysis of CT skills in Primary Education.

- E4: Analysis of intrinsic motivation, interests, persistence and behavior in response to rewards, in learning programming in early stages using the BAC environment.

The following sections summarise the main contributions, conclusions, and findings in the context of the thesis.

8.6. Beginners Computational Thinking Test (BCTt)

The main obstacle encountered at the beginning of the research was the non-existence, at that time, of an autonomous instrument (independent of a specific learning environment) for the assessment of CT in Primary Education, which made it very difficult to evaluate or validate new learning and non-autonomous assessment instruments, with no possibility of carrying out pre- and post-test type studies to assess the suitability of new environments and tools for the development of CT.

This lack of instruments made it very difficult to aseptically recognise skills at each age, as there could be biases due to the fact that certain skills could be more or less achievable due to the characteristics of the particular environment used. For this reason, the research framed in E1 (chapter 4) was carried out, in which an autonomous instrument, the Beginners Computational Thinking Test (BCTt) [7], [8], was developed and validated at the Primary Education stage, in three different schools. The data obtained showed a high reliability with a Cronbach's alpha = 0.824 across the entire primary education stage, although the final recommendation is that the BCTt is suitable mainly for the 1st and 2nd years of primary school. Since a decrease in test reliability was observed in older students, the competence Computational Thinking test (cCTt) [165] was developed in collaboration with a research team from the Ecole Polytechnique Fédérale de Lausanne (EPFL) in Switzerland, and validated in several schools in Switzerland. Furthermore, age limits for the administration of the BCTt and the cCTt are established through a study administering both tests to primary school students in several schools in Portugal. The results of the psychometric analysis indicate that the cCTt should be preferred for 3rd and 4th grade students. However, the BCTt can be used if the aim is to discriminate between low-ability students in these two grades.

On the other hand, in order to establish the lower age limit at which BCTt can be administered, and through international collaboration with Fontys University of Applied Science and Zuyd University of Applied Science, both in the Netherlands, the E2 case study (chapter 5) was carried out using robotic learning environments in Early Childhood Education, with children aged 4 and 5 years old, in a school in the Netherlands [6]. Through this study, the suitability of these learning environments for the development of CT is determined. At the same time, the reliability of the BCTt with 4-year-olds is confirmed (Cronbach's alpha = 0.802). However, four of the six computational concepts addressed in the test seem not to be achievable for 4-year-olds, as they were not able to answer the related questions and did not show any improvement in the post-test with respect to these concepts. Therefore, it is necessary to design another version of the BCTt for students aged 4 years and younger. This version is currently being developed in collaboration with a team from the Open University Netherlands (OU) in the Netherlands.

The BCTt and the cCTt are stand-alone instruments, independent of any environment, which have proven reliable for the assessment of CT in Primary Education and can be administered as a pre-test and post-test in research that require it. These tests can be considered a reliable extension of the CTt by Román et al. for younger students, and are part of the common international aim of developing stand-alone CT assessment instruments for all age ranges and populations. Table 48 shows the instruments applicable to each age range in the context of the OC aim.

Table 49 shows the achievable CT skills and/or possible level of acquisition, preliminarily derived from the simple administration of the test, in the E1 and E2 case studies.

In addition, the BCTt has been translated into numerous languages, several studies have been carried out and are currently being carried out in different countries (such as Portugal, Belgium, or Singapore) and future lines of action have been proposed, which are in different stages of development, these actions are also described in more detail in the following section.

8.7. CT assessment: current and future directions

The BCTt [7], [8] has had a significant international impact. The first collaborations with Switzerland, the Netherlands and Portugal have resulted in the development and validation of the competence computational Thinking test (cCTt) together with the Ecole Polytechnique Fédérale de Lausanne (EPFL): MOBOTS Group of the BIOROB Laboratory; and LEARN - Center for Learning Sciences; and Computer Human Interaction in Learning and Instruction (CHILI) [165].

In addition, a collaboration was established between the Ecole Polytechnique Fédérale de Lausanne (EPFL) and the NGO Tree Tree 2¹⁰ for a comparative study of BCTt and cCTt in schools in Portugal. This study is currently under review for publication at the ITiCSE 2022 (27th Annual Conference on Innovation and Technology in Computer Science Education) (section 8.5).

Furthermore, through collaboration with Fontys University of Applied Science and Zuyd University of Applied Science, both in the The Netherlands, it was possible to establish the lower age limit for the administration of the BCTt [6].

Other collaborations with other countries and populations are currently underway regarding the OC aim: Design and validation of CT assessment instruments covering a wide range of ages and populations, the most relevant of which are listed below.

Currently, other collaborations are being carried out with other countries and populations with respect to the CT: Design and validation of CT assessment instruments covering a wide range of ages and populations, the most relevant of which are listed below:

- Conducting a case study on the possibility of developing CT in Primary Education through Bebras activities [161], through a study in schools in Portugal, using the BCTt as a pre- and post-test tool. This study is currently in the drafting stage of a scientific paper.

- Development of a version of the test for children under 4 years of age in collaboration with the Open University (OU) in The Netherlands, including sensory aspects other than visual, such as sound. This project is in its initial phase and includes other sensory elements such as touch and hearing in the assessment instrument.

- Validation of the BCTt limits and study of CT skills in Primary Education: collaboration with the UNED and INTEF (Ministry of Education) in the administration of the BCTt to 2500 students between infant (5 years old) and 3rd grade of primary school. Preliminary studies indicate very good reliability (Cronbach's alpha = 0.87). This study is in the drafting phase.

- Validation of the BCTt in other populations, in collaboration with the University Institute of Lisbon (ISCTE), in which the BCTt was administered in schools in Portugal to +600 pupils. These results are currently being analysed.

- Integrating the concepts and ideas of CT education into the practices of primary school teachers is a difficult challenge. Teachers need to acquire new knowledge and skills related to how content, pedagogical strategies and ICT tools should be combined to introduce CT into their educational practices in a meaningful way [38]. In this context, a study of teachers' perceptions of CT about the school curricula is proposed. This research is the result of a collaboration between 4 countries: Spain, The Netherlands, Portugal, and Singapore, and is currently in the data collection phase. The study is expected to be presented at the CTE-STEM 2022 conference (Sixth APSCE-International Conference on Computational Thinking and STEM Education) with the title: "Understanding Teachers' Attitudes and Self-Assessment Towards Computational Thinking".

- Creación de un equipo internacional (España, Países Bajos, Bélgica, Portugal, Suiza), a través de una convocatoria Erasmus + para la inclusión del PC en el currículo de forma estandarizada, desarrollo y validación de instrumentos para su evaluación y atención especial a la diversidad, ya que la integración del PC en la educación formal puede ser una oportunidad para redefinir el currículo escolar en la educación informática, de forma hacia una mayor inclusión y diversidad [34]. Este proyecto se encuentra en fase de propuesta. Este proyecto se enmarca especialmente en el objetivo común de la tesis OC.

- Foundation of an international team (Spain, Netherlands, Belgium, Portugal, Switzerland), through an Erasmus + call for the inclusion of CT in the curriculum in a standardised way, development, and validation of instruments for its assessment and special attention to diversity, since the integration of CT in formal education can be an opportunity to redefine the school curricula in computer education, towards greater inclusion and diversity [34]. This project is at the proposal stage. This project is particularly framed by the common aim of the OC thesis.

In addition, there are some independent initiatives and studies using the BCTt in many countries, e.g., in Belgium, the Service d'Ingénierie Pédagogique et du Numérique éducatif, Umons-FPSE University has developed an unplugged version of the BCTt (see Figure 73) for 3- and 4-year-olds: Beginners Computational Thinking Test: Adaptation pour un public de moins de 5 ans [234], which has proved very reliable for the assessment of CT.

Large-scale research is also being carried out, for example, in schools in Italy using the BCTt, through a project funded by the Italian Ministry of Education, Universities and Research. In Switzerland, the Ecole Polytechnique Fédérale de Lausanne (EPFL) included the BCTt on the Roteco¹¹ website, which brings together the following institutions: Swiss Academies of Sciences, SUPSI, the EPFL, the ETHZ and schools St. Gallen, Lucerne, Schwyz, and Valais. There is also an initial collaboration with professors and researchers from China and the Ministry of Education of Singapore, as well as initial collaborations with Cornell University (USA), University of Hradec Kralove (Czech Republic), University of Tübingen (Germany), Vilnius University (Lithuania), and: Université Lumière Lyon 2 (ISPEF) (France), among others.

In addition, for this multi-population research, the BCTt has been translated into several languages, e.g., Italian, Dutch, Portuguese, German, French, Chinese, Farsi and other languages.

8.8. Blue Ant Code (BAC)

In the context of the thesis, a learning and assessment environment through learning analytics, has been developed. The development of the BAC environment includes a video game compatible with multiple platforms, a REST api for the connection with the relational database, and complementary modules for filtering, visualisation and data analysis with GLA. This environment was used in the case study E3: Validation and analysis of the educational environment based on the BAC game, in individual and collaborative modes and analysis of CT skills in Primary Education (see chapter 6). In addition, a second version of BAC was developed adapted to a non-school setting, including new features such as the inclusion of a rewards system, which was used in E4: Analysis of competences: intrinsic motivation, persistence and rewards through learning analytics with BAC (see chapter 7).

Research on how to teach CT and especially on the computational practices and perspectives dimensions of the 3D framework is scarce, so in this thesis we propose a learning system combining three of the most used learning strategies in the BAC environment, this being a collaborative game-based environment built on a constructivist problem-based strategy, which has been validated at the Primary Education stage in the E3 case study. The results of the research show that this approach is suitable for CT learning, especially at early ages, covering the three key dimensions of the 3D framework (see Figure 49), as posed in hypothesis HE2, and confirmed by the case study. The combination of different assessment methodologies, such as a system of assessments [148]: BCTt, GLA through GBL, and qualitative data (direct observation and questionnaires), allows for a three-dimensional assessment, as hypothesised in HE4, and confirmed by this case study (see Figure 50)

Regarding the validation of BAC in Primary Education, the results of the E3 case study confirmed BAC as an appropriate learning environment, with very significant improvements in CT development even with little interaction time with the environment. Moreover, it was also suitable as an assessment tool with results consistent with the analysis of the data collected by BAC and, in addition, it can provide real-time data that can be useful for detailed personal assessment, strong analysis of low-level data, and the identification of age-specific unachievable computational concepts not only at a specific moment in time, but throughout and at the end of a specific learning process. However, as was the case with BCTt, BAC seems to be better adapted to younger learners and even suitable for lower grades (Early Childhood Education), especially in collaborative mode. Moreover, BAC is especially suitable and motivating for students with special needs or low percentile especially in the collaborative play mode

Positive results were also obtained in the learning retention test, suggesting that this game-based methodology promotes long-lasting learning. Learning retention results were better in the collaborative mode than in the individual mode, especially in the lower grades, leading to the conclusion that the collaborative game mode promotes more effective and long-lasting learning. However, it is possible that the collaborative strategy may not be as effective for upper primary school students, as these students seem to be capable of more effective individual reflection. Further research is needed to test this hypothesis.

One of the most relevant findings is that students in the first years of Primary Education outperform the CT skills of students in higher grades at the end of the study, thus highlighting in this thesis the importance of developing CT from a very early age and the recommendation to incorporate certain concepts, such as nested loops or conditionals, in the school curricula already in the first year or even earlier, instead of waiting until the last year of Primary (or even secondary) Education, as it is currently the case in most schools in Spain.

The case study E4, mass deployment of BAC in a non-school setting, was carried out with the general aim of analysing strategies for the learning and assessment of CT, and identifying CT skills of Primary School students, with a special focus on dispositions and predispositions for CT based on intrinsic motivation and voluntary play. For this study, the adapted BAC mass deployment environment was used. The main findings of this study are summarised below:

- Initial interest: A priori there seems to be a greater interest in learning to program in boys than in girls, since boys play more often and spend more time playing until the age of 5; however, from the age of 6 onwards, the time spent playing and the number of games is much greater in girls than in boys.

- Performance: At all ages is significantly higher in girls than in boys, especially from 3 to 5 years old. However, from 6 to 8 years of age, although their performance is still better, girls spend much more time playing and, according to the results of the research carried out, this behaviour is not related to the mastery of the concepts addressed but is due to a persistence-reward behaviour.

- Specific computational problem solving: Girls have a higher interest and performance in the more difficult levels (nested loops and long sequences), compared to boys, who have a similar interest in each concept, but higher performance only in simple, medium-difficulty loops. Simple sequences seem to be mastered from the age of 3 years onwards, regardless of gender. However, due to working memory, long sequences are not mastered until the age of 5 years, regardless of gender. Simple loops are mastered around the age of 3 years for boys and 4 years for girls. On the other hand, nested loops seem to be mastered from the age of 5 years for girls and 6 years for boys. Finally, as far as conditionals are concerned, performance increases at the age of 4, regardless of gender. All the concepts addressed can be mastered between the ages of 3 and 6, which could be an aspect to be considered in the design of school curricula.

- Learning progression: Both girls and boys make learning progress, but boys' improvement is more significant in both time and performance. However, girls achieve significantly more than boys and reach full mastery at all levels in the first 10 attempts between the ages of 3 and 5, while this behaviour is not observed for boys. From the age of 6 years onwards, girls take more time and attempts to solve the challenges due to persistence-reward behaviour. It is noteworthy that, although nested loops seem to be difficult for primary school pupils, it is possible to fully master this concept by the age of 4, regardless of gender, if enough attempts are made and, in addition, children can persist with the challenge voluntarily, intrinsically motivated, until they master it. This result contrasts with that obtained in the E2 case study, in which many of these concepts were not attainable at age 4, thus confirming the importance of intrinsic motivation in learning processes. This could also be an aspect to consider for the development of school curricula, since all the concepts addressed (sequences, simple and nested loops, and conditionals) could be mastered at the age of 4 if there is intrinsically motivated persistence.

- Persistence-challenge and persistence-reward behaviours: Persistence-challenge (repetition despite losing) is clearly observed in both genders, but is more pronounced in boys, which may explain why their progress is highly correlated with persistence. In contrast, persistence-reward behaviour (repetition after winning) is observed significantly more in girls, especially between the ages of 5 and 7, when they have mastered all the concepts addressed. This is a relevant finding of our research because, although learning derived from this behaviour is possible, this learning could be reduced as the aim is not to overcome the challenge but to obtain rewards. Given that children are able to master the concepts addressed before the age of 5 through continued practice, we wonder whether it is not counterproductive to delay the acquisition of these concepts until after the age of 5, when persistence-challenge behaviour is lower and persistence-reward behaviour emerges, especially in girls. On the other hand, from the age of 5 onwards, motivation through rewards could also be suggested, especially in girls, but always bearing in mind not to exceed the limit at which rewards can be detrimental to learning, diminishing intrinsic motivation.

The potential applications derived from the research results are based on the recommendation to analyse aspects of persistence and rewards in educational games. The BAC environment could be used in school contexts, to develop and assess CT, as well as to analyse aspects related to intrinsic motivation, but, in this case, provided that its use is completely voluntary. In the same direction, it is recommended to incorporate educational games based on intrinsic motivation in school settings. School curricula could be designed considering that, through this motivation, children may be able to overcome computational concepts (including nested loops) before the age of 5, there being gender differences in the acquisition of these concepts. It is also recommended to design educational games with these aspects in mind, and in this way to enhance persistence-challenge behaviour in boys and, from the age of 5, to control or enhance persistence-reward behaviour with respect to girls. We believe that the findings could guide the design of better tools for learning programming and CT development. Since age and gender impact performance on these challenges, it is advisable to adapt learning tools accordingly, as well as the school curricula. In addition, learning environments could adapt the way reinforcement and rewards are provided, especially for boys in the more complex challenges and for girls from the age of 5. However, further studies are needed to take these complexities into account.

8.9. CT learning: current and future directions

New BAC modules are currently being developed to explore the learning and assessment of other CT concepts and skills, as well as data analysis and visualisation modules with GLA, in addition to the existing ones (section 6.4.3). Some of the projects currently under development are the following:

- Adaptive Sudoku game module: the aim is to develop problem-solving skills through decomposition by solving Sudokus. To this end, several levels of Sudoku games of increasing difficulty have been developed at as a playable and validated prototype (see Figure 74, left-hand side) [235].

- Code encryption and decryption game module: a playable prototype with two levels of difficulty has also been developed for future implementation (see Figure 74, right-hand side) [236].

- Module for automatic data analysis in terms of optimal paths in BAC, player progression and statistical evaluation This module is being developed in C# with the Visual Studio Code IDE.

Some of the future research lines are:

- Extending the BAC game environment with more computational concepts.

- Integration and evaluation of learning environments based on intrinsic motivation in school contexts, through new case studies

- Analysis of the code proposed by students in terms of optimality and programming styles using GLA.

- Extension of BAC to include findings on gaming behaviours.

- Extension of BAC to make use of adaptive GLA without reducing intrinsic motivation.

In addition, for the evaluation of new learning methodologies, a collaboration has been initiated with Harvard University to guide the creation of a video game for learning and assessing CT in secondary school students.

In general, the aim is to continue research and international collaboration for the creation of learning and assessment instruments and tools for the development of CT in early stages, with special attention to diversity.

8.10. Publications

This thesis has resulted in publications in journal papers, conference papers and a book chapter. In this section, the most relevant publications and actions are detailed below.

8.10.1. JCR journals (Q1 y Q2)

- M. Zapata-Caceres, E. Martin, and M. Roman-Gonzalez, "Collaborative Game-Based Environment and Assessment Tool for Learning Computational Thinking in Primary School: A Case Study," IEEE Transactions on Learning Technologies, 2021.

- M. Zapata-Cáceres and E. Martín-Barroso, "Applying Game Learning Analytics to a Voluntary Video Game: Intrinsic Motivation, Persistence, and Rewards in Learning to Program at an Early Age," IEEE Access, vol. 9, pp. 123588-123602, 2021.

- L. El-Hamamsy, M. Zapata-Caceres, E. Martín, F. Mondada, J. Dehler Zufferey, and B. Bruno, "The competent Computational Thinking test (cCTt): Development and validation of an unplugged Computational Thinking test for upper primary school," Journal of Educational Computing Research, 2022. In press.

8.10.2. International congresses

- M. Zapata-Cáceres, E. Martín-Barroso, and M. Román-González, “Computational thinking test for beginners: Design and content validation,” in IEEE Global Engineering Education Conference (EDUCON’20), Porto, Portugal, pp. 1905–1914, 2020, doi: 10.1109/EDUCON45650.2020.9125368.

- M. Zapata-Cáceres and N. Fanchamps, "Using the beginners computational thinking test to measure development on computational concepts among preschoolers," in Proceedings of the 5th APSCE International Computational Thinking and STEM in Education Conference 2021. Singapore: National Institute of Education, 2021, pp. 3237. Awarded as “Best full paper”.

- L. El-Hamamsy, M. Zapata-Caceres, P. Marcelino, J. Dehler Zufferey, B. Bruno, and E. Martin, “Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: the Beginners' CT test (BCTt) and the competent CT test (cCTt)” in ITiCSE 2022, 27th Annual Conference on Innovation and Technology in Computer Science Education. Under peer review.

- M. Zapata-Caceres, N. Fanchamps, I. H. Yeter, P. Marcelino, and E. Martín-Barroso, “Understanding Teachers’ Attitudes and Self-Assessment Towards Computational Thinking” in CTE-STEM 2022, Sixth APSCE International Conference on Computational Thinking and STEM Education. Work in progress.

8.10.3. National congresses

- Poster presentation “Computational Thinking at early stages: learning assessment tolos and school curricula approach”, in II Congreso de la Escuela Internacional de Doctorado de la Universidad Rey Juan Carlos, Nov. 2019.

- Presentation “Videojuegos para enseñar a programar: analíticas y motivación” in Congreso CE + Ed Educare + Educere de Formación del Profesorado, 2021.

8.10.4. Book chapters

- Zapata-Cáceres María, E. Martín-Barroso and M. Román-González, "BCTt: Beginners computational thinking test," in Raspberry Pi Foundation Research Seminars, Apr 2021, pp. 46-56. ISSN 2514-586X (16). Available: https://www.raspberrypi.org/app/uploads/2021/05/Understanding-computing-education-Volume-1-%E2%80%93-Raspberry-Pi-Foundation-Research-Seminars.pdf.

8.10.5. Dissemination

- Presentation “Experiencias de enseñanza de conceptos computacionales en Educación Primaria a través de un videojuego en modos colaborativo e individual” in las IV Jornadas de Innovación Educativa iEDU - CRIF Las Acacias, May 2019.

- Presentation “Experiencia de enseñanza y evaluación del pensamiento computacional en Educación Primaria a través de un videojuego colaborativo”, in SIMO Educación, Nov. 2019.

- Presentation y discussion panel Q&A “Computational thinking test for beginners” in the online seminars on computing education research from the Raspberry Pi Foundation, Cambridge UK, Julio 2020.

- Workshop “Recursos Tecnológicos en Educación Infantil, Primaria y NEE”, in XIX Semana de la Ciencia y de la Innovación de Madrid, Universidad Rey Juan Carlos, Nov. 2019.

- Workshop “Aprende a programar jugando a videojuegos con amigos”, in XIX Semana de la Ciencia y de la Innovación de Madrid, Universidad Rey Juan Carlos, Nov. 2019.

- Workshop “Cómo aprender a programar jugando”, in XI Noche Europea de los Investigadores de Madrid, Universidad Rey Juan Carlos, Nov. 2020.

- Workshop “Aprende a programar jugando a videojuegos”, in XI Noche Europea de los Investigadores de Madrid, Universidad Rey Juan Carlos, Nov. 2020.

Bibliografía

  [1]V. J. Shute, C. Sun, and J. Asbell-Clarke, "Demystifying computational thinking," Educational Research Review, vol. 22, pp. 142-158, 2017. Available: https://www.sciencedirect.com/science/article/pii/S1747938X17300350, doi: 10.1016/j.edurev.2017.09.003.

  [2]M. Berland and U. Wilensky, "Comparing virtual and physical robotics environments for supporting complex systems and computational thinking," Journal of Science Education and Technology, vol. 24, no. 5, pp. 628-647, 2015.

  [3]J. M. Wing, "Computational thinking and thinking about computing," Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 366, no. 1881, pp. 3717-3725, 2008. Available: http://rsta.royalsocietypublishing.org/content/366/1881/3717.abstract, doi: 10.1098/rsta.2008.0118.

  [4]T. Hsu, S. Chang, and Y. Hung, "How to learn and how to teach computational thinking: Suggestions based on a review of the literature," Computers & Education, vol. 126, pp. 296-310, 2018. Available: http://www.sciencedirect.com/science/article/pii/S0360131518301799, doi: //doi.org/10.1016/j.compedu.2018.07.004.

  [5]S. Grover and R. Pea, "Computational Thinking in K—12: A Review of the State of the Field," Educational Researcher, vol. 42, no. 1, pp. 38-43, 2013. Available: https://www.jstor.org/stable/23360476, doi: 10.3102/0013189X12463051.

  [6]M. Zapata-Cáceres and N. Fanchamps, "Using the beginners computational thinking test to measure development on computational concepts among preschoolers," in Proceedings of the 5th APSCE International Computational Thinking and STEM in Education Conference 2021, Singapore, National Institute of Education, 2021, pp. 32-37.

  [7]Zapata-Cáceres María, E. Martín-Barroso and M. Román-González, "BCTt: Beginners computational thinking test," in Raspberry Pi Foundation Research Seminars, Apr 2021, pp. 46-56. ISSN 2514-586X (16). Available: https://www.raspberrypi.org/app/uploads/2021/05/Understanding-computing-education-Volume-1-%E2%80%93-Raspberry-Pi-Foundation-Research-Seminars.pdf.

  [8]M. Zapata-Cáceres, E. Martín-Barroso, and M. Román-González, “Computational thinking test for beginners: Design and content validation,” in IEEE Global Engineering Education Conference (EDUCON’20), Porto, Portugal, pp. 1905–1914, 2020, doi: 10.1109/EDUCON45650.2020.9125368.

  [9]J. Kirriemuir and A. Mcfarlane, Literature Review in Games and Learning. Bristol: Futurelab, 2004. Available: https://telearn.archives-ouvertes.fr/hal-00190453.

 [10]M. Prensky, "Digital game-based learning," Computers in Entertainment (CIE), vol. 1, no. 1, pp. 21, 2003.

 [11]M. Qian and K. R. Clark, "Game-based Learning and 21st century skills: A review of recent research," Computers in Human Behavior, vol. 63, pp. 50-58, 2016. Available: https://www.sciencedirect.com/science/article/pii/S0747563216303491, doi: 10.1016/j.chb.2016.05.023.

 [12]L. S. Vygotsky, Mind in Society: The Development of Higher Psychological Processes. Cambridge, MA, USA: Harvard University Press, 1978.

 [13]J. Piaget, Psychology and Epistemology: Towards a Theory of Knowledge. CA, USA: Allen Lane, 1972.

 [14]D. H. Jonassen and L. Rohrer-Murphy, “Activity theory as a framework for designing constructivist learning environments,” Educational Technology Research & Development, vol. 47, no. 1, pp. 61–79, 1999. Available: https://www.jstor.org/stable/30220230, doi: 10.1007/BF02299477.

 [15]D. H. Jonassen, Learning to Solve Problems with Technology: A Constructivist Perspective. Upper Saddle River, N.J.: Merrill Prentice Hall, 2003.

 [16]C. Alonso-Fernandez et al, "Systematizing game learning analytics for serious games," in 2017 IEEE global engineering education conference (EDUCON), Apr 2017, pp. 1111-1118.

 [17]M. Freire et al, "Game Learning Analytics: Learning Analytics for Serious Games," Springer Nature Switzerland AG, pp. 1-29, Dec 1, 2016.

 [18]M. Zapata-Caceres, E. Martin, and M. Roman-Gonzalez, "Collaborative GameBased Environment and Assessment Tool for Learning Computational Thinking in Primary School: A Case Study," in IEEE Transactions on Learning Technologies, vol. 14, no. 5, pp. 576-589, 1 Oct. 2021, doi: 10.1109/TLT.2021.3111108.

 [19]M. Zapata-Cáceres and E. Martín-Barroso, "Applying Game Learning Analytics to a Voluntary Video Game: Intrinsic Motivation, Persistence, and Rewards in Learning to Program at an Early Age," in IEEE Access, vol. 9, pp. 123588-123602, 2021, doi: 10.1109/ACCESS.2021.3110475.

 [20]M. Binkley et al, "Defining twenty-first century skills," in Assessment and Teaching of 21st Century Skill. Dordrecht: Springer Netherlands, 2012, pp. 17-66.

 [21]B. Trilling and C. Fadel, 21st Century Skills: Learning for life in our times. San Francisco: John Wiley & Sons, 2009.

 [22]H. Kim et al, "Enhancing teachers' ICT capacity for the 21st century learning environment: Three cases of teacher education in Korea," Australasian Journal of Educational Technology, vol. 28, no. 6, 2012.

 [23]K. Tang, T. Chou, and C. Tsai, "A Content Analysis of Computational Thinking Research: An International Publication Trends and Research Typology," The Asia-Pacific Education Researcher, pp. 1-11, 2019. Available: https://search.proquest.com/docview/2195733480, doi: 10.1007/s40299-019-00442-8.

 [24]S. Grover et al, "A Framework for Using Hypothesis-Driven Approaches to Support Data-Driven Learning Analytics in Measuring Computational Thinking in Block-Based Programming Environments," ACM Transactions on Computing Education (TOCE), vol. 17, no. 3, pp. 1-25, 2017. Available: http://dl.acm.org/citation.cfm?id=3105910, doi: 10.1145/3105910.

 [25]S. Grover and A. Korhonen, "Unlocking the Potential of Learning Analytics in Computing Education," ACM Transactions on Computing Education (TOCE), vol. 17, no. 3, pp. 1-4, 2017. Available: http://dl.acm.org/citation.cfm?id=3122773, doi: 10.1145/3122773.

 [26]S. Grover. “The 5th ‘C’ of 21st Century skills? Try computational thinking (not coding)”, edsurge.com. http://bit.ly/346qK78 (accessed Feb. 1, 2018). Available: http://scholar.aci.info/view/1544f0f08b300170003/161ce93d34000012ef80942.

 [27]J. A. Velazquez-Iturbide, "Towards an analysis of computational thinking," in Sep 2018, pp. 1-6.

 [28]J.A. Velázquez-Iturbide, “Towards an analysis of computational thinking,” in Int. Symp. on Computers in Education (SIIE’18), pp. 1-6, Sep. 2018.

 [29]M. Israel et al, "Supporting all learners in school-wide computational thinking: A cross-case qualitative analysis," Computers & Education, vol. 82, pp. 263-279, 2015. Available: https://www.sciencedirect.com/science/article/pii/S0360131514002759, doi: 10.1016/j.compedu.2014.11.022.

 [30]C. Angevine et al, "Computational Thinking for a Computational World," 2017. Available: www.digitalpromise.org.

 [31]National Research Council, Report of a Workshop on the Pedagogical Aspects of Computational Thinking. National Academies Press, 2011.

 [32]Q. Burke, W. I. O'Byrne and Y. B. Kafai, "Computational participation: Understanding coding as an extension of literacy instruction," Journal of Adolescent & Adult Literacy, vol. 59, no. 4, pp. 371-375, 2016.

 [33]J. F. Sanford and J. T. Naidu, "Computational thinking concepts for grade school," Contemporary Issues in Education Research (CIER), vol. 9, no. 1, pp. 23-32, 2016.

 [34]K. Mills et al, Computational Thinking for an Inclusive World: A Resource for Educators to Learn and Lead. Digital Promise, 2021. Available: https://doi.org/10.51388/20.500.12265/140.

 [35]C. Sneider et al, "Computational thinking in high school science classrooms," The Science Teacher, vol. 81, no. 5, pp. 53, 2014.

 [36]N. Kelly and J. S. Gero, "Design thinking and computational thinking: a dual process model for addressing design problems," Design Science, vol. 7, 2021.

 [37]T. C. Pargman et al, "Teaching Computational Thinking in K-9: Tensions at the Intersection of Technology and Pedagogical Knowledge," in 14th International Conference of the Learning Sciences (ICLS) 2020, vol. 4, pp. 2409-2410, International Society of the Learning Sciences, 2020.

 [38]D. Kohen-Vacs and M. Milrad, "Computational thinking education for in-service elementary swedish teachers: Their perceptions and implications for competence development," in International Conference on Computational Thinking Education, 13-15 June 2019, Hong Kong, 2019.

 [39]V. Allan et al, "Computational thinking in high school courses," in Proceedings of the 41st ACM Technical Symposium on Computer Science Education, 2010.

 [40]P. Sengupta et al, "Integrating computational thinking with K-12 science education using agent-based computation: A theoretical framework," Education and Information Technologies, 2013, vol. 18, no 2, p. 351-380. Available: https://search.proquest.com/docview/1323979287, doi: 10.1007/s10639-012-9240-x.

 [41]V. Barr and C. Stephenson, "Bringing computational thinking to K-12: What is Involved and what is the role of the computer science education community?" ACM Inroads, vol. 2, no. 1, pp. 48-54, 2011.

 [42]P. Mishra, A. Yadav, and Deep-Play Research Group, "Rethinking technology & creativity in the 21st century," TechTrends, vol. 57, no. 3, pp. 10-14, 2013.

 [43]J. M. Wing, "Computational Thinking Test," CACM Viewpoint, pp. 33-35, 2006. Available: http://www.cs.cmu.edu/~wing/.

 [44]M. Román-González, J. Pérez-González and C. Jiménez-Fernández, "Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test," Computers in Human Behavior, vol. 72, pp. 678-691, 2017. Available: https://www.sciencedirect.com/science/article/pii/S0747563216306185, doi: 10.1016/j.chb.2016.08.047.

 [45]A. V. Aho, "Computation and computational thinking," The Computer Journal, vol. 55, no. 7, pp. 832-835, 2012.

 [46]K. Brennan, M. Resnick, and MIT Media Lab, "New frameworks for studying and assessing the development of computational thinking," American Educational Research Association Meeting, Vancouver, BC, Canada., 2012.

 [47]S. Papert, Mindstorms: Children, Computers, and Powerful Ideas. Basic Books, Inc., 1980.

 [48]J. J. Lu and G. H. Fletcher, "Thinking about computational thinking," in Proceedings of the 40th ACM Technical Symposium on Computer Science Education, 2009.

 [49]J. Wing, "Research notebook: Computational thinking—What and why," The Link Magazine, vol. 6, pp. 20-23, 2011.

 [50]S. Furber, "Shut down or restart? The way forward for computing in UK schools," The Royal Society, 2012. Available: https://royalsociety.org/-/media/education/computing-in-schools/2012-01-12-computing-in-schools.pdf.

 [51]M. Tedre and P. J. Denning, "The long quest for computational thinking," in Proceedings of the 16th Koli Calling International Conference on Computing Education Research, pp. 120-129, 2016.

 [52]J. Moreno et al, " No es lo mismo: un análisis de red de texto sobre definiciones de pensamiento computacional para estudiar su relación con la programación informática," Revista Interuniversitaria De Investigación En Tecnología Educativa, 2019, doi: 10.6018/riite.397151.

 [53]ISTE and CSTA, "Operational definition of computational thinking for K-12 education," 2011. Available: https://cdn.iste.org/www-root/Computational_Thinking_Operational_Definition_ISTE.pdf.

 [54]M. U. Bers et al, "Computational thinking and tinkering: Exploration of an early childhood robotics curriculum," Computers & Education, vol. 72, pp. 145-157, 2014. Available: https://search.datacite.org/works/10.1016/j.compedu.2013.10.020, doi: 10.1016/j.compedu.2013.10.020.

 [55]S. Atmatzidou and S. Demetriadis, "Advancing students’ computational thinking skills through educational robotics: A study on age and gender relevant differences," Robotics and Autonomous Systems, vol. 75, pp. 661-670, 2016. Available: https://www.sciencedirect.com/science/article/pii/S0921889015002420, doi: 10.1016/j.robot.2015.10.008.

 [56]N. D. Anderson, "A call for computational thinking in undergraduate psychology," Psychology Learning & Teaching, vol. 15, no. 3, pp. 226-234, 2016.

 [57]S. Rose, J. Habgood and T. Jay, " An exploration of the role of visual programming tools in the development of young children’s computational thinking," Electronic journal of e-learning, 2017, vol. 15, no 4, p. 297-309, Aug 1, 2017.

 [58]D. Weintrop and U. Wilensky, "Using commutative assessments to compare conceptual understanding in blocks-based and text-based programs," in ICER, pp. 101-110, Aug 9, 2015.

 [59]D. Statter and M. Armoni, "Teaching abstract thinking in introduction to computer science for 7th graders," in Proceedings of the 11th Workshop in Primary and Secondary Computing Education, pp. 80-83, 2016.

 [60]B. Zhong et al, " An Exploration of Three-Dimensional Integrated Assessment for Computational Thinking," Journal of Educational Computing Research, vol. 53, no. 4, pp. 562-590, 2016. Available: https://journals.sagepub.com/doi/full/10.1177/0735633115608444, doi: 10.1177/0735633115608444.

 [61]S. Y. Lye and J. H. L. Koh, "Review on teaching and learning of computational thinking through programming: What is next for K-12?" Computers in Human Behavior, vol. 41, pp. 51-61, 2014. Available: https://www.sciencedirect.com/science/article/pii/S0747563214004634, doi: 10.1016/j.chb.2014.09.012.

 [62]E. Hunsaker, Computational Thinking. The K–12 Educational Technology Handbook, A. Ottenbreit-Leftwich and R. Kimmons, Eds. EdTech Books, 2020, pp. 80–95. [Online]. Available: https://bit.ly/2UP0gZi.

 [63]P. J. Rich et al, "Coding in K-8: International Trends in Teaching Elementary/Primary Computing," TechTrends, pp. 1-19, 2018.

 [64]C. S. González-González, "State of the art in the teaching of computational thinking and programming in childhood education," Education in the Knowledge Society, vol. 20, pp. 1-15, 2019.

 [65]S. Bocconi, A. Chioccariello, G. Dettori, A. Ferrari, and K. Engelhardt, “Developing computational thinking in compulsory education,” Eur. Commission, JRC Sci. Policy Rep., pp. 68, 2016.

 [66]Unión Europea, “Plan de Acción de Educación Digital (2021-2027)”, Unión Europa, European Comission, European Education Area, 2021.[Online]. Available: https://ec.europa.eu/jrc/en/digcomp/digital-competence-framework. [Accessed: Dic. 12, 2021].

 [67]Unión Europea, “DigComp. The Digital Competence Framework 2.0”, EU SCIENCE HUB, 2021. [Online]. Available: https://ec.europa.eu/jrc/en/digcomp/digital-competence-framework. [Accessed: Dic. 12, 2021].

 [68]Code.org, “Learn computer science. Change the world”, Code.org. [Online]. Available: https://code.org/. [Accessed: Jan. 20, 2022].

 [69]La hora del código, “Celebramos las Ciencias de la Computación en todas partes”, hourofcode.com. [Online]. Available: https://hourofcode.com/es. [Accessed: Jan. 20, 2022].

 [70]J. Á Velázquez-Iturbide, "Report of the Spanish computing scientific society on computing education in pre-university stages," in Proceedings of the Sixth International Conference on Technological Ecosystems for Enhancing Multiculturality, 2018.

 [71]Instituto Nacional de Tecnologías Educativas y de Formación del Profesorado (INTEF), "Programación, robótica y Pensamiento Computacional en el aula. Situación en España y propuesta normativa", Ministerio de Educación y Formación Profesional, Gobierno de España, Oct, 2018. [Online]. Available: https://code.intef.es/wp-content/uploads/2018/10/Ponencia-sobre-Pensamiento-Computacional.-Informe-Final.pdf.

 [72]Junta de Castilla y León, “Proyecto de innovación 'TIC STEAM' - Curso 2016-2017”, educacyl Portal de Educación, 2017. [Online]. Available: https://www.educa.jcyl.es/profesorado/es/formacion-profesorado/proyectos-relacionados-formacion-permanente-profesorado/competencia-digital/convocatorias/proyecto-innovacion-tic-steam-curso-2016-2017.

 [73]Instituto Nacional de Tecnologías Educativas y de Formación del Profesorado (INTEF), “Escuela de Pensamiento Computacional e Inteligencia Artificial”, Ministerio de Educación y Formación Profesional, Gobierno de España, 2021. [Online]. Available: https://intef.es/tecnologia-educativa/pensamiento-computacional/.

 [74]Programamos, “Es necesario que todo el mundo desarrolle su Pensamiento Computacional, desde niños a mayores”, programamos.com. [Online]. Available: https://programamos.es/. [Accessed: Jan. 20, 2022].

 [75]Boletín Oficial del Estado, “Marco de referencia de la competencia digital docente”, Ministerio de la Presidencia, Relaciones con las Cortes y Memoria Democrática, Gobierno de España. [Online]. Available: https://www.boe.es/diario_boe/txt.php?id=BOE-A-2020-7775#top.

 [76]E. Hunsaker, Integrating computational thinking. K-12 technology integration. Pressbooks. Available: https://k12techintegration.pressbooks.com/chapter/integrating-computational-thinking, 2018.

 [77]M. Zapata-Ros, "Computational thinking unplugged," Education in the Knowledge Society, vol. 20, pp. 1-29, 2019.

 [78]T. Bell et al, "Computer science unplugged: School students doing real computing without computers," The New Zealand Journal of Applied Computing and Information Technology, vol. 13, no. 1, pp. 20-29, 2009.

 [79]S. Adak, "Effectiveness of Constructivist Approach on Academic Achievement in Science at Secondary Level." Educational Research and Reviews, vol. 12, no. 22, pp. 1074-1079, 2017.

 [80]V. Richardson, "Constructivist pedagogy," Teachers College Record, vol. 105, no. 9, pp. 1623-1640, 2003.

 [81]C. T. Fosnot and R. S. Perry, "Constructivism: A psychological theory of learning," Constructivism: Theory, Perspectives, and Practice, vol. 2, no. 1, pp. 8-33, 1996.

 [82]S. Papert and I. Harel, "Situating constructionism," Constructionism, vol. 36, no. 2, pp. 1-11, 1991.

 [83]C. W. Istikomah and C. Budiyanto, "The contribution of educational robotics and constructivist approach to computational thinking in the 21st century," in Proceedings of the the 1st International Conference on Computer Science and Engineering Technology Universitas Muria Kudus, 2018.

 [84]S. Hartikainen et al, "The concept of active learning and the measurement of learning outcomes: A review of research in engineering higher education," Education Sciences, vol. 9, no. 4, pp. 276, 2019.

 [85]R. T. Johnson and D. W. Johnson, "Active learning: Cooperation in the classroom," The Annual Report of Educational Psychology in Japan, vol. 47, pp. 29-30, 2008.

 [86]P. Dillenbourg, Ed., Collaborative Learning: Cognitive and Computational Approaches. Oxford, U.K.: Pergamon, 1999.

 [87]M. Laal and S. M. Ghodsi, "Benefits of collaborative learning," Procedia - Social and Behavioral Sciences, vol. 31, pp. 486-490, 2012. Available: http://www.sciencedirect.com/science/article/pii/S1877042811030205, doi: //doi.org/10.1016/j.sbspro.2011.12.091.

 [88]D. W. Johnson and R. T. Johnson, "Effects of cooperative and individualistic learning experiences on interethnic interaction," Journal of Educational Psychology, vol. 73, (3), pp. 444-449, 1981. Available: https://search.proquest.com/docview/614371667. DOI: 10.1037/00220663.73.3.444.

 [89]M. Laal and M. Laal, "Collaborative learning: what is it?" Procedia - Social and Behavioral Sciences, vol. 31, pp. 491-495, 2012. Available: http://www.sciencedirect.com/science/article/pii/S1877042811030217. DOI: //doi.org/10.1016/j.sbspro.2011.12.092.

 [90]D. Boud and G. E. Feletti, Eds., The Challenge of Problem-Based Learning. London, U.K.: Routledge, 1997, doi: 10.4324/9781315042039.

 [91]J. Roschelle and S. D. Teasley, "The construction of shared knowledge in collaborative problem solving," in Computer Supported Collaborative Learning, 1995.

 [92]P. C. Blumenfeld et al, "Motivating Project-Based Learning: Sustaining the Doing, Supporting the Learning," Educational Psychologist, vol. 26, no. 3-4, pp. 369-398, 1991. Available: http://www.tandfonline.com/doi/abs/10.1080/00461520.1991.9653139, doi: 10.1080/00461520.1991.9653139.

 [93]L. Zhang and J. Nouri, "A systematic review of learning computational thinking through Scratch in K-9," Comput. Educ., vol. 141, pp. 103-607, 2019.

 [94]J. Moreno-León, G. Robles, and M. Román-González, "Towards data-driven learning paths to develop computational thinking with scratch," IEEE Transactions on Emerging Topics in Computing, vol. 8, no. 1, pp. 193-205, 2017.

 [95]J. Nouri et al, "Development of computational thinking, digital competence and 21st century skills when learning programming in K-9," Education Inquiry, vol. 11, no. 1, pp. 1-17, 2020.

 [96]J. Fagerlund et al, "Computational thinking in programming with scratch in primary schools: A systematic review," Computer Applications in Engineering Education, vol. 29, no. 1, pp. 12-28, 2021.

 [97]G. Falloon, "An analysis of young students' thinking when completing basic coding tasks using Scratch Jnr. On the iPad," J. Comput. Assisted Learn., vol. 32, no. 6, pp. 576-593, 2016.

 [98]Massachusetts Institute of Technology: MIT, “Scratch”, scratch.mit.edu, [Online]. Available: https://scratch.mit.edu/.

 [99]P. Mozelius and L. Öberg, "Play‐based learning for programming education in primary school: The östersund model," in European Conference on E-Learning-ECEL 2017, Porto, Portugal, 26-27 October 2017, 2017.

[100]A. G. Soosai Raj et al, "Does native language play a role in learning a programming language?" in Proceedings of the 49th ACM Technical Symposium on Computer Science Education, 2018.

[101]M. Jancheski, "Improving teaching and learning computer programming in schools through educational software," Olympiads in Informatics, vol. 11, pp. 55-75, 2017.

[102]E. F. Silva, B. J. Dembogurski and G. S. Semaan, "A Systematic Review of Computational Thinking in Early Ages," arXiv Preprint arXiv:2106.10275, 2021.

[103]J. M. Cañas Plaza et al, "Entorno docente universitario para la programación de los robots," Revista Iberoamericana De Automática E Informática Industrial, vol. 15, no. 4, pp. 404-415, 2018.

[104]A. Hamidi, M. Milrad and S. Tavajoh, "Combining maker technologies to promote computational thinking and heart-ware skills through project-based activities: Design considerations and empirical outputs," in Cte-Stem 2021, 2021.

[105]R. Zerega et al, "A co-design approach for developing computational thinking skills in connection to STEM related curriculum in swedish schools," in Cte-Stem 2021, 2021.

[106]M. Riedmiller and T. Gabel, "On experiences in a complex and competitive gaming domain: Reinforcement learning meets robocup," in 2007 IEEE Symposium on Computational Intelligence and Games, 2007.

[107]N. L. Fanchamps et al, "The influence of SRA programming on algorithmic thinking and self-efficacy using Lego robotics in two types of instruction," International Journal of Technology and Design Education, vol. 31, no. 2, pp. 203-222, 2021.

[108]N. L. Fanchamps et al, "The Impact of SRA-Programming on Computational Thinking in a Visual Oriented Programming Environment," Education and Information Technologies, vol. 26, no. 5, pp. 6479-6498, 2021.

[109]M. Montessori, J. M. Hunt and J. Valsiner, The Montessori Method. Routledge, 2017.

[110]M. Elkin, A. Sullivan, and M. U. Bers, "Programming with the KIBO robotics kit in preschool classrooms," Computers in the Schools, vol. 33, no. 3, pp. 169-186, 2016.

[111]L. L. Román, "Metodología para la enseñanza aprendizaje de la lógica de la programación orientada a objetos," in 5ta. Conferencia Iberoamericana en Sistemas, Cibernética e Informática CISCI, International Institute of Informatics and Systemics, EE. UU, 2007.

[112]J. Urquiza-Fuentes and J. Á Velázquez-Iturbide, "A survey of successful evaluations of program visualization and algorithm animation systems," ACM Transactions on Computing Education (TOCE), vol. 9, no. 2, pp. 1-21, 2009.

[113]L. Seiter and B. Foreman, "Modeling the learning progressions of computational thinking of primary grade students," in Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, 2013, Available: http://doi.acm.org/10.1145/2493394.2493403, doi: 10.1145/2493394.2493403.

[114]S. Grover, R. Pea and S. Cooper, "Designing for deeper learning in a blended computer science course for middle school students," Computer Science Education, vol. 25, no. 2, pp. 199-237, 2015.

[115]A. Basawapatna et al, "Recognizing computational thinking patterns," in Proceedings of the 42nd ACM technical symposium on Computer science education, pp. 245-250 Mar 9, 2011.

[116]S. Papadakis and M. Kalogiannakis, "Evaluating a course for teaching advanced programming concepts with scratch to preservice kindergarten teachers: A case study in greece," in Early Childhood EducationAnonymous 2018.

[117]A. Eguiluz et al, "Exploring the progression of early programmers in a set of computational thinking challenges via clickstream analysis," Tetc, pp. 1, 2017. Available: https://ieeexplore.ieee.org/document/8093668, doi: 10.1109/TETC.2017.2768550.

[118]G. Petri and C. Gresse von Wangenheim, "How games for computing education are evaluated? A systematic literature review," Computers & Education, vol. 107, pp. 68-90, 2017. Available: http://dx.doi.org/10.1016/j.compedu.2017.01.004, doi: 10.1016/j.compedu.2017.01.004.

[119]A. Vahldick, A. J. Mendes and M. J. Marcelino, "Dynamic Difficulty Adjustment through a Learning Analytics Model in a Casual Serious Game for Computer Programming Learning," EAI Endorsed Transactions on Serious Games, vol. 4, no. 13, 2017.

[120]C. Kroustalli and S. Xinogalos, "Studying the effects of teaching programming to lower secondary school students with a serious game: a case study with Python and CodeCombat," Education and Information Technologies, pp. 1-27, 2021.

[121]M. Csikszentmihalyi, Finding Flow: The Psychology of Engagement with Everyday Life. United States: 1997.

[122]S. H. Chuah, "Why and who will adopt extended reality technology? Literature review, synthesis, and future research agenda," Literature Review, Synthesis, and Future Research Agenda, Dec. 13, 2018.

[123]M. Montola, "Exploring the edge of the magic circle: Defining pervasive games," in Proceedings of DAC, 2005.

[124]C. Bonillo, J. Marco, and E. Cerezo, "Developing pervasive games in interactive spaces: the JUGUEMOS toolkit," Multimedia Tools Appl, vol. 78, no. 22, pp. 32261-32305, 2019.

[125]N. M. Medina et al, "Hacia una propuesta de evaluación heurística de experiencias de juego pervasivas," Revista De La Asociación Interacción Persona Ordenador (AIPO), vol. 2, no. 2, pp. 42-53, 2021.

[126]J. Marco, C. Bonillo, and E. Cerezo, "A tangible interactive space odyssey to support children learning of computer programming," in Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces, 2017.

[127]S. SUKIRMAN et al, "A Strategy of Learning Computational Thinking through Game Based in Virtual Reality: Systematic Review and Conceptual Framework," Informatics in Education, 2021.

[128]Y. Ching, Y. Hsu and S. Baldwin, "Developing computational thinking with educational technologies for young learners," TechTrends, vol. 62, no. 6, pp. 563-573, 2018.

[129]C. P. Brackmann et al, "Development of computational thinking skills through unplugged activities in primary school," in Proceedings of the 12th Workshop on Primary and Secondary Computing Education, 2017.

[130]S. Suh, M. Lee and G. Xia, "Coding strip: A pedagogical tool for teaching and learning programming concepts through comics," in 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2020.

[131]G. Aranda and J. P. Ferguson, "Unplugged Programming: The future of teaching computational thinking?" Pedagogika, vol. 68, no. 3, 2018.

[132]J. Waite et al, "Unplugged computing and semantic waves: Analysing crazy characters," in Proceedings of the 1st UK & Ireland Computing Education Research Conference, 2019.

[133]A. Dengel, "Computer science replugged: What is the use of virtual reality in computer science education?" in Proceedings of the 14th Workshop in Primary and Secondary Computing Education, 2019.

[134]X. Tang et al, "Assessing computational thinking: A systematic review of empirical studies," Computers & Education, vol. 148, pp. 103-798, 2020. Available: http://www.sciencedirect.com/science/article/pii/S0360131519303483, doi: https://doi.org/10.1016/j.compedu.2019.103798.

[135]J. Denner, L. Werner and E. Ortiz, "Computer games created by middle school girls: Can they be used to measure understanding of computer science concepts?" Computers & Education, vol. 58, no. 1, pp. 240-249, 2012. Available: https://www.sciencedirect.com/science/article/pii/S0360131511001849, doi: 10.1016/j.compedu.2011.08.006.

[136]I. Cetin and E. Dubinsky, "Reflective abstraction in computational thinking," Journal of Mathematical Behavior, vol. 47, pp. 70-80, 2017. Available: https://www.sciencedirect.com/science/article/pii/S0732312316301511, doi: 10.1016/j.jmathb.2017.06.004.

[137]S. Jun et al, "Assessing the computational literacy of elementary students on a national level in Korea," Educational Assessment, Evaluation and Accountability, vol. 26, no. 4, pp. 319-332, 2014.

[138]L. Werner et al, "The fairy performance assessment," in Proceedings of the 43rd ACM technical symposium on Computer Science Education, pp. 215-220, 2012.

[139]V. Garneli and K. Chorianopoulos, "Programming video games and simulations in science education: exploring computational thinking through code analysis," Interactive Learning Environments, vol. 26, no. 3, pp. 386-401, 2018. Available: http://www.tandfonline.com/doi/abs/10.1080/10494820.2017.1337036, doi: 10.1080/10494820.2017.1337036.

[140]A. Mühling, A. Ruf and P. Hubwieser, "Design and first results of a psychometric test for measuring basic programming abilities," in Proceedings of the workshop in primary and secondary computing education, pp. 2-10, Nov 9, 2015.

[141]Ö Korkmaz, R. Çakir and M. Y. Özden, "A validity and reliability study of the computational thinking scales (CTS)," Computers in Human Behavior, vol. 72, pp. 558-569, 2017. Available: https://www.sciencedirect.com/science/article/pii/S0747563217300055, doi: 10.1016/j.chb.2017.01.005.

[142]M. Román-González et al, "Extending the nomological network of computational thinking with non-cognitive factors," Computers in Human Behavior, vol. 80, pp. 441-459, 2018. Available: http://www.sciencedirect.com/science/article/pii/S0747563217305563, doi: https://doi.org/10.1016/j.chb.2017.09.030.

[143]M. Román-González et al, "Can computational talent be detected? Predictive validity of the Computational Thinking Test," International Journal of ChildComputer Interaction, vol. 18, pp. 47-58, 2018. Available: http://www.sciencedirect.com/science/article/pii/S221286891730034X, doi: https://doi.org/10.1016/j.ijcci.2018.06.004.

[144]AERA, APA and NCME, Standards for Educational and Psychological Testing. 2014. Available: http://www.fachportal-paedagogik.de/fis_bildung/suche/fis_set.html?FId=1050398.

[145]G. Chen et al, "Assessing elementary students’ computational thinking in everyday reasoning and robotics programming," Computers & Education, vol. 109, pp. 162-175, 2017. Available: http://www.sciencedirect.com/science/article/pii/S0360131517300490, doi: https://doi.org/10.1016/j.compedu.2017.03.001.

[146]E. Relkin, L. de Ruiter and M. U. Bers, "TechCheck: Development and Validation of an Unplugged Assessment of Computational Thinking in Early Childhood Education," Journal of Science Education and Technology, vol. 29, no. 4, pp. 482-498, 2020. Available: https://search.proquest.com/docview/2413230852, doi: 10.1007/s10956-020-09831-x.

[147]Y. Li, S. Xu, and J. Liu, "Development and Validation of Computational Thinking Assessment of Chinese Elementary School Students," Journal of Pacific Rim Psychology, vol. 15, pp. 18344909211010240, 2021. Available: https://doi.org/10.1177/18344909211010240, doi: https://doi.org/10.1177/18344909211010240.

[148]S. Grover, R. Pea and S. Cooper, "«Systems of assessments» for deeper learning of computational thinking in K-12," in Proceedings of the 2015 Annual Meeting of the American Educational Research Association, 2015.

[149]G. Siemens and P. Long, "Penetrating the Fog: Analytics in Learning and Education," EDUCAUSE Review, vol. 46, no. 5, pp. 30, 2011.

[150]A. Mathrani, S. Christian and A. Ponder-Sutton, "PlayIT: Game based learning approach for teaching programming concepts," Journal of Educational Technology & Society, vol. 19, no. 2, pp. 5-17, 2016.

[151]C. Malliarakis, M. Satratzemi and S. Xinogalos, "CMX: The effects of an educational MMORPG on learning and teaching computer programming," IEEE Transactions on Learning Technologies, vol. 10, no. 2, pp. 219-235, 2016.

[152]C. Malliarakis, M. Satratzemi and S. Xinogalos, "Integrating learning analytics in an educational MMORPG for computer programming," in 2014 IEEE 14th International Conference on Advanced Learning Technologies, 2014.

[153]W. Greller and H. Drachsler, "Translating learning into numbers: A generic framework for learning analytics," Journal of Educational Technology & Society, vol. 15, no. 3, pp. 42-57, 2012.

[154]M. A. Chatti et al, "Learning Analytics: Challenges and Future Research Directions," eleed, vol. 10, no 1. Nov 6, 2014.

[155]M. Román González, "Computational thinking test: Design guidelines and content validation," in Proceedings of EDULEARN15 conference, Barcelona, pp. 2436-2444, 201.

[156]D. Căprioară, "Emotions and the Learning of School Mathematics," Bulletin of the Transilvania University of Braşov, Series VII: Social Sciences and Law, vol. 10, no. 1, pp. 9-18, 2017. Available: http://www.ceeol.com/search/article-detail?id=566146.

[157]C. Chen and P. Herbst, "The interplay among gestures, discourse, and diagrams in students’ geometrical reasoning," Educational Studies in Mathematics, vol. 83, no. 2, pp. 285-307, 2013. Available: 10.1007/s10649-012-9454-2, doi: https://doi.org/10.1007/s10649-012-9454-2.

[158]H. Y. Durak and M. Saritepeci, "Analysis of the relation between computational thinking skills and various variables with the structural equation model," Computers & Education, vol. 116, pp. 191-202, 2018. Available: http://www.sciencedirect.com/science/article/pii/S0360131517302087, doi: //doi.org/10.1016/j.compedu.2017.09.004.

[159]T. Watanabe, "Visual Reasoning Tools in Action: Double Number Lines, Area Models, and Other Diagrams Power Up Students' Ability to Solve and Make Sense of Various Problems," Mathematics Teaching in the Middle School, vol. 21, no. 3, pp. 152, 2015.

[160]C. E. Lance, M. M. Butts and L. C. Michels, "The sources of four commonly reported cutoff criteria: What did they really say?" Organ. Res. Methods, vol. 9, no. 2, pp. 202-220, 2006.

[161]M. Román-González, J. Moreno-León and G. Robles, "Combining assessment tools for a comprehensive evaluation of computational thinking interventions," in Computational Thinking Education, S. Kong and H. Abelson, Eds. 2019, Available: https://doi.org/10.1007/978-981-13-6528-7_6, doi: 10.1007/978-981-13-6528-7_6.

[162]A. A. De Souza et al, "Data mining framework to analyze the evolution of computational thinking skills in game building workshops," IEEE Access, vol. 7, pp. 82848-82866, 2019.

[163]R. F. DeVellis, "Classical test theory," Med. Care, pp. S50-S59, 2006.

[164]R. K. Hambleton and R. W. Jones, "Comparison of classical test theory and item response theory and their applications to test development," Educational Measurement: Issues and Practice, vol. 12, no. 3, pp. 38-47, 1993.

[165]L. El-Hamamsy, M. Zapata-Caceres, E. Martín, F. Mondada, J. Dehler Zufferey, and B. Bruno, "The competent Computational Thinking test (cCTt): Development and validation of an unplugged Computational Thinking test for upper primary school," Journal of Educational Computing Research, 2022. In press.

[166]S. Kong, "A framework of curriculum design for computational thinking development in K-12 education," Journal of Computers in Education, vol. 3, no. 4, pp. 377-394, 2016.

[167]F. Buitrago Flórez et al, "Changing a Generation's Way of Thinking: Teaching Computational Thinking Through Programming," Review of Educational Research, vol. 87, no. 4, pp. 834, 2017. Available: https://search.proquest.com/docview/1917804191, doi: 10.3102/0034654317710096.

[168]J. Alvarez et al, "Blue Bot Project Experiment," 8th International Toy Research Association World Conference Jul 11, 2018.

[169]C. Sáez Fernández, G. Viera López and D. Pérez Marín, "Propuesta metodológica de la enseñanza de la programación en Educación Infantil con Cubetto," IE Comunicaciones: Revista Iberoamericana De Informática Educativa, no. 28, pp. 1-8, 2018. Available: https://dialnet.unirioja.es/servlet/oaiart?codigo=6750332.

[170]W. Wu et al, "Investigating the learning‐theory foundations of game‐based learning: a meta‐analysis," Journal of Computer Assisted Learning, vol. 28, no. 3, pp. 265-279, 2012. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1365-2729.2011.00437.x, doi: 10.1111/j.1365-2729.2011.00437.x.

[171]E. Boyle, T. M. Connolly, and T. Hainey, "The role of psychology in understanding the impact of computer games," Entertainment Computing, vol. 2, no. 2, pp. 69-74, 2011. Available: https://www.sciencedirect.com/science/article/pii/S1875952110000200, doi: 10.1016/j.entcom.2010.12.002.

[172]M. Li and C. Tsai, "Game-Based Learning in Science Education: A Review of Relevant Research," Journal of Science Education and Technology, vol. 22, no. 6, pp. 877-898, 2013. Available: https://doi.org/10.1007/s10956-013-9436-x, doi: 10.1007/s10956-013-9436-x.

[173]W. Wu et al, "Re-exploring game-assisted learning research: The perspective of learning theoretical bases," Computers & Education, vol. 59, no. 4, pp. 1153-1161, 2012. Available: https://www.sciencedirect.com/science/article/pii/S0360131512001248, doi: 10.1016/j.compedu.2012.05.003.

[174]M. Csikszentmihalyi, "Flow and the psychology of discovery and invention," HarperPerennial, New York, vol. 39, 1997.

[175]T. Hainey et al, "A systematic literature review of games-based learning empirical evidence in primary education," Computers & Education, vol. 102, pp. 202-223, 2016.

[176]J. R. Savery and T. M. Duffy, "Problem based learning: An instructional model and its constructivist framework," Educational Technology, vol. 35, no. 5, pp. 31-38, 1995.

[177]J. R. Savery, "Overview of Problem-based Learning: Definitions and distinctions," Essential readings in problem-based learning: Exploring and extending the legacy of Howard S. Barrows, vol. 9, no 2, pp. 5-15, 2015. Available: https://library.biblioboard.com/viewer/620aa903-49eb-49d1-8fc2-b02dbeda193c.

[178]B. Haßler, L. Major, and S. Hennessy, "Tablet use in schools: A critical review of the evidence for learning outcomes," Journal of Computer Assisted Learning, vol. 32, no. 2, pp. 139-156, 2016.

[179]Y. Rogers et al, " Equal opportunities: Do shareable interfaces promote more group participation than single user displays?" Human–Computer Interaction, vol. 24, no. 1-2, pp. 79-116, 2009.

[180]D. Roldan-Alvarez et al, "DEDOS: An Authoring Toolkit to Create Educational Multimedia Activities for Multiple Devices," IEEE Transactions on Learning Technologies, vol. 11, no. 4, pp. 493-505, 2018. Available: https://ieeexplore.ieee.org/document/8244297, doi: 10.1109/TLT.2017.2788867.

[181]C. Lin, L. Wong and Y. Shao, "Comparison of 1: 1 and 1: m CSCL environment for collaborative concept mapping," Journal of Computer Assisted Learning, vol. 28, no. 2, pp. 99-113, 2012.

[182]M. Zapata-Cáceres, E. Martín-Barroso and M. Román-González, "Computational Thinking Test for Beginners: Design and Content Validation," 2020 IEEE Global Engineering Education Conference (EDUCON). in Press, 2020.

[183]R. S. Pressman, Software Engineering: A Practitioner's Approach. Palgrave macmillan, 2005.

[184]C. Larman, Agile and Iterative Development: A Manager's Guide. Addison-Wesley Professional, 2004.

[185]L. García-de-la-Fuente, "Tratamiento de ficheros generados por Blue Ant Code," Trabajo De Fin De Grado En Ingeniería Informática, Universidad Rey Juan Carlos, 2020.

[186]G. Matilla-Jiménez, "Desarrollo una aplicación web para facilitar la visualización y análisis se datos de Blue Ant Code," Trabajo De Fin De Grado En Ingeniería Informática, Universidad Rey Juan Carlos, 2020.

[187]J. Moreno and G. Robles, "Automatic detection of bad programming habits in scratch: A preliminary study," in 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, pp. 1-4, Oct. 2014.

[188]N. Diana et al, "An instructor dashboard for real-time analytics in interactive programming assignments," in Proceedings of the Seventh International Learning Analytics & Knowledge Conference, pp. 272-279, 2017.

[189]L. Gal et al, "Suggesting a log-based creativity measurement for online programming learning environment," in Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale, pp. 273-277, 2017.

[190]D. Olivares and C. Hundhausen, "Supporting learning analytics in computing education," in Proceedings of the Seventh International Learning Analytics & Knowledge Conference, pp. 584-585, 2017.

[191]M. Zapata-Ros and P. Pérez-Paredes, El Pensamiento Computacional. Análisis De Una Competencia Clave. New York: Create Space Independent Publishing, 2018.

[192]H. Søndergaard and R. A. Mulder, "Collaborative learning through formative peer review: Pedagogy, programs and potential," Computer Science Education, vol. 22, no. 4, pp. 343-367, 2012.

[193]M. R. Snodgrass, M. Israel and G. C. Reese, "Instructional supports for students with disabilities in K-5 computing: Findings from a cross-case analysis," Computers & Education, vol. 100, pp. 1-17, 2016.

[194]N. Ayuso et al, "Gender Gap in STEM: A Cross-Sectional Study of Primary School Students’ Self-Perception and Test Anxiety in Mathematics," IEEE Transactions on Education, vol. 64, no. 1, pp. 40-49, 2020.

[195]L. Sun et al, "Primary school students' perceptions of scaffolding in digital gamebased learning in mathematics," Learning, Culture and Social Interaction, vol. 28, pp. 100457, 2021.

[196]M. Cengiz et al, "Development of an interactive game-based learning environment to teach data mining," The International Journal of Engineering Education, vol. 33, no. 5, pp. 1598-1617, 2017.

[197]H. Lu, "Rethinking game-based learning from a gender perspective: a case study of a male English language learner in China," Gender, Technology and Development, vol. 16, no. 1, pp. 49-70, 2012.

[198]E. A. Boyle et al, "An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games," Computers & Education, vol. 94, pp. 178-192, 2016. Available: http://www.sciencedirect.com/science/article/pii/S0360131515300750, doi: //doi.org/10.1016/j.compedu.2015.11.003.

[199]R. J. Mislevy, L. S. Steinberg and R. G. Almond, "Focus Article: On the Structure of Educational Assessments," Measurement (Mahwah, N.J.), vol. 1, no. 1, pp. 3-62, 2003. Available: http://www.tandfonline.com/doi/abs/10.1207/S15366359MEA0101_02, doi: 10.1207/S15366359MEA0101_02.

[200]V. J. Shute and G. R. Moore, "Consistency and validity in game-based stealth assessment," Technology Enhanced Innovative Assessment: Development, Modeling, and Scoring from an Interdisciplinary Perspective, vol. 296, 2017.

[201]C. Alonso Fernández et al, "Improving serious games analyzing learning analytics data: lessons learned," in International Conference on Games and Learning Alliance. Springer, Cham Dec 5, 2018.

[202]G. Wallner and S. Kriglstein, "Comparative visualization of player behavior for serious game analytics," in Serious Games Analytics, Cham: Springer International Publishing, pp. 159-179, 2015.

[203]E. Zimmerman, "Narrative, interactivity, play, and games: Four naughty concepts in need of discipline," First Person: New Media as Story, Performance, and Game, vol. 154, 2004.

[204]E. Carratala Sánchez, "Análisis de la teoría de las metas de logro y de la autodeterminación en los planes de especialización deportiva de la Generalitat Valenciana." 2004.

[205]C. J. Ferguson and C. K. Olson, "Friends, fun, frustration and fantasy: Child motivations for video game play," Motivation and Emotion, vol. 37, no. 1, pp. 154-164, 2013.

[206]P. R. Pintrich and A. Zusho, "The development of academic self-regulation: The role of cognitive and motivational factors," in Development of Achievement Motivation, 2002.

[207]A. E. Gottfried, "Academic intrinsic motivation in young elementary school children." J. Educ. Psychol., vol. 82, no. 3, pp. 525, 1990.

[208]R. M. Ryan and E. L. Deci, "Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being." Am. Psychol., vol. 55, no. 1, pp. 68, 2000.

[209]E. L. Deci and R. M. Ryan, "Intrinsic motivation," The Corsini Encyclopedia of Psychology, pp. 1-2, 2010.

[210]J. R. Calvo-Ferrer, "Juegos, videojuegos y juegos serios: Análisis de los factores que favorecen la diversión del jugador," Miguel Hernández Communication Journal, no. 9, pp. 191-226, 2018.

[211]I. Granic, A. Lobel and R. C. Engels, "The benefits of playing video games." American Psychologist Journal, vol. 69, no. 1, pp. 66, 2014.

[212]M. Ventura, V. Shute and W. Zhao, "The relationship between video game use and a performance-based measure of persistence," Computers & Education, vol. 60, no. 1, pp. 52-58, 2013.

[213]R. M. Ryan, C. S. Rigby and A. Przybylski, "The motivational pull of video games: A self-determination theory approach," Motivation and Emotion, vol. 30, no. 4, pp. 344-360, 2006.

[214]M. Salminen and N. Ravaja, "Increased oscillatory theta activation evoked by violent digital game events," Neuroscience Letters - Journals, vol. 435, no. 1, pp. 69-72, 2008.

[215]R. Israel-Fishelson and A. Hershkovitz, "Shooting for the stars: Micro-persistence of students in game-based learning environments," in Early Warning Systems and Targeted Interventions for Student Success in Online Courses, 2020.

[216]C. Hart, "Factors associated with student persistence in an online program of study: A review of the literature." Journal of Interactive Online Learning, vol. 11, no. 1, 2012.

[217]R. Israel‐Fishelson and A. Hershkovitz, "Micro‐persistence and difficulty in a game‐based learning environment for computational thinking acquisition," Journal of Computer Assisted Learning, 2021.

[218]Y. Fang et al, "Online Learning Persistence and Academic Achievement." International Educational Data Mining Society, 2017.

[219]C. Alonso-Fernández et al, "Applications of data science to game learning analytics data: A systematic literature review," Computers & Education, vol. 141, pp. 103612, 2019. Available: https://search.datacite.org/works/10.1016/j.compedu.2019.103612, doi: 10.1016/j.compedu.2019.103612.

[220]P. R. Pintrich and A. Zusho, "Student motivation and self-regulated learning in the college classroom," in Higher Education: Handbook of Theory and ResearchAnonymous 2002.

[221]K. Weber and B. R. Patterson, "Student interest, empowerment and motivation," Communication Research Reports, vol. 17, no. 1, pp. 22-29, 2000.

[222]Á Serrano-Laguna et al, "Applying standards to systematize learning analytics in serious games," Computer Standards & Interfaces, vol. 50, pp. 116-123, 2017.

[223]Y. Tran, "Computational thinking equity in elementary classrooms: What third-grade students know and can do," Journal of Educational Computing Research, vol. 57, no. 1, pp. 3-31, 2019.

[224]R. Israel-Fishelson and A. Hershkovitz, "Persistence and achievement in acquiring computational thinking concepts: A large-scale log-based analysis," in E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, 2019.

[225]S. Grover et al, "Assessing problem-solving process at scale," in Proceedings of the Third (2016) ACM Conference on Learning@ Scale, pp. 245-248, Apr 25, 2016.

[226]M. Zapata-Cáceres and E. Martín-Barroso, "Blue Ant Code," 2020. Available: https://play.google.com/store/apps/details?id=com.artbabygame.blueAnt https://apps.apple.com/es/app/blue-ant-code/id1495021110.

[227]S. Kong, M. M. Chiu and M. Lai, "A study of primary school students' interest, collaboration attitude, and programming empowerment in computational thinking education," Computers & Education, vol. 127, pp. 178-189, 2018. Available: http://dx.doi.org/10.1016/j.compedu.2018.08.026, doi: 10.1016/j.compedu.2018.08.026.

[228]M. Guenaga et al, "How do students develop computational thinking? Assessing early programmers in a maze-based online game," Computer Science Education, vol. 31, no 2, pp. 259-289, 2021. Available: https://doi.org/10.1080/08993408.2021.1903248, doi: 10.1080/08993408.2021.1903248.

[229]T. P. Alloway, "How does working memory work in the classroom?" Educational Research and Reviews, vol. 1, no. 4, pp. 134-139, 2006.

[230]C. S. Dweck and D. C. Molden, "Their impact on competence motivation and acquisition," Handbook of Competence and Motivation: Theory and Application, vol. 2, pp. 135-154, 2017.

[231]L. S. Blackwell, K. H. Trzesniewski and C. S. Dweck, "Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention," Child Dev., vol. 78, no. 1, pp. 246-263, 2007.

[232]P. Sweetser and P. Wyeth, "GameFlow: a model for evaluating player enjoyment in games," Computers in Entertainment (CIE), vol. 3, no. 3, pp. 3, 2005.

[233]M. Lafuente Martínez et al, "Assessing Computational Thinking: Development and Validation of the Algorithmic Thinking Test for Adults," 2021.

[234]L. Narcisi et al, "Beginners Computational Thinking Test: Adaptation pour un public de moins de 5 ans." Service D'Ingénierie Pédagogique Et Du Numérique Éducatif, FPSE, UMONS, s.f., 2021.

[235]M. Lázaro-Álvarez, "Sudokus adaptativos para Educación Infantil o Primaria," Trabajo De Fin De Grado En Ingeniería Informática, Universidad Rey Juan Carlos, 2021.

[236]J. Plaza-Carro, "Blue-encode: Juego colaborativo de cifrado y descifrado de mensajes en Educación Primaria," Trabajo De Fin De Grado En Ingeniería Informática, Universidad Rey Juan Carlos, 2021.

_______________________________

⁶ https://www.uned.es/

⁷ https://www.treetree2.org/

⁸ https://www.roteco.ch/fr/nouveautes/comment-evaluer-lefficacite-de-la-pensee-computationnelle-chezles-eleves-du-primaire

⁹ https://www.uned.es/

¹⁰ https://www.treetree2.org/

¹¹ https://www.roteco.ch/fr/nouveautes/comment-evaluer-lefficacite-de-la-pensee-computationnelle-chez-les-eleves-du-primaire