Introduction to AI Sokrates Lesson Data Quality Evaluation Report
Issue Number: HB20240226EDate: 26 February 2024Author: Power Wu
One of the main functions of AI Sokrates Teaching Analytics Service is the collection and analysis of teaching and learning big data. Data collection is the first step in analyzing and applying data. Based on the accumulated teaching and learning big data of AI Sokrates, by the end of 2023, the Sokrates Platform has collected data from nearly 350,000 lessons, comprising lesson learning data of approximately 10 million students (person-time). The Sokrates Platform is rapidly accumulating lesson big data, and we believe that utilizing this valuable data can help us understand teaching and learning patterns more accurately, thereby enabling us to formulate education innovation and development directions and strategies more precisely.
AI Sokrates Lesson Data Quality Evaluation Report Overview Video
Analyzing lesson data is another main function of AI Sokrates. Lessons are the primary learning environments, and the data collected by AI Sokrates differs from asynchronous online learning data; it records the teaching and learning data from physical (or blended/hybrid) lessons. Through the statistical analysis of these lesson big data, AI Sokrates assists teacher training institutions, academic research organizations, and education policymakers in understanding the teaching technology application capabilities, teaching models, learning patterns, student engagement, and other data at the city, district, or school level. AI Sokrates can automatically collect lesson data and assess and analyze them for each lesson, including five indexes: Technology Interaction (T) Index, Pedagogical Application (P) Index, Content Implementation (C) Index, School-based Comprehensive Evaluation (D) Index, and Learning Engagement (E) Index (as shown in Figure 1), as well as relevant application analyses.
Figure 1: Five Indexes of Lesson Data Quality Evaluation Report
✨1. Introduction to AI Sokrates
AI Sokrates Teaching Analytics Service (AI Sokrates), a lesson data cloud platform service developed exclusively by Taiwan HABOOK Group, is an innovative educational application service that integrates multiple AI technologies, including AI teaching behavior data analysis, AI Text Analysis, AI Voice-to-Text, and AI GPT Service. It automatically collects lesson teaching and learning data, analyzes lesson videos, transcribes lesson transcripts, analyzes lesson speech rate, terminology, questioning, etc., and generates Sokrates Videos, Lesson Observation Forms, Lesson Data Quality Evaluation Reports (HABOOK, 2024), AI Sokrates Timeline (including voice transcripts, teacher-student interaction records), and other educational professional application data. HiTeach 5 is the data input package of AI Sokrates, with simple operation and high integration. By installing HiTeach 5 in various types of classrooms (multimedia classrooms, regular classrooms, TBL/PBL classrooms, recording classrooms, observation classrooms, etc.) and pairing it with recording/video equipment, lesson data can be collected, and educational professional application data can be generated. The data can then be stored in the exclusive channel of the Sokrates platform. The system operation architecture is shown in Figure 2.
Figure 2: AI Sokrates Operational Framework
✨2. Introduction to Lesson Data Quality Evaluation Report
The Lesson Data Quality Evaluation Report is an analysis report that supports teachers in discussing lessons. It includes lesson teaching data, learning data, and expert observation data, which are automatically generated by machines or given by experts. The main items include: 1. Technology Interaction (T) Index, 2. Pedagogical Application (P) Index, 3. Content Implementation (C) Index, 4. School-based Comprehensive Evaluation (D) Index, 5. Lesson Engagement (E) Index, 6. Lesson Analysis S-T Chart of Teaching Models, 7. Smart Lesson Main Learning Types, and 8. AI Sokrates Timeline (including observation markers/feedback, lesson transcripts, and interaction records). Below uses the data of a demonstration lesson to introduce the contents of a Lesson Data Quality Evaluation Report.
(1) Technology Interaction (T) Index
The Technology Interaction (T) Index has five analysis indexes, T1: Data channel, T2: Data feedback, T3: Decision by statistics, T4: Focus on students, and T5: Multi-assessment. AI statistically analyzes the frequency and effective combination application of each index, and gives sub-indexes and a total index. When technology integration is minimal, it is rated as a red light (<50); when technology integration is moderate, it is rated as a yellow light (50-70); when technology integration is proficient, it is rated as a green light (>=70). Figure 3 provides an example of the technology interaction index, distribution, usage time/count chart, and time accumulation chart for the demonstration lesson.
Figure 3: Distribution of technology interaction, frequency of technology usage, and time accumulation chart
Additionally, a technology interaction timeline chart (Figure 4) is generated to present the chronological use of technology tools, providing an initial understanding of the technological application characteristics of lesson teaching data.
Figure 4: Technology interaction timeline chart
When a more precise examination of the interaction between teachers and students during the lesson is needed, you can use the "AI Sokrates Timeline", which presents various lesson data in the form of a timeline, including expert observation marks/feedback, lesson transcripts, and teacher-student interaction records. (For more information please read (8) below)
(2) Pedagogical Application (P) Index
The Pedagogical Application (P) Index has six analysis indexes, P1: Group learning, P2: Whole-class interaction, P3: Student center decision, P4: Whole-class assessment, P5: Individual learning, and P6: Multi-approach assessment. AI uses these six indexes to evaluate the effectiveness of Teaching Behavioral Data Characteristics and gives sub-indexes and a total index. The more effective the teaching behavior data characteristics, the higher the index will be. When technology integration is minimal, it is rated as a red light (<50); when technology integration is moderate, it is rated as a yellow light (50-70); when technology integration is proficient, it is rated as a green light (>=70). Figure 5 presents an example of the teaching pedagogical application index and sub-indexes of the demonstration lesson.
Figure 5: Pedagogical Application (P) Index and sub-indexes
Global TEAM Model Education Research Institute (GTERI) collaborates with teacher training institutions to recode data based on the teaching research evaluation indicators of the collaborating institutions, to produce the required research evaluation data. For example, in cooperation with the Guangdong University of Education, the GTERI has developed an output of data based on a five-dimensional evaluation index: classroom interaction, higher-order thinking, classroom training, evaluation Incentives, and student-based orientation, as shown in Figure 6. This aims to cultivate and evaluate the technological teaching capabilities of teacher trainees (or practicing teachers).
Figure 6: Five-dimensional evaluation index for Guangdong University of Education
(3) Content Implementation (C) Index
The Content Implementation (C) Index has five analysis indexes, C1: Teaching Design, C2: Teaching Process, C3: Teaching Effectiveness, C4: Technology Integration, and C5: Innovation. These analysis indexes need to be entered manually by experts, and combined with AI Sokrates's Technology Interaction (T) Index and Pedagogical Application (P) Index to form the TPACK, a smarter lesson evaluation index that deeply integrates Technological (T), Pedagogical (P), Content (C). Figure 7 shows an example of the Content Implementation (C) index and its sub-indexes for the demonstration lesson.
Figure 7: Content Implementation (C) Index and sub-indexes
(4) School-based Comprehensive Evaluation (D) Index
The school-based Comprehensive Evaluation (D) Index is set by educational institutions based on their own customized school-based teaching models and lesson observation evaluation criteria. These evaluation indexes can be manually input by observers using the Sokrates Lecture Observation App at the end of lesson observations or entered by administrators after collecting expert data in the management interface. Figure 8 shows an example of the School-based Comprehensive Evaluation Index and its sub-indexes for the demonstration lesson.
Figure 8: School-based Comprehensive Evaluation (D) Index and sub-indexes
(5) Learning Engagement (E) Index
Learning Engagement is like an electronic pedometer, constantly recording learning performance to help teachers assess students' engagement time and participation levels (e.g., behavioral, emotional, cognitive, and proactive participation). In a HiTeach class where everyone has their own smart devices (e.g. tablets), the system records each student's participation data, including points, interactions, tasks, tests, peer assessments, collaborations, etc. It calculates the Learning Engagement Index of each student and aggregates the indexes for the whole class and groups, allowing teachers to further track the engagement data of groups or individual students after class. Figure 9 shows an example of whole-class participation, distribution, and group participation in the demonstration lesson.
Figure 9: Learning Engagement (E) Index (by class and group) and the distribution chart
(6) Lesson Analysis S-T Chart of Teaching Models
AI Sokrates automatically records lesson interactions between teachers and students and generates S-T and Rt-Ch charts based on the S-T lesson analysis method. By using HiTeach for teaching and combining it with AI Sokrates' data collection and analysis capabilities, S-T and Rt-Ch charts can be automatically generated. Figure 10 shows an example of S-T and Rt-Ch charts in the demonstration lesson, and the teaching model is evaluated based on numerical values such as Dialogue, Hybrid, Practice, or Lecture.
Figure 10: Analysis of the S-T Chart and Rt-Ch Chart
(7) Smart Lesson Main Learning Types
HiTeach supports multiple competence-oriented modern teaching methods and strategies. For HiTeach lessons, AI Sokrates automatically analyzes the teaching methods and strategies employed, including team-based, interactive, task, assessment, differentiated, peer assessment, and collaboration learning.
Figure 11 shows an example of the Lesson Learning Types Analysis in the demonstration lesson.
Figure 11: Smart Lesson Main Learning Types
(8) AI Sokrates Timeline
The AI Sokrates Timeline is centered around the timeline axis and presents various lesson data based on the progression of time, including expert observation marks/feedback, lesson transcripts, and teacher-student interaction records. See Figure 12 for reference.
Al Sokrates Timeline Example from Affiliated Experimental Elementary School of University of Taipei
(9) Summary of the Lesson Data Quality Evaluation Report
At the end of the Lesson Data Quality Evaluation Report, there will be a summary of the five indexes. Figure 13 shows an example of the summary of the demonstration lesson.
Figure 13: Example of Summary of the Lesson Data Quality Evaluation Report
✨3. Examples of Application of Lesson Data Quality Evaluation Report
AI Sokrates is an innovative educational service utilizing AI technology. After lessons, the Lesson Data Quality Evaluation Report automatically generated by AI Sokrates holds significant value for teacher professional development. It can be used for teaching discussions, teacher training, educational research, and teaching quality evaluation. Below are examples of its applications:
In smarter pedagogical review activities or lesson observation activities conducted in regular or micro-classrooms, teachers utilize HiTeach for instruction. Concurrently, observing teachers/experts use the Sokrates Lecture Observational App to give feedback. After the lesson, the teaching research team utilizes automatically generated Sokrates Videos, Observation Forms, and Lesson Data Quality Evaluation Reports for pedagogical discussions. They can seamlessly switch between different time points marked in the video, facilitating evidence-based teaching research discussions. Figure 14 shows a snapshot of Smarter Pedagogical Review Activities held at Yongfeng Elementary School in Dujiangyan.
Figure 14: Smarter Pedagogical Review Activities
(2) Application Scenario: District / School Teaching and Learning Data Analysis
The AI Sokrates system installed in classrooms of schools or whole district schools can routinely and automatically collect and accumulate classroom teaching-related data of the teaching team. Utilizing these automatically collected large volumes of lesson data, precise analysis of teaching behavior data features can be conducted. This includes T (data) statistical analysis, teaching model analysis, learning model analysis, learning engagement analysis, and technology interaction tool analysis. Through this process, course design and development can be optimized and refined. Examples of data analysis types are illustrated in Figure 15. Figure 16 shows the statistical chart of learning types at Chengdu Eldu Wisteria Primary School, analyzing the frequency and ratio of various learning strategies applied in regular teaching at the school.
Figure 15: Analysis of Teaching and Learning Data in School Districts (Schools)
Figure 16: Learning Type Statistics Chart of Chengdu Eldu Wisteria Primary School
(3) Application Scenario: Technological Enhancement Training Data Results (Teaching Competency Assessment/Evaluation)
The AI Sokrates system automatically collects data and generates Lesson Data Quality Evaluation Reports, which can serve as outcome data for technological enhancement training for pre-service and in-service teachers. It can also be used as reference data for assessing teaching capabilities, as shown in Figure 17.
Figure 17: The results of technology-enhanced training
.png)