While we are continuing to collect data, there have been several early findings that suggest that changes to general physics, organic chemistry and teaching methods courses are having a positive effect on student development.

Class Results

In General Physics, students viewed previously recorded think-alouds as well as recorded their own using the Livescribe smartpens. To assess students’ conceptual understanding, we used the Force Concept Inventory. The class average normalized gain on the FCI, which measures the amount of material learned that was not previously known, was statistically higher than any of the prior non-PENS courses taught by the same instructor. The gains of the two recent PENS classes were nearly 50% higher than the non-PENS prior years (0.55 versus 0.37).

Similar improvements were seen in students’ performance in organic chemistry where expert created think-alouds that focused on the content and best practices of problem solving were used. Students viewed and analyzed these think-alouds. Exam averages were across the board greater than previous years where recorded think-alouds were not used. The significance and degree of improvement vary but every exam average was greater as compared to previous years.

Case Studies

In addition to the quantitative, class-average data, we have studied select individuals who display a transformation in their problem-solving proficiency and/ or attitudes. One such student in general physics was ‘Isaac’ who had a Classroom Test of Scientific Reasoning score that indicated a lack of formal operational thinking & a GPA that was 0.3 lower than the class average. On the first three in-class tests, his scores were 18-24% below the class average. Then on the fourth test, where he reported that he was more aware of his thinking and would engage in metacognitive thinking, was only 6 points below the class average.

At the beginning of the semester I was quite skeptical of the Livescribe pen and how following a few simple guidelines/ approaches to problem solving could change the way I think....I stubbornly have to admit that the whole process took much longer than I had anticipated due to my unwillingness to embrace the pen as well as the process of talking out my actions.... It took almost the whole semester but I have finally come to a point where I find myself automatically dictating my comprehension process to prove to myself that I truly understand what is happening and the best method to approach a problem while utilizing key concepts.Ivan, Physics student
On a pre/ post-instruction survey, Isaac shifted his responses from one that showed no self-regulation (or valuing of self-regulation) in problem solving (continuously planning, monitoring and adjusting one’s thinking) to a more expert-like position. On the same survey, he reported that his test anxiety also was eliminated.

K-12 Teacher Development

As part of LMU’s graduate Teacher Leader certificate program, teachers enroll in a methods course where they refine their own teaching practices and explore other ideas, which may be of interest to their colleagues. As part of a revised version of this course, teachers viewed and discussed previously recorded student think-alouds. Teacher Leaders completed pre- and post-survey questions to assess their progress toward the above goals. It was seen that Teacher Leaders demonstrated significant gains (pre/ post instruction) in their modeling of self-regulation (p = 0.003) and teaching of self- regulation (p= 0.004). In open-response items, teachers explained how participating in this class affected how they approached problems themselves and instructed their students.
  • I try to talk out loud to make sure problems make sense which helps my students follow the same example.
  • Increased my conscious awareness of teaching strategies and timing
  • I try more to think about how my students would approach the problem.
  • Instead of self-regulation being discussed in passing every once in a while, I make sure I address it often throughout all lessons
  • I model how to go back and check my work when going through problems especially word problems. I often make mistakes at the beginning of a lesson when I introduce the "hook.” Then I talk thorough how to get the answer step by step.
  • I have students justify their work as well as use error analysis when they get questions wrong on tests. They go back and show the step that they missed and learn from their mistakes.
Teacher Leaders indicated that they spend more time on modeling problem-solving skills and teaching problem-solving skills than they did before participating in this class. All Teacher Leaders indicated that participation in the course affected how they require students to engage in problem solving, with many now requiring their students to check their work and state the strategies used more so than they previously did. Teacher Leaders also reported greater confidence in their ability to teach problem-solving in their own classes.

Student Researchers

Throughout the project a key goal has been to involve undergraduates as part of the research team. Mentoring young scholars as they begin their academic careers is important to the project team. Over ten different students have participated in collecting and analyzing data. These students are typically majoring in a STEM discipline- mathematics, chemistry, physics and civil engineering. The students learned many techniques in qualitative and quantitative research as well as given numerous opportunities to improve their communication skills. The topic of this project produced additional benefits as the students gained insight into their own problem-solving. When asked about the impact the research experience had on them, the students often pointed to changes in their own problem-solving:
  • “it is very easy to identify the specific part (ACE' M) of the problem that I am struggling with. This allows me to more quickly find what I need to know in order to solve to problem as thoroughly and completely as possible.”
  • “Now when I go on to solve problems, I conduct much more monitoring than I did before, mainly because I did not think much of its importance back then.”
  • “I had a great experience and it was instrumental to my success as a student and physics major at LMU. Contributed greatly to launching my career.”
Because of their experiences, they not only became better researchers, they also strengthened a skill vital for success in their own classes.