Scientific Reasoning Research Institute - Project

Strategies for Leading Classroom Discussions Aimed at Core Ideas and Scientific Modeling Practices

lstephens — Sat, 30 Jan 2016 00:35:15 +0000

Group Page:

CLSG

Starting date:

2015-08-01

Contact: John Clement
Funding: NSF DRK-12 Program # DRL-1503456

This NSF funded project is conducting classroom case studies in the areas of high school mechanics and electricity, and middle school life sciences in order to identify key discussion leading strategies used by exemplary teachers. It will also combine the results of the case studies with studies of expert scientists reasoning practices to develop an integrated theoretical framework for model based learning in science.

Identifying Science Teaching Strategies for Promoting Reasoned Discussions of Concepts and Simulations

lstephens — Sat, 30 Jan 2016 00:23:08 +0000

Group Page:

CLSG

Starting date:

2012-09-01

Contact: John Clement
Funding: NSF DRK-12 Program, DRL- 1222709

This project used classroom video tapes to identify discussion leading strategies that science teachers use to promote understanding in science classes, including classes that use computer simulations.

S-CASTS

konold — Thu, 09 Sep 2010 14:32:40 +0000

Group Page:

SERG

Funding:

National Science Foundation

S-CASTS: System for Collaboration among Students, Teachers and System is a collaboration between UMass, TERC, and Artificial Intelligence researchers at Harvard. We are investigating the use of models of collaboration, especially as embodied in collaborative human-computer interface systems, in the augmentation of existing flexible software tools for mathematics education. The mathematical domain is probability modeling and the educational software being used is the development version of TinkerPlots 2.0. As our part of the project, we have been adding logging capabilities to our beta version of TinkerPlots 2.0 that capture user actions as students use TinkerPlots to build and run models of various probabilistic situations.

Data Games

konold — Wed, 08 Sep 2010 18:07:58 +0000

Tools and Materials for Learning Data Modeling

Group Page:

SERG

Contact(s):

Konold, Cliff

Funding:

National Science Foundation

Starting date:

2009-09-01

by:

Cliff Konold

Students playing computer games generate large quantities of rich, interesting, highly variable data that mostly evaporates into the ether when the game ends.

What if in a classroom setting, data from games students played remained accessible to them for analysis? In software and curriculum materials being developed by the Data Games project at UMass Amherst and KCP Technologies, data generated by students playing computer games form the raw material for mathematics classroom activities. Students play a short video game, analyze the game data, conjecture improved strategies, and test their strategies in another round of the game.

The video games are embedded in TinkerPlots and Fathom , two data analysis learning environments widely used in grades 5–8 and 8–14 respectively. The game data appear in graphs in real time, allowing several cycles of strategy improvement in a short time. The games are designed so that these cycles improve understanding of specific data modeling and/or mathematics concepts. Lessons will be embedded in LessonLink from Key Curriculum Press to facilitate their integration into standard curricula. The three-year project expands research in students’ understanding of data modeling and their ability to learn mathematical content embedded in data-rich contexts.

For more project details see the Data Games Project Page at KCPT.

For the games developed at SRRI, see Games.

For games and activities, see Data Games product website - SRRI copy

Research: Statistical Intuitions

Thu, 14 Jan 2010 18:24:27 +0000

Investigate statistical intuitions of college students

Group Page:

SERG

Understanding of Basic Statistical Concepts
NSF Grant BNS-8509991 (1985 - 88)

Cognitive Skills Underlying Statistical Inference
NSF Grant SED-8113323 (1981 - 85)

Program of Applied Research on Scientific Reasoning Processes
NSF Grant SED-8016567 (1980 - 83)

Role of Preconceptions and Representational Transformations in Understanding Science and Mathematics
NSF Grant BNS-8509991 (1978 - 80)

In these four projects We used primarily clinical interviews to investigate statistical intuitions of college students. Statistical ideas we explored included randomness, sampling, weighted means, probability, conditional probability and inference, and sampling distributions. Results of this research are included in various articles listed among our available papers.

The TLT project is funded primarily by grant RED-9452917 from the National Science Foundation. Any opinions, findings, conclusions, and recommendations expressed here or in other project publications are those of the principal investigators and do not necessarily reflect the views of the NSF.

Critical Barriers

Thu, 14 Jan 2010 17:35:18 +0000

Students Analyzing Data: A Study of Critical Barriers

Group Page:

SERG

Funding:

US National Science Foundation RED-9452917

Starting date:

1995

This study was funded under the Small Grant for Exploratory Research Program at NSF. Researchers at the Universities of Massachusetts, Bielefeld, and Dortmund (see staff) independently analyzed a set of videotaped interviews of four students engaged in data analysis. The students had just finished a year-long course in probability and statistics at Holyoke High School (taught by Al Gagnon) in which they had performed at about the class average. Our interest was in exploring difficulties associated with doing fairly rudimentary data analysis by students who had had some introduction to statistics and had access to and familiarity with a data-analysis tool.

The interview comprised a series of open-ended questions concerning data the students had explored during the course. The data included information on 154 students in various mathematics courses at the school during that and a previous year; the 62 variables included information about age, gender, religion, job status, parents' education, stance on abortion, and time spent on a number of activities including studying, TV viewing, and reading. In both the course and the interviews, students worked in pairs using the data analysis software, DataScope®.

Results of our analyses of the interviews, which are summarized in Konold, Pollatsek, Well and Gagnon (1997), Biehler (1997), and Steinbring (1996), suggest at least three interrelated domains that cause particular problems for students learning data analysis. We describe these in the three sections below.

Group Properties

The most striking feature of the interviews was the limited ability of students to use and reason about group properties, such as percentages, means and medians. Typically, when reasoning about a single group, students spontaneously used these measures with what appeared to be good understanding (e.g., they used and reasoned intelligently about the divorce rate among parents of the students in the sample). In contrast, when they were comparing two groups (e.g., trying to discover whether boys or girls were more likely to have a driver's license) or exploring a relationship between two quantitative variables (e.g., whether there was a relation between time spent doing homework and course grades), they never spontaneously made use of appropriate group measures. Rather, they reverted to comparing the absolute frequencies in individual cells (e.g., comparing the number of boys and girls who had licenses, making no use of the number of boys and girls who didn't have licenses). Moreover, even though the appropriate statistics (e.g., the percent of boys who had licenses and the percent of girls who had licenses) were usually prominently displayed in the statistical display they were examining, the students did not switch to using them even after the interviewer suggested that the percentages might be relevant.

Although we do not fully understand this striking phenomenon, we are confident that it indicates an important problem, a major part of which is a lack of understanding that meaningful comparisons have to be of group tendencies rather than of properties of individuals within the groups. Interrelated barriers to using group tendencies include:

not understanding the concept of "variable." This is a difficult notion for many students, but critical for using group measures. For example, when comparing boys and girls in terms of driver's licenses, there is an implicit variable which includes the states of having a license and not having a license.
not understanding the imperfect yet lawful interrelation between group properties and individual observations. On the one hand, students need to see that statistical summaries such as medians, percents, standard deviations, and histograms, describe emergent group properties for which there may not be corresponding properties in the individuals (e.g., the fact that height is relatively normally distributed in the adult population could not be deduced from looking at the height of any one individual). On the other hand, they must also develop the understanding that these group properties are not arbitrary in that they both stem from relatively simple computations from individual data points, and also can often be related back to individual observations (e.g., if the median hours of study for students who get A's is higher than that for students who get C's, this says something about the behavior of individual students).
not having an appropriate model in which lawfulness and "randomness" can coexist. Statisticians have conceptual models of repeatable "random events" such as urns, coins, and dice, and models of "true values" plus or minus "random error". In contrast, novices usually have no conceptual model that explains variability, and instead rely on individual observations to draw conclusions (the "outcome approach," Konold, 1989).

Formation of Statistical Questions

In data analysis, a general question about the real world is transformed into an empirical question. This transformation guides the choice of data, coding decisions, data organization, how data are queried, and how observed patterns are related back to the original real-world question. Students in our interviews showed little awareness of the simplifying decisions inherent in this modeling process, and we expect that their tendency to not recognize the limits of their conclusions or those of others, and to not explore multiple possibilities during data analysis, stems in part from their not being fully aware of this transformation.

Causal Thinking

The belief seems widespread that results of analyzing observational data can be interpreted directly in causal terms. In the interviews, students' formulations of problems they wanted to investigate were typically framed in causal terms (e.g., "If you have a curfew it makes you ..."). Similarly, we find myriad causal relations in the media where observational studies are quoted in a way that seemingly support causal claims. Didactical constraints may be partly responsible in that students are often given observational data and are asked to investigate the "effects" of one variable on another. This language, which for the teacher may be just a manner of speaking, may be interpreted by the students as meaning "effect" in a causal sense. Furthermore, common language and the idea of a sequence of causes acting on individual cases do not provide adequate grounding for qualitative modeling in multivariate statistical situations.

In summary, our analyses suggest that the barriers we have uncovered in students' attempts at exploratory data analysis are remarkably parallel to those in understanding more advanced statistical concepts. Central to any adequate understanding of data are the ideas of "group measures" (such as medians, percents, inter-quartile ranges) together with their strengths and limitations, the imperfect relationship between verbal statements and statistical formulation, and the very imperfect relationship between intuitive causal thinking and statistical relationships.

Data Sharing

Thu, 14 Jan 2010 17:12:03 +0000

A Study of Student Investigations in Data-Sharing Projects

Group Page:

SERG

Contact(s):

Pollatsek, Alexander

Funding:

US National Science Foundation Grant REC-9725228

Starting date:

1997

Overview The primary objectives of this research project were to a) identify the core ideas in rudimentary data analysis, b) research the methods students typically employ to compare two groups or judge the relationship between two variables, and c) identify features of data and tasks that we should attend to in designing instruction. Our research was aimed at informing teachers as well as the development of future data analysis projects, materials, software, and teacher development efforts. We focused in particular on the data collection and analysis practices of students participating in "network science" projects. Network science projects use the Internet to link distant classrooms so that they can pool locally-collected data for aggregate analyses. In Global Lab, for example, at noon of the fall equinox students from all over the world collect a variety of local information and use these data to study such things as the relation between geographic location (sun angle) and measured light intensity. One important advantage of this approach is that the focal point of student investigations is not the data analytic techniques per se, but the phenomena students are exploring.

One of the things that prompted us to undertake this research were reports, both formal and informal, from several Network Science projects that their students were having considerable difficulty analyzing data. Given that the objectives of Network Science depend on students not only collecting and sharing data, but also reasoning about and learning from data, these difficulties present a serious barrier to fostering authentic science and mathematics learning. To explore these difficulties, we looked both at the projects (the data and materials they provided students) and at the students (the capabilities they brought to the task of data analysis).

We studied five Network Science projects: EnviroNet, GLOBE, Journey North and Water on the Web. We chose sites that a) ranged across various grade levels and b) used a variety of data types, including geographical based data, time series data, and case-based data sets that included multiple variables which students could explore. As part of the study conducted at TERC (and reported in Feldman et al, 2000) we also looked at Global Lab and EnergyNet.

In our studies of student reasoning, we are working with the following data sources:

Interviews and final project reports of 9th grade science students involved with EnviroNet in Derry, NH (4 groups of 2 students).
Interviews and final project reports of 7th grade students involved with EnviroNet in Rye, NH (3 groups of 4 students).
Interviews and final project reports of 12th graders at Holyoke High School, Holyoke, MA (2 groups of 2 students; about 12 final projects).
Pre-post instruction interviews of 7th and 8th grade students in a teaching experiment at Vanderbilt University in Nashville, TN (approximately 15 individual interviews, and 6 interviews with pairs of students.)
Analysis of a casebook of teachers' reflections on reasoning of K-6 students from the "Teaching to the Big Ideas" project directed by Deborah Shifter, Virginia Bastable, and Susan Jo Russell (34 written case studies).
Interviews with psychology students at LaTrobe University, Melbourne, Australia (13 individual interviews with 1st and 4th year undergraduates).

Major Findings

Network Science projects tend to underestimate the complexity of real data analysis and the amount of support they therefore need to provide to both students and their teachers. While we find some interesting and interpretable trends in some of their data, the skill required to detect these trends is typically beyond the unaided ability of even the best of students. This may help explain why we were able to find few participating classrooms that have an established history of analyzing data downloaded from Network Science sites. Our analysis of the problems and specific recommendations about how to address them are spelled out in Feldman, Konold, and Coulter (2000), and in a series of technical reports (see introduction).
Through our analysis of student thinking in classrooms, we have a much better understanding of the ideas that young students bring to their early experiences with data, and how these ideas might be further developed with instruction. Our research is described in various books and articles, several of them written specifically for teacher audiences. In two articles by Konold and Pollastek, we describe what we have come to regard as the core idea in data analysis and statistics -- the idea of central tendency. We analyze the nature of this idea in various contexts, trace its historical development, and suggest how the idea might be better developed instructionally.

Our understanding of students in the elementary grades how they learn to shift their focus from data as pointers, to case values, to aggregates was based primarily on our study of case studies written by elementary school teachers participating in the teacher development project "Teaching to the Big Ideas" directed by Deborah Shifter, Virginia Bastable, and Susan Jo Russell. These results are summarized in two articles by Konold and Higgins.

Our understanding of student thinking at the middle school, and how they begin to see data as an aggregate, was based primarily on our interviews with students participating in the RoadKill project at two schools in New Hampshire. These are being summarized in an article by Konold, Robinson, Khalil, Pollatsek, Well, Wing, & Mayr, (in preparation).

This project is supported in part by a grant from the National Science Foundation (REC-9725228). Opinions expressed here are those of the project staff and do not necessarily reflect the views of the Foundation.

ChancePlus

Thu, 14 Jan 2010 16:00:29 +0000

A Computer-Based Curriculum in Probability and Statistics

Group Page:

SERG

Funding:

US National Science Foundation Grant MDR-8954626

Starting date:

1989

ChancePlus was a four-year project to develop and field test materials for teaching probability and statistics at the high-school level using computers. The software and materials were influenced by NCTM's Curriculum Standards and by our prior research concerning the nature of student intuitions about probability and statistics. For information on our team, see ChancePlus [staff] .

Major Outcomes

We developed two Macintosh software tools: DataScope ® , a powerful and easy-to-use data-analysis tool with accompanying data sets, and Prob Sim ® , a generic, probability-modeling device designed for teaching probability and sampling via simulation. The programs, along with instructional materials and user's guides, were published by Intellimation and by the Australian Association of Mathematics Teachers, Inc. and are now available from us. Both programs were positively reviewed in The Mathematics Teacher (1996, 4 , 359-360) and in the Statistics Teacher Network (Winter, 1995).
A series of activity-based computer-supported instructional units on data analysis and probability for high-school or introductory college courses, published along with the software.
Research , which included several new studies of student intuitions about probability and implications for teaching probability and statistics, appeared in over 20 journal and book publications and 25 presentations and workshops given at various research centers and professional meetings. See our list of available papers.
Statistical Reasoning Assessment , a collection of assessment tools for monitoring changes in attitudes about and understandings of probability and statistics. These items, which were largely based on similar items used in our research, have been adapted and included in a number of subsequent studies and projects and in a number of different countries. This instrument was developed in collaboration with Joan Garfield at the University of Minnesota.

Evaluation of Materials

Our materials were designed to support an inquiry method of instruction, wherein students are encouraged to:

make predictions based on their intuitions
test and revise predictions through simulation and/or analysis of data
work collaboratively with other students
express themselves clearly both in oral and written communication
cultivate a reflective and questioning disposition.

The effectiveness of instruction using the software and curricular materials was assessed using the Statistical Reasoning Assessment at a number of sites. This instrument was designed to assess student understanding of a number of key concepts in probability and statistics. For most concepts we targeted, the number of students showing understanding grew by between 10% and 20% over instruction. However, the materials have been continually refined, and in the latest implementation (with high-school age students at SummerMath) the effectiveness appears to have improved. Whereas only about 20% of the students were judged as reasoning probabilistically before the two-week workshop, the proportion increased to 73% after the workshop. (In contrast, about 40% of surveyed college students who have had more traditional instruction in statistics show comparable understanding on the same set of items.

This project was supported in part by a grant from the National Science Foundation (MDR-8954626). Opinions expressed here are those of the project staff and do not necessarily reflect the views of the Foundation.

Deepening Conceptual Understanding in Middle School Life Science

root — Thu, 17 Jan 2008 23:47:26 +0000

Group Page:

CLSG

Contact(s):

Clement, John

Funding:

US National Science Foundation grant #9911401

Starting date:

2000-04-15

This NSF project is completing a model-based curriculum on Energy and the Human Body at the middle school level and investigating ways of teaching complex visual models in science.

Model Construction Processes in Experts

root — Thu, 17 Jan 2008 23:44:50 +0000

Contact(s):

Clement, John

This project complements and provides input to our science education projects by attempting to understand model construction and learning processes in expert scientists, with an emphasis on the roles of analogy, imagery, and thought experiments.

Visual Modeling Strategies in Science Teaching

root — Thu, 17 Jan 2008 23:40:20 +0000

Finding principles of instruction for developing students' visualizable models in science

Group Page:

CLSG

Contact(s):

Clement, John

Funding:

US National Science Foundation grant #0723709

Starting date:

2007-08-15

This NSF-funded project seeks principles of instruction for developing students' visualizable models in science, including design principles for curriculum development, technological tools, and new pedagogical principles. We are pursuing specific aspects of this goal by conducting detailed studies of teaching and learning in the context of innovative model based curricula in middle school biology, and in middle and high school physical science. Two parallel tracks of the research are:

studying classroom interaction patterns fostering the learning of visual models, and
analyzing the use of computer simulations and animations in large class discussions to scaffold students' ability to construct their own animated mental models.

Minds*On Physics Project

root — Sun, 13 Jan 2008 00:04:22 +0000

Developing and field testing a research-based curriculum for high school physics

Contact(s):

Leonard, William J.

Funding:

NSF MDR-9050213, NSF MDR-9255713

Starting date:

1993-03-01

To learn about the finished Minds·On Physics product, see the MOP entry in our Resources section.

When the Minds·On Physics (MOP) project began in 1990, there were no exemplary materials for high school physics instruction, the most popular textbooks were full of errors and oversimplifications, and college physics students seemed to be at a disadvantage if they had taken physics in high school. Results of cognitive research were starting to make their way into college instruction (by the researchers themselves, for the most part), but there had been little impact on high school physics instruction.

Pulling together multiple strands of educational research, such as cognitive overload, expert-novice differences, metacognition, and alternative conceptions, UMPERG created a framework for describing how knowledge is stored and used. We also identified 5 stages of cognitive development. Using the framework and stages of development, we proposed to create activities to help students explore existing concepts, interrelate concepts, analyze and reason, solve problems, and organize knowledge.

At first, the approach was so different from anything that had been done previously that NSF would fund only a small pilot project with four teachers in four very different settings. The activities were found to be useful and effective with the pilot teachers and classes. Now there are more than 180 activities in 6 volumes for students and over 2,000 pages of support materials for teachers.

The materials have many special features. For instance, activities are done first, with little or no preparation, and no need for any introduction by the teacher. The reading assignments are brief (1-2 pages per activity) and done after the activities are completed, rather than before. The activities demonstrate multiple paths to success, while encouraging thoughtfulness, communication, teamwork, and self-awareness as essential features of learning. The program is algebra-based, but it avoids over-generalizations by using graphs and mathematical principles to take into account changing quantities. The materials also support and promote a new approach to teaching that stresses modeling, conversation, advising, and mentoring over traditional, one-size-fits-all lecturing.

Having published the last Teacher's Guide early in 2003, the project has moved into an implementation and adoption phase, with visits to teachers and school systems interested in trying out a new style of teaching and learning. Grand Rapids (MI) Public Schools has recently adopted MOP, and Chicago Public Schools (among others) is currently considering the program.

Knowledge Broker

root — Sat, 12 Jan 2008 23:53:36 +0000

Exploring technology for next-generation classroom response systems

Contact(s):

Beatty, Ian D.

Funding:

Hewlett-Packard Co. (University Mobile Technology grant); Microsoft Corp. (University Relations grant)

Effective pedagogy --- identifying students' initial understanding and misconceptions, engaging their minds in the activity of learning, continually monitoring their individual progress, and adjusting one's teaching to the ever-changing circumstances of the classroom --- is difficult to practice with medium-sized and large classes. Technology can help, in the form of Classroom Communication Systems (CCS) and Classroom Response Systems. These are combinations of hardware and software that help an instructor give students questions to answer or tasks to accomplish in class, individually or in small groups; collect the answers; and immediately display a statistical summary or histogram of the answers for the instructor and potentially the entire class. In concert with appropriate pedagogic techniques and curriculum, such systems can enable interactive, active, dynamic pedagogy in large classes to a degree not previously possible.

Currently available systems, however, are pedagogically limiting and not ideally suited to forward-thinking university instruction. They are also closed and proprietary, not amenable to creative innovation, improvement, or extension. We believe a need exists for an open CCS that is:

designed according to sound, current thinking in educational research; and
not just a CCS product, but a platform for ongoing research, innovation, and growth in technology-assisted classroom pedagogy.

We have, therefore begun exploratory development of the Knowledge Broker, a next-generation CCS to serve both as a tool for practical teaching and as an instrument for research into technology-enhanced classroom pedagogy. In the process, we are finding other uses for Tablet PC technology in university instruction.

This project was born in collaboration with, and has provided support to, Prof. Gino Sorcinelli's ConferenceXP project in the UMass Isenberg School of Management.

Update: As of 2005, this project has been suspended indefinitely due to lack of support.

homeworkCentral

root — Sat, 12 Jan 2008 23:47:21 +0000

Online support for students doing electronic homework

Group Page:

PERG

Contact(s):

Leonard, William J.

Funding:

NSF ROLE-0106771

Starting date:

2003-09-01

At a large state university such as UMass, resources are necessarily limited, and students in a course with 200 or more classmates often think they cannot possibly get the assistance they need to answer questions and complete their homework. Course personnel are available for only certain hours and only during the day, and the departmental resource room has help that is regrettably spotty. A few students seek and receive help, but the majority of students work alone or in small groups, without having useful assistance.

We also use an electronic homework (eHW) system to administer out-of-class activities. While eHW has many advantages, such as immediate feedback and the opportunity to attempt each problem many times, there are also some drawbacks. For instance, students sometimes become overly focused on getting the answer correct and earning a perfect score on their assignments, and inevitably develop frenetic, superficial approaches to doing their homework.

So, even though analysis activities are being encouraged and modeled during class, since students spend most of their time outside of class, it is essential to help students engage in analysis activities while working on their homework. As part of the RRA project, we also despaired of ever monitoring what students are doing during homework, and therefore, we thought we would never be able to judge the level of engagement of individual students.

"It's very helpful most of the time; like having my own [professor]."

To address both of these concerns, we created a web site called homeworkCentral (hwC). For each homework problem assigned, there are typically 3-5 suggestions for activities designed to help students analyze the situation and solve the problem, with around a dozen hints for executing the suggested activities and a separate page of common pitfalls to avoid. The web site is available online, so students can get support from a trusted source any time they need it.

"Homework central helps us with our eHW when no one else is around to help (ie the professor) Helps us look at the problem in the correct way and names the common pitfalls so we are to avoid them."

By structuring the site with each hint on a separate page and pitfalls separate from suggestions, we anticipate being able to profile and follow students as they work through each homework problem. By correlating the hit logs in hwC with those from eHW, we hope to be able to construct a rather complete picture of how students — all students — begin, process, cope with, and finally complete their homework assignments.

"When doing homework, it's helpful to have the help immediately and from a source that is guarenteed [sic] to be reliable."

Every Decision Counts (EDC)

root — Sat, 12 Jan 2008 23:40:47 +0000

Developing and researching the impact of an alternative format for multiple-choice assessment

Contact(s):

Leonard, William J.

Funding:

NSF ROLE-0106771

Starting date:

2002-02-01

Standard multiple-choice assessments, for which there are 4 incorrect and exactly one correct choice, are efficient to implement but imprecise and difficult to interpret. To assess student course work, teachers generally accept the limitations of the standard multiple-choice format in order to take advantage of the low cost and minimal support needed to implement it.

For educational research, however, the standard format is unacceptable. Students guess; some of them guess correctly. Some are confident in their answers; others are not. With the standard format, we cannot distinguish between these states. Open-ended formats are therefore preferred, but open-ended questions and answers can be extremely time-consuming both to administer and to analyze.

_Every Decision Counts_ (EDC) is a compromise between open-ended and standard, multiple-choice formats. Students are allowed to mark more than one choice on a standard 5-bubble answer sheet, which has two consequences. The first is that students can communicate their confidence in their answer to a standard question by selecting two or more incompatible choices. The more marks they make, the less credit they earn and the less confidence they have in their answer. If they fill in all 5 bubbles, they are "just guessing."

The second consequence is that we can ask questions with possibly more than one mark in the correct answer. There are 31 possible combinations of answers, and therefore it is almost impossible for students to guess the correct one. Possible scores are 0, 1, 2, 3, 4, and 5, so there is a more precise categorization of students between completely correct (5/5), and completely incorrect (0/5). We also have a just guessing category for students who mark all 5 choices. As compared to standard multiple-choice questions, a lower percentage of students are completely correct (earning all 5 points), yet the average score tends to be noticeably higher, because nearly all students are earning at least 3 points (out of 5). In other words, in practice, almost nobody earns 0 or 1 point, and few earn only 2 points.

We would like to study the accuracy of EDC by comparing students' answers in this format to their answers in a more open-ended format. We would also like to study the potential of EDC for measuring problem-solving proficiency, which cannot currently be measured with the standard multiple-choice format. In 2006 or 2007, we anticipate being able to secure funding to study EDC, as well as identify and analyze question styles, develop and make available a database of exemplary questions, and work toward wider implementation of EDC by teachers and researchers who use multiple-choice assessments.

As a public service, we have created a website for instructors who want to use EDC. It includes web-based tools for making the scoring process simple.

EDC is a spin-off from our RRA project.

ConMap

root — Sat, 12 Jan 2008 23:34:28 +0000

Computer-based assessment tools for probing physics students' conceptual knowledge structures

Group Page:

PERG

Contact(s):

Beatty, Ian D.

Starting date:

1995-01-01

Traditional problem-based exams are not reliable tools for diagnosing students' knowledge and guiding pedagogical intervention; new tools grounded in cognitive science and educational research are needed. If one wishes to assess a students' knowledge in detail, rather than merely summarize items on which the student did or did not succeed, one needs a model of what a knowledge state is, instruments for probing a student's state of knowledge, and an understanding of the mechanism by which instruments probe the state. An effective diagnostic assessment must describe a student with reference to some suitably detailed model of physics knowing, learning, and application.

Our ConMap ("Conceptual Mapping") project has two interdependent objectives. One is developing a detailed, quantitative model of physics conceptual knowledge and its application to problem-solving and analysis. The other is creating practical tools to enhance teaching through frequent "formative assessment" and monitoring of learning. We have been investigating the potential of a set of simple, computer-administered term-association tasks for probing the concepts and interconnections within a physics student's knowledge store. By eliciting spontaneous associations between physics terms in a variety of ways and contexts, we hope to build up a map of the concepts within a topic area that a student has access to and of the web of associations providing structure to those concepts. Since experts and novices are known to structure their knowledge in qualitatively different ways - experts hierarchically around key principles with rich interlinking, novices chronologically with sparse interlinking - we hope to detect signatures of expertise in the patterns of associations elicited, and perhaps observe the onset of those signatures as learning occurs.

This project began with Ian Beatty's dissertation work in the 1990's. Since then, it has been pursued through the dissertation work of graduate students Jenny Chang and Edgardo Ortiz.

Preliminary work has established that the approach is "interesting and promising": We have found internal consistency within the data and partial correlations with other measures of knowledge, suggesting that the ConMap probes are in fact measuring something real, reproducible, and relevant. We have also experimented with quantitative dynamical models "explaining" some aspects of the observed data.

The original project is described in Ian's dissertation (2000) and, partially, in a paper published in 2002 the American Journal of Physics. Later work has not been written up (yet).

RRA

root — Tue, 27 Nov 2007 19:27:08 +0000

Researching the Role of Qualitative Analysis

Group Page:

PERG

Contact(s):

Leonard, William J.

Funding:

US National Science Foundation grant ROLE-0106771

Starting date:

2001-09-01

RRA was a research project on the combined impact of qualitative analysis and reasoning activities and formative assessment on the attitudes, conceptual understanding, skills, and problem-solving proficiency of introductory college physics students.

Too often, students in introductory college physics develop superficial problem-solving approaches - often manipulating equations and avoiding concepts. Meanwhile, their instructors generally have lofty goals for them, such as developing deep conceptual understanding and forward-looking strategic approaches to problem solving, yet somehow the day-to-day activities seem to discourage these outcomes from ever happening.

Analysis activities, in which students use concepts to reason and answer questions about problem situations, are being used as a bridge between the current reality of physics instruction and the desired state of helping students develop transferable thinking skills. The approach, called Analysis-Based Problem Solving, has been used by UMPERG members for many years, but only recently have we been funded to study the approach and its effect on student attitudes and skills.

The data consists of surveys (weekly attitudes, biweekly pretests, and end-of-semester posttests), performance (exams and quizzes using the EDC format; eHW), and hit logs (eHW and homeworkCentral). The goal is to classify students in terms of their prior knowledge and skills, as well as level and style of engagement, and then look for correlations of these variables with student attitudes and an array of performance indicators, such as conceptual understanding, analysis and reasoning ability, and problem-solving proficiency. The Every Decision Counts (EDC) format for multiple-choice assessments and the homeworkCentral (hwC) web site are critical elements of the project. EDC is a format invented to given us more precise performance data, and hwC is a web site created to allow us to monitor student engagement while working on electronic homework.

ViSoR

root — Sat, 24 Nov 2007 19:01:14 +0000

Advanced visualization tools for understanding statistics

Funding:

US National Science Foundation grant REC-0106654

Starting date:

2001-09-01

This project addresses the growing importance of data literacy as a fundamental skill for living in a democratic society and the disheartening fact that few people have a solid understanding of data. It addresses this need by studying how advanced visualization tools can affect teachers' and students' develop understanding of several crucial statistical concepts.

The project focuses on how people understand distributions of data, how they compare two groups of data, and how they think about convariation; it examines the ways in which powerful visualization tools can facilitate the learning of these concepts. As a collaboration between educational software developers and educational researchers, the project takes advantage of the expertise of both groups in order to:

develop a research foundation that elucidates teaching/learning processes in the area of statistical covariation;
develop a set of design principles for statistical education tools that best support statistical learning;
use the largely untapped design expertise of commercial software designers in educational research; and
leverage NSF's investment in educational software development.

The ultimate goal is to accelerate the development of both statistical education research and software in ways that would be impossible without such a collaboration.

This project was supported in part by a grant from the National Science Foundation (REC-0106654). Opinions expressed here are those of the project staff and do not necessarily reflect the views of the Foundation.

Data Modeling

Jang Kreetong — Sat, 24 Nov 2007 17:26:41 +0000

Constructing data, modeling worlds: Collaborative investigation of statistical reasoning

Group Page:

SERG

Funding:

US National Science Foundation grant REC-0337675

Starting date:

2004-02-15

This project aimed to change the way students -- and teachers -- think about math and science and was part of a larger endeavor by Peabody College to "reform the schooling of mathematics and science," said co-investigator and Professor of Science Education Rich Lehrer. This innovative project focused on learning rather than performance as the standard by which educational methods are judged. "The goal is to support teachers who can look at how kids are thinking and adapt instruction accordingly," said Lehrer. At the new math and science-focused Rose Park Magnet Middle School in Nashville, investigators used model-based reasoning to help teachers design a learning environment that was more in tune with "real world" math and science. These are not always familiar concepts to teachers, Lehrer said, and a key of the project was to "introduce ideas in a way they'll find fruitful."

At the heart of the investigation was the relationship between statistics (math) and biology (science). The discipline of statistics developed from a need to model natural systems using mathematical descriptions -- for example, measuring variation in organisms over time -- so it seemed only natural to reunite the two fields for an examination of how statistical reasoning develops. "We know, for example," said Lehrer, "that if you look at plants and take a measure of something like height, you'll find a lot of variation. There's a structure to that variation," and that's where statistics comes in. "If you take a group of plants, or of people," he continued, "and ask, what would happen if I grew them again? What would that look like? -- mathematical structures help you predict nature."

At the University of Massachusetts Amherst, we created new capabilities with TinkerPlots that help students develop statistical reasoning and learn new ways of representing data that don't exist except in a computer world. These tools aided investigators and teachers in developing lesson plans throughout the course of the three-year project. While development would be ongoing, teachers began introducing statistical concepts to students in fall 2004.

Assessment played a vital role in the project, though not in the traditional fashion: Evaluations of students' learning would serve as instructional tools for teachers. "If you're going to assess a kid," said Lehrer, "the assessment ought to inform instruction -- tell you how a kid is thinking about a set of problems so you know what the next step is." Researchers from the University of California-Berkeley helped develop techniques to assess students' learning. One of the most innovative aspects of the project involved not educational design or assessment, but the collaboration between investigators. Rather than dividing research into separate segments for each investigator to research independently and then combine at the conclusion of the project, all team members participated in multiple aspects of the investigation. "It's unusual to collaborate in this way," said Lehrer, but the result -- cohesive research that produces more comprehensive results -- may someday make it the standard.

This project is supported in part by a grant from the National Science Foundation (REC-0337675). Opinions expressed here are those of the project staff and do not necessarily reflect the views of the Foundation.

Model Chance

Jang Kreetong — Sat, 24 Nov 2007 16:56:02 +0000

Simulation software and classroom activities to help middle school students understand probability

Group Page:

SERG

Funding:

US National Science Foundation grant ESI-0454754

Starting date:

2005-05-01

Model Chance was a project funded by the National Science Foundation to develop simulation software and classroom activities to help middle school students learn about probability and data modeling. The simulation tool was integrated into our data-analysis software, TinkerPlots and was eventually released as TinkerPlots version 2.0. One of the needs the project responded to was that at the time curriculum materials on probability conveyed little sense of the importance and range of applications of probability, and that prior to high school we treated probability and data analysis as separate strands. The new software and materials allowed students to investigate real-world problems, such as the probability of false positives from medical screening tests, the probability of injury from repeated exposure to various risks, and chance-based processes, such as evolution and diffusion.

The project team included researchers from 10 universities. Materials had been field tested at the Lynch Middle School in Holyoke MA, and Rose Park Middle School in Nashville TN.

This project was supported in part by a grant from the National Science Foundation (ESI-0454754). Opinions expressed here are those of the project staff and do not necessarily reflect the views of the Foundation.

A2L Project

root — Sat, 27 Oct 2007 01:55:51 +0000

Assessing-to-Learn: Continuous Formative Assessment for Physics Instruction

Group Page:

PERG

Contact(s):

Gerace, William J.

Funding:

US National Science Foundation grant ESI-9730438

Starting date:

1998-05-01

There has been considerable interest in assessment, especially in view of the goals set forth in reform documents, such as the NRC's National Science Education Standards. Generally, those goals call for science learning to focus on conceptual understanding, problem solving, and science as inquiry. Most agree that there is a mismatch between those goals and summative tests used to rank students on those goals. Formative assessment practices provide a better match, but there are barriers to teachers adopting such practices. We argue that a good first step toward helping teachers adopt a program of formative assessment is the development of (simple-to-use) formative assessment activities and an efficient process of collecting formative assessment data.

Assessing-to-Learn (A2L) is a multifaceted project with the purpose of advancing our understanding and practice of formative assessment in the teaching of high school science. The original goal of the A2L project was to develop quality formative assessment materials (i.e., assessment materials that a teacher can use to make decisions about subsequent instruction) for teaching high school physics. Lessons learned during the development and pilot testing of formative assessment activities led us to broaden the scope of the project to address other significant issues (i.e., other than the lack of quality formative assessment materials) affecting the adoption and implementation of formative assessment practices, including the following issues:

How can one effectively integrate assessment and instruction to meet the wide variation in teaching practices among high school science teachers?
What factors influence a teacher's adoption of particular formative assessment practices?
What kinds of support do teachers need to implement a program of formative assessment?
How do we enable teachers to become authors of formative assessment activities?
What preparation do teachers need to be effective practitioners of formative assessment?
What is the effect of formative assessment practices on the classroom dynamic and the way teachers teach?

We have identified eight tangible outcomes for the A2L project (listed on the next page). Seven of the eight outcomes are framed in terms of a product that supports some aspect of formative assessment (i.e., classroom practice, teacher education, development of assessment materials, etc.). The remaining outcome concerns addressing certain research questions related to formative assessment — chosen specifically because we can make a contribution to these questions given the other goals of the project and the available resources.

A set of prototype formative assessment items (aimed at introductory high school physics) for use with a classroom communication system.
Teacher aids containing answers to the assessment items and other useful instructional information.
A design paradigm for formative assessment items.
Strategies for classroom formative assessment.
A guide to formative assessment for practicing teachers who are interested in a practical introduction to formative assessment and for teacher educators who wish to include a component of formative assessment in their teacher education programs.
Design of an on-line course.
A2L web site providing access to all published A2L materials.
Research improving the knowledge base on teacher adoption of classroom formative assessment.

As part of the project, we created a library of formative assessment "items" — multiple-choice questions for use with classroom response system teaching — designed as a vehicle for classroom interaction and learning, rather than for testing what students already know. This library is available to the public on the Assessing-to-Learn project website (A2L.physics.umass.edu or clickercentral.net). This site has a new location as of May, 2012.

The A2L project directly motivated and informed a current large project, Teacher Learning of Technology-Enhanced Formative Assessment.

TinkerPlots Project

konold — Sat, 27 Oct 2007 01:48:39 +0000

Developing tools and curricula for enhancing data analysis in the middle school

Group Page:

SERG

Contact(s):

Konold, Cliff

Funding:

US National Science Foundation grant DRL-9818946

Starting date:

1999-07-01

In the TinkerPlots project, funded by the National Science Foundation, we created a software tool and curriculum materials for teaching data analysis and statistics in middle schools. The software we developed, called TinkerPlots, offers a construction set rather than a menu of ready-made graph types, and helps orient students and teachers to the inquiry-driven nature of data analysis. The development of TinkerPlots was done in collaboration with teams who developed comprehensive instructional materials for middle school mathematics.

The completed software package, named TinkerPlots, is commercially available from Key Curriculum Press.

This project was supported in part by a grant from the National Science Foundation (DRL-9818946). Opinions expressed here are those of the project staff and do not necessarily reflect the views of the Foundation.

TinkerZeum

root — Sat, 27 Oct 2007 01:48:10 +0000

Involving museum visitors in data analysis explorations

Group Page:

SERG

Funding:

US National Science Foundation grant ESI-0437307

Starting date:

2004-09-01

Funded by the National Science Foundation, the Tinkerzeum Planning Project was a one-year collaborative project exploring the feasibility of involving museum visitors in data analysis. These studies, which took place at the Museum of Science in Boston and at the Naismith Basketball Hall of Fame in Springfield MA, have helped us understand the kinds of data and exhibits that lead to compelling museum investigations and the types of additional supports visitors require to begin exploring data.

To learn more about the project you can download and view the movie "Engaging Museum Visitors with Data". This seven-minute movie shows visitors investigating reaction-time data collected at the Museum of Science in Boston and at the Naismith Basketball Hall of Fame in Springfield MA. The software they are using is TinkerPlots, a tool we developed and currently publish.

download movie: large (43.3 MB), small (10.4 MB)

To view the the movie you must have QuickTime Player installed. The player can be downloaded free of charge from Apple.

In our final test at the Museum of Science, visitors spent 11 minutes on average exploring data with us at the computer. To engage visitors in this way, we learned that it was important to give them ample time to explore the question that initially draws them in: How do I compare with everyone else? In our initial field tests, we had been rushing visitors past this question to get to what we regarded as the more significant questions concerning general trends and group differences. But it was rare that visitors would ask these higher-level questions. We modified our procedure, introducing visitors to the data more slowly, and spending more time letting them see how they compared with other visitors. One important aspect of this initial phase was that it allowed the visitors to better understand the data and the graphs they were looking at. Once they were satisfied with this initial analysis, they showed considerable interest and ability in exploring other questions, which included whether reaction time was faster for males or females, or for young vs. older visitors.

By giving visitors data and tools for exploring that data at sites where they have been drawn by their interests in particular phenomena, we were able to both deepen their involvement in those phenomena and introduce them to the basic ideas of data analysis. These data inquiry skills are becoming critically important as increasingly more data become available to us and as we insist that all our institutions make objective decisions based on data. Our effort is part of a broader movement among educational institutions, including museums, to involve learners in active exploration. It also responds to the fact that while science topics are now creatively explored in many museum exhibits, museum mathematics is still too frequently a hands-off experience.

This project was supported in part by a grant from the National Science Foundation (ESI-0437307). Opinions expressed here are those of the project staff and do not necessarily reflect the views of the Foundation.

TLT

Jang Kreetong — Sat, 27 Oct 2007 01:39:05 +0000

Teacher Learning of Technology-Enhanced Formative Assessment

Group Page:

PERG

Contact(s):

Beatty, Ian D.

Contact(s):

Leonard, William J.

Funding:

US National Science Foundation grant ESI-0456124 (TPC)

Starting date:

2005-07-01

TLT was a five-year research project studying how secondary science and mathematics teachers learn to use an electronic "classroom response system" to implement a specific pedagogical approach called Technology-Enhanced Formative Assessment (TEFA). Background: Classroom response systems (CRSs) were technology that helped an instructor poll students' responses to a question, displaying a graphical chart of the class' aggregated answers. These systems, although simple in concept, could have a beneficial and even transformative effect on instruction.* TEFA was a pedagogy that Gerace, Leonard, Beatty, and colleagues had developed over 15 years to support effective science teaching with a CRS. Teachers at the university, high school, and middle school levels had succeeded with TEFA, but mastering it was often challenging and time-consuming, and took extensive support.

Objectives: The TLT project was designed with three goals: (1) to better understand teacher learning of CRS technology and TEFA, and consequent changes to their practice; (2) to better understand effective and efficient methods of teacher professional development in TEFA; and (3) to develop tools and techniques for the evaluation of teachers' TEFA mastery, of suitable design and quality for use in a controlled, randomized study of the effects of TEFA on student learning.

Professional Development: Project staff had conducted (and continue to conduct) intensive, sustained, on-site professional development (PD) programs for 40 middle- and high-school science and math teachers at six schools in three Western Massachusetts school districts. PD focuses on use of CRS technology (provided to the teachers by the project), practice of the TEFA pedagogy, development of supporting curriculum elements, and attendant teaching issues. It began with a four-day summer workshop, continued with a year of weekly or biweekly after-school meetings, and ended with one or two years of monthly after-school meetings. The PD was itself conducted according to the TEFA approach.

Research: The project used a longitudinal, delayed-intervention design. Data was collected via several channels, including interviews and regular questionnaires for participating teachers, surveys for their students, videotaping of classes being taught, and video- and audio-taping of professional development meetings. Analysis was mixed-methods, focused on detailed, heavily triangulated case studies and cross-case analysis. Significant new instrumentation had been developed, tested, and refined during the course of the project.

Outcomes: Project staff had compiled detailed case studies of four participants, with partial profiles of several others, and had developed an initial model of teacher learning and pedagogical transformation called the "model for the co-evolution of teacher and pedagogy." A TEFA PD program had been developed and iteratively improved, and the TEFA pedagogy had been more clearly articulated, defended, elaborated, and disseminated. Experiences, methods, and preliminary results had been presented at several different professional conferences.

* For references, contact Ian Beatty (idbeatty@uncg.edu).

The TLT project is funded primarily by grant TPC-0456124 from the National Science Foundation. Any opinions, findings, conclusions, and recommendations expressed here or in other project publications are those of the principal investigators and do not necessarily reflect the views of the NSF.

Additional project support has been provided by InterWrite Learning (now owned by eInstruction), makers of the PRS-RF classroom response system.