Critical Barriers
This study was funded under the Small Grant for Exploratory Research Program at NSF. Researchers at the Universities of Massachusetts, Bielefeld, and Dortmund (see staff) independently analyzed a set of videotaped interviews of four students engaged in data analysis. The students had just finished a year-long course in probability and statistics at Holyoke High School (taught by Al Gagnon) in which they had performed at about the class average. Our interest was in exploring difficulties associated with doing fairly rudimentary data analysis by students who had had some introduction to statistics and had access to and familiarity with a data-analysis tool.
The interview comprised a series of open-ended questions concerning data the students had explored during the course. The data included information on 154 students in various mathematics courses at the school during that and a previous year; the 62 variables included information about age, gender, religion, job status, parents' education, stance on abortion, and time spent on a number of activities including studying, TV viewing, and reading. In both the course and the interviews, students worked in pairs using the data analysis software, DataScope®.
Results of our analyses of the interviews, which are summarized in Konold, Pollatsek, Well and Gagnon (1997), Biehler (1997), and Steinbring (1996), suggest at least three interrelated domains that cause particular problems for students learning data analysis. We describe these in the three sections below.
Group Properties
The most striking feature of the interviews was the limited ability of students to use and reason about group properties, such as percentages, means and medians. Typically, when reasoning about a single group, students spontaneously used these measures with what appeared to be good understanding (e.g., they used and reasoned intelligently about the divorce rate among parents of the students in the sample). In contrast, when they were comparing two groups (e.g., trying to discover whether boys or girls were more likely to have a driver's license) or exploring a relationship between two quantitative variables (e.g., whether there was a relation between time spent doing homework and course grades), they never spontaneously made use of appropriate group measures. Rather, they reverted to comparing the absolute frequencies in individual cells (e.g., comparing the number of boys and girls who had licenses, making no use of the number of boys and girls who didn't have licenses). Moreover, even though the appropriate statistics (e.g., the percent of boys who had licenses and the percent of girls who had licenses) were usually prominently displayed in the statistical display they were examining, the students did not switch to using them even after the interviewer suggested that the percentages might be relevant.
Although we do not fully understand this striking phenomenon, we are confident that it indicates an important problem, a major part of which is a lack of understanding that meaningful comparisons have to be of group tendencies rather than of properties of individuals within the groups. Interrelated barriers to using group tendencies include:
not understanding the concept of "variable." This is a difficult notion for many students, but critical for using group measures. For example, when comparing boys and girls in terms of driver's licenses, there is an implicit variable which includes the states of having a license and not having a license.
not understanding the imperfect yet lawful interrelation between group properties and individual observations. On the one hand, students need to see that statistical summaries such as medians, percents, standard deviations, and histograms, describe emergent group properties for which there may not be corresponding properties in the individuals (e.g., the fact that height is relatively normally distributed in the adult population could not be deduced from looking at the height of any one individual). On the other hand, they must also develop the understanding that these group properties are not arbitrary in that they both stem from relatively simple computations from individual data points, and also can often be related back to individual observations (e.g., if the median hours of study for students who get A's is higher than that for students who get C's, this says something about the behavior of individual students).
not having an appropriate model in which lawfulness and "randomness" can coexist. Statisticians have conceptual models of repeatable "random events" such as urns, coins, and dice, and models of "true values" plus or minus "random error". In contrast, novices usually have no conceptual model that explains variability, and instead rely on individual observations to draw conclusions (the "outcome approach," Konold, 1989).
Formation of Statistical Questions
In data analysis, a general question about the real world is transformed into an empirical question. This transformation guides the choice of data, coding decisions, data organization, how data are queried, and how observed patterns are related back to the original real-world question. Students in our interviews showed little awareness of the simplifying decisions inherent in this modeling process, and we expect that their tendency to not recognize the limits of their conclusions or those of others, and to not explore multiple possibilities during data analysis, stems in part from their not being fully aware of this transformation.
Causal Thinking
The belief seems widespread that results of analyzing observational data can be interpreted directly in causal terms. In the interviews, students' formulations of problems they wanted to investigate were typically framed in causal terms (e.g., "If you have a curfew it makes you ..."). Similarly, we find myriad causal relations in the media where observational studies are quoted in a way that seemingly support causal claims. Didactical constraints may be partly responsible in that students are often given observational data and are asked to investigate the "effects" of one variable on another. This language, which for the teacher may be just a manner of speaking, may be interpreted by the students as meaning "effect" in a causal sense. Furthermore, common language and the idea of a sequence of causes acting on individual cases do not provide adequate grounding for qualitative modeling in multivariate statistical situations.
In summary, our analyses suggest that the barriers we have uncovered in students' attempts at exploratory data analysis are remarkably parallel to those in understanding more advanced statistical concepts. Central to any adequate understanding of data are the ideas of "group measures" (such as medians, percents, inter-quartile ranges) together with their strengths and limitations, the imperfect relationship between verbal statements and statistical formulation, and the very imperfect relationship between intuitive causal thinking and statistical relationships.
The TLT project is funded primarily by grant RED-9452917
from the National Science Foundation. Any opinions, findings, conclusions, and recommendations expressed here or in other project publications are those of the principal investigators and do not necessarily reflect the views of the NSF.Crtical Barriers Staff
NSF Grant RED-9452917 (1995 - 96)
PI and director of UMass research team:
Clifford Konold, Scientific Reasoning Research Institute, University of Massachusetts
Co-PIs:
Rolf Biehler, Senior Lecturer and Researcher, Institut für Didaktik der Mathematik, Universität BielefeldHeinz Steinbring, Professor, Institut für Didaktik der Mathematik, Universität Dortmund
University of Massachusetts research team:
Alan Gagnon, Holyoke High School, Holyoke, Massachusetts
Alexander Pollatsek, Professor, Psychology
Arnold Well, Professor, Psychology
Administrative assistant:
Dorothy Freeman, Scientific Reasoning Research Institute, University of Massachuse
The TLT project is funded primarily by grant RED-9452917
from the National Science Foundation. Any opinions, findings, conclusions, and recommendations expressed here or in other project publications are those of the principal investigators and do not necessarily reflect the views of the NSF.