Writing Assessment Case Study
Imagine a hypothetical random large university that is interested in improving the writing skills of its undergraduate students. So, a task force of writing experts is assembled with the goal of examining student writing samples and identify specific areas that need improvement. A rubric is created and hundreds of artifacts are collected from a common course. The assessment process is rigorous and employs all the hallmarks of ‘gold standard’ experimental or quasi-experimental research.
After completing their examination of the student writing samples, the task force publishes several articles and makes several presentations at a variety of conferences. The research methods were flawless, and the task force members were lauded for the quality and ground-breaking conclusions of their research.
At the end of the three-year task force, however, little to no changes in the curriculum have been made, and a further evaluation of the program shows that student writing has seen little to no improvement.
So, what happened? This brief case study illustrates what can happen when there is a lack of clarity about research purposes and goals. I categorize research into two types: empirical research, which is driven by curiosity, and action research, which is more oriented towards pragmatism and improvement.
Empirical Research vs. Action Research
As the figure below shows, the primary distinction is that action research is oriented towards making decisions and implementing change, whereas empirical research is more oriented towards generalizing, understanding, and theory.
Agricultural Research Example
In the book Evaluation Debates, Carol Weiss relates the story of an experience working in an experimental agriculture station. The researchers in this story generally didn’t like working in a farm setting because there were so many variables they could not control. A lab setting was more controlled, and thus reduced the impact of uncontrolled variables, and more adequately met the needs of researchers.
When the researchers communicated their findings to farmers, however, the data was not meaningful because the research didn’t occur in ‘real life.’ This is a case where the researchers intended the findings to be used by people in a practical setting, but the farmers’ needs and points of view weren’t addressed. This would be fine if the goal of the research was purely empirical in nature, but the point of the project was to influence agriculture decisions.
The Role of Evaluation in Research
Assessment and evaluation are the two most commonly used forms of action research. The main distinction between the two is that assessment is about collecting evidence and measurement, whereas evaluation is driven by making decisions and implementing change. The figure below shows where these two types of research happen.
Returning to the Writing Assessment Case Study
In the writing case study, the task force set up a goal of improving the writing skills of students. They did measure student writing, and even interpreted it in various publications and presentations. Thus, an assessment did occur.
The task force failed, however, in terms of measuring the impact and effectiveness of the program. One cannot assume that a perfectly-designed and rigorous methodology will equate to use. Nor can one assume that just because there is a lot of good data that it will magically translate into practical information via a report or email.
Most people define evaluation criteria (impact, efficiency, use, etc.) in terms that are meaningful to them, not in relation to some objective truth or reality. Additionally, data doesn’t drive. This is because data do not make decisions – people do. Someone or some group of human beings need to make a judgment about the evidence.
Questions to Ask to Ensure Research is Designed for Action
While the task force in this hypothetical case study did a good job of establishing methodological rigor, it neglected several important considerations that are unique to assessment and evaluation research. Some of these include:
- Identifying who is going to use the information.
- Clarifying where discussions about the evidence will be held.
- Identifying who needs to have input on the process.
- Considering whose values are going to be included.
- Identifying who the research is for: the researcher or the practitioner? Or both?
The ultimate value in action research is in whether it is used or not. When evaluating programs, clarifying the research approach in the beginning can lead to an evaluation process that is more efficient in terms of time and resources, more meaningful to the people who make program decisions, and – most importantly – more impactful on the intended beneficiaries of the program.