An Insider’s Take on Assessment Reaction

Every few months, someone writes an article exposing assessment for what it supposedly is: a waste of time and a destructive force. The latest installment is from the Chronicle of Higher Education, An Insider’s Take on Assessment: It May Be Worse Than You Think.

Any discipline should welcome critical thought and reflection. This article definitely provides material for interesting discussion. I have a reaction to three points made in the article:

  1. Assessment as a form of research.
  2. Assessment as a form of control.
  3. Assessment as a conspiracy.

Assessment as a Form of Research

The author quotes an article from Intersection stating “The whole assessment process would fall apart if we had to test for reliability and validity and carefully model interactions before making conclusions about cause and effect.”

Assessment isn’t the only discipline dealing with this. As the replication crisis shows, a lot of empirical research struggles with issues of reliability and validity. Assessment may be no worse than a lot of the social science research in peer-reviewed journals or conference presentations.

In Debates on Evaluation, Mike Patton provides an example to explain the tension between validity and utility. Most agriculture faculty like to conduct research in controlled settings. This is because it enhances validity and reliability.

Most of the time, this is good. Scientific methods and prescribed measurement principles have contributed to advancements in health care, psychology, and how we understand the world. On the other hand, advancements have been made by accident or specifically ignoring the  rules of empirical, “gold standard” research or serendipity.

This isn’t anti-science or anti-intellectual. It’s pragmatic. Is anyone really going to tell an art or music professor with 20 years of experience in their discipline that their assessment of learning is invalid or unreliable because it lacks the “gold standards” of empirical research design? Good luck with that.

As the son-in-law and grandson of farmers, however, I know that any farmer will tell you that controlled settings are always impractical and almost always impossible. Based on the criteria described in the Chronicle article, my father-in-law would be wise to ignore most agriculture research. In fact, based on the logic presented in the Chronicle, most farmers would be wise to avoid empirical agriculture research conducted in controlled settings.

So, of course assessment is going to “fall apart” when testing for validity and reliability, just like a lot of peer-reviewed, empirical studies and probably most social science research. But does that mean my father-in-law should ignore agriculture research because it doesn’t translate into the field? Conversely, do agriculture researchers have nothing to learn from farmers because their observations lack the gold standards of empirical research, like controlled settings, randomized trails, etc.? Does a lack of use of these methods imply

Of course not. That would be ridiculous.

Assessment as a Form of Control

I agree with the author of the Chronicle article on a major point. Here’s the quote:

He (the author of the Intersection article) also seems to be opening the door to a challenge to what is perhaps the single most implausible idea associated with assessment: that grades given by people with disciplinary knowledge and training don’t tell us about student learning, but measurements developed by assessors, who lack specific disciplinary knowledge, do.

I have never in my career met anyone would tell a faculty member how to measure learning in their class or even program. I would never do that. It sends the message “I have no idea what your job is, but whatever is, you’re doing it wrong.”

The Chronicle article also makes a point about grades and ignoring the judgment of disciplinary experts. I happen think grades are fine at the classroom level. Their utility at the program level is another matter.

Unfortunately, conference attendance and a review of institutional assessment websites makes me believe there are a lot of assessment administrators who do tell faculty how to measure learning. And they are telling them that grades are unreliable measures of learning, even at the classroom level. If the goal is to foster engagement with assessment, telling someone their wrong is a bad strategy.

Assessment as a Conspiracy

The final point from the Chronicle article is that assessment is somehow related to the rise of on-line learning and growth in adjunct faculty. The author does not specifically use the term “conspiracy,” but the theory is that assessment provides evidence of quality in these areas, thus justifying their existence.

This might be a matter of correlation and not causation. Assessment has its roots in psychological research in the 1960’s and 1970’s and the accountability movement of the late 1970’s and 1980’s. Assessment predates on-line learning and massive growth in adjunct faculty.

Is it possible that assessment was co-opted by administrators later? I don’t know. I’ve only worked in “traditional,” non-profit higher education institutions. All I can say is that I have never witnessed administrators place more emphasis on assessing on-line teaching and learning over “traditional” forms, like classroom or lab teaching and learning, as a scheme to justify lower costs.

If assessment is an effective strategy for articulating the benefits of dubious educational practices, then why not use assessment to articulate the benefits of “legitimate” educational practices? I would certainly support that approach, and I don’t know what’s stopping people from doing the latter.

A Way Forward

The Chronicle article didn’t really offer a lot in terms of positive solutions moving forward. Here some ideas for moving forward:

Give up the fight about grades. Grades are fine at the classroom level. Telling faculty they aren’t insults their disciplinary expertise. As a form of measurement and motivation, grades can be problematic. My assumption, though, is that almost all faculty are pretty good about tying learning outcomes to grades and communicating to students what a grade means, so I don’t know why assessment administrators make such a big deal about them. Grades at the program level are a different story. I have no idea if a student with a 3.0 is more knowledgeable than a person with a 2.86.

Consider ending grids and templates. The point of standardized grids and templates is to give administrators a view of learning and program effectiveness for the institution as a whole. In theory, the components of the template are complied into one document and reviewed. This almost never happens. And if it did, programs and courses are too varied and contextual to make sense out of it.

What if, instead, we asked faculty to pick one thing they care about – critical thinking, art criticism, research ethics, whatever – and asked them to spend a year researching it? And they get control over how the research is formatted and looks. It could be uploaded to an on-line respository for sharing. Faculty are doing a lot of creative and interesting work in the area of student learning. This approach could capitalize on that. This also addresses the rather arbitrary issue of quantity. Over 5-7 years, that’s a lot of research on student learning. Assessment administrators would call this assessment, but “research” would be a less threatening term.

There’s two potential problems with this kind of process. Convincing an accreditation peer reviewer or agency official to take a different approach that relies more on creativity and one outcome a year, and less on standardization and compliance, might be a hard sell. A second problem is change. Organizational processes that have been in place for many years provide certainty. Change is slow, effortful, and uncertain (like learning). It’s easier to fill out a mind-numbing and non-useful grid than to change a process. And it requires less thought.

Distinguish between student learning assessment and program evaluation. Student learning outcomes assessment looks at what students learn and do. Program evaluation looks at what the program does (space allocation, staffing, budgeting, etc.). Student learning assessment projects can be used in program evaluations and reviews. But a program evaluation should not exclusively focus on student learning or be evaluated solely on that criteria. Teaching and learning are two different activities. Faculty don’t have total control over learning, only what they teach. No one should be held accountable for something over which they only have partial control.

Perhaps accountability and compliance can be framed as program evaluation or program review? All of the standards, frameworks, forms, and templates can be used in a program evaluation. Since program evaluation has more to do with evaluation criteria than disciplinary criteria – and who can argue with program evaluation? – make program review the primary accounability and compliance vehicle.

This frees up student learning outcomes assessment. Provide few if any standards or guidelines, four or five at most. Make learning outcomes assessment an addendum to the evaluation. This would respect faculty disciplinary expertise and maybe enhance engagement with the process.

Be prepared for what’s coming next. Unlike higher education, K-12 teachers are held accountable for what students learn and the value they add. K-12 education has been dealing with this side of assessment for decades. It would be naive to think it’s not coming to higher education. Big data sets about faculty productivity and graduate salaries already exist. Value added measures should be next. Someday, people will be able to quantify how much value a faculty member adds to a graduate’s salary and other labor market outcomes. The data’s already there, just waiting for someone to match it.

There’s still a narrow window to get ahead of this if we engage in geniune, simple assessment and dialog about what it means (not two-way monologue, in which two people talk, but no one really listens). Even though the Chronicle article provided almost no guidance in terms of positive next steps, it did offer a chance for reflection and dialog.

One Last Story

In my early days of assessment and institutional research, the focus was on compliance combined with quality improvement processes. It appears like it still is in many places.

Getting faculty to “comply” and submit assessment reports was very difficult. And what was submitted to the assessment committee was subpar and boring. Some of the font-sizes in the standardized grids had 8-point font and were difficult to read.

A few years in, I attended an in-house retention symposium. Faculty and staff presented on strategies for learning and student success at the classroom, program, and institutional levels. I was blown away by the quality of the research and the creativity.

It was all what I would call assessment research, and no one had ever submitted it to the assessment committee. When I asked a group of faculty why they didn’t turn this creative work into the assessment committee, the reply was “well, they never asked, and this work isn’t assessment.”

A vast majority of faculty want to do good work. And they’ll share it, if they are asked in the right way. But I learned that most faculty will do anything you ask them to do, but almost nothing they are told to do. This Chronicle article made me think that perhaps it’s time, as a discipline, to engage in better dialog with faculty and administrators about assessment, make positive changes in how we organize assessment practices, and do a better job at telling these stories to the public, including accreditors and policy makers.

About rlsmith205

Bloomington-Normal, IL
This entry was posted in Illinois. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s