Michael Scriven

Papers and Publications (Michael Scriven)

 

Scroll down to read my first posted work in progress.
Click links on the left column to see other papers
(you may find it easier to navigate these papers if your browser
 is set to open pdf's automatically).

Again, please send comments to me at mjs@MichaelScriven.info, and I'll assume you don't mind my posting them here if I think of something useful to say about them. Thanks in advance!

THE THREE REVOLUTIONS

Michael Scriven
CGU, PAU, & WMU

 



My take on the situation in evaluation at the moment, in particular in program evaluation which is the kind most of us do, is that we're in good shape along one dimension--level of sophistication of our collective technical skill repertoire (with an important exception)--and in very bad shape in the other dimension of a discipline--conceptual clarity about the nature of the discipline. Moreover, we're mostly not aware of this defect; so let me begin by saying what I think we need to be aiming to achieve in this dimension. I'll only take the time to state my conclusions since it would take a book to give all the reasons for them; and it's more useful simply to lay out the targets and respond to the particular concerns about them raised by those who are interested enough to participate in our forum.

So I think we need to transform our thinking about evaluation to accommodate three revolutions in the way it is conceptualized (by us and those in other disciplines and the general literate public)--actually two and a half, since the first one is about half achieved by about half of us (my only quantitative comment for the day!).

The first revolution is to transform the status of evaluation from untouchable to respectable , i.e., from the days a century ago when the value-free doctrine held that there could be no place for the serious treatment of evaluation within the sciences (or in the company of other respectable disciplines like history, jurisprudence, mathematics, etc.) to the days when even the National Academy of Sciences is doing evaluations at the request of Congress without protest from leading scientific and other professional organizations, and everyone will have good reasons for this acceptance. The problem with this revolution is that the appearances suggest it has occurred, but in fact, the skeleton of the value-free doctrine is still in almost everyone's closet of unconscious beliefs, as one can see from discussions in which it appears that applied social science and evaluation are still taught and texted (and the competencies for evaluation enumerated) without any reference to how one is to identify, or validate, or integrate, the value premises that one logically must have before one can draw evaluative conclusions i.e., before one can actually do valid evaluation.

The second revolution (if we ever get to discussing it after arguing about the first one!) is to transform the status of evaluation from that of a respectable discipline to that of the alpha discipline. The alpha discipline is the one that has the power, in this case the keys to the kingdom of the disciplines. Evaluation has that power because all disciplines are completely dependent for their legitimacy on the quality of their intradisciplinary evaluation, i.e., their ability to identify good vs. bad theories, data, hypotheses, explan­ations, etc., the tools of every discipline's expertise. As the continuing scandals about fake results and bias in anaesthesiology, drug research, arbitrary funding, and effect sizes in general make clear, the classical disciplines are in trouble on this--see the special issue of NDE on research evaluation by Coryn and Scriven, for some details--and evaluation (although not the sub-area of program evaluation) owns that game. {No, Scriven is not just having dreams of glory, he really has reasons for this, and they will probably convince you because your practice already commits you to them.)

The third revolution is to extend the status of evaluation from that of alpha discipline to that of the paradigm applied discipline . A paradigm discipline is one that provides a valid model for other disciplines to emulate in constructing their own. (The difference between an alpha discipline and a paradigm discipline is like that between the Constitution and the Ten Commandments.) Evaluation is the paradigm applied discipline, when it operates as it should, because it shows how to (a) not only construct and fix where needed the foundations of an applied discipline (as in the second revolution), and (b) it exhibits in the good practices of every one of its seven named subareas (product evaluation, personnel evaluation, program evaluation, etc.) how to follow those foundational guidelines in doing the job of an applied discipline, which at least frequently involves answering evaluative questions (e.g., what's the best way to do this. is this approach worth what it costs, etc.) (No, Scriven is not etc.)