This fund (hereafter referred to as 3F) is being set up in the near future but instead of waiting till then and announcing exactly what it is by then legally defined as doing, good evaluation practice suggests we might do some 'preformative evaluation' and improve its eventual form via an interactive process. So please send in your suggestions or criticisms to mjs@michaelscriven.info After the legal entity is created, there will be a further period of refinement, reconsideration, and recruitment of ideas and support, for which minor funding will be available.
Eventual funding is already guaranteed at around the two million dollar mark (in terms of present market values). The present plan is to operate with the interest on the Fund's capital (on current data this will yield about $100,000 per annum), so that it can provide a continuing presence and influence towards its goals; further funding might be sought if suitable applications exceed the resources available, and capital can be used up if the income becomes too small to have the desired effects.
The basic idea of 3F is to provide support for three approaches within the evaluation field that are not the main foci of AEA activities, or of any other organization as far as is known (please send in any corrections and suggestions on this and later points here1):
1. To develop ideas -- which might mean concepts, methods, meetings, symposia, or organizations -- that show promise for significantly accelerating (not merely advancing) the theory or practice of evaluation, either overall; or in one or more of its sub-fields (e.g., product evaluation, personnel evaluation, program evaluation, intradisciplinary evaluation); or in one or more of its application areas (e.g., education, health, international aid, technology R&D). While AEA continues to provide an excellent and indispensable infrastructure for the profession, it has very limited resources for funding research, and none earmarked for efforts of the type described in 2 and 3 below.
2. To give particular support, within the range of 1, to efforts that are not part of the natural development of well-accepted paradigms or models. This means giving strong preference to highly original, revolutionary, maverick, or 'out of the box' ideas that are often dismissed as long-shots (i.e., as having very low probability of success), or as having exaggerated probable impact, by those reviewing funding requests submitted through the normal channels of existing foundations or agencies, from department or center heads to disciplinary associations and state or national review committees. This focus is an attempt to take seriously the implications of the common remark by Nobel laureates that the main flaw they see in standard scientific funding is the natural reluctance of the system to back revolutionary but long-shot proposals when doing so would deprive deserving and less risky proposals of support.2 Some examples are appended below and others are solicited.
3. To give special weighting, in evaluating proposals within the range of 1 and 2, to those whose development shows significant signs of benefitting people or peoples -- in the U.S. or elsewhere -- with serious unmet needs, whether physical, social, economic, or political in other ways. This would of course include many indigenous peoples but also people with less highly publicized needs such as those of children whose education is so constrained by implicit or explicit censorship as to render them seriously ill-prepared for dealing with ideas that are seen as radical, controversial, or subversive.
Here are a couple of examples of hypothetical proposals that currently seem relevant to the goals of 3F: more will be published here or available online if they are sent in and cleared by the director-elect and the advisory panel of experienced and widely representative evaluators who have already agreed to help develop 3F. As that group and the 3F crystallizes, those already included who feel comfortable with its emerging form will be identified for further contact; until then, their commitment is of course tentative and their identification might prove embarrassing if they choose to withdraw. Their assistance so far is much appreciated.
Example A (aimed mainly at criteria 1 and 2). Background: Two extremely valuable reference works in evaluation have appeared recently that serve as reminders of how far even leading evaluators are from agreement about the nature of their profession, despite the great progress that has occurred since the days when the value-free doctrine completely dominated the social sciences. The first is The Sage Handbook of Applied Social Research Methods, 2e, (2009), eds. Debra Rog and Leonard Bickman. It was published when DR was president of AEA, but its huge and excellent contents include almost nothing about evaluation, although evaluation is surely an essential element in the business of applied social research, since a good deal of that kind of research is aimed at answering questions about the best way or better ways to e.g., prevent crime, reduce poverty, educate slow learners, evaluate teachers, etc. The second book is The Handbook of Practical Program Evaluation, 3e, (2010) edited by Joseph Wholey, Harry Hatry, and Kathryn Newcomer. It contains 28 chapters and 700 pages of valuable material, and it (fortunately) contains a good deal about evaluation (including a chapter from Michael Quinn Patton and one about logic models) but virtually nothing about needs or values or how one is supposed to combine empirical research results with values in order to get an evaluative conclusion, such as an answer to one of the 'good/better/best questions' listed above. But practical program evaluation is essentially about answering such questions. This omission is perhaps less surprising when you find that their definition of program evaluation (p. 6) includes no reference to values at all. A third symptom of conceptual weakness is the continuing tendency of evaluators to claim that the difference between research and evaluation is that the former is aimed primarily at generating new knowledge, the latter at assisting decision-making. Both claims involve demonstrably false stereotyping and hence an inaccurate self-concept. It seems plausible to argue that forty years of argument about the nature of evaluation should have led to greater clarity amongst professionals than these examples exhibit, and hence that the pace of progress needs some acceleration.
Given this situation, it might be argued that evaluation as a discipline badly needs to decide and demonstrate more clearly to others that one can't virtually omit evaluation from a supposedly comprehensive reference work on how to apply social research, or the logic of evaluation from a handbook on how to apply evaluation. So a small proposal to 3F might request support for a doctoral level thesis or research monograph that undertakes to establish that these omissions result in serious weaknesses in one or both volumes, perhaps by showing that they fail to live up to their titles, e.g., by depending on many evaluative premises or conclusions that need support using methods they do not address. Or another proposal might request funding for a conference of editors and methodologists from evaluation and from the leading social science journals to consider guidelines that would hasten the ending of these residual echoes from the value-free days.
Example B. (aimed at criteria 1-3). Background: Consumers Union (the publisher of Consumer Reports) was founded in the Great Depression by an engineer and an expert on critical thinking, and it was tied to the union movement with the thought in mind that good product evaluation would help unemployed or underpaid workers get more or better goods for their very limited money. Consequently, the products reviewed included such products as cheap clothing and canned foods. For years CU was seen as an enemy of free enterprise (!) and was indeed blacklisted by the big newspaper publishers, so it could not advertise; it was even listed as a subversive organization on the Attorney General's list at the beginning of the second world war. Lately, despite its indispensability, it has been making some mistakes that involve serious errors in product evaluation (e.g., the child car seats and the hamburger reviews), and there is somewhat limited coverage of what our unemployed fellow-citizens most need help in evaluating, and a somewhat limited effort to get its contributions made accessible to them. It continues to hire engineers and statisticians, but no trained evaluators. Given this situation, it would be good to see proposals to 3F aimed at changing CU's focus or their skill set and quality control standards, perhaps by cooperating with AEA members interested in product evaluation and/or in helping evaluation create virtual income for the poor. Or they could be encouraged to submit a proposal themselves for some efforts in these directions using a grant in the four- or five-figure region.
Other examples of possible proposal topics, just as targets for discussion: (iii) A direct attack on the serious weaknesses in the evaluative underpinning of mainstream science that have been revealed in the recent scandals in anesthesiology and drug research, that consists in reviewing existing ways to prevent recurrences, and developing new approaches. (iv) Work on the psychology of evaluation aimed at e.g., developing training materials such as simulation games that demonstrably improve practitioner interpersonal skills; or perhaps aimed at the treatment of evaluation practice as (in part) a branch of rational-emotive psychotherapy. (v) The idea that competency requirements for evaluation licensure should, as with plumbing and medicine, involve closely supervised and fully certified hours of practice of the order of N thousand hours (as for plumbers and electricians), the supervisor(s) being (a) paid by the apprentice for this service; (b) previously licensed, (c) partially legally responsible for any subsequent technical incompetence of the apprentice. (vi) Developing a methodology for better design and use of the consumer product evaluations now appearing on vendor websites, which represent a huge new frontier for evaluation in the service of public enlightenment, where any improvement would pay off on a large scale.
Procedural issues for 3F:
(i) Management, legal. The current plan is for the Marin Community Foundation (MCF) to administer 3F (probably as a 501(c)3 organization), which keeps the overhead costs much lower than if 3F was set up as a foundation in its own right. We'll probably ask AEA to cooperate by (a) appointing a presidential representative, preferably a governing board member, to the 3F advisory board, since the 3F effort is complementary to AEA purposes; and (b) by allowing 3F space to deliver an annual report at the AEA annual meeting (subject to continuing approval of 3F's quality and impact); and (c) allowing announcements of 3F funding availability and awards to appear in their newsletter (subject to the same approval).
(ii) Management, practical. The director would probably use a small executive committee of board members chosen by him/herself with assistance from the board for some matters, and all of the board, in rotating groups, for proposal evaluation and major changes of direction within the general spirit outlined above. The board would elect replacements for any resignations, including that of the director, with the director having one vote ex officio, and a deciding vote in case of a draw. The director would charge for his/her time at up to his/her normal consulting rates, as s/he judges appropriate, within a limit of 20% of the income, if possible, with overage only if approved by the board.3 Other management expenditures should be limited to 10% of the income, also capable of being increased only with board approval. Investment changes are also possible subject to approval by MCF and the board.
(iii) Management, ethical. Transparency in announcements, procedures, activities, and expenditures should be a high priority, and an annual report to AEA would be published online, and at the 3F site and/or in JMDE and/or on the AEA site, and discussed at an open meeting chaired by the director, at the AEA annual meeting. The founder will not serve on the board, or have any other status with 3F, or receive funding from 3F, once it is set up. (iv) Proposal selection for funding. Proposals would be encouraged from all over the world, would be fully blinded (with disqualification if authorship is inferrable by any reviewers); would be rated by paid and calibrated raters; with follow-up to rate the raters. (v) Board members and the director would be able to apply for funding by recusing themselves from choice as a rater in the cycle in which they apply and (if necessary) being replaced pro tem. (vi) Grants extending for more than a few months would be payable in three equal installments: one after proposal approval and recipient commitment, one at mid-point on evidence of approximately proportional completion, and one after a final report demonstrating achievement of substantial progress as promised. (vii) Funding and management expenditures would normally be restricted to 110% of the year's income, while averaging 100% across three years starting with the year of overfunding. (viii) Funding would always depend on the submission of high quality proposals meeting 3F requirements, and hence might be zero in a given year, in which case the next year could involve funding up to two year's income plus interest. (ix) Further funding of 3F will be encouraged from donors sympathetic to the 3F mission, for specific or general use, and sought from granting institutions for particular projects. (x) Cooperation with other organizations will be encouraged, and can be funded. (xi) All the above, including the examples, are subject to refinement, removal, supplementation, or alteration by the board. While they involve an approach that is probably not universally accepted within the profession, it is perhaps one that is worth trying out -- and that is the basic justification for 3F's creation. [2555 words @24.jan.2011]
Michael Scriven, Founder
------------------
1The 2010 review of the Bill and Melinda Gates Foundation five year effort that involved putting half a billion into global health research, often backing long-shots (NYTimes 12.21.10), was studied with some care when designing 3F. They backed 43 out of 1600 submissions, each with an average support level of $11m. The results were significant but mostly very disappointing against the expectations of both the foundation and the researchers. Most notably, about half-way in, learning from experience, the Gates switched their new funding to much smaller, much-longer-shot grants (each being less than 1% of the original average, i.e., $100,000). In the target area 3F will explore � methodology and theory � the costs of research are typically only a few percent of those in health research, so downward scaling of funding by even an order of magnitude should still be compatible with valuable payoff. Grants will normally be intended to have �second-job� or consulting-work status, not full-time income replacement.
2There is no suggestion here that existing review panels are at fault in passing over long shots or projects with rather debatable impact: it is the present system, including the instructions to reviewers, not so much the practice within it that is at fault. It seems desirable to adjust the system to improve the optimization of funding, and several ways of doing this have been suggested, for example, a 5% (or 2% or 1%) set-aside of funds by each funding agency, this amount to be allocated with less negative weighting of the low probability of long shot/high payoff applicants. The Faster Forward Fund is a micro-funding pilot aimed in that direction, and may perhaps stimulate emulation.
3The figures here are suggested based upon present best estimates of a future income around $100,000.
