Evaluation Workshop: The Case for Project Evaluation
Continuing with the theme from the 2011 Winter meeting related to evaluation, this hands-on workshop will focus on 1) why one evaluates data and their associated enabling technologies for discovery and distribution and 2) some basic methods for getting started with evaluating the impact of your own research or products.
On day one we'll start with a short overview of some examples of evaluation and assessment projects, then move into a discussion of some basic methods for understanding two critical elements of evaluation: who are your main stakeholders and how you can interact with them so that you can understand better how your work is being used and what impact it's having. On day two, we'll work through a practical framework from the point of view of Public Value. In the most general sense, public value is linked to individual and societal interests and to the institutional forms and actions of government.
The CTG Public Value Framework uses the concept of public value to help government and those that work with government understand the tangible and intangible values that information can provide to multiple and often diverse stakeholders. The purpose of this workshop is to provide earth science information researchers and government practitioners with tools and strategies to help them better understand the value that the information they work with can or is providing to their various stakeholders. During the workshop, participants will learn about existing CTG methods and tools and how they can be used to help support the design, implementation, and evaluation of earth science information projects and ensure that the value these projects can provide is identified, communicated, and assessed.
Workshop sponsored by the Decisions Cluster and conducted by Brian Burke, CTG at SUNY Albany and Dan Ferguson, University of Arizona.
Day 1: Thursday, January 5th, City Center Ballroom
2:00 – 2:15PM ESIP Welcome
2:15 – 3:00PM Agenda and Goals for the Workshop
• Climate Assessment for the Southwest (CLIMAS) – Dan Ferguson (15 min)
• Center for Technology in Government – Brian Burke (15 min)
• Participant Introductions and Workshop Expectations (15 min)
3:00 – 4:00PM Making the Case for a Social Science Approach to Project Evaluation
• Evaluating the use of climate information by wildfire managers Case Example – Dan Ferguson (30 min)
• AIRNow-I Shanghai Case Example – Brian Burke (30 min)
4:00 – 4:15PM Break
4:15 – 5:30PM Understanding Stakeholders and Their Interests
• Stakeholder Identification Strategies – Brian Burke (15 min)
• Stakeholder Identification Strategies – Dan Ferguson (15 min)
• Small Group Exercise and Facilitated Discussion (45 min)
Day 2: Friday, January 6th, City Center Ballroom
8:30 – 9:00AM Review of and Reaction to Previous Day’s Work
9:00 – 10:00AM Public Value Framework Presentation – Brian Burke (40 min and 20 min Q&A)
10:00 – 10:15AM Break
10:15 – 12:00PM Mapping Stakeholders to Public Value
• Small Group Exercise where each group picks one project that a participant is working on and uses worksheet and instructions to identify stakeholders and identify the value delivered by their use of the earth science information provided. Each group will capture highlights of their work on a flip chart and be prepared to report out. Also be prepared to answer the following question: What if anything is currently preventing you from identifying stakeholders and identifying the value delivered to them by the earth science information you are providing. (60 min)
• Large Group Report Out and Discussion where each of the small groups shares the results of their exercise (45 min)
The workshop featured presentations by Brian Burke (Center for Technology in Government) and Dan Ferguson (U. of Arizona) who provided examples in project evaluation, followed by presentations and hands-on activities led by Brian Burke on the Public Value Framework
Hands-on exercises in project stakeholder mapping and identification of value types were particularly useful to attendees and generated discussion on how to identify, prioritize, and engage stakeholders and its significance for developing project metrics. There was also discussion on the Logic Model used in formal Project Evaluation (see links bellow) used in developing evaluation questions, and resources for using the model and/or finding professional evaluators such at the American Evaluation Association and the Environmental Evaluators Network (EEN). EEN has an annual event in D.C and has offered daylong evaluation workshops in the past (a few of us attended last year and found it very useful); all were encouraged to attend the 2012 event.
The ESIP January 2012 Evaluation Workshop was attended by EPA, NASA, NOAA and private sector entities, but it was clear that all were united by common theme/needs: 1) Learning how to identify project metrics and 2) methodologies/techniques for data collection and analysis that enable the evaluation. 3) How to effectively communicate these results to funding agencies and the public.
We then talked about 1) Developing a set of examples and Case Studies in evaluation as a resource to the ESIP community 2) A session at the next Summer or Winter ESIP meeting to include presentations within the ESIP membership who are currently (or newly) engaged project evaluation to share methodologies and HowTo’s with the ESIP community.
You can find the Workshop Evaluation presentations on the ESIP Decisions Wiki:
3) Evaluation Handbooks
NSF 02-057, The 2002 User-Friendly Handbook for Project Evaluation
From Kellogg Foundation
center/resources/2006/02/WK- Kellogg-Foundation-Evaluation- Handbook.aspx
CDC Evaluation Resources http://www.ams.usda.gov/AMSv1.
4) Logic Model Development Guides:
From Kellogg Foundation
center/resources/2006/02/WK- Kellogg-Foundation-Logic- Model-Development-Guide.aspx
From U. of Wisconsin Extension: http://www.uwex.edu/ces/
USDA Logic Model Template: http://www.ams.usda.gov/AMSv1.
5) Resources for finding professional evaluators and learning about project/program evaluation
Environmental Evaluators Network
American Evaluation Association