J. Michael Michael Bamberger & Linda S. S. Mabry 
RealWorld Evaluation [PDF ebook] 
Working Under Budget, Time, Data, and Political Constraints

List of Boxes, Figures, and Tables
List of Appendices
Foreword by Jim Rugh
Preface
Acknowledgments
About the Authors
PART I • THE SEVEN STEPS OF THE REALWORLD EVALUATION APPROACH
Chapter 1 • Overview: Real World Evaluation and the Contexts in Which It Is Used
1. Welcome to Real World Evaluation
2. The Real World Evaluation Context
3. The Four Types of Constraints Addressed by the Real World Approach
4. Additional Organizational and Administrative Challenges
5. The Real World Approach to Evaluation Challenges
6. Who Uses Real World Evaluation, for What Purposes, and When?
Summary
Further Reading
Chapter 2 • First Clarify the Purpose: Scoping the Evaluation
1. Stakeholder Expectations of Impact Evaluations
2. Understanding Information Needs
3. Developing the Program Theory Model
4. Identifying the Constraints to Be Addressed by RWE and Determining the Appropriate Evaluation Design
5. Developing Designs Suitable for Real World Evaluation Conditions
Summary
Further Reading
Chapter 3 • Not Enough Money: Addressing Budget Constraints
1. Simplifying the Evaluation Design
2. Clarifying Client Information Needs
3. Using Existing Data
4. Reducing Costs by Reducing Sample Size
5. Reducing Costs of Data Collection and Analysis
6. Assessing the Feasibility and Utility of Using New Information Technology (NIT) to Reduce the Costs of Data Collection
7. Threats to Validity of Budget Constraints
Summary
Further Reading
Chapter 4 • Not Enough Time: Addressing Scheduling and Other Time Constraints
1. Similarities and Differences Between Time and Budget Constraints
2. Simplifying the Evaluation Design
3. Clarifying Client Information Needs and Deadlines
4. Using Existing Documentary Data
5. Reducing Sample Size
6. Rapid Data-Collection Methods
7. Reducing Time Pressure on Outside Consultants
8. Hiring More Resource People
9. Building Outcome Indicators Into Project Records
10. New Information Technology for Data Collection and Analysis
11. Common Threats to Adequacy and Validity Relating to Time Constraints
Summary
Further Reading
Chapter 5 • Critical Information Is Missing or Difficult to Collect: Addressing Data Constraints
1. Data Issues Facing Real World Evaluators
2. Reconstructing Baseline Data
3. Special Issues Reconstructing Baseline Data for Project Populations and Comparison Groups
4. Collecting Data on Sensitive Topics or From Difficult-to-Reach Groups
5. Common Threats to Adequacy and Validity of an Evaluation Relating to Data Constraints
Summary
Further Reading
Chapter 6 • Political Constraints
1. Values, Ethics, and Politics
2. Societal Politics and Evaluation
3. Stakeholder Politics
4. Professional Politics
5. Political Issues in the Design Phase
6. Political Issues in the Conduct of an Evaluation
7. Political Issues in Evaluation Reporting and Use
8. Advocacy
Summary
Further Reading
Chapter 7 • Strengthening the Evaluation Design and the Validity of the Conclusions
1. Validity in Evaluation
2. Factors Affecting Adequacy and Validity
3. A Framework for Assessing the Validity and Adequacy of QUANT, QUAL, and Mixed-Method Designs
4. Assessing and Addressing Threats to Validity for Quantitative Impact Evaluations
5. Assessing Adequacy and Validity for Qualitative Impact Evaluations
6. Assessing Validity for Mixed-Method (MM) Evaluations
7. Using the Threats-to-Validity Worksheets
Summary
Further Reading
Chapter 8 • Making It Useful: Helping Clients and Other Stakeholders Utilize the Evaluation
1. What Do We Mean by Influential Evaluations and Useful Evaluations?
2. The Underutilization of Evaluation Studies
3. Strategies for Promoting the Utilization of Evaluation Findings and Recommendations
Summary
Further Reading
PART II • A REVIEW OF EVALUATION METHODS AND APPROACHES AND THEIR APPLICATION IN REALWORLD EVALUATION: FOR THOSE WHO WOULD LIKE TO DIG DEEPER
Chapter 9 • Standards and Ethics
1. Standards of Competence
2. Professional Standards
3. Ethical Codes of Conduct
4. Issues
Summary
Further Reading
Chapter 10 • Theory-Based Evaluation and Theory of Change
1. Theory-Based Evaluation (TBE) and Theory of Change (TOC)
2. Applications of Program Theory in Program Evaluation
3. Using TOC in Program Evaluation
4. Designing a Theory of Change Evaluation Framework
5. Integrating a Theory of Change Into the Program Management, Monitoring, and Evaluation Cycle
6. Program Theory Evaluation and Causality
Summary
Further Reading
Chapter 11 • Evaluation Designs: The RWE Strategy for Selecting the Appropriate Evaluation Design to Respond to the Purpose and Context of Each Evaluation
1. Different Approaches to the Classification of Evaluation Designs
2. Assessing Causality Attribution and Contribution
3. The RWE Approach to the Selection of the Appropriate Impact Evaluation Design
4. Tools and Techniques for Strengthening the Basic Evaluation Designs
5. Selecting the Best Design for Real World Evaluation Scenarios
Summary
Further Reading
Chapter 12 • Quantitative Evaluation Methods
1. Quantitative Evaluation Methodologies
2. Experimental and Quasi-Experimental Designs
3. Strengths and Weaknesses of Quantitative Evaluation Methodologies
4. Applications of Quantitative Methodologies in Program Evaluation
5. Quantitative Methods for Data Collection
6. The Management of Data Collection for Quantitative Studies
7. Data Analysis
Summary
Further Reading
Chapter 13 • Qualitative Evaluation Methods
1. Design
2. Data Collection
3. Data Analysis
4. Reporting
5. Real-World Constraints
Summary
Further Reading
Chapter 14 • Mixed-Method Evaluation
1. The Mixed-Method Approach
2. Rationale for Mixed-Method Approaches
3. Approaches to the Use of Mixed Methods
4. Mixed-Method Strategies
5. Implementing a Mixed-Method Design
6. Using Mixed Methods to Tell a More Compelling Story of What a Program Has Achieved
7. Case Studies Illustrating the Use of Mixed Methods
Summary
Further Reading
Chapter 15 • Sampling Strategies for Real World Evaluation
1. The Importance of Sampling for Real World Evaluation
2. Purposive Sampling
3. Probability (Random) Sampling
4. Using Power Analysis and Effect Size for Estimating the Appropriate Sample Size for an Impact Evaluation
5. The Contribution of Meta-Analysis
6. Sampling Issues for Mixed-Method Evaluations
7. Sampling Issues for Real World Evaluation
Summary
Further Reading
Chapter 16 • Evaluating Complex Projects, Programs, and Policies
1. The Move Toward Complex, Country-Level Development Programming
2. Defining Complexity in Development Programs and Evaluations
3. A Framework for the Evaluation of Complex Development Programs
Summary
Further Reading
Chapter 17 • Gender Evaluation: Integrating Gender Analysis Into Evaluations
1. Why a Gender Focus Is Critical
2. Gender Issues in Evaluations
3. Designing a Gender Evaluation
4. Gender Evaluations With Different Scopes
5. The Tools of Gender Evaluation
Summary
Further Reading
Chapter 18 • Evaluation in the Age of Big Data
1. Introducing Big Data and Data Science
2. Increasing Application of Big Data in the Development Context
3. The Tools of Data Science
4. Potential Applications of Data Science in Development Evaluation
5. Building Bridges Between Data Science and Evaluation
Summary
Further Reading
PART III • MANAGING EVALUATIONS
Chapter 19 • Managing Evaluations
1. Organizational and Political Issues Affecting the Design, Implementation, and Use of Evaluations
2. Planning and Managing the Evaluation
3. Institutionalizing Impact Evaluation Systems at the Country and Sector Levels
4. Evaluating Capacity Development
Summary
Further Reading
Chapter 20 • The Road Ahead
1. Conclusions
2. Recommendations
Glossary of Terms and Acronyms
References
Author Index
Subject Index

Linda Mabry is a faculty member at Washington State University specializing in program evaluation, student assessment, and research and evaluation methodology. She currently serves as president of the Oregon Program Evaluation Network and on the editorial board for Studies in Educational Evaluation. She has served in a variety of leadership positions for the American Evaluation Association, including the Board of Directors, chair of the Task Force on Educational Accountability, and chair of the Theories of Evaluation topical interest group. She has also served n the Board of Trustees for the National Center for the Improvement of Educational Assessments and on the Performance Assessment Review Board of New York. She has conducted evaluations for the U.S. Department of Education, National Science Foundation, National Endowment for the Arts, the Jacob Javits Foundation, Hewlett-Packard Corporation, Ameritech Corporation, ATT-Comcast Corporation, the New York City Fund for Public Education, the Chicago Arts Partnerships in Education, the Chicago Teachers Academy of Mathematics and Science, and a variety of university, state, and school agencies. She has published in a number of scholarly journals and written several books, including Evaluation and the Postmodern Dilemma (1997) and Portfolios Plus: A Critical Guide to Performance Assessment (1999).