EvaluLand

In this episode, I chat with the authors of The Garden of Evaluation Approaches about their innovative and helpful framework for thinking about evaluation theory and approaches. They each share their personal journeys into evaluation as well as into this work they have been collaborating on for years. We also discuss evaluation theory more broadly, including how to make it more accessible to new and emerging evaluators. About guests: Bianca Montrosse-Moorhead (https://education.uconn.edu/person/bianca-montrosse-moorhead/) is a Professor of Research Methods, Measurement, and Evaluation at the University of Connecticut, where she also directs the Partnership for Evaluation and Educational Research (PEER). As Co-Editor-in-Chief of New Directions for Evaluation and a fervent advocate for evaluation, Bianca has dedicated her career to bridging the space between evaluation theory and practice. Her work encompasses a broad spectrum of contributions, from evaluating various educational and social programs using diverse methodologies to enhancing the professional training of evaluators worldwide. Daniela Schroeter (https://wmich.edu/spaa/directory/schroeter-0) serves as a Presidential Innovation Professor and Associate Professor at Western Michigan University. With a Ph.D. in Interdisciplinary Evaluation and over twenty years of research, Daniela has contributed substantially to developing evaluation theories and methodologies. Her global work has involved diverse sectors, focusing on enhancing the capacity and effectiveness of evaluations. Daniela also co-edits the Teaching & Learning of Evaluation section of the American Journal of Evaluation. Lyssa Wilson Becho (https://wmich.edu/evaluation/directory/becho) serves as a Principal Research Associate at The Evaluation Center of Western Michigan University. Garnering the 2024 Marcia Guttentag Promising New Evaluator Award, Lyssa has made a significant impact in advancing evaluation methodologies, focusing on culturally responsive practices and promoting equity within evaluation processes. Lyssa is a co-Executive Editor for the Journal of MultiDisciplinary Evaluation. Collectively, their innovative work in evaluation spans continents, cementing their reputation as leaders in advancing the practical, theoretical, and methodological facets of the discipline. Contact information: Bianca Montrosse-Moorhead Email: bianca@uconn.edu (mailto:bianca@uconn.edu) LinkedIn: https://www.linkedin.com/in/BMMoorhead/ (https://www.linkedin.com/in/BMMoorhead/) ResearchGate: https://www.researchgate.net/profile/Bianca-Montrosse-Moorhead (https://www.researchgate.net/profile/Bianca-Montrosse-Moorhead) Daniela Schröter Email: daniela.schroeter@wmich.edu (mailto:daniela.schroeter@wmich.edu) LinkedIn: https://www.linkedin.com/in/danielaschroeter/ (https://www.linkedin.com/in/danielaschroeter/) ResearchGate: https://www.researchgate.net/profile/Daniela-Schroeter (https://www.researchgate.net/profile/Daniela-Schroeter) Lyssa Wilson Becho Email: lyssa.becho@wmich.edu (mailto:lyssa.becho@wmich.edu) LinkedIn: https://www.linkedin.com/in/lyssa-wilson-becho/ (https://www.linkedin.com/in/lyssa-wilson-becho/) ResearchGate: https://www.researchgate.net/profile/Lyssa-Becho (https://www.researchgate.net/profile/Lyssa-Becho) Resources mentioned: Project vita (biography or résumé) with links to all free and publicly available resources: https://tinyurl.com/EvalGardenVita (https://tinyurl.com/EvalGardenVita) Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2024). The garden of evaluation approaches visualization. Journal of MultiDisciplinary Evaluation, 20(48), 49–58. https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/1029 (https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/1029) Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2024). The garden of evaluation approaches. American Journal of Evaluation, 45(2), 166–185. https://doi.org/10.1177/10982140231216667 (https://doi.org/10.1177/10982140231216667) Bledsoe, K. L., & Graham, J. A. (2005). The use of multiple evaluation approaches in program evaluation. American Journal of Evaluation, 26(3), 302-319. https://doi.org/10.1177/1098214005278749 (https://doi.org/10.1177/1098214005278749) King, J. (2024) Garden of evaluation approaches with SROI and CBA https://linkedin.com/posts/julian-king-87a015a_cba-sroi-vfi-activity-7201338499776618497-J3-z/ (https://linkedin.com/posts/julian-king-87a015a_cba-sroi-vfi-activity-7201338499776618497-J3-z/) Miller, R. L. (2010). Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31(3), 390–399. https://doi.org/10/fmzjxp

In this week's episode, I talk with Dr. Jennifer Villalobos about evaluator education, the scholar-practitioner model, evaluation careers, and how the current administration is affecting evaluation. Contact information: Jennifer P. Villalobos - jennifer.villalobos@cgu.edu About Dr. Jennifer Villalobos: Jennifer P. Villalobos, PhD, is an Assistant Professor of Practice and Director of the Doctor of Evaluation Practice (D.Eval) program at Claremont Graduate University. As an evaluator, scholar-practitioner, and organizational psychologist, her work focuses on advancing socially responsive evaluation, helping organizations increase their DEI profile, positive psychological interventions, and the intersection of evaluation education and practice. Outside of work, Jennifer enjoys spending time with her husband and three kids, cheering them on in their many sports and theatrical events. She's also a firm believer in the power of community—whether that means mentoring students, collaborating with peers, or just gathering people around good food and conversation. Music by Matt Ingelson, http://www.mattingelsonmusic.com/

I share my reflections on teaching Foundations of Evaluation after a major course revision, including how it's been going for the first four weeks. Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This episode featured Kim Leonard and Sheila Robinson discussing their careers in evaluation and survey design. They shared how they met on Twitter in 2012 and began collaborating, writing blog posts that eventually became their book "Designing Quality Survey Questions (https://us.sagepub.com/en-us/nam/designing-quality-survey-questions/book249048)" published by Sage Publications. Sheila and Kim discussed their process for writing the book, starting with blog content and building it out over several years into a full manuscript. They also covered common issues they see in surveys and tips for developing effective survey questions. Sheila Robinson: https://www.sheilabrobinson.com/ & https://www.linkedin.com/in/sheilabrobinson/ Kim Leonard: https://leonardrande.com/ & https://www.linkedin.com/in/kim-firth-leonard-1ba9447/ Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This month I chat with Brittany Dernberger about her evaluation background and practice. Brittany Dernberger is a sociologist and gender inequality expert who has led research, evaluation, and organizational learning across academia, philanthropy, government, large international NGOs, and small nonprofits. Brittany currently leads global initiatives to measure systems-level change at CARE and co-chairs the American Evaluation Association Systems in Evaluation Topical Interest Group. Learn more about Brittany's work at http://brittanydernberger.com/. Access CARE's completed Systems-Level Impact evaluations at https://careevaluations.org/evaluation/keywords/systems-level-impact/. The AEA Systems in Evaluation TIG paper on Principles for Systems Thinking in is available at https://www.betterevaluation.org/sites/default/files/SETIG-Principles-FINAL-DRAFT-2018-9-9.pdf. Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This month I talk with Allison Prieur about her experiences as a graduate student in evaluation and running a business. Some resources mentioned or suggested for listeners: * Logic of evaluation (https://www.psychologytoday.com/ca/blog/evaluation-social-betterment/201810/the-application-the-logic-evaluation-the-real-world) * Evaluation of logic in practice (https://doi.org/10.1016/j.evalprogplan.2019.101681) * Destination Dissertation (https://rowman.com/ISBN/9781442246140/Destination-Dissertation-A-Traveler's-Guide-to-a-Done-Dissertation-Second-Edition) * Allison Prieur's LinkedIn profile (https://www.linkedin.com/in/allisonprieur) Music by Matt Ingelson, http://www.mattingelsonmusic.com/

In this episode, I chat with Corey Newhouse and Jessica Manta Meyer, both from Public Profit, about how to make evaluation capacity building sticky for the organizations we work with. Some resources mentioned: TIERS Framework (https://aea365.org/blog/ie-tig-week-laura-beals-and-rachel-albert-on-tiers-a-tool-for-allocating-evaluation-resources-at-nonprofit-agencies/) Tipping Point Community (https://tippingpoint.org/) Irvine Foundation (https://www.irvine.org/) Change Cadet (https://www.changecadet.com/) NonProfit AF (https://nonprofitaf.com/) article on capacity building does not work (https://nonprofitaf.com/2020/10/capacity-buildings-necessary-existential-crisis/) Info on the Public Profit's evaluation cohort (https://www.publicprofit.net/Driving-Toward-Impact) Contact information: info@publicprofit.net About Corey: Corey got her start as a teacher at Summerbridge Cincinnati in the early 1990s and has been involved in educational equity and social justice movements ever since. She founded Public Profit to build a team that would seamlessly blend social science research methods, organizational change strategy, and a deep commitment to supporting changemakers. As the Founder and Principal of Public Profit, Corey leads the team's strategic direction, external relationships, and business development. In addition, Corey serves as an internal thought partner to project teams, assisting with the design of Public Profit's engagements in evaluation, capacity building, and strategic program design. She is co-author of Public Profit's Creative Ways to Solicit Stakeholder Feedback and Dabbling in the Data, and a contributor to Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned. She is a co-editor of the volume, Measure, Use, Improve! Data Use in Out-of-School Time. About Jes: Jessica began her career as a peer volunteer on a national youth talkline providing referrals and support to youth in crisis. Challenged by what she heard on the talkline, and by some of her own experiences with the health care system, she developed an interest in health education, equity and advocacy, which launched a career in the social services sector. Eventually, Jessica combined that with a lifetime love of numbers and inquiry, which led her to program evaluation. Prior to joining Public Profit, Jessica had several years of experience creating, directing and evaluating a range of youth development, LGBT, and health programs. Her work has run the gamut of nonprofit and social service roles including nonprofit finance, human resources, development and agency-led evaluations. Jessica directs many of the projects at Public Profit. She designs evaluation studies, develops project strategy, and manages implementation including all aspects of data collection, analysis, and reporting. An expert facilitator, Jessica also facilitates large stakeholder meetings and evaluation capacity building trainings, and provides evaluation coaching to clients and staff alike.

In this episode, I chat with Michael Moses about theory-based approaches for navigating complexity, adaptive management, participatory strategy, and actions to shift power. Things and resources mentioned: Systems thinking in evaluation (https://www.bobwilliams.co.nz/ewExternalFiles/Wicked.pdf) (including a discussion of complexity) - Bob Williams and Sjon Van't Hof Outcome mapping (https://www.betterevaluation.org/methods-approaches/approaches/outcome-mapping) and outcome harvesting (https://www.betterevaluation.org/methods-approaches/approaches/outcome-harvesting) and emergent learning (https://emergentlearning.org/) Emergent Learning (https://emergentlearning.org/) (including before action reviews, after action reviews, emergent learning tables, and more approaches for supporting collective learning and action) Strategy Testing (https://asiafoundation.org/wp-content/uploads/2015/10/Strategy-Testing-An-Innovative-Approach-to-Monitoring-Highly-Flexible-Aid-Programs.pdf) - Deborah Ladner and the Asia Foundation Kat Haugh (https://www.katherinehaugh.com/) - visual facilitation, sensemaking, and notetaking Learning to Make All Voices Count Initiative - Summary (https://www.makingallvoicescount.org/supporting-local-learning-adaptation-understanding-effectiveness-adaptive-processes) and full paper (https://opendocs.ids.ac.uk/opendocs/bitstream/handle/20.500.12413/13351/MAVC_RR_Moses%20FINAL.pdf) Cynara training on Decolonizing M&E and Research (https://cynara.co/trainingstore/decolonize), led by Michelle Lokot (https://uk.linkedin.com/in/michellelokot) Contact information: Michael Moses mmoses@encompassworld.com (mailto:mmoses@encompassworld.com) LinkedIn (https://www.linkedin.com/in/michael-moses-a9bb9616/) About Michael Moses: Michael Moses is a strategist, facilitator, and evaluator with over 12 years of experience working with public and private sector partners to achieve social impact. He advises foundations, companies, governments, and nonprofits in their efforts to design and implement strategies for advancing change, including by capturing and using data to navigate the complex systems in which they work. In doing so, he helps changemakers learn how to improve programs and organizations, adapt, and over time, strengthen their impact.

In this episode, I chat with Dr. Tatiana Elisa Bustos on community-based participatory research (CBPR). We talked about what it is, how it compares to research and other similar forms of inquiry, and how to get started doing CBPR. Disclaimer: Views expressed here are personal and not reflective of the speaker's respective employers or agencies. Contact information Dr. Tatiana Elisa Bustos tbust002@gmail.com (mailto:tbust002@gmail.com) @TElisa72 (https://mobile.twitter.com/telisa72) https://www.linkedin.com/in/tebustos/ (https://www.linkedin.com/in/tebustos/) About Dr. Bustos Dr. Tatiana Elisa Bustos knows that community partner engagement is key to understanding social issues. She'll share her experience applying community-based participatory research approaches. Dr. Bustos innovates outside the box ways to do research that invite community participation, improving programs through implementation with a social justice lens. As a 1st generation college student and the daughter of Nicaraguan immigrants, equity is deeply important to her. She is an author and award-winning researcher. She leads professional development workshops on implementation science and community based participatory research. She received her PhD in Community Psychology from Michigan State University, an MS in Psychology from Nova Southeastern University, and a BA in Psychology from Florida International University. Connect with her on LinkedIn. (https://www.linkedin.com/in/tebustos/) Dr. Bustos also appeared on The Sci-Files on Impact 89FM and Beyond the Manuscript, the podcast of Progress in Community Health Partnerships. Resources Professional Organizations * Society for Community Research and Action (http://scra27.org/) * American Evaluation Association Connect (http://comm.eval.org/search?executeSearch=true&SearchTerm=community+based+participatory+action+research&l=1) (CBPR search) * Community Psychology TIG (http://comm.eval.org/communitypsychology/home) Training Institutes * https://www.detroiturc.org/programs-expertise/cbpr-capacity-building (https://www.detroiturc.org/programs-expertise/cbpr-capacity-building) * https://www.detroiturc.org/about-cbpr/online-cbpr-course (https://www.detroiturc.org/about-cbpr/online-cbpr-course) * https://www.mitrainingcenter.org/courses/cbprs0218noce (https://www.mitrainingcenter.org/courses/cbprs0218noce) Toolkits * https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/intervention-research/main (https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/intervention-research/main) Journals * Global Journal of Community Psychology (https://www.gjcpp.org/) * American Journal of Community Psychology (https://onlinelibrary.wiley.com/journal/15732770) * Collaborations: A Journal of Community Based Research and Practice (https://collaborations.miami.edu/)

In this episode, I provide a brief overview of the #Eval22 conference, describe the schedule at a glance, and provide tips for getting the most out of the conference. I hope to see you in New Orleans! Say hi to me at the conference to get an EvaluLand nametag ribbon. Eval22 Resources Register to attend the AEA Annual Business Meeting (https://www.evaluationconference.org/Programs/Annual-Business-Meeting) on November 3 at 2pm ET; a recording will be made available after the meeting. Register to attend TIG virtual meetings (https://www.evaluationconference.org/Programs/Topical-Interest-Groups) before and after the conference See the conference schedule at a glance (https://www.evaluationconference.org/Programs/Schedule-At-A-Glance) or the entire searchable program (https://evaluation.secure-platform.com/a/solicitations/2/sessiongallery/schedule?dayId=13&searchParams=%7B%22pageIndex%22%3A0,%22sortMode%22%3A%22SessionName%22,%22sortDirection%22%3A%22Ascending%22,%22sortByFieldId%22%3Anull,%22displayMode%22%3Anull,%22filterByFieldValues%22%3A%5B%5D,%22filterByTextValue%22%3Anull,%22filterByFavorites%22%3Afalse,%22filterByScheduleRoomIds%22%3A%5B%5D,%22filterBySessionTypeIds%22%3A%5B%5D,%22filterByScheduleDayIds%22%3A%5B%5D,%22filterByScheduleTimeSlotIds%22%3A%5B%5D,%22isScheduleOtherEventSearchAllowed%22%3Atrue%7D) Learn more about the professional development workshops (https://www.evaluationconference.org/Programs/Workshops) offered Monday, Tuesday, and Wednesday prior to the conference Attend all the social events (https://www.evaluationconference.org/Programs/Social-Events) at the conference: poster exhibit & meet the authors reception, TIG fair & reception, and the silent auction to benefit international presenters Attend all the plenary sessions (https://www.evaluationconference.org/Programs/Plenary-Sessions) and presidential strand sessions (https://www.evaluationconference.org/Programs/Presidential-Strand) focused on the theme of the conference (https://www.evaluationconference.org/About/2022-Theme) and watch the virtual pre-conference presidential town hall series (https://www.evaluationconference.org/Programs/Virtual-Pre-Conference-Presidential-Series) that occurred in the months leading up to the conference Check out the resources for attendees (https://www.evaluationconference.org/Attendees/Attendee-Resources), including guide to New Orleans, Zoom background options, PowerPoint templates, marketing toolkit, and tips for social media Check out the speaker resources (https://www.evaluationconference.org/Speakers/Presenter-Resources), including information about rooms and materials, printing and shipping, and the potent presentations information (https://www.eval.org/Education-Programs/Potent-Presentations) Check out the floor plan for the Exhibit Hall (https://floorplan.dc.smithbucklin.com/fxfloorplan/22AEA/exfx.html#floorplan), including the exhibitors, posters, and TIG fair This LinkedIn post by Sylvia Pu, PhD has a ton of great comments about how to get the most out of the conference (https://www.linkedin.com/posts/sylviapu_aeaconference-aea-activity-6985348540688433152-ssj1?utm_source=share&utm_medium=member_desktop)

I talked with Dr. Tamara Hamai of Hamai Consulting about contracting, subcontracting, and independent consulting in evaluation. We discussed the minor differences between contracting and subcontracting, cleared up a common misconception of what subcontracting relationships are like, how to get into subcontracting, and tips for setting up contracts. Contact information: Tamara Hamai Hamai Consulting assistant@hamaiconsulting.com https://sustainableimpact.co About Dr. Tamara Hamai: Tamara Hamai, Ph.D., has dedicated her career to empowering organizations and rebuilding our global systems to encourage children's holistic growth and well-being, from prenatal through the completion of higher education – especially those who are most vulnerable and facing the greatest challenges. In 2008, she founded Hamai Consulting as a platform to help organizations increase their impact, stability, and strength to make a bigger impact in children's lives. Dr. Hamai's work spans most aspects of child development, such as early childhood education, higher education, child welfare, parenting and discipline, trauma, and adverse childhood experiences. She has previously been featured by KTLA, KPIX CBS San Francisco, ABC 7 News, NBC Radio, American Psychological Association, Western Psychological Association, Institute for Violence, Abuse, and Trauma, American Evaluation Association, National Head Start Association, and several blogs. She is also a reviewer and on the Editorial Board for publications such as the Journal of Aggression, Maltreatment & Trauma, Journal of Child & Adolescent Trauma, Journal of Sexual Abuse, and the Journal of Child Custody. Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This episode I chatted with Carrie Tanasichuk and Harry Daley about how they have been using theories of change as their main process for facilitating strategic planning with non-profits. In the episode, they also mentioned: * The Community Foundation of Saint John's impact measurement and evaluation principles (https://thecommunityfoundationsj.com/impact/) * The Community Builder's Approach to Theory of Change (https://www.theoryofchange.org/pdf/TOC_fac_guide.pdf) * The Miles Nadal (https://www.mnjcc.org/) JCC Theory of Change (https://changeopenly.com/wp-content/uploads/2020/12/TOCsample-MNjcc_TheoryofChange.pdf) About Carrie and Harry: *Carrie Tanasichuk *(Twitter DrCarrieTee (https://twitter.com/DrCarrieTee) & carrie@sjfoundation.ca (carrie@sjfoundation.ca)) has 16 years of experience in evaluation. She is passionate about using her expertise to help organizations demonstrate impact, improve, and innovate. She has worked across sectors (non-profit, for-profit, government) in diverse areas, including poverty reduction, youth development, criminal justice, health promotion and screening, and technology. She has a Ph.D. in Applied Social Psychology from the University of Saskatchewan. Harry Daley (harry@sjfoundation.ca (harry@sjfoundation.ca)) was born and raised in Saint John, New Brunswick. He has dedicated his personal and professional life to working with organizations that focus on poverty reduction and amplifying the youth voice. He has used evaluation as a tool for designing experiential and participatory youth programming and believes evaluation is an integral aspect of program design and facilitation rather than something that lives outside the programs. Harry has a Bachelor of Philosophy in Interdisciplinary Leadership from the University of New Brunswick's Renaissance College. Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This episode I chatted with Bradlie Nabours about how he got into the field of evaluation and his experience applying for evaluation jobs. He also talks about his evaluation experience and provides great tips on applying for jobs in the evaluation space. Job search & application recommendations: Don't give up. Know your worth. The job process is difficult, but don't settle and remember it doesn't reflect necessarily on you. Keep applying! Don't limit yourself in what positions you look for. Expand your horizons. Network like your life depends on it! This includes having an online presence and building and leveraging connections with people in your field such as through LinkedIn, EvalYouth (LinkedIn (https://www.linkedin.com/company/evalyouth-north-america/) & Twitter @evalyouth_na (https://twitter.com/evalyouth_na)), and your professors! Be a versatile applicant. Build your resume. Seek out learning opportunities to grow as a professional. Be willing to experiment and try new things. Contact information: Email: bradlietnabours@gmail.com (bradlietnabours@gmail.com) Twitter: bradn98 (https://twitter.com/bradn98) LinkedIn: www.linkedin.com/in/bnabours About Bradlie Nabours: Bradlie is a perinatal health program evaluator at Johns Hopkins All Children's Hospital in St.Petersburg, Fl. He works for a federal Healthy Start program focused on reducing adverse perinatal outcomes for black women and their families. He holds a Bachelors of Science in Public Health from Georgia Southern University and completed his Master's in Public Health with a graduate certificate in Maternal and Child Health from the University of South Florida. He started his evaluation journey while in grad school where he worked as a graduate assistance on the evaluation of a non-profit family violence prevention program. While in grad school his research focused racial disparities in birth outcomes which led him to start his career at the intersection of his research interest and evaluation. Music by Matt Ingelson, http://www.mattingelsonmusic.com/

On this episode, I'm chatting with Dr. Tristi Nichols about her work conducting international evaluation and her journey into the international and evaluation spaces. Resources: Footprint evaluation (https://www.betterevaluation.org/en/footprint-evaluation-webinar) OECD DAC Criteria for Evaluation Framework (https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm) List of websites to find international evaluation: * https://www.ungm.org/ type “evaluation” in the title bar [for UN only] * https://reliefweb.int/ or https://reliefweb.int/jobs * https://www.evalcommunity.com/ Contact information: Tristi Nichols tnichols@manitouinc.com 914-414-8288 @tristiempo https://www.linkedin.com/in/drtristinichols/

On this episode, I'm chatting with Dr. Ayesha Boyce about the evaluation job market. Ayesha provides a ton of great insights and information about how to search for jobs, what to consider when looking for jobs, and tips for being a strong job candidate. Ayesha was previously on the podcast discussing "teaching evaluation and supporting students and colleagues of color (https://evaluland.fireside.fm/4)." Contact information: Ayesha Boyce Email: ayesha.boyce@asu.edu Twitter: @AyeshaBoyce Website: https://education.asu.edu/about/people/ayesha-boyce About GUEST: Ayesha Boyce is currently an associate professor in the Division of Educational Leadership and Innovation at Arizona State University. She also co-directs the STEM Program Evaluation Lab (http://www.stemprogramevaluation.org). Boyce's scholarship focuses on attending to value stances and issues related to diversity, equity, inclusion, access, cultural responsiveness, and social justice within evaluation—especially multi-site, STEM, and contexts with historically marginalized populations. She also examines teaching, mentoring, and learning in evaluation. She has evaluated more than 55 programs funded by the National Science Foundation (NSF), US Department of Education, National Institutes of Health, and Spencer and Teagle foundations. Boyce is a 2019 American Evaluation Association Marcia Guttentag Promising New Evaluator Award recipient. In her teaching and mentorship, Boyce encourages students to develop a strong methodological foundation, conduct studies based on democratic principles, and promote equity, fairness, inclusivity, and diversity. Music by Matt Ingelson, http://www.mattingelsonmusic.com/

On this episode, I'm chatting with Eric Persaud about evaluation at the National Institutes of Health, including his work in the Worker Training Program at the National Institute of Environmental Health Sciences. We also talked about his dissertation, how he got into the field of evaluation, and his experiences in conducting evaluation in general. About Dr. Eric Persaud: Eric Persaud received his doctorate in Public Health at the Department of Environmental and Occupational Health Sciences, State University of New York-Downstate Health Sciences University. He focuses on evaluating and researching training programs related to preparing workers for emergencies and disasters, and hazardous workplaces. He has been involved in evaluation and research associated with fentanyl and first responders, opioids and the workplace, and protecting workers from COVID-19. You can reach Eric Persaud at Eric.Persaud@NIH.gov You can learn more about the NIEHS Worker Training Program at https://www.niehs.nih.gov/careers/hazmat/about_wetp/index.cfm Resources: Biosafety and infectious disease occupational health training from the NIEHS Worker Training Program: A Historical look at capacity building that supported a COVID-19 response. Eric Persaud, Deborah Weinstock, Demia S. Wright. Journal of Emergency Management. 2022. (https://www.wmpllc.org/ojs/index.php/jem/article/view/3172/3440) Opioids and the Workplace Prevention and Response Awareness Training: Mixed Methods Follow-Up Evaluation - Eric Persaud, Aimee Afable, Laura A. Geer, Paul Landsbergis, 2021 (https://journals.sagepub.com/doi/abs/10.1177/10482911211010343) COVID-19 Biosafety Training and Infectious Disease Response Evaluation Report. NIEHS Worker Training Program. 2021. (https://tools.niehs.nih.gov/wetp/public/hasl_get_blob.cfm?ID=13421) Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This episode I chatted with Jan Noga about systemic design thinking. There's a wealth of resources and information provided below! Contact information: Jan Noga Jan.Noga@pathfinderevaluation.com www.pathfinderevaluation.com About Jan Noga: Jan Noga is an independent evaluation consultant based in Cincinnati, Ohio. She holds a bachelor's degree from Stanford in developmental and counseling psychology with specialization in early and middle childhood and a master's degree from the University of Cincinnati in instructional design and technology. Jan has worked in the non-profit and public sectors in human services and education for more than 30 years in roles spanning teaching, research, policy, and program planning and evaluation. As a program evaluator, Jan has planned and conducted both large and small-scale evaluations and provided organizational consulting and capacity building support to clients. She has also taught courses and workshops on such topics as systems thinking, systemic design thinking, research methods and techniques, program planning and development, and survey design and analysis. Jan has been a member of AEA since 2000 and was one of the founding members of the Systems in Evaluation TIG, serving as program chair and then TIG chair from 2004-2012. She is particularly interested in the use of systems approaches as a foundation for design, planning, implementation, and evaluation of change efforts in the human service and education arenas. Systems Thinking Resources for Evaluators: Hands on resources: * Williams, Bob. 2020. Systemic evaluation design: A workbook. Available for download from https://bobwilliams.gumroad.com/ * Williams, Bob. 2021. Systems diagrams: A practical guide. Available for download from https://bobwilliams.gumroad.com/ Good for starting out * Anderson, V. & Johnson, L. (1997). Systems thinking basics: From concepts to causal loops. Waltham, MA: Pegasus Communications. * Meadows, D.H. (2008). Thinking in systems: A primer. White River Junction, VT: Chelsea Green Publishing. * Ramage, M. & Shipp, K (2009). Systems Thinkers. New York: Springer. * Sweeney, L.B. & Meadows, D. (2010). The systems thinking playbook. White River Junction, VT: Chelsea Green Publishing. * Williams, B. & Hummelbrunner, R. (2011). Systems concepts in action: A practitioner's toolkit. Stanford, CA: Stanford University Press. * Williams, B. and Imam, I, eds. (2007). Systems concepts in evaluation: An expert anthology. Point Reyes, CA: EdgePress. * Williams, B. and Van't Hoft, S (2016). Wicked solutions: A systems approach to complex problems. Available at http://bit.ly/1SVoOH3 Good for more advanced reading: * Bamberger, M, Vaessen, J., & Raimondo, E. (eds.) (2016) Dealing with complexity in development evaluation: A practical approach. Thousand Oaks, CA: Sage Publications. * Cabrera, D., Colosi, L., & Lobdell, C. (2008) Systems thinking. Evaluation and Program Planning, 31(3), 299-310. * Cabrera, D. & Cabrera, L (2015). Systems thinking made simple: New hope for solving wicked problems. Odyssean Publishing. * Capra, F & Luisi, PL (2016). The systems view of life: A unifying vision (6th printing). New York: Cambridge University Press. * Checkland, P. (1999). Systems thinking, systems practice. New York: John Wiley & Sons, Ltd. Cunliff, E., (2002) Connecting systems thinking to action, The Systems Thinker, 15(2), 6-7. * Eoyang, G.H. & Holladay, R.J. (2013) Adaptive action: Leveraging uncertainty in your organization. Stanford: Stanford Business Books. * Karach, R, (1997) How to see structure, The Systems Thinker, 8(4), 6-7. * Patton M.Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press. * Patton, M.Q., McKegg, K., & Wehipeihana, N., eds. (2015). Developmental evaluation exemplars: Principles in practice. New York: Guilford Press. * Senge, P. (1990) The fifth discipline: The art and practice of the learning organization. New York: Doubleday. * Stroh, DP (2015). Systems thinking for social change. White River Junction, VT: Chelsea Green Publishing. * Ulrich, W & Reynolds, M (2010). Critical systems heuristics. In: Reynolds, Martin and Holwell, Sue eds. Systems approaches to managing change: A practical guide. London: Springer, pp. 243–292. * von Bertalanffy, Ludwig. (1950). The theory of open systems in physics and biology. Science, * 13, 23-29. * von Bertalanffy, Ludwig. (1968). General systems theory. New York: George Braziller, Inc. * Wolf-Branigin, M. (2013) Using complexity theory for research and evaluation. Oxford: Oxford University Press. Some other resources: * International Society for Systems Sciences * https://aea365.org/blog/systemic-design-thinking-for-evaluation-of-social-innovations-a-pd-for-intermediate-and-advanced-evaluators-by-jan-noga/ * http://www.epreconsulting.com/SETIG%202018%20Principles.pdf * https://systemic-design.org/ * https://modus.medium.com/what-the-is-systems-design-e005c1e9fef8 * https://rsdsymposium.org/ * Martin Reynolds Open University Music by Matt Ingelson, http://www.mattingelsonmusic.com/

This episode I chat with Martha Brown, PhD about trauma-informed evaluation, but more specifically about how we as evaluators can help organizations become more trauma-informed. We talk about trauma, the SAMHSA model of trauma-informed work, and much more! Martha talked about trauma-informed evaluation also on the Glass Frog podcast with Debbie Gowensmith and on the Community Possibilities podcast. Other resources mentioned include the AEA365 series on trauma-informed evaluation and the Center for Victims of Torture. For an introductory, self-paced course to learn more about trauma and what it means to be trauma-informed, visit https://www.rjaeconsulting.com/becoming-trauma-informed-how-to-apply-the-principles-of-trauma-informed-care-to-your-life-work-and-community- Suggested donation is $75-99 - special module for evaluators. Contact information: martha@hamaiconsulting.com martha@rjaeconsulting.com About Martha: Martha Brown is president and founder of RJAE Consulting and an active member of the American Evaluation Association (AEA). She is also the trauma informed and restorative justice specialist for Hamai Consulting, where she helps organizations and systems do the work of becoming trauma-informed and resilient. She received her doctorate in Curriculum & Instruction from Florida Atlantic University in 2015 and is an accomplished author, evaluator, program designer, presenter, trainer, and teacher. Dr. Brown's research has been published in numerous peer-reviewed journals and edited books. She also authored the best-selling book, Creating Restorative Schools: Setting Schools Up to Succeed, available from Living Justice Press. She has presented at numerous national and international conferences on the topics of educational policy reform, restorative justice, arts education, and culturally responsive evaluation. Martha's content expertise is in trauma informed organizations and practices, restorative justice, and arts evaluation. She brings a unique blend of creativity, caring, passion, and technical skills to her work. Martha utilizes a restorative approach to her work as an evaluator, always placing relationships at the center of her work. Music by Matt Ingelson, http://www.mattingelsonmusic.com/

I chat with Libby Smith and Tiffany Tovey, two of the three hosts of the Radical Re(imagining) (https://radicalreimagining.com/) podcast, to chat about what felt radical about the 2021 AEA virtual conference. We'd love to hear from you! What felt radical to you about the conference? Feel free to add your reflections on LinkedIn or Twitter. Be sure to use the #Eval21 hashtag, and tag us in your reflections! Twitter accounts: * @RadReImagining (https://twitter.com/RadReImagining) * @workwithlibby (https://twitter.com/work_with_libby) * @Tiffany7001 (https://twitter.com/Tiffany7001) * @EvaluLand (https://twitter.com/EvaluLand) * @danawanzer (https://twitter.com/danawanzer)

Jennifer Puma and I talk about project management in evaluation, including how she manages evaluation projects from start to finish and how she brings her project management training into her organizations and with the clients she works with. We discuss things like scoping projects (https://www.evalacademy.com/new-products/program-evaluation-scoping-guide), setting up ad hoc meetings, how project management aligns with various types of budgeting approaches, and so much more! Also, be sure to check out the Glass Frog Podcast (https://glassfrog.us/podcast/), which is another evaluation-related podcast (https://evaluland.fireside.fm/other-eval-podcasts) that I highly recommend! As a reformed management consultant, Jennifer (https://glassfrog.us/staff_trusted/jennifer-puma/) leverages more than 15 years of consulting and project management experience to manage program evaluations and research projects at Glass Frog (https://glassfrog.us/). Her functional specialty is in the development and practical application of theories of change and evaluation frameworks. Jennifer holds a B.A. degree from The College of New Jersey and a M.P.A. degree from The Maxwell School of Citizenship and Public Affairs at Syracuse University. She lives in Brooklyn, NY, with her husband and ridiculous rescue dog, Sammy. Here is the John Oliver clip that Jen mentions as well: Nothing Good Happens in Excel (https://www.youtube.com/watch?v=S46yX42IqaA). As Jen says, "When it comes to project management, we do some of our work in Excel and share with clients but are careful to check in, first, about their level of comfort with Excel. John Oliver's observation is funny because it's true."

Jennifer Bisgard and I talk about project management in evaluation, including how she manages evaluation projects from start to finish and useful tools in her company's practice. Jennifer has a wealth of information in her over 20 years of experience leading evaluation projects, and I learned a lot about how to better do this work from her. Khulisa has a wonderful library of resources including blog posts about QASP (https://www.khulisa.com/lesson-5-make-sure-that-the-data-you-collect-throughout-the-evaluation-process-is-high-quality-but-remains-private/), analyzing WhatsApp data (https://www.khulisa.com/a-step-by-step-guide-on-analyzing-whatsapp-data/), and using rubrics (https://www.khulisa.com/guest-blog-the-value-of-contribution-and-evidence-rubrics-for-evaluations/). Every Tuesday they publish #EvalTuesdayTip (https://www.khulisa.com/thought-leadership/eval-tuesday-tips/). Ms Jennifer Bisgard co-founded Khulisa Management Services (https://www.khulisa.com/) in 1993. An expert in monitoring and evaluation (M&E) and organisational development, she leads evaluations and capacity building assignments in the Education, Power, and Democracy and Governance sectors. Khulisa has about 100 staff, based in Johannesburg, Nairobi, Lusaka, Mbabane, and Bethesda. Prior to establishing Khulisa, Jennifer was the Senior Education Specialist at USAID/Pretoria from 1988 to 1993. Ms Bisgard has served on boards for the: African Evaluation Association (AfrEA) (https://afrea.org/), International Organization for Cooperation in Evaluation (IOCE) (https://www.ioce.net/) and South African Monitoring and Evaluation Association (SAMEA) (https://www.samea.org.za/home). She is a current board member of BetterEvaluation (https://www.betterevaluation.org/). Jennifer co-authored a chapter of “Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned (https://us.sagepub.com/en-us/nam/evaluation-failures/book260109)” published by Sage Publishers in August 2018. The book, edited by Kylie Hutchinson, includes a forward by Michael Quinn Paton. She has a Master's Degree in Social Change and Development from Johns Hopkins University. She is an American but has been in South Africa for 33 years. She is married to Marc (a dual Dutch-South Africa citizen) and has three boys: Zuko (adopted, now 23); Dylan (18) and Thomas (14). The family is completed by Border Collie, Riley and Africanis (https://southafrica-info.com/arts-culture/africanis_original_dog_africa/), Milly.

Elizabeth and I discuss her new independent consulting business (Elizabeth Grim LLC (https://elizabethgrim.com/)) and how she meets clients where they are, builds capacity and relationships, and leans into her values in the work. We also discuss Brene Brown's values exercise (https://daretolead.brenebrown.com/wp-content/uploads/2019/02/Values.pdf) and another version of the exercise as a card deck (https://bestself.co/products/core-values-deck), how we've embraced new ways of thinking about the world, and reporting in evaluation.

_Note: Sorry for the poor audio quality on my end. Recording is hard... _ Nina and I discuss her incredible journey through evaluation from research analyst to independent consultant to founder and principal of her own evaluation consulting firm, Intention 2 Impact (https://www.intention2impact.com/). We discuss starting up her business and how that's been going. Along the way, we also discuss her evaluation background, including her dissertation-in-progress on entrepreneurship in evaluation.

I'm interviewed by James Pann, PhD about my latest research on evaluation study in the American Journal of Evaluation. We discuss the difference between research and evaluation, the pros and cons of professionalization, the definition of evaluation, and other evaluation related topics. Link to the article in AJE (https://journals.sagepub.com/doi/abs/10.1177/1098214020920710) Link to the YouTube video of the interview (https://www.youtube.com/watch?v=KLcYU9xptwU) If you've got ideas for the podcast, or would like to be a guest on the podcast (students, practitioners, scholars, and others are all welcome!), then please email me at dana@danawanzer.com

After chatting with folks from EvalYouth about the Global Mentoring Program, they suggested a series of podcast episodes on what evaluation looks like around the world. This episode is one in which I chat with evaluators from the Eastern Europe, Central Asia, and South Caucasus (ECA) about what evaluation looks like in their region. I chat with Tamara Kabysh-Rybalka and Dana Reva, both of whom are involved with EvalYouth ECA, about their experiences evaluating in their region. We discuss a variety of topics, including how they define evaluation, how their countries and region affects how evaluation is done, what valuing looks like in their region, professional ethical guidelines, stakeholder involvement, how people get into the field of evaluation, ECA Summer Institute and other training and networking opportunities, institutionalization of evaluation in ECA, regional policies related to evaluation, and more. Resources discussed: The Eurasian Alliance of National Evaluation Associations (http://www.evaleurasia.org/_about_en_) Eval4Action Regional Consultation, Eurasia (http://www.evaleurasia.org/_about_en_) Eval Partners Global Forum (http://www.evaleurasia.org/_about_en_) Learn more about EvalYouth ECA here: ECA EvalYouth Website (https://www.eca-evalyouth.org/) EvalYouth Website (https://evalyouth.org/index.php/evalyouth-eca/) Facebook (EvalYouthECA) (https://www.facebook.com/EvalYouthECA) YouTube (https://www.youtube.com/channel/UC71edcbSrXVZHXXqUJ9WR4g) Twitter (EcaEval) (https://twitter.com/EcaEval) Special Guests: Dana Reva and Tamara Kabysh-Rybalka.

After chatting with folks from EvalYouth about the Global Mentoring Program, they suggested a series of podcast episodes on what evaluation looks like around the world. This episode is one in which I chat with evaluators from the Middle East and North Africa (MENA) about what evaluation looks like in their region. I chat with Hayat Askar and Sana Ben Salem about their experiences evaluating in their region, specifically Jordan and Tunisia, respectively. We discuss a variety of topics, including how they define evaluation, how their countries and region affects how evaluation is done, what evaluation looks like in their contexts (purpose, approaches, stakeholder involvement, recommendations, use, etc.), interesting trends and things going on in their local VOPEs, how people get into the field of evaluation, and more. Special Guests: Hayat Askar and Sana Ben Salem.

After chatting with folks from EvalYouth about the Global Mentoring Program, they suggested a series of podcast episodes on what evaluation looks like around the world. This episode is one in which I chat with evaluators from Latin America and the Caribbean (LAC) about what evaluation looks like in their region. I chat with Claudia Olavarría and Gerardo Sánchez-Romero about evaluation in the LAC region. They are the co-chairs of EvalYouth LAC (https://evalyouth.org/index.php/evalyouth-in-lac/), which you can contact via Twitter (https://twitter.com/evalyouth_lac), Facebook (https://www.facebook.com/EvalYouthLAC/), or email via evalyouthlac@gmail.com. We discuss a variety of topics, including how they define evaluation, stakeholder involvement, the governmental contexts they work in (e.g., CONEVAL (https://en.wikipedia.org/wiki/National_Council_for_the_Evaluation_of_Social_Development_Policy_(CONEVAL)), Mideplan (https://www.mideplan.go.cr/)), the purpose of doing evaluation, evaluation designs, evaluation standards (Estándares de Evaluación para América Latina y el Caribe (https://www.researchgate.net/publication/308143662_Estandares_de_Evaluacion_para_America_Latina_y_el_Caribe)), Evaluación Participativa (https://evalparticipativa.net/), training and education in evaluation, professionalization, Eval4Action (https://www.eval4action.org/), the SDG 2030 Agenda (https://sdgs.un.org/2030agenda), Evaluation without Borders, Blue Marble evaluation, and more. Special Guests: Claudia Olavarría and Gerardo Sánchez.

After chatting with folks from EvalYouth about the Global Mentoring Program, they suggested a series of podcast episodes on what evaluation looks like around the world. This episode is one in which I chat with evaluators from Europe about what evaluation looks like in their region. I chat with Tom Ling and Oto Potluka about European evaluation. They are both board members of the European Evaluation Society, the professional association for evaluators in Europe. We discuss a variety of topics, including the streams of evaluation happening in evaluation, the role of evaluators in program design, reflective practice, stakeholder involvement, impact evaluation, evaluation approaches like theory-driven evaluation, how the GDPR affects evaluation, and more. Special thanks to Lauren Weiss, the EES Adminstration and Communications Manager, for helping coordinate this episode. Special Guests: Oto Potluka and Tom Ling.

We’re getting meta today… looking at the influence of a major influencer in evaluation: Carol Weiss! I chat with Gregory Greenman II about his dissertation examining citation metrics of her work across disciplines. Special Guest: Gregory Greenman II.

I chat with Helen Salmon from SAGE Publishing about book publishing! We go over how to generate ideas for books, what the book publishing process looks like, and about some of the most recent evaluation books from SAGE! Sage has been a friendly publisher for evaluation books for decades. As Patton stated in the third edition of Qualitative Research and Evaluation Methods (2002): "[Sara Miller McCune's] vision and follow-through have made Sage publications the leading publisher of both evaluation and qualitative inquiry books." (p. xxiv). Some recent evaluation books mentioned: 1. Evaluation in Today's World (https://us.sagepub.com/en-us/nam/evaluation-in-today%E2%80%99s-world/book263463) by Veronica G. Thomas and Patricia B. Campbell 2. The Practice of Evaluation (https://us.sagepub.com/en-us/nam/the-practice-of-evaluation/book254842) by Ryan P. Kilmer and James R. Cook 3. Completing Your Evaluation Dissertation, Thesis, or Culminating Project (https://us.sagepub.com/en-us/nam/completing-your-evaluation-dissertation-thesis-or-culminating-project/book259160) by Tamara M. Walser and Michael S. Trevisan 4. Program Evaluation (https://us.sagepub.com/en-us/nam/program-evaluation/book253271) by Susan P. Giancola More information for interested book authors and editors: https://us.sagepub.com/en-us/nam/book-author-editors Special Guest: Helen Salmon.

In this week's episode, I chat with Sawsan Al-Zatari, Zach Tilton, and Teddy Theoharis about the EvalYouth Global Mentoring Program (EY-GMP). Links: EvalYouth (https://evalyouth.org/) YEEs in VOPEs (https://evalyouth.org/index.php/tf1-yees-in-vopes/) Mentoring program (https://evalyouth.org/index.php/tf2-mentoring-program/) E-Learning (https://evalyouth.org/index.php/tf3-elearning-conferences/) Youth participatory evaluation (https://evalyouth.org/index.php/tf4-youth-participatory-evaluation/) EY-GMP evaluations: 2017-18 Pilot Phase (https://evalpartners.org/sites/default/files/documents/evalyouth/EvalYouth%20TF2%20Mentoring%20Program%20-%20Pilot%20Phase%20Evaluation%20-%20July%202018.pdf) 2018-19 (https://evalpartners.org/sites/default/files/files/Evaluation%20of%20GMP%202019.pdf) 2020 in-progress How to get involved: Become a volunteer (https://volunteer.evalyouth.org/?page_id=10) for EvalYouth Join a Task Force Follow on social media: Twitter (https://twitter.com/Eval_Youth), LinkedIn (https://www.linkedin.com/in/evalyouth/), Facebook (https://www.facebook.com/EvalYouth), and YouTube (https://www.youtube.com/channel/UC5glSYDLvUub097giZRYWpA) Special Guests: Sawsan Al-Zatari, Theodore Theoharis, and Zach Tilton.

This week I chat with Tiffany Smith and Libby Smith about our experiences teaching a course on interpersonal effectiveness in the MS in Applied Psychology program (https://www.uwstout.edu/programs/ms-applied-psychology) at the University of Wisconsin-Stout. We also discuss why teaching interpersonal skills is important and why we believe these skills can and should be explicitly taught. Some background on the course The course is a 3-credit, 8-week course that focuses on developing self-awareness through personal and collaborative reflective practice. It is the third course in the evaluation concentration. Tiffany brought this course to our evaluation program and taught it from 2015-17. Libby then taught it from 2018-19, bringing in new elements. Dana taught it most recently in an online environment due to the COVID-19 pandemic. Resources: Radical (Re)imagining (https://radicalreimagining.com/): A collaborative reflective space Being Human at Work (https://workwithlibby.com/breathwork): A breathwork practice for integrating the personal and professional selves Tiffany Smith and colleagues' article "Demystifying reflective practice: Using the DATA model to enhance evaluators' professional activities (https://www.sciencedirect.com/science/article/pii/S014971891500049X)" Smith, Smith, & Wanzer (2020) A framework for purposefully fostering interpersonal skills (https://vimeo.com/470799423). Presented at the International Society for Evlauation Education (ISEE). Glass Frog podcast episode on Cultivating Interpersonal Effectiveness (https://glassfrog.us/s2e7/) Special Guests: Libby Smith and Tiffany Smith.

This week I chat with Dr. Aisha Rios to both get to know her a little better and to talk about social justice in evaluation. Topics & Resources Mentioned: Getting to know Dr. Rios and Coactive Change a little better What is social justice in evaluation? Emergent strategy as guiding Dr. Rios in framing social justice evaluation as about imagining different futures Abolition literature as another influence for Dr. Rios Studying up vs studying down: "Up the anthropologist-Perspectives gained from studying up (https://upwardanth.files.wordpress.com/2014/06/nader-studyingup.pdf)" by Laura Nader AEA session “Illuminating the future context through an equity lens: Why systems evaluators need to use foresight and futures (https://aea2020.pathable.co/meetings/virtual/y9wquEeCEFGM4H379)” by Dr. Jen Heeg and Dr. Jewlya Lynn AEA session “Evaluation as restorative practice: Exploring how collaborative and participatory methods can promote peace, justice, and healing in Guatemala (https://aea2020.pathable.co/meetings/virtual/ePj6Rc2rLHdQWuZ2x)” by Erica Henderson and Giovanni Dazzo Importance of building community and doing this work in relation Breathwork (https://workwithlibby.com/breathwork/) with Libby Smith Characteristics of white supremacy culture (https://www.showingupforracialjustice.org/white-supremacy-culture-characteristics.html) How to create, cultivate, and protect space for reflection Addressing the inadequate budget and time in proposals for appropriate diversity, equity, inclusion, and justice work Finding social justice evaluation work Thinking thoughtfully of evaluation questions and being flexible about the evaluation questions throughout the evaluation process Approaches, methods, and reporting in social justice evaluation Navigating a capitalistic society as a socialist: ownership, authorship, contracts, money, and more Contact: Dr. Aisha Rios: Coactive Change (https://www.coactivechange.com/) & Twitter @AishaRios17 (https://twitter.com/aisharios17) EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guest: Dr. Aisha Rios.

I've received multiple questions about my podcasting setup and process. Rather than answer the questions individually, I thought I would podcast about it! I discuss why I got into podcasting, my recording technology and room setup, the audio processing my husband (aka audio engineer) does, podcast hosting, the general timeline for me to create an episode, and some extra few questions. Technology * Old microphone: FIFINE Plug & Play Home Studio USB Condenser Microphone (https://www.amazon.com/gp/product/B01D4HTIOY) * Mic stand: Gator Frameworks Short Weighted Base Microphone Stand (https://www.amazon.com/gp/product/B00BPELU68) * New microphone: Audio-Technica AT 2020 Cardioid Condenser Studio XLR Microphone (https://www.amazon.com/gp/product/B0006H92QK/) * USB audio interface: Focusrite Scarlett Solo (3rd Gen) USB Audio Interface (https://www.amazon.com/gp/product/B07QR6Z1JB/) * Windows audio software: Audacity (https://www.audacityteam.org/) * Browser podcasting software: Zencaster (https://zencastr.com/) (personally, I don't recommend it) * Podcast hosting: Fireside (https://fireside.fm/) Also check out this blog post by Echo Rivera: The Ultimate Guide to Microphones (https://www.echorivera.com/blog/good-mic) Contact * Website: https://evaluland.fireside.fm/ * Twitter: https://twitter.com/evaluland (@EvaluLand)

This episode I chat with Ann Webb Price and Susan Wolfe about coalition evaluation, partnering together in evaluation, the independent consulting TIG of AEA, and so much more! Topics & Resources Mentioned: How Ann and Susan started working together What is coalition evaluation? How coalition evaluation uses a systems approach Advice for doing coalition evaluation, including not just "but why?" but "but why here?" Community Coalition Action Theory, which one book they recommend is Ignite! (https://www.amazon.com/Ignite-Getting-Community-Coalition-Change/dp/1491810122/) Tearless logic model (https://www.gjcpp.org/en/tool.php?issue=7&tool=9) approach How they are adapting their approaches to Covid-19 Workshops on coalitions and evaluation A new partnership: Positive Impact Consultants (https://positiveimpactconsultants.com/) The independent consulting TIG (https://comm.eval.org/independentconsulting/home) of the American Evaluation Association Contact: Ann Price: Community Evaluation Solutions (https://www.communityevaluationsolutions.com/), Positive Impact Consultants (https://positiveimpactconsultants.com/), and Twitter (@AnnWPrice (https://twitter.com/annwprice?lang=en)) Susan Wolfe: Susan Wolfe and Associates (https://susanwolfeandassociates.com/) or email susan@susanwolfeandassociates.com/ EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guests: Ann Price and Susan Wolfe.

In this episode, I chat with Aasha Joshi and Hannah Lantos about their recent chapter "Demystifying data: Strategies and tools for making data more meaningful in OST programs" in the recently released book Measure, Use, Improve! Data Use in Out-of-School Time (https://www.infoagepub.com/products/Measure-Use-Improve), edited by Christina Russell and Corey Newhouse. The book will be particularly useful to leadership-level staff in out-of-school time organizations that are thinking about deepening their own learning and evaluation systems, yet aren’t sure where to start. We talked about: What it means to demystify data in OST programs How OST program evaluation is unique How they and their OST programs are responding to the Covid-19 pandemic Contact: Aasha Joshi: http://www.claritywritingandresearch.com/ (http://www.claritywritingandresearch.com/) Hannah Lantos: hlantos@childtrends.org EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guests: Aasha Joshi and Hannah Lantos.

Thank you to everyone who participated in the call-in (https://evaluland.fireside.fm/voicemail) to share their own reflections about the 2020 AEA virtual experience (https://www.evaluationconference.org/)! In this episode, I start with thanking the conference organizers, sharing my own reflections, and then pass it off to the eight people who called in to share their own reflections: 1. Rachael Lawrence (@R8chLawrence (https://twitter.com/R8chLawrence)) 2. Rakesh Mohan (@RakeshMohanEval (https://twitter.com/RakeshMohanEval)) 3. Ann Price (@annwprice (https://twitter.com/annwprice)) 4. Elizabeth Grim (@ecgrim (https://twitter.com/ecgrim)) 5. Jeremy Danz (@DanzJeremy (https://twitter.com/DanzJeremy)) 6. Veronica Olazabal (@veroviews (https://twitter.com/veroviews)) 7. Tiffany Smith (@Tiffany7001 (https://twitter.com/Tiffany7001) 8. Libby Smith (@workwithlibby (https://twitter.com/work_with_libby))

Thank you to everyone who participated in the call-in (https://evaluland.fireside.fm/call-in) to share their own reflections about the 2020 AEA virtual experience (https://www.evaluationconference.org/)! After the conference, please sure your own thoughts and reflections by calling in (https://evaluland.fireside.fm/call-in). It's very similar to leaving a voicemail! In this episode, I start with my own reflection about the conference, how I'm going to make the most of the conference, and what I am personally excited about. Then, we get to hear from nine other evaluators (in no particular order): 1. Tom Archibald, associate professor at Virgina Tech (@tgarchibald (https://twitter.com/tgarchibald)) 2. Bianca Montrosse-Moorhead, associate professor at University of Connecticut (@b_mmoorhead (https://twitter.com/b_mmoorhead)) 3. Danielle Murillo, at the Claremont After School Program (@happyeval (https://twitter.com/happyeval)) 4. Kathleen Doll, independent evaluation consultant (part-time at the Chan-Zuckerberg Initiative and part-time at Intention 2 Impact (https://www.intention2impact.com/)) 5. Christian Lucchesi, project manager with Catalyst at University of Wisconsin-Stout (@luccheesey (https://twitter.com/luccheesey)) 6. Nina Sabarre, founder and principal of Intention 2 Impact (https://www.intention2impact.com/) (@ninasabarre (https://twitter.com/ninasabarre)) 7. Akihiko Ashimoto, National Institute for Educational Policy Research in Japan 8. Elizabeth Grim, director of community impact from the Connecticut Data Collaborative (@ecgrim (https://twitter.com/ecgrim)) 9. Libby Smith, evaluator and educator (@workwithlibby (https://twitter.com/work_with_libby))

This week, Chari Smith joins me to talk about nonprofit evaluation, how to build a culture of evaluation, the relationship between music and evaluation, and the AEA conference. Chari Smith wrote a book! Nonprofit Program Evaluation Made Simple: Get your Data. Show your Impact. Improve your Programs (https://evaluationintoaction.com/book/). Sign up for the mailing list to learn more about the book, including the launch date. Resources & People Mentioned: In 4 All (https://evaluationintoaction.com/2020/09/30/covid-19-cant-stop-progress-spotlight-on-education-and-evaluation/) blog post on adapting to Covid-19 Managing applied social research (https://www.wiley.com/en-us/Managing+Applied+Social+Research%3A+Tools%2C+Strategies%2C+and+Insights-p-9781118105474) by Russ-eft and colleagues (2017) Enhancing the effectiveness of logic models (https://journals.sagepub.com/doi/full/10.1177/1098214018824417) by Jones et al. (2020) Northwest Housing Alternatives concept paper: Program Evaluation and Data Culture in Resident Services (https://evaluationintoaction.com/resources/) Graphic mentioned - NW Housing Alternatives: Housing Stability Status (https://www.dropbox.com/s/pz6lw09dncsce8v/NWHousingAlt%20-%20Housing%20Stability%20Status.jpg?dl=0): This graphic was created as a result of the evaluation planning session with Resident Service Coordinator staff. We wanted to visually depict where their program activities fell on a spectrum of residents that are at a high risk of losing housing to a low risk of losing housing. This graphic was included in their program evaluation plan. Data into Dollars: The Intersection of Program Evaluation & Fundraising (https://evaluationintoaction.com/2020/01/13/data-into-dollars/) blog post Chari's presidential strand session (https://www.evaluationconference.org/p/cm/ld/fid=794) with AEA president Aimee White on Tuesday, October 27 New Directions for Evaluation: Organizational Capacity to Do and Use Evaluation (https://onlinelibrary.wiley.com/toc/1534875x/2014/2014/141) Two people mentioned: Michael Quinn Patton (https://www.utilization-focusedevaluation.org/our-team) and Stephanie Evergreen (https://stephanieevergreen.com/) Contact: Chari Smith: chari@evaluationintoaction.com EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guest: Chari Smith.

This week I am joined by David Keyes. We talked about R, why evaluators should learn R, tips for teaching courses, and more! Interested in getting started with R? Check out David's free course (https://rfortherestofus.com/courses/getting-started/)! Interested in learning how to do inferential statistics in R? Check our our new course (https://rfortherestofus.com/courses/inferential-statistics/)! Resources: Blog posts * Dana's R journey (https://rfortherestofus.com/2019/03/my-r-journey-dana-wanzer/) * What is a tidyverse-centric approach? (https://rfortherestofus.com/2019/06/what-is-a-tidyverse-centric-approach/) * How to make beautiful tables in R (https://rfortherestofus.com/2019/11/how-to-make-beautiful-tables-in-r/) * How to make functions in R (https://rfortherestofus.com/2019/10/how-to-make-functions-in-r/) * How to evaluate R packages (https://rfortherestofus.com/2020/07/how-to-evaluate-r-packages/) * Using Word reference documents with R Markdown to create custom reports (https://rfortherestofus.com/2020/07/word-reference-documents-rmarkdown/) * If you care about equity, use R (https://rfortherestofus.com/2019/07/equity/) Other resources * Daniel Lakens' course Improving your statistical inference (https://www.coursera.org/learn/statistical-inferences) * My manuscript written entirely in R (https://osf.io/a3zfj/) * Todoist (https://todoist.com/) * Savical (https://savvycal.com/) and Calendly (https://calendly.com/) * Figma (https://www.figma.com/) * QuickBooks (https://quickbooks.intuit.com/) * WordPress (https://wordpress.com/) and SquareSpace (https://www.squarespace.com/) * Evaluation Jobs (https://evaluationjobs.org/) * Teachable (https://teachable.com/) * Streamlabs (https://streamlabs.com/) Some packages: tidyverse (https://www.tidyverse.org/), beepr (https://cran.r-project.org/web/packages/beepr/index.html), praise (https://cran.r-project.org/web/packages/praise/index.html), janitor (https://garthtarr.github.io/meatR/janitor.html), gt (https://gt.rstudio.com/), gtsummary (https://github.com/ddsjoberg/gtsummary), lavaan (https://lavaan.ugent.be/), ggplot (https://ggplot2.tidyverse.org/reference/ggplot.html), rstatix (https://rpkgs.datanovia.com/rstatix/), papaja (https://crsh.github.io/papaja_man/) Contact: David Keyes: Twitter @dgkeyes (https://twitter.com/dgkeyes) and @RfortheRest (https://twitter.com/rfortherest) and email david@rfortherestofus.com EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guest: David Keyes.

ThisThis week I am joined by Ann K. Emery about her journeys in evaluation and entrepreneurship. Ann K. Emery is an internationally-acclaimed speaker who equips organizations to get their data out of dusty spreadsheets and into real-world conversations. She now runs Depict Data Studio (https://depictdatastudio.com/). If you’d rather watch this talk, you can do so on YouTube here (https://youtu.be/v1aRlVOnx0g). Topics & Resources Mentioned: #EvalTwitter hashtag on Twitter (also follow #eval and #dataviz) Company of One (https://ofone.co/) by Paul Jarvis Nina Sabarre (https://www.gotostage.com/channel/546a35201d924876a61df57d3b0d2259/recording/28b91d4463d843c1a78a4a97c3da8b80/watch?tos=true&ticket=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI4MDMwMjYyNjg5OTMwMjE3OTg5IiwiYXVkIjoiMTAxIiwibHMiOiJkNTEzMzI0ZC0yMjBkLTQzMTEtYWJiOC1iYmE0Y2U5MDUxMTkiLCJvZ24iOiJsaSIsImV4cCI6MTU5MDU4MjcxOSwidHlwZSI6IjEiLCJqdGkiOiJlYWIzZTE1Yi03ZWNmLTQ5YTYtOGM5MC0wY2E5YWRjM2VhYTYiLCJ0ZW5hbnQiOiIkZGVmYXVsdCIsImxvYSI6MiwidGd0ZXhwIjoxNTkwNTkzMzM5fQ.ryQP2302VMRhftv8W_hjH20TsNwjcfMzWihCkUH2fvE) presentation on getting into independent consulting Independent consulting vs. entrepreneurship Teaching (and blogging and YouTube-ing) as a way to give back to the community Multiple references to Chris Lysy (https://freshspectrum.com/) Your Money or Your Life (https://yourmoneyoryourlife.com/book/) by Vicki Robin All the books by Octavia Butler (https://www.fantasticfiction.com/b/octavia-e-butler/), and the podcast (https://podcasts.apple.com/us/podcast/octavias-parables/id1519024926) on her parables by Adrienne maree brown & Toshi Reagon with Kat Aaron Rethinking consulting from a mindset of abundance, not scarcity Farnoosh Torabi (https://www.instagram.com/farnooshtorabi) on Instagram 99 Tips for Creating Simple and Sustainable Educational Videos (https://styluspub.presswarehouse.com/browse/book/9781642670851/99-Tips-for-Creating-Simple-and-Sustainable-Educational-Videos) by Karen Costa Soar Beyond the Dusty Shelf Report (https://depictdatastudio.teachable.com/p/soar-beyond-the-dusty-shelf-report) course Contact: Ann K. Emery: Instagram (https://www.instagram.com/annkemery/?hl=en) and Website (https://depictdatastudio.com/) EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guest: Ann K. Emery.

This week, we’re chatting with AnnJanette, or Anjie, Rosga about objectivity, trust and numbers, truth and power, and more. AnnJanette (Anjie) Rosga, PhD (https://informingchange.com/about/team/annjanette-rosga/), co-directs the small, Berkeley-based consulting firm Informing Change (https://informingchange.com) which works to support strategic learning initiatives in the social sector. Most of Anjie’s projects involve facilitating strategy development, promoting equity and participatory research, and leading evaluation for hard-to-measure initiatives that cross disciplines, sectors, geographies, and/or cultures. This episode centers around Anjie's 3-part Medium series on Trust & Numbers. Read the 3-part series here: Part 1 (https://medium.com/@arosga/trust-in-numbers-bf585010224e), Part 2 (https://medium.com/@arosga/trust-numbers-part-2-8c4a8509f82f), and Part 3 (https://medium.com/@arosga/trust-numbers-part-2-8c4a8509f82f) Some resources, quotes, etc. mentioned: Power as electricity metaphor - Understanding "New Power" (https://hbr.org/2014/12/understanding-new-power) by Jeremy Heimans and Henry Timms Rethinking objectivity (https://www.dukeupress.edu/rethinking-objectivity) by Allan Megill Mariana Valverde: Truth as a "pragmatic ethical choice" (In Law's Dream of a Common Knowledge, page 10) Decolonizing Wealth (https://www.decolonizingwealth.com/thebook) by Edgar Villanueva Level Up: Growing Your Approach to Participatory Evaluation (AEA365) (https://aea365.org/blog/level-up-growing-your-approach-to-participatory-evaluation-by-elizabeth-diluzio/) by Elizabeth DiLuzio Grace Jones in I'll Never Write My Memoirs: "If you are a fan of doing the unexpected, and I am, then it is an advantage to be highly skilled at changing your mind. If you do not want to limit yourself, then be prepared to change your mind—often." Cultivating trust on Twitter - NYT Daily Podcast (https://www.nytimes.com/2020/08/07/podcasts/the-daily/Jack-dorsey-twitter-trump.html) Contact: Anjie Rosga: Contact her through the Informing Change (https://informingchange.com/about/team/annjanette-rosga/) website EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Special Guest: Anjie Rosga.

This week I discuss some of the latest Research on Evaluation (RoE) articles that have come out in 2020. Articles Mentioned: At the beginning I mention the RoE TIG (http://comm.eval.org/researchonevaluation/home)’s definition of RoE, which is summarized here (https://danawanzer.com/whatisroe/). Evaluation Journal of Australasia – Special Issue on Values in Evaluation Vol 19, Issue 4 (https://journals.sagepub.com/toc/evja/19/4): Guest Editors: Keryn Hassall, Kelly Hannum, Amy Gullickson and Mathea Roorda Vol 20, Issue 2 (https://journals.sagepub.com/toc/evja/20/2): Guest Editors: Ayesha S Boyce, Amy M Gullickson, Keryn Hassall and Kelly Hannum New Directions for Evaluation Volume 2020, Issue 166 - Examining issues facing communities of color today: The role of evaluation to incite change (https://onlinelibrary.wiley.com/toc/1534875x/2020/2020/166): Issue Editors: Leah C. Neubauer Dominica McBride Andrea D. Guajardo Wanda D. Casillas Melvin E. Hall Neubauer, McBride, Guajardo, Casillas, & Hall (2020) Examining issues facing communities of color today: The role of evaluation to incite change (https://onlinelibrary.wiley.com/doi/10.1002/ev.20406) Hall (2020) Blest be the tie that binds (https://onlinelibrary.wiley.com/doi/10.1002/ev.20414) Reid, Boyce, Adetogun, Moller, & Avent (2020) If not us, then who? Evaluators of color and social change (https://onlinelibrary.wiley.com/doi/10.1002/ev.20407) Ghanbarpour, Mercado, & Palotai (2020) A language justice framework for culturally responsive and equitable evaluation (https://onlinelibrary.wiley.com/doi/10.1002/ev.20412) Dhaliwal, Casey, Aceves-Iniguez, & Dean-Coffey (2020) Radical inquiry—liberatory praxis for research and evaluation (https://onlinelibrary.wiley.com/doi/10.1002/ev.20415) Guajardo, Robles-Schrader, Aponte-Soto, & Neubauer (2020) LatCrit theory as a framework for social justice evaluation: Considerations for evaluation and evaluators (https://onlinelibrary.wiley.com/doi/10.1002/ev.20409) Moss & Crewe (2020) The Black perspective: A framework for culturally competent health related evaluations for African Americans (https://onlinelibrary.wiley.com/doi/10.1002/ev.20413) Lemos & Garcia (2020) Promoting culturally responsive and equitable evaluation with Latinx immigrants (https://onlinelibrary.wiley.com/doi/10.1002/ev.20410) Bowman (2020) Nation-to-nation in evaluation: Utilizing an indigenous evaluation model to frame systems and government evaluations (https://onlinelibrary.wiley.com/doi/10.1002/ev.20411) McBride, Casilla, & LoPiccolo (2020) Inciting social change through evaluation (https://onlinelibrary.wiley.com/doi/10.1002/ev.20405) Neubauer & Hall (2020) Is inciting social change something evaluators can do? Should do? (https://onlinelibrary.wiley.com/doi/10.1002/ev.20408) Lemire, Peck, & Porowski (2020) The growth of the evaluation tree in the policy analysis forest (https://onlinelibrary.wiley.com/doi/full/10.1111/psj.12387) Hurteau, Rahmanian, Houle, & Marchand (2020) The role of intuition in evaluative judgment and decision (https://journals.sagepub.com/doi/10.1177/1098214020908211) Brown & Di Lallo (2020) Talking circles: A culturally responsive evaluation practice (https://journals.sagepub.com/doi/full/10.1177/1098214019899164) MERL Tech State of the Field Series (http://merltech.org/the-state-of-the-field-of-merl-tech/) Engage R&D (2020) Listening for change: Evaluators of color speak out about experiences with foundations and evaluation firms (https://www.engagerd.com/blog/listening-for-change) Contact: EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/

“If you are someone who is interested in bringing change, or making the world a better place, or if you would like the results of your work to get used, evaluation is something that you might be interested in.” This episode is a bit different. I asked three of my students to reflect on their experiences in the MS in Applied Psychology program at University of Wisconsin-Stout, as well as their experiences in the two evaluation courses. Emma, Omar, and Jade share their thoughts and reflections. Click here to learn more about the MS in Applied Psychology program (https://www.uwstout.edu/programs/ms-applied-psychology). If you have feedback about this episode, any previous episode, or the podcast in general, please email me at podcast@danawanzer.com or go to our website https://evaluland.fireside.fm (https://evaluland.fireside.fm). Special Guests: Emma Frenn, Jade, and Omar Albaraidi.

This week I am joined by Dr. Ayesha Boyce, assistant professor of educational research methodology at the University of North Carolina at Greensboro, to discuss teaching evaluation and supporting students and colleagues of color. Topics Covered: Reflections on five years of teaching evaluation The -ologies: ontology, epistemology, and axiology beyond just methodology Favorite evaluation activity (of course, it’s the evaluating the cookie activity!) Mentoring students of color. Five keys from Ayesha (see upcoming CJPE article): Consider impact of vicarious trauma Assist with facilitation peer mentors/squads Respect, honor, and celebrate students’ culture, religion, and family Be vigilant of microaggressions, and practice microvalidation Develop mentoring competence Supporting colleagues of color: helpful actions, silence as complicity, listening in, getting comfortable being uncomfortable Approaching institutional leaders about actions for supporting students, faculty, and staff of color Speaking out on the tenure track Having a family and being exceptional at work simultaneously (Ayesha encourages us all that it IS possible!) Resources mentioned: Boyce & McGowan (2018) article “An exploration of two novice evaluation educators’ experiences developing and implementing introduction to evaluation courses (https://journals.sagepub.com/doi/10.1177/1098214018778812)” Evaluating the cookie activity in Preskill & Russ-Eft’s Building Evaluation Capacity book (https://us.sagepub.com/en-us/nam/building-evaluation-capacity/book241833) Upcoming issue in CJPE on evaluator education Ibram X. Kendi (https://www.ibramxkendi.com/) Letter (https://twitter.com/AyeshaBoyce/status/1273662617035452419) from Dr. Boyce and students and faculty of color to their department Letter (https://racialequity.uncg.edu/letter-from-black-uncg-faculty/) from Dr. Boyce and other black UNCG faculty NDE upcoming issue on evaluator education (https://drive.google.com/file/d/16A60D7b7Rqajib1-Egx4FAh4TdQVdkne/view) Contact: Ayesha Boyce: Twitter @AyeshaBoyce (https://twitter.com/AyeshaBoyce) and ayesha.boyce@uncg.edu EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guest: Ayesha S. Boyce.

This week I am joined by Dr. Jennifer Ann Morrow, the program coordinator of the Evaluation Statistics and Measurement doctoral program (https://epc.utk.edu/evaluation-statistics-measurement/) (soon to be named "Evaluation Statistics and Methodology) at the University of Tennessee-Knoxville. We talked about teaching evaluation and data cleaning in evaluation. Topics Covered: Jennifer’s background, philosophy, and experience in teaching evaluation (01:58) Our favorite evaluation activities: evaluating cookies, one-pages on hot topics and people in evaluation, evaluation failures, self-reflection journals, practical experience outside of class (11:33) Personal professional development in data visualization and R (32:00) What’s giving Jennifer life in evaluation right now (38:40) Data cleaning book proposal by Jennifer and Gary Skolits (44:13) What’s next for Jennifer (50:55) Resources mentioned: Preskill & Russ-Eft (2015) Building Evaluation Capacity: Activities for Teaching and Training (https://us.sagepub.com/en-us/nam/building-evaluation-capacity/book241833) Wanzer (2020) What is evaluation paper (preprint) (https://osf.io/c9pf7/) Hutchinson (2018) Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned (https://us.sagepub.com/en-us/nam/evaluation-failures/book260109) Jennifer’s student’s blog on user experience for Ann K. Emery (https://depictdatastudio.com/three-takeaways-from-the-user-experience-ux-field-to-up-your-data-viz-game/) Great Graphs with Ann K. Emery](https://depictdatastudio.com/) R for the Rest of Us with David Keyes](https://rfortherestofus.com/) Amy Cesal Play-Doh data visualization activity (https://www.amycesal.com/day-doh-viz-all) Bianca Montrosse-Moorhead’s research on what ought to be included in the curriculum for master’s and doctoral eval programs (https://www.researchgate.net/publication/337032339_Evaluator_education_curriculum_What_ought_to_be_taught_in_master's_and_doctoral_programs) Jennifer’s handout on the 12 steps to data cleaning](http://comm.eval.org/qual/viewdocument/cbd146-a-brief-intr-1) Contact: Jennifer Ann Morrow: @evaluationdiva on Twitter (https://twitter.com/evaluationdiva) and jamorrow@utk.edu EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guest: Jennifer Ann Morrow.

This week I am joined by Zach Tilton, a Doctoral Research Associate at the Interdisciplinary PhD in Evaluation Program at Western Michigan University. We discussed research on evaluation and meta-evaluation. Topics Covered: How Zach got into evaluation and at Western Michigan University (01:12) What peacebuilding evaluation is (03:52) Evaluation specialists vs generalists (05:45) Research on evaluation and meta-evaluation (09:50) Prescriptive vs descriptive theories (22:35) Evaluation theory (25:00) Zach’s dissertation: RoE on peacebuilding evaluation (35:50) Adjusting to life amidst Covid-19 (40:10) What’s next for Zach (46:00) What’s giving life for Zach (50:55) Resources mentioned: Western Michigan University Interdisciplinary PhD in Evaluation (https://wmich.edu/evaluationphd), including Michael Harnar (https://wmich.edu/evaluationphd/directory/harnar) and Chris Coryn (https://wmich.edu/evaluationphd/directory/coryn) EPP article on knowledge syntheses (https://www.sciencedirect.com/science/article/abs/pii/S0149718919302605?via%3Dihub) AEA Research on Evaluation TIG (http://comm.eval.org/researchonevaluation/home) Blog describing working group’s updated RoE definition (https://danawanzer.com/whatisroe/) Weaver & Cousins article on participatory evaluation (https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/144) AEA Integrating Technology into Evaluation TIG MERL Tech conference (http://merltech.org/) Everyday Peace Indicators (https://everydaypeaceindicators.org/) Digital Impact Alliance (https://digitalimpactalliance.org/) Evaluation Practice Reconsidered by Thomas Schwandt (https://www.worldcat.org/title/evaluation-practice-reconsidered/oclc/47200552) EvalCentral Unwebinars (https://www.crowdcast.io/e/evalcentral/), Blogs (https://blog.evalcentral.com/), and Discussion Forum (https://evalcentral.com/) Equitable Evaluation Initiative (https://www.equitableeval.org/) Eval4Action Campaign (https://www.eval4action.org/) Black youth-led movements: Black Visions Collective (https://www.blackvisionsmn.org/) Reclaim the Block (https://www.reclaimtheblock.org/home), The Twin Cities Coalition for Justice for Jamar (https://www.facebook.com/TCC4J/), and Minnesota Freedom Fund (https://minnesotafreedomfund.org/) Contact: Zach Tilton: @ZachTilton on Twitter (https://twitter.com/zachtilton) and zachary.d.tilton@wmich.edu EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guest: Zach Tilton.

This week I am joined by Kathleen Doll, who recently completed her PhD at Claremont Graduate University. We discuss her dissertation on strategic planning in evaluation design. Topics Covered: Why did Kathleen choose to get a PhD in evaluation at Claremont Graduate University? (02:12) How did Kathleen get to her dissertation topic of strategic evaluation? (05:09) Dissertation design (07:56) What is strategic evaluation planning? (09:25) Results! What are the benefits of strategic evaluation planning? (10:44) Who should be doing strategic evaluation planning? (16:22) How does strategic evaluation planning align with developmental evaluation? (19:20) How does this work with funders? (21:47) How Kathleen’s work has been impacted by Covid-19 (23:34) Topics in evaluation giving Kathleen life: trauma-informed evaluation, research on evaluation working group, and working on an NDE volume (27:30) What’s next for Kathleen after the PhD (35:52) Resources mentioned: Kathleen's dissertation handout on Strategic Evaluation (https://tinyurl.com/drdoll2020) Martha Brown’s eStudy on Trauma Informed Evaluation (https://www.pathlms.com/aea/courses/15601) J. Bradley Cousins’ webinar on Getting RoE Published (http://comm.eval.org/researchonevaluation/tigresources) Contact: Kathleen Doll: LinkedIn (https://www.linkedin.com/in/kathleenmkdoll) & kathleendoll101@gmail.com EvaluLand: Website (https://evaluland.fireside.fm/) & Twitter (@EvaluLand (https://twitter.com/evaluland)) Music by Matt Ingelson, http://www.mattingelsonmusic.com/ Special Guest: Kathleen Doll.

Welcome to EvaluLand! This episode is a brief introduction to the podcast and to me, your host, Dana Linnell Wanzer. If you’d like to learn more about the podcast, check us out! * Website: evaluland.fireside.fm (https://evaluland.fireside.fm/) * Twitter: @EvaluLand (https://twitter.com/EvaluLand) * Email: podcast@danawanzer.com