Deven Wisner, our very own AZENet President-Elect, has written a blog post that talks a bit about Evaluation "Soft Skills." What are soft skills in terms of evaluation, you may ask? Read a short excerpt from his post below: 

"As I consider our efforts to establish the technical skills needed to be effective practitioners, I propose that we should be simultaneously focused on the other part of the formula: soft skills. Because after all is said and done, if we can’t have meaningful dialogue with stakeholders…meeting them where they’re at…it seems aspirational to think that a reliance on our technical skills alone will result in the use of findings. Let’s start thinking beyond certifications and traditional forms of expertise. You might call it back to the basics on effective human interaction!"

Click here to go to Deven's blog and read more!

Deven Wisner, M.S., is trained in applied psychology, with concentrations in evaluation research, and industrial-organizational psychology. Deven is a consultant focused on bringing data and research based decision making into a variety of organizations. Proudly, he identifies as a “dataviz” nerd and enjoys storytelling through infographics. Deven’s research interests are in evaluation methods and approaches, as well as interpersonal effectiveness among evaluators and stakeholders.

Reflective Practice – A Concept for Practitioners Interested in Constant Improvement! Let Me Show You How! Tiffany Smith, Ph.D. University of North Carolina at Greensboro

 

*Note, anything underlined you can CLICK ON to learn more!

 Who am I, and Why Do You Care?!

My name is Tiffany Smith and I am the Senior Evaluation Specialist in the Office of Assessment, Evaluation, and Research Services at the University of North Carolina at Greensboro. That’s a lot to say! What’s important about who I am is that I am an evaluator as well as an educator and researcher on evaluation. For the past four years I have spent hours in the classroom preaching the benefits of the concept of “reflective practice” in the evaluation field and beyond. Similarly, my dissertation surrounded reflective practice, my current efforts toward training and research on interpersonal effectiveness in evaluation are undergirded by it, and my growth and development has been all thanks to the concept itself operating in my daily life. But, reflective practice can seem like a mystical and nebulous concept for practitioners and scholars alike. Let’s explore what reflective practice is, how I’ve used it, and the ways in which it has developed in my life personally -- as well as some fun and useful resources for you to be more purposeful about using it in your daily practice.

“Insights and innovation await us only if we are capable of stepping outside of the frenzied worlds of data and distraction that wash over us… time for reflection is an open invitation to discover what awaits us…” – Daniel Forrester Reflective practice, for those of you unfamiliar with the term, is a fancy way of saying “think critically about what you’ve done in your practice in order to learn from it for next time.” The Essential Competencies for Evaluators outlines reflective practice as pivotal to our work, as does the Program Evaluation Standards, and so too do numerous seasoned evaluation professionals (Michael Quinn Patton, Jean King, Hallie Preskill, and many others!). I like to think of the reflection process as both a personal journey and a collaborative one. On my own, I can spend time truly reflecting, journaling or making note of problems and potential solutions during my week. But, I can also spend time opening myself and my practice up to others, seeking critical friendship from trusted colleagues and friends along the way. This sheds the light of multiple perspectives on my dilemmas of practice! I for one, believe in the power of collaborative reflection, as you will see below. However, both individualized reflection and collaborative reflection are just a part of healthy professional practice. “It’s hard to look at modern life and see our capacities for reflection or meaning-making.  We don’t use our gifts to be more aware or thoughtful.  We’re driven in the opposite direction.  Things move too fast for us to reflect, demanding tasks give us no time to think, and we barely notice the lack of meaning...  we cannot stop life’s dynamic of self-reference or the human need for meaning. If we want to influence any change, anywhere, we need to work with this powerful process rather than deny its existence.” – Margaret J. Wheatley Ways of Reflecting I have recently transitioned from a teaching faculty position to a full-time practicing evaluation position. When I moved to North Carolina this summer to begin my new job, it was, by nature, a reflective time in my life. As I have moved through the Fall semester, getting up to speed on evaluation projects, pitching our office’s services, and learning about our department’s culture, I have absolutely NEEDED reflective practice to grow as a professional. So, what have my reflective practices been? The answer is that the reflective process is embedded in everything I do, and I think making the process a “practice” means making a concerted effort toward including it in your way of doing things in order to improve yourself. Here are some areas where I see reflection in my life. Reflective Practice as Journaling: Journaling in any capacity is just good reflective practice. Keeping notes on projects and meetings, being able to review and reflect on them as you go. I have been a proponent of Bullet Journaling ever since two Christmases ago when one of my favorite past masters students, Deven Wisner (you might know him!) gifted me a Moleskine notebook and set of Staedtler pens to begin a journey of making note of my life. The bullet journal is freeing, in that you create your agenda as you go. You decide what goes in your journal, how, and why. If you’re interested in learning more about bullet journaling, go here. I like to interrupt my bullet journal every so often and take a few pages to write reflections on my day, week, or semester. Over the course of this summer and fall the way I have begun to bullet journal has shifted, and I’ve noticed! More future-oriented, more time breaking down tasks for better task management, and more time spent taking note of moments when further, more critical reflection need to happen. Speaking of which… Reflective Practice as Dialogue with Critical Friends: Journaling and reflecting on my own is great, but I think we can all agree that when we spend too much time thinking on our own about something, we tend to get really frustrated (even stuck). I have found that having a wealth of good “critical friends” I can connect with to bounce ideas off of, as I move through the daily challenges and learning opportunities associated with practice, is invaluable to my work. It is because of good critical friendship that I have stronger surveys, healthier logic models, more polished and thoughtful evaluation plans, and more productive conversations with stakeholders. And, generally, I feel like I have learned, and can grow, from my practice. How do you find good critical friends? They can be anywhere! I suggest finding people who challenge you, will shoot it to you straight, and are willing to spend the time thinking it through with you. If they know the subject matter, great! If not, your ability to explain it to them may unlock keys to your mind that you may not know exist. And, of course, you should be willing to return the favor and lend an ear when your critical friend needs it! “You could almost say there is a Zen of practice, and part of that is being present not just in the moment, but present all the way through the experience so that you can look back… it’s almost like there’s two of you so… it’s a cognitive process, it’s an intuitive process, and it’s an iterative process, where you review, you think, you study what you’re doing, you study what you have done. You look at the results that came out of that and then you make a judgment accordingly… So it really is the process of conscious and intuitive thought that come together.” – Dissertation Participant Utilizing the above two ways of reflecting are just a couple elements of reflective practice as a way of being. I think that in order to be our best selves, as well as our best evaluator-selves, we have to be reflective about what we do. To be thoughtful about what our actions were for that day, for that project, for that conversation. And, to take advantage of that thoughtfulness by delving deeper, asking ourselves why, and planning how to move forward having learned for next time. As one of my dissertation participants said, it is like a Zen of practice, that we are aware of ourselves, as ourselves and as evaluators. My reflection has led me to fun places these past few months, as I have been finding gems of books to read to further my passion for learning about science, communication, and social change. [Here are some books I have been reading, for fun : We Make the Road By Walking, Mystery of Mysteries, If I Understood You Would I Have This Look on My Face?!] Using the DATA Model for Reflection: See you at AEA? I’ve spent the past few paragraphs talking to you about my reflective experiences and their benefits, and I haven’t even gotten to the model that I have written about in the Journal of Evaluation and Program Planning. If you are an evaluator or educator who is looking for a systematic way to reflect on your practice, or a model to help walk through the reflective process alone or with others, the DATA Model for reflection is a very useful process. If you’re in Cleveland, come join Libby Smith, Deven Wisner , and I as we walk you through a skill-building workshop called “Reflective Practice Should Not Be Optional: Exploring an Essential and Unforgettable Competency for Evaluators”! The session is 8AM-9AM (hey! I didn’t pick this time!) on Friday November 2nd in Veteran’s Meeting Room A. See the session abstract and additional details here: AEA Reflective Practice Skill-Building Workshop. I am very much looking forward to this year’s conference. This theme is a very reflectively oriented one in my eyes, and as such I hope to dialogue with you about it there! Come join us, get your degree in evaluation (MS or PhD) in the Program Evaluation Track at UNCG. Learn more, here: Educational Research, Measurement, and Evaluation at UNCG Connect with me on Twitter or LinkedIn!

Do you want to make a positive social impact with your work? 

When you work with clients, when you present your research...why do you do it? 

Is it just a CV-builder? If so, then this probably isn’t for you :) But my guess is that it's because you actually want people to be inspired and motivated enough to use that information for social good. 

When you present about the strengths of a program or policy, I'm guessing it's because you your clients to keep those elements. 

When you present about the weaknesses of a program or policy, I'm guessing it's because you want them to implement changes to improve those things.

You get the point. 

But what if I told you that you'll probably never achieve that goal if you're presenting in the standard way. 

You've heard of the "Standard American Diet" (SAD) right? Well I'm calling this the "Standard Academic Delivery" (also SAD). 

Here's what the SAD looks like:

• Slides that consistently have 50+ words per slide.
• Presentations that follow the 1 minute per slide rule.
• Presentations with no storytelling, passion, excitement, or enthusiasm. 
• Overwhelming presentations that have too much information and not enough audience engagement. 
• Presentations that aren't well-rehearsed. 
• Presenters who say things like, "Oh that's a lot of text, I'm not going to read all that." *next slide*
• Dataviz that's just graphs/charts copied & pasted from a pub. 
• No visuals. 
• Visuals that are word clouds, puzzle pieces, hands shaking, or bubble men (WTF ARE THOSE OMG).
• Presentations where the main "visual" is a slide template. 

Evaluators aren’t immune from this just because they aren’t (always) “academics.”

So how can you prevent yourself from following the SAD? It is, after all, standard...which means it's the status quo. That's both good news and bad news. 

The bad news is that is there's a lot of pressure and nay-saying to keep people following the status quo. Any time you're going against the status quo, you will have people trying to convince you to "get back in line." Just remember: That's usually because they don't want the bar raised and they don't want to be required to put more effort into their presentations. When they tell you that effective, visual presentations are unprofessional, they're wrong. What's unprofessional about creating presentations that inspire people to act? That increase the retention of scientific or academic information? That increase the accessibility of complex info? That's exactly what professionalism is to me. 

Look. There's a reason #DeathByPowerpoint is the SAD--it's easy. 

The good news is that most presentations are boring, ineffective, and unengaging. If you get started on creating better presentations now, then you actually don't have to do all that much to create presentations that stand out for being awesome! You can learn the basics and stand out, and then increase your skills from there to be even more awesome.

Don't wait until most people are creating awesome presentations because then you'll be scrambling to catch up. 

This is something that even hummingbirds can demonstrate. Let's check in on my friend, the sword-billed hummingbird, so I can show you what I mean. 

Please keep in mind that I'm not a scientist who studies birds, so this is a highly simplified summary with a sprinkle of Echo Flair :) 

OK, so here's the setup.  

We're in South America, and there are lots of flowers and pretty hummingbirds here.

Which is kinda cool, we have a nice community of hummingbirds here...except there's also a lot of competition for food. A single flower's nectar only goes so far, amirite? 

That was like, super stressful, for some hummingbirds at one point. I can relate to that. It's kinda like being anti-capitalist in a capitalist society. I hate competing with folks. I'd rather we just, I dunno, collaborated and sh*t to end all these problems. But hey. I'm just one person.

Anyway. Then evolution happened and a new hummingbird came along: the sword-billed hummingbird. Unlike these standard hummingbirds, this one has a really long bill. It probably didn't want to mutate and be different. It probably just wanted more flowers to be available, but hey.Evolution, amirite?

So I want you to guess: how do you think that turns into an advantage for this hummingbird? 

If you guessed that it can essentially cut in line to reach that flower first, you'd be ...

wrong! 

Their long bill let's them bypass the whole competitive environment altogether. 

Their long bill let's them access whole new--really long--flowers, like those in the Passiflora and Datura family. 

As a result, this hummingbird effectively has no competition (from other hummingbirds) for food. 

Think about it. That's a less stressful existence for this hummingbird. It can float around the flowers, bypassing the crowd, and achieve its food goals without worrying about others. 

I want you to have that same advantage when you're presenting your work. 

The hummingbirds are born the way they are--they have no choice to have either a short bill or a long bill. Luckily, you have a choice when it comes to designing ineffective presentations or engaging presentations. 

Right now, the SAD is ineffective presentations. But, there's nothing to say that pretty soon the SAD will be effective, engaging, awesome presentations. In fact, I'm trying to change the SAD from #DeathByPowerpoint to #InspiredByPowerpoint. 

So now I ask you: Would you rather be part of this movement with me? Or, would you rather be sucker punched in 5 years and realize that now your slides stand out for being ineffective :( 

Clearly, I want you to join me. But here's some more illustrations to make my point even clearer (hopefully). 

An Illustrated Story of the Standard Academic Delivery (SAD)

People using the SAD start with slide templates. 

And then type their entire presentation and read from their slides to their audience, apologizing for this (but still doing it every time). When they realize they're running out of time they start saying things like, "Oh that's a lot of text, I'm not going to read that to you" ... and then skip a few slides.

Pretty soon, a pattern emerges across presenters. All templates, almost all text, no good storyboard or storytelling, and no inspiration. The audience is bored, stares at their phone, wondering why these always feel like a waste of time. 

Not a single audience member thinks, "Well, at least they're being professional!" 

But then a breath of fresh air comes along.

Someone who actually practiced the presentation beforehand and delivers it with enthusiasm. Someone who doesn't use a template, uses effective information design principles, and uses the slides as a visual complement rather than as their speaker notes. 

The audience is actually interested, engaged, and checking their phone approximately 50% less this time. They feel like they learned something new, feel empowered to act, and actually want to get to know this presenter more and make a note to network with them later. 

Which one do you want for you?

I know which one I want for you :)

Want more tips? I’ve made a free download available that has more effective presenting tips. You can down download those presentation tips here.  

About the author

Hi! I’m Dr. Echo Rivera, owner of Creative Research Communications, LLC via echorivera.com. My passion is helping researchers, academics, scientists, and evaluators become effective visual communicators. I love training folks on how to create astronomically awesome slide presentations for lectures, conferences, and workshops. I also love to draw comics and want to see more comics used in research, evaluation, and teaching. I’d love to connect with you on twitter, YouTube, Facebook, Instagram, or LinkedIn.

psst...this blog includes copyrighted illustrations I made. You can share this post widely, but if you want to use these, email me first!

This blog post is an adapted version of an article that originally appeared on Echo's blog. 

Rakesh Mohan has graciously agreed to share his insight on leveraging outside consultants to increase the evaluation capacity of his office. Rakesh has been the director of the Office of Performance Evaluations (OPE) since 2002. His office is an independent, nonpartisan agency of the Idaho State Legislature. OPE is a small shop of six evaluators, an administrative coordinator, and the director. The office was recognized twice by the American Evaluation Association (AEA):

• 2016 Outstanding Evaluation Award
• 2011 Alva and Gunner Myrdal Government Evaluation Award

Small evaluation shops shouldn’t shy away from taking on large, complex studies. They can tackle these studies with confidence and produce useful evaluations by strategically contracting with consultants.

In 2008, the Idaho legislature asked my office to conduct an evaluation of the state’s transportation department. My staff and I did not know anything about construction and maintenance of highways and bridges. We had no experience in conducting a complex, or even a simple, transportation study. Despite this limitation, I agreed to undertake the study.

Why did I tell the legislature that my office could do this study? Being fully aware of the risk of failure and possibility of losing my job, I agreed to do the study for two reasons: (1) I wanted to demonstrate to the legislature that my office can deliver large, complex evaluations and be responsive to its information needs, and (2) I had confidence that I could assemble a team of subject matter experts and experienced evaluators to do such a study.

I worked closely with key legislators and told them about my plans. They placed their trust in me and appropriated a significant sum of $550,000 to conduct the study. This amount was in addition to my annual appropriation of $822,200 to run the office and conduct other studies during the year.

I contracted with 11 consultants as one team and three individual consultants. Together these 14 consultants completed the evaluation on time in six months and 20% under budget. I managed the entire evaluation—from designing the scope to issuing the final report. The legislature and the governor used the report to make policy and budget decisions. The transportation department used the report to improve its operations.

Over the years I have used consultants for three main reasons: (1) acquire subject matter expertise for evaluations, (2) conduct quality control reviews before evaluations are publicly released, and (3) provide customized training for my staff.

Subject Matter Expertise. The transportation study shows how consultants can be used to conduct an entire study. My office also uses consultants as subject matter experts for specific areas within a study. These consultants work as team members with my staff. Recent examples of studies where we used consultants as team members include a wide range of topics:

Representation for Children and Youth in Child Protection Cases

State Jurisdiction in Indian Country

Child Welfare System

K–12 Education Funding

Risk of Bias in Administrative Hearings

Challenges and Approaches to Meeting Water Quality Standards

Quality Control. Before I migrated to evaluation work, I had been trained as a performance auditor. Audit standards require that audit shops go through a peer review once every three years. The purpose of a peer review is to assure that audit shops comply with audit standards and adhere to their audit policies and procedures. I found one critical limitation with this type of quality control—it is conducted after the audits become public. I believe a better peer review process would be the one that ensures the quality of an evaluation before it becomes public.

I began to use consultants for selective quality control work on our evaluations. This external review is in addition to the quality control review that evaluators perform in-house. Consultant reviews vary depending on project needs; they can take as few as 10 hours to as many as 100 or more hours. Below is a generic example of the scope of work for a consultant:

Quality Control Review Tasks The contractor agrees to conduct a quality control review and communicate her/his comments to the director and team lead via email, phone, or other means as necessary. Specifically, the contractor will do the following:
1	Review the project scope.
2	Review the initial project plan and any significant subsequent changes to the initial plan (this includes reviewing methodologies used to address study objectives).
3	Review key workpapers involving complex analyses (both qualitative and quantitative). Determine whether methodology used in the analysis was appropriate, the analysis was performed accurately, and the conclusions drawn from the analysis were logical.
4	Participate in a report message meeting with the team.
5	Review the report message write-up (the report write-up should clearly identify the main message based on the key findings).
6	Review the report draft for logic, flow, and accuracy based on evaluators’ work using the following criteria: (a)   Whether the report is logical (b)  Whether the methodology is sound (c)   Whether the conclusions and recommendations are adequately supported (d)  Whether the tone of the report is balanced (e)   Additional ideas to improve the quality of the report
7	Verify cross-referencing between the report draft and workpapers for accuracy by checking selected portions of the report.
8	Review the report’s one-page highlights and press release for accuracy, readability, and tone.
Professional Standards All work will be conducted in accordance with one of the following sets of standards: Generally Accepted Government Auditing Standards, US Government Accountability Office American Evaluation Association’s Guiding Principles for Evaluators Program Evaluation Standards, the Joint Committee on Standards for Educational Evaluation

Customized Training. I invite consultants to give my staff customized training. These training sessions have covered different types of analyses such as qualitative, quantitative, cost-benefit, and staffing and workload. Other trainings have included data visualization, leadership, and working in public policy environments. Trainings have lasted from two hours to two days. They have an added benefit—I can assess whether the trainer could be a good consultant for future evaluations.

There is no substitute for having a highly trained and motivated staff—and I’m fortunate to have such staff. By complementing them with the use of consultants, I have been able to take my staff to greater heights of evaluation capacity and excellence. I believe this approach could work well for any evaluation shop that has limited capacity to conduct large, complex studies. 

-Rakesh Mohan, Office of Performance Evaluations (OPE)

Hello! I’m Sheila B. Robinson, Ed.D of Custom Professional Learning, LLC. I’m an educator, consultant, and program evaluator with a passion for professional learning. I design and facilitate professional development courses on program evaluation, survey design, data visualization, and presentation design. My book, Designing Quality Survey Questions, was published by Sage Publications in 2018.

I’m an active American Evaluation Association member and currently Lead Curator and content writer for AEA365 Tip-A-Day By and For Evaluators, their daily blog on program evaluation, and Coordinator of the Potent Presentations Initiative (p2i). I’m Chair of both AEA’s Professional Development Workgroup, and Awards Committee. I’m also past Chair and Program Chair of the Pre-K12 Educational Evaluation Topical Interest Group (TIG).

I have taught graduate courses on program evaluation and professional development design and evaluation at the University of Rochester’s Warner School of Education where I received my doctorate in Educational Leadership and Program Evaluation Certificate. 

The Potent Presentations Initiative (p2i) is sponsored by the American Evaluation Association and exists for the explicit purpose of helping evaluators improve their presentation skills, both at conferences and in individual evaluation practice. Potent Presenters think about three key components of a compelling presentation: Message, Design, and Delivery, and our free resources are largely organized around these areas.

P2i got its start back in 2012 with Stephanie Evergreen, data visualization and presentations expert, and has continued to grow over the years, with several people contributing to the effort, including me! Initial identification of the three key components and inspiration for much of the early content and came from a study of a dozen highly regarded AEA presenters dubbed the “Dynamic Dozen,” who had the highest feedback scores from pre-conference professional development and Summer Institute workshops, and Coffee Break Webinars.

Our p2i website features a p2i Presentation Tools & Guidelines page with free, downloadable checklists and worksheets, along with webinars and slides on Message, Design, and Delivery. Check out the Poster Presentations page for additional guidance and tools for posters. And our Presentation Preparation Checklist is a perennial favorite that guides you along a well-paced schedule to a prepared and calm session. Our newest resources are the Audience Engagement Strategy Book and Guidelines for Handouts tool, my contributions to the p2i collection. Up next will be a resource dedicated to helping you use images in presentations.

As current p2i coordinator, I’m always on the lookout for ideas for new content, and engage with an advisory team a few times per year for input. In a recent effort to catalyze ideas I asked evaluators to participate in a brief, informal survey about presenting to help inform our future work. The survey asked about people’s past and future presentation work and what might help them improve their practice. I shared the link to “You, the Presenter: What Would Help You Up Your Game?” in an aea365 post and on EvalTalk, and 190 people responded! The results are quite informative, and I’ve been sharing them monthly in my p2i column in the AEA monthly newsletter.

Are you getting ready to submit a proposal to present at a conference? Have you been hired by an organization to give a presentation? Or, do you have an upcoming presentation you need to make to your colleagues? It’s a great time to check out AEA’s Potent Presentations Initiative and download free resources to supercharge your next presentation!

-Sheila B. Robinson, Ed.D., Custom Professional Learning, LLC

Happy monsoon season to all! This time of year is always a treat following a blistering couple of months. The AZENet Board of Directors is busy this summer diving into our 2018-2019 strategic plan for the Network. As a reminder, our work over this next year aligns with three key priorities: membership, sustainability, and community embeddedness. I welcome Network members to get involved to support the plan themselves in a few key ways.

Invite a colleague to your next AZENet membership event to get to know the network. Pass on professional development activities on your listserv or contact list. We welcome all evaluative types – not just those with the word “evaluator” in their name. Know an organization that does evaluative work who should be engaged in this Network? Share our website or contact the Membership Recruitment Chair to introduce them to us.

Consider joining our membership committee to expand your network and help us with outreach and recruitment. Contact Membership Recruitment Chair, Nicole Janich at Membership@azenet.org for more information and to get involved.

A key focus this year will be to connect with the next generation of evaluation professionals in post-secondary institutions in the state. Are you interested in participating in mentoring events for students and early career evaluators? The Network is also exploring opportunities to connect directly with our Arizona communities to provide professional development & outreach.

Interested in sharing your skills in a professional development workshop or have an idea for a community event? Contact the Professional Development Chair, Roseanne Schuster, at ProfessionalDevelopment@azenet.org.

I invite you to connect with us to grow your connections to evaluation in Arizona. Have a thought or comment? Email me directly at President@azenet.org.

In Evaluation,

Jenny McCullough Cosgrove, AZENet President/CEO