Rakesh Mohan has graciously agreed to share his insight on leveraging outside consultants to increase the evaluation capacity of his office. Rakesh has been the director of the Office of Performance Evaluations (OPE) since 2002. His office is an independent, nonpartisan agency of the Idaho State Legislature. OPE is a small shop of six evaluators, an administrative coordinator, and the director. The office was recognized twice by the American Evaluation Association (AEA):

• 2016 Outstanding Evaluation Award
• 2011 Alva and Gunner Myrdal Government Evaluation Award

Small evaluation shops shouldn’t shy away from taking on large, complex studies. They can tackle these studies with confidence and produce useful evaluations by strategically contracting with consultants.

In 2008, the Idaho legislature asked my office to conduct an evaluation of the state’s transportation department. My staff and I did not know anything about construction and maintenance of highways and bridges. We had no experience in conducting a complex, or even a simple, transportation study. Despite this limitation, I agreed to undertake the study.

Why did I tell the legislature that my office could do this study? Being fully aware of the risk of failure and possibility of losing my job, I agreed to do the study for two reasons: (1) I wanted to demonstrate to the legislature that my office can deliver large, complex evaluations and be responsive to its information needs, and (2) I had confidence that I could assemble a team of subject matter experts and experienced evaluators to do such a study.

I worked closely with key legislators and told them about my plans. They placed their trust in me and appropriated a significant sum of $550,000 to conduct the study. This amount was in addition to my annual appropriation of $822,200 to run the office and conduct other studies during the year.

I contracted with 11 consultants as one team and three individual consultants. Together these 14 consultants completed the evaluation on time in six months and 20% under budget. I managed the entire evaluation—from designing the scope to issuing the final report. The legislature and the governor used the report to make policy and budget decisions. The transportation department used the report to improve its operations.

Over the years I have used consultants for three main reasons: (1) acquire subject matter expertise for evaluations, (2) conduct quality control reviews before evaluations are publicly released, and (3) provide customized training for my staff.

Subject Matter Expertise. The transportation study shows how consultants can be used to conduct an entire study. My office also uses consultants as subject matter experts for specific areas within a study. These consultants work as team members with my staff. Recent examples of studies where we used consultants as team members include a wide range of topics:

Representation for Children and Youth in Child Protection Cases

State Jurisdiction in Indian Country

Child Welfare System

K–12 Education Funding

Risk of Bias in Administrative Hearings

Challenges and Approaches to Meeting Water Quality Standards

Quality Control. Before I migrated to evaluation work, I had been trained as a performance auditor. Audit standards require that audit shops go through a peer review once every three years. The purpose of a peer review is to assure that audit shops comply with audit standards and adhere to their audit policies and procedures. I found one critical limitation with this type of quality control—it is conducted after the audits become public. I believe a better peer review process would be the one that ensures the quality of an evaluation before it becomes public.

I began to use consultants for selective quality control work on our evaluations. This external review is in addition to the quality control review that evaluators perform in-house. Consultant reviews vary depending on project needs; they can take as few as 10 hours to as many as 100 or more hours. Below is a generic example of the scope of work for a consultant:

Quality Control Review Tasks The contractor agrees to conduct a quality control review and communicate her/his comments to the director and team lead via email, phone, or other means as necessary. Specifically, the contractor will do the following:
1	Review the project scope.
2	Review the initial project plan and any significant subsequent changes to the initial plan (this includes reviewing methodologies used to address study objectives).
3	Review key workpapers involving complex analyses (both qualitative and quantitative). Determine whether methodology used in the analysis was appropriate, the analysis was performed accurately, and the conclusions drawn from the analysis were logical.
4	Participate in a report message meeting with the team.
5	Review the report message write-up (the report write-up should clearly identify the main message based on the key findings).
6	Review the report draft for logic, flow, and accuracy based on evaluators’ work using the following criteria: (a)   Whether the report is logical (b)  Whether the methodology is sound (c)   Whether the conclusions and recommendations are adequately supported (d)  Whether the tone of the report is balanced (e)   Additional ideas to improve the quality of the report
7	Verify cross-referencing between the report draft and workpapers for accuracy by checking selected portions of the report.
8	Review the report’s one-page highlights and press release for accuracy, readability, and tone.
Professional Standards All work will be conducted in accordance with one of the following sets of standards: Generally Accepted Government Auditing Standards, US Government Accountability Office American Evaluation Association’s Guiding Principles for Evaluators Program Evaluation Standards, the Joint Committee on Standards for Educational Evaluation

Customized Training. I invite consultants to give my staff customized training. These training sessions have covered different types of analyses such as qualitative, quantitative, cost-benefit, and staffing and workload. Other trainings have included data visualization, leadership, and working in public policy environments. Trainings have lasted from two hours to two days. They have an added benefit—I can assess whether the trainer could be a good consultant for future evaluations.

There is no substitute for having a highly trained and motivated staff—and I’m fortunate to have such staff. By complementing them with the use of consultants, I have been able to take my staff to greater heights of evaluation capacity and excellence. I believe this approach could work well for any evaluation shop that has limited capacity to conduct large, complex studies. 

-Rakesh Mohan, Office of Performance Evaluations (OPE)