Few years ago, I was asked to carry out an impact evaluation (IE) of a donor-assisted project in one of the states in Nigeria as part of the project’s mid-term review. The first challenge I ran into was the unavailability of an approved and acceptable baseline study that would have provided the much-needed baseline data central to conducting an effective impact evaluation.
The State project management unit did conduct a baseline study at the beginning of the project. The study was however fraught with several issues that ended up invalidating many of the data reported in the study. I was frankly advised to ignore the report and figure out an alternate solution since using data presented in it would logically raise questions about the accuracy of the review I was hired to carry out.
The second challenge is the fact that even if the baseline study had been accepted and its data were accurate, they were mostly not the type of data I needed. Hence, the data in the study would still have been largely unusable. This is an instance of a case of data passing a test of reliability but failing that of validity.
So how does this information help anyone planning to conduct an impact evaluation of its project now or in the future? Well, it might not really be helpful for those who have gone beyond this stage in their project implementation, but should be for those who are just about starting out.
If you are yet to conduct a baseline study for your project. This is absolutely the time to make plans for your impact evaluation. Why you may ask? Well, the answer is not far-fetched. Planning for your IE at this stage enables you to avoid the pitfalls I enumerated earlier on.
You will get to determine the kind of data you need to generate from your baseline study. For example, considering your project objectives and the key performance indicators (KPIs) identified, what kind of data would you want to collect to evaluate the impact of your planned intervention at mid-term or at the end of the project? Would the data be quantitative or qualitative, secondary or primary? Do keep in mind the fact that you will essentially be collecting the same data again at a later period.
You will also get to determine the population to target and the kind of samples to draw that would be relevant for your IE too. This is particularly crucial if you plan to conduct a truly scientific IE where you will be employing the use of an intervention and control sample. This is because as we know it, a control group is a group that does not receive any intervention and is compared statistically with the group that does at a later stage in the project lifetime.
This is the perfect and ideal scenario for any development project that values and appreciates a truly scientific IE. However in reality, this opportunity is often missed due to a number reasons ranging from lack of proper understanding of the requirements of project IE to external factors and conditions beyond the control of the project management team such as problems with choosing control groups, political obstacles, and government policies, etc.
In coming articles, we shall be discussing at length how best to conduct IE of our projects in the light of all the prevailing circumstances and peculiar challenges commonly faced by project managers in this clime.