Logo of Big Brother Mouse, publishing books in Laos
 

How do evaluate the impact of a project?

Years ago, someone who worked at a small foundation commented to me, "I never knew it would be so hard to give away money!" At the time, I couldn't imagine why.

A home-based library in Laos. This woman is looking at

Now, living in Laos, I have quite a different perspective. I'm involved in Big Brother Mouse, which I naturally think is worthy of support. I often wonder how to convince others of that; periodically I try to step back and ask: If I were a donor or foundation, how would I decide where are the best places to put my money?

"There are no easy answers," as columnist Nicholas Kristof has repeatedly said. (Anyone else who's given the matter much thought has surely said the same thing. I mention Kristof because while I don't always agree with his conclusions, his thoughtful columns offer good insight on these questions.)

Many organizations in Laos are working in the fields of education, health, development, child protection, and so on. Anyone who looks around carefully can see a full range of approaches and theories in action. There is, of course, plenty of waste to be found. You probably don't need to look beyond your own country's government to find that. There are also small projects, with limited budgets and a few dedicated people, producing results far beyond what might be expected. Less obviously, every project, is exerting an influence in hidden ways. It's never easy to step back and weigh them all against one another.

Precisely because there are no easy answers, we need many people to contribute whatever experience and insights they have, and I've decided to add my observations to the mixture. This began as a list of "questions that I wish funders would ask." But they also apply to those of us involved in local projects, such as myself, who want to periodically step back and evaluate what we're doing. It reflects my experience in Laos, and some will not be relevant in other circumstances.

Who's hitting the jackpot?

I don't mean, "Is the bookkeeper making regular deposits to a secret bank account?" though that's something to keep an eye on, too. Rather, of the ultimate beneficiaries, who benefits, and how is that decided? Look at all the beneficiaries: Staff, who get jobs; members of the public, who get benefits; government officials, who may be getting something, somewhere along the way. You can't keep that all secret. Other people know who's getting money, and how, and are going to copy whatever behavior worked. Somewhere, your money is rewarding and encouraging certain behavior. What behavior are you encouraging?

Recently a young waiter at a restaurant, asked what I did, so I told him about Big Brother Mouse and showed him a few books we've published. He was quite interested.

"Do you have any jobs?" he asked.

"Yes, for people who have the skills that we need."

"Can I apply?"

"Yes. But you'll need to talk to Khamla."

Immediately his face fell. "Lao person?" he asked. I nodded. He promptly lost all interest.

I've had that experience a dozen times. Young people in Laos have learned that the best path to money and success is not to study hard or work hard, but to land a job with a foreign NGO or the government.

Big Brother Mouse sets up a home-based village library in Laos

What about us? Well, I contributed to that. Early in 2006, when Khamla and a few other college students had helped me get our first six children's books ready for press, we needed an entity to act as publisher. For many reasons, I preferred to be the adviser to a Lao-owned project, rather than to start an NGO or foreign-owned company. Khamla had greatly impressed me with his intelligence, ability to learn, and desire to help people, so I suggested that we form a business: He would be the owner, I would be the volunteer adviser.

We had no idea it would grow as fast as it did, and I didn't think about concerns I've explained above. Had I thought about it, I would probably have gone ahead anyway: The negative consequence seems greatly outweighed by what we've accomplished. But I did reinforce the perception that a good way to get ahead is to get involved in a project with a foreigner. That will need to be weighed into any evaluation of our impact.

An aid project continually influences behavior by the way it uses money. NGOs never pay bribes to corrupt officials, of course. But they often pay a "consulting fee". When bright young people turn their backs on teaching or business jobs, and compete for certain government positions, they're not doing it for the official salary.

What's the impact on local businesses?

NGOs and non-profit organizations begin with a sense of mission and purpose. That's a good start. But that can lead to a sense of superiority over anything so crass as a business. Agreed, businesses are sometimes destructive; the drive for private profit easily conflicts with public good. But it's usually businesses, within a framework established by government, that do the best job of distributing needed goods and services, establishing prices, and allocating resources.

Businesses also create an engaged middle class, a population that believes things can and should improve, and that is willing and able to pressure the government to greater responsiveness.

Western countries developed their economies as businesses competed against each other. In today's less-developed countries, money flowing in from abroad can become a monkey wrench in that process. Foreign organizations, using donated funds that don't have to be earned in the local economy, compete against struggling local businesses. NGOs freely outbid local businesses for what is often the scarcest resource of all: Skilled staff.

Yes, they're creating job opportunities. But they're helping a few individuals, at a cost to the local economy. A more positive contribution would be to search out those motivated and talented people who haven't yet had access to training, and thus increase the pool of skilled workers, rather than using superior resources to hire away workers who are needed by, and often were trained by, local businesses.

Foreign aid money can undercut the economy in other ways. The American Food AID program pays US farmers and shippers to send food to poor countries, where it can be sold at below-market cost. Who could be against sending food to hungry people? But in some areas of Africa, local farmers, who were undercut, weren't able to survive. Arguably, the U.S. should have given small subsidies to the local farmers, rather than big payouts to American farmers and shippers. But Africans don't vote in U.S. elections, and they don't contribute to American political campaigns. CARE deserves great credit for pulling out of this program in 2007, despite losing substantial income from food sales.

What about us? Big Brother Mouse falls somewhere between a business and NGO. Most of our income comes as donations from outside, which pay for us to deliver specific books and services to schools and villages. We could raise our prices on those books, and pay salaries competitive with NGOs. We don't: We pay at the high end of the local market scale. I prefer this because we shouldn't be competing against local businesses. Khamla, who ultimately sets salaries, sets them at this level because he knows the market here and doesn't overpay for anything.

If you provide funds to an organization in a poor country, is that money being used to compete against locally-owned businesses?

Is it sustainable?

Big Brother Mouse sets up a home-based village library in Laos

Has anyone ever started a project and not said it was sustainable? We can all become too optimistic about sustainability. A study of UN projects found a great many that were deemed highly sustainable at the proposal stage, but which collapsed when the UN funding and involvement ended.

First, a definition: "Sustainable" has been used in many senses, and often refers to environmental sustainability. Here, I mean a organization which, after some initial period of requiring outside assistance, will be run by local people, and will not be dependent on outside funding.

Evaluating long-term sustainability is, of course, "no easy task." Let's start by realizing that we'll all too easily join those UN proposal-writers in being too optimistic. Then we can look at two elements of sustainability: money and management.

Where does the funding come from now? Is that a good long-range source? Will new sources be needed? Can the costs, at some future point, by paid by the people who benefit? (Surely that's the best route to long-term sustainability, and it's one reason a shoestring operation often outlasts one that's heavily funded.) Whatever future sources are expected, to what extent are those expectations based on tangible evidence and trends that are visible now?

Similar questions can be asked about management: If an outside staff is handling key responsibilities now, what's the plan and time-scale for local staff to step into those roles? Do current trends support that projection? One revealing indicator is language. If the local and non-local staff speak different native languages, what's the language in which major decisions are discussed and made? Control and responsibility will move toward those who speak that language best.

What about us? Here at Big Brother Mouse, like everywhere else, hindsight and idle speculation are much easier than accurately predicting the future. The Lao economy is growing about 7-8% a year now. If that growth continues, even at a slower rate, it's realistic to expect that in 10 years, most villages will be able to support a small local library, based at a school, temple, shop, or home. The real danger isn't that they won't be able to, but that they'll be so in the habit of waiting for foreign NGOs to come in and do things for them, that they'll have no infrastructure, no leadership, and no confidence to do such things on their own.

Right now, with median rural incomes below $800 per year, we feel it's right to donate books to villages and schools, but in ways that develop, rather than discourage, local initiative and involvement. We've begun having festivals to initiate our home-based village libraries. Over the course of a few years, we believe these festivals have the potential to become annual village celebrations that raise funds for a library or other village projects.

* * *

I promised not to offer any easy answers, and that was an easy promise to deliver on. The process of writing this page was helpful to me, and I hope has been helpful to some readers. If you have experience in this type of work, I would be very happy to hear your comments and observations; you can reach me through the "Contact Us" page on this site.