“Throughout my career I have told people that I landed on my feet as an anthropologist when I became an evaluator.” (p. 76) In the phrase that seems to be not only the summary of her life and career, but also the leitmotif of her book, Mary Odell Butler expresses all that Evaluation is about: the meeting of the two disciplines on one hand, and their entanglement with her career path. In its eight chapters, the book ambitiously aims to introduce evaluation to anthropologists, anthropology to evaluators, and to map the difficulties, controversies, challenges and possible rewards of what doing evaluation anthropology means.
In her endeavor to fit together the two disciplines and introduce evaluation anthropology to beginners in either one of the two parts – evaluation or anthropology (p. 22) – the author gives all the details to understand the overarching picture. The eight chapters follow a clear order, where several threads (What is evaluation? What is anthropology? Who are the evaluators, and what do anthropologists do?) are woven into the re-emerging web of what evaluation anthropology is, how it can be done, and what important aspects to take into account, should one consider a career – academic or not – in or with evaluation anthropology. The “figure in the carpet” is re-emerging, after the author skillfully (albeit perhaps necessarily artificially) delimits the two approaches: the one of the anthropologist and the one of the evaluator. The book, in its clear linear structure can, thus be separated into two parts. The first part, which can be summarized as “understanding evaluation and anthropology”, comprises the first four chapters. It is a theoretical-introductory part, with some overview of the history and paradigms of the two fields, careful definitions and explanations of how she uses and defines the core concepts, such as evaluation, ethnography, culture, community, system and contexts. It is admirable that in her writing Butler maintains a reflexive stance and allows for the many other possible approaches, while in the same time is able to provide a conceptual and theoretical framework for beginners to work with. She thus stresses the necessity for evaluations to be scientific, but also opens paths for young researchers to find their own methods and theoretical stances through learning and unlearning some of the disciplinary certainties anthropologists (or, in fact, evaluators) might be biased towards (p. 19).
The next four chapters are within the framework of “how to do evaluation anthropology”, and discuss very practical methodological and ethical perspectives, while providing the reader not only with the know-how of doing research, but also that of building a career in evaluation anthropology. These general guidelines become especially lifelike through the author’s candid accounts of her own career experiences.
Evaluation being defined as “a study that seeks to scientifically determine the value of something for some kind of utilization” (p. 25), and anthropology as the “study of culture and its role in articulating what humans do as members of societies” (p. 72), the overlap that evaluation anthropology becomes (p. 71), is inevitably a discipline laden with ideas and relations of power. The two chapters on ethics and methodology raise important questions that anthropologists and evaluators have to engage with throughout their research. Whose interests are served? How do we protect vulnerable groups and people from being compromised by our research data? When do we have informed consent and how do we honor the trust of our informants? What are the conflicts between the necessity for scientific objectivity and negotiating through the needs and wants of stakeholders or participants? How do we deal with these? Who decides about how to use the data and about the consequences of research? Perhaps one of the most important statements in the book is that ethics “are not common sense. It requires training, experience, and judgment to make ethical decisions” (p. 108). Indeed, our professional training is (still) lacking in engaging discussions, seminars, and informative courses with regards to ethical guidelines, something the author also laments. The need for such discussions is, however, there – demonstrated by the multitude of online forums and conference discussions. One of the greatest merits of Evaluation is the way it further invites conversation about the relationship of the discipline and the solving of social problems, of evaluators and stakeholders, of decision-making and scientific research. Good research in evaluation, as in anthropology, depends on building trust with stakeholders (p. 12), and evaluators “have ethical obligations to stakeholders seeking scientific ‘truth’ or verisimilitude” (Harklau and Norwood 2005, 279). It is, therefore, vital to reflect on how subjectivities change in the evaluation process, how and with whom power-relations are negotiated, and the role and influence of the evaluator-anthropologist in decision-making processes, considering that “program evaluators hold the power to affect the very nature and future of the phenomena they investigate” (ibid., 278).
Considering all this, why evaluation anthropology, though? What new does it bring to the table? The author herself questions in the introduction the necessity to create a subfield by merging the two disciplines. Why not just evaluation? Or why not anthropology? Her answer is, that evaluation anthropology is stronger than its parts (p. 18), as academics and practitioners from both disciplines come together to mutually share and learn from each other’s expertise. Qualitative and quantitative data are both necessary for research that is on the one hand scientific enough to serve as the base for policy-change and to influence stakeholders in positions of power to make decisions in the right directions. On the other hand, such research is attentive to the sensibilities of wider social contexts and advocates for the needs of vulnerable groups. It is in this sense that there is a necessity for the two disciplines to converge, and bring ethnography as much as possible to the forefront of research. Constantly being in positions of negotiation not only with vulnerable research participants, but with powerful stakeholders as well, evaluation anthropology becomes one of the rare subfields that not only hold the possibility to investigate social problems and the effects of government and NGO programs on marginalized populations, but it also has a “playing area” that allows for researchers to “study up” (Hopson citing Nader 2005, 293), i.e. to study and ask questions of the groups who hold power, who make the all-important decisions. In reading Butler’s work, one gets the feeling that this tool and focus of evaluation anthropology is not yet applied. Practicing evaluators, and in most cases academic anthropologists also, depend on governmental or non-profit agencies for their research. In Evaluation, the author somewhat balances this moral dilemma in expressing her views on ethical best practice. Given, that “our work is neither scientifically, nor politically neutral” (Harklau and Norwood 2005, 281), picking up the thread of conversation on this matter is necessary in order for future practice not to end up compromising the needs of vulnerable groups, or indeed, researchers.
Compared to the 2005 NAPA Bulletin special issue, Creating Evaluation Anthropology: Introducing an Emerging Subfield, which she co-edited with Jaqueline Copeland-Carson, in Evaluation Butler focuses somewhat more on program evaluation – the area she has most experience in. In both cases, however, we only read about evaluation and evaluation anthropology done in the US or as commissioned in the US. As it turns out, the reader learns more about American society, programs and policy-making, than about the applicability and usefulness of doing evaluation anthropology to assess government and civil programs anywhere. The strong focus on US evaluation anthropology alone leaves the reader wanting more, even though this is a somewhat understandable bias, given the history and context of emergence of the subdiscipline. However, is there really no need for such research, or are there no similar endeavors elsewhere? Is evaluation anthropology not applicable outside the American social and political contexts? Perhaps being more concerned with ethics and methodology, rather than a complete review of the theories and history of the discipline, there is no room for such detours in Evaluation, it would, however, have been interesting if the book entered into conversations on this level too.
Mary Odell Butler proposes her book to be “about how to link evaluation and anthropology into a dynamic and flexible approach that brings out the vitally human nature of the ways people organize themselves to accomplish their goals” (p. 9). While an ambitious objective, she does manage to shed new light on how the two disciplines can inform and challenge each other, even subtly suggesting that evaluation anthropology is a necessary, but necessarily evolving transdisciplinary process. She does not intend to give a complete theoretical overview of the two parent-disciplines, or their hybrid, nor a complete methodological or ethical guideline, yet, through mapping possible approaches and career paths, and – often painfully honestly – narrating her own experiences, she provides the reader with the necessary starting points to understand anthropological and evaluation theory, and to develop their own methodologies. In doing so, she also advises about very practical issues, such as agencies, organizations, websites and tips on how to move on in their career. All in all, Evaluation is both an interesting and a useful read, where one is able to enjoy the personal narratives, gain an understanding of the basic conceptual tools and frameworks, and come back to the useful links and tips whenever they are needed.
References Cited:
Butler, Mary Odell, and Jaqueline Copeland-Carson (eds.) 2005. Creating Evaluation Anthropology: Introducing an Emerging Subfield. NAPA Bulletin 24 (1). Berkeley, CA: University of California Press.
Harklau, Linda, and Rachel Norwood. 2005. Negotiating Researcher Roles in Ethnographic Program Evaluation: A Postmodern Lens. Anthropology and Education Quarterly, 36 (3), pp. 278-288.
Hopson, Rodney K. 2005. Reinventing Evaluation. Anthropology and Education Quarterly, 36 (3), pp. 289-295.
© Imola Püsök
