6
Reviews Materials For Learning: How To Teach Adults At A Distance by Janet Jenkins, London, Routledge and Kegan Paul, ISBN. 0-7100-0808-2 198’1, pp. 209, g7.95 1 have found ~a~e~~u~s for Learnirrg a valuable manual in many respects. It contains a wide range of information particularly useful to the organiser and practitioner of distance education. The author’s assertions are solidly backed by citations from empirical research findings in various parts of the world, as well as being equally rooted in authorita- tive theoretical foundations. The author’s concept of the key terms of distance education and adult learning, her perception of the relationship between fanguage, culture and Iearning, particularly between culture and adult learning, aptly reflect recent trends of thinking in adult education. Apart from the merit derived from the validity and relevance of its content, Materials for Learning is remarkably lucid in presentation. The simplicity of the language coupled with the copious illustrative devices consistently employed by the author, saves the book from the handicap of any obscurities. Besides clarity of language, the preview and review portions of each chapter are remarkably didactic. While the preview tends to reduce the reader’s tendency to drift, the review acts, as an antidote against the reader’s forgetfulness. In Muteriais for Learning, both techniques are rendered more effective through a highly personalised tone and the association of the author’s self with the reader through the use of ‘we’ and ‘us’. Materials for Learning is, however, less successful in its structural features. The location of the notes at the end of the chapter makes reference work rather cumbersome. The method of inserting the reference numbers in-between sentences in the body of the book tends to disrupt the flow of reading and to distract the reader’s concentration. The reference notes themselves are rather incomplete particularly with regard to identification of publishers and date of publication. The periods generally lack nexus with the result that naturally related points are rather detatched from one another and the natural connections between ideas tend to be missed by the reader. In both content and structure the main body of Materials for Learning is considerably successful. Chapter Five on “Planning for Distance Teaching” is strikingly didactic in spite of its being rather fragmentary especially under the topic, “Resources and Constraints.” Chapter Six on “The different media’ is one of the most useful to practitioners. It contains several suggestions drawn from a rich source of information. Chapter Seven on “Writing Simply” contains highly informative and useful details but perhaps too many technicalities for easy grasping by the non-specialist reader. Chapter Eight on using pictures is perhaps the most heavily illustrated of all the chapters. It is a vivid and stimulating chapter but lacks explicit linkages to distance eucation. Chapter Nine on, “The Production of Printed Materials” discusses a material which is fundamental to the correspondence form of distance education. But as in Chapter Seven, over-elaboration of minute details of a technical nature tends to dominate the presentation and to outweigh the more relevant issue of the potential of the print media in distance education. The two chapters on programme production namely, chapters ten and eleven are moderately successful. Chapter Ten presents mostly the positive aspects of the use of educational radio programmes. Problems arising from this use are merely touched upon but not amply discussed particularly as they relate to developing countries. There is paucity of guidelines on the specific approaches to be folIowed by distance maths and science teachers in the use of materials for teaching adults at a distance. The chapters on approaching maths and science, however, contain a very useful background exposition on traditional maths and an in-depth clarification of the scientific method. In the final chapter the author is predominantly scepticat on blue-prints, while at the same time expressing a ray of optimism about the possibility of finding sdutions to the problems she raised in the course of ~ater~a~~ for Learning. The problems, however, appear to outweigh the anticipated solutions. But while it may be said that the conclusion of Materials for Learning portrays the author in a rather pessimistic light and does not appear to be perfectly consistent with the undeclared purpose, Materials for Learning was written to serve, namely that of a manual of special practical utility to practitioners of distance adult education, it is equally tenable to see Materials for Learning as one of the rare manuals in which the reader finds beautifully blended, both indications and contra-indications, prescriptions and precautions. Oliver 0. Anowor Evaiuation a Systemafic Approach Peter W. Rossi, Howard E. Freeman, Sonia R. Wright Sage Publications, 1979 Toward Reform of Program Evaluation Lee J Cronbach and Associates, Jossey - Bass, 198 1 Effective Evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches, Egon G Guba and Yvonna S Lincoln, Jossey-Bass, 1981 These three books by leading authorities iltustrate the range of approaches that can now be found within the general area of program evafuation. Their assumptions about purposes, processes, methods and utilisation are so different that one might be forgiven for thinking that these different species of evaluator lived on different planets. Anyone contemplating a close encounter with an evaluator 291

Effective evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches: Egon G Guba and Yvonna S Lincoln, Jossey — Bass, 1981

Embed Size (px)

Citation preview

Page 1: Effective evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches: Egon G Guba and Yvonna S Lincoln, Jossey — Bass, 1981

Reviews

Materials For Learning: How To Teach Adults At A Distance by Janet Jenkins, London, Routledge and Kegan Paul, ISBN. 0-7100-0808-2 198’1, pp. 209, g7.95

1 have found ~a~e~~u~s for Learnirrg a valuable manual in many respects. It contains a wide range of information particularly useful to the organiser and practitioner of distance education. The author’s assertions are solidly backed by citations from empirical research findings in various parts of the world, as well as being equally rooted in authorita- tive theoretical foundations. The author’s concept of the key terms of distance education and adult learning, her perception of the relationship between fanguage, culture and Iearning, particularly between culture and adult learning, aptly reflect recent trends of thinking in adult education.

Apart from the merit derived from the validity and relevance of its content, Materials for Learning is remarkably lucid in presentation. The simplicity of the language coupled with the copious illustrative devices consistently employed by the author, saves the book from the handicap of any obscurities. Besides clarity of language, the preview and review portions of each chapter are remarkably didactic. While the preview tends to reduce the reader’s tendency to drift, the review acts, as an antidote against the reader’s forgetfulness. In Muteriais for Learning, both techniques are rendered more effective through a highly personalised tone and the association of the author’s self with the reader through the use of ‘we’ and ‘us’.

Materials for Learning is, however, less successful in its structural features. The location of the notes at the end of the chapter makes reference work rather cumbersome. The method of inserting the reference numbers in-between sentences in the body of the book tends to disrupt the flow of reading and to distract the reader’s concentration. The reference notes themselves are rather incomplete particularly with regard to identification of publishers and date of publication.

The periods generally lack nexus with the result that naturally related points are rather detatched from one another and the natural connections between ideas tend to be missed by the reader.

In both content and structure the main body of Materials for Learning is considerably successful. Chapter Five on “Planning for Distance Teaching” is strikingly didactic in spite of its being rather fragmentary especially under the topic, “Resources and Constraints.” Chapter Six on “The different media’ is one of the most useful to practitioners. It contains several suggestions drawn from a rich source of information. Chapter Seven on “Writing Simply” contains highly informative and useful details but perhaps too many technicalities for easy grasping by the non-specialist reader. Chapter Eight on using pictures is perhaps the most heavily illustrated of all the chapters. It is a vivid and

stimulating chapter but lacks explicit linkages to distance eucation. Chapter Nine on, “The Production of Printed Materials” discusses a material which is fundamental to the correspondence form of distance education. But as in Chapter Seven, over-elaboration of minute details of a technical nature tends to dominate the presentation and to outweigh the more relevant issue of the potential of the print media in distance education. The two chapters on programme production namely, chapters ten and eleven are moderately successful. Chapter Ten presents mostly the positive aspects of the use of educational radio programmes. Problems arising from this use are merely touched upon but not amply discussed particularly as they relate to developing countries. There is paucity of guidelines on the specific approaches to be folIowed by distance maths and science teachers in the use of materials for teaching adults at a distance. The chapters on approaching maths and science, however, contain a very useful background exposition on traditional maths and an in-depth clarification of the scientific method.

In the final chapter the author is predominantly scepticat on blue-prints, while at the same time expressing a ray of optimism about the possibility of finding sdutions to the problems she raised in the course of ~ater~a~~ for Learning. The problems, however, appear to outweigh the anticipated solutions.

But while it may be said that the conclusion of Materials for Learning portrays the author in a rather pessimistic light and does not appear to be perfectly consistent with the undeclared purpose, Materials for Learning was written to serve, namely that of a manual of special practical utility to practitioners of distance adult education, it is equally tenable to see Materials for Learning as one of the rare manuals in which the reader finds beautifully blended, both indications and contra-indications, prescriptions and precautions.

Oliver 0. Anowor

Evaiuation a Systemafic Approach Peter W. Rossi, Howard E. Freeman, Sonia R. Wright Sage Publications, 1979

Toward Reform of Program Evaluation Lee J Cronbach and Associates, Jossey - Bass, 198 1

Effective Evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches, Egon G Guba and Yvonna S Lincoln, Jossey-Bass, 1981

These three books by leading authorities iltustrate the range of approaches that can now be found within the general area of program evafuation. Their assumptions about purposes, processes, methods and utilisation are so different that one might be forgiven for thinking that these different species of evaluator lived on different planets. Anyone contemplating a close encounter with an evaluator

291

Page 2: Effective evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches: Egon G Guba and Yvonna S Lincoln, Jossey — Bass, 1981

will need to consider carefully which species he prefers; and it is hoped that this essay-review will provide some assistance in clarifying the options.

Rossi, Freeman and Wright live on a planet governed by social scientists. Their background in quantitative positivist sociology but they also favour economics. The entities they evaluate are social intervention programs and most of their examples are chosen from health care, housing, welfare, criminology and education. Their prime concerns are summarised in the following excerpt:

Essentially, there are four sets of questions that one is concerned with in doing evaluations, corresponding to the four types of evaluation activities:

1. Program Planning Questions:

- What is the extent and distribution of the target pop~llation?

- fs the program designed in con- formity with its intended goals, and are chances of successful implementa- tion maximized?

2. Program Monitoring Questions:

- Is the program reaching the persons, households, or other target units to which it is addressed?

- Is the program providing the resources, services, or other benefits that were intended in the project design?

3. impact Assessment Questions:

- Is the program effective in achieving its intended goals?

- Can the results of the program be explained by some alternative process that does not include the program?

- Is the program having some effects that were not intended’!

4. Economic Efficiency Questions:

- What are the costs to deliver services and benefits to program participants’~

- Is the program an efficient use of resources compared with alternative uses of the resources? (~33)

Though elsewhere they define a comprehensive evaluation in terms of the last three categories, they still devote two substantial chapters to program planning. Their reason is clear:

The design and conduct of an evaluation can be undertaken competently only when the program is planned and implemented in ways that allow measurement of objectives, assessment of program implementation, and identification of the target popitIation. t’p.53)

292

Though their discussions of the political aspects of program planning and the problems of formula- ting objectives reveal many practical difficulties, Kossi et al. nevertheless claim that

unless goals can be clarified and objective\ operationalized, it is unlikely that an adequate evaluation can be attempted. (~63)

A second planning requirement is for an intervention model, a set of hypotheses which define how the program is expected to achieve its goals. However, since causal studies and logically developed theories of social behaviour are scarce, it is admitted that the sources of these hypotheses are usually either statistical associations or the impressionistic experience of professional practitioners in the relevant applied field. This seems a fundamental ffaw in an approach which consistently aspires to scientific status. Nor is one’s confidence enhanced by a chapter on ‘Research for Program Planning’ which advocates small research studies of a trouble- shooting nature. These are clearly necessary when plans are formulated at a remote distance from the context of application. But this catalogue of narrowly-averted disasters might lead some rentlcrs to wonder whether it is not the basic concept of the grand social intervention that lies at the heart of the problem.

Rossi’s chapter on ‘Monitoring Program Implementation’ is particularly welcome because there has been a tendency in the past to ignore this vital aspect of evaluation. T’he first question ib whether or not the program is reaching the ~~ppropriate target popLllation: Rossi discusses this in ierms of coverage and bias (defined as differential coverage of sub-groups), with attention also bcinp given to continuity of participation and dropout. Then second there is the question of the extent 10 which the program delivered corresponds to the original specifications. Rossi redefines this problem as one of finding adequate descriptions to indicate the salient characteristics of the program so that both contextual variation and conformity to the original design can be quantitatively analysed. The usefulness of this procedure depends wholly on the adequacy of’ the descriptions, but the authors are far too committed to the necessity for quantitative data to attend to this fundamental problem. Their xtrong commitment to using the original design as stand;tt-d :11so denies the possibility that program adaptation might be a beneficial process. Presumably the central planners always know best.

The most revealing statement about Rossi’s approach comes at the beginning of three chapter\ on impact assessment:

It is necessary to measure as rigorously iis possible the outcome of social interventions and to purify these results by removing the influences of forces other than the intervention hcing evaluated. (p 159)

This perhaps explains the authors’ strong preference fOi truing COflfi-01 groups. Consitle~~lt~~~ns of

Page 3: Effective evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches: Egon G Guba and Yvonna S Lincoln, Jossey — Bass, 1981

research design dominate the chapters on impact assessment to such an extent that discussion of measurement is confined to a few brief pages on reliability, validity and ‘proxy measures’. It is assumed that measurement normally presents no significant problems. Less rigorous methods discussed are limited to project records, site visits by experts, participants’ ratings and administrator’s reports. These approximate methods are cheap but definitely second best. The way has been cleared for a substantial discussion of cost-benefit and cost- effectiveness analysis which remains largely within the traditional economist’s framework of assumptions, with benefits being taken as estimated changes in income. The problem of evaluation has finally been reduced to a small set of figures, and it is vital that none of the questionable assumptions that have allowed us to get these should be allowed to cloud the clarity this brings to the decision-maker’s mind.

Rossi’s final chapter focusses on generalisation and on the utilization of evaluation by decision- makers. Here he urges that important policy issues need to be built into the design if the information is to have political utility, suggests that a program can be evaluated from several viewpoints and argues that there should be widespread consultation with interested parties before the evaluation begins. He does not discuss how this advice might conflict with the demands of his basic model for clearly specified goals, causal hypotheses and control groups; and the relegation to the very end of the book of these issues suggests an inability to come to terms with their implications. This becomes even more apparent when the authors admit that political time moves much #faster than evaluation time. Therefore, they suggest that the most technically complex evaluations (i.e. the only ones they are prepared to regard as competent) should be confined to

pilot and demonstration projects that are not likely to be expanded into large-scale programs in the near future;

and

before such programs appear upon the agendas of decision-making bodies. (~3101)

It looks suspiciously like a quick return to a planet governed solely by social scientists.

Meanwhile Cronbach and his co-authors have decided that the planet of social scientists is a mere figment of the imagination which has distracted evaluators from giving proper attention to the real world. ‘Indeed the need for reformation is so pressing that they begin their book with a list of 9.5 theses. Their criticisms of what they call the “stereotypical concept of evaluation” fall into two main categories. First, that the Rossi approach misunderstands the nature of the political process and is implicitly anti-democratic in its assumptions. Second, that it exaggerates the strengths, worth and relevance of social scientific knowledge. Cronbach’s book is explicitly directed at the U.S. policy-making context which makes it rather a long read for those

whose prime interest lies elsewhere. But the main substance of its argument has much wider significance and should command the attention of any government or international agency which seeks to commission and use social scientific research.

Cronbach’s starting point is that “a theory of evaluation must be as much a theory of political interaction as it is a theory of how to determine faCtS”(p3). Most previous theories of evaluation have assumed “a context of comn~and with a manager in firm control”, but such “rationalism is dangerously close to totalitarianism”. “An image of pluralistic accommodation more truly represents how policy and programs are shaped than does the Platonic image of concentrated power and respon- sibility” (~4).

“Information can change perceptions but is unlikely to bring all parties to agree on which facts are relevant or even on what the facts are. If the term decision is understood to mean a formal choice at a particular time between discrete alternatives, decision making is rarely to be observed. When there are multiple participants with multiple views, it is meaningless to speak of one action as the rational or correct action. The active parties jockey toward a politically acceptable accommodation.” (~84)

Like Rossi, Cronbach is primarly concerned with social intervention programmes and with events and decisions at the national level, but he avoids trying to mould the policy-making context to his own image. Hence his aspirations for the role of the social scientist are much more modest.

“The hope that an evaluation will provide unequivocal answers, convincing enough to extinguish controversy about the merits of a social program, is certain to be disappointed”. (P3)

“What is needed is information that supports negotiation rather than information calculated to point out the “correct” decision.” (~4)

Moreover while Rossi stresses the need for goal definition, Cronbach argues that it is unwise for evaluation to focus on whether a program has attained its goals. “Goals are a necessary part of political rhetoric” (~5). But they have to be kept broad and nebulous in order to obtain sufficient support. Hence to expect precise goals is to misunderstand the nature of the political process.

Evaluators in Cronbach’s model do not seek to achieve unambiguous recommendations by reducing complexity to a single cost-benefit analysis. They negotiate with, report to and seek to inform a policy-shaping covmunify - interested public officials, program personnel, constituents, and illuminators (reporters and academic commentators). The effective membership of this policy-making community varies according to people’s interest in a particular issue. The concerns of the community also vary over time quite rapidly and in a manner

293

Page 4: Effective evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches: Egon G Guba and Yvonna S Lincoln, Jossey — Bass, 1981

which it is difficult to predict. Hence negotiation

needs to be ongoing rather than confined to the

beginning and end of a particular study.

The communication of findings is also an

ongorng process which amounts to much more than a single report. Though the style and timing of a report arc vitally important, much of the communi- cation that matters will take place through other more informal channels; and not all of it will be

deliberate. Some of the most significant effects of an evaluation are indirect, and more likely to affect future projects than the immediate fate of the program studied. Where the impact of an evaluation is significant, it is more likely to be for its effect on understandings than on decisions, thus changing not so much the program itself but the prevailing view of social purpose. Hence a range of communi- cation devices need to bc considered that arc appropriate for the various audiences and their

information needs. “Commtlnication counts as much towards the contribution of an evaluation as the

adequacy of the data collection itself.”

‘this analysis of the relationship between evaluation and decision-making leads to a discussion of the allocation of research resources and the design of field studies that is radically different from Rossi’s. In particular the following principles are developed:

“It is better for an evaluative inquiry to launch :I small fleet of studies than to put all its

res0urces into a single approach.” (~7. see also p219).

‘. I he evaluator will be wise not to declare allegiance to either a quantitative-manipulativc- stimmative methodology or a qtialitative- naturalistic-descriptive methodology. He can dl-aw on both styles at appropriate limes and in appropriate amounts.” (p223 1. “External validity” - that is, the validity of inferences that go beyond the data - is the

crux; increasing internal validity by elegant clcsiyn often reduces rclcvance” (p7. see also p731).

It would bc unfair to the authors to attempt to pr-ccis their extensive discussions of research design. but a brief summary of their elegant analysis of external validity is worthy of special mention. Their argument on this istuc is that conventional practice attends to just one dimension of external validity -generating from :I sample of independent units. II. to the domain of all such units. U. Analysis then focuses on the choice of unit for analysis (perhaps individual pupil, or class. or school or community?), the selection of the sample of units, II, and the statistical procedures for making inferences about the domain. U. However. this common practice fails to recognise that the treatment. i.e. what happens to each unit. als0 varies. So vve also have to concern ourselves with the problem of generalisinp ft-om a sample of treatments. I. to the domain. 7‘. of all treatments that might be envisaged by the evaluation audicncc ;I\ fallin: M ithin the

relevant category. Similarly, the evaluation will be restricted to a sample set of observations, O, selected from the much larger domain, 0. of all possible observations. Hence the evaluator has to consider not just the problem of generalising from U/U to U/o. but that of genernlising from U~O to UTU. More problematic still is the possibility that the UT0 originally envisagcd will have changed to some new set of domains”’ UT0 by the time the evaluation has been completed and a wider range of audience concerns has developed.

Though Cronbach still uses the language and concepts of experimental research, he continually emphasises the limitations of evaluation evidence. “Comparison is invariably judgmental when program\ have multiple and perhaps dissimilar outcomes, and the judgements made are political. not analytic.” Even economists, he suggests “now see the cost- benefit model as a metaphor rather than as an algorithm”. (~287). One important conseqticncc. however, of interpreting the role of evaluation as the growth of understanding is that “evaluator\ should be trained to link facts to conceptual schemes and to criticize conceptualizations.” (p343). If taken seriously this would bring evaluation back into contact with thcorctical social science to the mutual

benefit of both.

Cuba and Lincoln attack the “stereotypical approach to evaluation” more fundamentally cvcn than Cronbach. Their argument is epistemologicat rather than political, and they advocate an alternative responsive and naturalistic paradigm to Rossi’s scientific experimentalism. Their book expounds and justifies the epistcmological basis of the natut-atistic paradigm and provides practical guidance on how to use it. Inevitably, some issues are dealt with in greater depth than others. Hut it remains. in this reviewer’s opinion. the clearest and most coherent exposition of the naturalistic approach yet published. My postgraduate education students have found the theoretical discussions readily comprehen- sible and the practical discu\\ion of direct relevance to their own evaluation projects. My main criticism would be that many thcorctical issues are somewhat over-simplified.

Pat-t One is historical anJ conceptual. An

introductory chapter briefly discusses several

cvaltiation models. Then the second. based Iargcly on the work of Stake. develops and defines a rc~.r/‘“‘r.\ilY~ approach to evaluation. which is organised around the concerns and issue5 of stakeholdinz

audiences and evolves during the course of an evaluation instead of being dependent on some I,,-c-o~tli/rtr/c research design. A CO~ICC~~I is dcfincd as “any matter of interest or importance to one 01. more party.” An ic.trrc~ is “any statcmcnt. proposition OI- focus that allows for the presentation of different points of view; any proposition about which

reasonable per-sons may disagree: or any point of contention.” (pp34-5 ). "'The audiences for an

evaluation arc those persons entitled by virtue of holding a stake to propose concerns and issues and to rccei\c a report rchpomivc to them.” (p?7 ). The

Page 5: Effective evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches: Egon G Guba and Yvonna S Lincoln, Jossey — Bass, 1981

third chapter develops Striven’s distinctions between formative and summative evaluation and between the determination of merit (intrinsic value) and the determination of worth (extrinsic value). The information-gathering process can then be summarised as follows:

“First, audiences who have stakes in concerns or issues must be identified, a task not always easy since some audiences are unaware of the stake they hold or, if they are aware, may prefer to keep a low profile, as in the case of some minority groups. An extensive commit- ment of time and resources is then required to work with these audiences in clarifying their concerns and issues. Next, several classes of standards must be identified to make the several judgments necessary to determine merit and worth. Finally, both descriptive and judgmental data must be elicited-data that are responsive to the issues and concerns that have been raised and that are interpretable within the sets of identified standards.” (P37).

Having introduced their concept of evaluation, Cuba and Lincoln then proceed to provide it with an epistemological base. Chapter 4 contrasts the assumptions of the ‘scientific’ and ‘naturalistic’ paradigms about reality, about the inquirer-subject relationship and about the nature of truth statements: then goes on to consider their distinctive postures towards quality, theory, causality, knowledge, purpose and methodology. The treatment is wide- ranging but rather shallow in places, with some inconsistency between the medium-defining a large number of categories and briefly contrasting the paradigms for each in turn - and the message - that the naturalistic paradigm handles complexity by not dividing it up into lots of simple categories.

However. Chapter 5. on ‘Naturalistic Solutions to Methodological ProbIems’ is much more thoroughly worked. Four major criteria of rigor are put forward as relevant to both paradigms: truth value, applicability, consistency and neutrality. The terms naming these concerns in the scientific paradigm are well-established; namely internal validity, external validity, reliability and objectivity. But these are inappropriate for the naturalistic paradigm, for which Guba and Lincoln propose the following alternatives: “credibility for truth value, fittirtg~ss for applicability, amknbility for con- sistency, and c~tJ~~~?3ffbi~j~y for neutrality”. (~104).

Part Three opens with an interesting chapter on “The evaluator as instrument” in which the role of the inquirer as data collector, analyst and interpreter is critically examined. The chapter is important because it establishes a link between epistemology and methodology which is lacking in the other two books. The qualities of a good interviewer and good fieldworker are discussed; and the concluding paragraph provides an excellent summary of the author’s position:

“The shortcomings of human beings as

instruments are more than compensated for by the quality and richness of the data they can gather. Isomorphism, a one-to-one correspon- dence with reality as it is lived by those in the research site, is achieved only through thick description -generous, fertile, abundant, and above all accurate portrayal of the events, persons, and contexts that make up the whole of the inquiry site. Paper-~ln~i-pencjl tests, test scores, and technical reports all have their place. but they cannot enter into the real lives of those that they so anonymously describe. Only human beings, depicting slices of their own lives in their own language, terms, and visions, can recreate reality.” (ppl.51-2).

This methodological introduction is followed by two long practical chapters: 7, Interviewing, Observation and Nonverbal Cue Interpretation; and 8, Using Documents, Records and Unobtrusive Measures. These are full of useful advice, especially on interviewing, though a tendency to overcate- gorise occasionally mars the discussion. Like many advocates of the naturalistic paradigm, Guba and Lincoln draw heavily on the methods and practices of social anthropologists without pausing to consider the different circumstances in which most of them were originally developed. It is unusual, and most will argue unethical, for an evaluator to enter a field context without explicit negotiation of purpose; and evaluators rarely spend anything like as much time in a field context as an anthropologist. The methodological consequences of these differences have yet to be properly explored; and those looking fat links with similar methodological issues in the area of classroom research -an important area for most evaluators working in education - will be dis- appointed. Here most of ail one finds the authors seriously handicapped by their ignorance of European work, a failing unfortunately shared with Cronbach and Rossi.

The last part of the book offers practical advice on organising, implementing and reporting evaluations. The content is we11 summarised by the following list of section headings: negotiating the evaluation contract, organising the evaluation team, getting established at the evaluation site, taking account of human and political factors; identifying stakeholders, making contact with informants, eliciting concerns and issues, inferring value frame- works, testing the rigor of the identification process; kinds of information required, sources of information; forms of reports, the case study, making judgements and recommendations. rnevitably, perhaps, this last part of the book is more patchy: but it still contains many useful points. Since it is dealing with matters that vary so much from one situation to another, the lesser degree of coherence may well be a strength rather than a weakness.

Taken together, these three books have a great deal to offer the reader who wishes to be aware of recent developments in evaluation; and Guba and Lincoln also provide practical help for evaluators. Rossi provides an exceedingly readable account of

295

Page 6: Effective evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches: Egon G Guba and Yvonna S Lincoln, Jossey — Bass, 1981

the still dominant ‘scientific paradigm’, which clearly presents its assumptions. Cronbach provides an alternative conception of the purpose and political role of evaluation; but confines his methodological discussion to research design. Gttba examines the ass~tmptions of the data-gathering process itself and provides a clear exposition of an alternative ‘naturalistic paradigm’.

However, there remains much more thinking to IX done. Cronbach, in particular, lacks epistemo- logical clarity. On the one hand. he couches his discussion of research design in the language of the experimenter, while, on the other, he discusses communications and policy making in the naturalistic mode. Guba lacks political clarity and leaves the political significance of his approach unexplored. Rocsi will need to defend his position against the dual onslaught. Whether he chooses to do so ncademicalty through CounterargL~ment or politically through maintaining the support of the social science eslablishment will be interesting to note.

Michael Eraut

CHURCH AND EDUCATION IN TANZANIA (Editor: Alan J. Gottneid. E. African Publishing House ) .

This book is one of six field-studies designed to evaluate particular types of educational, health and social action work in the context of one country or region in the Third World.

The Institute of Soc.ial Studies at the Hague and the Internat.ional Federation of Social and Socio- Religious Research [ISSlFERES) with a grant from the Ford Foundation have been jointly responsible for this study of the Church Schools system in Tanzania. But, of course. the s’tudy covers the period before Tanzanian independence in 1961, when the country was a colony of, first, Germn.ny and then Rritain fin fact it was not a British colony but a ‘man~i~lted territory’ under the League of Nations).

It begins with a historical survey: continues with a study of the attitudes of Tanzanian Africans to the Church’s involvement in secular education some six years after independence and ends with an attempt at interpreting the very detailed statistical data provided by the survey itself.

Perhaps this brief introduction is sufficient to indicate that we have in this book a massive amount of research covering a country and a period of vital significance to those who are concerned with education in the Third World. But it is also of great vntue to those who recognise that the Christian Church over the past century has made an enormous contribtitio!l in the field of education. and who, nevertheless, have little concrete evidence upon which to evaluate that contribution. Finally. it throws light upon many of the complex issues of Church- State relationships both before and after de- colonisation and independence. So. in all these ways. it is an invaluable piece of research. But, it has to be said. it is not a book intended for the general

reader. In a sense it is not a book which can he read with any pleasure from :I literary standpoint. Its large section of statistics - like all such cvidencc -- is very hard work indeed, though without it J suppose the study could have little value.

“The entrenched gradualism of ail thinking about Africa in the 1920’s must be noted. The progress of African peoples was something which. it was believed, could only come about very slowly A period of centuries was in Sir Donald Cameron’s mind when in 1Y26 he instituted the system of ‘indirect rule’ as a means of preparing the African people to share in their own government. When the major conference of government and missions met in 1925 to discuss educational policy it was specifically thinking in terms of what the country would become ‘several generations hence’.” (P. 3-l).

Only forty years later, however. “. in independent Tanzania we see a certain philosophy of education evolving. This has found its most concrete expression so far in President Nyerere’s document ‘Education for Self-Reliance’.” (P. 203 ).

‘J-he t-Gleof the Churches in those interveninp years was at the same time vital. complex and almost infinitely varied according to geographical, historical. cited religious differences.

1 was fortunate enough to be working in Tanzania (as Bishop of Masasi in the Southern Region. bordering on Mocambique) between 1960 and 1968. I therefore saw the transition from colonial rule to independence at grass-roots level. The Anglican Church at that time had a Secondary School. a Teachers Training College for girts and around one hundred Primary schools scattered over an arca the size of England. In the early 1960’s. although the Church was manager of its schools. the subsidy from government for teachers’ salaries was supposedly 100%. For all capital expenditure and building projects the Church was responsible. From our point of view there were three major problems. Fit-it, the financial problem caused when - as normaily happened - the local authority failed to collect its taxes and therefore failed to send its contribution (about 40% of the total if my memory serves me right) to the teachers’ salary cheque at the end of each month. As manager, the bishop was responsible in taw for paying the teachers in his Church schools. Result: an enormous ovcrdl-aft and heavy hank charges which were impossible for a poor diocese to sustain.

Secondly, the pastoral problem which constantly arose when the Church as manager of schools had to take disciplinary action against one of its teachers for moral turpitude. Result: an erosion of trtlst in the clergy and particularly the bishop as a caring and compassionate father-in-God.

Thirdly. the unending battle in :I poor countr) and a poor church, to raise standards: to secure better and more permanent buildings: to recruit teachers who would work under deteriorating roofs and in class-rooms with inadequate or non-existent materials. Result: a lack of confidence in the