56 Journal of Public Health Dentistry

REPORT OF COMMITTEE ON RESEARCH

EVALUATING THE EFFECTIVENESS OF PREVENTIVE DENTAL PROGRAMS* By John D. Suomi, DDS., Chairman,**

At the meeting of the American Association of Public Health Dentists last year in San Francisco, the report of the Research Committee concluded with a plea for critical, clinical evaluation of programs of preventive dentistry and health education. Some guidelines for such evaluation now appear in order.

Evaluation of the effectiveness of community or school health programs is probably one of the most neglected activities in the practice of public health. Many programs are initiated and continued for years without a critical analysis of effectiveness. The readiness of dental public health and school officials to accept and put into operation avariety of preventive dental programs, for which very little solid evidence of effectiveness exists, has been surprising It appears appropriate, hence, to review the criteria for investigators to consider when conducting their evaluations of the effectiveness of preventive programs.

The first step of the investigator, in planning an evaluation, should be to secure the assistance of a statistician, preferably one who has had some experience with dental data. Often, this step appears not to have been followed by the investigator; only after the data have been collected is a statistician asked to make some sense out of the accumulated data.

An important second step for the investigator is to prepare a written protocol which presents the background for the study, review of the pertinent literature, objectives of the study, the design of the study, methods employed, the timetable, the plan for analysis, the dummy tables that will be developed, the estimated cost, and the references which will be used for documentation. The preparation of a protocol, makes the investigator consider carefully the specific objectives, methods to follow, and the plan for analyzing the data. This preliminary procedure will save time and money during the conduct of one’s evaluation and enable the investigator to delete steps clearly nonessential to the aims of his study. When a carefully prepared protocol has been followed druing an investigation, another investigator should be able to perform the same study precisely.

A common mistake of investigators is an attempt to answer too many questions with one study. The investigator, hence, should state his main objective and not permit his study to become cluttered with interesting side-objectives which hinder achievement of the major objective. Some studies fail to answer one question satisfactorily, after considerable effort has been expended.

Questionnaires, when used, are particularly susceptible to undue length. If they are to be analyzed at all, thought has to be utilized in preparation for the eventual analysis when the study is being planned. The questionnaire is an important tool of the behavioral scientist when effectively used. To evaluate the effectiveness of a preventive dental progam, however, there is no good substitute for an assessment of oral conditions directly.

A large number of children in a community often will be participating in a preventive dental program. An important question to settle then is how many need to be examined in order to obtain a valid estimate of the effectiveness of the program. For this task, a statistician can be most helpful. The number of persons to be studied cannot be determined precisely, however, unless the variation between individuals and the magni-

*Presented in Houston, Texas, at the 36th Annual Meeting of the American Association of Public

**Chief, Community Appkations Section, Preventive Practices Branch, Division of Dental Health, Health Dentists, October 27, 1973.

U.S. Public Health Service, RockviUe, Md. 20852.

Vol. 34, No. 1-Winter Issue 57

tude of the expected change can be estimated from a pilot-study or similar studies reported in the literature.'

As a rule, when large differences between test- and control-groups are expected, the samples can be smaller than when small differences are expected. When calculating the size of the sample, also, one must take into account the losses that will occur even during a short-term study. For example, losses from 30 to 50 percent have been observed in a three-year study. It is necessary, therefore, to add to the estimated sample the number of individuals who may be lost during the period of study.

Another important consideration in studies or their evaluations concerns the construction of study-groups. The objective of this procedure is to obtain test- and control-groups that are as alike as possible in all factors that may affect the outcomes of the study.' All subjects must be derived from the same population and each participant should have an equal chance of falling into the test- or control-group.

Random assignment of subjects offers the best hope for obtaining comparable groups when studying known and unknown factors which may affect the outcome of the study. Occasionally, it may be desirable to stratify according to two or three relevant factors. When stratification is employed, however, it still is necessary to provide for the random assignment of subjects to test- and control-groups within the subgroups formed.

A group of controls, not receiving the treatment in the study, is an essential ingredient of any study designed. Without controls, it is impossible to determine how much of a beneficial result can be attributed to the treatment or to other factors such as bias of the examiner because of his knowledge of the design of the study. Many times a beneficial result will be detected in the controls as well as in the group treated.2

The selection of the proper dental index to evaluate the status of oral conditions in the population studied is critical in importance. Selection not always is easy when many indexes are available to the investigator. If possible, an existing index or indexes should be selected. For most dental studies, satisfactory tested indexes are available. A recent conference at Temple University reviewed available indexes for periodontal disease and oral hygiene suitable for use in clinical tria1s.j The proceedings from this conference, when available, should prove helpful.

Indexes particularly useful in studies of oral hygiene and periodontal disease include the Oral Hygiene Index - Simplified of Greene and Vermillion? the Periodontal Disease Index of Ramfjord,' modified to give separate measures of gingivitis and loss of attachment, the Patient Hygiene Performance Index of Podshadley and Haley,6 the DHC Index of gingival inflammation,' the probe-method for assessing calculus, developed by Volpe, Manhold, and Hazen,8 the Calculus Surface Index of Ennever, Sturzenberger, and Radike,g and the indexes for plaque and gingivitis developed by Loe and his co- w0rkers.l o, '

During a clinical assessment, everything possible should be employed to ease the task of the examiner. The form for recording should be laid out in such a way that it conforms to the sequence of the teeth and surfaces being examined so that needless movement by the examiner and the shifting from facial to lingual surfaces while examining can be minimal. Forms frequently are designed for convenience in processing data but incon- venient for the examiner.

No more than four or five hours per day should be devoted to examining. Longer periods tire the examiner and results become of questionable value.

The number of examiners should be kept to a minimum. A separate examiner to assess each variable is preferable, however. Organized this way, each examiner is influenced to the least extent possible by the other variables being recorded. During examinations, subjects in test- and control-groups should be ihtermingled and the examiners should not know to which group any subject belongs.

Examiners should be trained and given an opportunity to engage in an examination

58 Journal of Public Health Dentistry

of nonstudy participants in advance of the real examinations. These practicing exami- nations, however, should be completed for subjects similar to those who will be examined in the study. The same examiners should perform both the base-line and follow-up examinations.

Whenever possible, it is important that all examinations take place in the same quarters, that lights and instruments remain the same, and that the same atmosphere be maintained. Ideally, each examiner should have his own mirrors, explorers, and other needed instruments. Preferably, these items should not be exchanged among examiners.

In clinical studies of attempts to reduce caries-experience, a period of three years often is recommended to demonstrate a positive effect.' In studies of the effectiveness of programs to improve oral hygiene and gingival health, much shorter periods may be used. Two to three months may be sufficient to determine the initial impact of such a program. In determining the effectiveness of mouth-rinses or different methods of toothbrushing for removing plaque, even shorter studies may suffice.

In addition to base-line and final examinations, it is desirable to have a follow-up examination at some point in time after the dental program has ended. In this way, the investigator can determine whether a carry-over effect has been obtained beyond the end of the original project.

In some studies, follow-up examinations of this type have yielded disappointing results. Koch and Lindhe,' following a study in Sweden, determined that improved dental health behavior in children existed only during the supervised periods of instruction. In other studies reported by Suomi e t and Clark et al.14 follow-up results were more positive.

When a difference between test- and control-groups is judged to be of no clinical or practical importance during analysis of data, a statistical test for significance generally will not contribute much to the interpretation. A small nonclinical difference between study-groups may be statistically significant when a study-population, much too large, has been used. If, however, a difference is thought to be clinically meaningful, it usudly is important to establish whether the difference is of statisticd significance. At the same time, a statistically significant result does not indicate that the difference between test- and control-groups is of any clinical or practical importance.

Many times, the statistical methods used for analyzing the data are not applicable because the conditions for their use have not been met. The mete presence of an elaborate statistical evaluation in reported research quarantees, in no way, that the results are any more meaningful or more definitive. If no statistical tests are used, however, no guarantee exists that they should not have been used.

As a final warning to the present report on evaluation of effectiveness, it should be noted that not all points of importance for evaluating preventive dental programs have been discussed. This report only has attempted to cover briefly some of the major factors thought to be of value by the Committee.

Bibliography

1. Backer-Dirks, O., et al. Principal requirements for controlled clinical trials. Intetnat. Dent. J.,

2. Fischman, S.L., and Picozzi, A. Clinical evaluation of anti-calculus agents; methodology and

3. Conference on Clinical Periodontal Trials, Temple University, Philadelphia, Pa., April 1-3, 1973.

4. Greene, J.C., and Vermillion, J.R. The Simplified Oral Hygiene Index. Am. Dent. A. J., 68L7-13,

17:93-103, Mar. 1967.

placebo effect. Pharmacol. and Therap. in Dent., 1:16-23, Oct. 1970.

(Proceedings to be published in the Journal of Periodontal Research)

Tan. 1964. 5. Ramfjord, S.P. Indices for prevalence and incidence of periodontal disease. J. Periodont.,

30:51-9, Jan. 1959.

Vol. 34, No. 1-Winter Issue 59

6.

7.

8.

9.

10.

11.

12.

13.

14.

Podshadley, A.G., and Haley, J.V. A method for evaluating oral hygiene performance. Pub. Health Rep., 83:259-64, Mar. 1968. Suomi, J.D., et al. The effect of controlled oral hygiene procedures on the progression of periodontal disease in adults; results after two years. J. Periodont., 40:416-20, July 1969. Volpe, A.R., Manhold, J ?I., and Hazen, S.P. In vivo calculus assessment. Part I. A method and its examiner reproducibility. J. Periodont., 36: 292-304, July-Aug. 1965. Ennever, John, Sturzenberger, P.O., and Radike, A.W. The calculus surface index method for scoring clinical calculus studies. J. Periodont., 32: 54-7, Jan. 1961. Loe, Harald, and Silness, J . Periodontal disease in pregnancy. I. Prevalence and severity. Acta Odont. Scand., 21:533-51, 1963. Silness, J., and Loe, Harald. Periodontal disease in pregnancy. 11. Correlation between oral hygiene and periodontal condition. Acta Odont. Scand., 22: 121-35, 1964. Koch, G., and Lindhe, J. The state of the gingivae and other caries-increment in school children during and after withdrawal of various prophylactive measures. p. 271-81. (In McHugh, W.D., ed. Dental plaque. London, Livingstone, 1970, 298 p.) Suomi, J.D., Leatherwood, E.C., and Chang, J.J. A follow-up study of former participants in a controlled oral hygiene study. J. Periodont. In Press. Clark, C.A., Fintz, J.B., and Elwell, K.R. Eliminating dental plaque in the sixth grade. J. Publ. Health Dent., 33:70-4, Spring 1973.

Two In One?

Dr. Roger Sperry, a t the California Institute of Technology, after studying a 48-year-old veteran of World War 11, concludes that Homo sapiens is the only mammal in Planet World with two independent hemispheres or half-brains linked together by the corpus callosum to gain some unity. The veteran, wounded in the head by fragments of a shell, developed such a severe epilepsy that the corpus had t o be cut, but the surgery cured the fits immediately. Then it was discovered that only the dominant left brain could speak while the right brain remained mute and could perform no tasks which required judgment or interpretation based on language. Once the veteran threatened his wife with his left hand while his right hand tried to come to her rescue and bring the belligerant hand under control. (Maya Pines in the Detroit Free Press for September 16, 1973)

Bottled Chemicals?

Says Melvin Calvin, President-Elect of the American Society for Advancement of Science, science is not a gadget or plaything or curio, and scientists are not mere bottles of chemicals. The concern of the public and its officals for proper information makes the scientist an essential cog in decision-making. (AAAS Bde t in for September, 1973)

Human Bites Staff members f rom the Department of Oral Surgery at Loyola University in

Maywood, Illinois, have prepared a review of the wounds inflicted by human bites. They point out that the multitutde of micro-organisms introduced by such bites makes treatment a serious problem. No instances of infection by tetanus have been reported, although the organisms have been isolated in oral cavities. Syphilis, however, has been transmitted by the bite of an infected person. (Journal of Oral Surgery for Oct. 1973)