Volume 15 Number 1
October 1975

Lena Canyon, Sandy Gibbs and David Churchman

The psychologists, for example, have tended to attempt experimental and quasi-experimental methods without the controls necessary for successful research; further, they generally have relied on published tests without taking into account the inadequacies of such instruments for evaluative purposes (Klein, 1970, 1971). For their part, the anthropologists tended to write about the difficulties of doing cross-cultural research and to use this as a rationale for avoiding conclusions. The remaining experts have judged the special programs against their own experience--a logically inappropriate approach to evaluating innovative schools attempting to meet the needs of a special population.

During the past three years, Tribal American Consulting Corporation [TACC] has been working not only toward development of a special educational program to meet the needs of Native American children in Los Angeles, but also toward the development. of suitable methods for evaluating this program. TACC, an Indian-owned, nonprofit corporation, established a preschool for urban Indian children in 1972 (Long, Canyon and Churchman, 1973). The school was expanded to include a day-care center and moved to improved quarters in 1973.

The Children’s Center is unique in the flexibility that it provides to meet the needs of individual children. The day-care center, which uses a Montessori approach during instructional periods, is available to the children of working mothers or single parents. The remaining children attend school for one-half day either in the morning or in the afternoon and are assigned to one of two classrooms. One of these provides a traditional educational program, emphasizing small group activities under the direction of one certificated teacher and two aides, and is popular with younger or quieter children. The second classroom provides an educational program developed directly from Indian cultures yet seeking traditional educational goals necessary to prepare the children for public school, and generally is preferred by older and more active children (D. Churchman, J. Herman and T. Hall).

California state funding included money for and required evaluation of the program after September 1973. TACC recognized the opportunity to use the funds mandated for this purpose not only to complete the required evaluation, but also to train Native Americans in evaluation skills. Developing these skills would contribute to the increasing independence of Native American educational programs and in addition would address in the most direct way the concerns of the anthropologists for the difficulties of cross-cultural research. Aware of the expertise needed, TACC turned to a member of the senior research staff of the Center for the Study of Evaluation [CSE]. Located at UCLA an easy half-hour drive from the Children’s Center, CSE is the only research center in the country concentrating exclusively on the problems of educational evaluation.

An appropriate evaluation plan was developed and four Native American undergraduates at UCLA were selected to assist in carrying out the plan. James Young is a Sioux from the Standing Rock Reservation and an economics major. Wally Strong, a Yakima and Nez Perce, is a sociology major who is working on a master’s degree in education. Russel Begaye, a graduate political science major now working on a law degree, and Lena Canyon, an economics major, are both Navajo.

Their primary task was to test children individually each month of the school year. Four different tests were in use as part of the evaluation effort. The first of these was the Peabody Picture Vocabulary Test [PPV]. The examiner shows the child four pictures printed on a single page and asks the child to point to the thing named by the examiner. For example, the child may be shown line drawings of a crib, a woman, a kitten and a spoon, and asked to indicate the picture of the crib.

The third test used was a locally developed pre-mathematics instrument [MT] developed because no published test covered the areas of interest. The test is used to measure counting ability, knowledge of mathematical properties of objects such as size, weight and length, and knowledge of time and money. Various aspects of each concept are explored. For example, the counting score is based on the ability of the child to count objects as well as to recite numbers in sequence.

The fourth test was the Cooperative Primary Inventory [CPI]. Three of the four scales were used. The Social Responsiveness score, interpreted as a measure of self-concept, is based on the ability of the child to respond to questions such as "What is your first name?" and "[Can you] show me your shoulder?" The Associative Vocabulary score is based on questions such as "If you want gas for your car, where would you go?" The Concept Activation-Sensory score, interpreted as a measure of motor development, is based on judgments of the quality of the child’s response to directions such as "Copy this [triangle) for me."

The first task of the apprentice Native American evaluators was to learn to administer these tests. Individualized testing is a strange experience to inflict on a child who is just starting school and for the first time finds himself in large groups of children and under the control of strange adults. There was much for the evaluators to learn: how to overcome the child’s fear; the exact degree of friendliness required so that the child feels comfortable yet still takes the testing seriously; to handle testing equipment efficiently without distracting the child while simultaneously keeping his interest and recording his answers; to judge between correct and incorrect answers when, particularly on the CPI, the distinction is not always clear; to work with a variety of children, some unresponsive, some bright, some dull, some hyperactive; to treat each child the same as all others, six hours a day, regardless of how tired or bored they became repeating the same questions a dozen or more times a day.

Through this repetition, they learned that the tests were less than perfect. The PPV contained many items beyond the experience of the children and lacked items that tested English vocabulary related to Native American cultures. The PASS makes the odd assumption that dependency and motivation are independent--that, for example, a highly dependent child will not be motivated to achieve in school to gain parental approval. Further, the test is normed on only 480 children from suburban Chicago, about whom no information is given regarding age, sex, race or socio-economic status (Buros, 1972). It is difficult to know how to interpret the scores of Native American children in southeastern Los Angeles when the "norms" are based on 480 suburban midwesterners.

The CPI includes an "associative vocabulary scale that is primarily a test of information rather than vocabulary." The proportion of children passing each item suggests that it is not very sensitive to changes in student knowledge over short periods of time. There is no sampling plan to justify the norms that are provided and the useful idea of providing norms for various social classes breaks down in practice because there are no criteria for determining the class to which a particular child belongs. Further, the lack of any information on how items were selected to measure the various domains makes the instrument difficult to interpret (Buros, 1972). The locally developed MT can be justified only by face validity and no information as to its validity or reliability is available.

Through direct experience, the evaluators learned to judge tests in terms of their administrative usability, appropriateness to examinees, technical qualities such as validity and reliability and the extent to which they measure the educational goals sought by the program. They learned where to find reviews of tests and they came to know something of the process of selecting tests for a specific situation. They learned how to interpret individual scores and that all tests are not equally sensitive to changes in student learning.

However, they learned that tests probably are more objective, more reliable and more easily interpreted than alternative means of evaluating educational programs. They learned how a stratified random sample is drawn, about means and standard deviations, and how test data is analyzed and interpreted through inferential statistics, Through hard experience they learned the validity of Murphy’s Law, which asserts that "If anything can go wrong, it will!" Absentees, dropouts, class changes and new students insured that data was incomplete and difficult to analyze; scores were recorded incorrectly and supplies ran out unexpectedly or were lost; test equipment was broken; factors not included in the evaluation plan were discovered late in the school year to have been important.

The testing and evaluation program will continue, with improvements based on experience, during the 1974-1975 school year. Among these improvements are (1) replacement of the PASS with the new science scale from the Tests of Basic Experience; (2) inclusion of two additional independent variables in the data analysis; (3) improvement of the sampling plan to provide more stable and accurate data; (4) collection of information needed for development of a test related to the cultural curriculum; and (5) appointment of one of the Indian students as administrator of the testing program.

The last change is perhaps the most important. The responsibilities include selecting personnel to replace the two who have graduated, training, ordering and checking of tests and supplies, maintaining up-to-date rosters of students, and scheduling of work each month. In this way, his apprenticeship will teach him many of the skills related to administering an evaluation program. Subsequent training will involve learning more about the analysis and interpretation of data as well as the design of evaluation studies and the reporting of results. In this way, TACC is not only developing a model evaluation program for Native American educational programs, but is providing sophisticated and realistic on-the-job training that will increase the ability of Native Americans to evaluate their own schools.