Volume 33 Number 1
October 1993

Jim MacAvoy, Stephanie Orr, and Craig Sidles

Court cases beginning in the 1970s and continuing through the present time have consistently held that minority school pupils should not be placed in special education programs on the basis of test results utilizing instruments normed primarily on majority Anglo populations (Reschley, Kicklighter, & McKee, 1988). The concerns expressed by the courts as well as by many researchers center around the cultural loading on the tests, the language and linguistic bias, and the test sophistication/readiness of the minority child (Pfeiffer, 1979; Sattler, 1988).

In response to these concerns, the present investigators and several others have reviewed reportedly culture-fair tests in terms of their applicability with American Indians (Cattell, 1979; Martin & Sidles, 1986; Sidles & MacAvoy, 1987). One instrument having considerable supporting evidence for use with American Indian students was found to be the Raven Progressive Matrices tests (Gearhait 1984; Sidles, MacAvoy, Bernston, & Kuhn, 1987). Strong support for its culture-fair status can be found in the analysis of related research by Raven and Summers (1986). They report that the Raven Matrices minimize error variance contributed by ethnicity and socioeconomic status (SES) factors. While the combined influence of ethnicity and SES was found to account for 15% of test variance, 48% of Raven test performance was directly attributed to age and developmental factors.

Additional reasons for its favorable status include the fact that the Raven Matrices are untimed and use a nonverbal format. Although no comparison is intended, Dash and Rath (1986) found that assessment measures incorporating "testing the limits" procedures resulted in improved test performance with East Indian children. Studies of the intellectual ability of American Indian children uniformly indicate that they obtain higher scores on performance (nonverbal) tests than on verbal ability measures (McShane & Plas, 1984). The Raven's nonverbal format appears to have applications, for gifted assessment as well. It has been found to play a useful screening role in identifying gifted children in general populations (Matthews, 1988) and talented math students in other culturally diverse populations (Robinson, 1990).

This study represents a cooperative project between the Psychological Services Department of the Flagstaff Unified School District #1 and the Center for Excellence in Education at Northern Arizona University. Its purpose is to examine the Raven performance of elementary and secondary Navajo students attending the Flagstaff schools. By developing normative data, the Raven Coloured and Standard Progressive Matrices can be better utilized in a culture-fair manner as a cognitive reasoning measure during special education and gifted assessments. Additional goals of the study were to examine any differences in the performance of Navajo males and females as well as students living "on" and "off' the reservation. A final goal addressed the relationship between students' scores on the Raven Matrices and the tests of academic achievement selected for this study with respect to age and school location variables.

A total of 908 elementary and secondary Navajo students served as the student population for this study. Students selected attended grades 2 through grades 12 and ranged in age from 7 to 17 years. Care was taken to select only full-blood Navajo students to reduce the chance of extraneous variance that could arise from exposure to a background of multiple cultural experiences or language experiences which may have included more than the Navajo and/or English languages.

By virtue of its close proximity to the Navajo reservation, the Flagstaff Public Schools are faced with a unique challenge in providing educational services to large Navajo student populations "on" and "off" the reservation. For the purpose of this study, "off" reservation students are defined as those Navajo students attending schools which are located off the reservation. There are 12 elementary (K-6), 2 middle schools (7-8), and 3 high schools (9-12) in the Flagstaff area. Navajo students attending local Flagstaff Schools are integrated with Anglo and other minority students and represent 14% of the population. "On" reservation students are defined as those Navajo youngsters attending Leupp Public School, a Flagstaff District school, which is located in a small town on the reservation 40 miles east of Flagstaff. This school serves kindergarten through grade 8 and contains a student population that is virtually all American Indian; 88% of which are Navajo. Public school students in Leupp typically commute to Flagstaff, a trip of 45 minutes, to complete their secondary education. Schools "on" and "off" the reservation base instruction on a board approved district-wide curriculum that is consistent for all schools.

The Raven Coloured Progressive Matrices (RCPM, the Raven Standard Progressive Matrices (RSPM), the Iowa Tests of Basic Skills (ITBS), and the Tests of Achievement and Proficiency (TAP) were the assessment instruments used in the study. Both the Coloured and Standard Progressive Matrices, developed in 1947 and 1938, respectively, have been described as nonverbal tests of reasoning ability (Sattler, 1988) and as providing an index of intellectual capacity while minimizing the effects of cultural environment (Raven, Court, & Raven, 1983). The subject is required to select a missing figure through analogy reasoning to complete a matrix-like arrangement. The Coloured Matrices test, which was designed to assess the reasoning ability of younger children, consists of 36 problems with matrices detailed in a colored background (Raven, Court, & Raven, 1986). The Standard Matrices test was intended for use with children and adults and can be administered on an individual or group basis (Raven, Court, & Raven, 1983). It consists of 60 items divided into 5 sets of 12 questions. Matrices are presented in a black and white pictorial context. The test uses an untimed format and the items in each set progress in terms of their level of difficulty. Whereas the focus of earlier test items is to find a missing piece to complete a pattern, items contained in later sets measure one's ability to form comparisons and reason by analogy (Raven, Court, & Raven, 1983). Raven (1986) also reports that problem solving can take the form of a verbal-analytic approach or one employing gestalt visual-perceptual discovery. Test-retest reliabilities range from .71 to .92 and concurrent validity estimates from .55 to .86 for these instruments (Sattler, 1988).

The Iowa Tests of Basic Skills (ITBS) is a standardized, norm-referenced test designed to reflect a continuous developmental achievement sequence from kindergarten through grade 8. It was administered in grades 2 through 8 to students participating in this study. The ITBS batteries provide for comprehensive measurement of achievement across the following fundamental skill areas: 1) listening, 2) word analysis, 3) vocabulary, 4) reading, 5) language, 6) work study, and 7) mathematics. According to the ITBS Manual for School Administrators (Hieronymus & Hoover, 1990), some specific purposes of the test include determining the developmental achievement level of the student and diagnosing specific qualitative strengths and weaknesses in the pupil's educational development.

The complete ITBS battery was given at grade 2 for both subject populations in this study. At this level, the tests are essentially "power" tests. They are untimed and there is no emphasis on speed. Nine of the thirteen subtests covering the seven fundamental skill areas are orally administered in order to lessen the importance of reading mechanics. The complete battery was also administered to grades 3 through 8 in separate level booklets. Each level is used to test a designated grade level. Eleven timed subtests measuring the seven fundamental skill areas are administered. Internal consistency reliabilities for the ITBS typically range from .73 to .97 (Hieronymus & Hoover, 1990).

The Tests of Achievement and Proficiency (TAP) were administered to students attending "off" reservation schools in grades 9 through 12. The tests are given in separate level booklets by grade level and the basic battery administered includes achievement subtests in the areas of reading comprehension, mathematics written expression, and using sources of information. Working time for each subtest is approximately 40 minutes. Internal consistency reliabilities for the TAP range from .87 to .95 (Scannell, 1990).

The assessment and data collecting phase took place during a 2 year period. In the spring of 1989, the population of elementary Navajo students (grades 2 through 6) was administered the Raven Coloured Progressive Matrices (RCPM). These scores were, in turn, conrrelated with the results of the Iowa Tests of Basic Skills (MS) also administered that spring. The RSPM, the ITBS, and the TAP were given to secondary level (grades 7 through 12) Navajo students in the spring of 1990. The total student body of Navajo students was targeted for assessment. The subjects were tested with the RCPM and RSPM in small groups with a maximum of 25 students per group. Students were grouped according to grade level for each administration. An untimed testing format was utilized and standard test directions were given based on instructions contained in the Raven test manuals. Group administrations were also used with the ITBS and the TAP. These tests, which incorporate a timed format, were administered in their grade level classrooms.

Descriptive analyses of the RCPM and RSPM raw scores were conducted to obtain normative data. Specifically, means, standard deviations, and percentile ranks were computed for students grouped according to grade level. For the elementarymentary students, normative data was also derived on the basis of school location (e.g., "on"/"off" reservation). This information was converted into tables for ease of interpretation.

Correlation analyses were also conducted to examine the relationship of Raven, the ITBS, and the TAP test data. To accommodate this process, Raven raw scores were converted to T-scores (x = 50, SD = 10) and then correlated with NCE scores based on the same scale from the ITBS and the TAP. In addition to total group analyses, correlations were derived for "on" and "off 'reservation school groups at the elementary and secondary grade levels.

Finally, differences in Raven performance between male and female groups, and "on" and "off" 'reservation subgroups were analyzed by means of T-test analysis. An alpha level of significance was set at p < .01 for all statistical analyses conducted in this study.

A primary objective of this study was to develop Raven norms for Navajo students attending grades 2 through 12. Table 1 contains RCPM percentile rank norms for the total sample of Navajo students attending grades 2 through 6. Listed at the bottom of the table are the means and standard deviations for each grade level.

Table 1
Percentile Ranks Corresponding to RCPM Raw Scores by Elementary Grade
Level for Flagstaff Public Schools (N 369)

Table 2
Percentile Ranks Corresponding to RCPM Raw Scores by Elementary Grade
Level for Flagstaff Public Schools (N=138)
("On" Reservation Norms)

Table 2 lists the RCPM grade norms for students attending Leupp Public School. This data represents the elementary "on" reservation Navajo sample in the study. Also listed are mean and standard deviation scores for each group.

Grade norms for the RSPM were also developed for secondary level students attending grades 7 through 12. These scores are displayed in Table 3. "On" reservation norms were not available for the secondary level as services at Leupp Public School are limited to kindergarten through grade 8.

A second area of investigation addressed the relationship between Raven and ITBS/TAP performance. Separate correlations were computed for the RCPM (elementary level) and the RSPM (secondary level). As noted earlier, ITBS scores were available for grades 2 through 8 and seven individual skill areas were preselected for analysis in this study. Group achievement scores for the TAP were available for grades 9 and higher and three skill areas were preselected for analysis.

Table 4 lists correlations between the RCPM and the ITBS for grades 2 through 6.

In Table 5, the correlational results between the RSPM and the ITBS are displayed for grades 7 and 8.

The correlational results between the RSPM and the TAP for grades 9 and 10 are profiled in Table 6.

Table 3
Percentile Ranks Corresponding to RSPM Raw Scores by Secondary Grade
Level for Flagstaff Public Schools (N 374)

Table 4
Summary of Correlations Between RCPM and ITBS for Grades 2-6

Table 5
Correlations Between RSPM and ITBS for Grades 7-8

Table 6
Correlations Between RSPM and TAP for Grades 9-10

Table 7 contains a summary of RCPM/RSPM and ITBS correlations for "on" and "off' reservation schools at the elementary and junior high (grades 7 and 8) levels.

Table 7

Correlations Between RCPM/RSPM and ITBS for "On" and "Off"
Reservation Elementary (2-6) and Junior High Levels (7-8)

As noted earlier, the final phase of the study addressed differences in RCPM and RSPM performances in relation to gender and school location factors. T-test analysis revealed a significant difference in male/female performance at the elementary but not at the secondary level. In the former case, the RCPM scores for male students were noted to significantly exceed those of their female counterparts. These results are displayed in Table 8.

T-test comparisons between "on" and "off" reservation schools revealed a significant difference in the test performance at the elementary and junior high levels. Specifically, the RCPM and RSPM performance of students attending grades 2 through 8 at Leupp Public School on the reservation was noted to exceed those of Navajo students attending elementary and junior high schools located in the City of Flagstaff. These results are displayed in Table 9.

Table 8
Results of T-Test Analysis Comparing RCPM/RSPM Performance for Males vs.
Females at Elementary (Grades 2-6) and Secondary (Grades 7-12) Levels

Table 9
Results of T-Test Analysis Comparing RCPM/RSPM Performance for
"On" and "Off" Reservation Schools at the Elementary (Grades 2-6)
and Junior High (Grades 7-8) Levels

The gains noted in mean scores across grade levels support the hypothesis that the RCPM and RSPM are developmentally sensitive measures. In line with previously reported research, these gains in raw scores are considered to be a primary function of increasing age and/or educational experience (Raven & Summers, 1986). As noted earlier, part of the rationale for developing local norms was to address the possible use of the RCPM and the RSPM as nonverbal measures of giftedness. The present study suggests that the RCPM does not contain the necessary item difficulty to serve such a purpose for grades 3 and higher. When applying a gifted criteria (two standard deviations above the mean), it becomes evident that the RCPM does not provide a sufficient score ceiling for this population. Similar limitations appear to exist for the RSPM when applied to secondary level students. Examination of the norms suggests that the RSPM has a sufficient ceiling up through grades 9 but not beyond. These findings would lend support to the use of the RCPM through grades 2 and the RSPM for grades 3 through 9. Additional study appears to be needed to address the possible use of the Advanced version of the Raven Matrices for Navajo students attending grades 10 and up who are being considered for gifted eligibility.

Analysis of correlational data suggests low to moderate positive correlations between the RCPM and the ITBS. Correlations between the RSPM and the ITBS/TAP with secondary level Navajo students were noted to be generally higher. This trend may be a reflection of test characteristics or student factors. For instance, while the RCPM emphasizes visual pattern completion, the higher degree of sequential/analogous reasoning incorporated into the RSPM may more closely approximate the kinds of cognitive demands inherent in the academic tests of the ITBS and TAP. Secondly, as Navajo students become older and more oriented to verbal/analytical reasoning through their educational experiences, the kinds of cognitive strategies they bring to both the Raven Matrices and achievement tests at that higher level may result in higher concordance among their scores in those measures.

At the elementary level, males were found to score significantly higher than females on the RCPM. In contrast, secondary students tested with the RSPM earned mean estimates that were relatively commensurate for both gender groups. Similarly, Sidles and MacAvoy (1987) found no significant difference in RSPM scores for older male and female students. These differences in gender findings may reflect the interaction of factors specific to student and test characteristics. Specifically, younger male Navajo students may be more unilateral in their mental processing relying more heavily on visual strategies during their early elementary years. Following this line of reasoning, the RCPM, with its emphasis on visual completion, may present an easier task for males than for females employing a bilateral mental processing approach. As a result of undergoing a shaping process through their educational experience, older male students may adopt a more bilateral (e.g., analytical/holistic) processing approach similar to their female peers by the time they enter their secondary grades. This change could account for the equivalent estimates obtained for both gender groups on the RSPM.

A significant difference in Raven performance was also observed between "on" and "off"' reservation students at both the elementary and secondary levels. A superior performance by "on" reservation students is consistent with other studies of American Indian intelligence which reflect a higher performance in nonverbal measures of cognitive reasoning (McShane & Plas, 1984). The experiences that Navajo students encounter when they reside and are educated in a majority Anglo environment may lead to a greater emphasis on verbal/sequential thinking as opposed to a reliance on visual reasoning strategies. This could account for the significantly lower Raven performance recorded for "off" reservation Navajo students in this study. In addition, differences in Raven performance may be partly attributed to differences that are likely to exist in teaching styles employed by teachers in "on" and "off" reservation school settings. There is a higher representation of Navajo teachers at Leupp Public School than is the case for local Flagstaff schools. Differences in teaching styles may be a viable topic for additional research in this area.

In terms of educational implications, it is apparent that Navajo students possess analytical thinking skills but may best express these strengths when a visual model is used. In order to enhance their academic performance to a level commensurate with their reasoning capabilities, it is seen as important that linkage be established between visual and verbal models of instruction to promote that bilateral reasoning capacity. To supplement their strong visual reasoning abilities, it is also important to introduce Navajo children to critical thinking and sequential/analytical problem solving at an early age. Finally, it is important to de-emphasize the element of timed completion with Navajo children during work assignments or test administration to encourage their best performance.

Jim MacAvoy, Ed.D., is a Psychologist with the Flagstaff, Arizona, Unified School District. He received his doctoral degree in Educational Psychology from Northern Arizona University in 1986. His previous research efforts have examined issues related to the assessment of intelligence and auditory memory with American Indian students. His professional practice includes a specialization in the assessment of American Indian children as well as developmentally disabled populations.