Volume 37 Number 1
Fall 1997

Given the rapidly changing demographic face of the United States (i.e., the increased number of people who are representative of non-White racial origins), the recent attention given by researchers and practitioners to the importance of analyzing validity issues embedded in the assessment process is timely. One such group of people, consisting of American Indians/ Alaska Natives, is indigenous to the Americas. American Indian/Alaska Native (AI/AN) children are often raised within family systems that have language, value sets, traditions, spiritual convictions, and child-rearing practices that differ from the majority society. As a result, they need assessment instruments and procedures that reflect their heritage (Benner, 1992; Tafoya, 1989). The lack of empirical information on early childhood assessment practices for AI/AN children and their families is a major impediment to the provision of effective services for this population (Division for Culturally and Linguistically Diverse Exceptional Learners, 1995; Indian Nations at Risk Task Force, 1991). Experts in the field have deemed the "best practices" in early intervention to be those that are amenable to the diverse interests and needs of family systems. Given that assessment is the foundation upon which Individualized Family Service Plans (IFSPs) or Individualized Education Programs (IEPs) are based, it is essential that the instruments and procedures used reveal pertinent information that reflects both appropriate outcomes for this age group and relevance for the cultural setting.

Assessment practices (standardized norm-referenced tests) traditionally used with school-aged children are inappropriate when one considers their use with very young culturally diverse exceptional populations (Bagnato & Neisworth, 1991; Benner, 1992; Bracken, 1989; Neisworth & Bagnato, 1992). The use of standardized norm-referenced tests with populations other than those on whom the test was normed is inappropriate. In addition, standardized testing of American Indian students has been shown to underestimate their potential (Brescia & Fortune, 1989).

Assessment practices in early intervention are varied, and often present a challenging maze of choices for both professionals and families as they strive to design quality individualized programs. The primary goal of assessment is to obtain information that leads to the development of program plans and evaluations that not only identify curriculum goals, but also match the needs of families (Bailey & Wolery, 1992; Bagnato & Neisworth, 1991; Benner, 1992). Therefore, evaluating the validity of assessment practices that are utilized in early intervention programs serving AI/AN populations may serve to assist professionals and parents/caregivers through the assessment maze and facilitate reaching the goal of quality service delivery. The validity of assessment practices are especially important when one considers: (a) the multidimensionality of infant and toddler assessment (e.g., an assessment approach that incorporates an interdisciplinary or transdisciplinary team, emphasizes family involvement and collaborative decision-making, uses convergent information, formal and informal methodologies, etc.) (Bagnato & Neisworth, 1991; Messick, 1989; Moss, 1994); (b) the multitude of purposes for which assessment in early childhood is conducted (i.e., screening, diagnosis, eligibility, program planning, and evaluation); and (c) the expanse of cultural variation across Indian Nations. Given the lack of empirical information and the critical need to address cultural issues in early childhood assessment for American Indian/Alaska Native (AI/AN) populations, a descriptive study of current practices and the relationship to validity provides the basis for this investigation. The following research questions served to guide this investigation:

1. What are the current assessment practices reported by parent/ caregiver participants and professional participants?
2. What are parents’/caregivers’ and professionals’ perceptions regarding the ecological validity of reported assessment practices?
3. What are parents’/caregivers’ and professionals’ perceptions regarding the social validity of reported assessment practices?
4. What are professional participants’ perceptions regarding the treatment validity (utility) of reported assessment instruments and procedures for program planning and evaluation?

Review of the Literature

The research related to validity and "best practices" for assessment in early intervention provides a conceptual framework for the present research. The following narrative presents a review of literature related to: (a) examining overall "construct validity" and related component evidences of ecological, treatment, and social validity; (b) examining assessment and validity issues in early intervention; and (c) examining validity issues inherent in assessing culturally and linguistically diverse populations, with further examination of research specific to early childhood populations and AI/AN children and their families.

Construct Validity
"Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment" (Messick, 1989, p.13). Messick provides a model for conceptualizing validity as encompassing a unitary concept. The test developers and test users are responsible for gathering evidence of different types in order to support the overall "construct validity" of a given instrument. "A construct can be defined as an unobservable trait of interest, ‘constructed by professionals’ to explain some phenomenon. Constructs are useful in that they help explain differences among people" (Suen, 1990 p. 147).

Table 1 provides a graphic representation of Messick’s model. Messick (1989) proposes that when attempting to define and measure a construct of interest, one must account for the evidential basis as well as the consequential basis for both test interpretation and test use. Simply relying on reliability indices alone is insufficient (Moss, 1994).

Table 1
Facets of test validity, as portrayed in Fig. 3,1 (Messick, 1980)

The extent to which construct validity is established is contingent upon the degree to which the evidence gathered either refutes or substantiates the construct under investigation. Constructs, by definition, are not absolutes; rather, they continue to structurally evolve throughout the validation process. As such, "construct validation is essentially an ongoing process of conducting various studies to confirm various hypotheses regarding the internal structure of the construct and its relationship with other variables" (Suen, 1990, p. 147).

Ecological validity. The nature of the educational process indicates that an ecological research perspective should be used, rather than a contrived or artificial/laboratory research perspective (Bronfenbrenner, 1986; Gannon, 1986). Ecological validity, a component piece of evidence that can contribute to the overall construct validity of assessment, is based on examining an assessment instrument and procedures using environmental analysis. Ecological validity becomes a critical area of evidence for researchers who strive to develop educationally relevant assessment devices that reflect a given construct of interest. This type of evidence seems to be applicable to all facets of validity proposed by Messick (1989).

Ecological experimentation in education needs to reflect a system of assessing the natural environment which includes the following four subsystems: (a) the microsystem, (b) the mesosystem, (c) the exosystem, and (d) the macrosystem (Brofenbrenner, 1976). Researching these subsystems consists first of evaluating the immediate direct environments of an individual and then progressing on to evaluate those environments which are further removed from the individual. For example, when you are assessing a young child, the microsystem would include the child’s home environment and developmental preschool; whereas, an examination of the child’s macrosystem (the other end of the continuum) would include examining the effects of unemployment on the region, the different individual lifestyles, ethnic customs, and values of the region, etc. It has been further theorized that when an individual’s behavior is considered to be appropriate, or at least tolerable, within these subsystems, he/she is said to be experiencing ecological congruence (Thurman & Widerstrom, 1989). Therefore, "ecologically valid assessment refers to a system of child assessment, environmental analysis, and the identification of strengths and needs of the child’s family" (Benner, 1992, p. 20). The extent to which an assessment device is deemed to be ecologically valid adds to the pool of evidence compiled to substantiate the construct validity of that assessment device.

Treatment validity/utility. Treatment validity/utility reflects attention to the evidential basis for test use that is embedded in Messick’s model. Construct validity and relevance/utility are the key areas to be examined by test developers and users to provide evidential basis for test use. Treatment validity of assessment is established when the benefits relate directly to treatment outcomes (Hays, Nelson, & Jarrett, 1987). The degree to which assessment results can be used to develop functional, relevant objectives is then a measure of treatment validity. Functional, relevant objectives are defined as objectives that relate to useful and practical skills necessary for success in a given environmental setting. The treatment utility of assessment practices, inclusive of procedures, is one fundamental component that must be examined by both researchers and practitioners in early intervention to ensure the provision of appropriate, quality services. This is particularly challenging in light of the numerous test instruments and accompanying procedures from which early intervention practitioners must choose.

Social validity. Social validity involves addressing the consequential basis for test interpretation and use; as such, evidence regarding value implications and social consequences must be obtained and analyzed (Messick, 1989). Social validity of early intervention involves addressing: (a) the needs and wants of society, (b) the social appropriateness of intervention procedures, and (c) the acceptability and satisfaction of those who are recipients of the given intervention procedures (Wolf, 1978). Socially valid assessment then, refers to the acceptability of test instruments and procedures by practitioners and direct recipients (i.e., children, parents, family members). Subsequent development of objectives and intervention procedures that are viewed as acceptable to parents, families, and practitioners are also indices of socially valid assessment, when one considers the linkage of assessment practices to program planning and evaluation. Evaluating the acceptability of test instruments and their corresponding procedures, objectives, and interventions is an area that must not be overlooked in the validation process.

Assessment and Validity Issues in Early Intervention Three major assessment issues are embedded and intertwined in the assessment process and impact on validity. Within the assessment process, one needs to consider the purpose of the assessment, standards for assessment, and recommended types of assessment approaches from which selections may be made (Bailey & Wolery, 1992; Bagnato & Neisworth, 1991; Benner, 1992; McLoughlin & Lewis, 1990; Meisels & Provence, 1989). Matching the purpose of the assessment with corresponding standards and recommended approaches may result in ecological, treatment, and social validity of assessment practices. Valid assessments are critical to fulfilling the "spirit" of the federal mandate for early childhood service delivery, that is, the development of quality Individualized Family Service Plans (IFSPs) and/or Individualized Education Programs (IEPs) that are truly individualized, according to the strengths and needs of families (Bailey & Wolery, 1992; Bagnato & Neisworth, 1991; Benner, 1992; Garcia and Pearson, 1994; Harry, 1992; McLoughlin & Lewis, 1990; Meisels & Provence, 1989).

Purpose of assessment. One of the first steps within the planning phase of the assessment process is determining the purposes of a given assessment and matching that purpose with appropriate practices (i.e., corresponding purpose with recommended standards and approaches inclusive of instrument selection) (Bailey & Wolery, 1992; Bagnato & Neisworth, 1991; Salvia & Hughes, 1990; Suen, 1990). Assessment in early intervention is multi-faceted. Assessment is used to identify children who are "at risk" or who demonstrate developmental delays (i.e., screening); determine the type and extent of disabilities (i.e., diagnostic); determine appropriate goals and objectives for intervention programs (i.e., prescriptive); and provide ongoing information for monitoring program effectiveness (i.e., evaluative) (Bagnato & Neisworth, 1991; Bailey & Wolery, 1992; Benner, 1992). Information obtained from screening and diagnostic tools, usually standardized norm-referenced instruments, indicate a child’s developmental level in relation to other children of the same age. Such tests are administered in a standard manner by highly trained examiners (Salvia & Ysseldyke, 1995). The focus is directed on whether a child can or can’t perform the test item as administered. These tests are designed to verify the nature and extent of a child’s delay. The instructional relevance (i.e., treatment utility) for teachers is minimal at best (Bailey & Wolery, 1992; Neisworth & Bagnato, 1992; Salvia & Ysseldyke, 1995). Additionally, the practice of administering standardized tests in a standardized manner with young children, for whom behavior is usually anything but standard, is suspect (Neisworth & Bagnato, 1992). For the purposes of the remainder of this literature review, therefore, assessment practices that pertain to the purpose of assessment for program planning and evaluation will be summarized and discussed.

Recommended standards and "best practices". Given the seemingly endless numbers of assessment instruments and procedures from which researchers and practitioners must choose in order to meet the individual needs of children and families, members of the Council of Exceptional Children: Division of Early Childhood, in collaboration with researchers and practitioners, have recommended the following standards for assessment in early intervention:

1. Assessment is to be authentic; that is, the child must demonstrate real performance in real situations as opposed to contrived performance on decontextualized tasks. Assessment should be conducted in the child’s natural environments as opposed to one shot in-clinic assessment (Benner, 1992).
2. Assessment is to be convergent; that is, provide for the synthesis of information gathered from several sources, instruments, settings, and occasions to attain a wide base of corroborative evidence (Bagnato & Neisworth, 1991).
3. Assessment is to be collaborative; that is, family-centered where parents participate in all aspects of the assessment process to the extent they desire (i.e., parent/professional partnership).
4. Assessment must be equitable; that is, assessment items and administration must be conducted in such a way as to accommodate cultural and disability differences among children and their families.
5. Assessment is to be sensitive; that is, there are to be enough items across small increments to track program progress (i.e., to allow for the evaluation of social significance of change of a child/family).
6. Assessment is to be congruent; that is, the assessment instrument and procedures used must be field tested on the very children and situations that the instrument purports effectiveness.

Given the large number of component considerations for assessment in early intervention, a convergent assessment model is recommended. "Convergent assessment refers to the synthesis of information gathered from several sources, instruments, settings, and occasions to produce the most valid appraisal of developmental status and to accomplish the related assessment purposes of identification, prescription, and progress evaluation" (Bagnato & Neisworth, 1991, p. 57). Individualized convergent assessment batteries may include any combination of the following formal and informal assessment types: normbased, curriculum-based, curriculum-based dynamic, dynamic assessment, judgment-based, ecological appraisals, anecdotal records, home/classroom observations, self-reports, and interviews (Bagnato & Neisworth, 1991; Benner, 1992; Lidz, 1991; Southern Association on Children Under Six, 1990). Individualized assessment batteries comprised of these conventional recommended measures are used in order to collect information needed to assess the "whole" child, that is, all contexts and skill domains related to the daily functioning of the child (Vacca, 1993). These assessments are administered periodically throughout the duration of service delivery to provide both ongoing data for evaluating effectiveness and additional information regarding changes that may need to take place (Cambell, 1991). Therefore, the ecologies of children and families are interwoven within early childhood assessment batteries, necessitating the use of family-centered, collaborative, team decision making models that facilitate program planning and evaluation that matches the needs of a given child/family.

Validity Issues: Assessing Culturally and Linguistically Diverse Populations Cultural variations (factors), including: (a) language and communication styles; (b) experiential background differences; and (c) variations in values, learning styles, and child-rearing practices (Harry, 1992; Samuda, Kong, Cummins, Lewis, & Pascual-Leone, 1989; Voltz, 1994), have presented numerous challenges for practitioners responsible for testing, placing, and serving children and families. The linkage between testing, placement, and service delivery is plainly evident in the consequences historically experienced by people from culturally diverse backgrounds. Among those consequences highlighted in research studies, are the overrepresentation of minority populations in educational programs serving students with disabilities and their under-representation in educational programs serving students identified as gifted and talented (Harry, 1992; Reshley, 1988; Reynolds, Chastain, Kaufman, & McLean, 1987). Such inequities are associated with the long-standing practice of using Standardized Normreferenced assessment instruments and procedures with populations not represented in the norming process; thus, bias is systematically introduced into the assessment process (Salvia & Ysseldyke, 1995; Garcia and Pearson, 1994; Reynolds, Wang, & Walberg 1987).

Service provision for young AI/AN children and their families must reflect consideration of the cultural and linguistic diversity among AI/AN populations in the assessment process. Valid assessment practices that guide culturally relevant program planning and evaluation that meets the needs of young AI/AN populations, presents a host of problems for those who attempt to use traditional standardized norm-referenced assessment as their foundation. There is a rich research base documenting the impact of traditional assessment practices employed with culturally/linguistically diverse learners at the school-age level and beyond (Reshley, 1988; Brescia & Fortune, 1989; Samuda et al., 1989; Harry, 1992; Garcia & Pearson, 1994; Deyhle & Swisher, 1997, to name only a few). Research on the impact of the extension of those practices to culturally/linguistically diverse learners at the early childhood level is just beginning to emerge. AI/AN populations were not, for the most part, represented in such research endeavors. This is of great concern given that the Indian Nations At Risk (INAR) Task Force (1991) identified educational assessment at both the local and national levels as critical to improving education for AI/AN children and their families. A descriptive research base must be laid as the foundation for addressing assessment and validity issues in early intervention service delivery for AI/AN populations. It is important to empirically document and analyze current assessment practices utilized by practitioners serving young AI/AN children and their families, as well as parent/caregiver and professionals’ perceptions of those practices in terms of ecological, treatment, and social validity.

Method

Instrumentation
Data were collected using a parent/caregiver and professional practitioner questionnaires. The parent/caregiver questionnaire was designed to gather specific information regarding demographics, assessment practices, and parent/ caregivers’ perceptions regarding both the ecological and social validity of assessment practices. The parent/caregiver questionnaire was designed to be as free of educational jargon as possible and to incorporate few items per page with large print to facilitate readability. The questionnaires were written in English (all participants were able to understand English) and questions were read to those participants who needed such accommodation. The professional questionnaire was also designed to obtain information pertaining to demographics, assessment practices, ecological validity, and social validity. In addition it was designed to elicit the professional participants’ perceptions of the treatment validity (utility) of the specific instruments they utilized with young AI/AN children.

The nature of the questionnaire item determined the type of response. Four types of response opportunities included were: (1) Likert scale response to elicit a rating response (e.g., Rate the acceptability of currently used assessment instruments and procedures for AI/AN children and families? 1= not useful; 2= minimally useful; 3= useful); (2) choice responses that were indicated by circling the number to the left of the list of choices to indicate which service was provided; (3) short answer/fill-in-the-blank response to identify specific information that would be discrete (e.g., test instruments used or age of child when first tested, etc.); and (4) open-ended responses to provide the opportunity for additional information beyond structured closed-response opportunities

Subject Sampling
The participants in this study were family members of young AI/AN children (eight years and under) who assume the majority of child-rearing responsibilities, and assessment personnel or service providers (i.e., early childhood educators, physical therapists, school psychologists, speech and language pathologist, etc.) who conduct assessments within the AI/AN community schools and centers. Five states with large AI/AN populations were selected from each major geographical area of the United States (i.e., one western state, one southwestern state, two mid-western states, and one eastern state). One AI/AN Pennsylvania State University alumnus was contacted in each state and asked to act as an intermediary and to make referrals to administrators of early childhood programs serving young AI/AN children with special needs and their families residing in that administrator’s service area. Administrators facilitated data collection and mailed the completed questionnaires to the researcher. A total of 31 questionnaires were returned; 20 were completed by parents/caregivers and 11 were completed by professionals. The return rate for the parent/caregiver participants was 20 out of 34, or 59%, and for professionals 11 out of 13, or 85%.

Table 2
Frequency of responses to demographic questions given by Parent/Caregiver Participants(N=20)

Data Entry and Analysis Procedures for Questionnaire Item Analysis All responses to questionnaire items were coded and entered on a computer. Interrater reliability was conducted by having a doctoral student, who was not involved with this research project check for the accuracy of data entry on 10 randomly selected completed questionnaires. Interrater reliability between the doctoral student and this researcher was 100 percent agreement. Descriptive statistics, including the computing of percentages and means, were generated using the SPSS Statistical Data Analysis (1990) package. In the text, all data results are reported in terms of percentages of responses to each questionnaire item. Additionally, for items common to parent/caregiver and professional participants, the Fisher’s Exact Probability Test was employed to test whether or not differences among parent/caregivers’ and professionals’ perceptions of reported practices were statistically significant.

Results

Data results are summarized within the following categories: (a) assessment practices as reported by parents/caregivers and professionals; (b) perceptions of ecological validity; (c) perceptions of treatment validity; and (d) perceptions of social validity. Additionally, within each of the categories data are subcategorized according to parent/caregiver participants and professional participants. Demographic data were also collected. Table 2 presents a summary of demographic data for parent/caregiver participants and Table 3 presents a summary of the demographic data for professional participants.

Assessment Information The following results are presented in terms of the percentage of participants who responded that the individual practice under each item was provided. For those practices reported by parents/caregivers and professionals that differed significantly, the Fisher’s Exact Probability Test was employed to test whether or not these differences were significant. An alpha level of .05 was used for all statistical tests. Table 4 presents a summary of assessment practices and procedures reported by parent/caregiver and professional participants.

Items unique to parent/caregiver participants. The two items unique to parent/caregiver participants were: (1) What language was used during your child’s testing session? and (2) The testing of my child was done in ______ sessions. Most parent/caregiver participants (65%) indicated that English was the language used during their child’s testing session; however, several participants (30%) did not respond to this item. One parent/caregiver participant indicated that both their Native Language and English were used. The majority of parent/caregiver participants (60%) reported that they did not know how many testing sessions their child had received; of those that did know, 30% reported that one testing session was provided.

Items unique to professional participants. The five items unique to professional participants were those items that addressed the type of service delivery model employed, the number of assessment approaches that the team used, the type of testing format used, and the types of tools the team used for assessments. The fifth item addressed the degree to which a child’s primary language was used during testing sessions. Slightly over half of the professional participants (54.5%) indicated that their team provided assessments according to a multidisciplinary model, and the remainder of the professional participants reported using either an interdisciplinary or transdisciplinary model. The number of assessment approaches reportedly used ranged from one to five; 56.6% of the professional participants who reported that three or more approaches were used, and 36.4% who reported that two or less approaches were used. All the professional participants reported using a standardized format for testing, and some (36.4%) reported that they also used an environmental format. Only one professional participant reported that an arena format was also used. The most frequently used tools for team assessment included standardized norm-referenced instruments (100%), checklists (90.9%), and observational tools (90.9%). Interviews, criterion-referenced instruments, medical evaluation tools, and curriculum- based assessment were reported to be used less frequently, ranging from 72.7% to 54.5% respectively. Specialized testing tools were reported to be used the least (36.6%). Fifty-four percent of professional participants reported that assessments were always conducted in the child’s primary language; the remainder reported that the child’s primary language was occasionally or never used.

Table 4
Frequency of "yes" responses to questions given by parent/caregiver and professional participants regarding their reported assessment practices and procedures

Items common to parents/caregivers and professionals. Responses to four items were elicited from both the parent/caregiver and professional participants. Those questions addressed the services received or offered, the settings where testing occurs or occurred, the type of parental involvement in the assessment process, and the amount of parental input in determining goals for IEPs/ISFPs after assessment information was collected. In responding to the types of services received/offered, parents/caregivers and professionals reported that counseling, outpatient therapy, and support group services were received/offered the least, ranging from (0% to 30%) respectively. Thirty percent of the parent/ caregiver participants and 63.6% percent of professional participants reported that classroom-based intervention services were received/offered. Parent/ caregiver participants and professional participants reported that the following services were received/offered: screening services, 30% and 72.7% respectively; assessment services, 65% and 100%, respectively; home-based intervention services, 5% and 36.4% respectively. Parent/caregiver participants and professional participants differed significantly with respect to reporting IEP/ISFP services, 30% and 100%, respectively. In contrast to the professionals, a significantly smaller proportion of parents/caregivers reported that IEP/IFSP services were received.

The second item addressed the settings where testing was provided. All of the participants reported that testing within the daycare environment is rare, and that testing within a clinical environment is not done at all. Nearly all participants (95% of parents/caregivers and 100% of professionals) reported that testing occurred within the school environment. Testing within the home was reported by 10% of the parent/caregiver participants and 45.5% of the professional participants. Testing within the community environment was reported differently by the two groups of participants. A significantly larger proportion of the parent/caregiver participants reported that testing occurred in their communities (30%), in contrast to professionals, who reported that testing never occurred in community environments (0%).

The third and fourth items addressed the types of parental involvement incorporated in the assessment process, and the amount of parental input utilized in determining the goals to be included in a child’s IEP/ISFP. All the participants indicated that interviews were the primary means of obtaining parental involvement (50% of parents/caregivers and 81.8% of professionals). The participants reported that parental involvement, either through observing assessments or parents assisting in the administration of test items, was rare. Parents/ caregivers reported being somewhat involved in the assessment process through completing home inventories (30%), in contrast to professionals who reported frequently using home inventories (72.7%). Thirty percent of parents/caregivers reported that they were not typically involved in the assessment process, while 18.2% of the professionals who reported that parents were not typically involved in the assessment process. Finally, the amount of parental input in determining goals for their child’s IEP/IFSP was reported as being rarely extensive by both parents/caregivers and professionals.

Perceptions of Ecological Validity Information regarding eight ecological factors (e.g., socio-economic level, education level, community resources, etc.) was obtained when the participants indicated whether or not the specific factor was addressed in the assessment process. Participants rated three additional items dealing with the assessment of ecological systems. The percentage of "yes" responses given by participants who indicated that the individual practice under each item was included in the assessment process or who indicated that the item had been thoroughly examined will be reported. All of the items were common to both parent/caregiver and professional participants. For differences between the parent/caregiver percentage and the professional percentage statistical significance was determined using an alpha level of .05. Table 5 and Table 6 contains a summary of the data regarding ecological validity.

All of the responses given by the participants indicated that socio-economic level, educational level, community resources, cultural perceptions of disabilities, educational priorities of families, and educational priorities of the respective Indian Nations were rarely, if ever, addressed in the assessment process ("yes" responses ranged from 0% - 27.3%). A significantly smaller proportion of "yes" responses were given by parents/caregivers regarding the inclusion of State/Federal guidelines and Parental Rights/Special Education Laws in the assessment process, 35% and 40% respectively; while responses given by professionals indicated that they were both included (90.9%). The first item that addressed ecological systems was designed to obtain information about the degree to which the interrelationships across the child’s immediate settings (i.e., school, park, pow-wows, church, etc.) were examined. Parent/caregiver participants rarely responded "yes", the interrelationship across their child’s immediate settings had been examined; professional participants also responded that it was rare for interrelationships across children’s immediate settings to be examined. All of the responses given by participants also indicated that the impact of larger social systems (i.e., public school system, healthcare system, etc.) upon child development and family strengths and needs were rarely examined. The final item examined the impact of broad culture/subcultures (i.e., regional, ethnic customs and values) on child development and family strengths and needs. Ten percent of the responses given by the parent/caregiver participants and 36% of the responses given by the professional participants indicated that cultural and subcultural impacts had been thoroughly examined.

Table 5
Frequency of "yes" responses to ecological validity questions given by parent/caregiver and professional participants regarding their reported inclusion in the assessment process

Perceptions of Social Validity Three items which addressed perceptions of social validity were specific to parent/caregiver participants, and five items were common to parent/caregiver professional participants. The percentage of "yes" responses to social validity questions given by parent/caregiver and professional participants regarding their reported testing program or tests respectively will be reported. For differences between the parent/caregiver percentage and the professional percentage statistical significance was determined using an alpha level of .05. Table 7 and Table 8 contains a summary of the data regarding social validity.

Items unique to parent/caregiver. Parent/caregivers responded to the following items: Did test information match your family’s wants and needs?, Were you satisfied with how information about your child and family was obtained?, and Upon reassessment, did your child show changes that were meaningful? The responses given by parents/caregivers indicated that the test information rarely matched their families’ wants and needs (15% "yes" responses) and that they were rarely satisfied with how information was obtained (5.0% "yes responses".) None of the parent/caregiver participants indicated that changes their child demonstrated at the time of reassessment were meaningful.

Table 7
Frequency of "yes" responses to social questions given by parent/caregiver and professionals regarding their reported testing program or tests respectively

Table 8
Frequency of "yes" responses to social questions given by parent/caregiver and professionals regarding their reported program

Items common to parents/caregivers and professionals. The five items common to all the participants included: acceptability of program goals based on tests, cultural fairness of test items, culturally sensitive test administration, test instruments selected specifically based on child/family needs, and instrument administration designed specifically to meet the individual child/family needs. Fifteen percent of responses given by parent/caregiver participants regarding instrument selection and administration were "yes" responses, which indicated that instrument selection and administration was rarely individualized; while 45.5% of the responses given by professionals were "yes" responses, indicating that their instrument selection and administration to be somewhat individualized. A significantly smaller proportion of "yes" responses were given by parents/caregivers regarding acceptability of program goals based on tests (10%), cultural fairness of test items (20%), and that test administration was conducted in a culturally sensitive manner (15%). This is in contrast to "yes" responses given by professionals, 51.5%, 81.8%, and 81.8% respectively.

Perceptions of Treatment Validity
Items that addressed treatment validity (i.e., the development of functional goals and objectives/program planning and evaluation) were specific to the professional questionnaire and sought information regarding the specific assessment instruments used and corresponding perceptions of the instruments’ usefulness. Professionals identified screening instruments and assessment instruments by name (fill-in-the-blank response items). The five following items were rated separately for their relationship to program planning and program evaluation: item comprehensiveness for assessing AI/AN populations, utility of scoring methods for obtaining information, utility of procedures for obtaining information, acceptability of instruments and procedures for AI/AN children and families, and the degree of usefulness for developing appropriate instructional methods. The percentage of "yes" responses to each item given by professional participants regarding each instrument’s utility for program planning and evaluation were reported. Table 9 contains a summary of the data regarding treatment validity.

Screening and assessment instruments were rated separately. Eighty-nine percent of the screening instruments reportedly used were standardized normreferenced, and 98% of the assessment instruments listed were standardized norm-referenced. The percentage of "yes" responses given by professional participants regarding their reported instruments’ item comprehensiveness, scoring method, procedures, and instruments and procedures acceptability for use with AI/AN populations, ranged from 54.5% to 75.8%, respectively, for program planning, and ranged from 63.6% to 81.8%, respectively, for program evaluation. Finally, responses given by professional participants regarding whether instruments and procedures were useful for determining instructional methods were analyzed. One third of responses given by professional participants indicated "yes," instruments were useful in this regard. Screening instruments were rated in terms of the degree of precision they had demonstrated when testing AI/AN children and families for the purposes of identification and eligibility. Half of the responses given by professional participants indicated that, "yes," their screening instruments were precise for program planning and program evaluation, as related to the identification/eligibility of AI/AN children and families who were in need of further testing.

Table 9
Frequency of "yes" responses to treatment validity questions given by professionals regarding their reported screening instruments or assessment instruments respectively

Discussion

This study was conducted to examine the assessment process in programs that provide service delivery for young AI/AN children and their families. The analysis of the data revealed distinct trends across the assessment practices and perceptions of ecological and social validity reported by both parent/caregiver and professional participants, as well as across the results regarding professionals’ perceptions of treatment validity of specific instruments that they indicated were commonly used to assess AI/AN children. The specific findings of this study will be presented in terms of: (a) interpretations of the findings and how findings contrast to literature regarding recommended "best practices" for the assessment process inclusive of validity issues, (b) implications of the findings for research and practice, (c) limitations of this study, and (d) suggestions for future research.

Findings Interpreted and Contrasted to Recommendations

The majority of the reported practices revealed gaps between recommended practices and "in use" practices. There were also some significant differences between the parent/caregiver and professional participant groups regarding assessment practices and corresponding perceptions of validity. This section, "Findings Interpreted and Contrasted to Recommendations," will be organized in the following manner: the reported practices will be discussed and compared to "best practices" for the assessment process in relation to the research questions that guided this study: What are the current assessment practices reported by parent/caregiver participants and professional participants?; What are parents’/caregivers’ and professionals’ perceptions regarding the ecological validity of reported assessment practices?; What are parents’/caregivers’ and professionals’ perceptions regarding the social validity of reported assessment practices?; and What are professional participants’ perceptions regarding the treatment validity (utility for program planning and evaluation) of reported assessment instruments?

Assessment Practices and Procedures The assessment process is an integral part of early intervention service delivery; as such, the practices and procedures utilized impact the type and quality of services rendered. The following narrative addresses reported services received/ offered as related to the assessment process and "best practices." Additionally, it provides a contrast between current practices reportedly in use for serving AI/AN children and families and recommendations for practices in relation to instrument/approach selection, authentic assessment, equitable and collaborative assessment, convergent and congruent assessment.

Services Received/Offered as Related To The Assessment Process Professional participants reported providing services in accordance with federal guidelines; that is, conducting screenings and assessments, developing IEPs/ IFSPs, providing classroom and home-based interventions, and related services (i.e., counseling and support groups). However, a significantly smaller proportion of parents/caregivers reported that IEP/IFSP services were received. This difference may have occurred because of AI/AN families’ under-utilization of services, as previously documented in a research study by (Arcia et al., 1993). Alternatively, these differences may have occurred as a result of the parents’/caregivers’ lack of involvement in the assessment process. That may account for their subsequent lack of knowledge about IEP’s/IFSP’s, beyond the act of signing the document.

Instruments/Approaches Used and Recommended Practices For The Assessment Process
Professional participants reported the use of multiple testing instruments/ approaches. They reported that, in addition to using standardized norm-referenced instruments, they used criterion-referenced instruments (63.6%) and curriculum-based assessment (54.5%). However, the results of this study indicate conflicting reports regarding test instrumentation/approaches. When the professionals listed the screening and assessment instruments that they commonly used for providing services to AI/AN children, only 11% of the screening instruments and 2% of the assessment instruments were not standardized norm-referenced instruments. A heavy reliance on standardized-norm referenced assessment instruments/approaches and standardized testing formats, with little attention to other assessment approaches and formats, are practices which parents, practitioners, and researchers have collaboratively cautioned against (American Speech and Hearing Association, 1990; DEC, 1993; National Association for the Education of Young Children, 1990). Professionals were also given the opportunity, within an open-ended question, to indicate other assessment methods they used to obtain information for the development of objectives, instructional methods, and evaluation. No professionals indicated using ecological-based assessment, performance-based assessment, or dynamic assessment. Professionals’ responses to the open-ended question included one participant who said, "observations of the child and interviews with parents & teachers;" another who said, "observational techniques & interviewing informally;" and a third who said, "Regular classroom teachers’ observations are helpful. Student’s learning modalities information is helpful. Kinesthetic learner, visual, auditory etc." The use of judgment-based assessment may be inferred by professionals’ responses to the open-ended question.

Recommendations for Authenticity Within the Assessment Process
Parent/caregiver and professional participants agreed that testing primarily occurred in school settings; yet, approximately half of the professional participants reported providing additional testing within home settings; while only, 10% of parent/caregiver participants reported that this service was provided. It may be that testing within home settings was offered to AI/AN parents/ caregivers, but the service was not utilized for a variety of reasons, or parents/ caregivers may not have been aware that such services were available. Parent/ caregiver and professional participants did agree that testing rarely occurred in daycare settings, and, although a significantly larger proportion of parent/ caregivers reported that testing was conducted within their community settings (i.e., church settings, pow-wow’s, community work events, etc.) professionals reported that this practice was not part of the services that they provide. It may be that persons who were not part of the assessment team observed and talked to the respective parents/caregivers in community settings, or that individual team members may have attended community functions and talked informally with parents/caregivers. This particular finding was surprising and contradictory to responses elicited by other questions regarding participants’ perceptions of ecological validity. These results do merit attention with respect to recommended authenticity of the assessment process; assessments need to be conducted in the child’s natural environments, as opposed to one shot in-clinic assessment (Bagnato & Neisworth, 1991; Benner, 1992; Einhorn, Hagan, Johnson, Wujek, & Hoffman, 1991). In order to get to know the child’s and family’s testing needs, assessment should consistently be conducted in settings beyond the school walls. It is critical that the primary mode of obtaining authentic assessments ensures that the process be both equitable and collaborative in nature.

Recommendations for Equity and Collaboration within the Assessment Process
Federal guidelines and recommended "best practices" stress that the assessment process be an equitable one and that parents/caregivers be active assessment team members throughout all aspects of the assessment process. The results of this study indicated a gap between the recommended practices that incorporate equity and parental involvement and reported practices that are currently in use. The equitable nature of the assessment process may be questionable given that of the parent/caregiver participants who knew what language was used during their child’s testing sessions, 65% reported that testing was conducted in English, while only 35% of parents/caregivers reported that English was the primary language spoken in their homes. It also must be noted that 15% parents/caregivers reported that both their Native language and English (bilingual) was used at home, but this research did not obtain information about Native language or English fluency. Therefore, it is unclear as to the extent of inequity (i.e., parent/caregiver participants who reported that their Native Language was spoken at home, ranged from 30% to 45% and parent/caregiver participants reporting English spoken at home, range from 35% to 50%) based on the language in which tests were administered. However, for those who are bi-lingual, it is important to determine English proficiency prior to testing (Duran, 1989). In terms of parental involvement, participants agreed that parents rarely observed assessments, and they were not afforded the opportunity to participate in the administration of test items. Professional participants reported that their teams used checklists and interviews in addition to standardized-norm referenced instruments; however, when reporting what procedures were used to involve parents in the assessment process, differences emerged. In contrast to professional participants, a smaller proportion of parent/caregivers indicated parental involvement through the use of interviews and home inventories. Finally, a larger proportion of parents/caregivers reported that they were not typically involved in the assessment process, in contrast to professionals’ indications of parental involvement. These differences although not statistically significant (perhaps due to small sample size), are important in terms of the recommended collaborative nature of the assessment process. Collaboration, by definition, involves equal input of all team members including parents/ caregivers. Several parents/caregivers responded to these issues on the openended question that dealt with suggestions the parent/caregiver had for professionals who were working with AI/AN children and families. One parent/ caregiver wrote, "I would suggest that parent/caregiver involvement be uppermost on their agenda-as they are the person most important to the student..."(the remainder of her response will be quoted under perceptions of social validity). Others responded saying; "they (meaning professionals) should know us more", "they need to look at our needs", "they need to know us good", and "we need a Native speaker".

Recommendations for Congruence and Convergence within the Assessment Process
Congruence of assessment (Bagnato, Neisworth, & Munson, 1997) refers to using test instruments and procedures with populations similar to those upon which the instruments and procedures were field tested, and in situations where the instruments have been purported to be effective. The professionals reported practices of instrument/approach selection and formats, and parent/caregiver report of language used during testing, raise questions as to the congruence of the assessment process. The extensive use of standardized norm-referenced instruments, including the revised versions for assessing young minority children (NAEYC, 1990; Washington & Craig, 1992), as well as conducting testing in a language other than the child’s/family’s primary language, do not result in congruent assessment.

In addition to issues of congruence, the assessment process also needs to provide for the synthesis of information gathered from several sources, instruments, settings, and occasions to attain a wide base of corroborative evidence (i.e., convergent assessment) (Bagnato & Neisworth, 1991). The professionals reported practices of instrument/approach selection and format, parent/caregiver and professional participants report of limited settings where testing occurred, and the limited amount of parental involvement are problematic in terms of a match between recommendations for convergent assessment, and current practices in use.

Implications

Construct validity of the assessment process in relation to early intervention service provision for AI/AN populations involves analyzing the interrelationships among reported assessment practices and perceptions of ecological, social, and treatment validity to identify potential road blocks in the provision of assessments that integrate recommended "best practices" and federal guidelines. In analyzing the interrelationships among assessment practices and perceptions that emerged from this study, distinct differences (i.e., gaps) between assessment practices utilized and recommended "best practices" were readily apparent. These gaps have several implications for providing assessment services that are perceived as comprehensive and valid by AI/AN families and the professionals who provide services to young AI/AN children and their families.

Service delivery models that address the assessment process for young AI/AN children and their families need to be developed; such models could benefit by focusing more attention on parental/caregiver involvement throughout the assessment process. Parental/caregiver involvement is critical because it systematically affects all other practices and subsequent perceptions of validity by parents/caregivers. The parents/caregivers who responded to the open ended questions pointed out the need for parental input, cultural sensitivity of professionals, and the opportunity for identifying their family’s strengths and needs. Once parental involvement is established, attention to assessment practices that reflect ecological and social validity can be added. For example, assessments across multiple settings, occasions, and dimensions to obtain information that reflects attention to the "whole child" (including family strengths and needs, cultural variations, resources available, etc.) and shared decision-making in the assessment process could be added. Providing specific training regarding the assessment process for professionals who are responsible for assessing young culturally diverse exceptional learners and their families, with particular attention being given to training regarding the purposes and limitations of standardized norm-referenced testing, may also help close the gap between recommended and actual practices. Such tests are not designed to facilitate program planning and evaluation, yet most of the professionals in this study indicated that they were being used for that purpose. One professional commented regarding the strengths and weaknesses of the instruments and procedures available for assessing AI/AN children and families by stating:

Parents/caregivers also offered their suggestions regarding the testing of their children: "It would be better if professionals had better testing tools to be able to help and work with families." "Children should be tested on things that they should know because of where they live. For example, if they live in Alaska, don’t give them words that are used in a different region because there is no way they should be expected to know what is not in their own environment." and "If the professionals really want to help children and families they need to use a test that shows children and families familiar things from their culture."

The additional finding that only one of the professionals in this study was AI/AN supports previous ongoing documentation regarding the need to increase AI/AN representation in the field of education and related support services professions (Deyhle & Swisher, 1997; Indian Nations at Risk Task Force, 1991; Smith, 1980). Due to the impact of examiner familiarity on student performance, this has particular relevance for the assessment process (Fuchs & Fuchs, 1989). One parent/caregiver spoke to this issue: "The professionals should be American Indian as most of us relate to our own before other nationalities. Our people are more aware of American Indians and relate better and with understanding."

Limitations

This study has several limitations. First, given the small sample size these descriptive data results, subsequent analysis and implications warrant caution. The results cannot be generalized beyond those sites that participated. Early childhood services vary from state to state and further confounding the variability of service provision for AI/AN children and families is the type of schools (i.e., BIA schools, public schools, contract schools, tribally controlled schools, American Indian Head-Start programs, etc.) in which services are provided. Second, the usual limitations of questionnaire methodology that warrant caution are: ambiguity of items, interpretation of items by respondents, less than 100% return-rate, honesty of responses, and interests of respondents in the topic. Additionally, the degree to which the protocol was followed at each site was not controlled by this researcher; rather, the integrity of the data collection procedures was based on the verbal reports of Alumni and administrators. The interrelationship among questions that were categorized into assessment practices, perceptions of ecological validity, perceptions of social validity, and perceptions of treatment validity may have confounded the reliability and validity of these measures. Acknowledging these limitations, this study does begin to address an area where previous research is limited to non-existent and provides some directions for future research with respect to early childhood assessment and AI/AN children and families.

Suggestions for Future Research