Avoiding Potential Problems
ACES Validity
- Significant changes
- Systematic exclusions
- Correlations of predictors with each other
- Reliability of the criterion
- Range of predictor variables
- Number of students
A number of factors affect the usefulness of a validity study. The following is a brief discussion of the factors that affect both admission and placement validity studies.
Significant changes
Validity studies are based on studying the relationship between predictor variables and a criterion variable for a current group of students. This relationship is then used to predict what will occur when a future group of students takes the test. The key assumption is that the relationship between the predictor and criterion that is developed using the current group can be used for prediction with future groups. This will be true only if important factors impacting the relationship do not change over time. For this reason, it is very important to periodically check the original relationship (prediction equations) on a new group of students.
Systematic exclusions
The students that are used for a validity study should be representative of the group of students for whom you are ultimately interested in making predictions. For example, if you are interested in developing equations that will be used to predict college first-year grade point average, students who left college because of failing grades should not be systematically excluded from your study sample.
Correlations of predictors with each other
The ideal prediction equation has multiple predictors that measure relatively different characteristics, and consequently, are not highly correlated with each other. In such a situation, the correlations between the individual predictors and the criterion are, more or less, "additive." A less-than-ideal situation occurs when predictors measure similar constructs and are, consequently, highly correlated. The worst case is using two predictors that are perfectly correlated. It is statistically impossible to determine the best predictor weights as long as both variables remain in the prediction equation.
If two variables are highly correlated (0.9 or better) but not perfectly correlated, the weights assigned to them make little difference in the overall effectiveness of the prediction equation. However, the weights are quite unstable because they are affected by very small differences in the student records; the addition or deletion of a single record could alter the weights considerably. Consequently, it is very important to avoid using predictor variables that are highly correlated.
Possible consequences of correlated predictor variables
The ACES user should exercise caution when interpreting ACES study results that include highly correlated predictor variables (multicollinearity). The analyses performed by ACES operate under the assumption that the predictor variables are independent (uncorrelated), and violating this assumption may result in less precise prediction estimates with large standard errors. A typical situation where correlation of the predictor variables exists is when a composite variable, such as an admission index, is used as a predictor in the same analysis where any of the individual variables comprising the composite are also used. For instance, if the composite variable includes SAT (SAT) scores, then the models including both the composite variable and the SAT scores as predictors may yield results where the SAT scores seem to be contributing little, if anything, to the prediction. This outcome will occur because some of the predictive information contained in the SAT scores is attributed to the composite variable.
Correlations in Placement Studies
The correlations utilized in placement studies are logistic biserial correlations. A biserial correlation is computed when one of the variables to be correlated is dichotomous (e.g., course grade of C or higher vs. course grade lower than C), while the other variable has many possible values—enough to be considered, for practical purposes, as a continuous variable (e.g., a test score). The calculation of a biserial correlation assumes that the dichotomous variable is the observable indication of an underlying continuous variable. The biserial correlation is an estimate of the correlation of that underlying continuous variable (e.g., the tendency to earn a course grade of C or higher) with the continuous variable that could actually be observed (e.g., the test score).
Correlations are particularly unstable if they are estimated from a small number of students or from a group of students with a very high proportion of students who succeeded or a very high proportion of students who did not succeed.
In addition, correlations can often be underestimated because of restriction of range due to placement decisions based on scores from one test, which restricts the amount of variation in the predictor. Correlations may also be deflated when a criterion, such as "college success," is dichotomized as "pass" or "fail," which eliminates meaningful variations in course performance.
Reliability of the criterion
An underlying assumption of all validity studies is that the criterion is perfectly reliable and that the criterion scores contain no error. However, we know that no criterion is perfectly reliable. For example, students receiving a grade from one instructor might have received a different grade for the same work if they had been given a different instructor.
In the case of the criterion of college grade point average, students take different programs of study, different courses, varying numbers of courses, various sections of the same course taught by different instructors, etc. Each of these factors influences grade point average in some way, weakening the relationship of the criterion and the predictor variable(s).
Range of predictor variables
Predictor variables that are used for an admission validity study are typically the same test scores that are used for admission into your institution. Consequently, they typically do not include uncharacteristically low scores because students with these scores may not have been admitted to your college or university. Furthermore, students with either very high or very low scores may not have applied to your institution. As a result, the range (spread from high to low) of the scores on the predictor variables you are investigating may be restricted. The range on the criterion may also be restricted if students who think they may be failing withdraw from a course or from school before they receive their grades.
Students with scores in either extreme of the score distribution (very high or very low scores) are usually the students with the most predictable successes or failures. If students become more alike, the range of scores on the predictor variables is reduced, and the predictive effectiveness of the test is reduced. An extreme example would be if all students had the same score on the predictor; it would not be possible to develop prediction equations for the test.
ACES™ admission studies adjust for range restriction on the high school academic measure and SAT® predictors by using national statistics for these variables. Due to the need for a population sample to complete the adjustment, ACES does not adjust for restriction of range on the criterion variables. ACES is also unable to adjust for range restriction in placement studies, as the population sample for the classes you are placing students into is unknown.
Number of students
It is important to have a reasonably large sample of students to conduct a validity study, since the likelihood of a sample being similar to the population it is intended to represent is increased if the sample is a large, random sample from this population. In addition, validity coefficients used in the predictor equations are extremely sensitive to chance fluctuations—in some cases, a change in one score can make a large difference. For this reason, ACES requires a minimum sample size of 75 students for an admission validity study and a minimum of 30 students for a placement validity study. (Refer to the specific type of placement validity study you are planning to do for guidelines regarding the minimum number of students required for that type of placement study.)