Glossary
Glossary of ACES™ Key Terms
Familiarity with the key terms and concepts of a validity study will enable a better understanding of study results. Definitions and/or descriptions of some key terms are below. Either scroll through the following text or use the direct links provided:
- Adjusted coefficient
- Correlation
- Cross validation and replication
- Cut score
- Multiple regression
- Negative weights
- Normal distribution
- Regression and regression equation
- Sample and sampling error
- Standard error of estimate
- Variable
- Variance and standard deviation
Adjusted coefficient
An adjusted coefficient is a statistic that has been revised to estimate its value under conditions other than those in the sample on which it has been calculated. In the Admitted Class Evaluation Service™ (ACES™) some correlations, multiple correlations, and r-squared statistics are adjusted for restriction of range in the sample. Because the data sent to ACES are based on enrolled students at your institution, the range of scores of these students may be different from the range of scores from the total applicant pool from which they were selected*. When a range of scores is artificially restricted in this manner, correlation results are lower than they would be for the total applicant population. Since admission decisions are made on the entire applicant pool, the relationship between variables should be modeled as if the entire applicant population had been used.
The ACES statistical system can also adjust the criterion for course-taking patterns. If the institution submits individual course grades for a given first-year class, statistics can be obtained for an individual course. These statistics characterize courses in terms of the stringency of grading standards at each institution.
*ACES assumes that the applicant pool is all college-bound seniors who took the SAT™.
Correlation
The correlation coefficient is basic to communicating validity study results. It summarizes in a single number the linear relationship between two variables and indicates the degree to which a change in one variable is related to a change in the other. The value of the correlation coefficient ranges from -1 to +1, inclusively, where
- -1 indicates a perfect negative relationship with a high score on measure A associated with a low score on measure B, or a low score on measure A associated with a high score on measure B;
- 0 indicates no relationship; and
- +1 indicates a perfect positive relationship with a high score on measure A associated with a high score on measure B, or a low score on measure A associated with a low score on measure B.
Cross validation and replication
ACES recommends that you continue to collect data for both placement and admission validity studies, and that these studies be replicated to be sure that the results of the initial study remain applicable to the current population.
Replication is simply a repetition of the study with a new sample.
Cross validation is the application of scoring weights or prediction equations derived from one sample to a different sample. This process allows estimation of the extent to which chance factors determined the weights or equations or inflated the validity estimated in the analysis sample.
In high-stakes situations, such as college admission, we recommend a new study at least every three years but suggest a new study every year.
Cut score
A cut score is a point on a score scale at and above which test-takers are classified in one way and below which they are classified in a different way. For example, if a cut score is set at 60, then students who score 60 and above may be classified as "ready for college level courses," and students who score 59 and below classified as "needing developmental courses."
Multiple regression
In some situations, one factor can be used to predict a certain outcome (e.g., upper body strength predicts the amount of weight one can bench press). In complex, real-life situations, such as admission decisions, there is no one score or piece of information that can adequately predict who will be successful in college. Therefore, several different types of information are needed; the more types of relevant information you use, the better you are able to predict success. In the case of college admission, success may be measured using first-year GPA. That is why colleges use a combination of predictors (SAT scores, high school GPA, interviews, class rank, etc.) in admission decisions. A statistical process known as multiple regression is used when more than one set of information (test scores, high school GPA, etc.) are used to predict one criterion (such as first-year GPA).
Multiple regression assigns relative values, or weights, to the various predictors to determine the optimal combination of factors to predict the desired outcome, or criterion. For example, at a particular college the most accurate combination of predictors may give more weight to the critical reading score on the SAT than to the mathematical reasoning score, while the high school GPA gets more weight than both test scores combined. In another college which emphasizes quantitative skills, more weight may be assigned to the SAT mathematical reasoning score than to either the SAT critical reading score or high school GPA. The best combination of predictors will result in the least error in predicting the criterion.
The Admitted Class Evalution Service (ACES) computes the multiple regression equations for you, using standard statistical measures such as correlations, prediction equations, and the likely error in prediction.
The following example shows how the predictors SAT critical reading score, high school grade point average, and an Ability rating in writing (an optional ACES predictor generated from the SAT Descriptive Questionnaire) would be combined in a multiple prediction analysis.
One equation derived would use only the SAT critical reading score to predict first-year GPA:
First-year GPA = A (SAT critical reading score) + B, where ACES provides the appropriate values for A and B.
A is often referred to as a computational weight for the predictor, and B is often referred to as the constant.
The second equation would use SAT critical reading score and high school grade point average:
First-year GPA = C (SAT critical reading score) + D (HS GPA) + E.
ACES provides the appropriate numbers for C, D, and E in order to combine the two predictors optimally with the least average squared error.
For three predictors, the form of the prediction equation is the same as for two predictors, except for one additional term. ACES supplies the numbers of F, G, H, and J in the equation below in order to combine the three predictors optimally with the least average squared error.
First-year GPA = F (SAT critical reading score) + G (HS GPA) + H (Ability rating in writing) + J.
The prediction equations for both multiple and single predictors are used in similar ways; values are entered for each of the predictors used. ACES provides the option to use up to five additional predictors beyond SAT critical reading, SAT mathematical reasoning, and high school GPA or rank in class. For each student in the study, the difference between the predicted value on the criterion and the obtained value is error. The average squared error for all the students is the unexplained variance, and the square root of the unexplained variance is the standard error. The difference between the total variance and the unexplained variance is the explained variance. Finally,
The multiple-correlation coefficient varies from 0 to +1; unlike the single-predictor correlation coefficient, it cannot be negative. The multiple-correlation coefficient generally will not decrease when a predictor is added; it will stay the same or, more commonly, will increase. The standard error, on the other hand, usually decreases.
Negative weights
In a regression equation, a negative weight may occur because a predictor is inversely related to the criterion (e.g., if a high score on a certain reading test was correlated with a lower first-year GPA).
Negative weights occur for a variety of reasons. In some cases, the predictors may be too similar or too highly correlated. This is seen with test scores for courses that are highly correlated. Adding the second course test score can result in a negative weight for one of the variables. In this case, it is not helpful to use both test scores as predictors since the addition of the second test score does not make the prediction more accurate. (Remember that the idea is to account for more of the prediction error with each additional variable.)
Negative weights may also occur when the predictor or criterion is on a scale, such as from 1 to 5, with 1 representing the highest score. This is opposite of most scales where the greater number would represent the higher score, as with GPA, SAT® scores, etc. When this situation occurs (e.g., high school rank), the first scale is often inverted to make calculations easier and to demonstrate the effectiveness of the predictors more clearly.
Negative weights may also be associated with a predictor that is completely unrelated to the criterion. Most negative weights of this type are very small. It is advisable to drop the predictor with the negative weight from the equation because, frequently, it does not maintain its negative weight during cross validation, and it is difficult to justify penalizing a student for doing well on a predictor.
Normal distribution
A model called the normal distribution can represent most real-life distributions of student data. Normal distributions have the following characteristics:
- Most of the numbers are close to the mean (average). Approximately two-thirds (68%) are within one standard deviation from the mean; nearly 96% are within two standard deviations from the mean; and virtually all (99.7%) are within three standard deviations from the mean.
- The distribution is symmetrical above and below the mean. For example, 34% of the numbers are one standard deviation above the mean and 34% are one standard deviation below. The normal distribution can be represented in the shape of a bell curve, with its center located at the distribution's mean.
Regression and regression equation
The predictor and criterion value for any student can be plotted on a graph. For example, if the SAT critical reading score and the college first-year GPA of 200 students were plotted on a graph, the results would resemble those found in Figure 1. Note that numbers along the bottom of the graph (the horizontal axis) represent SAT critical reading scores and numbers along the side of the graph (vertical axis) represent college first-year GPA. Each student is represented by a point in the plot shown in Figure 1.
Students represented by the points in the upper right hand portion of the plot obtained high SAT critical reading scores and have a high first-year GPA. Conversely, students represented by points in the lower left hand corner of Figure 1 have both low SAT scores and low first-year GPAs. Notice that there are a few outliers on this plot (i.e., students who may have obtained low SAT scores and have fairly high first-year GPAs, represented by point A in Figure 1, or students who have high SAT scores but low first-year GPAs, represented by point B in Figure 1). For the most part, however, there is an upward trend, indicating a positive correlation between the SAT critical reading score and college first-year GPA. The higher the SAT critical reading score, the higher the first-year GPA, and the lower the SAT critical reading score, the lower the first-year GPA.
To predict the average first-year GPA for any student in Figure 1, first predict the average first-year GPA for the group. The average GPA for this group is 2.5, represented by the horizontal line shown in Figure 1. Predicting 2.5 as the first-year GPA (using the horizontal prediction line) would be an accurate prediction for some students but not others, such as those that are represented by the points that fall away from the horizontal prediction line in Figure 1. Prediction of these scores can be improved by finding a new prediction line. Because the points predominate along a line from the lower left to the upper right of the graph, we would improve our prediction substantially by turning the horizontal line counter clockwise and using the new line as a prediction line, as shown in Figure 2.
There are many prediction lines that could be drawn and used to predict first-year GPA. ACES develops and tests these prediction lines until it produces the line that minimizes the error in predicting GPA using SAT critical reading scores. In some cases, only some of the points lie close to the diagonal line (the prediction line). ACES chooses the line that passes as close as possible to the largest number of points, as seen in Figure 2. If the correlation between SAT critical reading scores and first-year GPA is very high, the points will follow the prediction line closely, and there will be very little error in the estimated or predicted values of first-year GPA. If the correlation is low, the points will be quite scattered, and points falling away from the diagonal line will be poorly estimated (i.e., have a large error of estimate) when the diagonal line is used as the prediction line.
For example, the prediction line shown in Figure 2 to predict first-year grade point average from SAT critical reading scores should be used in the following way: Suppose a student had an SAT critical reading score of 370. Locate that score on the horizontal axis of the diagram. Now move vertically to the prediction line. At the prediction line, draw a horizontal line to the left to meet the vertical axis, as represented by the dotted lines in Figure 3. The number you intersect on the vertical axis (approximately 1.6) is the predicted GPA for all students who have SAT critical reading scores of 370. Notice that a number of students with SAT critical reading scores of 370 have a first-year GPA that is greater or less than 1.6. Consequently, the predicted value of 1.6 will contain some error.
All straight lines can be described by the formula Y = AX + B, where X is the variable on the horizontal axis and Y is the variable on the vertical axis. In Figures 1-3, X is SAT critical reading score and Y is college first-year GPA. Typically X, the predictor variable, is represented on the horizontal axis and Y, the criterion variable, is represented on the vertical axis.
The prediction equation for predicting college first-year GPA from SAT critical reading scores can be represented as:
First-year GPA = A (SAT critical reading score) + B.
Numerical values for A and B define a single line. The choice of the best line becomes the choice of the appropriate values for A and B. The A and B values describing the line in Figure 2 are 0.005 and –0.2381, respectively. Consequently, the equation for predicting first-year GPA from SAT critical reading scores can be written as:
First-year GPA = 0.005 (SAT critical reading score) + (-0.2381)
The A value is called the slope of the line. It is the expected increase in the number of first-year GPA units for every one-point increase in the SAT critical reading score. For our equation, we would expect an increase of 100 SAT critical reading score points to add about (100) (0.005) = 0.5 points to the first-year GPA.
The B value is called the Y-intercept. This is where the line will cross the vertical axis (i.e., the point on the vertical axis when the horizontal axis is zero). In our illustration, this is the value of first-year GPA when SAT critical reading is zero.
You can predict the first-year GPA for an applicant to your institution by using the prediction formula: insert the X predictor value (SAT critical reading score) into the equation and determine the predicted first-year GPA. A student applying to your institution with an SAT critical reading score of 290 has a predicted first-year GPA of (0.005) (290) + ( -0.2381) = 1.21.
Sample and sampling error
A sample is a subset of a larger population. For example, a few hundred high schools may be selected to represent the more than 20,000 high schools in the United States. Samples differ in how well they represent the larger population. Generally, the care with which a sample is chosen has a greater effect on its ability to represent the entire population than does the size of the sample. Sampling error refers to the difference between the statistic derived from a particular sample and the corresponding parameter for the population from which the sample was drawn. In other words, in order to be reasonably certain that the results of the validity study are applicable to other students with similar characteristics, it is important that the sample of students studied should be representative of the population from which they are drawn.
Often the first question asked of a validation expert is, "How large does my sample need to be?" Because it is dependent on how representative the sample is, this is a difficult question to answer. ACES requires 75 students for an admission study and 30 students (or 10 times the number of predictors, whichever is greater) for a placement study, but these should be considered the lower limits.
Standard error of estimate
Despite using the best statistical processes, all the factors that lead to a particular criterion value can never be accounted for. The standard error of estimate is a measure of how different actual scores are from the scores predicted by a regression equation. It is the amount of error expected in predicting criterion scores.
There are several ways to derive the standard error of estimate for a predicted score. One way is to use the following equation:
Standard error of estimate = (standard deviation of criterion)where r is the correlation coefficient between the predictor and the criterion.
In most circumstances, the difference between the actual and the predicted GPAs for all students with the same score on the predictor are distributed normally. The actual GPA can be thought of as being distributed according to the normal distribution with a mean equal to the predicted GPA.
For example, our predicted first-year GPA for a student with an SAT critical reading score of 540 is:
First-year GPA = (0.005) (540) + (-0.2381) = 2.46.
Some of the students with an SAT critical reading score of 540 would obtain a GPA of exactly 2.46; however, some will obtain GPAs higher or lower than that. A reasonable approximation is that the mean of these GPAs will be the predicted mean, in this case 2.46. Most of the GPAs will be close to that mean and will be as likely to fall above it as below it. The normal distribution is a good approximation and the standard deviation of the normal distribution is the standard error of the estimate.
This information is helpful in knowing how close the prediction is likely to be. For example, if we predict a first-year GPA of 2.46 for students with an SAT critical reading score of 540, about two out of three of those students are likely to have a true first-year GPA within one standard error of the prediction, in the range from 1.90 to 3.02; about 96% of those students will be within two standard errors, from 1.34 to 3.58.
Variable
A variable is an attribute that can take on different values, such as scores, grade point average, family income, age, height, or weight. For validity studies the predictor variable is often a test score or other measure used to predict relative success or failure in the future, and the criterion variable is a measure of that success, often a GPA or a course grade. Some examples follow:
- Suppose you use high school grades, SAT critical reading, and SAT Subject Test Math Level 1 to determine whether to admit students to your institution. In an admission validity study, the high school grades, SAT critical reading, and SAT Subject Test Math Level 1 are the predictor variables. You will then need to determine a criterion variable that tells you whether or not you admitted the correct students — usually first-year GPA.
- Suppose you use ACCUPLACER ESL™ Language Usage Scores to determine whether to require students to take an English-as-a-Second-Language class. Since you are concerned about their success in a college-level English class, you may wish to collect information from the college-level classes and use ACCUPLACER ESL Language Usage scores as your predictor variable and final grades in the class as your criterion variable.
- Suppose you use CLEP® College Algebra to determine whether students should be allowed to receive credit for their college algebra requirement. You will probably want to analyze data from students in the regular college algebra course, using the CLEP College Algebra score as your predictor variable and final grades in the course as your criterion variable.
Variance and standard deviation
Variance and standard deviation are important descriptors of any distribution of numbers, such as a set of test scores. Both describe how numbers within a set differ from each other. The variance is the average of the squared differences from the mean (average) of the distribution of numbers, and the standard deviation is the square root of the variance.
The significance of these numbers is in their relationship to the number of students at various distances from the mean. Approximately two-thirds (68%) are within one standard deviation from the mean; nearly 96% are within two standard deviations from the mean; and virtually all (99.7%) are within three standard deviations from the mean. This allows you to have a good idea of the distribution of the scores by looking at just one number.
The following example shows how these statistics are calculated.
The college first-year grade point average (GPA) for five students is shown in the table below. If you sum the five GPAs, the number you get is 15.0. Divide this number by five (the total number of GPAs) to get the mean GPA, which is 3.0. You can now determine a difference from the mean for each student. These differences are shown in the third column of the table. Next, square each of the differences. The squared differences are then summed to obtain 2.50. This number is then divided by five (the number of GPAs) to get an average squared deviation (difference) from the mean, which in this case is 0.50. This number is called the variance of the GPAs. The square root of the variance, called the standard deviation, is approximately 0.7.
First-Year GPA
| Student | GPA | Difference from the Mean | (Difference from the Mean)² |
|---|---|---|---|
| #1 | 4.0 | +1 | +1 |
| #2 | 3.5 | +0.5 | 0.25 |
| #3 | 3.0 | 0 | 0 |
| #4 | 2.5 | -0.5 | 0.25 |
| #5 | 2.0 | -1.0 | 1.00 |
| (Sum) 15 | 0 | (Sum) 2.50 |
Note that squaring the difference from the mean results in relatively large, positive numbers for large differences from the mean, and relatively small, positive numbers for small differences from the mean. Therefore, the average of these—the variance—reflects how the original numbers differ from the mean and from each other. If the original numbers are very different from each other, the variance (and hence, the standard deviation) will be large. If the original numbers are close to each other, the variance (and the standard deviation) will be small.