About Admission Validity Studies
Understanding ACES Admission Validity Studies
- How can I benefit from an Admitted Class Evaluation Service (ACES) admission validity study?
- How ACES analyzes your data in an admission validity study
- Subgroups in admission validity studies
- How do I design an ACES admission validity study?
- What does an ACES admission validity report look like?
How can I benefit from an ACES admission validity study?
An ACES admission validity study will
- Provide information on the most useful predictors of success at your institution.
- Provide information allowing you to narrow the number of factors considered in the admission process without loss of predictive ability.
- Provide optimal equations for predicting the success of future students.
- Provide a list of the students most at risk for dropping out.
- Provide a matched student-level data set for use in follow-up studies.
The admission validity report includes a one-page comprehensive executive summary, Results at a Glance.
When you receive your admission validity report, you have the option to download an electronic copy of the student data used to process your ACES study. It contains the information you sent to ACES, to which has been added information on the students' scores for the SAT™, SAT Subject Tests™, Advanced Placement Program® grades, and demographic information extracted from the ACES database, as well as the statistics generated during the processing of your study. The file is available in both a text format and Microsoft EXCEL® format. This enhanced data file provides a rich resource for investigative research.
How ACES analyzes your data in an admission validity study
For an admission validity study, ACES begins by matching your students to the SAT database. If a match cannot be found, the student will not be included in the ACES admission study.
ACES provides you with an analysis based on each individual predictor, as well as combinations of your predictors. ACES calculates the prediction statistics two different ways. ACES will calculate the statistics using every student for whom there is data on any of the respective variables. This is reported as the "individual admission measures." ACES will also calculate the statistics for all students with SAT scores, a high school measure, and at least one other predictor in the model. These are the "combined admission measures." Using all of the students with a given predictor for the calculations will generally result in the best, most stable, results for those predictors. Using the students who have taken all of the predictors for the calculations, however, allows you to compare the results of the various predictors using the same sample. Using the same sample will allow a more direct, fairer comparison.
Subgroups in admission validity studies
ACES admission studies break down results on the basis of gender, ethnicity, and English Best Language (EBL) if your sample has at least 75 students for two or more levels in each of these categories. For example, you must have at least 75 males and 75 females. You have the option of requesting two additional subgroups from ACES-supplied data, your own data, or a combination of the two.
There are a number of reasons for considering subgroup results. For example, the criterion might be different for each subgroup, such as when the college first-year GPAs for engineering majors differ from the first-year GPAs of education majors.
It is also appropriate to consider subgroup results when it is suspected that one or more of the predictors might function differently for each subgroup. This could be the case with high school GPA, which may give a different result for older applicants, who have been out of high school for a few years, than for younger applicants recently graduating from high school.
Subgroup information can also be used to look at the fairness of your admission criteria for various populations, such as ethnic groups. If the relationship between the criterion and the predictor is quite different for any two groups, it may result in significant under- or over-prediction of the criterion for one of the groups. ACES provides separate equations for the subgroups of students you chose to study and will also supply equations for gender, ethnicity, and EBL subgroups upon request. However, many colleges and universities have noted that policy considerations and conceptions of fairness have prevented them from using separate prediction equations.
Although you will probably not use separate prediction equations for admission purposes, information provided about subgroup performance may be of interest to you in monitoring how special groups of students, perhaps ethnic or language groups, perform after they are admitted into your institution. If the particular subgroup's actual performance is considerably above or below its predicted performance, you may want to pursue possible explanations for this outcome.
How do I design an Admitted Class Evaluation Service (ACES) admission validity study?
- Admission studies: predictive validity
- Choosing a criterion
- Selecting predictor variables
- Whom to include in the study
- How to collect data
- How to submit data
- Format for data submission
Admission studies: predictive validity
Predictive validity is the basis of the ACES admission validity study. ACES evaluates your admission model so you can better predict those applicants who are likely to be successful at your school.
Choosing a criterion for an admission validity study
In an admission validity study, a general rule is that the criterion for the study should be some measure of the outcome of the admission decision, a measure of the success of the accepted applicants. The criterion must be quantifiable. In other words, it must be described by a set of numbers that is meaningful and, for ACES, along a continuum with at least four points. ACES cannot utilize a dichotomous criterion, such as Pass/Fail. Some examples of a criterion are grade point average or teacher ratings.
Good criteria must be measurable, reliable, comparable for all students, and related to the admission objectives. In these types of studies, there is always a conflict between the use of an intermediate or a long-range criterion, such as first-year versus graduation grade point average.
ACES uses college first-year GPA as the default criterion for success in the admission validity studies. If you choose to submit individual course grades, ACES can use this information to adjust the first-year grade point average to more accurately reflect the course-taking patterns of the first-year students at your institution.
Research has shown that adjusting the grade point average for course selection provides a more accurate portrayal of the validity of the predictors than the standard first-year grade point average. This adjustment accounts for high-performing students who take more difficult courses and may receive a lower-than-expected first-year GPA. Conversely, it also accounts for lower-performing students who take easier courses and as a result may obtain a higher first-year GPA than expected.
Selecting predictor variables
The primary purpose of an admission validity study is to evaluate the measures that are used in admission decisions—the predictors—to determine how well they work alone and in combination to predict student success. A good predictor, however, has several other important qualities. It should be widely available, reliable, and fair to all students. Predictors should show sufficient variation in scores to differentiate student ability, without large clumps of students at the top or the bottom scores.
When selecting predictors to include in your study, you should include any possible contributors to your admission decisions. ACES can help to sort through them for redundant or weak contributors, as well as providing prediction equations for future students.
The ideal prediction equation has multiple predictors that measure relatively different characteristics, and consequently, are not highly correlated with each other. In such a situation, the correlations between the individual predictors and the criterion are more or less "additive." A less-than-ideal situation occurs when the individual predictors measure similar constructs, and consequently, are highly correlated (e.g., high school rank and high school GPA); the worst case is when two predictors are perfectly correlated.
Whom to include in the study
General guidelines
Usually you will want to include in your study anyone for whom you have data for the various predictors. There are, however, a few cautions. First, if the students you include in your sample are not representative of the students in your admission pool, the results are not likely to generalize to a new group of students. Second, if the students you include do not have most of the variables you wish to evaluate, comparisons will be difficult.
Specific guidelines
Students with Pass/Fail grades
In order to effectively evaluate how well a predictor is working, ACES requires a range of at least four points on a scale for the criterion. Thus, students cannot have a criterion that is a dichotomous variable (e.g., Pass/Fail for the entire first year). However, students can have individual classes graded on a Pass/Fail or Credit/No Credit basis.
Part-time students
Part-time students can be included in your ACES study. However, part-time students can be very different from full-time students (e.g., they may be, on average, much older than traditional full-time students right out of high school). Therefore, you might want to include a school-supplied subgrouping variable stating the status of the student (e.g., 1= full-time, 2= part-time) so that you can compare them.
Students with a Fail, Incomplete, or Withdraw grade
Generally, we have found that at most institutions failing grades, such as F or E, are calculated into the students' grade point average and are equivalent to zero grade/quality points; and Incomplete or Withdraw grades, such as I or W, are not incorporated into the GPA calculation. If this is true for your school, then you should include students who have received any (or all) of these grades. If you are providing individual course information, you will have an opportunity, when submitting your data, to identify how many grade/quality points each one of your grades is worth. At that time you would simply omit I and W from the grade translation table and ACES will filter out those grades before analyzing your data.
These are only given as general guidelines. If you have a large number of students who failed courses for any reason other than doing poorly on the tests and/or class work (such as, dropping a class or leaving the school without officially withdrawing), or if your Incomplete or Withdraw grades are calculated into the GPA, please contact the ACES staff at aces@info.collegeboard.org or by calling (609) 921-9000 to discuss your specific situation.
Students who do not complete their first year
If the student completed at least one semester, you should definitely include their data; if not, it is probably still a good idea. Although these students' records will not be used in the validity analyses, they will be included on the electronic files you get back from ACES. This might help you to do internal retention analyses for students who drop out.
Students who are missing SAT® scores
If you do not have SAT scores for some students, you should still include these students in your study. ACES requires that SAT scores must be present for each student record. However, we sometimes have SAT scores for students who did not release them to you. In these cases, we will add the score and use the student record in the analyses done for your study. ACES matches student data based on name, social security number, date of birth, and gender. If any of this information is missing, ACES will try to match the student with our database based on the available information, but we cannot guarantee that we will be able to find the student's information.
International students who are missing rank-in-class, high school GPA, SAT scores, and/or any of the ID variables required by ACES (name, SSN, date of birth, and gender)
You should include records for international students even if they are missing one or more of these predictors or identification variables. We will attempt to match these records to the ACES database. It may have information for these students that was not released to you by them. In that case, we will include the students' records, along with their scores and demographic information, on the electronic files we return to you. However, ACES will not include in its analyses any student that does not have a criterion score, a high school measure of achievement, or SAT scores.
Note: If your school assigns "fake" Social Security Numbers to international students or students without an SSN, please do not include this number in your ACES layout design.
How to collect data
Students included in an ACES admission study should have four pieces of identifying information: name, social security number, date-of-birth (DOB), and gender, as well as a criterion score, SAT scores, and a high school performance measure. The high school performance measure may be either high school grade point average or high school rank. If you use high school rank, ACES requires the exact rank and the size of the high school class for each student. You may decide to submit two separate study requests, one using high school grade point average and the other using high school rank.
A school can request up to five additional predictors. These additional predictors can be provided by the school or can be chosen from the SAT Questionnaire. The SAT Questionnaire is an optional questionnaire completed by the students at the time they register for the SAT. The variables chosen by the school, whether they are provided by the school or chosen from the SAT Questionnaire, are referred to as "school-provided predictors."
Up to six different samples are analyzed in an ACES admission study. The six samples are:
- Students with SAT™
- Students with SAT and high school information
- Students with SAT, high school, and school-provided predictors
- Students with SAT and SAT Subject Tests™
- Students with SAT, SAT Subject Tests, and high school information
- Students with SAT, SAT Subject Tests, high school, and school-provided predictors
Not all school-provided predictor information must be present for a student to be included in samples 3 and 6. If a student has at least one school-provided predictor, they will be included. Also, only one SAT Subject Tests test score is needed for a student to be included in samples 4, 5, and 6.
Occasionally, students will have a predictor score or criterion that is significantly different from the scores of other students in the sample. ACES will remove students from the sample if they differ from the mean score by more than three standard deviations. This prevents outlying scores from unduly influencing the results.
How to submit data
If you are submitting course grades for an admission study, you may submit the files of student data as one merged file or two separate files. If two files are being submitted, the first file containing high school information, the predictors, and/or the grouping variables must be in a horizontal format. The second file containing the course information can be in a stacked (vertical) or horizontal format. Each file must contain the key student identifying fields of student name, social security number, date of birth, and gender. The student identifying data must be in the same format for both files, including the layout of the social security number, format of the birth date, the gender designation, and the upper/lowercase format for the name. The ACES data submission form provides a space for you to identify how the student names are formatted in your files.
Example of stacked format:
| Jane Smith | 11/28/81 | 111-11-2345 | F | Math | 3 | A |
| Jane Smith | 11/28/81 | 111-11-2345 | F | English | 3 | B+ |
| Jane Smith | 11/28/81 | 111-11-2345 | F | Biology | 3 | B |
| Jane Smith | 11/28/81 | 111-11-2345 | F | French | 3 | F |
Example of horizontal format:
| Jane Smith | 11/28/81 | 111-11-2345 | F | Math | 3 | A | English | 3 | B+ | Biology | 3 | B | French | 3 | F |
Format for data submission
Data may be submitted as a fixed-length ASCII file, a comma-delimited ASCII file, a tab-delimited ASCII file, a Microsoft ACCESS® database, a Microsoft EXCEL® workbook, a SAS® transport file, or an SPSS® portable file.
What does an ACES admission validity report look like?
View a sample admission validity report (.pdf/1M)
Require Adobe Reader (latest version recommended).