Predictive Placement Validity Studies
ACES Predictive Placement Validity Studies
- How can I benefit from a predictive placement validity study?
- How do I design a predictive placement validity study?
- What does an ACES™ placement validity report look like?
How can I benefit from a predictive placement validity study?
A predictive placement validity study will help you to translate a student's test score into the probability that a student will succeed in a specific course at your institution.
Making appropriate course placements for students benefits the institution and the individual student. Appropriate course placement ensures that valuable course time, student time, and other resources are not wasted because students are not prepared to benefit from a course, or because their level of preparation is such that they would benefit from a higher-level course.
ACES™ provides reports that can be shared with policymakers, complete with documentation that supports placement decisions and policies at your institution.
How do I design a predictive placement validity study using nationally administered SAT™ and SAT Subject Test™ scores, or ACCUPLACER® scores?
There are several decisions you will need to make when designing a predictive placement study. The information below will help you design your predictive placement study.
Choosing a criterion
The criterion used for your placement validity study will be used to determine whether or not students are correctly placed. Determining whether or not a test is valid for placement purposes depends on choosing the correct criterion against which to evaluate the test. To choose your criterion for a predictive placement validity study:
- Decide into which course you will place your students (rather than the course out of which you will place students).
ACES uses end-of-course grades as the criterion in placement validity studies. All the problems of GPA reliability as a criterion apply as well to the criterion of course grade. Use of only one grade from one instructor per student increases the problems. The course grade, however, is often the only available or feasible criterion.
A more reliable criterion than course grades, if available, would be a well-designed common departmental examination, graded centrally without indication of a student's instructor or section. This increases the chance that the study results will be useful, regardless of the instructor to which future students are assigned.
Selecting predictor variables
For both predictive and concurrent placement validity studies, you are predicting success on the criterion (end-of-course grade) using the predictor variables. In a predictive placement validity study, the predictor information is typically gathered at some point before a student enters the course of interest. It is important that the predictor is current and that little or no instructional intervention has occurred between the predictor and the start of the course.
In addition to SAT, SAT Subject Tests, and ACCUPLACER, other predictors that may be useful are local tests, high school subject grades, and high school GPA or rank. ACES can be used to help validate any of these predictors or to determine their best combination.
The specific predictors you choose should first pass content and face validity examinations. In addition, you will want to think through the availability and cost of the predictors you might choose. It may be that a test with several open-ended questions matches your course material, is deemed appropriate by your colleagues, and does a wonderful job of predicting success in your course. However, the cost of such a test may be more than your department or the students are willing to pay, or the test may take so long to score that decisions cannot be made in time for class assignments.
Whom to include in the study
General guidelines
To be included in the sample for a validity study, students must have scores on the predictor variables and on the criterion. To predict the probability of success of future students, data derived from the sample of students are used to generate the validity study results. Predictions are more likely to be accurate if the sample of students used for the study is similar to the group of students whose success you want to predict, both in ways that are and are not measured by the predictors. Characteristics of the population, such as gender, ethnic/racial makeup, and age, should be represented in the sample.
For the study to be accurate, the predictor and criterion need to be collected from all students at the same point in the course sequence. Many users want to increase their sample size by including students who took the predictor, were then given a remedial course, and then took the regular course. Adding these students to the data containing students who took the predictor and then immediately took the regular course is like mixing apples and oranges. The intervention of the remedial course changes the relationship between the predictor and the criterion, making the two sets of data incompatible. You will get an answer from ACES, but it won't correctly reflect the relationship between the predictor score and either the remedial or the regular course. For assistance in analyzing the validity of a predictor for a sequence of courses, please contact the ACES staff at ACES@info.collegeboard.org.
Specific Guidelines
Students with Pass/Fail grades
In order to effectively evaluate how well a predictor is working, ACES requires a range of at least four points on a scale for the criterion. Students with Pass/Fail grades should not be included in the study.
Students with a Fail, Incomplete, or Withdraw grade
Generally, we have found that at most institutions failing grades, such as F or E, are calculated into the students' grade point average and are equivalent to zero grade/quality points; and Incomplete or Withdraw grades, such as I or W, are not incorporated into the GPA calculation. In these cases, students with failing grades should be included in your placement validity study, while students with Incomplete or Withdraw grades should not.
These are only given as general guidelines. If you have a large number of students who failed the course you wish to study for any reason other than doing poorly on the tests and/or class work (such as, dropping the class or leaving the school without officially withdrawing), or if your Incomplete or Withdraw grades are calculated into the GPA, please contact the ACES staff at aces@info.collegeboard.org or by calling (609) 921-9000 to discuss your specific situation.
Number of students in the study
The minimum number of students required for a study depends on the number of predictors used. If 1 to 3 predictors are used, a minimum of 30 students are required; for 4 predictors, a minimum of 40 students are required; and for 5 predictors, a minimum of 50 students are required.
Student records used in the analysis
National SAT™ or SAT Subject Test™ data
For a placement study using the nationally administered SAT or SAT Subject Tests, ACES begins by matching your students to the national SAT® database based on name, social security number, date of birth, and gender. If any of this information is missing, ACES will try to match the student with our database based on the available information, but we cannot guarantee that we will be able to find the student's information. If a match cannot be found, the student will not be included in your predictive placement study.
How to collect data
The ideal way to obtain data for a placement validity study would be to assign students randomly to the alternative learning situations (different courses, sections, levels, etc.) so that students with a full range of academic ability are in each learning situation. However, since this is rarely feasible, a study using the chosen test, without any randomization, can still provide useful information.
For a predictive placement validity study, information from the tests or measures that are used as predictive variables is collected prior to the student entering the course of interest. The time between the testing of the student and the entrance into the course should not be lengthy. There should also not be any instructional intervention between the predictive test or measure and the start of the course.
How to submit data
For a predictive placement study the college may submit their files of student data as one merged file or two separate files. If two files are sent, both must be in a horizontal format. Each file must contain the key student identifying fields of student name, social security number, date of birth, and gender. The student data must be in the same format for both files, including the layout of the identifying fields: social security number, birth date, gender designation, and the upper/lowercase format for the name. The ACES data submission form provides a space for you to identify how the student names are formatted in your files. The first file must contain the test scores, if they are being provided by your institution; whereas, the second file will consist of the course data. Note that social security numbers are not required for ACCUPLACER placement validity studies.
If two files are submitted, ACES begins by matching the data in the two files. If students are found in one file but not the other, these students will not be included in your study.
You can also request an ACES placement validity study (using SAT™ scores as predictors) without submitting a new data file when you have already submitted an admission validity study data file (using SAT scores as predictors) that includes students’ coursework information.
You can take advantage of this efficient option provided that the course(s) you are interested in examining has at least 30 students with valid grades in that course. (Note that it is often necessary to have at least 50 students in your sample, depending on the number of predictors chosen, as the results of your analyses will be more stable.) The ACES placement validity study request web page will prompt you for your admission study request number and password in order to access your previously submitted data file. If you have misplaced this information, please contact aces@info.collegeboard.org.
Format for data submission
Data may be submitted as a fixed-length ASCII file, a comma-delimited ASCII file, a tab-delimited ASCII file, a Microsoft ACCESS® database, a Microsoft EXCEL® workbook, a SAS® transport file, or an SPSS® portable file.
What does an ACES™ placement validity report look like?
View a sample placement validity report . (.pdf/889K)
Requires Adobe Reader (latest version recommended).