WaPo: New Studies Say AP Works

possibly of interest for HSTS. Mentions article from previous days paper as well.


New Studies Say AP Works
By Jay Mathews
Washington Post Staff Writer
Tuesday, January 30, 2007; 12:44 PM

The College Board releases its annual Advanced Placement Report to the Nation next week. For us AP and IB dorks, it is the equivalent of the State of the Union address. No, delete that. The State of the Union is usually a bore. AP's Report to the Nation is more like the Academy Awards, for a small group of socially awkward fans like me.
For those of you new to this obsession, Advanced Placement and International Baccalaureate are courses and tests given to high school students that are designed to be the equivalent of introductory college courses in about two dozen subjects. They impress selective-college admissions offices. If the students do well on the tests, they can earn college credits and skip introductory courses for more advanced stuff when they get to college.
This year there was a bonus in the Report to the Nation. In the advance copy for reporters, I saw a reference to two new studies of AP in Texas that appeared to break new ground. Since these were reports by independent researchers, I was not breaking the College Board embargo if I sought them out and asked to see their work.
The result was a story I wrote for yesterday's editions of The Post. Here are the first paragraphs:
"In the midst of a national debate over whether Advanced Placement courses place too much pressure on American high school students, a team of Texas researchers have concluded the difficult courses and three-hour exams are worth it.
"In the largest study ever of the impact of AP on college success, which looked at 222,289 students from all backgrounds attending a wide range of Texas universities, the researchers said they found 'strong evidence of benefits to students who participate in both AP courses and exams in terms of higher GPAs, credit hours earned, and four-year graduation rates.'
"A separate University of Texas study of 24,941 students said those who used their AP credits to take more advanced courses in college had better grades in those courses than similar students who took college introductory courses instead of AP in 10 ten different subjects.
" 'Both of these papers are home runs. They definitely settle a lot,' said Joseph Hawkins, an AP expert who is senior study director for the private research firm Westat in Rockville.
"The new studies run counter to an unpublished Harvard and University of Virginia study that casts doubt on the worth of AP science courses, and contradict some critics who say that high school courses, even with an AP label, cannot match the depth of the college introductory courses."
Once these studies are available online, which may take awhile, I will have the links in this column. In the meantime, I want to use the extra space here, cruelly denied me whenever I write a news story for the paper, to identify what I think are the most interesting parts, and to let the experts I consulted have more than the paltry 56 words -- I just counted -- that I had room to quote in the news story.
1. AP students v. non-AP students
Both studies attempted to put this competition on a level playing field by comparing the college success of AP students to those non-AP students (or AP students who did not take the AP exam) who were similar in important ways. The larger study, by University of Texas-Austin researchers Linda Hargrove and Barbara Dodd and Texas Higher Education Coordinating Board researcher Donn Godin, compared students with about the same SAT or ACT scores, or similar family economic circumstances.
But Saul Geiser, the University of California scholar who produced his own massive AP and college success study in 2004, said these comparisons are still a problem.
"The main technical issue that bedevils not only the Texas studies but all research of this kind concerns the question of selection bias: To what extent is AP students' success in college a reflection of the AP program itself (a 'program effect') as opposed to the personal characteristics of the students selected for the program (a 'selection effect')?" he asked. "This question is particularly important in evaluating AP because the program historically has targeted 'better,' academically stronger students who could be expected to perform well in college even without AP.
"Both of the Texas studies attempt to address the problem of selection bias by 'matching' AP and non-AP students on other measures, primarily SAT scores, in order to control for student background characteristics. But neither study controls for a sufficiently robust set of student characteristics to rule out the hypothesis that selection effects account for most of the relationship between AP and college outcomes. The larger study of all Texas public institutions by Hargrove, Godin and Dodd matched AP and non-AP students only on two measures, SAT I scores and participation in free-lunch programs, while the smaller study matched students only on SAT I scores and high-school class rank. As a result, the studies control for far fewer variables than previous researchers have considered, leaving open the possibility, and indeed probability, that systematic differences in student motivation, academic preparation, family background and high-school quality account for much of the observed difference in college outcomes between AP and non-AP students."
2. How AP affects success in the next college course
This is the part of the second Texas study, by University of Texas-Austin researchers Leslie Keng and Barbara Dodd, that attempts to shed light in the so far clueless debate over whether AP and IB students deserve college credit and a chance to skip to a more advanced college course.
Many selective schools allow only students with the highest grade, a 5, on the AP exam to get credit. Some insist that their introductory courses are so special that no AP grade is high enough. But the amount of research they have devoted to proving their point is laughingly small. Harvard made a rule that only 5s will count -- and only if a student is going for sophomore standing -- based on one study of three courses that showed AP students not doing as well as students who first took the college's intro course. A few colleges have data from a similarly small number of courses that show the AP or IB students do better on the advanced course than students who have taken the college's intro course. But most colleges have no data. When you ask them on what they rest their restrictive rules, they shrug and say the faculty wanted it that way.
The Texas study has to be read very carefully, because although it concludes that students who received AP credit in 10 subjects at UT did better in the advanced course than students who first took the intro course, picking the right advanced course for the researchers to gather data from and study was difficult, and in some cases, with some categories of student, the triumph of the AP kids is not so clear.
3. The worthiness of AP science
In my Nov. 28 column, I discussed in detail a yet unpublished study by Harvard researcher Philip M. Sadler and University of Virginia researcher Robert H. Tai. They surveyed a relatively small sample of AP science students at 63 colleges and concluded that their AP experience did not help them much, if at all, in similar college introductory courses. They sadly did not look at AP students who took advanced college courses, so their work can not be directly compared to the UT study, but theirs and the Texas study are going to be the principal pieces of evidence in debates on this issue.
In the second Texas study, Sadler found some of those same patterns among the AP science students.
"I note that in the Dodd and Keng study that non-AP students (taking the prerequisite course in college biology) get better grades than AP-Credit students in the subsequent (Bio 303 ) courses. I would like to see their chemistry data on subsequent course grade, which does not appear to be included. This can be interpreted as AP courses not being the equivalent of taking the university's biology prerequisite to second year biology. . . . This all may differ by subject matter area.
"I also worry about the methodology of matching rank and SAT/ACT scores. It is unclear whether high school rank was done with AP weighting or not, since this will impact rank. I would much rather use actual course grades in relevant subjects and actual SAT/ACT scores in regression models. I remain unconvinced that the study controlled for enough variables to rule out alternative hypotheses to that of AP being responsible for better performance seen in GPA and graduation rates. I think it is best to control for performance and level of high school math, performance in prerequisite courses to AP in HS, parental education, community affluence, etc., if possible."
4. Extra grade points for AP
Geiser shares with me, and most of the AP teachers I know, the view that AP students should be encouraged to take the AP exams. This is sometimes difficult because the seniors who often take AP have their exams scheduled in May, after they have gotten into college, after the weather has turned warm, and just as the season of prom/senior-cut-day/senior-prank-day/senioritis extremis reaches its peak.
The Texas researchers, like Geiser and others who have looked at this, have convincing evidence that it is working hard to understand the material and getting good grades on the AP exams that correlates with college success, not just taking the course. Geiser sees this as an education policy issue of great importance:
"Though the authors of the Texas studies interpret their findings as showing that the rapid expansion of the AP program has not harmed program quality, some of the Texas data would seem to suggest a different conclusion. Among the tables appended at the end of the larger statewide study, data show that the number of students in the 'AP Course Only' category -- students who enrolled in AP courses but failed to take the AP exams -- increased both overall and in each AP subject area studied, without exception, over the four years studied.
"College Board and AP officials will be understandably thrilled, as they should be, with the results of the new Texas studies and the association between AP exam scores and college outcomes that both studies demonstrate. But those officials should also emphasize to their constituents in U.S. schools and colleges what the studies do NOT show. The new studies are consistent with a growing body of evidence that, while mastering the material taught in AP classes and performing well on the AP exams is correlated with later success in college, mere enrollment in AP classes is not. The widespread practice of 'weighting' students' high-school GPAs simply for taking AP courses is not justified by the research evidence and has had perverse consequences for both high schools and colleges."
5. AP vs. Dual Enrollment
Dual enrollment -- a common term for courses taken at local colleges, or conducted by local colleges, for high school students -- takes a major hit in the Keng-Dodd study. The researchers say students who took dual enrollment courses in high school did not do as well in college as those who took AP courses.
This is a very important issue, worthy of another column as soon as dual enrollment's defenders get a chance to read the full report. I have been talking about the relative rigor of AP, IB and dual enrollment with many educators who think Newsweek's America's Best High Schools list should count dual enrollment courses, as it now counts AP and IB exams. The Keng-Dodd data convinces me Newsweek needs a lot more information before it takes that step.
Former U.S. Education Department researcher Cliff Adelman, the guru of college completion data, had these thoughts:
"I am not overly surprised that dual-enrollment courses don't have the same impact as AP. I'll put good money on the table that dual-enrollment courses are capturing a somewhat different population. What would be interesting would be to compare the education histories of students who took dual-enrollment course X at a local community college while in high school with those who entered the community college and took it there, and then to divide the latter group into those who transferred to the 4-year sector and those who didn't."
6. AP and students who speak Spanish at home
I occasionally get e-mails from readers who think it is unfair that Spanish-speaking students are allowed to take the AP Spanish exam designed for English-speaking students struggling to master that language in high school. My standard answer is that being bilingual, as these students are, is a valuable academic skill and they should get some credit for it, just as I would not begrudge the 5 on AP biology credited to a girl whose mother the DNA expert has been letting her help in the lab since she was five.
But there is more interesting data in the Texas AP studies that touches on this, since there were plenty of Spanish speaking AP students in their large sample. Many students -- almost 2,000 in one cohort -- took the AP exam but not the AP course, extremely rare in AP courses other than Spanish.
Chrys Dougherty, director of research at the National Center for Educational Accountability in Austin, noticed something else. In the larger of the two studies, "the correlation between AP exam grades and college outcomes seems to go away for students with lower prior SAT scores. This may well be due to the large number of native Spanish speakers in Texas who earn credit on the AP Spanish exam, but who may not be well prepared in or take AP exams in other subjects, and who thus would populate the statewide group of high AP exam scorers with low SAT scores."
7. What does it all mean? It will take some time to figure that out. There is too much data here to absorb all at once, and we are likely to see an acceleration of research on AP and IB, given their growing importance, that is going to make it hard for even us devotees to keep up.
One of my favorite fellow AP addicts is Hawkins, the senior study director I quoted in my news article yesterday. He is an educational activist who has made a study of AP results among minority students. His view on these numbers has strongly influenced my thinking. He believes that it is not so much what is learned in AP, but the act of struggling with a difficult course that adds the most value.
"So taking AP courses and exams is highly associated with college outcomes down the road," he said, summing up this research. "But I think this happens not just because the kids who take these courses learn stuff and master knowledge (college level stuff), but because these kids also in the process become better learners, students, scholars. They end up mastering the importance of working hard, clearly something that will serve them well in college and later in life."
That seems right to me, but I await the next deluge of data to see if Hawkins can be proved right.