By Jonathan E. Martin, Principal, JonathanEMartin Ed. Services • @JonathanEMartin
From The Yield, Winter 2014

32 Schools Join The Enrollment Management Association in Pilot Study on Non-Cognitive Assessment

Independent school admission directors might someday refer to it as “The Homewood Summit.” Perhaps that is being boastful, but most participants felt, about the two days working together at the Princeton Homewood Suites in late September, that they were participating in something that might become very important—something which will genuinely mark a shift in the way we manage the admission process, communicate the value of our schools, and shape our student bodies. “I left thinking I could now tell my colleagues we had glimpsed the future, and it was full of possibilities that would ad- vance our profession,” said Tom Sheppard, Director of Admission at the Lawrenceville School.

The project, launched this fall by a unanimous resolution of the The Enrollment Management Association Board of Directors upon the recommendation of the Think Tank on the Future of Assessment, is to build a new assessment, a supplement to the current SSAT, which can report on the character qualities, or “non-cognitive” attributes, of individual student applicants. No such tool currently exists in K-12 education.

The group of educators who came together for the summit from 32 schools—those schools participating in the Think Tank, recognized for innovative practices, and/or having expressed an interest in this work—represented an intentionally wide diversity of school types: boarding and day; K-12, 9-12, K-8, and 6-12; coed and single sex (all boys and all girls schools); schools with varying kinds of academic programs; U.S. and international, including schools from Canada and Shanghai; and schools from a wide array of U.S. regions, from New Hampshire to Hawaii, from Texas to Washington state. Doug Price from Middlesex School expressed his appreciation for this diversified collaboration by asserting, "Although the participants came from a diverse group of schools, it was apparent that we shared the belief that current assessment tools are limited in predictive value, and the work we had gathered to do could only improve our ability to assess students.”

Joining the group of 32 school representatives were The Enrollment Management Association senior staff and a team from the Center for Academic and Workforce Readiness and Success at Educational Testing Service (ETS), including Dr. Patrick Kyllonen, Ph.D. Kyllonen, who was featured in both of the Think Tank’s special reports, is an internationally recognized expert on non-cognitive assessment who has worked with OECD/PISA, NAEP, and the College Board.

The term “Forced Choice,” as explained by Dr. Kyllonen and his team, offers considerable advantages in the complicated work of evaluating non-cognitive attributes. Respondents are not asked to rate themselves high or low on individual qualities (“I am responsible in my work;” “I treat others fairly,”) but rather must run through a long list, perhaps 50-100, of paired positive qualities, choosing for example either “I am responsible” or “I treat others fairly.” “Faking” such a “test” is close to impossible. Every item in every choice is socially valued, but the creators can design the test with a wide array of combinations and generate meaningfully valid and reliable results. Another more familiar technique which is fast becoming standard in this field is called the “Situation Judgment Test,” in which respondents are presented an authentic scenario presenting a social or personal challenge, and they are asked to rate various options for how it can best be handled.

Garth Chalmers from the University of Toronto Schools asserted, “Learning about types of assessment for which students cannot be coached was the most interesting aspect of the summit. I think it is a huge challenge for admission offices— that is, to determine whether the student really is a great fit or just well coached through the process.”

Determining exactly which set of non-cognitive attributes to assess was the other key component of the summit—and is an ongoing project for the pilot schools. Working in small groups and then full group discussion, admission directors, school heads, and academic deans drew deeply from their experiences in their schools to discuss and debate the comparative merits of openness, assertiveness, and curiosity, to name a few. The group’s intention is that the set of attributes that will drive this work will be established in the next six to nine months.

Participants in this meeting reported an enthusiasm and a commitment to carry forward this initiative and the broader work of elevating the importance of social and emotional learning in our schools. To quote one admission director from Atlanta, Galloway’s Elizabeth King, “The Enrollment Management Association is well on its way in developing the next generation of assessments that more accurately predict success in our 21st century globally-fused economy. In a few years, the non-cog working group will look back and be proud to say it had a hand in transforming assessments, and consequently, improving admission, and our schools. The Enrollment Management Association and the non-cog group is onto something evolutionary.”