Assessing Character: A Meeting With ETS' Patrick Kyllonen
Here at the Think Tank we’ve been combing the country far and wide to discover the best minds and the sharpest thinking in non-cognitive assessment. We are learning that some of the very best work in this field is in our own backyard, on the sprawling Pcampus of ETS (Educational Testing Services).
In previous posts, we’ve reported on the ETS research of Rich Roberts, architect of the Mission Skills Assessment (MSA), which is being used in a growing number of independent middle schools. Last month, SSATB Executive Director and Think Tank member Heather Hoerle visited with Patrick Kyllonen, Director of ETS’ New Constructs Center. Kyllonen is probably most recognized for the development of the PPI, Personal Potential Index, a system for assessing a graduate school applicant’s suitability for graduate study in the following areas: Knowledge, Analytical Skills, Communication Skills, Teamwork, Motivation, Self-Organization, and Professionalism & Maturity. His white paper about the PPI is free, easily accessible, and very informative.
Kyllonen shared with Heather his delight that character and non-cognitive assessment is becoming so much more prominent. This is something, he explained, that has been his life’s work for several decades now, and something he is eager to assist organizations with in any way that he can. He emphasized that beyond selection, organizations should be using these tools for evaluation of their own success: are you meeting objectives in developing your people? Assess as they come in, assess along the way, assess as they depart, and take note: are you adding value?
Currently, Kyllonen and his colleagues are working with a group of Fortune 100 companies seeking to select college graduates, as well the Los Angeles Unified School District (LAUSD). LAUSD is working with a NCLB waiver to develop its own assessment focused not only on the common core academic standards, but also on the district’s school climate and students’ non-cognitive attributes. He had recently returned from a meeting with L.A. administrators and happily shared pictures with Heather from his iPhone depicting the process that was used to hone in on the non-cognitive attributes to study. Would anyone be surprised to hear that they simply asked administrators to place a sticker next to those attributes they are most interested in?
Because a non-cognitive “construct” is another kind of skill, an attribute such as motivation or creativity can be put on a scale and psychometrically studied and quantified like any other “cognitive construct,” such as the ability to solve algebraic equations. Because rich psychological research already exists in areas like motivation or empathy, Kyllonen emphasized that an organization wishing to assess non-cognitive attributes simply needs to define the construct – what’s important and/or what they want – and to determine the method of assessment – usually some combination of self-ratings and ratings by someone else.
The existing psychological research is used to define, for example, what “highly skilled in empathy” means. In fact, Kyllonen referred to the work of Pellegrino and Hilton, published by the National Research Council, on 21stCentury competency clusters – Cognitive Domain, Interpersonal Domain, and Intrapersonal Domain (which has been referenced in this blog).
Like Rich Roberts, Kyllonen also described innovative item types that are being developed in this area – forced choice, anchoring vignettes, and situational judgment – and emphasized the need to get around “fake-ability.” According to Kyllonen, as designed these items allow for scalability in much the same way that a math question on the SAT does. Of course, sophisticated Item Response Theory statistics also help when dealing with the “ipsative” nature of these types of items.
For Kyllonen, the true stakes of his work are enormous. If we can use better assessment of the skills and attributes which make the biggest impact on success in the workforce and in life, we can better educate and train people, and improve their lives considerably. Like others in this field, he is inspired and informed by the work of Nobel Prize winning economist James Heckman,Ph.D. Heckman, a giant in his field, has many accomplishments to his credit, among them the trailblazing research on the power of early childhood education and its impact especially on non-cognitive skill development. To quote Heckman, “There is hard evidence that non-cognitive — or character — skills matter greatly. And, there is widespread evidence that failing to systematically develop, measure and reward positive character traits is failing America — in schools and in the workforce.”
-----
In the News
“They’re Watching You at Work,” an article published in The Atlantic last month, is simultaneously disturbing, creepy, challenging, encouraging, and enlightening. It is useful to be reminded that there is nothing new about using surveys and the like to evaluate employee and candidate traits and select among them; it is also interesting to see that we have better research than ever before about how to evaluate these things and that we can use them wisely to strengthen our organizations and improve the experience of our “subjects,” be they employees or students. The article argues that decision-making with improved data doesn’t, in the long run, diminish the significance of human judgment—it just improves it.
