Perspectives on Methodological Issues

Mark Wilson; Isaac Bejar; Kathleen Scalise; Jonathan Templin; Dylan Wiliam; David Torres Irribarra

doi:10.1007/978-94-007-2324-5_3

In this chapter the authors have surveyed the methodological perspectives seen as important for assessing twenty-first century skills. Some of those issues are specific to twenty-first century skills, but the majority would apply more generally to the assessment of other psychological and educational variables. The narrative of the paper initially follows the logic of assessment development, commencing by defining constructs to be assessed, designing tasks that can be used to generate informative student responses, coding/valuing of those responses, delivering the tasks and gathering the responses, and modeling the responses in accordance with the constructs. The paper continues with a survey of the strands of validity evidence that need to be established, and a discussion of specific issues that are prominent in this context, such as the need to resolve issues of generality versus contextual specificity; the relationships of classroom to large-scale assessments; and the possible roles for technological advances in assessing these skills. There is also a brief segment discussing some issues that arise with respect to specific types of variables involved in the assessment of twenty-first century skills. The chapter concludes with a listing of particular challenges that are regarded as being prominent at the time of writing. There is an annexure that describes specific approaches to assessment design that are useful in the development of new assessments. Perhaps one of the most important, yet often overlooked, choices in assessment is how results are to be presented to various types of stakeholders. This is of prime importance since decisions that will influence the future learning of test takers are based on these results. Reflecting on the kinds of assessment reports that we want to provide is an excellent way to start thinking about the challenges that we face in designing assessment structures to support the development of twenty-first century skills. There have been several efforts to create lists of such skills indeed, a companion paper provides a perspective on a range of these,(1) some examples being: creativity and innovation, collaboration (teamwork), and information literacy. Why are assessment reports a good starting point? Because they encourage us to think about the topics that we want to assess, invite us to consider what kind of inferences we want to promote to users, and lead us to ponder what kind of evidence we should deem appropriate to support those inferences. The kinds of reports that we aspire to provide will be directly useful in enhancing instruction by targeting the teaching of the skills being assessed. Ideally, we want these reports to provide timely and easily interpretable feedback to a wide variety of users, including students and teachers, parents and principals, administrative authorities, and the general public. Finally, we want these reports to be valid and reliable by adhering to high technical standards in the development of the assessments and the analysis of the data. A brief look at some of these topics leads to questions that need to be addressed. A few of the issues we face are: The selection of the constructs to be evaluated: Are these skills defined as domain-general or closely associated with specific contexts or disciplines? The age span of the skills: Will they be confined to K I 2, higher education, or beyond? The level of analysis at which we want to provide feedback: for individuals, teams, classes, or large groups? The question of the universality or cultural specificity of the skills. The answers to these and other questions will shape decisions about the characterization of the constructs to be assessed, the kinds of instruments that will be developed, and the level of information that will be gathered. Ultimately, these decisions will delineate the available evidence and so will constrain the kinds of inferences that can be supported and communicated to users. It is for this reason that it is extremely important to ensure that the development of our assessments is guided by the kinds of inferences that we want to encourage. In this chapter, we present an overview of the assessment design process. The first section addresses the role of evidentiary reasoning, as the starting point of a sound assessment. Sections Two through Six review the different steps involved in the development of an assessment, respectively: (a) defining the constructs to be measured, (b) creating the tasks that will be used to elicit responses and performances, (c) assigning values (codes or scores) to the student responses to these tasks, (d) gathering and delivering the responses, and (e) the modeling and analysis of those responses. Section Seven summarizes the various elements involved in constructing a validity argument to support the claims that will be based on the collected data. Section Eight discusses three general issues that need to be addressed in the design of assessments for twenty-first century skills, namely, the relation between content and process, the interactions between classroom-based and large-scale assessments, and finally, the opportunities that technology offers in the construction of assessments. Section Nine reviews examples of measures that can help visualize potential forms of assessments. A final section summarizes the issues and open challenges raised in the previous sections.

Perspectives on Methodological Issues

View Online

Abstract

Details

Metrics