 |
Dr. James Popham
Minnetonka Summer Institute
June 26, 2012
|
Presentation hand-outs:
Determining a high stakes Test's instructional sensitivity
Right Task. Wrong Tools: The Flawed Appraisal of America's Teachers
Presentation Slides
Notes from discussion:
What is the students learning progression?
How many sub-skills will a student need to master to reach your overall unit target?
FA is more about teaching than testing. It is an instruction enhancing process.
—-----------------------------
Chapter 1:
Formative assesment is a carefully planned process (carefully planned steps) aimed at enhancing student learning.
Applications of FA (when to use it):
Make an immediate instructional adjustment
-gather data, analyze it real time and adjust instruction appropriately
- could be a quick quiz or student reported level of understanding
*planning includes a prediction of what students might get wrong - this is made easier by the fact that student mistakes are not random and the more you know what mistakes they might make the better your planning and instruction will be.
Make a near future instructional adjustment
- collect data and analyze at end of class to inform the next day or few days of instruction. Need to be aware of sub-sets within learning progression. Want students to support arguments with evidence from text? Better teach them to paraphrase. How do you paraphrase? Teach them to focus on vocab, re-structuring a sentence and determining the essential meaning of the sentence - can't restructure? Teach them parts of speech! The learning progression must adapt to what the evidence in their work suggests they need.
For last chance instructional adjustments
-before the summative assessment or unit end - discover the gaps in learning by using a "dress rehearsal exam"
Student makes a learning tactical adjustment
-Teachers foster and promote this skill. Students who reflect on how they learn best, or make adjustments to how they process and retain information are active learners. Students need to connect the outcome of their achievement with what they put into it!
To promote a classroom climate shift
-see learning as possible for all students (this kind of classroom isn't competitive)
-students bear significant responsibility for their own learning and the learning of their classmates
-change in purpose of assessments from of learning to for learning -- Vast majority of tests are not graded. Students make tactical adjustments based on feedback and teachers inform future instruction
Much thinking of formative assessment takes place before instruction
Question for reflecting (QFR) if you had to come up with your own brief but accurate explanation of formative assessment (suitable for non educator audience) how would you describe it?
------------------------------
Chapter 2: learning progressions: blueprints for the formative assessment process
Establish learning target for a unit and then determine the learning progression (LP). the LP is the blue print and the teacher is the architect of that blueprint.
If A ship w/out a rudder is rudderless then a formative assesment without a learning progression is plan-less.
LP defined:
A sequenced set of sub-skills or bodies of enabling knowledge that, it is thought, students must master en route a more remote target or curricular aim.
enabling knowledge - enabling knowledge - sub-skill- target curriculum aim
The further students are away from reaching the curricular aim the more enabling knowledge and sub-skills they'll need (the longer the learning progression).
Use FA to determine student mastery of each building block in the LP
-at minimum, FA is used to assess mastery before moving on to next building block- collecting this evidence helps planning
5 choices to make when developing a learning progression:
1. Which curricular aims? - what curricular outcomes are worthy of a learning progression? - must choose higher cognitive skills (these take more than a class session or two to teach) Is the skill applicable for future learning? Is mastery of the skill going to help them on a high stakes assessment?
2. What the grain size of the building blocks will be
The larger the grain the less # of FAs and planning - consider the time frame you have to develop student mastery of a target.
3. The # of building blocks in the progression
10 is too ambitious. Master CAST (4 building blocks to creating a well written thesis) MASTER VEST (4 building blocks to mastering paraphrasing) Both CAST and VEST are sub-sets of the curricular aim of using evidence from text to support argument in writing.
4. What the building blocks should be
Must think how a student's mastery Of the target aim will be measured at conclusion of instruction and then engage in a rigorous backward analysis
"what must a student know or be able to do in order to master this aim?" that question leads an architect to the building block. They then Repeat the question to determine how they'll master the building block.
5. How the building blocks will be sequenced
QFR - which of your current curricular aims seem like good candidates for formative assessment and why? Take a look at your plans for a unit (or units) and see if you can map out a learning progression for it.
------------------------------
Chapter 3
Immediate Instructional adjustments based on assessed performance
Essence of this type of FA is the teacher's making an immediate instructional adjustment.
Gathering data through conventional kinds of classroom assessment
Making along-the-way corrections as quickly as possible
Adjustment necessary when:
Students don't get it
Students are misunderstanding
Students have already reached level of understanding the lesson is aiming for
Preparation imperative!
*need to identify when in lesson to assess student understanding and what the assessment should look like -- the data generated must support immediate and accurate diagnosis of student learning - Then need to decide what will happen instructionally depending on the nature of the results
*5 key choice points
*What kind of assessment tool to use
Gathering evidence in just a few questions and/or student responses
Can use clickers...
1. Thumbs up/thumbs down - questions must assess understanding of content rather than simply asking students if they get it or if they feel they're ready to move on. These questions should be scripted.
2. letter card responses - usually card stalk or index with A-E printed on them. Some architects also provide ? Card so students can express uncertainty. (don't see why this can't be done with wipe boards). Must have distracters that signify a particular kind of confusion -- but also need options that convey subtle differences in "correctness"
3. Whiteboard responses - this is for constructed responses - answers must be short because teachers need to quickly gather visual data
*When to collect assessment evidence - must collect evidence of learning on each building block. Must ask 2 questions:
When would it make the most sense to make an instructional adjustment?
What effect would pausing to collect assessment evidence have on the planned instructional sequence
This is difficult because immediate adjustment formative assessment takes extensive planning - you have to prepare for the unknown. Doubling or tripling preparation time will lead to teachers doing less immediate formative assessment.
*How many items to include in the assessment
This is dependent upon the significance of the building block toward achieving mastery. All building blocks are important but some are more crucial to achieve absolute mastery than others.
An architect might use a well crafted single question to determine mastery of a simple building block but need multiple questions for a more complex sub-set of skills.
Teachers must ask "how much data will I need to be comfortable that a student either has or had not achieved mastery."
*When to make an Instructional adjustment
This needs to be set in advance to ward off the tendency to conclude student mastery is "good enough."
Teachers must set a required level of understanding and express that level of expectation to the students.
Examples of a pre-set level of understanding:
75% of Students will get a correct thumbs up/thumbs down on 5 of 6 questions or an instructional adjustment will be triggered.
Note: really important sub-skills or whole building blocks require a more stringent level of understanding for individuals and whole group percentages.
*What kind of instructional adjustment to make?
most difficult decision to make is what to do after you realize an adjustment needs to take place.
Popham recommends making a template with all potential student confusion
This can be found in figure 3.2 on P. 58
QFR - If you were recommending one of three assessment procedures (thumbs, letter card, wipe board) to a teacher new to FA, which one would you suggest and why?
A teacher's friend, a teacher's foe?
Educational assessment - friend or foe? Thats up to us!
The curse:
"May you live in an interesting time." In an interesting time things change and the habits and practices that worked well for us don't anymore.
So many changes in education - widely held view that schools are in much worse shape than in 1960s. American schools were the highest of social achievement - why have we fallen?
Chief issue - profound pervasive problem with assessment literacy.
We've made pivotal mistakes that have eroded the caliber and perception of our educational programming
5 mistakes currently being made in the field of Education:
1. Make big educational/academic decisions based on supposed numerical accuracy of Education tests.
2. We almost never measure student attitudes, interests, or values - that which is assessed promotes instructional changes! We can measure students pleasure in reading before and after! We will then do things to enhance their pleasure throughout the year.
3. we teach and test and require student mastery of too many curricular aims (Schmoker like). When you try to measure to many things you become overwhelmed. Too many targets become no targets at all.
4. spend too much money on interim tests (NWEA). There are zero evidence that they work.
5. Worst mistake - we have allowed the success of American educators to be determined by educational tests that are incapable of accomplishing this profoundly important measurement mission.
The tests we devised from NCLB produce comparative score interpretations.
Popham used to push behavioral objectives - posted objectives to the student describing what they'll be able to do post-instruction. He wanted to stamp out non-behavioral objectives.
start with a smaller number of more salient targets
How can we remedy these 5 mistakes?
Too many educators are assessment illiterate
Assessment literacy - consists of an individual's understandings of the fundamental assessment concepts and procedures deemed likely to influence educational decisions.
Assessment literacy is not:
Not off-putting quantitative - most of the significant understandings needed by an assessment literate person aren't mind-numbing numerical.
Assessment literate people:
Know about measurement accuracy of standardized tests
Have comprehension of what's needed for a test to be diagnostic (need to gain instructional insight from the test to be diagnostic)
We just need to understand what the tests we are giving can or are intended to do for us.
We are apt to do some really harmful things regarding teacher evaluation
Instructional sensitivity - degree to which students' performances on a test accurately reflect the quality of instruction specifically provided to promote students' mastery of what is being assessed.
Validity refers to the accuracy of a test-based inference. Tests are not valid or invalid themselves.
Can't infer that a student with a high score was better taught than a student with a lower score. Can't use student performance to evaluate teachers -
Feds want teacher evaluation to make student growth and test scores as evidence of teacher efficacy. We can't make this invalid inference. The tests used to judge the caliber of instruction aren't designed to be used in that way.
What test items might be instruction ally insensitive
1. Alignment leniency - need rigorously congruent test items
2. Excessive easiness - can this item distinguish poorly taught and well taught students?
3. Excessively difficult - no matter how well it's taught, students can't get it right
4. Confusion-engendering item flaws
5. Test item linked to SES (if an item gives a meaningful advantage to higher SES kids, then the item tends to measure what students bring to school rather than how well they were taught).
Tests that are not instructionally sensitive should not be used to evaluate teachers. What tests that MV gives are instructionally insensitive?
If a test is being used to evaluate teachers (&principals) that isn't instructionally sensitive we should create a rumpus. We should be outraged.
Mastering Assessment - Pearson (2011)
Everything school leaders need to know about assessment(Corwin 2010)
Classroom Assessment: what teachers need to know (2011) Pearson
What it is/isn't
Formative assesment is a carefully planned process (carefully planned steps) in which assessment elicited evidence of students' status is used by teachers to adjust their ongoing instructional procedures or by students to adjust their current learning tactics.
It is a process by which we use assessment outcomes to inform instruction
A formative assessment is not a test. It is not a benchmark or periodic test administered every few months
It is not the unplanned, serendipitous use of student cues to adjust teaching!
What it can do/what it can't do
It improves student learning
Follow this paradigm:
Have an objective, teach, then measure (assess) the extent to which students met that objective and adjust instruction accordingly
FA can't sufficiently raise scores on instructionally insensitive state accountability tests.
Should we have School improvement goals (SMART goals) on MCA tests if we know they're instructionally insensitive?
4 levels
Teacher adjustments (decision)
Student learning-tactic adjustments
Classroom shifts
Strategies
5 applications of FA:
Immediate instructional adjustment
gather data, analyze it real time and adjust instruction appropriately
- could be a quick quiz or student reported level of understanding
*planning includes a prediction of what students might get wrong - this is made easier by the fact that student mistakes are not random and the more you know what mistakes they might make the better your planning and instruction will be.
Near future instructional adjustments
- collect data and analyze at end of class to inform the next day or few days of instruction. Need to be aware of sub-sets within learning progression. Want students to support arguments with evidence from text? Better teach them to paraphrase. How do you paraphrase? Teach them to focus on vocab, re-structuring a sentence and determining the essential meaning of the sentence - can't restructure? Teach them parts of speech! The learning progression must adapt to what the evidence in their work suggests they need.
Last chance instructional adjustment
-before the summative assessment or unit end - discover the gaps in learning by using a "dress rehearsal exam"
Student makes a learning tactical adjustment
-Teachers foster and promote this skill. Students who reflect on how they learn best, or make adjustments to how they process and retain information are active learners. Students need to connect the outcome of their achievement with what they put into it!
Classroom climate shifts
see learning as possible for all students (this kind of classroom isn't competitive)
-students bear significant responsibility for their own learning and the learning of their classmates
-change in purpose of assessments from of learning to for learning -- Vast majority of tests are not graded. Students make tactical adjustments based on feedback and teachers inform future instruction
Dylan William's 5 strategies of FA
1. Clarify and share learning intentions
2. Engineer effective classroom discussion, activities and learning takes that elicit evidence of learning
Learning progression - sub-skills and bodies of enabling knowledge
Enabling knowledge - enabling knowledge - sub-skill A, - target curricular aim.
Learning progressions: blueprints for the formative assessment process
Establish learning target for a unit and then determine the learning progression (LP). the LP is the blue print and the teacher is the architect of that blueprint.
If A ship w/out a rudder is rudderless then a formative assesment without a learning progression is plan-less.
LP defined:
A sequenced set of sub-skills or bodies of enabling knowledge that, it is thought, students must master en route a more remote target or curricular aim.
enabling knowledge - enabling knowledge - sub-skill- target curriculum aim
The further students are away from reaching the curricular aim the more enabling knowledge and sub-skills they'll need (the longer the learning progression).
Use FA to determine student mastery of each building block in the LP
-at minimum, FA is used to assess mastery before moving on to next building block- collecting this evidence helps planning
5 choices to make when developing a learning progression:
1. Which curricular aims? - what curricular outcomes are worthy of a learning progression? - must choose higher cognitive skills (these take more than a class session or two to teach) Is the skill applicable for future learning? Is mastery of the skill going to help them on a high stakes assessment?
2. What the grain size of the building blocks will be
The larger the grain the less # of FAs and planning - consider the time frame you have to develop student mastery of a target.
3. The # of building blocks in the progression
10 is too ambitious. Master CAST (4 building blocks to creating a well written thesis) MASTER VEST (4 building blocks to mastering paraphrasing) Both CAST and VEST are sub-sets of the curricular aim of using evidence from text to support argument in writing.
4. What the building blocks should be
Must think how a student's mastery Of the target aim will be measured at conclusion of instruction and then engage in a rigorous backward analysis
"what must a student know or be able to do in order to master this aim?" that question leads an architect to the building block. They then Repeat the question to determine how they'll master the building block.
5. How the building blocks will be sequenced
Transformative Assessment in Action
book notes:
1:Ask yourself what is pivotal to be taught and mastered and in what sequence?What is the students learning progression?
How many sub-skills will a student need to master to reach your overall unit target?
—-----------------------------
Chapter 1:
Formative assesment is a carefully planned process (carefully planned steps) aimed at enhancing student learning.
Applications of FA (when to use it):
Make an immediate instructional adjustment
-gather data, analyze it real time and adjust instruction appropriately
- could be a quick quiz or student reported level of understanding
*planning includes a prediction of what students might get wrong - this is made easier by the fact that student mistakes are not random and the more you know what mistakes they might make the better your planning and instruction will be.
Make a near future instructional adjustment
- collect data and analyze at end of class to inform the next day or few days of instruction. Need to be aware of sub-sets within learning progression. Want students to support arguments with evidence from text? Better teach them to paraphrase. How do you paraphrase? Teach them to focus on vocab, re-structuring a sentence and determining the essential meaning of the sentence - can't restructure? Teach them parts of speech! The learning progression must adapt to what the evidence in their work suggests they need.
For last chance instructional adjustments
-before the summative assessment or unit end - discover the gaps in learning by using a "dress rehearsal exam"
Student makes a learning tactical adjustment
-Teachers foster and promote this skill. Students who reflect on how they learn best, or make adjustments to how they process and retain information are active learners. Students need to connect the outcome of their achievement with what they put into it!
To promote a classroom climate shift
-see learning as possible for all students (this kind of classroom isn't competitive)
-students bear significant responsibility for their own learning and the learning of their classmates
-change in purpose of assessments from of learning to for learning -- Vast majority of tests are not graded. Students make tactical adjustments based on feedback and teachers inform future instruction
Much thinking of formative assessment takes place before instruction
Question for reflecting (QFR) if you had to come up with your own brief but accurate explanation of formative assessment (suitable for non educator audience) how would you describe it?
------------------------------
Chapter 2: learning progressions: blueprints for the formative assessment process
Establish learning target for a unit and then determine the learning progression (LP). the LP is the blue print and the teacher is the architect of that blueprint.
If A ship w/out a rudder is rudderless then a formative assesment without a learning progression is plan-less.
LP defined:
A sequenced set of sub-skills or bodies of enabling knowledge that, it is thought, students must master en route a more remote target or curricular aim.
enabling knowledge - enabling knowledge - sub-skill- target curriculum aim
The further students are away from reaching the curricular aim the more enabling knowledge and sub-skills they'll need (the longer the learning progression).
Use FA to determine student mastery of each building block in the LP
-at minimum, FA is used to assess mastery before moving on to next building block- collecting this evidence helps planning
5 choices to make when developing a learning progression:
1. Which curricular aims? - what curricular outcomes are worthy of a learning progression? - must choose higher cognitive skills (these take more than a class session or two to teach) Is the skill applicable for future learning? Is mastery of the skill going to help them on a high stakes assessment?
2. What the grain size of the building blocks will be
The larger the grain the less # of FAs and planning - consider the time frame you have to develop student mastery of a target.
3. The # of building blocks in the progression
10 is too ambitious. Master CAST (4 building blocks to creating a well written thesis) MASTER VEST (4 building blocks to mastering paraphrasing) Both CAST and VEST are sub-sets of the curricular aim of using evidence from text to support argument in writing.
4. What the building blocks should be
Must think how a student's mastery Of the target aim will be measured at conclusion of instruction and then engage in a rigorous backward analysis
"what must a student know or be able to do in order to master this aim?" that question leads an architect to the building block. They then Repeat the question to determine how they'll master the building block.
5. How the building blocks will be sequenced
QFR - which of your current curricular aims seem like good candidates for formative assessment and why? Take a look at your plans for a unit (or units) and see if you can map out a learning progression for it.
------------------------------
Chapter 3
Immediate Instructional adjustments based on assessed performance
Essence of this type of FA is the teacher's making an immediate instructional adjustment.
Gathering data through conventional kinds of classroom assessment
Making along-the-way corrections as quickly as possible
Adjustment necessary when:
Students don't get it
Students are misunderstanding
Students have already reached level of understanding the lesson is aiming for
Preparation imperative!
*need to identify when in lesson to assess student understanding and what the assessment should look like -- the data generated must support immediate and accurate diagnosis of student learning - Then need to decide what will happen instructionally depending on the nature of the results
*5 key choice points
*What kind of assessment tool to use
Gathering evidence in just a few questions and/or student responses
Can use clickers...
1. Thumbs up/thumbs down - questions must assess understanding of content rather than simply asking students if they get it or if they feel they're ready to move on. These questions should be scripted.
2. letter card responses - usually card stalk or index with A-E printed on them. Some architects also provide ? Card so students can express uncertainty. (don't see why this can't be done with wipe boards). Must have distracters that signify a particular kind of confusion -- but also need options that convey subtle differences in "correctness"
3. Whiteboard responses - this is for constructed responses - answers must be short because teachers need to quickly gather visual data
*When to collect assessment evidence - must collect evidence of learning on each building block. Must ask 2 questions:
When would it make the most sense to make an instructional adjustment?
What effect would pausing to collect assessment evidence have on the planned instructional sequence
This is difficult because immediate adjustment formative assessment takes extensive planning - you have to prepare for the unknown. Doubling or tripling preparation time will lead to teachers doing less immediate formative assessment.
*How many items to include in the assessment
This is dependent upon the significance of the building block toward achieving mastery. All building blocks are important but some are more crucial to achieve absolute mastery than others.
An architect might use a well crafted single question to determine mastery of a simple building block but need multiple questions for a more complex sub-set of skills.
Teachers must ask "how much data will I need to be comfortable that a student either has or had not achieved mastery."
*When to make an Instructional adjustment
This needs to be set in advance to ward off the tendency to conclude student mastery is "good enough."
Teachers must set a required level of understanding and express that level of expectation to the students.
Examples of a pre-set level of understanding:
75% of Students will get a correct thumbs up/thumbs down on 5 of 6 questions or an instructional adjustment will be triggered.
Note: really important sub-skills or whole building blocks require a more stringent level of understanding for individuals and whole group percentages.
*What kind of instructional adjustment to make?
most difficult decision to make is what to do after you realize an adjustment needs to take place.
Popham recommends making a template with all potential student confusion
This can be found in figure 3.2 on P. 58
QFR - If you were recommending one of three assessment procedures (thumbs, letter card, wipe board) to a teacher new to FA, which one would you suggest and why?
No comments:
Post a Comment