Notes from Data Wise by Kathryn Parker Boudett

Chapter 1: Organizing for Collaborative Work

There are three activities that can support a "data culture" within schools: (a) creating and guiding a data team, (b) enabling collaborative work among faculty, (c) and planning productive meetings.

Data team - having a few people responsible for organizing and preparing the data means that you can dedicate the full faculty's time to discussing the data.

3 Questions to ponder when considering school improvement:

What data do we already have? What initiatives are we already implementing? What is our current approach to improvement?

Three tasks to begin:

1. Create a data inventory (external and internal assessments)

2. Take stock of data organization
3. Develop an inventory of the instructional initiatives currently in place

Are you satisfied with the way you capture the information generated from each of your assessments?


Four Helpful Strategies for Planning Productive Meetings
1. Establish group norms (i.e. no blame, no wrong answers)
2. Use protocols to structure conversations
3. Adopt an improvement process
4. "Lesson plan" for meetings (repackage data results so they can be easily understood)

Chapter 3: Creating a Data Overview

Preparing for a faculty meeting:

1. Decide on the educational questions
2. Reorganize your assessment data (simple is better)
3. Draw attention to critical comparisons
4. Display performance trends

The underlying educational questions should also drive every aspect of the presentation of the assessment data and provide a rationale for why it is important to present the data one way rather than another.


Understanding how students outside your school perform on the same assessment can provide benchmarks against which to compare the performance of your school's students.

In labeling and explaining graphs showing student performance, it is very important to be clear about whether the display illustrates trends on achievement for the same group over time, or whether it illustrates cohort-to-cohort differences over a number of years in the performance of students at the same grade level.

Components of Good Displays

1. Make an explicit and informative title for every figure in which you indicate critical elements of the chart, such as who was assessed, the number of students whose performance is summarized in the figure, what subject specialty, and when.

2. Make clear labels for each axis in a plot, or each row and column in a table.

3. Make sensible use of the space available on the page, with the dimensions, axes, and themes that are most important for the educational discussion being the most dominant in the display.

4. Keep plots uncluttered and free of unnecessary detail, extraneous features, and gratuitous cross-hatching and patterns.

Actively involve teachers with the data by giving them an opportunity to make sense of the data for themselves, encouraging them to ask questions, and offering them a chance to experience and discuss the actual questions on the test.

In reality, student assessment data is neither weak nor powerful. The real value in looking at this kind of data is not that it provides answers, but that it inspires questions.

 

Chapter 4: Digging Into Data

Without an investigation of the data, schools risk misdiagnosing the problem.

There are two main steps when using data to identify the learner-centered problem in your school: looking carefully at a single data source and digging into other data sources.

The first thing to consider is: What questions do you have about the student learning problem, and what data will help answer those questions?

The next consideration is context: What data will be most compelling for the faculty?

Understanding how students arrived at a wrong answer or a poor result is important in knowing how to help them learn to get to the right answer or a good result.

Challenging assumptions is critical for three reasons:

 
1. Assumptions obscure clear understanding by taking the place of evidence
2. Teachers have to believe that students are capable of more than what the data shows
3. Solutions will require change

Starting with data and grounding the conversation in evidence from the data keeps the discussion focused on what we see rather than what we believe.

By triangulating your findings from multiple data sources- that is, by analyzing other data to illuminate, confirm, or dispute what you learned through your initial analysis- you will be able to identify your problem with more accuracy and specificity.

Students are an important and underused source of insight into their own thinking, and having focus groups with students to talk about their thinking can have a positive impact on your efforts to identify a problem underlying low student performance.


Chapter 5: Examining Instruction

Reframe the learning problem as a "problem of practice". It should:

-include learning and teaching
-be specific and fine-grained
-be a problem within the school's control
-be a problem that, if solved, will mean progress toward some larger goal

There are four main tasks to help you investigate instruction and articulate a problem of practice:

1. Link learning and teaching: With this particular learning problem, how does instruction impact what students learn?
2. Develop the skill of examining practice: How do we look at instructional data?
3. Develop a shared understanding of effective practice: What does effective instruction for our learning problem look like and what makes it effective?
4. Analyze current practice: What is actually happening in the classroom in terms of the learning problem, and how does it relate to our understanding of effective practice?

If teachers don't fundamentally believe that their teaching can make a difference for student learning, then it's going to be difficult to convince them to change their teaching.

Components of examining practice:

1. Evidence, data about teaching
2. Precise, shared vocabulary
3. Collaborative conversation with explicit norms

Hearing others' responses to the same lesson helps challenge individual assumptions, helps us notice different things and see the same things Ina new way, and leads to a better understanding of the practice observed..

Connecting best practices to data serves multiple purposes: it increases the likelihood that the practice is effective rather than simply congenial; it reinforces the discipline of grounding all conversations about teaching and learning in evidence rather than generalities or assumptions; it's more persuasive-teachers are more likely to try something for which there's evidence that it works; and it reinforced the link between learning and teaching.

Chapter 6: Developing An Action Plan

To create a successful action plan, four tasks are required.

1. Decide on an instructional strategy or strategies to address the identified area of need.

2. Agree on what the plan will look like in the classroom. Be sure to include what others should expect to see teachers and students doing if the plan is being implemented.

3. Put the plan on paper. Document the roles and responsibilities of team members and the specific steps that need to occur. This will create internal accountability. Also include the Professional Development time and instruction that the team will need to ensure support.

4. Plan how you will know if the action plan in working.

What type of student outcome data will be collected? Is it a school-wide problem? Is it a grade-level issue? Is the problem specific to a content area? Is the issue specific to a group of students or teacher(s)?

Brainstorm solutions to the problem: Engage teachers in conversations about addressing problem.

Develop a common vision for implementation: Establish implementation indicators: 1. Description of what teacher will be doing. 2. Description of what students will be doing. 3. Description of classroom environment when strategy is in place.

Put the plan to paper. Identify specific tasks that need to be completed for success. Assign responsibilities and time frames. How will school support teachers as they carry out new work? Making sure the chosen instructional strategy is broad enough to cover a number of different content areas or grade levels and specific enough to ensure discussion, improvement, and concrete classroom practices.

Chapter 7: Planning to Assess Progress

Assessment plans address the following:

            What assessments will be used to measure progress? When will each type of assessment data be collected? Who is responsible for collecting and keeping track of the data? How will the data be shared among faculty and administrators? What are the goals for student improvement and proficiency?

Assessments to Measure Progress

Short-term data can be used to make assessments of students’ progress that includes:

·         Student’s classwork and homework – data is closely aligned with classroom instruction so teachers keep track of which students seemed to learn the critical skills of the lesson. Teacher can use this data to drive their instruction.

·         Classroom observations of student performance – this can provide teachers with rich data on what tasks students are engaged in and how they talk about texts, concepts and problems they are working on.

·         Conferences with students about their learning – insights from students can be gleaned through individual conferences, small focus groups, surveys or written reflections.

 

Individual classroom teachers should be responsible for collecting the short-term data.

 

Using Benchmark or Interim Assessments

In selecting benchmark assessments, leaders must consider the kinds of diagnostic information they want the tests to provide. It is important to keep in mind the amount of effort needed to administer the assessment, the speed with which results can be obtained and the richness of the information they provide.

Using In-House Assessments

Developing in-house assessments provides flexibility. Schools can design assessments closely aligned to their goals of the action plan. To ensure validity and reliability each school must:

1)    Ensure that each version of the test must measure the same skill.

2)    Difficulty levels should be consistent from one version of the assessment to the next.

3)    Tests must be administered under standardized conditions.

4)    A consistent scoring system must be established.

Long-term Data

Long-term data (statewide assessments) are collected on an annual basis. They are often generated by an external accountability system connected to state and federal mandates. This is the data that seems to “count” for schools and places pressure on them.

Long-term data can be helpful for evaluating progress over a longer time frame, such as several years and reflect trend data.

It is important to set short-term, medium term and long-term goals so teachers have targets to aim for and benchmarks by which to assess their students’ progress.

Chapter 8: Acting and Assessing

To help bring the action plan alive, school leaders need to communicate the action plan clearly, integrate the plan into the ongoing work of the school, and use teams for support and internal accountability.

Communicate the Action Plan Clearly

One effective means of communication is the creative use of school documents. Offering a one-page summary that sets out the key components of school improvement work can be an important step in helping create an environment in which all teachers are able to describe in specific and concrete language how the school is working to improve student performance.

Integrate the Action Plan into Ongoing School Work

Work with staff to develop a school wide curriculum map and describe how your new strategies fit into this overall plan.

Use Teams for Support and Internal Accountability

The system of interlocking teacher teams created for the collaborative data analysis is critical. As teachers become accustomed to checking in with their colleagues regularly, they make a greater effort to be prepared for meetings. They become more motivated to use strategies shared in their classrooms and to make positive changes to their instruction. Teachers also develop an increased sense of personal responsibility for their students’ performance.

 

References

Boudett, K.P., City, E.A., & Murnane, R.J. (Eds.). (2005). Data wise: A step-by-step guide to using assessment results to improve teaching and learning. Cambridge, MA: Harvard Education Press.