Running Head: DBQ ASSESSMENTS

Generating Student Performance Data with Document-Based Question (DBQ) Assessments:

Results from Two Urban Schools

by

Scott M. Petri

Los Angeles Unified School District

Submitted for TSU International Conference on Teaching and Learning

December 13, 2013

Track 1: Learning /Teaching Methodologies & Assessment

Presentation: This presentation and paper will describe how educators can incorporate the Common Core Standards for Writing into History-Social Science instruction with Document-

based Question Assessments (DBQs). This process will also generate new metrics and evaluation techniques that can improve the study of student writing.

Format: Interactive multi-media with slides displaying data, audio clips of student perceptions,

video clips of student & teacher reflections, samples of student work, and an activity for participants to evaluate an online peer-review rubric with their mobile devices.

Author Note:

Scott M. Petri, Ed. D.

6050 Colfax Avenue, North Hollywood, CA 91606.

scottmpetri@gmail.com 818-319-2581

DBQ ASSESSMENTS

2

Abstract

This pilot was conducted at two PHABO secondary schools in Los Angeles. Both schools had

API scores in the lowest decile of public schools in the state of California. A series of Document

Based Question Assessments (DBQs) were administered to a sample of students from each

school (N=315). The resulting performance data, additional factors, and implications for

Common Core writing instruction in Social Studies suggest that educators will be successful in

implementing common core writing tasks if they are supported by instructional leaders who

explicitly teach writing scaffolds across the curriculum, endorse the use of peer review grading

policies and incorporate revision memos into the writing process, and allow teachers to

collaboratively develop, design, and assess document based question assessments into all social

studies subjects.

DBQ ASSESSMENTS

3

Generating Student Performance Data with Document-Based Question (DBQ) Assessments: Results from Two Urban Schools

The Common Core State Standards call for teachers to emphasize argumentative and

informative writing into all subjects. Many teachers across the content areas are unsure how to

respond to these new standards. Should secondary social studies teachers stop delivery of subject

content to explicitly teach spelling, vocabulary, and sentence construction? Should professional

learning communities devote a specific amount of time to writing instruction in each subject?

How many writing projects should be delivered in each subject? Educators will struggle with

these questions over the next few years as they implement the Common Core writing and literacy

standards, however, this article will present a method for how teachers can implement the

Common Core Standards for writing into History-Social Science instruction immediately.

After a brief review of the literature on writing and formative assessments, it will report

the results of a pilot program that was conducted at two Predominantly Hispanic, Asian, Black

and Other (PHABO) secondary schools in California. Both schools had API scores in the lowest

decile of public schools in the state of California. A series of Document-Based Question (DBQ)

assessments were administered to a sample of 315 students at the two different high schools. The

resulting performance data and implications for Common Core writing instruction in Social

Studies will be discussed in this article.

Literature Review

Under Common Core, all teachers need to be writing teachers (Rothman, 2011). Social

Studies teachers can increase student literacy skills by incorporating document-based questions

as formal assessments. DBQ word production may be used as a goal setting strategy to help

communicate a key measure of student engagement to parents and students. In short, we can use

DBQ ASSESSMENTS

4

our students writing to show them where they are and use data from a national sample to show

them where they need to be.

There is concern that a majority of adolescents do not develop the competence in writing

they need to be successful in school, the workplace, or their personal lives (Graham & Perin,

2007). Blanton (1986) found English language learners (ELL) students often respond to each act

of writing as if it were a test, denying themselves space to practice with written language. ELL

students want to get it right the first time, a perception that often prevents them from becoming

proficient writers. Rogers & Graham (2008) found goal setting for productivity effective in a

meta-analysis of single subject writing interventions. Other researchers (Bissex & Bullock, 1987;

Calkins, 1994; Graves, 1983;) have noted a connection between increased reading and writing

and higher levels of academic achievement. Hence, Common Core and an increasing number of

assessments including the ACT, CRWA, and SAT employ writing-from-sources tasks that

integrate reading and writing.

De La Paz (2005) compared 8th-grade students (N=70) in an integrated social studies and

language arts unit designed to promote historical understanding and argumentative writing to a

control group of students (N=62) who did not receive writing intervention or instruction. Results

indicated the students who demonstrated mastery of the target strategies during instruction wrote

historically more accurate and more persuasive essays regardless of their initial learning profile.

Similarly, De La Paz & Felton (2010) compared 11th grade students who learned a pre-

writing strategy (N=81) for composing argumentative essays related to historical events to a

control group (N=79) that read the same primary and secondary source document sets. They

found that the essays written by students who received pre-writing instruction were longer, were

rated as having significantly greater historical accuracy, were significantly more persuasive, and

DBQ ASSESSMENTS

5

claims and rebuttals within each argument became more elaborated. The word count for the pre-

writing instruction group increased from 195.32 to 327.86 on average, an increase of 132.54

words. Yet for the control group, word production only increased by 14.45 words.

This research suggests that writing instruction that is focused on goal setting strategies,

argumentative claims and rebuttals, and historical accuracy may be effective when introducing

common core writing tasks to students. Hence, instructional leaders should encourage teachers

to design, develop, and analyze DBQs as formative assessments in common planning time, or

department professional development.

Complex writing assignments, or DBQs, are essential for improving adolescent literacy

(Fisher & Frey, 2007). DBQ units align with plans for increasing writing proficiency, critical

thinking, and creating a college-going culture. DBQs can be designed to give students a preview

of Advanced Placement curriculum. Increased use of DBQs should lead to greater English

proficiency and help students avoid costly and demoralizing remedial coursework that has an

adverse effect on college completion rates. DBQs should be jointly developed and graded by

History and English teachers to ensure that students will meet the new Common Core standards

for Writing. Students may be more motivated when they get credit in both classes for the same

assignment. This credit can be given in multiple stages for planning, writing, and revising

DBQs.

The College and Career Readiness Anchor Standards in Writing (See Figs. 1 & 2)

demand that students are able to write arguments on discipline-specific content, developing

claims and counterclaims, while establishing a formal tone and objective style citation. Using

controversial, or debatable content, teaching students to write in the third person, and using their

historical content knowledge to qualify them as an expert may provide motivation when

DBQ ASSESSMENTS

6

teaching this type of writing.

Figure 1

64

| 6

-12

| HIS

TOR

Y/SO

CIA

L ST

UD

IES

, SC

IEN

CE

, AN

D T

EC

HN

ICA

L S

UB

JEC

TS |

WR

ITIN

G

WHST

COMMON CORE STATE STANDARDS FOR ENGLISH LANGUAGE ARTS & LITERACY IN HISTORY/SOCIAL STUDIES, SCIENCE, AND TECHNICAL SUBJECTS

Writing Standards for Literacy in History/Social Studies, Science, and Technical Subjects 612The standards below begin at grade 6; standards for K5 writing in history/social studies, science, and technical subjects are integrated into the K5 Writing standards. The CCR anchor standards and high school standards in literacy work in tandem to define college and career readiness expectationsthe former providing broad standards, the latter providing additional specificity.

Grades 68 students: Grades 910 students: Grades 1112 students:Text Types and Purposes1. Write arguments focused on discipline-specific

content.a. Introduce claim(s) about a topic or issue,

acknowledge and distinguish the claim(s) from alternate or opposing claims, and organize the reasons and evidence logically.

b. Support claim(s) with logical reasoning and relevant, accurate data and evidence that demonstrate an understanding of the topic or text, using credible sources.

c. Use words, phrases, and clauses to create cohesion and clarify the relationships among claim(s), counterclaims, reasons, and evidence.

d. Establish and maintain a formal style.e. Provide a concluding statement or section

that follows from and supports the argument presented.

1. Write arguments focused on discipline-specific content.a. Introduce precise claim(s), distinguish the

claim(s) from alternate or opposing claims, and create an organization that establishes clear relationships among the claim(s), counterclaims, reasons, and evidence.

b. Develop claim(s) and counterclaims fairly, supplying data and evidence for each while pointing out the strengths and limitations of both claim(s) and counterclaims in a discipline-appropriate form and in a manner that anticipates the audiences knowledge level and concerns.

c. Use words, phrases, and clauses to link the major sections of the text, create cohesion, and clarify the relationships between claim(s) and reasons, between reasons and evidence, and between claim(s) and counterclaims.

d. Establish and maintain a formal style and objective tone while attending to the norms and conventions of the discipline in which they are writing.

e. Provide a concluding statement or section that follows from or supports the argument presented.

1. Write arguments focused on discipline-specific content.a. Introduce precise, knowledgeable claim(s),

establish the significance of the claim(s), distinguish the claim(s) from alternate or opposing claims, and create an organization that logically sequences the claim(s), counterclaims, reasons, and evidence.

b. Develop claim(s) and counterclaims fairly and thoroughly, supplying the most relevant data and evidence for each while pointing out the strengths and limitations of both claim(s) and counterclaims in a discipline-appropriate form that anticipates the audiences knowledge level, concerns, values, and possible biases.

c. Use words, phrases, and clauses as well as varied syntax to link the major sections of the text, create cohesion, and clarify the relationships between claim(s) and reasons, between reasons and evidence, and between claim(s) and counterclaims.

d. Establish and maintain a formal style and objective tone while attending to the norms and conventions of the discipline in which they are writing.

e. Provide a concluding statement or section that follows from or supports the argument presented.

Figure 2

65

| 6

-12

| HIS

TOR

Y/S

OC

IAL

STU

DIE

S, S

CIE

NC

E, A

ND

TE

CH

NIC

AL

SU

BJE

CTS

| W

RIT

ING

WHST

COMMON CORE STATE STANDARDS FOR ENGLISH LANGUAGE ARTS & LITERACY IN HISTORY/SOCIAL STUDIES, SCIENCE, AND TECHNICAL SUBJECTS

Writing Standards for Literacy in History/Social Studies, Science, and Technical Subjects 612

Grades 68 students: Grades 910 students: Grades 1112 students:Text Types and Purposes (continued)2. Write informative/explanatory texts, including

the narration of historical events, scientific procedures/ experiments, or technical processes.a. Introduce a topic clearly, previewing what

is to follow; organize ideas, concepts, and information into broader categories as appropriate to achieving purpose; include formatting (e.g., headings), graphics (e.g., charts, tables), and multimedia when useful to aiding comprehension.

b. Develop the topic with relevant, well-chosen facts, definitions, concrete details, quotations, or other information and examples.

c. Use appropriate and varied transitions to create cohesion and clarify the relationships among ideas and concepts.

d. Use precise language and domain-specific vocabulary to inform about or explain the topic.

e. Establish and maintain a formal style and objective tone.

f. Provide a concluding statement or section that follows from and supports the information or explanation presented.

2. Write informative/explanatory texts, including the narration of historical events, scientific procedures/ experiments, or technical processes.a. Introduce a topic and organize ideas,

concepts, and information to make important connections and distinctions; include formatting (e.g., headings), graphics (e.g., figures, tables), and multimedia when useful to aiding comprehension.

b. Develop the topic with well-chosen, relevant, and su!cient facts, extended definitions, concrete details, quotations, or other information and examples appropriate to the audiences knowledge of the topic.

c. Use varied transitions and sentence structures to link the major sections of the text, create cohesion, and clarify the relationships among ideas and concepts.

d. Use precise language and domain-specific vocabulary to manage the complexity of the topic and convey a style appropriate to the discipline and context as well as to the expertise of likely readers.

e. Establish and maintain a formal style and objective tone while attending to the norms and conventions of the discipline in which they are writing.

f. Provide a concluding statement or section that follows from and supports the information or explanation presented (e.g., articulating implications or the significance of the topic).

2. Write informative/explanatory texts, including the narration of historical events, scientific procedures/ experiments, or technical processes.a. Introduce a topic and organize complex ideas,

concepts, and information so that each new element builds on that which precedes it to create a unified whole; include formatting (e.g., headings), graphics (e.g., figures, tables), and multimedia when useful to aiding comprehension.

b. Develop the topic thoroughly by selecting the most significant and relevant facts, extended definitions, concrete details, quotations, or other information and examples appropriate to the audiences knowledge of the topic.

c. Use varied transitions and sentence structures to link the major sections of the text, create cohesion, and clarify the relationships among complex ideas and concepts.

d. Use precise language, domain-specific vocabulary and techniques such as metaphor, simile, and analogy to manage the complexity of the topic; convey a knowledgeable stance in a style that responds to the discipline and context as well as to the expertise of likely readers.

e. Provide a concluding statement or section that follows from and supports the information or explanation provided (e.g., articulating implications or the significance of the topic).

3. (See note; not applicable as a separate requirement)

3. (See note; not applicable as a separate requirement)

3. (See note; not applicable as a separate requirement)

Note: Students narrative skills continue to grow in these grades. The Standards require that students be able to incorporate narrative elements e"ectively into arguments and informative/explanatory texts. In history/social studies, students must be able to incorporate narrative accounts into their analyses of individuals or events of historical import. In science and technical subjects, students must be able to write precise enough descriptions of the step-by-step procedures they use in their investigations or technical work that others can replicate them and (possibly) reach the same results.

Developing the Pilot Program

In order to strengthen social studies teachers abilities to support writing across the

curriculum, several professional development seminars were developed to implement a

Document Based Questions (DBQ) assessment program. Teachers were skeptical about

assigning more frequent writing tasks and creating more work for themselves. To address these

concerns, revision memos and peer review rubrics were employed, so that teacher workloads

would not significantly increase. Mean word counts were charted for each class period. Essays

were skimmed by teachers to make sure students were on topic. After administering five DBQs,

students significantly increased their word production (See Figs. 3 & 4). Student performance

DBQ ASSESSMENTS

7

was charted for each writing task so parents could see where their student performed in relation

to their peers (See Fig 5). At the conclusion of this exercise, all of the teachers agreed to

implement DBQs into their classroom assessment practices the following semester.

Figure 3

!"#$%&'#"$()*$&'*#&+"(#&,-&."+"#/&0&1/($*2/%&&

3456789&:*,;&*$C?2@ &6BD&

7&

677&

DBQ ASSESSMENTS

8

Tenth grade World History students were given a series of complex writing tasks or

DBQs. In order to measure student effort, words produced in a 60 minute time period were

counted and analyzed on a per class basis. The results from the final writing task were compared

to the totals from the first writing task to determine how much student effort improved. It is

important to note that there were a total of 106 members of the tenth grade cohort, but only 73

writing tasks were completed on the first and last administrations of in-class DBQs. Student

absences were problematic and student motivation tended to be low, particularly during complex

writing tasks. DBQ teachers may reverse these factors by constantly displaying student effort

data in class, scaffolding pre-writing organizational tasks, and by increasing the number of DBQ

assessments given each semester.

Figure 4

!"#$%&'%()"(*+$,(-$+,.#/+"(

0(

100(

200(

300(

400(

5000(

5100(

5( 6( 7( 8( 9( 55( 56( 57( 58( 59( 15( 16( 17( 18( 19( 65( 66( 67( 68( 69( 25( 26( 27( 28( 29( 75( 76( 77( 78( 79( 35( 36( 37( 38( 39( 85( 86(

:;?(

:%&"@ ( (573(

:%,)&"@ (571(

A+B@ (031(

C)DE@ (686(

By May 2012, the word production generated by this cohort of students had increased to a low of 76 words per hour and a high of 990 words per hour.

The median was 315 words per hour

and the cohort mean was 342 words per hour.

This level of production was 68 words

per hour below the national mean. These results indicate that additional

practice has benefitted students and further practice could lift word production levels over the national mean.

Complicating Factors

At the end of the Spring 2012 semester, 105 students completed the class with 29 of these

students failing, yielding a course pass rate of 72% and a course failure rate of 28%. These

students had an average of 9.69 absences and 7.87 tardies. Assuming a loss of 82 minutes for

DBQ ASSESSMENTS

9

each class period and 15 minutes lost due to each tardy, this resulted in the loss of over 912

minutes, or 15 hours of instruction in Social Studies.

Cohort II

This study was replicated at a second high school location with a larger cohort of 216

students, 121 were enrolled in 10th grade World History, and 95 were enrolled in 11th grade US

History. Word production was tracked for three DBQ assignments. The remaining DBQs were

peer graded by students using rubrics designed for evaluating thesis statements and use of textual

evidence. Students who demonstrated increases in word production were given bonus points in

addition to the points awarded for the assignment. Also, students were able to submit revisions

of their DBQ essays for extra-credit during the semester.

Figure 5

!"#$

%$ %&&$%%'$

%("$ %("$%'%$

()"$($

(%"$ ((*$ (*#$ (*"$ (')$ ('!$ ('+$ ('"$ (+"$*#&$

*%"$

*+"$ '))$ ')!$'#"$ '&*$

'%+$ '%"$''#$ ''"$

,-.//0$12341$

-5674880$-9648:42;$

827.2150$;;/770$,26429$

82;-9/?0$82,

>/7$

:4/66/?0$/;64D>/$

72.5B0$,2642;;/$

4.7/84280$E6/11B$

E4;72B85;0$=4727$

:>-@/60$89/,26$

A/6/?0$-9648:5A9/6$

F47742,

80$12;1622615$

957D>4;0$2;:95;B$

DBQ ASSESSMENTS

10

period and 15 minutes lost due to each tardy, this resulted in the loss of approximately 648

minutes, or 11 hours of instruction in Social Studies. See (Figs. 7 & 8) for a comparison of the

final grades from both cohorts.

Conclusion

This article has described a Common Core writing instruction program that requires the

students to work harder than the teachers. The teacher delivers controversial social studies

content from multiple sources to students. The teacher demonstrates strategies for detecting bias,

evaluating rhetoric, and determining validity in a series of texts. The students then write an

argument in response to a complex question, where there is no correct answer. They utilize

sections of the texts to strengthen the claims in their argument. Then, the student writing is

assessed by an elbow partner. By utilizing peer-reviewed feedback, teachers are free to use their

instruction time encourage student debate and discussion on the topic, to demonstrate pre-writing

strategies, and to provide examples of effective student writing.

Figure 7

!"#"$%&'&()*+,&-$+./&0)1%$)234"*&

567898:&!"#$%&' (' )' *' +' ,'

8& 9& ;& & 9& ?&

DBQ ASSESSMENTS

11

that increase student engagement, lower tardy and absence rates, and increase student literacy

skills. DBQ instruction may boost student literacy rates and student achievement overall,

however, unless carefully scaffolded, this may result in increased number of incomplete

assignments and increased course failure rates, particularly among male students (See Figs. 9 &

10). Additional research may shed light on how to ameliorate this problem.

Figure 9

!"#"$%&''&()*+*,-&.%/01,%2&&

34&51,01$&67899:&

40! 15 !

Figure 10

!"#$%&'()*!"+"&,-*

!"#$%&'!()&'

*+,'

-..(/0#&01'

2#$%13'4"1&'

567'

809:&".&'

2;/3'?';@'

A(..(0/'

B"C.'

->/3'?';@'

D":E(&.'

A(..(0/'

80.1:910%'

A(0

DBQ ASSESSMENTS

12

References

Bardine, B., & Fulton, A. (2008). Analyzing the benefits of revision memos during the writing

and revision process. The Clearing House, 81(4), 149-154.

Bissex, G. L., & Bullock, R. (1987). Seeing for ourselves: Case-study research by teachers of

writing. Portsmouth, N.H.: Heinemann.

Blanton, L.L. (1986). Reshaping ESL students perceptions of writing. ELT Journal. 41(2), 112-

118. DOI: 10.1093/elt/41.2.112

Brookhart, S. M. (2013). How to create and use rubrics for formative assessment and grading.

Teacher Librarian, 40(4), 52.

Calkins, L. (1994). The art of teaching writing. Portsmouth, N.H.: Heinemann.

De La Paz, S. (2005). Effects of historical reasoning instruction and writing strategy mastery in

culturally and academically diverse middle school classrooms. Journal of Educational

Psychology, 97(2), 139-156.

De La Paz, S., & Felton, M. (2010). Reading and writing from multiple source documents in

history: Effects of strategy instruction with low to average high school writers. Journal of

Contemporary Educational Psychology, 35, 174-192.

Graves, D. (1983). Writing: Teachers and children at work. Exeter, N.H.: Heinemann

Educational Books.

Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve writing of

adolescents in middle and high schools - A report to Carnegie Corporation of New York.

Washington, DC: Alliance for Excellent Education.

DBQ ASSESSMENTS

13

Grant, S. G., Gradwell, J. M., & Cimbricz, S. K. (2004). A question of authenticity: The

document-based question as an assessment of students knowledge of history. Journal of

Curriculum and Supervision, 19, 309-337.

Fisher, D., & Frey, N. (2007). Checking for understanding: Formative assessment techniques for

your classroom. Alexandria, VA: Association for Supervision and Curriculum

Development.

O'Toole, R. (2013) Pedagogical strategies and technologies for peer assessment in Massively

Open Online Courses (MOOCs). Discussion Paper. University of Warwick, Coventry,

UK: University of Warwick. (Unpublished)

Rogers, L., & Graham, S. (2008). A meta-analysis of single subject design writing intervention

research. Journal of Educational Psychology, 100(4), 879-906.

Rothman, R. (2011). Something in common: The common core standards and the next chapter in

American education. Cambridge, MA: Harvard Education Publishing.

DBQ ASSESSMENTS

14

Author Bio

Scott Petri has worked as a social studies teacher since 2003; five of those years in

middle school and the rest in high school. He has served as a teacher, coordinator, and small

school principal in the Los Angeles Unified School District. He earned a Bachelor of Arts in

Political Science from the University of San Diego, a M. A. in Education Administration, and a

Doctorate in Educational Leadership from California State University Northridge, where he

developed an Entrepreneurial Orientation instrument for educators that evaluated 729 California

charter, private, and traditional public school teachers along domains of innovativeness,

proactiveness, and risk taking.