ALTOGETHER OBVIOUS:

HINDSIGHT BIAS AND THE EVALUATION OF INNOVATIONS

David Tan

Foster School of Business

University of Washington

davidtan@uw.edu

January 20, 2017

Please do not circulate without permission

* I am grateful for helpful comments on earlier versions from Chris Rider, Ming Leung, Deepak Hegde,

Ezra Zuckerman, Giacomo Negro, Matt Marx, and seminar participants at the University of Washington,

Washington University in St. Louis, and the Strategy Research Forum.

1

ALTOGETHER OBVIOUS:

HINDSIGHT BIAS AND THE EVALUATION OF INNOVATIONS

Abstract

Searching for new ways to use existing ideas is an important aspect of innovation and can involve a great

deal of ex ante uncertainty. The paradox is that the outcomes of this search often seem obvious when

viewed in hindsight. I examine the degree to which hindsight bias affects the evaluation of obviousness in

the patent examination process. If examiners are relying only on ex ante knowledge to assess

obviousness, then inventions should seem less obvious when examiners are less familiar with the

underlying areas of technology. If, on the other hand, examiners rely on ex post knowledge of the

outcome when they have less ex ante knowledge, i.e. hindsight bias, then inventions will seem more

obvious when examiners are less familiar with the underlying areas of technology. I test this hypothesis

using a novel data set on claim rejection rates for granted and rejected patent applications. I find that

claim rejection rates due to obviousness are higher when examiners are less familiar with the underlying

technologies in an application. Additional analysis of the kinds of citations that examiners use to make

obviousness rejections, the kinds of objections that applicants raise in response to rejections, and the rates

at which examiners reverse rejections in light of applicants’ objections provide further evidence

consistent with hindsight bias in the evaluation of innovations.

2

“Now that it has succeeded, it may seem very plain to any one that he could have done it as well. This is

often the case with inventions of the greatest merit.” -- Loom Co. v. Higgins, 105 U.S. 580 (1882)

“this inner circle of the audience knows the technical problems of the craft and the difficult problems…

they can respond to the work with a fuller understanding of what has been attempted” -- (Becker, 1982)

INTRODUCTION

Innovation has long been portrayed as a process of discovering novel ways to use and recombine

existing ideas (Schumpeter, 1939). While most, if not all, inventions can be linked to some basis in pre-

existing ideas and technologies, the act of discovering new ways to use existing ideas is often a difficult

inventive step in and of itself (Fleming, 2001). Aside from the ultimately successful end-product, there

are numerous more unsuccessful possibilities that an inventor must rule out to discover one that is

successful. The less familiar the search space, the less guidance there is from prior experience and related

knowledge to help narrow the range of possibilities (Nelson & Winter, 1982; March, 1991; Stuart &

Podolny, 1996). Searching for unfamiliar new ways to use existing ideas therefore represents an important

source of uncertainty in innovation and an important way in which inventors use inventive insight.

Given the uncertainty around innovation, it would be troubling if the ex ante difficulty of working

through this uncertainty were to be systematically discounted by audiences who evaluate innovations ex

post. Anecdotal evidence suggests that this may indeed be happening in one of the most economically

consequential and large-scale evaluative contexts for technological innovations: the examination of patent

applications. In the U.S., patent applicants must demonstrate not only that their inventions are novel, i.e.

different from previous technologies, but that these differences represent non-obvious inventive steps (35

U.S.C. §103). In other words, patents are not supposed to be granted for inventions that, while admittedly

novel, are known to be inevitable next steps. In principle, this is meant to discourage the patenting of

3

predictable technological problem-solving and to reward inventive efforts with less guidance from prior

knowledge. Indeed, in line with the academic literature, courts have long recognized that combining

existing ideas in novel ways is an important aspect of innovation: “Virtually all inventions are

combinations and virtually all are combinations of old elements” (Envtl. Designs, Ltd. v. Union Oil Co.,

713 F.2d 693, 698, 1983).

Yet courts have observed a tendency for patent examiners to discount the difficulty of inventions

solely on the basis of finding related elements in pre-existing technologies. For instance, a commonly

observed practice is for examiners to “break an invention into its component parts (A + B + C), then find

a prior art reference containing A, another containing B, and another containing C, and on that basis alone

declare the invention obvious” (Ruiz v. A.B. Chance Co., 357 F.3d 1270, 2004). This is potentially

troubling because unlike inventors facing ex ante uncertainty (Fleming, 2001), examiners have the benefit

of seeing only successful inventions in hindsight. As a result, rejecting an application by decomposing it

into familiar components “simply takes the inventor’s disclosure as a blueprint for piecing together the

prior art to defeat patentability—the essence of hindsight” (In re Dembiczak, 175 F.3d 994, 1999). If

patent applications are systematically rejected on this basis, then this may in effect “discount the value of

combining various existing features or principles in a new way to achieve a new result—often the very

definition of invention” (Ruiz v. A.B. Chance Co., 357 F.3d 1270, 2004).

Should such cases be interpreted as random imperfections in a necessarily imperfect system, or

could they be indicators of a more systematic bias against an aspect of innovation that the system is

intended to reward? Despite numerous landmark cases on the issue, there has been no prior empirical

study of how examiners evaluate the obviousness of recombinant innovations. This is due at least in part

to the lack of systematic data on claim rejections. In particular, an important first-order question that

remains unanswered is whether examiners are more likely to reject an invention as obvious when they are

more or less familiar with the underlying technologies.

4

To answer this question, I rely on a novel data set to examine claim rejection rates for a sample of

granted and rejected patent applications filed with the USPTO between 2001 and 2002. A unique feature

of this data set is claim-level data coded from the text of rejection and allowance notices mailed by

examiners as part of the application process. The text of rejection notices indicates the statutory basis for

each rejection, allowing me to distinguish rejections based on obviousness from rejections for other

reasons. The data also allow for additional analysis of the kinds of prior art citations that examiners use to

reject a claim for obviousness. The text of rejection notices allows me to identify which prior art citations

were used to make obviousness rejections and to match these to rejected claims. It also allows me to

determine the ultimate disposition of each claim. Some rejected claims are abandoned by the applicants in

light of examiners’ rejections, but others are ultimately allowed in light of applicants’ responses to

rejections.

Using this data set, I test for evidence of hindsight bias in the evaluation of obviousness. If

examiners are relying only on ex ante knowledge to assess obviousness, then inventions should seem less

obvious when examiners are less familiar with the underlying technologies. If, on the other hand,

examiners rely more on ex post knowledge of the outcome when they have less ex ante knowledge, i.e.

hindsight bias, then inventions will seem more obvious when examiners are less familiar with the

underlying technologies. The results are more consistent with this latter interpretation.

The main result is that claim rejection rates due to obviousness are higher when the technologies

in an application are less familiar to the examiner. To explore the mechanism behind this result, I examine

how the presence of unfamiliar technologies affects the kinds of citations examiners use to make

obviousness rejections and the objections that applicants raise in response to obviousness rejections.

When examiners are less familiar with the technologies in an application, the prior art citations they use to

make obviousness rejections are more likely to teach away from the invention. Consistent with the

hypothesized mechanism behind hindsight bias, this suggests that examiners overestimate the

5

obviousness of an invention by overlooking ways in which the prior art pointed away from rather than

towards the invention.

Finally, using data on individual claims matched to the examiner citations used to reject them for

obviousness, I show that when an examiner citation teaches away from the focal invention, claims that

were rejected based on that citation are more likely to be ultimately allowed in light of the applicant’s

objection. This is consistent with hindsight bias representing an impermissible basis for making

obviousness rejection and valid grounds for objection by applicants.

Non-obviousness in the patent system

Each year, the U.S. Patent and Trademark Office (USPTO) receives more than 500,000

applications for patent rights on technological innovations (Hegde, 2012). On average, about 70 percent

of these ultimately result in a granted patent (Lemley & Sampat, 2012). During the application process, an

examiner may initially reject some or all of the claims in an application for being previously patented or

for being obvious steps given the prior state of the art. The applicant then has an opportunity to respond

and challenge the basis for a rejection. Based on the applicant’s response, the examiner may maintain the

initial rejection or decide to allow some or all of the previously rejected claims. The reject-response

process may go on for several rounds. The process ends with a granted patent when all of the claims are

allowed or some claims are allowed and the applicant decides to abandon the rejected claims. Or the

process ends with an abandoned application if all claims are rejected and the applicant decides not to

further pursue any of the rejected claims.

Figure 1 shows the distribution of rejection rounds for 72,252 patent applications filed in 2001

and 2002 that were abandoned or resulted in a granted patent as of 2014. The sample represents

applications numbering from 10/000,001 to 10/099,999. In this sample, 76 percent of filed applications

ultimately resulted in a granted patent. Among these applications, only 21 percent were granted without

undergoing at least one round of rejection. The vast majority of applications underwent one or more

rounds of rejection before being ultimately granted. Disagreements about patentability between examiners

6

and applicants—which must be worked out through additional rounds of examination—add to the time

and cost of granting patent rights. The sources of these disagreements have not been previously studied

but are important for understanding the efficiency of the patent system.

The two principle criteria for patentability in the U.S. are that an invention must be novel (35

U.S.C. §102) and non-obvious (35 U.S.C. §103) in light of the prior art 1. Of these, non-obviousness has

long been considered the “ultimate condition of patentability” (Witherspoon, 1980). Determining whether

an invention is novel is relatively straightforward. An invention can be said to be novel so long as there is

no prior art that is identical, i.e. contains each and every single element of the claimed invention

(Continental Can Co. v. Monsanto Co., 948 F.2d 1264, 1991).

The non-obviousness requirement, on the other hand, is far more controversial. Even if an

invention is novel, it is not necessarily non-obvious. After establishing that an invention is not identical to

any prior invention, establishing non-obviousness under §103 requires examiners to determine whether

the differences between the invention and the prior art represent significant—i.e. non-obvious—inventive

steps. This is often the most ambiguous and contested aspect of the examination process. As legal

scholars have argued, there is very limited guidance from the statutory language or case law about what

quantum of inventive insight is needed for a novel difference to be considered non-obvious (Durie &

Lemley, 2008; Mandel, 2008).

One of the core challenges is that instead of being rooted in contemporaneous evidence, the non-

obviousness requirement calls upon examiners to reconstruct, in hindsight, a legal fiction. What

examiners are asked to determine is not whether an invention seems ex post obvious to them after reading

the application. When submitting an application, applicants are required to provide a sufficiently detailed

description of an invention so that it is entirely clear to anyone who reads the application how the

inventors were able to make the various aspects of the invention work. Instead, what 35 U.S.C §103 asks

examiners to determine is whether an invention would have been “obvious at the time the invention was

11

The other two requirements not associated with the originality of an invention are that it must contain patentable

subject matter (35 U.S.C. §101) and disclose enough information to enable others to practice the invention (35

U.S.C. §112).

7

made to a person having ordinary skill in the art” (emphasis added). In other words, despite already

knowing the outcome, examiners must determine whether the steps taken by the inventor would have

been the only plausible ones to attempt before knowing what the outcome would be. As the Supreme

Court observed, evaluating non-obviousness “requires the oft-difficult but critical step of casting the mind

back to the time of invention” (In re Dembiczak, 175 F.3d 994, 1999).

Evaluating recombinant uncertainty in hindsight

Like any evaluation system, the patent examination process has its imperfections, and not all are

necessarily cause for concern (Lemley, 2001). However, the potential for hindsight bias is particularly

important to investigate because it reflects a discount against one of the fundamental reasons why

invention is difficult and is supposed to be rewarded by the patent system (e.g. Envtl. Designs, Ltd. v.

Union Oil Co., 713 F.2d 693, 698, 1983; In re Dembiczak, 175 F.3d 994, 1999; Ruiz v. A.B. Chance Co.,

357 F.3d 1270, 2004). The core issue in these cases is that examiners sometimes fail to appreciate the ex

ante uncertainty behind an invention (Fleming, 2001). Examiners are able to see a successful outcome of

innovation, but they may fail to recognize the broader search space of successful and unsuccessful

possibilities that an inventor would have had to work through to arrive at the successful one. Innovations

seem more obvious in hindsight if examiners cannot imagine the extent of the uncertainty that faced

inventors before the discovery of the successful outcome.

From the perspective of assessing hindsight bias, an important first-order question is whether

examiners are more likely to reject an invention as obvious when they are more or less familiar with the

underlying technologies. A key argument from the academic literature on innovation is that individuals

should in general perceive greater uncertainty in unfamiliar areas, far from the locus of their prior

experience (Nelson & Winter, 1982; March, 1991; Stuart & Podolny, 1996; Fleming, 2001). Prior

experience provides guidance about next steps in closely related areas. When individuals remain close to

the locus of prior experience, there are clearer expectations about which next steps are and are not likely

to turn out successfully.

8

From this perspective, it is less indicative of systematic hindsight bias if examiners are more

likely to view inventions in familiar areas of technology as obvious. This is consistent with examiners

judging obviousness based on their own ex ante knowledge. When examiners are more familiar with the

underlying technologies, they have clearer guidance from prior knowledge about which next steps are

likely to turn out successfully. Examiners should perceive less uncertainty and therefore be reasonably

expected to view inventions as more obvious. If this result holds, then there might not be reason to be

concerned about systematic hindsight bias. In this case, examiners are making obviousness rejections in

areas that prior theory suggests they would have perceived more ex ante uncertainty if they had attempted

these inventions without the benefit of hindsight (Nelson & Winter, 1982; March, 1991; Stuart &

Podolny, 1996; Fleming, 2001).

On the other hand, it is more indicative of hindsight bias if examiners are more likely to view

inventions in unfamiliar areas of technology as obvious. For inventions involving unfamiliar

technologies, examiners should have less guidance from their own prior knowledge about which steps are

and are not likely to turn out successfully. But hindsight bias suggests, counterintuitively, that the less ex

ante knowledge individuals have about a domain, the more likely they are to overestimate the obviousness

of outcomes in that domain (Fischhoff, 1975; Hawkins & Hastie, 1990). The logic is that seeing an

outcome ex post only provides information about the successful possibility, whereas ex ante knowledge

about a domain provides more balanced information about the broader space of possibilities that includes

successful as well as unsuccessful outcomes. The less ex ante knowledge individuals have about a

domain, the more their perceptions are weighted towards ex post knowledge about the successful

outcome. As a result, they overestimate the certainty of the successful outcome.

This can be seen in cases when examiners justify the obviousness of an invention based on prior

art that actually points away from the invention (In re Dembiczak, 175 F.3d 994, 1999; Ruiz v. A.B.

Chance Co., 357 F.3d 1270, 2004). Lack of ex ante knowledge about an invention’s areas of technology

means that examiners may be learning about the prior art in these areas for the first time. As a result, they

9

may be inadvertently learning about this prior art through the hindsight lens of the successful invention.

They become vulnerable to seeing only those aspects of prior art that point towards the steps that they

know in hindsight to be successful and vulnerable to overlooking aspects of prior art that point away.

Courts have observed this in cases when examiners justify the obviousness of an invention based on prior

art that actually “teaches away” from the invention (W.L. Gore & Associates, Inc., v. Garlock, Inc., 721

F.2d 1540, 1983; In re Grasselli, 713 F.2d 731, 1983). These cases are notable because they stem from

prior art citations that examiners themselves picked as the best ones to support their conclusions of

obviousness. In these cases, an unbiased reading of the prior art would have revealed that while it may

have contained related elements, the underlying logic advised against the kinds of next steps needed to

arrive at the focal invention. To take these steps in a direction contrary to the guidance of prior art, with

low prior expectations that a workable solution could be found, would have required non-obvious insight

from the inventor. So to conclude that that these steps were obvious would require a biased reading of the

prior art, reflecting hindsight knowledge of the successful outcome.

Qualitative examples

Application 10/071,840 provides an example of how this looks in practice. The invention in this

case is an implantable medical device that inhibits microbial growth. It does so through a combination of

two mechanisms: i) using an antimicrobial agent to kill microbes that come in contact with the device and

ii) preventing these microbes from permanently attaching to and forming a film on the device. In rejecting

several of the claims as obvious, the examiner reasoned that there were already prior inventions that

described: i) using antimicrobial agents on medical devices and ii) adding non-stick coatings to medical

devices. The applicants responded by pointing out that, while literally matching elements of the invention

in question, bringing these elements together is contrary to underlying logic of the prior art:

“… Farber teaches against the incorporation of antimicrobial agents for the following reasons: (a)

even broad-spectrum antibiotics eventually lead to the selection of resistant organisms…

Consequently, Farber resorts to an alternative approach based on interference with the adherence

10

of bacteria to polymeric surfaces of medical devices, thereby eliminating the need for

antimicrobials in his devices.” (emphasis added)

As noted by the applicants, the prior art suggested against using the two mechanisms together. The prior

wisdom for using a non-stick coating was to avoid having to use antimicrobial agents. In contrast, the

applicants’ invention uses the two mechanisms together and in a synergistic way. When microbes attach

to a device, they form a protective film that prevents the antimicrobial agent from killing any further

microbes. The non-stick mechanism is used to maintain a clear surface so that the antimicrobial agent can

continue to act for as long as the device is implanted. Given the contrary prior wisdom about how these

elements should be used, using them together in the focal invention could not be viewed as an obvious

extension of the prior art. The examiner subsequently reversed these rejections and allowed the claims.

Application 10/043,849 provides a similar example. In this case, the invention is an organic

optoelectronic device. In rejecting several of the claims as obvious, the examiner reasoned that the

various mechanisms and components of the invention could be found in the prior art. As in the preceding

example, the applicants pointed out that while the elements could be found, the underlying wisdom

attached to each element would have suggested against combining and adapting them in the manner

necessary to produce the invention in question:

“In this case, however, the invention only appears obvious after studying the here claimed

invention and combining the disclosures of the references with the application of undue

hindsight… For example, without undue hindsight, there is no reason why one of ordinary skill in

the art would be motivated take the layer taught by Hora, which is an electrical isolation layer that

is disposed between the electrodes (i.e., it is a key component of the active region) of an

electroluminescent lamp, and insert it into the OLED structure of Affinito… Indeed, it is

respectfully submitted that one of ordinary skill in the art would actually have been dissuaded

from providing particles within one of the polymer smoothing layers of Affinito… because

adding solid particles to a layer whose function is to cover and smooth underlying features is

antithetical to achieving that function.” (emphasis added)

11

The examiner subsequently reversed these rejections, citing the applicants’ clarification.

While examiners will oftentimes simply include a reference to the applicant’s response when

reversing a rejection (e.g. “Please refer to the Applicant's arguments filed on…”), there are cases when

examiners will explicitly acknowledge their own hindsight bias. Application 10/071,596 provides an

example. In this case, the invention is a method for cutting a hole in a piece of ceramic-matrix composite.

Cutting holes in ceramic composites is difficult using conventional drilling techniques because the

material is not internally uniform and prone to fraying, splintering, and cracking. The invention proposes

a solution that uses a cutting head smaller than the size of the desired hole and creating the hole by

removing successive layers in a manner akin to milling. In rejecting the claims as obvious, the examiner

reasoned that because i) numerous prior inventions describe methods for cutting holes in composite

materials and ii) techniques for milling metal are well-known, it would have been obvious to adapt

milling techniques to the particular material properties of ceramic-matrix composites for the purpose of

cutting holes. As the examiner subsequently acknowledged, however, the act of combining and adapting

these disparate insights to this unique problem required significant inventive insight and only seemed

obvious in hindsight:

"…the milling operation claimed by Applicant is something of a hybrid operation of drilling and

milling, which Examiner concedes would likely have a different set of parameters than the

parameters utilized in a pure milling operation… to combine the teachings of Handbook with the

teachings of Eriksson to arrive at the presently-claimed invention would require the use of

impermissible hindsight to first modify the teachings of Handbook (before applying them to

Eriksson), since Handbook is providing feed guidelines for a different type of machining

operation (face milling as shown in Figure 10-78) than for the hybrid drilling/milling operation of

the present claims, and since Handbook does not provide any feed guidelines for the selection of

a feed for a composite material.” (emphasis added)

As these examples illustrate, seeing an invention in hindsight primarily provides information about a

successful path of inventive steps. Faithfully judging obviousness from the ex ante perspective of the

12

inventor requires an appreciation of not just the ex ante reasons to expect these steps would work but also

ex ante reasons to expect they would not. When examiners are less familiar with the field of prior art

behind an invention, they are more susceptible to using the invention as a roadmap for learning about this

prior art and inadvertently cherry-picking only those aspects of prior art that point towards the successful

path.

Constraints on examiner familiarity

Ideally, every invention can be matched to an examiner whose prior experience is perfectly

aligned with the various areas of technology in the invention. In practice, however, there are constraints

on the ways in which examiners gain familiarity with the prior art. Each examiner belongs to one of

several technology centers, corresponding to major fields of technology. In the interest of administrative

efficiency, these divisions are necessarily more static and coarse-grained than the evolving universe of

inventive activity. Nested within these technology centers are hundreds of art units, each corresponding to

a narrower subfield and headed by its own supervisory patent examiner. When applications are received

by the USPTO, the Office of Initial Patent Examination makes an initial classification of the content and

dispatches it to the most appropriate art unit. Applications received by an art unit are then divided

randomly amongst examiners to balance workload.

This formal division of labor has important consequences for the evaluation of obviousness

because it constrains how examiners accumulate familiarity with domains of technology. Examiners gain

exposure to technologies based on the applications that are assigned to them in accordance with the

administrative boundaries between technology centers. Within a domain of specialization, examiners are

more likely be aware of not just what technologies exist but beliefs are about the directions in which

technologies are and are not likely to progress (Nelson & Winter, 1982; March, 1991; Stuart & Podolny,

1996). This helps examiners to simulate an inventor’s prior knowledge about the field of search behind an

invention. This includes beliefs not only about which paths of innovation are likely to be successful but,

just as importantly, which paths are likely to be unsuccessful. For inventions that do not align closely with

these domains, examiners are less likely to have prior knowledge about all of the underlying technologies

13

and less likely to be aware of the multitude of unsuccessful steps that inventors could plausibly have

taken. Their views about obviousness are therefore more likely to be overestimated as a result of seeing

only successful outcomes in hindsight. In the language of the obviousness statute, when an invention

aligns with an examiner’s prior experience, the examiner more closely approximates a “person having

ordinary skill in the art”, whereas when it does not align, the examiner must judge obviousness “with

reference to a hypothetical construct that virtually by definition could not exist” (Durie & Lemley, 2008).

Hindsight bias predicts that in these cases, when examiners lack an ex ante reference point, they are more

likely to overestimate obviousness in hindsight.

METHOD

To test this hypothesis, I use data on claim rejections and acceptances for a sample of 12,100

patent applications filed with the USPTO between 2001 and 2002. This sample includes applications that

ultimately resulted in a granted patent (77 percent) and applications that did not (23 percent). Claim-level

data are coded from the text of rejection and allowance notices mailed by examiners as part of the

application process. These can be found in the image file wrapper for each application, available through

the Patent Information Retrieval System (PAIR). Claim-level data permit analysis at a finer-grained level

than the acceptance or rejection of an application as a whole. The data reveal not only which claims are

ultimately accepted but which claims are accepted without rejection and which are accepted after being

initially rejected. Additionally, the text from rejection notices indicates the statutory basis for each

rejection, allowing me to distinguish rejections based on 35 U.S.C. §103 obviousness from rejections for

other reasons.

The sample of 12,100 consists of those applications numbering from 10/000,001 to 10/099,999

for which the image file wrapper was available and there was at least one applicant patent citation2. The

sample excludes cases in which inventions in the same family underwent prior examinations before the

2 The sample size turned out to be a round number purely by chance.

14

focal one, including continuations and non-U.S. applications entering the USPTO via the Patent

Cooperation Treaty (PCT).

Obviousness rejections. The primary dependent variable in my analysis is the percentage of

claims in a patent application that were rejected for being obvious. This is calculated using total number

of pending claims as the denominator and number of claims rejected due to obviousness as the numerator.

In this sample, 25 percent of applications were granted with all claims, and 20 percent resulted in no

granted claims. The remaining applications were granted with some claims allowed and others rejected

and abandoned. Figure 2 shows the distribution of claim allowance rates for applications that did not have

all or no claims allowed.

Final allowance rates understate the rate at which claims were rejected by examiners during the

application process. In particular, they mask rejections that examiners subsequently reversed in light of

applicants’ objections. To test the hypothesized mechanisms in this study, it is important to consider rates

of rejections. On average, 73 percent of the claims in an application are rejected. This is significantly

higher than what can be seen in the rate of ultimately allowed claims. The majority of these rejections are

due to obviousness. This can be seen in Figure 3. On average, 41 percent of an application’s claims are

rejected for being obvious under §103; 32 percent are rejected for not being novel under §102; 15 percent

are rejected for providing insufficient disclosure under §112; and only 2 percent are rejected for being

non-patentable subject matter under §101.

Invention’s technological overlap with examiner experience. The primary independent variable in

my analysis is the degree to which an examiner’s prior experience aligns with the areas of technology in a

focal invention. I measure this based on the technology classes of prior art cited in the applicant’s

information disclosure and the technology classes of the patents that the examiner previously examined3. I

3 I use the technology classes of applicant cited prior art because these are available for applications that were

ultimately granted as well as applications that were not. Unless it results in a granted patent, an application is only

classified under one technology class by the Office of Initial Patent Examination for the purposes of assigning it to

15

construct two vectors: 𝑇𝑎, whose elements are the percentages of an applicant’s prior art in each

technology class; and 𝑇𝑒, whose elements are the percentages of an examiner’s past patents in each

technology class4. The uncentered correlation between these two vectors, 𝑇𝑎 and 𝑇𝑒, provides a measure

of overlap between the technologies in an application and the technologies in which an examiner has prior

experience:

𝑇𝑎′𝑇𝑒

√𝑇𝑎′𝑇𝑎√𝑇𝑒

′𝑇𝑒

This measure ranges from 0, when an examiner has no prior experience with any of the technology

classes cited in an application, to 1, when an examiner has prior experience in all of the technology

classes cited in an application. Figure 4 shows the distribution of this overlap measure in my sample.

Controls. In all multivariate models, I include controls for attributes of the examiner, the

application, and the law firm that prepared the application. To account for differences between examiners,

all multivariate models include fixed-effects and clustered errors at the level of the examiner5.

At the application level, I include controls for total number of pending claims, number of prior art

references in the applicant’s information disclosure, the average year of the applicant’s prior art, and total

number of technology classes cited by the applicant’s prior art.

the relevant examiner. The full range of technology classes seen on published patents is determined at the end of

examination process for applications that have been granted. 4 This is consistent with prior measures of technological similarity and measures of examiner specialization, which

express technological experience as vectors of percentages across technology classes (Jaffe, 1986; Cockburn,

Kortum, & Lerner, 2003). 5 All results remain consistent when fixed-effects are at the level of the art unit. Prior studies suggest that there are

persistent differences between examiners and that a significant portion of variation in patenting outcomes can be

attributed to examiner fixed-effects (Cockburn et al., 2003; Lemley & Sampat, 2012). Consistent with these prior

findings, examiner dummies explain a significant portion of variation in obviousness rejections in my sample. An

OLS model with only art unit dummies has an adjusted R-squared of 0.12, while a model with only examiner

dummies has an adjusted R-squared of 0.31, and a model with both examiner and art unit dummies has an R-squared

of 0.32. This suggests that examiners vary substantially within art units, and examiner fixed-effects explain most of

the variation among art units. Since the focus of the analysis is not to estimate individual examiner effects, the main

analyses use within-examiner FE estimators to net out examiner differences, rather than including the full set of

more than 2,000 examiner dummies.

16

To control for variation across law firms that prepare these applications, I construct a measure of

average client size. Law firms representing larger clients that patent more heavily may accumulate

experience differently or have different application strategies compared to firms representing smaller

client that patent less heavily. For each law firm, I identify the assignees whose patents it was responsible

for filing. I count the number of patents granted to each assignee in the prior twenty years and take the

average across assignees for each law firm. This measures the extent to which a law firm works with large

clients6. Table 1 provides descriptive statistics.

Transfer requests

An invention’s technological overlap with examiner experience is meant to capture the degree to

which an examiner has prior familiarity with the areas of technology in the invention. To provide a basic

validity check that this is reflected in the measure, I examine how overlap with examiner experience

affects the likelihood of transfer requests.

During the process of matching an application to the most appropriate examiner, transfer requests

reflect cases when it is hard to find an examiner who has familiarity with all the technologies in the

application. When an application is received by the USPTO, it first goes to the Office of Initial Patent

Examination, which makes the initial decision about the most relevant art unit to examine the application.

In the majority of cases, a natural fit can be found and the application is examined in the art unit to which

it is first assigned. In some cases, however, examiners may file transfer requests when they do not feel

that an application fits with their expertise. As the USPTO has recognized, there will inevitably be some

applications that do not align well with any examiner’s expertise:

“A class of cases exists in which either no art or a divided art is found and in which no rule or

principle is involved. Such cases are placed where, in the judgment of the classification TC

6 Data are not available for clients in the case of applications that did not result in a granted patent. Unless an

application results in a granted patent, there is no assignment process and no identifiable assignee. However, for the

granted patents in my sample, the correlation between assignee size (number of patents granted in the prior twenty

years) and average client size of the assignee’s law firm is 0.73. In other words, large assignees tend to work with

law firms that work with other large assignees.

17

representative panel, they will be best searched and adjudicated. It is often impossible to so

explain a decision in this class of cases as to satisfy, or in any way aid, the examiners interested.

Indeed, the reasons for or against sending such cases one place or another may be so evenly

balanced that no reason of any value can be given.” (MPEP §903.08(e))

But: “Every application, no matter how peculiar or confusing, must be assigned somewhere for

examination” (MPEP §903.08(d)).

As a basic check that the measure of technological overlap is capturing alignment with examiner

expertise, I examine whether applications with low levels of overlap tend to experience higher numbers of

transfer requests. Figure 5 compares the number of transfer requests across low, medium, and high levels

of technological overlap with examiner experience. Overall, 16 percent of the applications in my sample

experienced at least one transfer request. As expected, applications with low levels of overlap with

examiner experience have significantly higher numbers of transfer requests than applications with high

levels of overlap.

Model (1) in Table 2 reports results from a fixed-effects Poisson model of the number of transfer

requests that an application received before being ultimately assigned. Fixed-effects and clustered

standard errors are at the level of the examiner. In this model, an invention’s technological overlap with

examiner experience has a significant negative relationship with number of transfer requests. This

provides assurance that the measure of technological overlap is correlated with an outcome that is known

to reflect an application’s alignment with examiner expertise.

MAIN RESULTS: OBVIOUSNESS REJECTIONS

Figure 6 shows univariate results comparing claim rejection rates across low, medium, and high

levels of an invention’s technological overlap with examiner experience, split at the terciles. Each panel

shows the average and 95 percent confidence intervals. For applications with low levels of technological

overlap with examiner experience, the percentage of claims rejected for obviousness is significantly

higher than for applications with high levels of overlap.

18

Table 4 shows results from regression models of claims rejected due to obviousness. All models

include fixed-effects and clustered errors at the examiner level. In model (1), the dependent variable is the

percentage of total claims in an application that were rejected due to obviousness. This is computed using

total claims as the denominator and number of claims rejected due to obviousness as the numerator. The

model is estimated using linear regression. In this model, an invention’s technological overlap with

examiner experience has a significant negative effect on percentage of claims rejected due to obviousness

(p<0.05)7. Model (2) shows that this result is robust when the dependent variable is modeled as a count.

In this model, the dependent variable is number of claims rejected due to obviousness. Total number of

claims is included as a control. It is estimated using Poisson regression. In this model, an invention’s

technological overlap with examiner experience has a significant negative effect on number of claims

rejected due to obviousness (p<0.05).

ADDITIONAL ANALYSIS

Examiner citations used to make obviousness rejections

To explore these results further, I examine how an invention’s technological overlap with

examiner experience affects the attributes of citations that examiners use to support their obviousness

rejections. Examples of hindsight bias from case law suggest that, when faced with unfamiliar

technologies, examiners may rely on the application as a blueprint for decomposing an invention into

elements that can be linked to pre-existing technologies and then cherry-pick these from otherwise

unfamiliar prior art sources to reject the invention as obvious (In re Dembiczak, 175 F.3d 994, 1999; Ruiz

v. A.B. Chance Co., 357 F.3d 1270, 2004). There would be greater evidence that examiners are relying on

hindsight in this manner if examiners cite prior art farther outside their prior experience to justify

obviousness rejections when they have less prior experience with the areas of technology in an invention.

Prior art citations are used by examiners for a variety of reasons, not all of which have to do with

claim rejections or specifically with claim rejections due to obviousness. From the text of claim rejection

7 Predicted values are roughly normally distributed around 0.41, with no values below 0.37 or above 0.49, well

within the 0 to 1 range of the dependent variable.

19

letters, I was able to determine which prior art citations examiners used to make obviousness rejections.

Using these citations, I measure the degree to which an examiner’s citations are to familiar areas. This

measure is computed in the same way as the previous technological overlap measure. In this case, overlap

is between the technology classes of the patents that the examiner previously examined and the

technology classes cited in the examiner’s obviousness rejections for a focal application. The measure

ranges from 0, when an examiner has no prior experience with any of the technology classes cited in the

examiner’s obviousness rejections, to 1, when an examiner has prior experience in all of the technology

classes cited in the examiner’s obviousness rejections.

Figure 7 shows univariate results comparing the examiner citation overlap measure at high,

medium, and low levels of an invention’s technological overlap with examiner experience. As the figure

shows, inventions with low levels of overlap with an examiner’s prior experience also result in

obviousness rejections based on examiner citations with low levels of overlap with the examiner’s prior

experience.

Model (1) in Table 5 confirms that this same pattern holds in a multivariate model. In model (1),

the dependent variable is technological overlap between examiner experience and the examiner citations

used to make obviousness rejections. It is estimated using linear regression with fixed-effects and

clustered errors at the level of examiners. The model controls for percentage of claims rejected for

obviousness and number of examiner citations used for obviousness rejections. In model (1), an

invention’s technological overlap with examiner experience has a significant positive effect. This implies

that when examiners are less familiar with the areas of technology in an application, examiners use

citations to less familiar areas of technology to make obviousness rejections.

On face, it may seem that this is what should be expected of the patent examination process:

examiners searching for prior art in a way that mirrors the areas of technology in an application. But it is

important to distinguish between examiners searching for prior art to assess novelty and examiners using

20

prior art to conclude obviousness. When assessing novelty (§102), it is acceptable for examiners to use

the technologies in an application as a roadmap to search the prior art. In this case, they are trying to

determine simply whether any prior art exists that is identical to the claimed invention in order to avoid

duplicate patenting of the same invention. For the purposes of determining novelty, it does not matter

whether an examiner would or would not have thought to find this prior art before reading the application.

In contrast, when assessing obviousness, case law and the Manual of Patent Examining Procedure

explicitly forbid examiners from using an application as a roadmap for finding and making connections to

prior art in a way that they would not otherwise have imagined before reading the application (e.g. In re

Dembiczak, 175 F.3d 994, 1999). It is impermissible for examiners to reject inventions as obvious based

on unfamiliar prior art that they would only have thought to find and connect based on seeing these

inventions in hindsight.

Applicant responses to rejections

To provide an even closer look at the hypothesized mechanism, I examine the objections that

applicants raise in response to examiners’ obviousness rejections. The hypothesized mechanism behind

hindsight bias in general is that when individuals have ex post knowledge of a successful outcome, they

tend to overlook ex ante reasons to expect the opposite and therefore overestimate the obviousness of the

successful outcome (Fischhoff, 1975; Hawkins & Hastie, 1990). In other words, their ex post knowledge

biases their perceptions about was known and believed ex ante. In the context of patent examination, this

can be seen in cases when the prior art that examiners use to justify an invention’s obviousness actually

“teaches away” from the invention (W.L. Gore & Associates, Inc., v. Garlock, Inc., 721 F.2d 1540, 1983;

In re Grasselli, 713 F.2d 731, 1983).

For the purpose of making obviousness rejections, examiners use prior art citations to show that

there was an ex ante basis in prior knowledge to expect that the inventive steps described in an application

would turn out successfully. In other words, these were predictable, the only plausible next steps to try

given the prevailing wisdom at the time. When examiners fall victim to hindsight bias, however, they

21

become vulnerable to inadvertently noticing and cherry-picking only those aspects of prior art that seem

to point towards the inventive steps in an application. They may overlook aspects of prior art, sometimes

in the same sources, that suggest the opposite: that there were other preferred alternatives or reasons to

expect these inventive steps would turn out unsuccessfully. When examiners view prior art as pointing

towards an invention when it actually points away from the invention, this provides an indication that

examiners may have allowed their ex post knowledge to color their perceptions about what was known

and believed ex ante.

As a window on this mechanism, I collected data on instances when the prior art cited by

examiners to make obviousness rejections actually teach away from the inventive steps in the application.

This was done by manually reading through the text of applicant responses to examiner rejections. Prior

art that teaches away from an invention is legally impermissible as a basis for obviousness rejections

precisely because it indicates a biased reading of prior art that is colored by hindsight (W.L. Gore &

Associates, Inc., v. Garlock, Inc., 721 F.2d 1540, 1983; In re Grasselli, 713 F.2d 731, 1983). This is a

valid basis for applicants to object to obviousness rejections and therefore appears in applicants’ response

letters. I was able to collect data on these objections for 96 percent of the applications in my sample with

obviousness rejections. Table 6 provides examples.

In the case of application 10027417 (Ex. 3), the examiner had argued that the applicant’s

invention simply took an obvious step of modifying an existing catheter invention (Miki et al.) with the

use a heat curing adhesive. As the applicants pointed out, however, the prior art cited by the examiner

explicitly advised against this modification: “the use of a heat curing adhesive agent is not advised”.

Taking this step—and making it work—was therefore contrary to the ex ante expectations in this prior art.

Similarly, in the case of application 10046568 (Ex. 7), the examiner had argued that the applicant’s

invention took an obvious step of modifying an existing nail polish invention (Graves et al.) with the

addition of a polymer. As the applicants point out, however, “a skilled artisan would have had no

reasonable expectation of success in incorporating such a polymer into the compositions of Graves et al.”.

22

Explicit statements in this prior art indicated that the outcome would not be “commercially viable or

useful” due to adverse effects from such a modification. In the case of application 10053085 (Ex. 10), the

applicants noted that the prior art cited by the examiner “expressly criticizes” the inventive step described

in the application. In the case of application 10097257 (Ex. 20), the applicants noted that the prior art

cited by the examiner points away from inks with small particle sizes: “The applicants have, surprisingly,

found that the use of very low particle size polymeric nanoparticles actually reduces the viscosity of the

dispersion, which is counter-intuitive from the known art because smaller particle size usually results in

higher viscosity”.

As these examples illustrate, examiners are often able to find prior inventions containing some

but not all elements of a focal invention. They must then make a judgment about whether the

modifications needed to arrive at the focal invention would have been obvious next steps given the prior

art. When examiners decide that these inventive steps were obvious but support their judgments using

prior art that explicitly advises against these steps, this suggests that examiners’ judgments may reflect

biased readings of the prior art, due to their knowledge about how the focal invention was able to make

these steps work in hindsight.

As an empirical window on hindsight bias, these “teach away” objections are especially

informative because they are less subject to gaming and manipulation on the part of applicants than other

kinds of objections that applicants may raise. First, a “teach away” objection requires that a prior art

citation contain suggestions against a particular inventive step. As the examples illustrate, applicants

cannot freely invoke this objection as a response to every rejection if no such suggestion is actually

present in the prior art cited by the examiner. Second, these are objections to prior art chosen by the

examiner. It is certainly plausible that applicants can always dig hard enough and find some prior art that

teaches away from a given invention. But these are citations chosen by examiners as the best examples to

demonstrate that prior art pointed towards the focal invention as the only plausible next step. To invoke

this objection, it is not enough for applicants to argue that the examiner’s examples are not clear or strong

23

enough. They must show that the examiner’s citations support the opposite conclusion. Applicants

therefore cannot freely invoke this objection in response to every rejection.

Figure 8 compares the number of examiner citations with “teach away” objections across

different levels of technological overlap between an invention and examiner experience. As the figure

shows, when inventions have low levels of overlap with examiner experience, examiners cite significantly

more prior art that teaches away from the focal invention when making obviousness rejections.

Model (1) in Table 7 confirms that this result holds in a multivariate model. The dependent

variable in this model is the number of examiner citations used for obviousness rejections that teach away

from the focal invention. The model is estimated using Poisson regression with fixed-effects and clustered

errors at the examiner level. The model controls for percentage of claims rejected for obviousness and

number of examiner citations used for obviousness rejections. In this model, technological overlap

between an invention and examiner experience has a significant negative effect. This implies that when an

examiner has less familiarity with the areas of technology in invention, the examiner is more likely to

make obviousness rejections based on prior art that teaches away from the invention.

These results are consistent with the hypothesized mechanism behind hindsight bias. Examiners

with less prior familiarity with a field of technology are less likely to have ex ante familiarity with

wisdom in the field, some of which may point towards and some of which may point against the inventive

steps taken by the focal invention. As a result, when they read the prior art in these areas for the first time,

their readings of the prior art are more heavily biased by knowing in hindsight which inventive steps

turned out successfully. By highlighting the successful paths, this hindsight knowledge paints a less

balanced picture than ex ante knowledge. It makes examiners more vulnerable to overlooking aspects of

prior art that pointed away from the invention and therefore vulnerable to overestimating obviousness.

24

Reversals of obviousness rejections

As a final step to explore this mechanism, I examine the likelihood of obviousness rejections

being subsequently reversed. As described above, it is impermissible to reject a claim as obvious based on

a prior art citation that teaches away from the invention. When examiners’ obviousness rejections are

subject to this objection, there should be a higher likelihood that these rejections are subsequently

reversed and the rejected claims ultimately allowed.

To explore this, I matched each individual claim to the examiner citation that was used to reject it

for obviousness. I was able to do this for 156,991 claims. These cover 93 percent of the applications in

my sample and 77 percent of the claims in these applications that were rejected due to obviousness. After

matching rejected claims to examiner citations, I then determined how many of these claims were

ultimately allowed. For 34 percent of the examiner citations used for obviousness rejections, all of the

claims rejected based on these citations were later allowed. For 37 percent of examiner citations used for

obviousness rejections, none of the claims rejected based on these citations was later allowed. Figure 9

shows the distribution of percentage of claims allowed for the remaining 29 percent of examiner citations

for which some but not all of the claims rejected were later allowed.

After matching claims to examiner citations, I then determined whether the examiner citation

used to reject a claim for obviousness was objected to by applicants as teaching away from the focal

invention. Figure 10 compares examiner citations that teach away from the focal invention and examiner

citations that do not. As expected, when examiners reject claims based on prior art citations that teach

away from the focal invention, a significantly higher percentage of these claims are ultimately allowed.

Model (1) in Table 8 confirms that this result holds in a multivariate model. The dependent

variable in this model is the percentage of claims rejected for obviousness that were later allowed. The

unit of observation is an examiner prior art citation. Percentages are calculated specifically for claims

25

rejected based on that examiner citation8. The denominator is the number of claims rejected for

obviousness by a given examiner citation, and the numerator is the number of these rejected claims that

were ultimately allowed. The key variable of interest is an indicator variable that takes a value of one

when the focal examiner citation teaches away from the invention in question (and zero otherwise). In this

model, this variable has a significant positive effect on percentage of claims allowed9. This implies that

when claims are rejected for obviousness based on prior art that teaches away from the focal invention,

these claims are more likely to be later allowed. This provides further support for the mechanism

suggested by prior results: rejections based on prior art that teaches away from the focal invention reflect

hindsight bias by examiners and are more likely to be later reserved.

Model (2) adds a variable that controls for the percentage of claims rejected for obviousness that

were also rejected for other reasons beside obviousness. As before, percentages are calculated at the level

of a given examiner citation, so the denominator is the number of claims rejected for obviousness based

on that examiner citation, and the numerator is the number of these claims that were also rejected for

other reasons besides obviousness. Model (3) splits this into separate variables corresponding to the three

other rejection criteria: lack of novelty (§102), insufficient disclosure (§112), and non-patentable subject

matter (§101). Each variable is a percentage, in which the denominator is the number of claims rejected

for obviousness based on a given examiner citation, and the numerator is the number of those claims that

were also rejected for one of the other three reasons besides obviousness. In models (2) and (3), with

these controls included, the variable indicating that an examiner citation teaches away from the invention

in question continues to have a significant positive effect on percentage of claims allowed.

8 Results are robust when aggregated at the application level. The models here are at the examiner citation level

because the key variable of interest—whether an examiner citation does or does not teach away from the invention

in question—is at the examiner citation level. The analysis could also be done at the level of the individual claim.

Since the variable of interest is at the examiner citation level, it will not vary across claims rejected by the same

examiner citation. Clustering errors across claim-level observations to account for these duplicate values would

produce the same result as analysis at the examiner citation level. 9 Predicted values are roughly normally distributed around 0.53, with no values below 0.35 or above 0.68, well

within the 0 to 1 range of the dependent variable.

26

Finally, model (4) shows that the effect is robust in a count model of claim allowances. In this

model, the dependent variable is the number of claims that were rejected for obviousness based on a given

examiner citation but were ultimately allowed. The model controls for total number of claims rejected for

obviousness based on that examiner citation. As before, the variable indicating that an examiner citation

teaches away from the invention in question has a significant positive effect on number of claims allowed.

The effect of the “teach away” variable provides a useful window on hindsight bias because it

reflects a relatively concrete reason for overturning obviousness rejections. When studying the allowance

of previously rejected claims, a potential concern is that allowances might reflect examiners being worn

down by especially stubborn applicants. However, as described above, an especially determined applicant

could not freely invoke the “teach away” objection if no such suggestion is actually present in the prior art

cited by the examiner. It is not enough for applicants to argue that the examiner’s examples are not clear

or strong enough. They must show that the examiner’s citations support the opposite conclusion.

Moreover, when applicants can show this, an especially stubborn examiner could not freely deny the

objection10

. The effect of the “teach away” variable indicates cases when rejections are reversed because

examiners had overlooked aspects of the prior art that pointed away from the invention in question. Taken

together, these and the prior results provide evidence consistent with hindsight bias in the evaluation of

obviousness.

DISCUSSION

Searching for new ways to use existing ideas is an important aspect of innovation and can involve

a great deal of ex ante uncertainty (Fleming, 2001). The paradox is that the outcomes of this search often

seem obvious when viewed in hindsight. In this study, I examine the degree to which hindsight bias

affects the evaluation of obviousness in the patent examination process. If examiners are relying only on

ex ante knowledge to assess obviousness, then inventions should seem less obvious when examiners are

10

Note that examiner fixed-effects ensure that the results are not being driven by between-examiner differences in

stubbornness.

27

less familiar with the underlying areas of technology. If, on the other hand, examiners rely more on ex

post knowledge of the outcome when they have less ex ante knowledge, i.e. hindsight bias, then

inventions will seem more obvious when examiners are less familiar with the underlying areas of

technology. I test this hypothesis using a novel data set on claim rejection rates for granted and rejected

patent applications. I find that claim rejection rates due to obviousness are higher when examiners are less

familiar with the underlying technologies in an application. Additional analysis of the kinds of citations

that examiners use to make obviousness rejections, the kinds of objections that applicants raise in

response to rejections, and the rates at which examiners reverse rejections in light of applicants’

objections provide further evidence consistent with hindsight bias.

These results have important implications for understanding the evaluation of innovations. Prior

research has recognized that the value of innovations is not always straightforward to demonstrate. For

instance, innovations with unfamiliar features may be difficult to understand and evaluate and therefore

diffuse slowly (e.g. Rogers, 1962). There may be reluctance to adopt innovations with unfamiliar

characteristics because of doubts about economic value, practicality, or desirability. But these problems

center on doubts about the value of innovations—not about whether an idea is in fact a non-obvious

inventive insight or discovery in the first place. The results in this study suggest, paradoxically, that when

evaluators are less familiar with the underlying areas of technology, an invention may suffer a discount in

terms of perceived innovativeness.

This finding seems counterintuitive because unfamiliarity would seem to imply that an innovation

is contrary to prior expectations. For instance, an insight from prior studies of category-spanning products

is that unfamiliar combinations fly in the face of audience expectations (Zuckerman, 1999; Negro &

Leung, 2013; Bowers, 2015; Leung & Sharkey, 2014). Unfamiliar combinations run counter to audience

beliefs about what does and does not make sense together. This can be a liability when evaluators must

accurately assess a product’s economic value and determine how much there is to gain from investing in

it (Zuckerman, 1999; Bowers, 2015; Leung & Sharkey, 2014). But in a context such as the patent system,

28

where innovations are rewarded precisely for being non-obvious,—i.e. inconsistent with prior

expectations—unfamiliarity would seem on face to be an advantage. Unlike innovation adopters or

investors, patent examiners are not evaluating the economic value of an invention. The evaluation of

obviousness is strictly about the degree to which an invention was an ex ante predictable outcome. If

unfamiliar sets of attributes run counter to audience expectations about what typically goes together, then

this would seem to be an asset in the context of demonstrating non-obviousness.

The theory behind hindsight bias helps to explain why unfamiliar sets of attributes may,

counterintuitively, appear more obvious to evaluators. Prior knowledge about a domain helps to provide

individuals with a clear baseline, which includes both ex ante reasons to expect an outcome will be

successful as well as ex ante reasons to expect it will not. When individuals lack this ex ante baseline,

their perceptions are more likely to be biased in hindsight by their ex post knowledge that the outcome

turned out to be successful (Fischhoff, 1975; Hawkins & Hastie, 1990). In the context of evaluating

innovativeness, an invention may not appear in hindsight to a be an extraordinary achievement if

evaluators have no ex ante reference point for what ‘ordinary’ looks like in that field of technology. Prior

familiarity with a field of technology helps to mitigate this potential bias by making evaluators more

aware of reasons why ‘ordinary’ inventive insight might have suggested many other paths besides the

ultimately successful one. However, when inventions involve unfamiliar areas of technology, evaluators

may not have prior knowledge to provide a relevant benchmark. In these cases, the relevant benchmark

for “ordinary skill in the art” may be “a hypothetical construct that virtually by definition could not exist”

(Durie & Lemley, 2008). In the context of patent examination, there has been a great deal of attention on

the possibility that examiners might be too lax in granting patents as a result of being unfamiliar with new

and changing fields of technology (e.g. Eisenberg, 2004). Indeed, an intuitive expectation is that

innovativeness should be easier to demonstrate to an audience that is ignorant of what else has been done

in a field. This study suggests, however, that there is at least a countervailing tendency. Ignorant

audiences are more likely to discount an innovation as facile: merely and un-extraordinarily novel.

29

Outside of the context of patent examination, these insights shed light on the role of the relative

comparison mechanism in evaluation processes. Familiarity is often thought to affect evaluation outcomes

via evaluator beliefs about whether the underlying elements—e.g. product features, lines of business,

genres, etc.—tend to or ought to go together. But familiarity with particular sets of attributes may also

affect evaluations because evaluators have a better understanding of variation among products with those

attributes, including both successful and unsuccessful variants. Greater experience with variation within a

category helps evaluators to form clearer benchmarks for what an ordinary level of success look like. This

allows evaluators to make relative comparisons among category members (Zuckerman, 1999; 2004).

When a set of attributes does not cohere with any domain of evaluator experience, there is greater

ambiguity about what the relevant benchmark is. Only recently has research begun to isolate the effects of

this relative comparison mechanism. Recent findings show, for instance, that rather than appear

extraordinary for not fitting in, unfamiliar combinations are less likely to be rewarded as extraordinary

(Bowers, 2015).

This study yields new insights about the relative comparison mechanism by showing how

evaluators form benchmarks when they are less familiar with a domain. The results suggest that

evaluators may fail to appreciate achievements in unfamiliar domains because they lack prior familiarity

with what an ordinary level of success looks like. As a result, they lack a clear reference point against

which extraordinary levels of achievement can be contrasted. In the context of patent examination,

obviousness rejections represent an evaluative outcome that directly reflects an examiner’s relative

comparison of an invention to the prior art. The prior art citations that examiners use to make obviousness

rejections shed light on how examiners form benchmarks for “ordinary skill in the art”. The results

suggest that when examiners are less familiar with a domain, reference points tend to be constructed

based on guidance from successful outcomes and therefore biased towards reasons to expect success. By

providing fine-grained evidence of these previously unstudied micro-mechanisms, this study contributes

30

novel insights about how reference points and relative comparison shape evaluation processes in

unfamiliar domains.

In terms of isolating these micro-mechanisms from other related explanations for discounts

against unfamiliar features, this context provides several advantages. A common explanation for

evaluative discounts against unfamiliar features is greater cognitive confusion. The context of

obviousness rejections is useful for isolating the relative comparison mechanism because the very nature

of an obviousness rejection implies the opposite of cognitive confusion. For a claim to be rejected for

obviousness, an examiner must first have determined that the claim does indeed contain a novel inventive

step—indicating that examiners are not failing to recognize novel features. An obviousness rejection then

implies that not only did these inventive steps seem easy to understand but there was so little ambiguity

about these steps that they were the only conceivable ones to take. Indeed, hindsight bias in patent

examination is essentially rooted in the tendency for examiners to experience less cognitive confusion

than they should. After seeing an invention in hindsight, examiners are only able to see how the

underlying features make sense. They are unable to imagine reasons why these would not have made

sense ex ante. When applicants argue that prior art teaches away from a novel feature they add, they are

essentially trying to demonstrate that this would have seemed more confusing ex ante than it may seem ex

post. They are explicitly pointing to ways in which these features clash with prevailing beliefs about what

does and does not make sense, trying to recreate a sense of the cognitive confusion that is no longer

apparent to examiners in hindsight.

A general insight stemming from these findings is that an achievement is more likely to appear

extraordinary to an audience that appreciates the ease of failure than to an audience that knows only of the

focal success. Similar observations have been made in other evaluative contexts in which savvy audiences

seem to be “the most understanding and forgiving audience” (Becker, 1982, p. 54). An example of

success can appear unremarkable if that is the only example audiences have seen. Inexperienced or

outside evaluators may be less capable of appreciating the ex ante difficulty of work in a domain because

31

they are less aware of how easily it could have been done less successfully. The generality of these

insights should be tested in other contexts, but they have the potential to shed new light on how

experience in a domain affects the way individuals conduct evaluations.

32

REFERENCES

Becker, H. S. 1982. Art Worlds. Berkeley: University of California Press.

Bowers, A. 2015. Relative comparison and category membership: the case of equity analysts. Organization Science, 26:

571-583.

Cockburn, I. M., S. Kortum, and S. Stern. 2003. Are all patent examiners equal? The impact of characteristics on patent

statistics and litigation outcomes. In Patents in the Knowledge-based Economy, W. M. Cohen and S. A. Merrill

(eds). National Academies Press: Washington, DC: 19-73.

Durie, D. J. and M. A. Lemley. 2008. “A realistic approach to the obviousness of inventions.” William and Mary Law

Review, 50: 989-1020.

Eisenberg, R. S. 2004. Obvious to whom? Evaluating inventions from the perspective of PHOSITA. Berkeley Technology

Law Journal, 19: 885-906.

Fischhoff, B. 1975. “Hindsight ≠ foresight: the effect of outcome knowledge on judgment under uncertainty.” Journal of

Experimental Psychology: Human Perception and Performance, 1: 288-299.

Fleming, L. 2001. “Recombinant uncertainty in technological search.” Management Science, 47: 117-132.

Hall, B. H., A. B. Jaffe, and M. Trajtenberg. 2001. The NBER patent citations data file: lessons, insights, and

methodological tools. NBER Working Paper 8498.

Hawkins, S. A. and R. Hastie. 1990. “Hindsight: biased judgments of past events after the outcomes are known.”

Psychological Bulletin, 107: 311-327.

Hegde, D. 2012. “Funding and performance at the US Patent and Trademark Office.” Nature Biotechnology, 30: 148-

150.

Lemley, M. A. and B. Sampat. 2012. “Examiner characteristics and patent office outcomes.” Review of Economics and

Statistics, 94: 817-827.

Lemley, M. A. 2001. Rational ignorance at the patent office. Northwestern University Law Review, 95: 1495-1532.

Leung, M. D. and A. J. Sharkey. 2014. “Out of sight, out of mind? Evidence of perceptual factors in the multiple-category

discount.” Organization Science, 25: 171-184.

Mandel, G. N. 2008. “Another missed opportunity: the Supreme Court's failure to define nonobviousness or combat

hindsight bias in KSR v. Teleflex.” Lewis and Clark Law Review, 12: 323-342.

March, J. 1991. Exploration and exploitation in organizational learning. Organization Science, 2: 71-87.

Negro, G. and M. D. Leung. 2013. “’Actual’ and perceptual effects of category spanning.” Organization Science, 24:

684-696.

Nelson, R. R. and S. G. Winter. 1982. An Evolutionary Theory of Economic Change. Cambridge, MA: Harvard

University Press.

Rogers, E. M. 1962. Diffusion of Innovations. New York: Free Press.

Stuart, T. J. and Podolny, J. 1996. Local search and the evolution of technological capabilities. Strategic Management

Journal, 17: 21-38.

Witherspoon, J. F. (Ed.). 1980. Nonobviousness: the Ultimate Condition of Patentability. Bureau of National Affairs.

Zuckerman, E. W. 1999. “The categorical imperative: securities analysts and the illegitimacy discount.” American

Journal of Sociology, 104: 1398-1438.

Zuckerman, E. W. 2004. “Structural incoherence and stock market activity.” American Sociological Review, 69: 405-432.

33

Figure 1. Rounds of rejection for accepted and rejected patent applications

This figure shows the distribution of rejection rounds for 72,252 patent applications filed in 2001 and 2002 that were ultimately

accepted or ultimately rejected and abandoned as of 2014. The sample represents cases with application numbers from 10/000,001 to

10/099,999. A rejection round refers to the mailing of a rejection notice by the examiner. Applicants always have the opportunity to

respond to and challenge a rejection notice. An application is only ultimately rejected if an applicant chooses to abandon the

application instead of responding. In this sample, 76 percent of applications ultimately resulted in a granted patent.

Figure 2. Percentage of claims in an application that were ultimately allowed

This figure shows the percentage of claims that were ultimately allowed for a sample of 12,100 patent applications. The figure does

not include applications that resulted in all claims being granted (25 percent) or applications that resulted in no claims granted (20

percent).

0

5

10

15

20

25

30

35

40

45

0 1 2 3 4 5 6 7 8 or more

% o

f a

pp

lica

tion

s

Rounds of rejection

Accepted applications Rejected applications

34

Figure 3. Percentage of claims in an application that were rejected

This figure shows the percentage of pending claims that were rejected in a sample of 12,100 applications. All percentages are

calculated using total number of pending claims as the denominator. In the first column, the numerator is the number of claims

ultimately allowed. In the second column, the numerator is number of claims rejected for any reason. In the next four columns, the

numerators are the numbers of claims rejected based on each of the four patentability standards.

Figure 4. Invention’s technological overlap with examiner experience

This figure shows the distribution of technological overlap between the prior art cited by an applicant’s information disclosure and the

patents that an examiner has previously examined. A value closer to 1 indicates high overlap, and a value closer to 0 indicates low

overlap.

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

Ultimately allowed Rejected:

All reasons

Rejected:

Obvious§103

Rejected:

Not novel§102

Rejected:

Insufficientdisclosure

§112

Rejected:

Non-patentablematter

§101

% o

f p

en

din

g c

laim

s

35

Table 1. Descriptive statistics

These tables show descriptive statistics and correlations for the key independent variables used in all analyses. The unit of observation

is the application.

N=12,100 Mean SD 25th %tile 50th %tile 75th %tile

(1) ln(total claims) 3.272 0.695 2.944 3.258 3.689

(2) ln(law firm's average client size) 7.586 1.699 6.963 8.005 8.608

(3) # technology classes in applicant prior art 4.017 4.337 1 3 5 (4) Average year of applicant prior art 1994.385 6.930 1991.833 1996.500 1999.000

(5) ln(applicant prior art citations) 1.809 1.247 0.693 1.792 2.639

(6) Invention's technological overlap with examiner experience 0.721 0.279 0.515 0.820 0.969

(1) (2) (3) (4) (5)

(1) ln(total claims) (2) ln(law firm's average client size) 0.008

(3) # technology classes in applicant prior art 0.200 -0.034

(4) Average year of applicant prior art 0.065 0.114 -0.067 (5) ln(applicant prior art citations) 0.249 -0.057 0.754 -0.098

(6) Invention's technological overlap with examiner experience -0.057 -0.004 -0.281 0.047 -0.215

Figure 5. Number of transfer requests

This figure shows the number of times that an assigned examiner requested that an application be transferred elsewhere before it was

ultimately assigned and examined. The unit of observation is the application. Dotted lines show the 95 percent confidence interval.

The figure compares low, medium, and high levels of an invention’s technological overlap with examiner experience, split at the

terciles.

Table 2. Fixed-effects Poisson model of number of transfer requests

This table reports results from a Poisson model of transfer requests. The unit of observation is the application. The model includes

fixed-effects and clustered standard errors at the examiner level.

(1)

ln(total claims) 0.156**

(0.057)

ln(law firm's average client size) 0.003

(0.018)

# technology classes in applicant prior art 0.046***

(0.012)

Average year of applicant prior art 0.001

(0.005)

ln(applicant prior art citations) -0.194***

(0.042)

Invention's technological overlap with examiner experience -0.811***

(0.107)

Observations 7,191

Number of examiners 1,146

Chi-sqr 78.98 df 6

Standard errors clustered by examiner; *** p<0.001, ** p<0.01, * p<0.05, + p<0.10

0.14

0.19

0.24

0.29

0.34

Low Medium High

Nu

mb

er o

f tr

an

sfer r

eq

uest

s

Invention's technological overlap with examiner experience

36

Figure 6. Percentage of claims rejected due to obviousness

The vertical axis in this figure shows the percentage of pending claims in an application that were rejected due to obviousness during

the examination process. Percentages are calculated using total number of pending claims as the denominator and number of claims

rejected due to obviousness as the numerator. The figure compares applications with low, medium, and high levels of an invention’s

technological overlap with examiner experience, split at the terciles. The unit of observation is the application. Dotted lines show the

95 percent confidence interval.

Table 4. Regression models of claims rejected due to obviousness

This table reports results from models of claims rejected due to obviousness. In column 1, the dependent variable is the percentage of

total claims in an application that were rejected due to obviousness. The model is estimated using linear regression. In column 2, the

dependent variable is number of claims rejected due to obviousness with a control for total number of claims. The model is estimated

using Poisson regression. In all models, the unit of observation is the application. All models include fixed-effects and clustered

standard errors at the examiner level.

(1) (2) Linear Poisson

% of total claims that were

rejected for obviousness

# of claims rejected for

obviousness

ln(total claims) -0.000 0.914***

(0.006) (0.017)

ln(law firm's average client size) 0.002 0.007 (0.002) (0.005)

# technology classes in applicant prior art -0.000 -0.001

(0.001) (0.003) Average year of applicant prior art -0.001 -0.001

(0.001) (0.002)

ln(applicant prior art citations) -0.003 0.005 (0.004) (0.012)

Invention's technological overlap with examiner experience -0.033* -0.085*

(0.014) (0.038) Constant 2.176+

(1.144)

Observations 12,100 10,257

Number of examiners 2,707 1,901

F-stat 1.614 . Chi-sqr . 2892

df 5 6

Standard errors clustered by examiner; *** p<0.001, ** p<0.01, * p<0.05, + p<0.10

0.38

0.39

0.4

0.41

0.42

0.43

0.44

Low Medium High

% o

f cla

ims

reje

cte

d f

or o

bvio

usn

ess

Invention's technological overlap with examiner

experience

37

Figure 7. Examiner citations’ technological overlap vs. invention’s technological overlap with examiner experience

This figure shows the technological overlap between examiner experience and the examiner citations used to make obviousness

rejections. The figure compares low, medium, and high levels of an invention’s technological overlap with examiner experience, split

at the terciles. The unit of observation is the application. Dotted lines show the 95 percent confidence interval.

Table 5. Technological overlap between examiner citations and examiner experience

This table reports results of a linear regression model in which the dependent variable is technological overlap between examiner

experience and the citations used by the examiner to make obviousness rejections. The unit of observation is the application. The

model includes fixed-effects and clustered standard errors at the examiner level.

(1)

Technological overlap

between examiner citations

and examiner experience

ln(total claims) -0.005

(0.006)

ln(law firm's average client size) -0.001 (0.002)

# technology classes in applicant prior art -0.007***

(0.001) Average year of applicant prior art 0.000

(0.001)

ln(applicant prior art citations) 0.031*** (0.004)

% claims rejected for obviousness -0.005

(0.013) Number of examiner citations used for obviousness rejections -0.005**

(0.001)

Invention's technological overlap with examiner experience 0.289*** (0.015)

Constant 0.498

(1.278)

Observations 6,582

Number of examiners 2,001

F-stat 57.66

df 7

Standard errors clustered by examiner; *** p<0.001, ** p<0.01, * p<0.05, + p<0.10

0.6

0.65

0.7

0.75

0.8

0.85

Low Medium High

Exa

min

er c

ita

tio

ns'

tech

no

log

ica

l

overla

p w

ith

exa

min

er e

xp

erie

nce

Invention's technological overlap with examiner experience

38

Table 6. Examples of applicant objections to examiner citations that teach away from focal invention

These examples come from the text of applicant responses to examiners’ rejection letters. They point out ways in which the prior art

cited by examiners to make obviousness rejections actually teach away from the inventive steps described by the applicants.

Example

number

Application

number

Text from applicant response to examiner rejection (emphases added)

Ex. 1 10002145 In summary, Vlodavsky explicitly teaches against the sugars of the currently claimed invention by teaching that to retain

high inhibitory activity against metastasis N-substituted oligosaccharides require a molecular size of about 4000 daltons or

more… Moreover, Vlodavsky has only negative things to say about the metastatic inhibitory effect of sugars within the

scope of the claimed invention (i.e., sugars with a molecular weight of 3000 daltons or less), such as the pentasaccharide was

completely devoid of activity against tumor metastasis.

Ex. 2 10024633

Rather, the very fact that Saitoh is an ink jet printing paper requires it to be water absorbing - if this were not the case, you could not ink jet print with the inks as disclosed in that patent. Furthermore, Saitoh does not suggest or motivate one to

employ such a weatherproof coating and, in fact, teaches away from the same since such a coating would render the ink

jet printing paper of Saitoh inoperative for its intended purpose - that is, to absorb the aqueous-based ink applied thereto.

Ex. 3 10027417 Moreover, Miki et al. teach away from using a heat curing adhesive at column 21, lines 33-39: "When a heat curing

adhesive agent is used, the catheter shaft and balloon are inevitably exposed to heat during the heating required for curing. As

a result, it is entirely possible that the balloon diameter will shrink, the balloon bursting pressure will decrease, and the catheter shaft will undergo thermal degradation, and these all lead to diminished balloon catheter performance, so the use of a heat

curing adhesive agent is not advised."

Ex. 4 10029584

Because Houser describes a graft that achieves an effective seal and fixation within a vessel without auxiliary means, and McDermott describes using the inflatable channels as fixation means without the use of a stent, the skilled artisan reading the

references would not look to the teachings of McDermott to modify the graft of Houser. Further, since McDermott is

explicitly desirous of minimizing the profile of the graft and states that the graft lacks a stent for maximal compression,

McDermott teaches away from the inclusion of a stent to the graft.

Ex. 5 10035420

Applicants submit that, contrary to the Examiner's assertion, the cited passage from Dorin et al., when taken in context, does

not provide motivation to adjust the pH of a formulation of IFN-p3 to about 3.0 to about 5.0. … to the contrary, this cited

passage specifically teaches away from a pH range of about 3.0 to about 5.0… Upon understanding the teachings of Dorin

et al. that during storage of an IFN-p3 polypeptide, maintenance of pH is "critical," and that the pH of the formulation is most

preferably "adjusted to 6.5," one of skill in the art (who would understand that each pH unit represents a tenfold difference in hydrogen ion concentration) would not be motivated to adjust the pH of a formulation of IFN-p to about 3.0 to about 5.0,

which is fifty to five thousand times more acidic than the preferred pH of 6.5 taught by Dorin et al.

Ex. 6 10043392 As noted above, injection molding and press-fitting processes operate on much different principles, which are not compatible with one another… If the socket housing of Borgen et al. was cooler than the socket liner, then it would be inoperable for its

intended purpose. Similarly, if the socket liner of Borgen et al. was hotter than the socket housing, then it would be inoperable

for its intended purpose. Accordingly, one of ordinary skill in the art would be discouraged from combining the injection

molding process of Jackson with the press-fitting process of Borgen et al.

Ex. 7 10046568 Moreover, Graves et al. actually teaches away from upsetting its careful chemical balance because of the adverse bubbling effects of certain additives, like film formers, polymers, and solvents. See col. 1, lines 39-67, discussing how the incorporation

of a particular copolymer into the nail formulations therein caused such adverse effects that "the product was not commercially

viable or useful." … In light of this teaching, one of ordinary skill would not have been motivated to incorporate at least one first polymer, as presently claimed, into the nail enamel compositions of Graves et al., for fear that it may render them

unsatisfactory for their intended purpose. See MPEP § 2143.01. And certainly a skilled artisan would have had no

reasonable expectation of success in incorporating such a polymer into the compositions of Graves et al. in an effort

achieve the claimed invention, in light of this teaching.

Ex. 8 10050432 Thus, Orthoefer et al. '122 teaches that sulfurous ion, such as salts of sulfite or bisulfite, should be added during the water

extraction step of the soy starting material to decrease the viscosity of the extract which, in turn, decreases the viscosity of the final soy isolate product. On the other hand, Sailer '554 explicitly teaches against the use of sulfurous ion in the water

extraction step of the soy starting material or any subsequent steps in the Sailer '554 process. For the foregoing reasons, Sailer

'544 and Orthoefer et al. '122 clearly teach away from one another regarding how to obtain a soy product having a desired viscosity. Therefore, one of ordinary skill in the art would not combine the disclosures of Sailer '554 and Orthoefer et al. '122

in the manner relied upon by the Examiner.

Ex. 9 10053013 Furthermore, Goertzel et al. teaches way from the claimed invention. Goertzel et al. assigns access based on virtual locations,

where "access is assigned based upon the least trusted location (i.e., the "weakest link") through which a user's packets are

being routed." Col. 6, lines 14-17. Goertzel et al. recites "[i]ndeed, a RAS 681, 682 dial-up user may be closer in physical

distance than [a] user at a remote office 641, connecting via a T1 line, even though the dial-up user will ordinarily be considered less secure." Col. 5, lines 20-23. Goertzel et al. thus teaches away from reliance on physical locations to

determine and select security levels, providing strong indicia of nonobvious of the present invention…

Ex. 10 10053085 A review of Cable and Isenberg makes clear that the cited references teach away from combination with one another. The Background section of Cable '903 expressly criticizes the electrodes described by Isenberg in US Patent 4,582,766, noting that

in devices made according to Isenberg, the electrolyte is bound to the electrode, which bonding results in undesirable

mechanical and structural complications (see Cable '903 at col. 2, lines 38-41)… The cited references likewise teach away from the claimed invention. While the claimed invention recites that the anode and electrolyte are secured to one

another, the cited Cable '903 reference expressly criticizes such bonding and instead advocates the disposition of so-called

"microslip zones" between the electrolyte and the electrode components (Cable '903 at col. 5, lines 53-68).

39

Ex. 11 10057198 Thus, Shanbrom conforms to the standard approach within the pharmaceutical industry of not heating and therefore not

vaporizing the medicament subject to thermal decomposition. Thus, Shanbrom teaches away from the present invention in

which degradable physiologically active compound is heated to form a vapor. As Shanbrom teaches to avoid heating of the

medicament, Shanbrom cannot be said to teach a method for decreasing the decomposition of the medicament upon heating…

and thus teaches away from the instant invention.

Ex. 12 10058495

Indeed, the very art on which the Examiner relies teaches the commonly known need to anneal UHMWPE to remove free radicals therefrom subsequent to irradiation crosslinking… No one skilled in the art would be motivated to radiation

crosslink the finished acetabular cup of Devanathan, as proposed by the Examiner, since to do so would require post-

irradiation annealing at a temperature which closely coincides with the melting temperature of PMMA. Unlike UHMWPE which remains workable above its melting temperature, PMMA turns to a liquid (with the approximate viscosity of

honey) above its melting temperature. As such, post-irradiation quenching would completely distort, and effectively destroy,

Devanathan's acetabular cup.

Ex. 13 10083205 In addition, the Ward et al. and Beaupre references teach away from perforating the aluminum vapor barrier layer of Beaupre.

Ward et al. teaches that any holes in the vapor barrier material are undesirable. Ward et al. states that "vapor can

penetrate [pin holes] to wet the insulation or condense on the underside of the superimposed metal decking. As is well known, wetted insulation has significantly reduced resistance to thermal conductivity than does the same insulation when dry." Ward

et al. col. 1, lines 43-52. Beaupre teaches that it is undesirable to deteriorate the vapor barrier characteristics of a foil vapor

barrier, as would certainly be the case if the foil layer were perforated… Claim 8 is not obvious in view of Beaupre, Ward et al. and Denenberg, because the Beaupre and Ward et al. patents teach away from perforating the aluminum vapor barrier layer

of Beaupre.

Ex. 14 10087662 The skilled reader of this reference would not consider it obvious to replace the silicon nitride taught by Glaser with silicon dioxide… Glaser repeatedly lauds the benefits of using high refractive index material for the lower coating… Further, Glaser

expressly indicates that the refractive index of the described layers should not be substantially modified… Thus, Glaser

teaches against any modification that would involve a substantial change in the refractive index of the described

films… Since replacing silicon nitride (or bismuth oxide or tin oxide) with silicon dioxide would result in a very substantial

decrease in refractive index, Applicant submits that this modification would be far from obvious.

Ex. 15 10090293 No such suggestion or teaching is provided by Reiley or Ferree. Rather, Ferree does not mention facet joint replacement or facet joint ailments, and Reiley teaches against the use of artificial discs where facet joint ailments are present… Indeed, not

only Reiley, but the general state of the prior art teaches away from the combination of facet joint replacement with use

of an artificial disc. As expressed in Applicants' specification, "contraindications for artificial discs include arthritic facet joints, absent facet joints, severe facet joint tropism or otherwise deformed facet joints."

Ex. 16 10090358 Substituting the invention of Scherson into the invention of Henley would defeat this purpose: creation of a sustained, oxygen-

enriched environment through oxygen generation from ambient air under hyperbaric pressure. Indeed, the negative pressure supplied by Henley to drain wound surfaces would entirely teach away from the hyperbaric pressure teaching of Scherson for

maintenance of an oxygen-rich environment. Accordingly, one of skill in the art would not think to modify Henley by

combining it with the device of Scherson, because, at the least, the two inventions teach away from each other.

Ex. 17 10093199 In fact, Yamaguro teaches to the contrary. As explained above, the invention of Yamaguro is designed for low temperature

engine environment. If a heater were installed at the hydrogen generating unit, the heat would dramatically affect the catalyst operation. Specifically, as shown in Figure 7 of Yamaguro, even a slight amount of additional heat at the catalyst inlet is

detrimental to NOx conversion efficiency (col. 6, lines 20-24). … He teaches away from the introduction of any additional

heat in the exhaust or near the exhaust inlet (Figure 7; col. 6, lines 20-24).

Ex. 18 10094253 Additionally, the primary teaching of Hossom is that components of the test device should be removable to increase the

flexibility of the device and to optimize the device for the particular assay to be performed. This runs directly counter to the

claimed apparatus, i.e., a guide tray having a one-piece construction. Thus, Hossom teaches away from providing a guide

tray having a one-piece construction.

Ex. 19 10094265 As stated above, Melis et al, clearly states that antagonists (or blockers) of the dopamine receptors reduce or eliminate the

sexual stimulation seen with agonists in mammals (see at least col. 2 on page 20, col. 1 on page 22, and col 2, under fluman Studies, page 32). Therefore, Kulagowski et al, in US 5,792,768 not only does not present a motivation to use the claimed

antagonists in treating sexual dysfunction, but also teaches away from the present invention. Accordingly, one skilled in the

art would not have a reasonable expectation of success in using the compounds of Kulagowski et al, in US 5,792,768 in

treating sexual dysfunction, especially in view of the teachings of Melis et at.

Ex. 20 10097257 However, those skilled in the art know that there are a multitude of reasons why the 1-15 nm particle size regime is generally

not accessible to traditional emulsion polymerization or dispersion polymerization techniques… For the invention described in Chandler, the practitioner would want the maximum effective concentration range to be as large as possible, and thus the

particle size to be as large as possible or practical. The applicants have, surprisingly, found that the use of very low particle

size polymeric nanoparticles actually reduces the viscosity of the dispersion (paragraph 0015), which is counter-intuitive

from the known art because smaller particle size usually results in higher viscosity… Given that the fields of invention

are so diverse, Chandler teaches away from our invention, and that one would anticipate increased viscosity by moving to

lower particle size, anyone of ordinary skill in the art would not consider Chandler to be prior art for the inventive step of very low particle size (1-15 nm) cross-linked polymeric nanoparticles with attached chromophores.

40

Figure 8. Examiner citations that teach away from focal invention

This figure shows the number of citations used by examiners to make obviousness rejections that actually teach away from the focal

inventions. Data come from applicants’ responses to examiners’ rejection letters. The figure compares high, medium, and low levels

of technological overlap between the focal invention and the examiner’s prior experience. The unit of observation is the application.

Dotted lines show the 95 percent confidence interval.

Table 7. Examiner citations that teach away from focal invention

This table reports results of a Poisson regression, in which the dependent variable is the number of examiner citations used for

obviousness rejections that teach away from the focal invention. The unit of observation is the application. The model includes fixed-

effects and clustered standard errors at the examiner level.

(1)

# examiner citations that teach away from

focal invention

ln(total claims) 0.270+ (0.154)

ln(law firm's average client size) -0.024

(0.061) # technology classes in applicant prior art -0.062*

(0.025)

Average year of applicant prior art -0.007 (0.015)

ln(applicant prior art citations) 0.486***

(0.100) % claims rejected for obviousness 0.819*

(0.334)

Number of examiner citations used for obviousness rejections 0.215*** (0.031)

Invention's technological overlap with examiner experience -0.949**

(0.321)

Observations 1,331

Number of examiners 242 Chi-sqr 149.7

df 8

Standard errors clustered by examiner; *** p<0.001, ** p<0.01, * p<0.05, + p<0.10

0.025

0.035

0.045

0.055

0.065

0.075

0.085

Low Medium High

# e

xa

min

er c

ita

tio

ns

tha

t te

ach

aw

ay

fro

m f

oca

l

inven

tio

n

Invention's technological overlap with examiner experience

41

Figure 9. Claims rejected due to obviousness that were later allowed

This figure shows the percentage of claims rejected due to obviousness that were later allowed. The unit of observation is the

examiner prior art citation, and percentages are calculated specifically for claims rejected based a given examiner citation. The

denominator is the number of claims rejected for obviousness by a given examiner citation, and the numerator is the number of these

rejected claims that were ultimately allowed. For 34 percent of the examiner citations used for obviousness rejections, all of the claims

rejected based on these citations were later allowed. For 37 percent of examiner citations used for obviousness rejections, none of the

claims rejected based on these citations were later allowed. This figure shows the distribution of percentage of claims allowed for the

remaining 29 percent of examiner citations for which some but not all of the claims rejected were later allowed.

Figure 10. Claims rejected due to obviousness that were later allowed

The vertical axis in this figure shows the percentage of claims rejected due to obviousness that were later allowed by the examiner.

The unit of observation is the examiner prior art citation, and percentages are calculated specifically for claims rejected based a given

examiner citation. The denominator is the number of claims rejected for obviousness by a given examiner citation, and the numerator

is the number of these rejected claims that were ultimately allowed. The figure compares allowances for examiner citations that do and

do not teach away from the focal invention. Dotted lines show the 95 percent confidence interval.

49.0%

51.0%

53.0%

55.0%

57.0%

59.0%

61.0%

63.0%

65.0%

67.0%

69.0%

Does not teach away Teaches away from focal invention

% o

f cla

ims

reje

cte

d f

or o

bvio

usn

ess

th

at

were l

ate

r a

llo

wed

Examiner citation used to reject claims for obviousness

42

Table 8. Regression models of claims allowed after being rejected

This table reports results from regression models of claims allowed after being rejected due to obviousness. The unit of observation is

an examiner prior art citation. All variables at the examiner citation level are computed using claims that were rejected for

obviousness using that examiner citation. In columns 1 through 3, the dependent variable is a percentage, in which the denominator is

the number of claims rejected for obviousness by a given examiner citation, and the numerator is the number of these rejected claims

that were ultimately allowed. These models are estimated using linear regression. In column 4, the dependent variable is the number of

claims that were rejected for obviousness using a given examiner citation but were ultimately allowed. Fixed-effects and clustered

errors are at the level of the examiner.

(1) (2) (3) (4)

Linear Poisson

% of claims rejected for obviousness

that were ultimately allowed

# of claims rejected due to

obviousness that were

ultimately allowed

Application level variables

ln(law firm's average client size) 0.006 0.005 0.005 0.003

(0.004) (0.004) (0.004) (0.010) # technology classes in applicant prior art -0.002 -0.002 -0.002 -0.004

(0.002) (0.002) (0.002) (0.006)

Average year of applicant prior art 0.002 0.002 0.002 0.003 (0.001) (0.001) (0.001) (0.003)

ln(applicant prior art citations) 0.013 0.014 0.014 0.032

(0.009) (0.009) (0.009) (0.020)

Examiner citation level variables

ln(# claims rejected due to obviousness) 0.006 0.006 0.006 0.995*** (0.005) (0.005) (0.005) (0.016)

Examiner citation teaches away from focal invention 0.104*** 0.103*** 0.102*** 0.143**

(0.025) (0.025) (0.025) (0.048) % of claims rejected due to obviousness that were also rejected for

other reasons 0.020

(0.017) % of claims rejected due to obviousness that were also rejected for

lack of novelty (§102) 0.002 -0.031

(0.016) (0.042) % of claims rejected due to obviousness that were also rejected for

insufficient disclosure (§112) 0.018 0.011

(0.021) (0.051) % of claims rejected due to obviousness that were also rejected for

non-patentable subject matter (§101) 0.037 0.130

(0.047) (0.108) Constant -3.312 -3.337 -3.303

(2.569) (2.570) (2.567)

Observations 19,673 19,673 19,673 18,469

Number of examiners 1,991 1,991 1,991 1,593

F-stat 4.755 4.378 3.350 . Chi-sqr . . . 4444

df 5 6 8 9

Standard errors clustered by examiner; *** p<0.001, ** p<0.01, * p<0.05, + p<0.10