Upload
catalyst-repository-systems
View
220
Download
0
Embed Size (px)
Citation preview
877.557.4273
catalystsecure.com
Using TAR 2.0 to Reduce Review Costs Moderator: Michael Arkfeld Speakers:
David Stanton
John Tredennick
Thomas C. Gricks
A primer for legal professionals
Webinar
v
John is a former trial lawyer and litigation
partner with a large national law firm, and has
written or edited five books and countless articles on litigation and
technology issues. He was recently named one of the top six e-
discovery trailblazers by The American Lawyer. He was also named
one of the “Top 100 Global Technology Leaders” by London's
CityTech magazine. John served as chair of the ABA Law Practice
Management Section and editor-in-chief of its flagship magazine.
Speakers John Tredennick
A prominent e-discovery lawyer and one of the
nation's leading authorities on the use of TAR in
litigation, Tom joined Catalyst in June. He advises corporations and law
firms on best practices for applying Catalyst's TAR technology, Insight
Predict, to reduce the time and cost of discovery. He has more than 25
years’ experience as a trial lawyer and in-house counsel, most recently
with the law firm Schnader Harrison Segal & Lewis, where he was a
partner and chair of the e-Discovery Practice Group.
Thomas C. Gricks III
David Stanton is a partner in the law firm’s
Litigation practice. He leads the firm’s nationally
recognized Information Law and Electronic Discovery group, oversees
the firm’s nationwide Litigation Support department, and he is a
member of Pillsbury’s Privacy, Data Security & Information Use
group. Mr. Stanton is also a member of the firm's Professional
Responsibility Committee and he serves as Pillsbury’s Executive
Partner for Anti-Bribery/Anti-Corruption Compliance.
David Stanton
Michael Arkfeld is a consultant, litigator, educator and
author. Michael is the Founder and Director of
Education for eDiscovery Education Center and Director of the Arkfeld
eDiscovery and Digital Program at the Sandra Day O’Connor College of
Law at Arizona State University. This program hosts the annual ASU-
Arkfeld eDiscovery and Digital Evidence Conference held in March of
each year.
Michael Arkfeld
CEO & Founder
Partner, Pillsbury Winthrop Shaw Pittman
Managing Director
Moderator
Founder, eDiscovery Education Center
Case Study: Bank Production Review
Collection: 2.1 million documents
Initial relevance (richness): 1%
Review team used CAL
Tagged relevance increased to 25 to 35%
Then relevance dropped toward zero
What is Technology Assisted Review?
1. A process through which humans work with a
computer to teach it to identify relevant
documents.
2. Ordering documents by relevance for more
efficient review.
3. Stopping the review after you have reviewed a
high percentage of relevant documents.
How Does it Work?
Support Vector Machines
Naïve Bayes
K-Nearest Neighbor
Geospatial Predictive Modeling
Latent Semantic
How Does it Work?
Support Vector Machines
Naïve Bayes
K-Nearest Neighbor
Geospatial Predictive Modeling
Latent Semantic
"I may be less interested in the science
behind the "black box” than in whether it
produced responsive documents.“
Judge Andrew Peck (SDNY)
What Are the Savings?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Percen
tageofR
elevan
tDocum
entsFou
nd(R
ecall)
PercentageofDocumentsReviewed
YieldCurve
Linear Review
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
YieldCurve
%ofDocuments
%Relevan
t
What Are the Savings?
Linear Review
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
YieldCurve
%ofDocuments
%Relevan
t
What Are the Savings?
Review 12% and get 80% relevant documents
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
YieldCurve
%ofDocuments
%Relevan
t
What Are the Savings?
Review 24% and get 90% relevant documents
What is TAR 2.0?
1. Continuous Active Learning is key
(one bite at the apple is not enough).
2. Reviewers train while they review.
No SME required.
3. Ranking is against all the documents.
No control set involved.
4. Properly tokenizes foreign language
documents.
5. Contextual diversity helps find what you
don’t know. Random sampling not required.
Vulnerabilities of TAR 1.0
Continuous Active Learning is key
(one bite at the apple is not enough).
Reviewers train while they review.
No SME required.
Ranking is against all the documents.
No control set involved.
Properly tokenizes foreign language
documents.
Contextual diversity helps find what you
don’t know. Random sampling not required.
TAR 2.0 – Harnessing the Wisdom of the Crowd
A large group's aggregated answers to
estimation questions have been found to be
as good as, or better than, the answer given
by any of the individuals.
TAR 2.0 operates as a mechanism for
aggregating the wisdom of the crowd.
A crowd's individual judgments can be
modeled as a probability distribution of
responses with the mean centered near the
true mean of the quantity to be estimated.
TAR 2.0 - Defining Responsiveness
Crowds tend to work best when there is a correct answer to the
question being posed, such as a question about geography or
mathematics.
Critical to define and train/drill team on meaning of responsiveness.
Provide context
Present examples
Pose hypotheticals
Dialogue and debate
Issue tags as “categories of responsiveness”
Inclusive
Complete
Quality Control – TAR 1.0 v. TAR 2.0
Lawyers have an ethical obligation to
appropriately supervise document reviews
and productions.
In TAR workflows, supervisory responsibility
requires a mechanism to obtain insight into
the quality of the assessments being made
by the algorithm, and to respond to them.
Sampling
TAR 2.0 ranking/categorization details.
About the Collection
1. Predict operates on text
2. Any collection, any time
Rolling collections are seamless
3. Incorporated into Ranked Collection
Initial collection is random
Rolling collections rank by existing coding
Certain documents are unranked
Training is Immediate, Ubiquitous & Constant
1. You can use any documents to seed
Random or judgmental seeds
Synthetic Seeds
2. You can use any number of documents
3. EVERY attorney decision is used
Review
Search
“One-touch” coding
Ranking Occurs Every Several Minutes
Reviewer decisions between rankings:
1. Rank every 10 minutes
10 reviewer decisions per ranking
2. Rank once per day
480 reviewer decisions per ranking
Waiting until the end of the day loses the
benefit of 48x the reviewers decisions…
Separate QC for positive calls and negative calls
Catalyst Algorithmic Quality Control
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
1. Model the NEGATIVE documents
2. Rank only positive-coded documents
by their likelihood of being negative
1. Check the Positive Calls 2. Check the Negative Calls
1. Model the POSITIVE documents
2. Rank only negative-coded documents
by their likelihood of being positive
X
X
X
X
X
X
X
X
X
X
Overturn
Overturn
Overturn
Overturn
Overturn
Overturn
“In evaluating whether search terms or search methods
employed to carry out the search were appropriate, the
court applies a reasonableness test to determine the
adequacy of search methodology. An adequate search is
one that could . . . have been expected to produce the
information requested.”
Treppel v. Biovail Corp., 233 F.R.D. 363, 373-374 (S.D.N.Y. 2006)(Francis, J.)
Eurand, Inc. v. Mylan Pharms., Inc., 266 F.R.D. 79, 85 (D. Del. 2010)
Rule 26(g)(1) - “certifies that to the best of the person’s knowledge, information, and belief formed after a reasonable inquiry . . .”
The Law of Search – “Reasonably Comprehensive”
“This judicial opinion now recognizes that
computer-assisted review is an
acceptable way to search for relevant ESI
in appropriate cases.”
Da Silva Moore v. Publicis Groupe
(S.D.N.Y. 2012)
Go Ahead, Dive In!
Unanimous Approval
Nat’l Day Laborer Org. Network v. U.S. Immigration & Customs Enforcement
Agency, No. 10 Civ. 2488 (SAS), 2012 WL 2878130 (S.D.N.Y. July 13, 2012).
EORHB, Inc. v. HOA Holdings, LLC, No. 7409-VCL (Del. Ch. Oct. 15, 2012).
Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL 61040 (Vir. Cir. Ct. Apr.
23, 2012).
In re Actos (Pioglitazone) Prods. Liab. Litig., MDL No. 6:11-MD-2299 (W.D. La.
July 27, 2012).
2012
Unanimous Approval
Gordon v. Kaleida Health, No. 08-CV-378S(F), 2013 WL 2250579 (W.D.N.Y.
May 21, 2013).
In re Biomet M2a Magnum Hip Implant Prods. Liab. Litig., 2013 U.S. Dist.
LEXIS 84440 (N.D. Ind. Apr. 18, 2013).
In re Biomet M2a Magnum Hip Implant Prods. Liab. Litig., 2013 U.S. Dist.
LEXIS 172570 (N.D. Ind. Aug. 21, 2013).
EORHB, Inc. v. HOA Holdings, LLC, No. 7409-VCL, 2013 WL 1960621 (Del.
Ch. May 6, 2013).
Gabriel Techs., Corp. v. Qualcomm, Inc., No. 08CV1992 AJB (MDD), 2013
WL 410103 (S.D. Cal. Feb. 1, 2013).
2013
Unanimous Approval
Bridgestone Americas, Inc. v. Int. Bus. Machs. Corp., No. 3:13-1196 (M.D.
Tenn. July 22, 2014).
Dynamo Holdings Ltd. P’ship v. Comm’r of Internal Revenue, Nos. 2685-11,
8393-12 (T.C. Sept. 17, 2014).
FDIC v. Bowden, No. CV413-245, 2014 WL 2548137 (S.D. Ga. June 6, 2014).
In re Bridgepoint Educ., Inc., No. 12cv1737 JM (JLB), 2014 WL 3867495 (S.D.
Cal. Aug. 6, 2014).
Progressive Cas. Ins. Co. v. Delaney, No. 2:11-cv-00678-LRH-PAL, 2014 WL
2112927 (D. Nev. May 20, 2014).
2014
“The case law has developed
to the point where it is black
letter law that if a party wants
to utilize TAR for document
review, courts will permit it.”
The Law
Outbound productions
Reduce review costs and time dramatically by
establishing a cutoff point and stopping review.
Save up to 80-90% or more on review costs!
Run a privilege QC before production.
Sleep better at night!
How Can I Use it?
In-bound productions
In-bound productions allow your team to find
hot documents for depositions and trial.
Find relevant documents in a fraction of the time!
How Can I Use it?
Early case assessment
The SEC and DOJ actively use TAR to get a quick
handle on documents produced in their
investigations.
Greg Buckles, eDJ Analyst
How Can I Use it?
Focus team on multiple issues during review to
save time and effort
Issue/custodian review
How Can I Use it?
Non-English documents
TAR works effectively on any language so long
as you properly process the language.
How Can I Use it?
v
John is a former trial lawyer and litigation
partner with a large national law firm, and has
written or edited five books and countless articles on litigation and
technology issues. He was recently named one of the top six e-
discovery trailblazers by The American Lawyer. He was also named
one of the “Top 100 Global Technology Leaders” by London's
CityTech magazine. John served as chair of the ABA Law Practice
Management Section and editor-in-chief of its flagship magazine.
Speakers John Tredennick
A prominent e-discovery lawyer and one of the
nation's leading authorities on the use of TAR in
litigation, Tom joined Catalyst in June. He advises corporations and law
firms on best practices for applying Catalyst's TAR technology, Insight
Predict, to reduce the time and cost of discovery. He has more than 25
years’ experience as a trial lawyer and in-house counsel, most recently
with the law firm Schnader Harrison Segal & Lewis, where he was a
partner and chair of the e-Discovery Practice Group.
Thomas C. Gricks III
David Stanton is a partner in the law firm’s
Litigation practice. He leads the firm’s nationally
recognized Information Law and Electronic Discovery group, oversees
the firm’s nationwide Litigation Support department, and he is a
member of Pillsbury’s Privacy, Data Security & Information Use
group. Mr. Stanton is also a member of the firm's Professional
Responsibility Committee and he serves as Pillsbury’s Executive
Partner for Anti-Bribery/Anti-Corruption Compliance.
David Stanton
Michael Arkfeld is a consultant, litigator, educator and
author. Michael is the Founder and Director of
Education for eDiscovery Education Center and Director of the Arkfeld
eDiscovery and Digital Program at the Sandra Day O’Connor College of
Law at Arizona State University. This program hosts the annual ASU-
Arkfeld eDiscovery and Digital Evidence Conference held in March of
each year.
Michael Arkfeld
CEO & Founder
Partner, Pillsbury Winthrop Shaw Pittman
Managing Director
Moderator
Founder, eDiscovery Education Center
Catalyst designs, builds and hosts the world’s fastest
and most powerful document repositories for large-
scale discovery and regulatory compliance.
We back our award-winning technology with a highly skilled Professional Services team
and a global partner network to ensure the best e-discovery experience possible.
To view more the recording of this – and other Catalyst webinars – please visit:
http://catalystsecure.com/resources/events-and-webinars/on-demand-webinars