• Home

  • Internet

  • Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource
18
Ian Milligan (@ianmilligan1) Assistant Professor of History [email protected] Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Embed Size (px)

DESCRIPTION

Here are the slides for the talk I gave at the Digital Humanities 2014 conference in Lausanne, Switzerland. Paper abstract is http://dharchive.org/paper/DH2014/Paper-83.xml.

Citation preview

Page 1: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Ian Milligan (@ianmilligan1) Assistant Professor of History [email protected]

Clustering Search to Navigate A Case

Study of the Canadian World Wide Web as a

Historical Resource

mailto:[email protected]
Page 2: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Why? !

Historians need to think about Computational Methods in an era of

web archives.

Page 3: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

INTERNET ARCHIVE~ 10,240 TBs

LIBRARY of CONGRESS~ 200 TBs

est. HOLDINGS:

Page 4: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

The 80TB Wide Web Scrape

[March - December 2011]

Page 5: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Wayback Machine

or WARC files?

Page 6: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Building a .ca sample: !

622,365 distinct URLs / 8,512,275 overall URLs =

7.31% in case study

Page 7: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

WARC Web ARChive file format

ISO 28500:2009

Page 8: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

filesdump.py available at https://github.com/ianmilligan1/Historian-WARC-1/tree/master/WARC/warc-tools-mandel

WARC File WARC-Tools/Lynx!(warcfilter.py,

warchtmlindex.py and filesdump.py)

Indexing

CDX Files !(finding aids)

https://github.com/ianmilligan1/Historian-WARC-1/tree/master/WARC/warc-tools-mandel
Page 9: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Full Text Index

Clustering Workbench

Other sorts of text

analysis

Page 10: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource
Page 11: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource
Page 12: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource
Page 13: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource
Page 14: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

https://github.com/ianmilligan1/Historian-WARC-1/tree/master/WARC/warc-tools-mandel

WARC File WARC-Tools/Lynx!(warchtmlindex.py and filesdump.py)

Indexing

https://github.com/ianmilligan1/Historian-WARC-1/tree/master/WARC/warc-tools-mandel
Page 15: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Downside is you still have to know what you’re looking for.

Page 16: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource
Page 17: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Playing with images?

Page 18: Clustering Search to Navigate A Case Study of the Canadian World Wide Web as a Historical Resource

Ian Milligan Assistant Professor of History [email protected]

Thanks (to you all and to funders).

!

http://ianmilligan.ca/

mailto:[email protected]
http://ianmilligan.ca/

Affinity Clustering: Hierarchical Clustering at Scalepapers.nips.cc/paper/7262-affinity-clustering-hierarchical-clustering-at-scale.pdf · Affinity Clustering: Hierarchical Clustering

Affinity Clustering: Hierarchical Clustering at Scalepapers.nips.cc/paper/7262-affinity-clustering-hierarchical-clustering-at-scale.pdf · Affinity Clustering: Hierarchical Clustering

Documents
FUZZY CLUSTERING 2009/2010. 2 What is Data Clustering? Fuzzy C-Means Clustering Subtractive Clustering Data Clustering Using the Clustering GUI

FUZZY CLUSTERING 2009/2010. 2 What is Data Clustering? Fuzzy C-Means Clustering Subtractive Clustering Data Clustering Using the Clustering GUI

Documents
Text-Mining: Clustering - Philosophische Fakultät · Clustering im TM Flaches Clustering Hierarchisches Clustering Erweiterungen, LabelingLiteratur Cluster-Hypothese \Documents in

Text-Mining: Clustering - Philosophische Fakultät · Clustering im TM Flaches Clustering Hierarchisches Clustering Erweiterungen, LabelingLiteratur Cluster-Hypothese \Documents in

Documents
Bi-Clustering. 2 Data Mining: Clustering Where K-means clustering minimizes

Bi-Clustering. 2 Data Mining: Clustering Where K-means clustering minimizes

Documents
Lecture 12: Clustering May 5, 2010. Clustering (Ch 16 and 17) Document clustering Motivations Document representations Success criteria Clustering

Lecture 12: Clustering May 5, 2010. Clustering (Ch 16 and 17) Document clustering Motivations Document representations Success criteria Clustering

Documents
Chapter19 Clustering Analysis. Content Similarity coefficient Hierarchical clustering analysis Dynamic clustering analysis Ordered sample clustering analysis

Chapter19 Clustering Analysis. Content Similarity coefficient Hierarchical clustering analysis Dynamic clustering analysis Ordered sample clustering analysis

Documents
Clustering Supervised vs. Unsupervised Learning Examples of clustering in Web IR Characteristics of clustering Clustering algorithms Cluster Labeling 1

Clustering Supervised vs. Unsupervised Learning Examples of clustering in Web IR Characteristics of clustering Clustering algorithms Cluster Labeling 1

Documents
Tutorial 8 Clustering 1. General Methods –Unsupervised Clustering Hierarchical clustering K-means clustering Expression data –GEO –UCSC –ArrayExpress

Tutorial 8 Clustering 1. General Methods –Unsupervised Clustering Hierarchical clustering K-means clustering Expression data –GEO –UCSC –ArrayExpress

Documents
Clustering: Partition Clustering

Clustering: Partition Clustering

Documents
Collaborative Clustering for Entity Clustering

Collaborative Clustering for Entity Clustering

Documents
ASA Clustering within VMDC Architecture - Cisco€¦ · ASA Clustering within VMDC Architecture ASA Clustering Overview ASA Clustering Overview The clustering feature for the ASA

ASA Clustering within VMDC Architecture - Cisco€¦ · ASA Clustering within VMDC Architecture ASA Clustering Overview ASA Clustering Overview The clustering feature for the ASA

Documents
Vladyslav Kolbasin Stable Clustering. Clustering data Clustering is part of exploratory process Standard definition: Clustering - grouping a set of

Vladyslav Kolbasin Stable Clustering. Clustering data Clustering is part of exploratory process Standard definition: Clustering - grouping a set of

Documents
Navigate sampler

Navigate sampler

Leadership & Management
Segmentation and Clustering · – Divisive and hierarchical clustering – k-means clustering – Mean shift clustering – Graph cuts •Applications – Image processing, object

Segmentation and Clustering · – Divisive and hierarchical clustering – k-means clustering – Mean shift clustering – Graph cuts •Applications – Image processing, object

Documents
DATA MINING - CLUSTERING. Clustering 4 Clustering - unsupervised classification 4 Clustering - the process of grouping physical or abstract objects into

DATA MINING - CLUSTERING. Clustering 4 Clustering - unsupervised classification 4 Clustering - the process of grouping physical or abstract objects into

Documents
Modul clustering data mining modul clustering

Modul clustering data mining modul clustering

Data & Analytics
The Canadian Trade Commissioner Service - NEIAneia.org/wp-content/uploads/2013/01/6-Steps-to-CE-Marketing.pdfThe Canadian Trade Commissioner Service helps companies navigate the complexities

The Canadian Trade Commissioner Service - NEIAneia.org/wp-content/uploads/2013/01/6-Steps-to-CE-Marketing.pdfThe Canadian Trade Commissioner Service helps companies navigate the complexities

Documents
Guest Lecture: Clusteringcvml.ist.ac.at/talks/clustering-core2018.pdfsingle linkage clustering, complete linkage clustering, average linkage clustering Graph-based clustering spectral

Guest Lecture: Clusteringcvml.ist.ac.at/talks/clustering-core2018.pdfsingle linkage clustering, complete linkage clustering, average linkage clustering Graph-based clustering spectral

Documents
Clustering in Ratemaking: Applications in Territories ... · Clustering in Ratemaking: Applications in Territories Clustering OVERVIEW OF CLUSTERING ¾Purpose of Clustering in Insurance

Clustering in Ratemaking: Applications in Territories ... · Clustering in Ratemaking: Applications in Territories Clustering OVERVIEW OF CLUSTERING ¾Purpose of Clustering in Insurance

Documents
BIRCH: Balanced Iterative Reducing and Clustering Using Hierarchies A hierarchical clustering method. It introduces two concepts : Clustering feature Clustering

BIRCH: Balanced Iterative Reducing and Clustering Using Hierarchies A hierarchical clustering method. It introduces two concepts : Clustering feature Clustering

Documents
Navigate La

Navigate La

Documents
Navigate Family Series - Episode 4 Navigate Family Series. · Navigate Family Series - Episode 4 Hello and welcome to the resource material for our Navigate Family Series. Thank you

Navigate Family Series - Episode 4 Navigate Family Series. · Navigate Family Series - Episode 4 Hello and welcome to the resource material for our Navigate Family Series. Thank you

Documents
Ward’s Hierarchical Clustering Method: Clustering

Ward’s Hierarchical Clustering Method: Clustering

Documents
RISK TAKING IN UNIVERSITY MULTICULTURAL CLASSROOMS · 2018-05-31 · engagement with indigenous knowledges, and help navigate cultural appropriation. We, as a mixed-race Canadian

RISK TAKING IN UNIVERSITY MULTICULTURAL CLASSROOMS · 2018-05-31 · engagement with indigenous knowledges, and help navigate cultural appropriation. We, as a mixed-race Canadian

Documents
LNCS 3776 - Data Clustering: A User’s Dilemmabiometrics.cse.msu.edu/Publications/Clustering/JainLawClustering05… · Data Clustering: A User’s Dilemma 3 performing clustering,

LNCS 3776 - Data Clustering: A User’s Dilemmabiometrics.cse.msu.edu/Publications/Clustering/JainLawClustering05… · Data Clustering: A User’s Dilemma 3 performing clustering,

Documents
Clustering Clustering

Clustering Clustering

Documents
Clustering IV. Outline Impossibility theorem for clustering Density-based clustering and subspace clustering Bi-clustering or co-clustering

Clustering IV. Outline Impossibility theorem for clustering Density-based clustering and subspace clustering Bi-clustering or co-clustering

Documents
Navigate Tomorrow

Navigate Tomorrow

Documents
A STUDENT GUIDE TO USING NAVIGATE NAVIGATE STUDENT GUIDE

A STUDENT GUIDE TO USING NAVIGATE NAVIGATE STUDENT GUIDE

Documents
CSE601 Clustering Ensemble - University at Buffalojing/cse601/fa12/materials/clustering_ensem… · – Clustering ensemble – Clustering in MapReduce – Semi-supervised clustering,

CSE601 Clustering Ensemble - University at Buffalojing/cse601/fa12/materials/clustering_ensem… · – Clustering ensemble – Clustering in MapReduce – Semi-supervised clustering,

Documents