Fried sp techcon hybrid search deeper dive

Hybrid SharePoint with the new Cloud Hybrid Search

Jeff Fried CTO, BA Insight

Cloud Search Service Application

• Unified index with on-premises and cloud content

• Feeds Office Graph/Delve experiences

• Supports Search as a Service

• Reduces search server footprint

Audio textAnd search

indexpropertiessignalsMetadata extraction and processing

2 Hybrid SharePoint sessions on Monday 4-5:15 – pick one, review the other later

SharePoint Hybrid: The Sure Path Forward - Ben Curry

Debunking the Hybrid SharePoint Infrastructure Dilemma - Jill Hannemann & Adam Levithan

2 Hybrid search sessions on Tuesday – go to one or both

11:45am The Future of Microsoft Search is Here! Cloud SSA - Jeff Fried & Ben Curry

3:45pm Hybrid SharePoint with the new Cloud Hybrid Search - Jeff Fried

Hybrid Sessions at

Focused on Search and

SharePoint since 2004

Longtime

Search Nerd

• CTO, BA Insight

• Senior PM, Microsoft

• VP, FAST

• SVP, LingoMotors

About Jeff Fried

Passionate About

• Search

• SharePoint

• Search-driven

applications

• Information Strategy

Blog:

DoMoreWithSearch.com

Technet Column

“A View from the

Crawlspace”

jeff.fried@bainsight.com

About BA Insight

– Connectivity

– Applications - Im

– Classification -

– Analytics

–

–

–

–

–

Why Hybrid SharePoint?

7

Approaches to Hybrid – by Workload

Split Workload

different tools in

different places

Split User

task uses content or

sites across ‘the divide’

Exchange, SharePoint, Lync

OneDrive, Yammer, PowerBI, Delve

Extranet, Mysites, Team Sites, Project Sites

Portals, Intranet, Services/Applications

Links Search

Online

On-Prem

Cloud Hybrid Search

Cloud

SSA

Text &

Metadata

Online

On-Prem

Logical Architecture: Crawling

Cloud SSACloud SSA

ParseCrawl

SCS

ACL Map Process

Blob

storequeue

Online

On-Prem

Query

processing

Logical Architecture: Query

Cloud

SSA

Online

On-Prem

Logical Architecture: Query

Cloud SSA

Query

processing

Online

On-Prem

Failure mode: what if you can’t reach the cloud?

Cloud

SSA

Online

On-Prem

Combination: double crawling

Cloud

SSA

Text &

Metadata

Mechanisms

Cloud SSA

Remote Result Source

Cloud App model

Add-ins

External Content

Federation

Identity and Directory Sync

Benefits of Cloud Hybrid Search

External Content

(on-premises and/or

in the cloud)

SharePoint Server

(On-premises or Hosted)Office 365

SharePoint Online Content

Onedrive for Business Content

Co

nnect

ors

SharePoint Content

Adding External Content

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

BA Insight ConnectorsMailbox and Archiving Systems• Microsoft Exchange

• Microsoft Exchange Online

• IBM Lotus Notes

• Symantex Evault

• Autonomy EAS / (Zantaz)

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

ERP and Portal Systems•

•

•

•

•

•

•

•

•

•

•

•

Plus a proven architecture and process for creating new

connectors to complex systems

External Content in O365 UX

Unified view across all content

- on-premises and on-line

- inside and outside SharePoint

Scaling

External Content

(on-premises and/or

in the cloud)

Custom

Processing

CEWS

Bottlenecks:

1) Source systems

2) Content Processing

3) Indexer

….

External Content

(on-premises and/or

in the cloud)

Bottlenecks:

1) Uplink

2) Source systems

….

24

Performance

External Content

(on-premises and/or

in the cloud)

CEWS

Custom

Processing

Bottlenecks:

1) Uplink

2) Source systems

3) Content Processing

….

Performance Monitoring and Bandwidth (Get-Counter -ListSet "Search Gatherer Azure Plugin - SharePointServerSearch").counter

500K items crawled on an Azure D3

50 DPS 100 DPS

1 hour

Less servers is OK

•

•

•

•

•

Directory Synchronization

SID S-1-5-21-1212121212-1212121212-1212

jaden@corp.hybridsearch.com

msOnline-

OnPremiseSecurity

Identifier

S-1-5-21-1212121212-1212121212-1212

PUID PUID-XXXX-XXXXXXXXXX

Mapping of Access Control Lists

Allow: S-1-5-21-1212121212-1212121212-1212

PUID-XXXX-XXXXXXXXXX

• User SIDs are mapped to PUIDs

• Group SIDs are mapped to Object IDs

• «Everyone» and «Authenticated users» are mapped to

«Everyone except external users»

SUPPORTED

– Custom IFilter

– BCS connectors

– Partner connectors

Customizations: Supported & Unsupported

SUPPORTED

– Tenant level schema mapping

– Query rules

– Result sources

On-premises In the cloud

NOT SUPPORTED

• Content that requires custom security trimming

NOT SUPPORTED

• Site collection level schema mapping

• Custom security trimming

• Custom entity extraction

• Content enrichment web service

1)

2)

3)

Cloud Hybrid Search Limitations + Workarounds

33

Feature OOB Limitation BA Insight

CEWS not available with Cloud SSA available via connector framework

Entity Extraction not available with Cloud SSA available via autoclassifier

Custom Security Trimming not available with O365 index can 'map down' to AD groups

ThesaurusSharePoint Online doesn't support a thesaurus

can use Federator - with SP server-

based search center

Removal of on-premises

search results not available with Cloud SSA (could provide a custom solution)

External Content

(on-premises and/or

in the cloud)

SharePoint Server

(On-premises or Hosted)

SPO Content

OneDrive Content

Co

nnect

ors SharePoint Content

Connector

Framework

Office 365

AutoClassifier

(app version)

CEWS

Custom

Processing

External Content

(on-premises and/or

in the cloud)

SharePoint Server

(On-premises or Hosted)

SPO Content

OneDrive Content

Co

nnect

ors SharePoint Content

Connector

Framework

AutoClassifier

Office 365

AutoClassifier

(app version)

CEWS

DLP Sensitive Data Search works with hybrid

Search for sensitive data across on-premises and SharePoint Online

All Built-in sensitive types

Identification and export

Extends to data in OneDrive

Sensitive Information type detection through KQL searches

Get instant statistics

Preview & export results

Right now: only when you query for it

A global single index solution

Cloud SSA

Cloud SSA

Cloud SSA

Cloud SSA

Cloud SSA

–

–

–

NOT OOB …. but there’s a way to handle them all

Scenarios

40

41

Connectors

Federator

OOB Federated Search User Experience

Results from Cloud

Results from SharePoint On-Premise

Refiners from Cloud only

No termset

synchronization

Result Blocks

(not interleaved)

BA Insight Federator

45

Full Range of Hybrid Search Configurations

ScenarioMost systems and portals

hosted on-premises

Most systems and portals

hosted in the cloud

Must work across borders but

maintain data residency

Single Single MultiSearch

Search Index Search Index Across Multiple

in SP Server in SP Online * Search Indices

How it works

Crawls SP Online and

other sources from SP

Server

Crawls SP Server and other

sources from Cloud SSA, pushes

text & metadata to SP Online

Searches SP Online and SP

Server simultaneously;

combines the results

AdvantagesSimplest approach;

best search experience

Low footprint on-premises;

can use online features

(Delve, DLP)

Only solution for some

scenarios

BA Insight

Improvement over OOBno OOB solution

Adds content outside SP

Preview content outside SP2013

Supports content enrichment

Provides single interleaved

result set and refiners

* requires Microsoft Cloud SSA

Approaches for Hybrid SharePoint

Configuration

Should I run index reset?

NO!

Best Practice:Content Source Naming & Deletion

{ { { {

Action CrawlDB state Office 365 index

state

User’s view

Create contentsource1_v1

Crawl doc1

doc1 in crawldb Doc1 indexed Doc1 is searchable

“index reset” <empty> Doc1 indexed Doc1 is searchable

Create a result source to exclude

contentsource1_v1 from the tenant &

search center site collections

<empty> Doc1 indexed Doc1 is no longer searchable

Tenant Admin opens SR to delete ALL

cloud SSA content.

<empty> <empty> All external content has been

removed

Create contentsource1_v2

Crawl doc1

doc1 in crawldb Doc1 indexed Doc1 searchable again

Orphaned Content

50

Customer Example: ACE

Built on SharePoint 2013 – but couldn’t run as-is in O365

SharePoint Server in Azurein hybrid configuration with O365 Tenant

Virtual Network

Cloud Service

Availability Set

Active Directory & DNS

Cloud Service Cloud Service

Availability Set

Front End

Availability Set

App server

Availability Set

Database

Microsoft Azure

Gatewaysubnet

Active VPN

On-premises environment

Optional!

Example: Using Search-First Migration with Hybrid

Cloud Service

Availability Sets

SharePoint Services Farm

Microsoft Azure

SharePoint Online

Site collections

Office 365 Tenant

SharePoint 2013 Content Farms

SharePoint 2010 Farm(s)

2) Migrate / UpgradeContent Farms

Each site collection can be moved independentlyCan be on-premises, in O365, or hosted in Azure

3) Decommission old farm(s)

1) Establish Search Service(using Azure IaaS)

Key Considerations for Hybrid: Workloads, Environment, Data, Customizations

Availability of features Online versus

On-Premises on particular workloads

Significant investments in

customization of On-Premises

workloads

Concerns over global network

performance with remote sites

Regulatory

considerations

Manageability concerns

References

http://technet.microsoft.com/en-us/library/dn197172(v=office.15).aspx

http://sp2013searchtool.codeplex.com/

https://github.com/OfficeDev/PnP-Tools/tree/master/Scripts/SharePoint.Hybrid.Search.Configuration

References - Blogs

http://blogs.msdn.com/b/spses/archive/2015/09/15/cloud-hybrid-search-service-application.aspx

http://blogs.msdn.com/b/spses/archive/2013/10/22/office-365-configure-hybrid-search-with-directory-synchronization.aspx

http://blogs.msdn.com/b/spses/archive/2014/01/05/office-365-configure-hybrid-search-with-directory-synchronization-password-sync-part2.aspx

http://blogs.msdn.com/b/spses/archive/2014/01/07/identity-federation-amp-single-sign-on-deployment-for-hybrid-search-in-office-365-sharepoint-online-part3.aspx

http://blogs.msdn.com/b/spses/archive/2015/03/19/configuring-microsoft-web-application-proxy-server-for-inbound-hybrid-topology-with-office-365-and-microsoft-sharepoint-server-2013-part7.aspx

https://www.youtube.com/watch?v=JWEZx9SHDb0&list=PLvmwu6WYeFdjNbiy7SISJAZd1HjzIJoz5

https://azure.microsoft.com/en-us/documentation/articles/active-directory-aadconnect/

https://azure.microsoft.com/en-us/documentation/articles/active-directory-aadconnect/

http://blogs.msdn.com/b/spses/archive/2015/09/15/cloud-hybrid-search-service-application.aspx

References – Installing with SP2016

Tools

62

New Sites to bookmark

63

Contact:Jeff.Fried@BAinsight.comwww.BAinsight.com

Questions /Discussion