Network Topology for Search Service

DeploymentSunita Shrivastava

Protection with AuthZ in AT with Load Balancers sending requests to AT

AT

Job

ES

Load Balancer

Private NetworkLoad

Bal

ance

r

Questions : Is a Load Balancer in front of ES Required?NEST Client seems to support load balancing of requests across a set of nodes

Protection with a Gateway (App Insight Model?)

AT

Job

ES

Load Balancer

Gateway

Load

Bal

ance

r

Questions : Gateway Availability, AT and Job in Separate Private Network

Private Network

References• http://azure.microsoft.com/blog/2014/01/07/new-windows-azure-ne

twork-security-whitepaper/

Shared Elastic Search Cluster across VSO Services• Requirements for supporting agility/isolation

• Separate Data Nodes• Separate Query/Aggregation Nodes for each Service• Separation of workload/capacity planning

• Motivation• Required for on-prem• Some of the work/analysis is mandated by ALM Search Search• Belief that it will simplify monitoring

• We need to ensure this • What are the key metrics that each workload would require monitoring?

• Aggregations across queries will be facilitated, in certain scenarios

• Concern • ALM Search Uses plugins

• How do we deal with fixes in plugins• Are our operational models really the same• Impact of capacity planning or index management excercises• Shared asset would be the network

• Do we understand the implications of the Test Results traffic (or bandwidth use)• If we can build some understanding of the network bandwidths consumed by our workloads?

ES Inside the ALM Search VNET (Opt 1)

AT

Job

ES

Load Balancer

Private Network

Inte

rnal

Loa

d Ba

lanc

erALM Search Service

AT

Job

VSO Service (Test Results Service)

TR Data Nodes

AT

Job

AT

Job

Search Data

Nodes

TR Query/Indexin

g Nodes

ALM Search Query/Indexing

Nodes

Shared Master Nodes

Load

Bal

ance

r

TR Query/Indexin

g NodesTRS

Query/Indexing Nodes

TR Data NodesTR Data

Nodes

ALM Search Query/Indexing

NodesALM Search

Query/Indexing Nodes

Search Data

Nodes

Search Data

Nodes

Option 1• Search• Authentication for user in ALM Search Service• Authorization for indexes is in Auth Checks in ALM Search Service AT

• Test Results• Authentication of the user

Securing the ES cluster[TBD] Authentication at ES Cluster

Comments

Jetty SSL/Basic Auth Replacement for NettyHttpServerTransport,Caller is authenticated, Certs are required

NGinx A Reverse Proxy soln used by other ES customers

VNET to VNET Gateway

None 1) Allow only nodes within the vnet in which TRS Service resides access to ES nodes

VSSF Based Proxy Use S2S calls between clients and gateway

Salyh/Elastic Search Security Plugin

If we use ALM Search indexing pipelines for TRS as well

AT

Job

ES

Load Balancer

Private Network

Inte

rnal

Loa

d Ba

lanc

er

ALM Search Service

AT

Job

TR Data Nodes

Search Data

Nodes

TR Query/Indexin

g Nodes

ALM Search Query/Indexing

Nodes

Shared Master Nodes

TR Query/Indexin

g NodesTRS

Query/Indexing Nodes

TR Data NodesTR Data

Nodes

ALM Search Query/Indexing

NodesALM Search

Query/Indexing Nodes

Search Data

NodesSearch Data

Nodes

VSO Service (Test Results Service)

AT

Job

AT

Job

Crawl

VSO Service (TFS)

AT

Job

AT

Job

Crawl Test Results