Europe's largest Internet Service Platform · Physical Configurations (SP,IP-Net, Storage) Distributed Service to support 50 Mio customers Web Cache Service Why are Telco's invest

Dr.Stefan Radtke

Dr. Stefan Radtke

Europe's largest Internet Service Platform powered by RS/6000SP Systems -A Technical Overview -

[email protected]

Dr.Stefan Radtke

TopicsInternet Service Platform - Introduction

Customer Groups of the PlatformCore Components of DT's Platform developed by IBM

Billing Broker ComponentBilling Broker Component as a Core ServiceCharacteristics of the Dial In ServiceUser Dial In ScenarioHow high availability is implementedPhysical Configurations (SP,IP-Net, Storage)Distributed Service to support 50 Mio customers

Web Cache ServiceWhy are Telco's invest in caching Object size distribution vs. bandwidth requirementsHow does the Websphere Edge Server Cluster workHardware platformGrowth of IP trafficStatistic and performance tools

Miscellaneous (...only when sufficient time left)Caching of Streaming Data (RTSP)Software distributionPhysical RS/6000SP configurations

Dr.Stefan Radtke

Customer Groups of the ISP

IP Platform

Open Communication

Closed Communication

Virtual Internet Provider

- T-Online- ....- ....

Virtual Private Networks

- Corporate Networks- ...

Realization ofVPNs

based on the samearchitecture

Today more than 150Virtual Internet Provider

are served by the platformwith about 10 Mio. users

Authentication, Authorization, Accounting

Content Provider

- Email- Web Sites- Video On

Demand - ASP...

Content Provider offer Services

to known Users

Services

Dr.Stefan Radtke

Core Components of the ISPBilling Broker Component(Protocol = RADIUS, Remote Access Dial In User Service) Distributed and higly available application for AAA-Services: - Authentication - Authorization - Accounting Handles about 10 Million users today but designed for 50 Mil.

Web Caching Scalable solution for - Saving bandwith - Faster Web access - Saving peering costs Serves more than 500.000.000 HTTP Requests per day to endusers

- Video on Demand Video platform for vendor independent video formats

Dr.Stefan Radtke


Customer Groups of the PlatformCore Services of DT's Platform developed by IBM



Miscellaneous (...only when sufficient time left)Caching of Streaming Data (RTSP)Software distribution

Physical RS/6000SP configurations

Dr.Stefan Radtke

Central Component of the ISP

Billing Broker Component

ISDN

Network Access Server

NAS

ADSL

RADIUS

E-MailServer

POP,IMAP

RADIUS *)

Video Server

RTSP

Video Provider

CustomerCareCustomerCare

Billing

VIP / VPN / ASP

CustomerCareCustomerCare

BillingCustomerCareCustomerCare

Billingftp

Analog

ISP Dial In ServiceASP E-Mail

ASP Video on Demand

ASP 1

ASP 3ASP 2

e.g. Media Hosting,Web Hosting

RADIUS *)

RADIUS *)

Mobile

MQSeries RADIUS

Dr.Stefan Radtke

Characteristics of Dial In Service Distributed

POPs are georaphically distributed 13 POPs with RADIUS-Proxy and RADIUS-LocalServer Instances Central Database with Millions of Data Records are replicated to 13 POPs Distributed Database

Scaleable Distributed Architecture is designed for more than 50 Million Users Distributed Database & Web-Cache 3 Tier RADIUS Architecture

Highly Available Service are all highly available through Application redundance All important infrastructure components are redundant

Driven by IBM TechnologyRADIUS-Proxy Server and Local Server running on AIX, PSSP, RS/6000SPIBM Middleware for highly available Services: DB2 UDB, HACMPIBM Storage Technology for performance and availablilityCentral RADIUS Server running on zSeries-OS/390 in parallel Sysplex with DB2

Intregration of 3rd Party Components Cisco Routers Network Access Routers from Lucent, Shiva, Accent, Siemens

Dr.Stefan Radtke

User Dial In Scenario (1)

ISDN


NAS

ADSL

Internet

RADIUS-Proxy

Local Server/DB

RADIUS-ProxyRADIUS-ProxyIP-Addresses

User Data

Firewall

Central Server

Caching Proxy Cluster

httpd

NASUser Data

POP (13x)

1. User Request to NAS2. NAS talks to RD-Proxy3. Proxy asks Local Server4. Auth.accept to Proxy5. Proxy allocates IP-Address6. Auth. accept Proxy to NAS7. Accounting start to Central- Server8. User is connected to Network with dynamic IP-Address

Analog

Dr.Stefan Radtke

ISDN


NAS

ADSL

Internet

Central Server

User Data

PO

P (

13x)

HA Implementation on RS/6000SP

RS

/600

0SP

zSeries

Central Server Central Server Central Server

Parallel Sysplex

HACMPCluster

Radius-Proxy

7133-600

Mirror

Radius-Proxy(Backup)

7133-600Radius-Svr Radius-DB

(Backup)

Firewall Firewall

7133-600Radius-Proxy Radius-Proxy

(Backup)

Analog

Dr.Stefan Radtke





Miscellaneous (...only when sufficient time left)Caching of Streaming Data (RTSP)Software Distribution


Dr.Stefan Radtke

Caching Proxy Cluster on RS/6000SP

1. Reduce Response Time for Customers

2. Reduce required network bandwidth

3. Reduce costs for IP traffice over peering points

4. Reduce load on overloaded web servers

Why are Telco's investing in Web Cache ?

Dr.Stefan Radtke

HTTP Traffic growth

Today the WES cluster serves about 500.000.000 HTTP Requests per day

Reqeuests/Day

050000000

100000000150000000200000000250000000300000000350000000400000000450000000

Jan

98

Apr

98

Jun

98

Aug

98

Oct

98

Jan

99

Mar

99

May

99

Jul 9

9

Oct

99

Dec

99

Feb

00

Apr

00

Jun

00

Sep

00

Nov

00

Feb

01

Dr.Stefan Radtke

HTTP Traffic growth

... and about 4 TB of data through the caching proxy clusters

Bytes/Day

0500000000000

100000000000015000000000002000000000000250000000000030000000000003500000000000400000000000045000000000005000000000000

Jan

98

Apr

98

Jun

98

Sep

98

Nov

98

Jan

99

Apr

99

Jun

99

Sep

99

Nov

99

Jan

00

Apr

00

Jun

00

Sep

00

Nov

00

Feb

01

Dr.Stefan Radtke

Object Size vs. Bandwidth Consumption

<1K

1-10

K

10-1

00K

100K

- 1M

1 - 1

0M

10 -

100M

0102030405060

Percent of Repeat Objects

<1K

1-10

K

10-1

00K

100K

- 1M

1 - 1

0M

10 -

100M

0

10

20

30

40

Percent of Repeat Traffic ... those objects consume over 50% of network bandwidth!

Very few (<1%) object requests are for objects larger than 1 MB...

Dr.Stefan Radtke

WWW-Performance for 8 Million Users

2 Tier Load Balancing Telephone Network routes call to nearest POP Load Balancing within the POP by Network Dispachtcher

'Local' DNS Server on each Cluster Node DNS Server on ND-Nodes answer user queries DNS Server on WTE-Nodes answer WTE queries

Caching Proxy Cluster with dynamic Load Balancing Using IBM Interactive Session Support (ISS) Takes dynamically into account different CPU capabilities Streaming formats like Real Video/Audio can be integrated

Dr.Stefan Radtke

2 Tier Load Balancing

ND

TelephoneNetwork

NAS

NAS

NAS

NAS

ND

ND

ND

Internet

Network Dispatcherand 'local' DNS Cache Server

User connections are routed to the geographically nearestPOP & Proxy Cluster.

Dr.Stefan Radtke

IBM Websphere Edge Server

Case 1: Requested data already in local cache

Case 2: Requested data saved in neighbour cache

Case 3: Requested data not yet saved within cluster

Caching Web with Remote Cache Access

Dr.Stefan Radtke

Case 1: Requested Data is already in Cache

WTE1

WTE2

WTE3

ND

ND

WTE4

WTE5

WTE6

Internet

www-proxy.dtag.deGEThttp://www.abc.com

1

2

4 - Direkte Antwort

WTE7

WTE8

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

3

Dr.Stefan Radtke

WTE1

WTE2

WTE3

ND

ND

WTE4

WTE5

WTE6

Internet

GEThttp://www.abc.com

1

2

6 - Direkte Antwort

WTE7

WTE8

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

4

53

3,5 = HTTP

Case 2: Data resides in Neighbour Cache

Dr.Stefan Radtke

Case 3: Data not yet available in Cache

WTE1

WTE2

WTE3

ND

ND

WTE4

WTE5

WTE6

Internet

GEThttp://www.abc.com

1

2

7 - Direkte Antwort

WTE7

WTE8

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

160MB/s

5

63

4

Dr.Stefan Radtke

Caching Proxy Cluster: Physical Configuration Deutsche Telekom AG runs about 50 Caching Proxy Clusters based on RS/6000SP technology with about 400 nodes.

A1A2B1B2

A1A2B1B2

B2B1A2A1

B2B1A2A1

B2B1A2A1 7133-600

4

5

8

9

8 123456

9 10 1112 13 14 15 16

1

12

13

16

7

9,1 9,19,19,1

9,1 9,19,19,1

9,1 9,19,19,1

9,1 9,19,19,1

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

7133-600

4

5

8

9

8 123456

9 10 1112 13 14 15 16

1

12

13

16

7

9,1 9,19,19,1

9,1 9,19,19,1

9,1 9,19,19,1

9,1 9,19,19,1

WTE + ND

WTE + ND

WTE

WTE

WTE

WTE

WTE

WTE

- 8 x SP Nodes- each node with 1 GB RAM- each node has 4x 9,1 GB SSA Disks - CacheSize per cluster = 8x 36 GB = 288 GB- ND and Backup-ND are running co-located

Dr.Stefan Radtke

Dynamic Load Balancing with ISSDifferent nodes have different performance andcan handle different number of requests/time.

Typ T-Web-Proxy-Prod. SpecWeb99-----------------------------------------------------------------------------------------------------------Power2SC 160MHz 1 CPU: ca. 4.500 req/min.PowerPC 332Mhz 2 CPUs ca. 6.000 req/min.Power3 200Mhz 2 CPUs 14.000 req/min. 440Power3-II 370 Mhz 4 CPUs ~ 40.000 req/min*) 1359 *) estimated

Number of HTTP-Reqeusts per Minute

The test nodes were equiped with - 1 GB RAM - 4 oder 5x 9.1 GB SSA Disks

DynamicLoad Balancingis required !!

Dr.Stefan Radtke

eND Cluster with ISS

ISS Agent

ISS Agent

ISS Agent

ISS Agent

ISS Agent

ISS Agent

ISS Agent

ISS Monitor

ISS Agent

Dispatcher

Load Information

Requests

Customized ND-Proportions 4 % Active Connections 3 % New Connections 3 % Advisor90 % System load

advisoradvisor

advisor

manager

executor

ndcontrol

ndserver

Manager: sets weights based on - internal counters in executor - feedback from servers/advisors - system monitoring, e.g. ISSAdvisors: query servers and calls Manager for setting weights.

- exist for HTTP, SSL, POP,SMTP,NNTP,IMAP,... - runs optional (but recommended !!)Executor: does the load balancing. - mandatory

Dr.Stefan Radtke

Customized Dispatcher Weights

Customized ND-Proportions: 4 % Active Connections 3 % New Connections 3 % Advisor90 % System load (ISS)

Default ND-Proportions: 90 % Active Connections 5 % New Connections 5 % Advisor 0 % System load

We changed the default weight proportions since we are usingdifferent nodes. The system load*) takes automatically into account thedifferent I/O and CPU capabilities.

*) Load Average is the number of processes which were ready to run (though not waiting for I/O) but waiting to be dispatched to CPU ... during the last 1, 10, 15 minutes.

Dr.Stefan Radtke

Average Response Times

Average response time over 24 hours for 20 http GET requests

Result: Responses over Caching-Proxies are about 2 times faster than direct requests !!

24 Stunden Durchschnitt

0

5

10

15

20

25

30

35

Proxy Direkt

Seku

nden

für 2

0 R

equ

ests

0

5

10

15

20

25

30

35

Seku

nden

für 2

0 R

equ

ests

Resonse Timefor 20 http get requests:

over Proxy: 15 sec.direct: 32 sec.

Dr.Stefan Radtke

Statistics

Detailed statistics are available for each cluster and each node. - Special plugin developed (performance) - Available in table- and grafic format through Web interface.

Dr.Stefan Radtke





Miscellaneous (...only when sufficient time left)Caching of Streaming Data (RTSP)Software distribution


Just for backup

Dr.Stefan Radtke

Caching of Streaming formats

RTSP = Real Time Streaming ProtocolThe Real Time Streaming Protocol is defined in RFC 2326. It is an Internet standard protocol for controlling data streams. It does not include technology for delivering streams

RTSP Redirector im WTEThe RTSP redirection feature allows Web Traffic Express to redirect requests for any streaming media sessions controlled by RTSP. These include the following types of media:

RealNetworks recorded audio RealNetworks recorded video RealNetworks live streams (audio and video) Microsoft Media Player files Apple Quicktime media files

Any player that can be configured to contact a proxy server on its RTSP port (typically 554) can use this framework in WTE to have its requests handled by the RTSP redirector.

Dr.Stefan Radtke

RTSP Redirect (1/2)

WTE

ND

ND

WTE

WTE

WTE

1

2

RS/6000SP

GET http://..../wantit.ra&proto=rtsp HTTP/1.0

HTTP/1.1 200 OK.Server: Lightweb/0.00 (Linux)Content-type: audio/x-pn-realaudio<CR><CR>rtsp://g2...com/.../wantit.ra

3

4

5

Cache f. Streaming Daten

Streaming Proxies Real-Networks,Microsoft,Quicktime,...

Dr.Stefan Radtke

WTE

ND

ND

WTE

WTE

WTE

1

2

RS/6000SP

Streaming Proxies Real-Networks,Microsoft,Quicktime,...

5

rtsp://g2...com/.../wantit.ra

3

46

Cache f. Streaming Daten

RTSP Redirect (2/2)

3.) RTSP/1.0 305 UseProxy Location: rtsp://193.158.128.70:554/

4) OPTIONS rtsp://g2-rbn-nfs06.rbn.com:554 RTSP/1.0 User-Agent: RealMedia Player Version 6.0.7.19

5) Autorisierungs Session

6) Auslieferung Video/Audio aus Cache

Dr.Stefan Radtke

RTSP Trace between (1/3) Client->WTE->Real-Proxy->RealServer

Client > WTE:80GET http://play.rbn.com/?url=gmusic/gmusic/g2demand/peeps/bboys/wantit.ra&proto=rtsp HTTP/1.0Referer: http://www.peeps.com/bboys/media.htmlProxy-Connection: Keep-AliveUser-Agent: Mozilla/4.5 [en] (Win98; I)Host: play.rbn.com..Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*Accept-Encoding: gzip..Accept-Language: enAccept-Charset: iso-8859-1,*,utf-8<CR><CR>

WTE > Real-Server:80GET /?url=gmusi/gmusic/g2deman/peeps/bboys/watit.ra&proto=rtp HTTP/1.1Hos: play.rbn.com.Connection: Kee-Alive, TERefrer: http://www.peeps.com/bboys/media.htmlAccept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*Accept-Encoding: gzipAccept-Language:enAccept-Charset: iso-8859-1,*,utf-8TE: chunkedVia: HTTP/1.0 spcss47. (IBM-PROXY-WTE)Client-IP: 193.158.128.84User-Agent: Mozilla/4.5[en] (Win98; I) <CR><CR>

Dr.Stefan Radtke

Trace (2/3)Real-Server:80 > WTEHTTP/1.1 200 OK.Server: Lightweb/0.00 (Linux)Content-type: audio/x-pn-realaudiortsp://g2-rbn-nfs06.rbn.com/gmusic/gmusic/g2demand/peeps/bboys/wantit.ra

WTE:80 > ClientHTTP/1.1 200 OKConnection: closeProxy-Connection: closeVia: HTTP/1.1 spcss47 (IBM-PROXY-WTE)Server: Lightweb/0.00 (Linux)Content-type: audio/x-pn-realaudio<CR> <CR>

WTE :80> Clientrtsp://g2-rbn-nfs06.rbn.com/gmusic/gmusic/g2demand/peeps/bboys/wantit.ra.

Client > WTE.554 (RTSP-Redirector)OPTIONS rtsp://g2-rbn-nfs06.rbn.com:554 RTSP/1.0CSeq: 1User-Agent: RealMediaPlayer Version 6.0.7.194 (win32)ClientChallenge: ab8bd4308b03704d9773d0eb3890dfb6PlayerStarttime: [11/07/2000:20:18:27 00:00]CompanyID: W9LrkNVmrQ4si74ItX7pcQ==GUID: caafabe0-5588-11d3-a6fd-0000e8557559RegionData:ClientID: Win98_4.10_6.0.6.99_play32_MF60_en-US_586<CR> <CR> WTE.554 (RTSP-Redirector) > Client RTSP/1.0 305 UseProxyCSeq: 1L ti t //193 158 128 70 554/

Dr.Stefan Radtke

Trace (3/3)

WTE.554 (RTSP-Redirector) > Client RTSP/1.0 305 UseProxyCSeq: 1Location: rtsp://193.158.128.70:554/Date: TueJul 11 20:19:482000Server: Web Traffic Express/Edge Server RTSP Redirector 0.1<CR> <CR> Client > Real-Proxy-Server.rtsp: OPTIONS rtsp://g2-rbn-nfs06.rbn.com:554 RTSP/1.0CSeq: 1User-Agent: RealMedia Player Version 6.0.7.19

Real-Proxy-Server.3253 > Real-Origin-Server .rtspOPTIONS rtsp://g2-rbn-nfs06.rbn.com:554 RTSP/1.0CSeq: 1User-Agent: RealMedia Player Vers

Dr.Stefan Radtke

3-Tier Software DistributionSoftware Packages are replicated (rdist) over night to about50 Control Workstations (with ssh).

Only one repository must bemaintained.

Installation to nodes fromCWS does not consumeWAN capacity.

Dr.Stefan Radtke

Physical Configuration on RS/6000SP

RARs

neue RARs

Cisco Router

LAN-Switch 1

LAN-Switch 2

Control-Ethernet (en0)(all Nodes)

VPN-Netz (en5,en6)

AGB-Netz (en3,en4)

100/100 Mbit Ethernet Adapter

SSA-Adapter

PCI-Steckplatz leer

HACMP-Stby-Net (en2)

HACMP-Service-Net (en1)

LAN-Switch 3

Ethernet-Hub

1

3

5

7

9

11

13

15

IP-Net

Dr.Stefan Radtke

1

3

5

7

9

11

13

15

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

B2B1A2A1

4

5

8

9

1

12

13

16radiusdb3

8 45 13

9 10 12 13 15 16

3 2

14

7 6

11

4

5

8

9

1

12

13

16

rad1

vg

rad2

vg

rad4

vg

rad3

vg

8 45 13

9 10 12 13 15 16

3 2

14

7 6

11

radiusdb1

4

5

8

9

1

12

13

168 45 13

9 12 13 16

3 2

14

7 6

1110 15

B2B1A2A1

radi

usdb

ls1vg

ls2vgls2vg

Physical Configuration on RS/6000SPStorage (SSA)

Documents

Europe's largest Internet Service Platform · Physical Configurations (SP,IP-Net, Storage) Distributed Service to support 50 Mio customers Web Cache Service Why are Telco's invest