54

Spca2014 advanced share point troubleshooting hessing

  • Upload
    nccomms

  • View
    144

  • Download
    1

Embed Size (px)

Citation preview

Page 1: Spca2014 advanced share point troubleshooting hessing
Page 2: Spca2014 advanced share point troubleshooting hessing

Advanced SharePoint Troubleshooting

Donald Hessing, @dhessing, #SPCon14

Page 3: Spca2014 advanced share point troubleshooting hessing

Who am I?

Donald Hessing Principal Consultant | Thought Leader SharePoint @Capgemini

Netherlands Microsoft Certified Master (MCM) - SharePoint (Virtual) Technology Solution Professional for Microsoft Work full time on SharePoint since 2007 | #DEV | #ITPRO | #STRATEGY [email protected] | @dhessing | ##SPCon14

Page 4: Spca2014 advanced share point troubleshooting hessing
Page 5: Spca2014 advanced share point troubleshooting hessing

Let’s investigate….

Who

What

What

When when

Where

Why Why

Page 6: Spca2014 advanced share point troubleshooting hessing

…defines the scope

Lab\User3

30 seconds render time

Mon-Fri between 0900-1100 CST, but not on weekends

WFE1

Typically achieves results between 5-7 seconds

When user is searching for “Widget” and hits WFE1

Page 7: Spca2014 advanced share point troubleshooting hessing
Page 8: Spca2014 advanced share point troubleshooting hessing

It’s not all about SharePoint!

Load Balancer

Page 9: Spca2014 advanced share point troubleshooting hessing

Understand response time

Page 10: Spca2014 advanced share point troubleshooting hessing

Response Time

1414 KB

1024 KB (10Mb)

20

ms

(44

/2)300 ms 300 ms 2.6 sec.

1414KB

100 KB (1Mb)

150

ms

(44

/2)300 ms 300 ms

15.42

sec

IE8, FireFox 3 – 6

connections max.

TCP Slow Start

Page 11: Spca2014 advanced share point troubleshooting hessing
Page 12: Spca2014 advanced share point troubleshooting hessing

ULS Viewer

Page 13: Spca2014 advanced share point troubleshooting hessing

Fiddler

Page 14: Spca2014 advanced share point troubleshooting hessing

Fiddler – Troubleshooting WCF services

http://www.andrewconnell.com/blog/SP2013-Workflow-Advanced-Workflow-Debugging-with-Fiddler

http://127.0.0.1:8888

Page 15: Spca2014 advanced share point troubleshooting hessing

Fiddler – Troubleshooting Search

Page 16: Spca2014 advanced share point troubleshooting hessing
Page 17: Spca2014 advanced share point troubleshooting hessing

LOGPARSER 2.2

Page 18: Spca2014 advanced share point troubleshooting hessing

LOGPARSER

Page 19: Spca2014 advanced share point troubleshooting hessing

IIS LOGS

CS-URI Client Side URI

Page 20: Spca2014 advanced share point troubleshooting hessing

Addition Fields

Page 21: Spca2014 advanced share point troubleshooting hessing

Example – Find slow pages

Page 22: Spca2014 advanced share point troubleshooting hessing

Join data with AD with PowerQuery

Page 23: Spca2014 advanced share point troubleshooting hessing
Page 24: Spca2014 advanced share point troubleshooting hessing
Page 25: Spca2014 advanced share point troubleshooting hessing

Customer Case

Page 26: Spca2014 advanced share point troubleshooting hessing

Questions?

Page 27: Spca2014 advanced share point troubleshooting hessing

LOGPARSER "SELECT QUANTIZE(TO_TIMESTAMP(date, TO_LOCALTIME(time)), 3600) AS Hour, DIV(SUM(sc-bytes),1048576) AS TotalBytesSent, DIV(SUM(cs-bytes),1048576) AS TotalBytesReceived FROM *.log GROUP BY Hour ORDER BY Hour" -i:w3c

Determine network capacity..

Number of users 5.000 10.000 20.000 50.000

Bandwidth (Mbit/s) 25 50 100 250

Page 28: Spca2014 advanced share point troubleshooting hessing

Response times

SC-STATUS AVGTIME MAXTIME HITS

200 215 33882 30581

302 56 3104 2949

400 58 78 13

401 16 3166 14811

403 31 31 1

404 81 2745 1472

500 63 109 12

Page 29: Spca2014 advanced share point troubleshooting hessing

Response Times Web Frontend Servers

Page 30: Spca2014 advanced share point troubleshooting hessing

Response times per DOMAIN

Page 31: Spca2014 advanced share point troubleshooting hessing

Response times for DOMAINB per WFE

Page 32: Spca2014 advanced share point troubleshooting hessing

Why are my responses for DOMAINB users SLOW on WFE2?

Page 33: Spca2014 advanced share point troubleshooting hessing
Page 34: Spca2014 advanced share point troubleshooting hessing

Win32Status

Page 35: Spca2014 advanced share point troubleshooting hessing

System outage analysed - Win32Status

Win32-Status Description Occurred

2 The system cannot find the file specified. 4788

64 The specified network name is no longer available. 1663

121 The semaphore timeout period has expired. 964

995The I/O operation has been aborted because of either a thread exit or an application

request.85

1236 The network connection was aborted by the local system. 3934

1330 The password for this account has expired. 6

2148074252 The logon attempt failed 271

2148074254 No credentials are available in the security package 117831

2148074257 No authority could be contacted for authentication. 9379

LogParser.exe -i:w3c

"SELECT sc-win32-status,

WIN32_ERROR_DESCRIPTION(sc- win32-status) as Description,

Count(*) AS Occurred FROM *.log WHERE sc-win32-status>0

GROUP BY sc-win32-status ORDER BY Total DESC

Page 36: Spca2014 advanced share point troubleshooting hessing
Page 37: Spca2014 advanced share point troubleshooting hessing

Performance Counters

SharePoint Servers Threshold

% CPU SharePoint Servers <75%

# Available Mbytes < 20%

% Network bandwidth – bytes total < 41 - 65%

Disk (Instance)% Idle Time < 20%

SQL Server Threshold

% CPU SQL Server <75%

Memory: SQLServer:Buffer Manager--Buffer

Cache hit ratio

< 90%

Memory: SQLServer:Buffer Manager--Page Life

Expectancy(PLE):

(Total Mem/4) * 300

16GB > (16/4) * 300

Network bandwidth – bytes total < 40%

Avg. Disk Write Queue Length < 2 Per Spindel

Avg. Disk Read Queue length < 2 Per Spindel

Avg. Disk Sec/Read SQL Server Disks < 12 ms – 0.012

Avg. Disk Sec/Write SQL Server Disks < 12 ms – 0.012

Page 38: Spca2014 advanced share point troubleshooting hessing

Virtualization - Hypervisor

Host - Counter Healthy Caution Critical

Hyper-V Hypervisor Logical Processor

(_Total)\% Total Runtime

< 60% 60-89% 90-100%

Memory\Available Mbytes 50% 25% 10%

Memory\Pages/sec (swap rate) <500 500-1000 >1000

% Network bandwidth – bytes total < 40% 41 – 64% 65% - 100%

Network Interface(*)\Output Queue Length 0 1-2 >2

Page 39: Spca2014 advanced share point troubleshooting hessing

Logical versus Virtual Processor

Page 40: Spca2014 advanced share point troubleshooting hessing

Inaccurate perfmon results - VMWare

Page 41: Spca2014 advanced share point troubleshooting hessing
Page 42: Spca2014 advanced share point troubleshooting hessing

Validate disk configuration prior to deployment

Page 43: Spca2014 advanced share point troubleshooting hessing

Numbers to Remember - Spindles

Page 44: Spca2014 advanced share point troubleshooting hessing

Determine maximum throughput

Kind Threads Seconds Drive Stripe Outstanding Size IOs MBs Latency_min Latency_avg Latency_max

R 2 60 G random 1 8 105.05 0.82 4 18 666

R 2 60 G random 2 8 106.96 0.83 6 36 684

Page 45: Spca2014 advanced share point troubleshooting hessing

SQL IO Characteristics

Operation Random / Sequential Read/Write Size Range

OLTP – LOG Sequential Write 512 bytes – 64KB

OLTP- Data Random Read / Write 8KB

Bulk Insert Sequential Write 8KB – 128KB (in

multiples of 8KB)

Read Ahead Sequential Read 8KB – 128KB (in

multiples of 8KB)

Backup Sequential Read/Write 1MB

Restore Sequential Read/Write 64KB

Page 46: Spca2014 advanced share point troubleshooting hessing

Validate configurations prior to deployment

Page 47: Spca2014 advanced share point troubleshooting hessing

Customer Example – IOMonitor.exe

Design TEST - 2 KB Random - Read/Write (67%/33%)

Drive

Expected

IO

Total

IO

Average

Read Time

(ms)

Average Write

Response Time

(ms)

Maximum

Read Time

(ms)

Maximum

Write

Time(ms)

C:\ 60 314 3 1 197 6

L:\ 300 419 2 3 31 11

S:\ 600 386 3 3 115 28

T:\ 1200 51 35 52 89 2501

U:\ 1000 43 5 59 41 1817

Page 48: Spca2014 advanced share point troubleshooting hessing

Monitoring Pending Disk IO

http://henkvandervalk.com/sql-under-the-hood-part-1-monitoring-current-pending-disk-ios

Page 49: Spca2014 advanced share point troubleshooting hessing

The common…

Page 50: Spca2014 advanced share point troubleshooting hessing

…Less well known

Page 51: Spca2014 advanced share point troubleshooting hessing
Page 52: Spca2014 advanced share point troubleshooting hessing

Thank you!

Page 53: Spca2014 advanced share point troubleshooting hessing
Page 54: Spca2014 advanced share point troubleshooting hessing