Empiricism and Guessing Among Oracle Professionals Jeremiah Wilton

Empiricism and GuessingEmpiricism and GuessingAmong Oracle ProfessionalsAmong Oracle Professionals

Jeremiah Wiltonhttp://www.ora-600.net

ORA-600 Consulting

Independent Consulting Multiple clients Emergencies Seminars Recruiting

Me and Guessing

Started out guessing a lot Felt my work was sound Insufficient mentoring/oversight No overall method to draw from No tradition of excellence

Oracle and Guessing

7.3 Tuning Guide is a manual in guessology Where are wait events described? Emphasis on long lists of stuff

What sounds good? BCHR – OK “Spin Count” - Cool!

Seemed more art than science

Erroneous Literature

Literature was full of nonsense – written by mischievous elves

Still 80% correct Would have been helpful but I could get by

without it so I mainly did

Compensation Issue

One of my problems was compensation Late '90s steep increase in DBA salaries Low pay = low motivation

Amazon and Me

Wild guesses and misinformation fell flat High expectations for accuracy and knowledge Gifted colleagues set level of excellence Better compensation Afraid to let colleagues down

Sad state of our profession

Poor quality is common Unqualified DBAs are often well compensated Driven by law of supply and demand Insufficient resources for poor DBAs get better Oracle is specialized and complex and it is easy

to fool people

Role of hiring in low quality

Non-technical/non-DBAs hiring DBAs Unqualified DBAs get hired at good salaries Creates resentment among colleagues Waste of money No valid review process without senior technical

leadership Hire someone to help do this right!

You do you know you are unqualified?

Poor DBAs don't know there's a problem Manual? What manual? No mentoring Only exceptional people can self-develop in

these circumstances

The boys' club

Personal circumstances play into work hours and availability

80-hour DBAs can be really good 40-hour should be equally acceptable Freebie for company Encourages low DBA headcount Emphasis should be on quality Reasonable work levels ≠ unqualified

More opportunities for improvement

IOUG.org OTN Oracle-L comp.databases.oracle.server

Has candidate ever posted to a forum? Goes to motivation, interest in learning or

interest in sharing knowledge

State of DBA candidates

So many unqualified candidates Actually working! Unbelievable!

TABOS

Ask any DBA candidate the “slow database” question

Answer is usually a punchlist in no particular order

TABOS = Try A Bunch Of Stuff Oracle Support uses TABOS

Origins of TABOS in Oracle docs

Oracle 7 Tuning Guide Insufficient CPU Insufficient memory Insufficient I/O Network constraints Software constraints

Pointed DBAs at v$sysstat Scant and muddled references to wait events

Candidates echo the manual

Candidate espousing TABOS sounds like the Oracle7 Tuning Guide

Check CPU usage

Check memory usage

Check for high BCHR

Check for some other hit/miss ratios you found on the Internet

Check for full disks

Check for too many extents

Check for “fragmentation”

Check for disk sorts

Check for locks

Check for swapping

Check for FT scans

Check for bad plans

etc...

TABOS... works?

You might stumble upon the problem Works enough to just keep practitioners

employed Not fast or clever, but... Sounds “technical” - lots of jargon Makes you look busy! But might never find the problem if not on your

list Waste of time

Is TABOS for real?

Just a figment of the job interview Even underqualified DBAs use some

experience to order their list of checks Experience trumps a wild guess I wouldn't use even experience-driven TABOS Using waits and ASH still faster and more

reliable

Good Guesses

Some educated guesses are effective All DBAs guess Educated guess is as good as the education Empirical from εμπειρισμός = experience Experience is a valid form of evidence/data

Pure Empiricism

Are there purely empirical DBAs? Yes: BAAG – Battle Against Any Guess

http://www.battleagainstanyguess.com “Eliminate guesswork from the decision-making

process” BAAG in the tradition of BAARF

BAAG has a point

Oracle's instrumentation is the best way to solve most problems

“Slow database” question is the poster child for ASH

Starting with ASH/waits assumes that Oracle is the problem

For reality-based troubleshooting, we need equally good instrumentation on app server

How empirical can you really be?

Time and resources limit empiricism Many decisions don't merit exhaustive means A good DBA knows when to make an educated

guess

The Axis of Guessing

Empiricism TABOSEducatedGuessing

Most DBAs

There are no pure empiricists

So why BAAG? An effort to distance from messy thought Large number of unqualified and undertrained

DBAs Polarizing effect on skilled DBAs Real axis is learners vs. slackers

A meaningless axis

Separatism is destructive Us vs. Them Imaginary division Anti-guessing ≠ Science The key to a scientific approach is verification Oracle's instrumentation is ample

Issue of incentive

Most DBAs won't improve without incentive High DBA demand reduces incentive Non-DBAs can't tell a DBA is unqualified

Manuals are the first step

Oracle7 Tuning Guide notwithstanding Manuals are pretty good now Require patience and persistence Some people are not book learners “Experiential learners” not compatible with

production DBA work

Incentives for Excellence

I was motivated by high expectations and corporate culture

Can also be naturally bright and curious Most need organizational incentive

State of Oracle Education

No critical thinking taught Emphasis on syntax and rote facts Which top Oracle gurus ever came out of OCP? OCP and non-OCP indistinguishable in

interviews

Senior review and mentoring

Underqualified DBAs often have no role models Pro-mentoring is cost-effective

Junior DBAs are half price Not primadonnas Smart junior DBAs become senior in just few years

Role of technology

Middle tiers and applications need instrumentation

Opportunities at: Design time Purchase time

Enables analytical methods

Conclusions

Problem is not empiricism vs. guessing Problem is sad state of excellence in DBA work Solutions:

High expectations Critical thinking skills Meaningful education (mentoring, NoCOUG) Instrumentation at the app layer Don't hire unqualified people