Building configurable applications for the web

Tom Melendez, Yahoo! 1

B u ild in g c o n f ig u r a b le a p p lic a t io n s f o r t h e

w e b


Building configurable applications for the web

About Me

• Using PHP since 1998 (3.0.5)

• Lead Infrastructure Engineer, Yahoo! News

• Native New Yorker (so, I’m skeptical of everything)



What does “configurable” mean?

What are we really talking about?

Am I in the right place?



Intended audience

• Developers (who don’t want to annoy their Operations folks)

• Ops folks (who don’t want to constantly harass their Developers)

• QA folks (who want to make sure Dev and Ops have their sh*t together)

• Managers and Execs (who want to tell their bosses, “Yeah, we can do that”)

• and…



Anyone who doesn’t want to get paged in the middle of the night.



We’re talking about application and system configuration

• What's on the box?

• What state is the box in?

• What is it allowed to do?

Not personalized user configuration of your web app.



Why highly configurable systems?

The Need for:• Multiple Deployment Platforms

• Multiple Experiences

• Different Architectures

• Multiple installations for different purposes

• Different data and/or presentation of that data

• Unique states of some instances of the application

• Real-time tuning

• Backwards compatibility; Deployment and Rollback Situations



Why now?

A bug in production costs 200 times more to fix than what it would have if caught prior. *

Something similar could be said about adding in more flexibility and configuration, because really, it’s a bug.

Plus, when your app gets Oprahified or Slashdotted you’ll have plenty of levers to pull and knobs to turn.

* Sebastian and Stefan's talk on PHP and CI on Monday at IPC 2009



I’m a one-man band: do I really need all this stuff?

photo courtesy of origamidon@ on Flickr



YES.(Besides, the alternative is that we sit here in silence.)



These are goals, not implementation requirements!The Need for:

• Multiple Deployment Platforms

• Multiple Experiences

• Different Architectures

• Multiple installations for different purposes

• Different data and/or presentation of that data

• Unique states of some instances of the application

• Real-time tuning

• Backwards compatibility/Deployment and Rollback Procedures



Our ultimate goal is allow the system to be as configurable as possible without changing code and without doing too much work.



I do have some requirements though.photo courtesy of duncan@ on Flickr



• Automated deployment:Host, Application and Configuration

• All of this needs to be in SCM

• Going forward without the above is bad



Other nice to haves would be:

• Automated Builds

• Monitoring

• Profiling

Despite the logos above, even a poor man’s shell script will do. We want to build our software without effort, know that the app is up, and make sure performance didn’t drop dramatically.



So today, things that we’ll touch on:

• The Cloud (well, not really, but you’ll be cloud-deployable)

• CI

• Caching

• Performance

• Proxies

• Defensive coding/Debuggable code



Now lets make our app configurable.

Question: What should be made configurable?

Answer: Everything.



Seriously.

Look at your app and prioritize how important it is for the business to have that feature/widget right where it is showing the data it is currently showing.

Examples that come to mind:• Form Fields• Payment screens• AJAX updates• Account Information• Summary Data

• Data about your contacts• Previous transactions• Recommendations• Photos/Multimedia• Ads



Lets look at some examples:

Twitter

Facebook

Y! News



I would argue that the previous exercise will probably make you come up with a better solution.

It also sets expectations with your stakeholders as to what points you would degrade your service and at which points you would fail.



Now that we have our priorities for app features, we begin to make them configurable.

Take the lowest priority items first

• If increased load occurs, it is a no-brainer to turn these off

• Since they are low priority, they are likely to be changed for something else that could be a higher priority

You’ll want to apply some high value/low cost logic to the above.



You will also use the priority list you created as the basis for your DEFCON levels. Example:

• Level 5 – normal operation

• Level 4 – planned high traffic event

• Level 3 – unplanned high traffic

• Level 2 – component failure

• Level 1 – site failurehttp://en.wikipedia.org/wiki/Defcon



What about a given feature should be configurable? It depends, but generally:

• Hostnames/Ports

• Database names

• Usernames/Passwords

• Cache times

• Base file locations

• Timeouts

• Application defaults/Flags

• Display settings

• Minimums and Maximums



Is there a strategy to figure this out? Well, ask yourself:

• Could it ever change, and be changed, without affecting anything else? (u/p, db names)

• Would someone else ever need to change it? (hostnames, different platforms, diff envs)

• Do I really want to get out of bed, edit some code, build and deploy just for this change?



Further strategy (and a little detour). Please write debuggable code

• Or else



Writing debuggable code

• We need extended information available to us for troubleshooting/diagnostics

• Any service calls need terse logging available to show:

• the request made to me

• The URL that I requested

• What parameters/headers/cookies I sent

• the HTTP code I got back

• The response I got back

• Ideally need to do this while box is in use, but definitely without changing code



It is 3am.

I don’t know why something isn’t working.

I don’t want to spend an hour trying to figure it out.

I should be able to flip a switch and get this information in the log.

Better yet, someone else should be able to do it for me.



Now that your code is debuggable, strive for Runbooks

• Step by step decision tree on how to handle a situation and how to escalate if necessary

• When you’ve reached this stage you now have given someone else control over the maintenance of your application.

• Congrats!

http://en.wikipedia.org/wiki/Runbook



Writing defensive code

• Expected objects need to be validated

• is_object($foo) is not enough

• instanceof FooClass may not be enough

• You should ensure that what you have is a valid object that you can useif (is_object($xml_resp) ) {

//site blows up $xml_resp->xpath(“/something/that/does_not/exist”);}

if (is_object($xml_resp) && $xml_resp instanceof SimpleXMLElement) { //better, but you still don’t know if it is valid as you see it. Could be an error message in XML format $xml_resp->xpath(“/something/that/does_not/exist”);}



Degrading Service

• A mechanism to request data without killing your connections

• Your code depends on something which might not be available

• Or, you’re just a smart cookie and know that at any time, anything might not be available

• Can you still serve traffic? Are you eating up connections by connecting to something that probably won’t be there?

• Should you be serving traffic?

• Review your priority list as to when you should fail



Degrading Service

• Assuming that you shouldn’t fail, the service should be degraded

• At some point, you should stop requesting that service, you’re just eating up connections.

• How it works

• How many times should I try to fetch and have it fail?

• How long should I wait until I try again to fetch?



Managing your cache

• File cache, local cache, distributed cache, proxy cache, CDN

• Do you have/use tools to manage your caches (CRUD operations)?

• Can’t tell you how many issues I’ve seen to bad caching, whether it be expiration or poision

• All Devs and Ops need to know how to use these tools (Runbooks, remember?)



Storing and reading your configuration

• Single PHP file, auto prepended?

• Apache SetEnv (requires restart)?

• Database?

• APC?

• Beware of apc cache slam



Configuration via Proxy: Introducing Apache Traffic Server• “Traffic Server is a high-performance web proxy cache that improves network

efficiency and performance by caching frequently-accessed information at the edge of the network.”

• http://incubator.apache.org/trafficserver/docs/admin/

• How do we know which INTL to show? We use an Apache TS remap rule to proxy the user to the front end passing in the INTL in the URL

• Super-high performing: Unofficial “test” had it handling 90k rps (!) on 12 boxes in 3 colos

• Enabling caching on TS is our DEFCON 1. We’ve never needed that.

• Allows for other cool things like bucket testing and URL rewriting.



Configuration via Proxy: configured with YAMLvars: story_fe_ep: <internal dns name>

templates: news.template paths:... /s/: endpoint: %{story_fe_ep}/news/story/v3/en-US/s ...hosts: news.yahoo.com template: news.template



Wishlist: If I could have it my way• Dev writes tests and code, commits• Commit build runs on build server, passes OK• Packages are deployed to Integration periodically and BAT and Smoke Tests are run• Packages are deployed to a QA env, Selenium and regression tests are run• In parallel, packages are deployed to another environment and perf tests are run• All looks good, we automatically deploy to a bucket• See good performance, high clicks, user satisfaction in the metrics, we initiate the process to go live.



Summary

• Give others the options, tools and info necessary to administer the app

• An ounce of prevention is worth a pound of cure

• Don’t get paged!

photo courtesy of red5standingby@ on Flickr



Questions?

[email protected]

Technology

Building configurable applications for the web