Modern Web technologies (and why you should care): Megacomm, Jerusalem, February 2012

Modern Web Technologies (and why

you should care)Reuven M. Lerner • [email protected]

MegacommFebruary 15th, 2012

mailto:[email protected]


Who am I?

• Web developer since 1993

• Software architect/consultant/trainer

• Linux Journal columnist since 1996

• Mostly Ruby on Rails + PostgreSQL, but also Python, jQuery, MySQL, and more...

• PhD candidate at Northwestern University

How does the Internet (TCP/IP) work?

Client Server“Socket”

Port Port



Client opens connection

Port Port



Client opens connection

Port Port

Server accepts connection

Protocols

• Communication standards built on top of TCP/IP, typically text-based

• SMTP (e-mail)

• FTP (file transfer)

• NNTP (transfer of “news” articles)

WWW:Three technologies

• Markup format: HTML

• URL: protocol + server + port + doc

• Protocol: HTTP

How the Web works

Browser Server

How the Web works

Browser Server

HTTP Request

How the Web works

Browser Server

HTTP Request

HTTP Response

How the Web works

Browser Server

HTTP Request

HTTP Response

Stateless — after the response is sent, the connection is broken and forgotten

Simple request

Browser Server

Simple request

Browser Server

GET /

Simple request

Browser Server

GET /

200 OK<html><head>...</head><body>...</body></html>

Not found?

Browser Server

Not found?

Browser Server

GET /blahblah

Not found?

Browser Server

GET /blahblah

404 Not found

Three things are certain:

Death, taxes, and lost data.

Guess which has occurred.

— David Dixon, Salon magazine contest

Submitting forms

Browser Server

Submitting forms

Browser Server

POST /loginname=reuven&password=secret

Submitting forms

Browser Server


200 OK<html>...<p>Thank you!</p>...</html>

Document = request

• If an HTML page contains images, then each is retrieved in a separate HTTP request

• Page with 30 images = 31 HTTP requests

• Page with 20 images, 10 JavaScript files, and 5 CSS files = 36 HTTP requests

Idea: Lie to the browser

• Don’t return a document to the user

• Rather, run a program when the user makes a request, and send the program’s output

• If the output is in HTML, then the browser will show it no differently than a static doc

Just in time production

• Wait as long as possible to create pages for the user

• Ideally, create them when the user requests them

• “Mass customization”

Dynamic document

Browser Server

Dynamic document

Browser Server

GET /

Dynamic document

Browser Server

GET /


Dynamic document

Browser Server

GET /


Program output

What we return

• Usually HTML

• Image (e.g., stock graphs, Google Analytics)

• Word/Excel doc (e.g., from Google docs)

• PDF (e.g., PDF reports)

• XML, JSON (for computers, not people)

• Basically, any defined MIME type

APIs and mobile apps

Computer B (server)

Computer A (browser)


Computer B (server)

GET /



Computer B (server)

GET /

200 OK<some-xml><talk>JWP</talk></some-xml>



Computer B (server)

GET /



Computer B (server)

GET /



Computer B (server)

GET /



Computer B (server)

GET /


What is a Web app?

• Receives its inputs via HTTP

• Sends its output via HTTP

• That’s it! A Web application can do anything you want, within these limits

That’s it!

• Now you understand how the Web works.

• Really, that’s it.

• Go home.

OK, perhaps not...

• How do we write these programs?

• Where (and how) do we store user data?

• How have the underlying technologies (URLs, HTTP, and HTML) advanced?

Early Web applications

• First server-side programs were in C

• They were compiled into binaries

• So you had the CGI source (cgi-src) directory...

• ... and the CGI binary (cgi-bin) directory

• No explicit compilation

• Cross platform

• Built-in, powerful text functions

• Do a lot in a little bit of code

• Perl, Python, PHP, Ruby

• Typically open source

“Scripting” languages

Frameworks

• DRY (Don’t repeat yourself)

• Get rid of the drudgery

• Concentrate on your business, rather than worrying about common Web issues

MVC paradigm

• Most modern frameworks use MVC

• From Smalltalk in the 1980s!

• Model — communicates with database

• View — HTML/JavaScript/CSS for user

• Controller — handles requests

• Clear division of labor

Web frameworks in dynamic languages

• Programmer speed trumps execution speed

• Community support

• Plugins for commonly requested features

• Create a domain-specific language (DSL) for your specific needs

Ruby on Rails

• Ruby language

• Rails Web app framework (MVC)

• Designed for writing Web DSLs

• “Convention over configuration”

• Thousands of little improvements

• ActiveRecord — ORM

Person model

class Person < ActiveRecord::Base

end

Where’s the definition?

• The computer takes care of it automatically

• ActiveRecord knows what columns you have defined, and what their types are

• (More on columns later)

• Only write things that cannot be understood automatically

Not only Rails

• Python (Django)

• PHP (Symfony)

• Perl (Catalyst)

• Groovy (Grails)

• Even if you don’t use Ruby on Rails, you have benefitted from its “opinions”

Plugins

• Rails (and other systems) have open-source plugins to handle common issues

• Authentication

• E-commerce

• Social networking

• Don’t write these yourself! Customize existing code that has proved itself

Storage

• Applications are great!

• But what if we want to store information about our users?

• Name, e-mail address, account balance

• We could use text files, but most people will use a database

What is a database?

Database

Store data confidently

Retrieve data flexibly

Relational databases

Define tables, store data in them

Database

Retrieve data from related tables

Relational database

SQL goes hereCREATE TABLE

INSERTUPDATEDELETE

Database

Relational databases

• Everything is stored in 2-dimensional tables

• Data should appear only once (normalized)

• “Join” tables to connect tables

• Technology is extremely robust, fail-safe

• Not all data is a good fit for this paradigm

id first_name last_name phone

1 Reuven Lerner 054-496-8405

2 Charlie Kalech 02-671-9918



id first_name last_name email

1 Reuven Lerner [email protected]

2 Charlie Kalech [email protected]

person_id phone_number

1 054-496-8405

1 847-230-9795

2 02-671-9918

2 054-803-3356

2 501-629-8620








id type

1 work

2 mobile

3 fax

4 home

person_id phone_number_type_id phone_number

1 2 054-496-8405

1 1 847-230-9795

2 1 02-671-9918

2 2 054-803-3356

2 3 501-629-8620








id type

1 work

2 mobile

3 fax

4 home

person_id phone_number_type_id phone_number

1 2 054-496-8405

1 1 847-230-9795

2 1 02-671-9918

2 2 054-803-3356

2 3 501-629-8620

SELECT P.first_name, P.last_name, P.email, PN.phone_number, PNT.typeFROM People P, Phone_Numbers PN, Phone_Number_Types PNTWHERE PN.person_id = P.id AND PNT.id = PN.phone_number_type_id





Another language!

• SQL is the language of relational databases

• So a Web app will use a language (e.g., Ruby, Python, or PHP) + SQL

• Or use an ORM, which automatically translates your language into SQL

Person.first.phone_number

Extending our diagram

Browser


Browser

HTTP Request


Browser Server

HTTP Request


Browser Server

HTTP Request


Browser Server

HTTP Request

Database


Browser Server

HTTP Request

Database


Browser Server

HTTP Request

HTTP Response

Database

Popular databases

• PostgreSQL (my favorite)

• MySQL

• Microsoft SQL Server

• Oracle

• Most popular: SQLite!

Scaling problems

• Lots of requests?

• Optimize

Scaling problems


• Optimize

• Even more requests?

• Buy a bigger server

Scaling problems


• Optimize

• Even more requests?

• Buy a bigger server

• What next?

• Panic! (Or spend lots of money)

Sharding

• Split your data across multiple databases

• This works, but...

• Requires rewriting a lot of code

• Maintenance is a big issue

• Re-sharding as each server gets overwhelmed can be expensive, time-consuming

Non-relational databases

• Don’t store things in tables!

• Don’t pre-define a schema

• No SQL!

• Indeed, known as “NoSQL” databases

• Only common factor: No SQL, non-relational

Examples

• Key-value stores

• e.g., Memcached, Redis, Tokyo Cabinet

• Columnar databases

• e.g., Cassandra

• Document databases

• e.g., MongoDB, CouchDB

MapReduce / Hadoop

• Google and Yahoo do it like this:

• Split data across many servers

• Run a function on all of those servers

• Collect the results

• Display to the user!

• (Too slow? Add more servers!)

NoSQL: Good news

• Often easier to administer, configure

• Scaling to multiple servers is a no-brainer

• No new programming language (SQL)!

• Better fit for certain kinds of data

• Better performance than a relational DB

• Lots of options to choose from!

NoSQL: Bad news

• Speed is in the eye of the beholder

• Is “eventually consistent” good enough?

• Non-normalized data — ugh!

• Reporting can be harder

• Less tested than relational databases

• Can you trust your data to them?

Meanwhile...

• Our browsers are displaying HTML

• HTML had several problems:

• Standardized set of tags

• Making it easy for programs to parse

• Semantic, display content were mixed

HTML standards

• HTML — several versions, several standards, none universally accepted

• XML — lets us create HTML-like languages, for computer conversations

• XHTML — HTML for pedantic people

• It was a big mess!

HTML5

• Relaxes much of the formality of XHTML, while remaining easy for computers to read

• Backward compatible to a large degree

• Adds a number of tags for better semantics

• Best of all: Lots of new JavaScript goodies

• More on this in a moment

HTML5 declaration

HTML5 declaration

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd




HTML5 declaration

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<!DOCTYPE html>





New “email” tag

<input type=”text” name=”email” />

<input type=”email” name=”email” />

New “url” tag

<input type=”text” name=”url” />

<input type=”url” name=”url” />

New “date” tag

<input type=”date” name=”date” />

Color picker!

<input type=”color” name=”color” />

And more

• Validation — built-in validation of form element inputs, via regular expressions

• No more JavaScript plugins!

• Sliders — more natural numeric inputs

• Canvas — draw any picture you might like, and detect/change it with software

• Hints in text fields (e.g., “search”)

My favorite

• Private data in attributes!

• Any attribute starting with “data-” is considered valid

• A great way to stash information inside of HTML elements without violating standards

Oh, yeah

• These don’t all work.

• Many of them don’t work on any browser

• Most work on only some browsers.

• What to do? Wait. Or use Modernizr, which uses JavaScript to detect features.

• If a feature isn’t there, you can fall back

CSS

• Cascading Style Sheets

• Split semantic markup from presentation

• One CSS file can apply to an entire site

• No more “style” tags in your text!

• Easy to move place things, create effects

Ids are unique

<p id=”important”>Agenda</p>

p#important {

font-size: 13p;

font-weight: bold;

}

Classes can repeat

<p class=”important”>Agenda</p>

<p class=”important”>Lunch</p>

p.important {

font-size: 13p;

font-weight: bold;

}

It gets better

• You can set styles for when the user’s mouse hovers over or clicks on an element

• In other words: Cheap animation!

• Many uses of JavaScript (e.g., some menus) can now be done with simple CSS

• You can make beautiful sites with CSS

CSS3: Cool effects

• Rounded corners

• Transparency

• Text shadows and drop shadows

• Gradients

CSS3: Cool selectors

• If you love regular expressions, then these selectors will be second nature to you:

p[id=”foo”] { background: green}

p[id^=”foo”] { background: green}

p[id$=”foo”] { background: green}

p[id*=”foo”] { background: green}







Equals







Starts with







Ends with






p[id*=”foo”] { background: green}Contains

OK, that’s nice.

• But really, the big news with HTML5 doesn’t have to do with HTML at all.

• Instead, it has to do with JavaScript.

• Remember JavaScript?

• It’s the programming language that we love to hate. (At least, I used to.)

JavaScript

• Originally “LiveScript,” a language that executes programs in the browser

• Renamed “JavaScript” when an unrelated language (“Java”) stole the thunder

• Renamed (officially) ECMAScript for standardization purposes

• No one actually calls it this

HTML5 turns the browser into a smart, powerful, networked application platform.JavaScript makes it

possible.

Powerful? Huh?

• Didn’t I just say that I love to hate JavaScript?

• And then I said that it was powerful?

• What gives?

JavaScript isthe new hotness

• Browsers are competing for fastest, best

• Google’s V8

• Mozilla’s TraceMonkey (and JägerMonkey)

• Safari’s Nitro

• IE’s Chakra

• JavaScript is faster, more stable than ever!

Server-side JavaScript!

• The latest JavaScript development

• Run it on your server!

• Why? Because it’s super-fast

Also: frameworks

• Remove the drudgery of JavaScript

• Handle differences between browsers

• Make it easy to perform common tasks

• Lots of plugins available

• For me, it’s the difference between pain and pleasure when working with JavaScript

JavaScript frameworks

• Dojo

• YUI

• MooTools

• Prototype

• jQuery (the 900-pound gorilla)

jQuery

• jQuery has taken the world by storm

• Super-easy to use

• Extremely fast

• Open source (of course!)

• Easy to write plugins

• Lots of plugins are available

Ajax

• One reason for JavaScript libraries: Ajax!

• Make HTTP requests from within the page

• No refresh! Just get results from the server, and modify the page accordingly

• This has revolutionized our view of Web pages

Ajax

Browser Server

Ajax

Browser Server


Ajax

Browser Server



Ajax

Browser Server



Ajax

Browser Server



Ajax

Browser Server



Ajax

Browser Server



Ajax isn’t everything

• What if I want a chat application, or something else that stays open?

• What if I want to execute more than one JavaScript function at a time?

• What if I want to store things locally?

• HTML5 provides all this — and more!

Canvas

• A complete drawing area, in your browser!

• Use JavaScript to:

• Draw arbitrary shapes

• Detect the mouse

• Detect the drawing

• The end of Flash? Maybe...

Geolocation

• Your browser can know where you are!

• It can send this info to the server

• It’s not perfect, but still pretty good

• To avoid privacy issues, users are always asked if their location should be shared

Inter-page communication

• Modern Web apps can span multiple pages

• HTML5 makes it easy for two pages (from the same server) to communicate

• The receiver knows which server sent the data — so it can filter incoming messages, as well as screen them for security

Web sockets

• This is potentially the biggest deal of all

• Ajax allows for server connections. But:

• High overhead

• Stateless

• Web sockets have low overhead, and they stay open as long as you need!

Using Web sockets

Using Web socketsvar weatherSocket = new WebSocket("ws://localhost:8080");


weatherSocket.onopen = function(e) { alert("Opened weather socket");};



weatherSocket.onmessage = function(e) { alert("Got message: " + e.data);};



weatherSocket.onmessage = function(e) { alert("Got message: " + e.data);};

weatherSocket.onclose = function(e) { alert("Closing socket..."); };

What can sockets do?

• Chat servers

• Stock feeds

• Teleconferencing

• Who knows?

• Remember, HTTP was only invented after sockets had been around for 15 years

Web workers

• Execute more than one thing at a time

• In other words: You can run JavaScript functions in the background

• Process text

• Open Web sockets

• Perform calculations

Local storage

• Now Web apps can store data

• A little database of name-value pairs

var foo = localStorage.getItem("bar");

localStorage.setItem("bar", foo);

Local storage





var foo = localStorage["bar"];

Local storage





var foo = localStorage["bar"];

localStorage["bar"] = foo;

Media

• Standard (well, sort of) ways to play audio and video

• No more Flash!

• Problem: No one format is supported by all browsers

Rich client-side frameworks

• Backbone

• Ember

Design frameworks

• Twitter bootstrap

• Blueprint

• Compass

Don’t forget mobile!

• iOS and Android are growing massively

• Web site vs. native app?

• (Ask Jakob Nielsen — for now, apps are better, but that won’t last for long)

• Ignore these users at your peril

Want a Web app?

• It used to be:

• “What operating system, language, and database will I use?”

• Or:

• “How can I produce an interesting Web site?”

Now it’s:

• What experience do we want to give people?

• What will they be using to access our system?

Those lead to a wide variety of questions:

• What server language and framework? JavaScript framework? Hosted or cloud?

• What type of database (relational, NoSQL)? Which one? Hosted or cloud?

• Do we offer an API? A mobile app? Both?

• Which HTML5 features do we want to use, and how do we gracefully degrade?

The key takeaway

• Web sites are far more than just static text, blogs, or forums

• They’re full-fledged software applications

• Take advantage of these technologies, and you can create a fabulous experience

• (Don’t take advantage of them, and your competitors will!)

Oh, yeah: Testing

• Automated testing is amazing

• Your Web site should use it

• Most modern frameworks support some sort of testing — if not, change framework

• Catch dumb bugs and issues before your customers do!

• Faster and cheaper than people

My brain is too small!

• Yes, there’s a lot to learn

• Most of it can wait a little bit

• There are oodles of tutorials and books that can help you

• Besides a lot of this is still highly transitional

Whew!

• There’s a lot to the modern Web

• It’s fun and exciting, and continues to move forward at breakneck speed

• Understanding as many of these parts as possible will help you make better decisions, and better applications!

Any questions?

• Call me: 054-496-8405

• E-mail me: [email protected]

• Interrupt me: reuvenlerner (Skype/AIM)

