HTTP::Parser::XS - writing a fast & secure XS module

HTTP::Parser::XSwriting a fast & secure XS module

Cybozu Labs, Inc.Kazuho Oku

Pros & Cons of XS

ProsFaster (well, not always, but…)

ConsDifficult and time-consumingHigher security risk

Mistakes in C programming leads to remote code injection

May 28 2010 HTTP::Parser::XS - writing a fast & secure XS module 2

The Best Practice, in General

Don’t use XSUntil you hit performance bottleneck

Rewrite the bottleneck (only) using XS


HTTP::Parser::XS


Summary

An HTTP request parserDesigned and implemented to be

simple, fast & secureStatelessPSGI-compatible

Required or recommended by many Plack serversStarman, Starlet, Twiggy, etc.


Two-layered Approach

PicohttpparserHTTP request / response parser written in Chas its own test suitedoes not parse request / response content

request line (response line) and headers onlycopy-less

faster, lesser probability of security holes and memory leaks

HTTP::Parser::XSglue code to access picohttpparser from

perlMay 28 2010 HTTP::Parser::XS - writing a fast & secure XS module 6

Speed and Complexity


reqs/sec. lines of code

HTTP::HeaderParser::XS

116,000 1,166

HTTP::Parser::XS 140,000 487

Plack::HTTPParser::PP 10,100 104

HTTP::Parser::XS is simple, and fastmost of the time is not spent in

picohttpparser, but in the glue code constructing hashrefpicohttpparser/trunk can handle >1Mreqs/sec.

Why is it a Stateless?

Faster and simpler than a stateful parserlower security risks

most HTTP requests / responses arrive in a single packet, anyway

if written optimally in C, the cost of re-parsing is smaller than storing headers into a perl hashref


Why is it Stateless? (cont’d)

Easy to determine the end of an multi-packet HTTP requestby looking for “\r\n\r\n” within the last

packet (and preceding three bytes)mainly as a countermeasure for Slowloris


Consistent Design

Functions take same arguments buf – points to current char buf_end – points to end of buffer *ret – error value returns pointer to the next char

or null on error (the reason will be stored in *ret)


const char* parse_http_version(const char* buf, const char* buf_end, int* minor_version, int* ret){ EXPECT_CHAR('H'); EXPECT_CHAR('T'); EXPECT_CHAR('T'); EXPECT_CHAR('P'); EXPECT_CHAR('/'); EXPECT_CHAR('1'); EXPECT_CHAR('.'); return parse_int(buf, buf_end, minor_version, ret);}

Macros

Consistent design is essential to heavy use of macrosGood abstraction ⇒

safe code


#define CHECK_EOF() \ if (buf == buf_end) { \ *ret = -2; \ return NULL; \ }#define EXPECT_CHAR(ch) \ CHECK_EOF(); \ if (*buf++ != ch) { \ *ret = -1; \ return NULL; \ }

const char* parse_http_version(const char* buf, const char* buf_end, int* minor_version, int* ret){ EXPECT_CHAR('H'); EXPECT_CHAR('T'); EXPECT_CHAR('T'); EXPECT_CHAR('P'); EXPECT_CHAR('/'); EXPECT_CHAR('1'); EXPECT_CHAR('.'); return parse_int(buf, buf_end, minor_version, ret);}

Micro-optimization (only in picohttpparser)

Reduce # of conditional branches, optimize for pipeline (27% faster)

Unroll loops (36% faster)


#define unlikely(x) __builtin_expect(!!(x), 0)

for (; ; ++buf) { CHECK_EOF(); if (unlikely((unsigned char)*buf <= '\r') && (*buf == '\r' || *buf == '\n')) goto EOL_FOUND;}

Release History of HTTP::Parser::XS

Current version: 0.07No security hole found since initial

releaseTwo memory leaks were found and

fixed

Please let me know if you find any security holes (especially the ones that lead to arbitrary code execution)


Conclusion – KISS

Keep it simple, stupidfor fast developmentsimple design leads to more secure codeuse perl whenever possible

simple operations (like tokenization) is worth converting to XS

complex operations (from handling of strings to database queries) are not so slow in perl


Technology

HTTP::Parser::XS - writing a fast & secure XS module