WEB APPLICATION PENETRATION TESTING

HTTP BASICS A Deeper analysis of the HTTP Protocol

HTTP is a stateless protocol. The web works in and around this

protocol. To manifest and understand the HTTP Protocol and its

derivatives keeping web application penetration testing in scope,

we move on to study this particular protocol to its root.

To the hardworking people and to ninjas!

The HTTP Protocol

The Basics of HTTP

Protocol in Security.

HTTP Versions

HTTP Authentication

Security Centric

HTTP Research

Paper

OPENFIRE TECHNOLOGIES

Author: Shritam Bhowmick

22nd February, 2014

INTRODUCTION

This paper drafts out the required knowledge about the HTTP protocol in use by

web technologies in a brief format to study the security risks affected to the

same protocol in later versions of the undergoing research documents. HTTP

protocol is the widest and most recently adopted protocol since 1996. The first

versions of HTTP were version 0.9 and quickly changed to 1.0 after IETF (Internet

Engineering Task Force) recommended protocol developers to maintain strict

streamlined compression due to huge traffic flow it has to go through. After the

version 1.0 in 1997, year 2000 saw the dawn of web 2.0 and soon version 1.1

came out to be the proposed standard HTTP protocol in use. HTTP version 1.1 is

still prevalent today (year 2014) and most discussion on development of version

2.0 of the same protocol is undergoing development and experimentation.

One can go through RFC’s at IETF to study HTTP and other internet based

protocols and their massive progression. HTTP also plays an important role in

web applications, as a web application has to be accessed via text based

protocol such as HTTP itself and therefore is prone to security flaws. Most security

centric web application flaw either come due to injection attacks at the server

side itself (from misconfigured, unsanitized input and poor coding practices) or

HTTP method tampering.

The paper contains a brief study on both version 1.0 and 1.1 of HTTP. HTTP 2.0 is

also added to this document for further studies. HTTP/2.0 would be

implemented to the recent HTML5 standards and hence the required

knowledge of the protocol is almost mandatory and recommended. In

continuation to this document, other document releases might pop up in

conjunction to the present current topics being researched. However all the

topics being researched is kept in pace of series and in a step deduction

format to maintain legacy and generic security centric problems. All the

papers are concerned with web application flaws, web application

vulnerability research, web application vulnerability vectors and web

application exploitation and penetration testing.

HTTP BASICS

What is HTTP?

HTTP is a stateless protocol. Protocols are a set of rules which has to be

followed to maintain a strict boundary of instructions for a networking system.

HTTP stands for Hypertext Transfer Protocol which in itself suggests it is a protocol

based out of text commands and used to transfer data via these pre-set of

command instructions. The commands itself are test supported and therefore it

makes easy for a networking point of view to collaborate, distribute, interact

and communicate among hypertext networking systems. Statelessness of HTTP

protocol means the protocol supports being on different state from the original

one. This means if a request is sent from a client to a server via a medium; this

newly introduced medium could tamper with the request methods and then

send the newly formed request to the server. Now this has security concerns;

which is why this paper is all about. It’s to know the HTTP protocol closely.

The World Wide Web largely depends on the HTTP protocol, as the protocol

itself is light, fast, efficient and dependent on use of status codes, headers and

request methods. This use of dependent headers, status code and request

methods makes the HTTP protocol suited the best for any packet based

communication in-between a network, group of network or an isolated

network. The study of HTTP is a must for any web application security enthusiast

as later for HTTP tampering, HTTP based attacks, HTTP CR/LF attacks, and other

HTTP attacks; HTTP based required knowledge would be needed. It’s to be

added that HTTP is an application level protocol and works at the application

level of the OSI model. Find the references of HTTP below.

HTTP is said to be asymmetric request-response client-server protocol. It’s also a

‘pull’ based protocol which means the client pulls out information from the

server rather than the server ‘pushing’ the information back to the client. To

study HTTP protocols, we must have a basic idea of an URL or Uniform Resource

Locator. There are URI and URN too. That comes later.

References:

1.) http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol

2.) http://www.w3.org/Protocols/HTTP/AsImplemented.html

Terminology:

Referenced from: http://www.w3.org/Protocols/rfc2616/rfc2616-

sec1.html#sec1

Connection: A transport layer virtual circuit established between two programs

for the purpose of communication.

Message: The basic unit of HTTP communication, consisting of a structured

sequence of octets matching the syntax defined in section 4 (of the referenced

RFC) and transmitted via the connection.

Request: An HTTP request message.

Response: An HTTP response message.

Resource: A network data object or service that can be identified by a URI, as

defined in section 3.2. Resources may be available in multiple representations

(e.g. multiple languages, data formats, size, and resolutions) or vary in other

ways.

Entity: The information transferred as the payload of a request or response. An

entity consists of metainformation in the form of entity-header fields and

content in the form of an entity-body, as described in section 7.

Representation: An entity included with a response that is subject to content

negotiation, as described in section 12. There may exist multiple representations

associated with a particular response status.

Content negotiation: The mechanism for selecting the appropriate

representation when servicing a request, as described in section 12. The

representation of entities in any response can be negotiated (including error

responses).

Variant: A resource may have one, or more than one, representation(s)

associated with it at any given instant. Each of these representations is termed

a `variant'. Use of the term `variant' does not necessarily imply that the resource

is subject to content negotiation.

Client: A program that establishes connections for the purpose of sending

requests.

User agent: The client which initiates a request. These are often browsers,

editors, spiders (web-traversing robots), or other end user tools.

Server: An application program that accepts connections in order to service

requests by sending back responses. Any given program may be capable of

being both a client and a server; our use of these terms refers only to the role

being performed by the program for a particular connection, rather than to

the program's capabilities in general. Likewise, any server may act as an origin

server, proxy, gateway, or tunnel, switching behavior based on the nature of

each request.

Origin server: The server on which a given resource resides or is to be created.

Proxy: An intermediary program which acts as both a server and a client for the

purpose of making requests on behalf of other clients. Requests are serviced

internally or by passing them on, with possible translation, to other servers. A

proxy MUST implement both the client and server requirements of this

specification. A "transparent proxy" is a proxy that does not modify the request

or response beyond what is required for proxy authentication and

identification. A "non-transparent proxy" is a proxy that modifies the request or

response in order to provide some added service to the user agent, such as

group annotation services, media type transformation, protocol reduction, or

anonymity filtering. Except where either transparent or non-transparent

behavior is explicitly stated, the HTTP proxy requirements apply to both types of

proxies.

Gateway: A server which acts as an intermediary for some other server. Unlike a

proxy, a gateway receives requests as if it were the origin server for the

requested resource; the requesting client may not be aware that it is

communicating with a gateway.

Tunnel: An intermediary program which is acting as a blind relay between two

connections. Once active, a tunnel is not considered a party to the HTTP

communication, though the tunnel may have been initiated by an HTTP

request. The tunnel ceases to exist when both ends of the relayed connections

are closed.

Cache: A program's local store of response messages and the subsystem that

controls its message storage, retrieval, and deletion. A cache stores cacheable

responses in order to reduce the response time and network bandwidth

consumption on future, equivalent requests. Any client or server may include a

cache, though a cache cannot be used by a server that is acting as a tunnel.

Cacheable: A response is cacheable if a cache is allowed to store a copy of

the response message for use in answering subsequent requests. The rules for

determining the cacheability of HTTP responses are defined in section 13. Even

if a resource is cacheable, there may be additional constraints on whether a

cache can use the cached copy for a particular request.

First-hand: A response is first-hand if it comes directly and without unnecessary

delay from the origin server, perhaps via one or more proxies. A response is also

first-hand if its validity has just been checked directly with the origin server.

Explicit expiration time: The time at which the origin server intends that an entity

should no longer be returned by a cache without further validation.

Heuristic expiration time: An expiration time assigned by a cache when no

explicit expiration time is available.

Age: The age of a response is the time since it was sent by, or successfully

validated with, the origin server.

Freshness lifetime: The length of time between the generation of a response

and its expiration time.

Fresh: A response is fresh if its age has not yet exceeded its freshness lifetime.

Stale: A response is stale if its age has passed its freshness lifetime.

Semantically transparent: A cache behaves in a "semantically transparent"

manner, with respect to a particular response, when its use affects neither the

requesting client nor the origin server, except to improve performance. When a

cache is semantically transparent, the client receives exactly the same

response (except for hop-by-hop headers) that it would have received had its

request been handled directly by the origin server.

Validator: A protocol element (e.g., an entity tag or a Last-Modified time) that

is used to find out whether a cache entry is an equivalent copy of an entity.

Upstream/downstream: Upstream and downstream describe the flow of a

message: all messages flow from upstream to downstream.

Inbound/outbound: Inbound and outbound refer to the request and response

paths for messages: "inbound" means "traveling toward the origin server", and

"outbound" means "traveling toward the user agent"

What is an URL?

URL is short for Uniform Resource Locator. A generic URL has four parts to it:

1. Protocol

2. Hostname

3. Port

4. Path and file

To breakdown all of the mentioned 4 entities of an URL, let’s take how an

actual URL would look like.

http://www.mytestsite.com:80/destination_path/index.htm

Here:

1. Http is the protocol.

2. www.mytestsite.com is the host.

3. 80 is the port, by default if it’s not mentioned, 80 is taken as default.

4. Destination_path/index.htm is the destination directory with a file within.

Broadly, HTTP is the stateless protocol used to communicate or request a

resource from the server from the client. HTTP works on TCP suite and 80 is the

default port number for the server to listen on which acts as a HTTP server.

Hostname is www.mytestsite.com which is again the DNS host and it could be

an IP too such as for localhost, it could be something like “192.168.78.11”. The

rest is ‘path and the filename’ of the requested resource on the server. The

server may have directories and sub-directories and end up with a particular

file that is being requested. In this case it’s index.htm. There could had been an

index.htm file under the document root directory of the server too, however

here for illustration purposes, the ‘directory’ path is shown for conveying

directories could be included as well. HTTP is an application level protocol.

Application level protocols.

There are different application level protocols for the client and the server to

talk to each other. These protocols are varied and few of them are mentioned

below for reference purposes only.

1. SMTP: Smart Mail Transfer Protocol.

2. GMTP: Group Mail Transfer Protocol.

3. FTP: File Transfer Protocol.

4. TFTP: Trivial File Transfer Protocol.

5. ARP: Address Resolution Protocol.

6. RARP: Reverse Address Resolution protocol.

7. Telnet: Remote Terminal Access Protocol.

8. ADC

9. AFP

10. BACnet

11. BitTorrent

12. BGP: Border Gateway Protocol.

13. BOOTP: Bootstrap Protocol.

14. CAMEL

15. DiCOM: An Authentication, Authorization and Accounting Protocol.

16. DICT: Dictionary Protocol.

17. DNS: domain Name System.

18. DSM-CC: Digital Storage Media Command and Control.

19. DNSP: Dynamic Host Control Protocol.

20. ED2K: A peer to peer protocol.

21. Finger: User profile Information Protocol.

22. Gnutella: Peer to peer file swapping protocol.

23. Gopher: Hierarchical Hyperlinkable protocol.

24. HTTP: Hypertext Transfer Protocol.

25. HTTPS: Hypertext Transfer Protocol Secure.

26. IMAP: Internet Message Access Protocol.

27. IRC: Internet Relay Chat.

28. ISUP: ISDN User Part.

29. LDAP: Lightweight Directory Access Protocol.

30. MIME: Multipurpose Internet Mail Extensions.

31. MSNP: Microsoft Notification Protocol.

32. MAP: Mobile Application Protocol.

33. NetBIOS: File sharing and Name Resolution Protocol for Windows.

34. NNTP: Network News Transfer Protocol.

35. NTCIP: National Transportation Communications for Intelligent

Transportation System Protocol.

36. POP3: Post Office Protocol Version 3.

37. RADIUS: An Authentication, Authorization and Accounting Protocol.

38. RDP: Remote Desktop Protocol.

39. Rlogin: A UNIX remote login protocol.

40. Rsync: File Transfer Protocol for backups, copying and mirroring.

41. RTP: Real-Time Transport Protocol.

42. RTSP: Real-Time Transport Streaming Protocol.

43. SIP: Session Initiation Protocol.

44. SISNAPI: Siebel Internet Session Network API.

45. SNMP: Simple Network Management Protocol.

46. SMB: Microsoft Server Message Block Protocol.

47. STUN: Session Traversal Utilities for NAT.

48. TUP: Telephone User Part.

49. TCAP: Transaction Capabilities Application Part.

50. WebDAV: Web Distributed Authoring and Versioning.

51. XMPP: Instant-Messaging Protocol.

52. SSH: Secure Shell.

53. SOAP: Simple Object Access Protocol.

54. DHCP: Dynamic Host Control Protocol.

This does not end the application level protocols however, because

application protocols keep emerging from recent times, new applications

implement their own protocols and therefore is updated in continuation.

Application protocol take the support of the underlying transport protocol in

the OSI model. There are 7 layers in the OSI model; application and transport

layer are one of them. In web application exploitation, we will generally

confront and will be restricted in and around these two specified layers. The

application layer protocol generally varies with programs or software’s in use

because they require different environment, different negotiations beside

packet synchronization at the TCP/IP level and there are much more factors

which results in a varied protocol for certain ‘software’s’ or ‘programs’. That

been said, we will now confront with the most used protocol which is ‘HTTP’ and

could be easily used by a browser or any other client which supports the HTTP

protocol. The first of its kind being HTTP 0.9 and later extended to version 1.1 for

better performances and added functionality!

References: http://www.slideshare.net/Eacademy4u/application-protocols

HTTP/1.0

HTTP/1.0 is the older HTTP standard in succession to HTTP/0.9. HTTP/1.0 is still

under use other than HTTP/1.1 used widely. Besides HTTP/0.9, HTTP/1.0 had major

improvements to it. HTTP/1.0 is a hypertext based protocol which could be

tested via telnet on windows or on a Linux machine. Other operating system

works too. All the requests made to a server for the availability of a certain

resource is requested via telnet and the commands end with a CR/LF. CR is

Carriage Return and LF is Line Feed. When a site doesn’t support this protocol

because it’s old, a forced response is bounced back to the telnet session.

Below, attached are screenshots taken while communication via a program

called ‘telnet’ which supports the ‘telnet’ protocol. It could communicate to a

remote server which is based off the HTTP protocol and the default port as

shown is ‘80’ because port number ‘80’ is the default HTTP server port. Telnet

could be also used to connect to other ports. Later we would see how we

could achieve the same via a Swiss –army knife tool called ‘netcat’. It’s

dubbed as Swiss-Army Knife as it provides major additional functionality along

with listening to certain ports and communicating to remote ports on basis of

the situation. We will further take it to Nmap (a network mapper) and using its

internal NSE (Nmap Scripting Engine) determine the services running, detection

of web applications running, determine if there is a web application firewall

present and a variety of tasks to automate a penetration test task.

In HTTP/1.0 each connections are closed after a request/response session. The

connection is closed by the server itself. HTTP/1.0 has TCP slow-start

disadvantage, references to TCP Slow start has been documented at the

reference section of this document. HTTP/1.0 supports caching via its header ‘If-

Modified-Since’. It is not mandatory to specify the HOST header in HTTP/1.0 but

it allows to add one. Overall HTTP/1.0 is faster than 0.9 and comparatively

slower than the newly introduced HTTP version 1.1.

References: TCP-Slow-start: http://en.wikipedia.org/wiki/Slow-start

HTTP/1.0 Header fields are different and could be classified as the following:

HTTP_ACCEPT (or ACCEPT)

HTTP_USER_AGENT (or User-Agent)

Content-Type

Date

Expires

From

If-Modified-Since

Last-Modified

Location

Referer

Server

Various fields describes various functions. Because field information is

information about the information that is being sent through the protocol, it is

termed as ‘meta-information’. Therefore HTTP fields are meta-information.

Anything preceding the actual data (in HTML, CSS or other form) is the HTTP

Header Fields. The HTTP header fields are responsible for crypto-identification,

authentication and user cookies. The above header fields are supported by

HTTP version 1.0. A brief description of what these headers do are mentioned in

this document.

HTTP_ACCEPT (or Accept) header specifies what MIME-types the client would

accept provided the client request a resource from a remote server. The MIME-

type has to be divided with type and subtype along with the Accept: header

specification. Each item on this specified header must be separated by a

comma. Examples could be: Accept: txt/xml, xml/css, …

HTTP_USER_AGENT header (or User-Agent) header specifies the browser agent in

use at the client side to get the resources from the server. The general

representation of the User-Agent header is in format: User-Agent:

Software/version_library/version; the User-Agent is used to traceback protocol

violation errors, statistical uses to record what client software is in use and for

tailoring response in accordance to the User-Agent to attempt more efficient

browsing experiences. The study of User-Agent in web application security is

due to the fact that the servers being able to intercept the User-Agent as

different format makes the User-Agent vulnerable to User-Agent Injections such

as Cross Site Scripting attacks and SQL Injection attacks. Malware analyst

frequently use the User-Agent fields to study botnets because generally a

malware source would have a different User-Agent string which might look

abnormal. The specification of User-Agent header could be blank or kept

omitted, but the RFC’s say it is mandatory.

Content-Type header field states the media type of the body sent by the server

to a client. The media types are in accordance to the RFC’s. Example

specification for Content-Type could be as: Content-Type: txt/html;

charset=ISO-8859-4

Date header field represents the HTTP-date and the time the HTTP message

originated. There are three standards to mention the date field according to

the RFC. All these three standards are later supported by the recent HTTP

version 1.1; the first standard and the third standard provides the year, however

the second standard commonly in use doesn’t say the year of the message

originated. Syntax for use is Date: HTTP-date, example:

Expires header field gives the notion of the content to be expired or cease to

be valid after a certain period of time. This time stamp and the data is given by

the Expire header field. After this data and time, the content of the HTTP

message could not be accessed by any caching servers, proxies or a client

until and unless the originating party updates the expire field and lets the

content to be accessed or extends the date and time stamp for validity.

Format for specifying the Expire header is as: Expires: HTTP-expire-date-time,

example:

From header among the HTTP fields are used to specify an email address for

reference and to provide details of the human factor involved in making the

request. Robot spiders use this field to let the servers know the responsible head

to make such a request containing the request parameters to access any

particular resources on a remote server. The format for using the From header is,

From: example_email_address@example_domain.com

This header is generally for reference purposes and should not be taken as an

access token for any authorization.

If-Modified-Since header field is to request a resource which has been modified

in relation to the date and time mentioned on the header field. If the resource

is not modified in accordance to the time and date stamp mentioned in the

header, then the server responds with a 304 which states ‘Not Modified’ error.

Here ‘304’ is status code. Status code will be discussed in this document at later

stages. It is to check if the content is modified and older content is not served

to the client. The format is If-Modified-Since: If-Modified-Since_data_year_time,

for an example:

If-Modified-Since is a conditional HTTP header field.

Last-Modified header field is a HTTP header which states the date and time of

the content which were last modified. If the content was modified at an older

outdated date; then the requested content which was outdated, the cache

would take care of it and thus not requiring the content transfer once again. It

saves page rendering and requesting obsolete content which were not

modified and the content which the client already had. It saves unnecessary

data transfers. The format could be Last-Modified: Date_year_time; for an

example:

The Last-Modified header saves unnecessary data transfer and hence cache

comes into play while data packet transfer via the HTTP protocol.

Location header field is a HTTP header which is triggered by the server to let

know the client that the URI (Uniform Resource Identifier) the client has

provided is moved to another location and send the ‘location’ of the path to

this header field (Location). The client then picks up the path and request

another ‘request’ to the server including the specified ‘path’ to the server to

achieve the resource. One can use ‘curl’ with ‘-IL’ switch to inspect the

‘Location’ header and see how it works.

The diagram shows, http://google.com was requested to from the google

server, the server replied with a ‘301’ status code which meant ‘moved

permanently’ and added a ‘Location’ header to its response adding ‘www’ to

the original requested URI. Now, the client picks up this path and requests the

server again to query if the resource was available, to which the server

responded ‘moved temporarily’ this time but with a status code of ‘302’ which

exactly meant ‘moved temporarily’ (we’ll look to status codes later in this

document). The server responded with a ‘Location’ header again and the

client picks up the path and sends the 3rd query as a request for the resource

and this time the server responds with a ‘200 OK’ status code, which meant the

resource the client was looking for was found on the server and could now

serve the resource.

Referer in the HTTP header field is generally for the servers benefit. Referer might

look like a mis-spell and originally it ‘was’ a mis-spell (ftp://ftp.isi.edu/in-

notes/rfc2616.txt RFC 2616, section 14.36) but due to it being mis-spelled at the

first place and the authors not caring to alter the RFC went ahead to keep it

official. Referer field is for maintenance, optimized caching and logging

requests. The Referer field is asked (rather requested!) by the server web

application to the client to provide the server with the originating ‘request URI’.

This could be for inspection purposes, could be for determining from where the

traffic comes and might be for back-linking.

The Referer header is prone to spam, injections, scripting attacks from a web

application security aspects. References are attached below for a read.

References: http://resources.infosecinstitute.com/sql-injection-http-headers/

References for malware inspection purposes:

http://isc.sans.edu/diary/The+Importance+of+HTTP+Headers+When+Investigati

ng+Malicious+Sites+/10279

Server HTTP header field is a response header field from a server upon

responding to a request. This field is used to provide information on the web

technologies being used on the server side to handle the requests sent by a

client. The server header from a web application security aspect, provides

informational data like the version numbers of the web technology (HTTP

Daemon being used {httpd}) being used and more. A sample server header on

a HTTP communication between a client and a server would look like:

That being said, we would now move on to HTTP 1.1 (version 1.1) which are the

current official HTTP communication protocol. We had taken look onto how

HTTP/1.0 works for most of the clients and how the server responds to the query

of the clients does. We also looked at how the HTTP protocol itself is a ‘pull’

protocol and not a ‘push’ protocol. By that it is meant, the client requests and

pulls out information from the server and the server responds back to the

client’s requests. The server itself does not ‘push’ the information to the client.

Furthermore, we went to look at some HTTP/1.0 headers which will be still at use

on HTTP/1.1 headers. This is because a vast number of clients and the internet

itself depends on these header fields. For an instance, the HTTP version and the

GET/POST requests containing headers like ‘server’,’User-Agent’,’Content-Type’

and many more are of prime importance.

HTTP/1.1

Welcome to HTTP/1.1, a further enhancement made for HTTP/1.0 protocol.

Since we will mostly be dealing with HTTP/1.1 security concerns and later; we

would take a much closer look to HTTP/1.1 then HTTP/1.0

The HTTP daemon is a HTTP server which accepts and client requests and

responds to these requested based on the availability of the resource,

authorization of the resource asked for and location of the resource been

asked for. All requests are answered via an appropriate status code as we saw

earlier with HTTP/1.0

The overall operation could be much more complicated if a tunnel, proxy or

other clients were involved in the process. As implied in the ‘terminology’

section of this document, ‘Origin Server’ is the server which ‘has’ the resource

asked for by the client. The client may be asking for the resource via a

forwarding agent which could be a proxy. A proxy in this case could then be a

mediator, it accepts the requests made by the client, parses it and adds its own

attributes or re-writes the HTTP messages and then forwards it to the server. Or

this might not happen, because if the proxy was supposed to forward this to a

gateway, it could forward to a gateway as well. A gateway is another

mediator here, which accepts the requests made by the proxy, and then

without any modifications sends this message to other involved in the process.

The last client or the party receiving the request but not before making this

original request to the ‘Origin Server’ is the ‘party’ which would get the respond

back from the server and the server has to respond back to only this party. Now

the original client which made the request is called the ‘User Agent’ and the

one which this request was meant for originally is the ‘Origin Server’. There

might be at times that the connection or the request chain could be denied to

other parties provided the rules that they do not use any ‘tunnels’ or do not

modify the original request or be in range of the neighboring ‘User Agent’.

To keep it simple, one should know that the request chain is always in some or

other way falls under complex variety of ‘mediator’ proxies, tunnels, other

clients, other servers etc... And hence this request could be modified in several

factors if security was to be a point. We’ll come to tunnels later. Basics first, we

would discuss HTTP/1.1 broadly.

The diagram and the protocols doesn’t matter as long as you know what’s

going and what is attempted to make the connections clear. Here the ‘peer0’

could consist bunch of other clients as well, finally reaching to the ‘server’

which is the ‘origin server’.

The HTTP/1.1 is designed to maintain the wide diversity among the web

technologies used and being deployed. This happens without un-necessary

steps to change the protocol itself, because the protocol is maintained in such

a way that it negotiates the connection and data transfer to maintain the

diversity which is in use. And if any ‘errors’ occur at any point, the HTTP should

be able to diagnose the problem and should be able to throw back the ‘error’

via error codes or via any form which could be normally presented to the UA

(User Agent). From here on we’ll look at certain header fields when the topic

pops in. The general HTTP field headers as mentioned above for HTTP/1.0 are

common and would remain valid unless notified here in the document. We

look at the HTTP/1.1 protocol from a security prospective and look at its access

controls, authentication mechanisms provided, its caching, the range requests,

conditional requests, semantics and content, messaging syntax and routing.

We’ll also look at the encoding parameters for ‘basic’ authentication, Mutual

Authentication Protocol, use if HTTP signatures, the way HTTP manages sessions,

and beyond the HTTP basics. This specific section ‘beyond the basics’ will be

covered in this document for informational purposes and should be studied for

references and to know more about the protocol which falls under the web

application developers, as well as web application exploiters scope of

references. That been said, we would divide the HTTP/1.1 to number of chunks

of topic to pick up and to be discussed about!

HTTP/1.1 Authentication

The most relevant and broadly authentication mechanism used by the client-

server model falls under three types:

1. Basic

2. Digest

3. NTLM

HTTP Access Authentication Framework: Before moving any further onto basic,

digest or NTLM authentication schemes, one should consider studying the basic

authentication framework which the HTTP protocol itself provides. This proposed

framework is to chalk out the architectural layout of how an authentication

could be achieved whilst being on HTTP protocol over a TCP/IP stack

connection. This framework is the root of all other authentication scheme which

would follow.

First, the HTTP is a challenge-response based authentication protocol, which

means the server provides the client with a ‘challenge’ to which the client has

to solve by picking up the ‘authentication tokens’ provided by the origin server

in the headers while responding back to a client’s request. The authentication

framework of HTTP relies on a 2-way credential share, which means the client

sends a request first and then receives a 401 Unauthorized from the server ends

with a ‘realm’ value and the ‘schemes’ over which the challenge depends

and has to be solved by the client. Now if the client goes ahead and solves

these scheme provided by the server and then the server verifies the

credentials provided by the user agent that is the client, and matches the

values from the server’s internal databases and identifies a valid user, the user

agent is allowed to browse through the resource. If that fails, either the

connection terminates or the server keeps serving 401 unauthorized until the

connection itself is timed-out or the client closes up the connection.

Hitting on cancel would close the connection from the user agent or the client

side. This is how the HTTP authentication architecture works. Now we will move

on to discuss how the specifics work on basic, digest and NTLM authentication

which are common among HTTP authentication.

Basic HTTP Authentication: Over basic HTTP authentication, the client sends the

username and the password as requested by the server upon the clients

‘request’ to access a particular resource which resides in the web server. But

this username and password is encoded (not encrypted!) as Base64 and sent

to the server. This might be a security concern. If the mode being used is a pure

HTTP connection over TCP/IP and not via HTTPS (HTTP secure), the username

and the password could be decoded (not decrypted!) in clear text if any

malicious user/client/proxy sites in-between the client and the server (Origin

Server). The tokens used over this authentication protocol is always ‘case-

insensitive’ and does not require the validation to be on ‘case-sensitive’ basis.

This token is only used to identify among which of the various authentication

schemes the server and the client both negotiates and move ahead to the

actual authentication process. After this token being stated, authentication

parameters are set for ‘actual authentication’ process and consists of attribute:

value set with more parameters if necessary with commas in place between

them. For the origin server to provide an authentication challenge to the

requesting client which is a user-agent, the origin server must throw back a ‘401

status code’ which is ‘Unauthorized’ in terms of HTTP protocol status code

(more on status codes later!). The stating of ‘401’ should contain a ‘WWW-

authenticate’ HTTP header field in response to challenge the user agent.

When the same is dealt with proxy, the proxy sends ‘407 status code’ which

means ‘Proxy Authentication required’ in HTTP protocol status code terms. This is

only possible when the ‘proxy’ itself sends the HTTP header ‘Proxy-

Authentication’ with the status code mentioned. Now, the client or the original

user agent which requests the protected resource must solve the challenge;

looking at the authentication scheme token, choose if it’s compatible scheme

or not and then looking at the attributes and the parameters which is followed

after the scheme has been stated. As said before, the authentication scheme

is resolved by ‘non case sensitive’ string as a ‘token’ in the HTTP header.

There might be more than one WWW-authenticate schemes to be solved as a

challenge as the attribute and the value pair may contain other challenges to

solve via a comma separated list. The client or the user agent takes care of this

and solves them. Now this does not confine to WWW-authenticate header but

to the Proxy-Authenticate header as well. Below is a snap how it would look if a

TELNET client was used. One could do the inspection of the same using any

proxy such as ‘Httpwatch’ or ‘Burp Suite’. Here the WWW-authenticate was not

set by the server. It throws us with a status code of ‘401’ with a ‘Basic realm’. So

what is Basic Realm? And how does in come under the authentication

framework used by the HTTP protocol. We will come back to the Basic Realm

after we had solved our authentication headers and the authentication

parameters.

Now that we know that various authentication headers could be in use for

authorization to be sent to the server and hence the user agent has to decide

which authentication scheme to choose; basically the one with the strongest

scheme is chosen and then solved. But various authentication schemes in

place with different auth-scheme could have its own problems which are

security centric to web applications such as the following:

1. Online Dictionary attacks.

2. Man in the middle attacks.

3. Chosen plaintext attacks.

4. Precomputed Dictionary attacks.

5. Batch Bruteforce attacks.

6. Spoofing via 3rd party counterfeit servers.

References: http://tools.ietf.org/html/rfc2617#section-4.6

The digest authentication scheme requires the client to store passwords which

could be fetched easily if the client was compromised. But this was not

covered here as this is the BASIC authentication section. We will look to security

concerns for the digest authentication scheme whilst being on the digest

authentication topic and discuss the reasons it could be compromised.

Unlike the snap above, there could be more WWW-authenticate or Proxy-

Authenticate HTTP headers with different challenges which needs to be solved

first to request and successfully get the requested resource residing in the web

server. The user agent as discussed above has to parse the response in a way

to solve these challenges separated by commas and list of other auth-param

after it has identified the auth-scheme which includes the ‘token’.

So this goes like the following:

Auth-scheme = token {where token could be of a “string” or any value!}

Token = {parameters, “strings”}

Here the [Token] Authentication-parameter could be customized by the web

developer or the web framework itself in their entire own fashion, such as

“App_token = “thisisthetokenValUechallenge””

This ‘Token’ or the ‘Auth-scheme’ generally takes form of ‘nonstandard’ HTTP

header fields such as ‘X-auth-token’, etc… common nonstandard HTTP request

header fields could be the following:

X-Requested-With

DNT

X-Forwarded-For

X-Forwarded-Proto

Front-End-HTTPS

X-Wap-Profile

X-ATT-deviceid

Proxy-Connection

There are dozens of nonstandard HTTP response headers as well, we will look at

them later when we discuss about various framework which relies on HTTP

headers and use custom header fields, and sometimes this leads to information

disclosure keeping web application in mind. Some examples of custom

response HTTP headers are:

Content-Type

Content-Range

Etag

P3P

Pragma

The customized authentication we are specifically taking an example over

here is the ‘x-auth-token’ for Authentication token which has to be reused later

on each request being sent to the origin server.

That being said, different web frameworks might use different challenges with

different HTTP auth-param headers for solving its challenges. The WWW-

Authenticate however states if it’s a BASIC authentication level

challenge/DIGEST authentication level challenge or NTLM challenge which the

client must solve.

We will now come to the HTTP ‘realm’ part which the HTTP header field uses for

authentication. The HTTP authentication provides the ‘realm’ to the client for a

specific purpose. The reason and the purpose being the server stores a

protected ‘area’ where all the ‘realms’ are stored and they have their own

‘names’ to it. The names could be of any “string” type and hence enclosed in

double quotes. The realm however is case-sensitive in relation to the ‘realm

directive’ which is a ‘directory’ saved on the server side to which the client

wants access and is protected by a ‘realm value’. Here the realm directive

might be case-insensitive, but the realm value has to be case-sensitive

because there might be several other protected spaces for different resources

all placed on the same protected space or a different protected area, space,

page, group, etc…

The realm value might have similar semantics which relates to the

authentication scheme provided by the server. The origin server sets the realm

value. A realm looks like the following:

This was for the digest authentication scheme, however for a basic

authentication scheme, it would look same but with a WWW-authenticate:

Basic HTTP header in place.

References on realm directives are the following:

For HTTPD Apache: http://httpd.apache.org/docs/2.2/howto/auth.html

For SQUID: http://www.squid-cache.org/Doc/config/auth_param/

For Ngnix: http://wiki.nginx.org/HttpAuthBasicModule

Now, how does the user agent (the client) goes ahead and provide the

credentials to the origin server? The answer is it sends a special HTTP header

field called ‘Authorization’ not ‘authentication’ as presented by the origin

server with 401 status code. The ‘Authorization’ HTTP header is sent by the user

agent (the client) to a server only after it receives a 401 (Unauthorized) or 407

(Proxy-Authentication) response from a foreign remote origin server. It is to be

noted that the client does not necessarily have to send the ‘Authorization’ but

only if it wishes to, failing to which the server would endlessly (until timeout) send

the client a 401 status code with ‘Unauthorized’ in its response. The HTP header

would look like the following when the user agent sends the origin server with a

authorization HTTP header!

We can check the client has now added an ‘Authorization’ HTTP header along

with the other necessary HTTP headers for its authentication scheme and the

value for this HTTP attribute (which is Authorization) has been Base64 encoded.

This Base64 encoded value is basically the credentials we had provided in a

username:password form separated by a ‘:’ in place between them. The server

sends back the response as 401 yet again because the credentials I had

provided must be invalid and not applicable to the requested protected

resource under a ‘protected area’ based out of a ‘realm’ for that particular

resource being requested by the user agent (or the client!).

Now how does one move along to decode the Base64 encoding? The answer

is simply using an online Base64 encoder or using python or C libraries. A simple

program which is programmed in ‘C’ could be useful if a user has no access to

the internet and hence on practical scenarios could create his own C program

and compile to run the decode check to an encoded Base64 string:

#include <stdint.h>

#include <stdlib.h>

static char encoding_table[] = {'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H',

'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P',

'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X',

'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f',

'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n',

'o', 'p', 'q', 'r', 's', 't', 'u', 'v',

'w', 'x', 'y', 'z', '0', '1', '2', '3',

'4', '5', '6', '7', '8', '9', '+', '/'};

static char *decoding_table = NULL;

static int mod_table[] = {0, 2, 1};

char *base64_encode(const unsigned char *data,

size_t input_length,

size_t *output_length) {

*output_length = 4 * ((input_length + 2) / 3);

char *encoded_data = malloc(*output_length);

if (encoded_data == NULL) return NULL;

for (int i = 0, j = 0; i < input_length;) {

uint32_t octet_a = i < input_length ? (unsigned char)data[i++] : 0;

uint32_t octet_b = i < input_length ? (unsigned char)data[i++] : 0;

uint32_t octet_c = i < input_length ? (unsigned char)data[i++] : 0;

uint32_t triple = (octet_a << 0x10) + (octet_b << 0x08) + octet_c;

encoded_data[j++] = encoding_table[(triple >> 3 * 6) & 0x3F];

encoded_data[j++] = encoding_table[(triple >> 2 * 6) & 0x3F];

encoded_data[j++] = encoding_table[(triple >> 1 * 6) & 0x3F];

encoded_data[j++] = encoding_table[(triple >> 0 * 6) & 0x3F];

}

for (int i = 0; i < mod_table[input_length % 3]; i++)

encoded_data[*output_length - 1 - i] = '=';

return encoded_data;

}

unsigned char *base64_decode(const char *data,

size_t input_length,

size_t *output_length) {

if (decoding_table == NULL) build_decoding_table();

if (input_length % 4 != 0) return NULL;

*output_length = input_length / 4 * 3;

if (data[input_length - 1] == '=') (*output_length)--;

if (data[input_length - 2] == '=') (*output_length)--;

unsigned char *decoded_data = malloc(*output_length);

if (decoded_data == NULL) return NULL;

for (int i = 0, j = 0; i < input_length;) {

uint32_t sextet_a = data[i] == '=' ? 0 & i++ : decoding_table[data[i++]];

uint32_t sextet_b = data[i] == '=' ? 0 & i++ : decoding_table[data[i++]];

uint32_t sextet_c = data[i] == '=' ? 0 & i++ : decoding_table[data[i++]];

uint32_t sextet_d = data[i] == '=' ? 0 & i++ : decoding_table[data[i++]];

uint32_t triple = (sextet_a << 3 * 6)

+ (sextet_b << 2 * 6)

+ (sextet_c << 1 * 6)

+ (sextet_d << 0 * 6);

if (j < *output_length) decoded_data[j++] = (triple >> 2 * 8) & 0xFF;

if (j < *output_length) decoded_data[j++] = (triple >> 1 * 8) & 0xFF;

if (j < *output_length) decoded_data[j++] = (triple >> 0 * 8) & 0xFF;

}

return decoded_data;

}

void build_decoding_table() {

decoding_table = malloc(256);

for (int i = 0; i < 64; i++)

decoding_table[(unsigned char) encoding_table[i]] = i;

}

void base64_cleanup() {

free(decoding_table);

}

However using Python; the program becomes a lot simpler and has less

dependencies, an example of this python script could be:

#!/usr/bin/python

str = "this is string example....wow!!!";

str = str.encode('base64','strict');

print "Encoded String: " + str;

print "Decoded String: " + str.decode('base64','strict')

Or if on a bash environment, use the following to get the output!

root@coded:~# echo QWxhZGRpbjpvcGVuIHNlc2FtZQ== | base64 --decode

Where, QWxhZGRpbjpvcGVuIHNlc2FtZQ== is the base64 encoded string.

Now the great feature about using realm is that it uses ‘protection space’

which means if a particular user credential was authorized for this ‘realm’, the

same credentials would hold ‘valid’ for the ‘realm’ identically same as that of

the other realm, which means the resources are shared over the same ‘realm’

name and the browser (or the user agent or the client) could go ahead and

may not ask for credentials when trying to access the same realm. Now it is to

be kept in mind that these credentials might be valid only for certain amount of

time for that ‘protected space’ set by the authentication scheme in use, the

authentication parameters, and the user preferences. Unless defined by the

authentication scheme itself, the scope is only limited to that particular

‘protected space’ and does not extend beyond this scope.

Now, if the origin server does not accept the credentials sent by the user agent,

it must send 401 (Unauthorized) with WWW-Authenticate header back to the

client (or the user agent). The same applies to the Proxy-Authenticate header, if

the proxy does not accept the credentials sent by the user agent, the proxy-

server must send a 407 (Proxy-Authenticate) header back to the user agent.

This happens with providing newer authentication schemes to the user agent

on each response and is liable to both proxy-server and the origin server.

Hence overall the basic authentication scheme is all about the user ID and the

password being sent to the origin server or a proxy-server along with base64

encoded credentials separated by a colon. The left part of the colon could

represent the user ID and the right side of the colon could represent the

password. This being said, the whole credential would be only valid for a

particular realm to which the realm value is set by the origin server or the proxy-

server itself and keeps the resource at a ‘protected space’. This protected

space has its own realm to which the client needs an access to! The URI is sent

by the client first, to which the server sends a 401 Unauthorized with WWW-

Authenticate with mentioned authentication scheme back with its realm for

the protected space, the client picks up these headers, and sends the

credentials over the HTTP Authorization HTTP header field encoded with base64

encoding along with Connection header if the connection is to keep ‘alive’ or

close at authentication. The ‘Connection’ is generally kept ‘alive’. The same

happens with the proxy-server, instead the proxy-server sends Proxy-

Authenticate HTTP header and accepts Proxy-Authorization as a request to

access the protected resource (along with the Referer if present!)

When the Authorization header is sent to the origin server, it also states what

authentication scheme it uses, for instance it would say “Authorization: Basic

Y29kZWQzMjppYW10ZXN0Cg== “

The ‘basic’ states the client had chosen the basic authentication. The string

followed by the ‘basic’ is the username:password base64 encoded value. To

get the base64 value, one could use the echo command in a Linux machine (I

am using Kali Linux) and quote out the string to be encoded with a ‘pipe’ to

filter it out to the base64 command. The returned string is the base64 encoded

equivalent to the username:password string.

The header should look like the above. Now, the rest is it should be

remembered by the user agent that the credentials it just provided to the origin

server with the relative URI should be valid for the directories falling under that

URI. This means the client should assume that the request-URI the client had

provided, the links deeper than the depth of the same URI falls under the same

‘protection space’ and hence belongs to the same ‘realm’ and therefore the

same credentials could be valid under the directory previously specified by the

request-URI. If the file being requested ends at that particular URI and is

mentioned in the URI, it’s then absolute URI path which the client requested

and hence it was the last resource which could be fetched via the credentials

provided by the user agent (or the client).

Digest Access HTTP Authentication: The Digest access authentication is a need

over basic HTTP authentication due to security concerns. Because the basic

authentication provides no encryption (only encoding to the plain text), this

could fall under MITM attacks. MITM is short for Man in the Middle attack which

would be discussed in other coming documents. For now we shall stick to what

is digest access authentication and how does it work. The digest access

authentication itself provides no encryption but was originally meant to cover

up the flaws of the basic authentication over HTTP connections.

The digest access authentication is same as the basic access authentication

with minor changes in the way the username and the password is verified by

the origin server. The digest access authentication uses the same scheme as

the basic access authentication which is challenge-response paradigm. The

origin server provides a challenge with a ‘nonce’ header to the user agent or

the client which asks for the protected resource. This ‘nonce’ header is

detected and picked up by the client (or the user agent) and sends the

credentials in hashed format (which is generally MD5 hash). The origin server

picks up the hash values, matches it up and verifies if the same hash is present

in the database of the server or not, if the hashed credentials been served by

the user agent (or the client) is valid, the credentials would be accepted or

otherwise not authorized to access the resource asked for. But this happens in a

deeper way. The hash is being send as a hash checksum of the username and

the password. The challenge-response consists of these checksums and the

response to this could be the MD5 checksum (generally unless specified by the

server itself when first contact was made by the user agent or the client!) of the

username, the password, the URI the client or the user agent sent, the ‘nonce’

and the HTTP ‘method’ via which the resource is requested over. Now the HTTP

‘method’ is something we are yet to discuss about. There are various methods

supported such as GET, PUT, DELETE, TRACE, OPTIONS, and POST etc...

If at the first client request the serer does not find any credentials, the sever

generates a 401 Authenticate status code and sends this to the client with

HTML Etag headers (for different supported web application) as to where the

URI requested could be found and the ciphers the server supports and lets the

client choose the appropriate cipher the client finds comfortable with.

Everything happens same as the Basic Access Authentication, just that the

nonce header is added and the checksums are sent by the client instead of

encoding the plain text credentials. The verification is also two way as the

server verifies the stored checksums and sends matches with the one client

sent! Etag headers are the non-standard headers which different web

application generates for different purposes, these purposes could serve

information about the server itself, or as a hint of the ciphers it supports, or

maybe any resource which has to be followed before the authentication takes

place or for any different reason as suited by the application framework the

web application itself was built upon.

Now how this prevents replay attacks which were caused by the basic access

authentication? The answer is the origin server responds with a different

‘nonce’ value to which then the client according calculates the MD5 hash

checksums and thereby generating a different credential digest every time it

sends this request to the server. The general calculation via which the client

calculates the MD5 checksum is as follows:

md5(md5(username:realm:password):nonce:md5(httpMethod:uri))

This again could be implemented over python or suitable language of choice

to verify the integrity of the different requests sent to the origin server when

trying to access the protected resource. No request is the same as the previous

one because the ‘nonce’ header sent by the origin server mitigates the issues

related with the basic access authentication scheme.

References: http://en.m.wikipedia.org/wiki/Cryptographic_nonce

The whole operation over a digest access authentication in-between a user

agent or a client and the origin server or a proxy server would go as follows. On

original request, if the user agent or the client does not send any credentials

(however the server would drop the credentials because the nonce will be

computed only when the origin server sends the nonce value), the response

from the server’s end would look like:

Upon, getting the response, the client will take a note of the WWW-

Authenticate header, grab the realm, will take a note of the URI, take a note of

the request method, will see the algorithm used to compute the challenge

provided by the server, would grab the nonce value which is supposed to be

generated by the origin server each time on each request (no duplicates

happen here!), and then altogether will computer the MD5 checksum of the

username:realm:password and then combine this with the :nonce: value and

then merge the MD5 of HTTP_method:URI; and hence the algorithm derived

would be:

md5(md5(username:realm:password):nonce:md5(httpMethod:uri))

Which will be sent to the origin server to access the protected resource. The

origin server would accept if the credentials provided were valid and this way

the credentials are never stored in the server itself, only the MD5 checksums of

the credentials are stored which is requested and accessed each time via the

combination of the credentials themselves along with the HTTP URI, the HTTP

METHOD and the realm in-between the credentials.

References: http://technet.microsoft.com/en-

us/library/cc780170%28v=ws.10%29.aspx