Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
ServersServers
Servers allow web sites to do things, not just displaythings
choices test server URLs routes
1
ChoicesChoicesFor a dynamic web site, you have to choose a webserver, with a framework and a scripting ordevelopment language for writing the pages:
CGI Perl or C (obsolete)
ASPVBScript orC#
(proprietary/1-platform/obsolete)
JSP Java (clunky)LAMP PHPDjango PythonRails RubyNode JavaScript
2
Two choicesTwo choicesFor this course, two choices are recommended, and willbe covered in this chapter:
RawRaw Node, with standard libraries only
ExpressExpress Node, with express and/or other libraries
Take the raw approach to learn more, take the expressapproach to get further
3
Development serverDevelopment serverIf you are using Node (raw or express), the developmentserver server.js provided at the start of the coursecontains some potentially useful details
But remember that it was designed for development andnot production:
It only allows connections from the same computer, andits security is otherwise weak and, in particular it has aserious security vulnerability
4
Preliminary securityPreliminary securitySecurity is essential for a potentially public-accessservice - we will come back to it several times
Step one is to have a dedicated computer for a web site,and put it somewhere physically secure, e.g. a lockedmachine room
Step two is to restrict login to that computer to alimited number of other computers or people or both
Step three is to establish a firewall, especially for anintranet
5
ExperimentsExperimentsIn this chapter, we will concentrate on the rawapproach, in order to work out what the various issuesare, and summarise the other approaches
The first thing to do is to find out what kinds of requestthere are, and what information they contain
So, let's set up a minimal Node server which prints outrequest details - it may be helpful to keep it around forfuture use as well
6
DocumentationDocumentationThe documentation is in the Node docs for the httpand later the https modules
The documentation is very difficult to make sense ofbecause (a) it includes very low level things we don'tneed (b) it mixes client and server info (c) it isn't explicitabout types (d) it is poorly explained (e) it assumes thecallback style
Two important pieces of information are that a requesthas type IncomingMessage and a response has typeServerResponse
7
server1.jsserver1.js
Minimal serverMinimal server// Minimal server// Minimal server letlet HTTP = HTTP = requirerequire(('http''http');); start(start(80808080);); ......
Download server1.js to experiment
The 'http' module comes with Node, and so does 'https'
Use 8080, to add firewall security, to avoidroot/administrator privileges, or if port 80 is being usedby another program
8
server1.jsserver1.js
Starting the serverStarting the server...... // Provide a service to localhost only.// Provide a service to localhost only. functionfunction startstart((portport) ) {{ letlet service = HTTP.createServer(handle); service = HTTP.createServer(handle); service.listen(port, service.listen(port, 'localhost''localhost');); }} ......
The server is now listening for browser requests, butonly from the computer it is running on (do this whiledeveloping, until you are happy with your security)
Every request will cause a call to the handle function
9
server1.jsserver1.js
Handling requestsHandling requests...... // Deal with a request.// Deal with a request. functionfunction handlehandle((request, responserequest, response) ) {{ consoleconsole.log(.log("Method:""Method:", request.method);, request.method); consoleconsole.log(.log("URL:""URL:", request.url);, request.url); consoleconsole.log(.log("Headers:""Headers:", request.headers);, request.headers); reply(response); reply(response); }}
Print details - the most important fields of the request
Look up http.IncomingMessage in the Node APIdocs for the 'http' module to get request details
10
server1.jsserver1.js
RespondingResponding...... // Send a reply.// Send a reply. functionfunction replyreply((responseresponse) ) {{ letlet hdrs = { hdrs = { 'Content-Type''Content-Type': : 'text/plain''text/plain' }; }; response.writeHead( response.writeHead(200200, hdrs); , hdrs); // 200 = OK// 200 = OK response.write( response.write("OK""OK");); response.end(); response.end(); }}
Send a minimal response (as if delivering a plain textfile with one line containing OK)
Look up http.ServerResponse in the Node APIdocs for the 'http' module to get response details
11
TestingTestingWhen you visit http://localhost:8080/ withyour browser, the server will describe the request
You are quite likely to see two requests, one for URL /which you asked for, and one for URL /favicon.icowhich is the browser asking for the site icon
The test server responds to every request the same way
12
server2.jsserver2.js
Delivering pagesDelivering pagesletlet HTTP = HTTP = requirerequire(('http''http');); letlet FS = FS = requirerequire(('fs''fs').promises;).promises; letlet OK = OK = 200200, NotFound = , NotFound = 404404, BadType = , BadType = 415415;; start(start(80808080););
The next server server2.js delivers HTML pages
Names are defined for success and failure codes
start can't be called until global data has been defined(some people call it at the bottom of the script)
13
Handling page requestsHandling page requests
Add index.html if a URL ends with /, and give anerror if it is not .html, return after calling fail tostop further processing, convert the url into a file path
Read the file and use the content to reply
server2.jsserver2.jsasyncasync functionfunction handlehandle((request, responserequest, response) ) {{ letlet url = request.url; url = request.url; ifif (url.endsWith( (url.endsWith("/""/")) url = url + )) url = url + "index.html""index.html";; ifif (! url.endsWith( (! url.endsWith(".html"".html")))) returnreturn fail(response, BadType, fail(response, BadType, "Not .html""Not .html");); letlet file = file = "./public""./public" + url; + url; letlet content; content; trytry { content = { content = awaitawait FS.readFile(file); } FS.readFile(file); } catchcatch (err) { (err) { returnreturn fail(response, NotFound, fail(response, NotFound, "Not found""Not found")) reply(response, content); reply(response, content); }}
14
ReplyingReplying
the content comes from readFile
Deliver file content as application/xhtml+xml(can only handle web pages)
server2.jsserver2.jsfunctionfunction replyreply((response, contentresponse, content) ) {{ letlet hdrs = { hdrs = { 'Content-Type''Content-Type': : 'application/xhtml+xml'application/xhtml+xml response.writeHead(OK, hdrs); response.writeHead(OK, hdrs); response.write(content); response.write(content); response.end(); response.end(); }}
15
ErrorsErrors
In the case of an error, deliver a one-line plain textresponse, with a given error code
server2.jsserver2.jsfunctionfunction failfail((response, code, messageresponse, code, message) ) {{ letlet hdrs = { hdrs = { 'Content-Type''Content-Type': : 'text/plain''text/plain' }; }; response.writeHead(code, hdrs); response.writeHead(code, hdrs); response.write(message); response.write(message); response.end(); response.end(); }}
16
URL ValidationURL ValidationSecurity inside the server is a difficult topic; the firstline of defence is to validate the URL in each request
Warning:Warning: the original server.js provided doesn't dofull URL validation
Warning:Warning: expressexpress doesn't do full URL validation
Warning:Warning: other servers/frameworks don't do full URLvalidation
Security is youryour responsibility
17
File systemsFile systemsThe main problem is that servers convert the URL pathinto a file path by prefixing the root folder
Then they get the file system to find the resource
That means there are security loopholes, depending onodd features of the file system
18
Dot dotDot dotAn obvious attack is to use .. in URLs to try to reachfiles outside the root folder
Suppose you have a root folder /site/public and asensitive file /site/secret.txt
An attack might be to visit/products/../../secret.txt
This must be prevented and it is stronglystrongly recommendedthat you test it (express and other servers do preventit, if you use them properly)
19
TestingTestingTesting can be difficult, e.g. if you type a URL directlyinto your browser, and then get the server to print outthe URL, you may find that the browser has changed theURL before sending the request (especially /../)
The same happens with tools like wget or curl
For total control, write a program such as get.js
19a
SynonymsSynonymsSuppose you have an admin folder insideinside your site
And you have password protection of the form "if a URL starts with "if a URL starts with /admin//admin/ then require a password" then require a password"
Attacks may use synonyms to bypass the protection /products/../admin/adduser /./admin/adduser //admin/adduser
Express and other servers (and browsers) handle /../and /./ but notnot // so test this!!!
20
Restricting URLsRestricting URLsThe simplest approach to solve these problems is tovalidate the URL
Simplest is to reject a URL if it contains /. or //, orcontains non-ascii characters, or doesn't start with /, oris too long, see dotdot, hidden, spaces
Safer is to check that the URL matches one of the stringsor patterns that you regard as OK, and reverse yoursecurity checks
21
DotdotDotdotIf you reject URLs containing /., then you are rejecting/./, which is good, but also /../ which is used quitecommonly
But that's probably OK, because /../ is usuallyresolved by the browser before the request gets to theserver
If you prefer, you can canonicalise the URL, and thenreject it if it starts with .. which indicates that it istrying to climb out of the public folder, and thenapply security restrictions to the canonical URL
21a
Hidden filesHidden filesIf you reject URLs containing /., then you are rejectinghidden files /.hidden, which may be a good thing
On Linux and MacOs, any file or directory whose namestarts with a dot is automatically hidden
On Windows, a flag is used for hidden files, and namesstarting with a dot have no special meaning, but manytools such as git create files or folders that start withdot andand are marked as hidden
So probablyprobably a dot file is hidden, even on Windows
21b
SpacesSpacesSpaces are officially only allowed in URLs if they areescaped as %20
If you are going to allow them, you need to work out (a)how different browsers handle them (b) what gets sentto your server (c) how to handle them safely in yourserver on your file system (d) what effect, probablyconfusion, they will have on your users
It is almost always simpler and better to ban them
21c
Need to knowNeed to knowSecurity can be improved by reversing it (on the 'needto know' principle)
Suppose your site has just two folders, products andadmin
Instead of "if a URL starts with "if a URL starts with /admin//admin/ then require a then require apassword"password" do "if a URL doesn't start with "if a URL doesn't start with /products//products/then require a password"then require a password"
That would catch more of the problems without havingto think too hard
21d
CaseCaseAttackers might try /ADMIN/adduser to get pastyour password protection on /admin/...
Usually Linux servers are case sensitive and Windowsservers are case insensitive (the filestore decides!)
I recommend making all requests case sensitive, as inthe provided server, and using a whitelist or blacklistfor exceptions
22
AccessAccessAnother thing that needs to be done is to preventvisitors from accessing files which are not intended tobe accessible
That means e.g. being careful not to put the serverprogram, encryption keys, database files, config files,source code, editor backups, hidden files, and so oninside the public root folder
It is also a bad idea to have hard links, symbolic links,shortcuts, or anything equivalent, inside a root folder
23
jsjs
Content typesContent typesEach successful response needs a content type: the normis to maintain a table in the server code which maps fileextensions to content types (or use the mime module)
letlet types = { types = { csscss : : 'text/css''text/css',, jsjs : : 'application/javascript''application/javascript',, ... ... }} ...... let type = types[extension];let type = types[extension]; letlet hdrs = { hdrs = { 'Content-Type''Content-Type': type };: type }; response.writeHead(response.writeHead(200200, hdrs);, hdrs); ......
24
ExtensionsExtensionsWhat happens with a URL like /project/Makefilewhich doesn't have an extension?
The server doesn't know, just from the URL, whether itis a file with no extension or a folder with no /
If it is a file, the server doesn't know what type to use,so the default is often to treat it as a folder
If it is folder, you mustmust redirect the browser, so it issimpler to reject such URLs (maybe with exceptions)
25
RedirectionRedirectionWhat happens if you treat /x/y as /x/y/ and deliverthe file /x/y/index.html?
If there is a relative link on that index page, the browserthinks it is relative to /x/ and not /x/y/ so the linkfails
So you must redirect the browser, by sending aresponse with the corrected URL (using code 301, 302,303, 307, 308 - good luck working out which is correct)
I think it is better to avoid the inefficiency of multiplerequests by banning the original URL
25a
jsjs
Content negotiationContent negotiationFor XHTML, content negotiation is needed to serve an.html page to old and new browsers
// For the .html extension:// For the .html extension: letlet otype = otype = "text/html""text/html";; letlet ntype = ntype = "application/xhtml+xml""application/xhtml+xml";; letlet header = request.headers.accept; header = request.headers.accept; letlet accepts = header.split( accepts = header.split(","",");); ifif (accepts.indexOf(ntype) >= (accepts.indexOf(ntype) >= 00) type = ntype;) type = ntype; elseelse type = otype; type = otype;
Warning:Warning: express and other servers deliver all HTMLpages as "text/html" by default
26
Dynamic URLsDynamic URLsFor dynamic resources, e.g. pages generated fromdatabase data, it is up to you to invent URL conventions
As well as dynamically generated HTML pages, dynamicURLs may present requests to the server to update thedatabase, or deal with an upload - you could call theseaction URLs
There are lots of possibilities, including using express
27
ExpressExpressIn an express server, I personally find it extremelydifficult to understandunderstand what happens to a particularrequest, or to debugdebug what is happening to a particularrequest, or work out how to do 'unusual' things such ascontent negotiation
On the other hand, express has a very good 'mix-and-match' style, with a wide range of off-the-shelf tools
Maybe the best approach is to use those tools directly,without using express's obscure routing
27a
URL RewritingURL RewritingURL rewriting allows you to make logical and intuitiveURLs available to users
Suppose you publish a URL /pets/dogs/odie.html
When it arrives in a server request, you can change it to/pets/dogs.html?name=odie so you can use atemplate for all dogs, filled in with data about aparticular dog
Or rewrite to /pets.html?type=dog&name=odieso you can use a single template for all pets
28
RoutesRoutesA route is a rule for sending a dynamic URL to the righthandler function for dealing with the request
The handler function may use a template and a databaseto construct a web page to be sent back to the browser
OROR the handler may carry out some update to thedatabase, or deal with an upload, and send aconfirmation message back to the browser
Routes are most often based on URL prefixes
29
PrefixesPrefixesIn the prefix case, you recognise any URL which startswith a fixed set of handler prefixes, e.g /set/x/42 or/set?x=42 need to be handled by a function set
The server contains code to recognize the action URLsand call the right handler function, passing it therequest, response, etc.
The advantage is simplicity when there are fewhandlers, and the danger is a poor, non-modular,monolithic design, i.e. all in a one-piece program
30
ModularityModularityA good way to make routing more modular in a DIYserver is for dynamic or action URLs to be of the form/do/set?x=42
The /do/ prefix is used to recognize the URL as anaction URL
The next part set is used as the name of a moduleset.js in your server module folder (notnot insidepublic!)
You pass it the parameters and it handles the request,see table of handlers
31
jsjs
jsjs
Table of handlersTable of handlersYou can maintain a table of handlers
letlet handlers = ...; handlers = ...; letlet f = handlers[name]; f = handlers[name]; f(request, response);f(request, response);
If you are using modules, you can load a module if thehandler hasn't been used before (using node's old non-standard module system)
letlet m = m = requirerequire(("./do/""./do/" + name + + name + ".js"".js");); letlet f = m.handle; f = m.handle; handlers[name] = f;handlers[name] = f;
31a
MethodsMethodsThe method GET represents a normal request for a pageor file - it has no body
The method POST sends data from a form - it has abody with the parameters and their values
The method PUT is very useful for simple uploads - youcan arrange for the filename to be in the URL, and thebody to be the file contents (the alternative, POST withtype multipart/form-data, is complex andinefficient)
32
Going furtherGoing furtherTo avoid nasty callback issues and allow theasync/await keywords to be used, a promise versionof the http library is needed
At present, there isn't one built into node, so try this:
npm install q-io -savenpm install q-io -save
It provides require("q-io/fs") equivalent torequire("fs").promises, and require("q-io/http") which is like http withpromises
33
Reading the bodyReading the body
Like server 1, this responds OK to every request
The body is not instantly available - it has to be read in
server1b.jsserver1b.jsasyncasync functionfunction handlehandle((request, responserequest, response) ) {{ consoleconsole.log(.log("Method:""Method:", request.method);, request.method); consoleconsole.log(.log("URL:""URL:", request.url);, request.url); consoleconsole.log(.log("Headers:""Headers:", request.headers);, request.headers); letlet body = body = awaitawait request.body.read(); request.body.read(); consoleconsole.log(.log("Body:""Body:", body.toString());, body.toString()); response.status = response.status = 200200;; response.headers = { response.headers = {'Content-Type''Content-Type': : 'text/plain''text/plain' response.body = [ response.body = ["OK""OK"];]; returnreturn response; response; }}
34
Test pageTest pageTo test, create a minimal form page
Double click to view with browser (with a file: URL,not via the server) then press Send
htmlhtml<!DOCTYPE html><!DOCTYPE html> <<htmlhtml xmlnsxmlns=="http://www.w3.org/1999/xhtml""http://www.w3.org/1999/xhtml">> <<headhead>><<titletitle>>TestTest</</titletitle>></</headhead>> <<bodybody>> <<formform methodmethod=="post""post" actionaction=="http://localhost:8080/"http://localhost:8080/ <<inputinput namename=="pet""pet" valuevalue=="dog""dog"/>/> <<inputinput namename=="car""car" valuevalue=="bmw""bmw"/>/> <<inputinput typetype=="submit""submit" valuevalue=="Send""Send"/>/> </</formform>> </</bodybody>> </</htmlhtml>>
35
QuerystringQuerystringThe body of the request posted from a form contains: pet=dog&car=bmw
This can be unpacked with the querystring module
letlet QS = QS = requirerequire(("querystring""querystring");); ...... let params = QS.parse(body);let params = QS.parse(body); consoleconsole.log(params.pet, params.car);.log(params.pet, params.car);
36
MiddlewareMiddlewareIn the express world, middleware refers to a piece ofsoftware which sits between the main handle functionand the individual handlers which do the real work -they adapt the request in some way, e.g. checkpasswords and permissions
A middleware function is given the request andresponse by handle, chooses a handler, and passes itthe possibly modified request and response
Middleware functions are also called filters
37
ChoicesChoicesRoughly speaking, your choices are:
write your own server, using callbackswrite your own server, using async/await(maybe using flows)use express, with callbacks (as tutorials tell you todo)use express, with async/await (much better,but harder to find out how)
38
TemplatesTemplatesA template is a web page, reused with different data
Some convention is used to indicate points in thetemplate where data is inserted
If you are taking the raw approach to building a server,you could invent your own conventions, but you maywant to use some ready-made template system
Express allows you to choose between template engines(ejs, pug, haml, handlebars, combyne, swig, blade, ...)
39
Password protectionPassword protectionPasswords must not be sent over the network withoutencryption
Using http, passwords are sent using base64 encoding
This is mostly to turn special characters into ascii, andto prevent accidental prying, but it is not encryptionit is not encryption
To encrypt properly, you mustmust switch to https
encryption protocols, saving passwords, HTTPS sites, HTTPS forwarding
40
Encryption protocolsEncryption protocolsFor serious https work, you need to configure yourserver to avoid encryption algorithms or protocolswhich are now thought to be weak
Look for a web site (e.g. ssllabs) which will carry out an"SSL server test" on the encryption facilities offered byyour site
40a
Saving passwordsSaving passwordsHow are passwords saved in databases?
Usually, they are 'encrypted' first with MD5
Note (a) MD5 is really a digest (hash) mechanism and not anencryption (b) it is now regarded as weak and something likeSHA256 or bcrypt is recommended instead (c) it is only secondary security - the main security isnot letting anyone access the database in the first place
40b
Https sitesHttps sitesIt used to be common for a site to provide both HTTPand HTTPS (technically two servers, but running in thesame program)
It is becoming common to provide only HTTPS
HTTP/2 based on SPDY, and Let's Encrypt, are likely toincrease this trend
40c
Https forwardingHttps forwardingSuppose your server provides both an http service andan https service for the same site
And suppose that some areas of the site are protectedand some are not
If an http request is made for a protected area, it isimportant that the request should be redirected tohttps
That means sending a response to the browser, asking itto try again on the https URL
40d
CertificatesCertificatesTo use https, you need to provide a private key (from apublic/private pair) and a certificate
For this unit, a self-signed certificate is fine (though abrowser will ask the user to confirm trust), or if youhave a domain you can use Let's Encrypt
These can be loaded from files before the server starts(or even embedded in the code) but must be kept secret
41
Form submissionForm submissionSuppose your site has a form page for a privileged userto fill in
The form data will be sent to an action (specified in theaction attribute), normally a dynamic URL
The form page should be protected, so the user can seea comforting protection icon
But the actionaction URL really mustmust be protected, because itis the one that carries sensitive data and/or asks theserver to do updates
42
Input ValidationInput ValidationData validation can optionally be done on a form page(by HTML5 or script) for quick response
But validation mustmust also be done on the server side, onthe action URL that receives the data, for security anddata consistency
For good security, your attitude should not just be"what damaging data could a hacker send" but "whatrange of data is safe"
It is better to start very restrictive and expand than theother way round
43
Output ValidationOutput ValidationSuppose a user can enter text which ends up displayedon a web page (e.g. in a blog post or even in an errormessage)
Care needs to be taken that it can't do any harm byhaving any special HTML/CSS/JS meaning
The main danger is XSS (cross-site scripting) attacks
The issues are subtle, so it is best to use a security-aware library feature
44
SessionsSessionsA session is needed to track multiple requests comingfrom the same user, and keep some state data
This is particularly necessary for protected pages, just toremember that the user has already logged in andneedn't login again
It is possible to use OAuth to delegate logins
But otherwise, how do sessions work?
45
Shared secretsShared secretsIt is not enough to use ip addresses and port numbers,partly because they can be forged, and partly becausethey are not always specific to users and their browsers
It is normally done by having a shared secret, i.e. anunguessable random number held by both the browserand server
Since nobody must see it, https must be used, even ifpassword login isn't used
46
Generating a shared secretGenerating a shared secretIt is the server which must generate the shared secreton a user's first visit
A username or fixed id number or pseudo-randomnumber is notnot good enough
A time-of-day random number is very often used, whichis OK but not excellent
An id signed with an encryption key is good, and so is acrypto-random number (usually entropy-based)
47
CookiesCookiesA cookie is a name/value pair which the server sets in aresponse header
After that, the browser will send the same cookie in arequest header every time it visits the same server
This is ideal for a shared secret and is then called asession cookie and it lasts as long as you choose
There is no need to store any data in the browser otherthan a session id - everything else can be in the server
(The main exception is using browser localStorage whenthere is no configurable server)
48