DevDays2008 - Qt Network AccessFlexible and Powerful Access to Data on the Internet

1

Qt Network Access:Flexible and Powerful Access to Data on the Internet

Thiago Macieira

Qt Developer Days 2008

2

Who am I?

•Senior Software Engineer at Qt Software / Nokia

•Degrees in Engineering and an MBA

•Almost 2 years with the company

•Part of the Core Team• Work mostly with Networking, I/O, IPC, Threading, Tool Classes

•Other official duties:• Liaison to the KDE Community

• Release Manager for Qt

3

Agenda

What’s coming in Qt 4.5What’s coming in Qt 4.5

The API in useThe API in use

The Qt Network Access APIThe Qt Network Access API

Background and rationaleBackground and rationale

4

Agenda

What’s coming in Qt 4.5

The API in use

The Qt Network Access API

Background and rationaleBackground and rationale

5

Data on the Internet

•The Internet is full of data• Some good, some bad, but that's not the point

•Intranets and Extranets too• Not just the World Wide Web

•Applications often need to access data

6

Data on the Internet

•The Internet is full of data• Some good, some bad, but that's not the point

•Intranets and Extranets too• Not just the World Wide Web

•Applications often need to access data• Download updates or optional features

• Upload results

• Obtain reports, weather forecast

7

Before Qt Network Access...

•There was QHttp and QFtp

•Low-level access to HTTP and FTP communication• Ability to send any command

• Full access to the response (non-standard replies)

•In Qt 3.x, there was QUrlOperator

8

But QHttp and QFtp could be better

•Operations are too low-level• Developer must know how HTTP and FTP work

• Easy to make mistakes

•Fragile• Easy to break

•Almost no parsing of replies• You must implement the rest

9

Sample QHttp usage

•Things to remember:• Read everything at the requestFinished signal

• Match the contents with the “id” you saved

QHttp http;http.setHost(“trolltech.com”);int id1 = http.get(“/”);int id2 = http.get(“/qtdevdays2008”);connect(&http, SIGNAL(requestFinished(int,bool)), ...);

10

And QHttp could support more of HTTP

•The HTTP protocol is full of features

•QHttp supports a small subset only

•Improving support was on the plans• It would require a full refactoring

11

Then QtWebKit came along...

•WebKit required an abstraction layer• Early code worked with QHttp

• No support for FTP

• Separate code for dealing with files

•WebKit comes from KDE’s KHTML• Originally, KHTML used KIO

12

We had no choice, did we?

•Requirements were:• Easy to use, but possible to extend

• HTTP/1.1, FTP, local files, “data:”

• Abstract the differences away

•WebKit also required some more features• Cookies

• Proxy handling

• Authentication cache

• Simultaneous requests

13

Agenda


The API in use

The Qt Network Access APIThe Qt Network Access API

Background and rationale

14

Enter Qt Network Access

•Three main classes

QNetworkAccessManager

QNetworkRequest QNetworkReply

15

QNetworkRequest

•Represents one request to be sent

•Main component: one URL

•Meta-data to the request:• HTTP headers

• SSL options

• Settings controlling behaviour

16

QNetworkReply

•Read-only sequential-access QIODevice

•Data of the reply

•Meta-data:• URL

• Request associated

• HTTP headers and status

• Encryption status

• Error conditions

17

QNetworkReply (cont.)

•Already open, but not finished yet

•Emits readyRead() when data arrives

•Emits finished() when done

•Emits error() if problem occurs

•Progress signals: downloadProgress() and uploadProgress()

18

QNetworkAccessManager

•Operations supported:• HEAD – obtain status

• GET – download

• PUT – upload

• POST – HTTP-only

•For each operation, there’s one method• QNetworkRequest as input

• Data to be uploaded (if appropriate)

• QNetworkReply as output

19

QNetworkAccessManager (cont.)

•Manages request queue

•Centralises settings• Cookie Jar

• Proxy

• More in 4.5

•Caches previous authentication successes

•Keeps connections open for some time

20

Cookies

•Documentation defines as:• “Cookies are small bits of information that stateless protocols like HTTP use

to maintain some persistent information across requests.”

•One class: QNetworkCookie

•Does the parsing for you

•Recreates the raw form

21

Getting and setting cookies

•Set cookies in a QNetworkRequest

•Retrieve them from a QNetworkReplyQNetworkCookie cookie(“key”, “value 42”);qDebug() << cookie; // prints: key=“value 42”

QNetworkRequest request;request.setHeader(QNetworkRequest::SetCookieHeader, cookie);

QList<QNetworkCookie> cookies = reply->header(QNetworkRequest::CookieHeader) .value<QList<QNetworkCookie> >();

22

Where do you keep your cookies?

•In the cookie jar!

•QNetworkCookieJar

•Basic security only

•Two virtual functions:• cookiesForUrl

• setCookiesFromUrl

23

Extending Qt Network Access – Why?

•Handling special requests

•Using a different backend

•Applying policies/filtering to requests allowed

•Rewriting URLs

•Changing default behaviour

24

How do I extend it?

•QNetworkAccessManager has one virtual function:• createRequest()

• All requests are queued and sent by this function

•QNetworkReply is abstract• Implement pure virtuals

• Emit readyRead() and other signals

25

Agenda


The API in useThe API in use



26

How easy is it?

•Simplest ways:

QNetworkAccessManager manager;QNetworkReply *reply = manager.get(QUrl(“http://trolltech.com”));connect(reply, SIGNAL(finished()), ...);

QNetworkAccessManager manager;connect(&manager, SIGNAL(finished(QNetworkReply*)), ...);manager.get(QUrl(“http://trolltech.com”));manager.get(QUrl(“http://trolltech.com/qtdevdays2008”));

http://trolltech.com/


27

Old versus new

•Code does the same operation

QHttp http;http.setHost(“trolltech.com”);int id1 = http.get(“/”);int id2 = http.get(“/qtdevdays2008”);connect(&http, SIGNAL(requestFinished(int,bool)), ...);

QNetworkAccessManager manager;connect(&manager, SIGNAL(finished(QNetworkReply*)), ...);manager.get(QUrl(“http://trolltech.com”));manager.get(QUrl(“http://trolltech.com/qtdevdays2008”));


28

Full example – Downloader classclass Download: public QObject{ Q_OBJECT QNetworkAccessManager manager;public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); }

void download(const QUrl &url) { manager.get(url); }

private slots: void finished(QNetworkReply *reply) { reply->deleteLater(); emit done(reply->url(), reply->readAll()); }

signals: void done(const QUrl &url, const QByteArray &data);};

29





QNetworkAccessManager manager;public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); }

30






signals: void done(const QUrl &url, const QByteArray &data);

31






32

Full example – Downloader class

class Download: public QObject{ Q_OBJECT QNetworkAccessManager manager;public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); }




33

Downloader class – improving it

•Handle authentication challenges

•Handle error conditions

•Handle SSL issues

•Allow access to the meta-data

•Use a shared QNetworkAccessManager

34

QNetworkAccessManager manager;public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); }

Downloader class – authentication challenges

•Just connect to the extra signals

QNetworkAccessManager manager;public: Download() { connect(&manager, SIGNAL(finished(QNetworkReply*)), SLOT(finished(QNetworkReply*))); connect(&manager, SIGNAL(authenticationRequired(QNetworkReply*,QAuthenticator*)), ...); connect(&manager, SIGNAL(proxyAuthenticationRequired(QNetworkProxy,QAuthenticator*)), ...); }

Downloader class – authentication challenges

•Just connect to the extra signals

private slots: void finished(QNetworkReply *reply) { reply->deleteLater(); if (reply->error() != QNetworkReply::NoError) { // do something about this error } else emit done(reply->url(), reply->readAll()); }

Downloader class – error handling (alternative 1)

•The “finished” signal is still emitted

•The error code is saved in the reply

37

Downloader class – error handling (alternative 2)

•Or you can use the “error” signal

•That one is per reply

void download(const QUrl &url) { QNetworkReply *reply = manager.get(url); connect(reply, SIGNAL(error(QNetworkReply::NetworkError)), ...); }

38

Synchronous downloading

•No “waitFor” functions – you must use the event loop

QByteArray synchronousDownload(const QNetworkRequest &request){ QNetworkReply *reply = manager.get(request);

QEventLoop loop; connect(reply, SIGNAL(finished()), &loop, SLOT(quit())); loop.exec();

return reply->readAll();}

39

Agenda

What’s coming in Qt 4.5What’s coming in Qt 4.5

The API in use



40

What’s coming in Qt 4.5?

•Page caching

•Proxy factory

41

Data caching

•Problematic: • Network transfers are costly

• Therefore, we should avoid re-transferring

•New abstract base class: QAbstractNetworkCache

•Integrated with QNetworkAccessManager • setCache(QAbstractNetworkCache *)

• Automatically uses the cache

42

Cache access policies

•Four policies to choose from:• AlwaysNetwork

• PreferNetwork

• PreferCache

• AlwaysCache

•In a web browser• Default: PreferNetwork

• Reload button: AlwaysNetwork

• Offline browsing: AlwaysCache

43

Caching on disk

•Public class: QNetworkDiskCache

•Sample implementation of data caching

•Saves all data to one directory on disk

•Controls cache size

•Should work from multiple processes...

44

Proxy factory

•Problematic:• Different destinations accessed via different proxies

• Support for Proxy Auto Configuration (PAC) scripts

•One single proxy for all requests doesn’t cut it...

•New abstract class: QNetworkProxyFactory

•Integrated with QNetworkAccessManager: • setProxyFactory(QNetworkProxyFactory *)

45

How does it work?

•For each request, obtain a list of proxies to try

•Currently we use the suitable one

Proxy factoryRequestQNetworkReplyQNetworkReplyQNetworkReplyQNetworkProxy

46

Proxy Auto Configuration scripts

•Basically JavaScript code to choose a proxy

•Maps to the QNetworkProxyFactory model

•Interpreter provided on Labs

JavaScriptHostname,URL

QNetworkReplyQNetworkReplyQNetworkReplyQNetworkProxy

47

Qt 4.5 is just around the corner

•Technical Preview just released

•Updates and fixes available via Snapshots

48

Agenda

SummarySummary

49

Summary

•Qt Network Access is very powerful

•Easily extensible

Extra slidesExtra slides

Appendix

51

Uniform Resource Locator (URL)

•Data is identified by a URL

•The Qt class for this is QUrl

•Examples:http://www.example.net/faq.html#question13

ftp://ftp.example.com/pub/something/

ftp://tray:[email protected]:2021

mailto:[email protected]

http://www.example.net/faq.html#question13

ftp://ftp.example.com/pub/something/

ftp://tray:[email protected]:2021/

mailto:[email protected]

Documents

DevDays2008 - Qt Network AccessFlexible and Powerful Access to Data on the Internet