Monitoring the acquisition process by web widgets Leonardo Tininini and Antonino Virgillito ISTAT Meeting on the Management of Statistical Information

Monitoring the acquisition processby web widgets

Leonardo Tininini andAntonino Virgillito

ISTAT

Meeting on the Management of Statistical Information Systems

(MSIS 2011)

Luxembourg, 23-25 May 2011

Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011 2

Outline

• Motivations• The web widgets

– The "General picture" widget– The "Regional thematic map" widget– The "Latest news histogram" widget

• Enhancing performances by materializing data• The widgets' software architecture


Motivations

• The web is increasingly becoming a fundamental means for supporting several phases of the statistical data life cycle, particularly data acquisition and dissemination, however...

• little interest has been devoted so far to use the web for publicly monitoring the acquisition process

• Objectives:– providing, as far as possible, fresh data – do not disturb... (the acquisition process itself)– make the components “embeddable” in any web page


Web widgets

• A widget is a small web component that can be freely “embedded” into any web page (e.g. a blog), without requiring specific technical knowledge

• Widgets can be composed and organized in personalized dashboards according to the user's specific needs and preferences

• The widgets described here display some key data about the 2010 Italian Agriculture Census acquisition process


The “General picture” widget

• Shows the numeric values of 5 main aggregates:– the number of operators– the number of interviews completed by operators and

recorded in the Census Management System– the number of questionnaires (fully) compiled by

enterprises by using the web– the number of questionnaires returned (compiled) by

operators by the web– the number of questionnaires under compilation (i.e.

only partially compiled) on the web by enterprises.

• Accepts an additional parameter to select the territory of interest (Italy / a specific region)

• Refreshed every minute


The “General picture” widget (snapshot)


The “Regional thematic map” widget

• Same 5 aggregates as those in the “General picture” widget, each selectable by its own tab

• Aggregates are represented on the map of Italy using classes of values

• Exact aggregate values, along with names, can be obtained by rolling over the region on the map

• Each region's color is faded in proportion to the value of the specific aggregate for that region

• The maps are built using the freely-available Google Interactive Chart Tools API (specifically, the Geo Map component)


The “Regional thematic map” widget (snapshot)


The "Latest news histogram" widget

• Shows the time distribution of the questionnaires compiled over the last 24 hours

• Both enterprises and operators– the fraction corresponding to enterprises is depicted in

darker green– the one corresponding to operators in lighter green

• Details can be obtained by rolling over some sections of the widget– the roll-over on each bar displays the details of the two

component values (enterprises vs. operators)– the roll-over on the two legend items displays two

different histograms, corresponding to questionnaires compiled only by enterprises or operators, respectively


The "Latest news histogram" widget (snapshot)


How to show the enterprise-only histogram


Enhancing performances by materializing data

• Data displayed by widgets are expensive to be computed (counts on several hundreds of thousands of records)

• Unpredictable number of users (due to free embedding)• Potential performance degradation for both:

– the operators and enterprises working on the Census Management and Acquisition System and

– the widget users

• This potential degradation has been avoided by adapting two techniques, commonly known as view materialization and query rewriting, widely used in data warehousing environments– Displayed data are obtained by using (and possibly further

processing) a few pre-computed data, instead of aggregating data from scratch


Materialized views and query rewriting


The widgets' software architecture

• Single Java web application, directly accessing the Census database

• High scalability: minimum (and bounded) overhead on the database against unpredictable request traffic

• Decoupling the access to the widgets made by the user from the access to the database (with aggregation) made by the widgets– One component (Update Job) computes the aggregates

(storing them into the application memory) and is scheduled to automatically run every minute

– The front-end component (widget in stricter terms) displays the aggregates stored in the application memory


High-level architecture of the web application


The widget embedding process

• Each widget is implemented as a separate JSP page• The embedding starts form a single row of predefined HTML

code that has to be included in the page (Embedding code)

• the code produces a call to a JavaScript fragment (Embedding script) that is stored on the server and is downloaded and executed on the client browser

• The execution of the Embedding script creates a frame within which the JSP page rendering the widget is actually loaded

• Through this technique, the widgets can be incorporated into virtually any web site or blog regardless of the underlying technology and without any programming knowledge

<script src="https://censimentoagricoltura.istat.it/CensWidget/caTable.js" type="text/javascript"> </script>


Conclusion and future works

• The proposed design can be considered as a general pattern for the acquisition process web monitoring:– easily embeddable “lightweight” components– high level of scalability, by...– decoupling the acquisition data from the monitoring

data provided to users– price to pay: “freshness” of the information provided

• Next (more severe) testbed: monitoring the acquisition process of the Population Census– one/two order(s) of magnitude larger– the trade-off between freshness and overload will be

much harder to achieve


Questions?

Documents

Monitoring the acquisition process by web widgets Leonardo Tininini and Antonino Virgillito ISTAT Meeting on the Management of Statistical Information