Enabling Active Site Analytics in WebSphere Portal 7.0

A Tutorial on Active Site Analytics in WebSphere Portal 7.0

Mike TaylorLotus Business Partner Technical EnablementIBM Software Grouptaylormi@us.ibm.com

Table of ContentsIntroduction..........................................................................................................................3Overview of Active Site Analytics.......................................................................................4Writing an aggregator for active site analytics..................................................................10Adding an ASA aggregator to a portal page......................................................................16Instrumenting a theme for Active Site Analytics...............................................................22Troubleshooting.................................................................................................................24About the author................................................................................................................25Trademarks.........................................................................................................................25

Introduction

In WebSphere Portal 7.0, IBM has enhanced it's Site Analytics Support. Analytics is the process of capturing and measuring the user activity primarily to understand the end user needs and behaviors and site usability so that site can be better designed and targeted.As customers create Portal Applications, there is a great need to analyze actual usage in order to determine impact. Data collected will be used as input to determine return on investment (ROI) which in turn will be used to increase or decrease investment or just tune the site. Data is collected using passive site analytic (logging) or active site analytic (tagging). Today, integration with site analyzer tools is performed by manually imbedding tags into portlets and themes or creating reports based on the portal site analyzer logs. In WebSphere Portal 7.0, we have enhanced the user experience by automating the imbedding of tags.

In this tutorial, we will show you how to integrate active site analytics into you WebSphere Portal 7.0 environment.

3

Overview of Active Site Analytics

With active site analytics the web browser becomes part of the event processing system. The user requests portal pages from the portal server. Pages are delivered back to the user for rendering within the browser. Embedded within these page responses are active elements (e.g. Java script elements) that are processed by the web browser. Based on the configuration, the active elements embedded in the page trigger remote requests to the analytics server that report usage events to that server. The analytics server is part of the selected reporting solution and not part of your WebSphere Portal deployment. The analytics server needs to be up and running and accessible from the users browser. The analytics server processes the events and stores the event data in the system specific event data sink. This event data is than processed to generate usage reports for the Portal site.

Figure 1

The high-level flow is shown in figure 1. It starts with the browser requesting a portal page (step 1). The portal will then return the markup of the portal page (step 2), including a reference to the javascript file (the aggregator) and data that is relevant for Active Site Analytics. This data is tagged using a microformat. Once the page is parsed and rendered in the client's browser, the aggregator will execute (step 3). The aggregator will first retrieve all data from the page's HTML representation, and will then use that data to format a request to an external analytics service. The request is then sent to a server of the analytics service, carrying over all the data that was previously collected from within the HTML page (step 4). Typically, but not limited to it, the data submission is done by dynamically injecting an image into the page.

At this point, the responsibility of Active Site Analytics ends. It is the responsibility of the analytics service to process the data and generate reports from it. For example, the image request could be written to an HTTP server log file (step 5). This file could then be processed by a report generation software (step 6).

WebSphere Portal comes with out-of-the-box themes and skins that are already instrumented with basic analytics data. Each item is present in the HTML markup of a portal page, tagged with a CSS class (see Table 2 below). The aggregator can use regular JavaScript operations to retrieve that data from the HTML code.

5

The AggregatorThe main purpose of the aggregator is to collect all relevant analytics information and aggregate it into a single string of information that can then be sent to the analytics server in one request.

An aggregator is written as regular JavaScript code, and there are two interfaces it has to work with: First, the format of data that holds information about the current page, and second, the format of the data that is expected by the analytics server.

The aggregator will, once invoked in the browser, typically iterate over all data instances in the DOM tree of the markup as present in the client's browser, and collect all required information into a representation that is internal to the aggregator (e.g. a JavaScript array). There are numerous ways to select nodes and iterate over them in JavaScript, but we have found the query function of Dojo to be very easy to use and appropriate for the task. The sample aggregator below (see section "Writing An Aggregator") uses that technique. It iterates over all instances of span elements that have a certain class attribute and puts the text of matching elements into an array.

In a second step, the aggregator the takes the collected data and formats it in a way that it corresponds to the interface the analytics server defines. Therefore, the aggregator will be specifically made for a certain analytics server (e.g. there is a dedicated Coremetrics aggregator, another one for Omniture, one for Webtrends, etc.). The same is true for the transmission technique that is used to carry over the collected data to the analytics server. Due to the security models of modern browsers, no JavaScript is allowed to submit data to third-party sites. Since the portal page came from the portal server, the browser sees the analytics server as a third party and prevents direct submission of data to it. However, browsers do allow the retrieval of images from third parties. This exemption from the security rule allows us to send the analytics data as part of the src attribute of the image element. For example, if the aggregator found that the current portal page’s name is “Home”, the aggregator can transmit that information to the analytics server by injecting the following image element into the markup: <img src=”http://server/page.png?name=Home”/>. The analytics provider can then look at the query strings that appear in the HTTP server log file and can derive the page information from that.

There are other ways to carry the information from the browser to the third-party server, but the technique outlined above is common and also used by the sample aggregator that is listed later on.

MicroformatAs the aggregator is running in the clients browser, it has no access to the server-side APIs of WebSphere Portal (unless remote APIs like REST services are used, but this beyond the scope of this paper). Therefore, all information that might be of interest to an aggregator must be embedded into the HTML markup of the portal page. Once the markup has arrived in the browser, the aggregator can only use what’s already in the page.

In order to provide a standardized way to access the embedded data in the page, WebSphere Portal defines a microformat (basically a set of CSS class names that can be added to about any HTML element in a portal page) that is used to tag data that is relevant for Active Site Analytics (see table 2). The data instances are added in the theme and skin JSPs of the portal default theme.

7

Name Description Tagged as Injected inPage Title Title of the page in portal

default language. asa.page.title Theme

Page ID ObjectID of the current page.

asa.page.id Theme

Navigation Breadcrumb

Pseudo URL showing the page’s position in the navigation hierarchy

asa.page.breadcrumb Theme

Friendly URL

Clickable URL of the page without state

asa.page.url Theme

Visitor ID ObjectID of the user currently logged on, or empty.

asa.visitor Theme

Portlet Window Title

Title of the portlet as delivered to the client

asa.portlet.title Skin

Portlet Window ID

Unique identifier of the portlet

asa.portlet.id Published to dojo topic queue*

Portlet Screen ID

Unique identifier of the screen / view that is displayed in a portletasa.portlet.screen.id

asa.portlet.screen.id Custom Portlet

Portlet Screen Title

Title (localized) of the screen / view that is displayed in a portlet

asa.portlet.screen.title Custom Portlet

WCM Content Querystring

Unique identifier of the WCM content item that is displayed in a portletasa.wcm.asa.wcm.content_item.path

asa.wcm.content_item.path WCM Rendering Portlet

WCM Content Title

May not be the same as the portlet window's title.

asa.wcm.content_item.title WCM Rendering Portlet

Search term query Published to dojo topic queue

Number of search results

results Published to dojo topic queue

Search Scope ID

scope.id Published to dojo topic queue

Search Scope Label

scope.label Published to dojo topic queue

*Topic name: "com.ibm.portal.theme.portlet_ready"

** Topic name: "com.ibm.portal.search.RESULTS_ANALYTICS“Data:{

"query": "<query_text>", "results": <total_number_of_results>, "scope": { id: "<scope_id>" label: "<scope_name>“ }}

Table 1

Some of this data may be visible to the end user, some may not. Since we use CSS classes to tag the data, it is easy to add instrumentation to an already existing data element. In addition to that, custom CSS definitions can be added so that there is fine-grained control over the appearance and behavior of the microformat instances.

The default theme of WebSphere Portal 7.0 comes with default instrumentation of analytics data, so that no changes to themes or skins are required. Table 2 lists all data elements that are part of WebSphere Portal 7.0. We may extend the number of tagged information in the future. IBM treats the microformat as public API and will apply proper deprecation techniques if necessary.

9

Writing an aggregator for active site analytics

Note: If you are using Coremetrics, Webtrends, or Omniture Analytics, there are aggregators already available. Please refer o the following wiki pages for information on how to download and install these. (Coremetrics, Webtrends and Omniture) You can write your own scripts to retrieve the data for active site analytics from the portal themes and skins. Such scripts are called aggregators. The portal Tab Menu - Page Builder theme includes a sample aggregator.

The portal themes and skins come with plug points that allow you to inject custom Javascript snippets. These scripts are called aggregators. You can write such an aggregator to retrieve the instances of the microformat in which you are interested. For example, this can be all CSS classes that start with asa. You can extract that data and submit it to the external analytics service where the data are then recorded and processed for evaluation.

The aggregator script is typically a JavaScript file, but can be augmented with a JSP.

The portal is shipped with a sample aggregator that works in the pagebuilder themes and skins. As these already rely on the Dojo framework, the sample aggregator also makes use of Dojo.

However, there is no general dependency on Dojo in the overall Active Site Analytics framework. Instead, the aggregator depends on the implementation details of the themes and skins. For instance, if the themes and skins do not use Dojo, there is no need to use Dojo in the aggregator.

Consequentially, the aggregator should be developed with respect to and managed as part of the themes and skins.

The Sample code

Below is the listing for the asa_sample.js file shipped with Websphere Portal 7.0. We will walk through this code to see how to build your own custom aggregator.

Note: the sample code has been reformatted to fit into this document.

// This is a sample for an Active Site Analytics aggregator. // // It injects a 1x1 pixel image into every page that has the metadata// item "asa_aggregator" set to "asa_sample.js"

// The aggregator will assemble the image URL, containing a list of// portlet ids and titles, as well as a number of additional data, as// HTTP GET parameters, in the query part of the URL. The image is then// added to the page (and requested by the browser, subsequently). // // Testing the sample aggregator // // 1. Follow the Infocenter topic "Adding an Active Site Analytics// aggregator to a portal page" // // 2. In this script, change the value of the variable 'trackingImg'// to point to location that is reachable for a browser via HTTP// (e.g. local Apache server). // // 3. Using a browser, load the page to which an aggregator was// assigned in step 1 // // 4. Check the log file of the HTTP server where the URL from step 2// points to. // It looks something like this: // http://example.com/tracking.gif//asa.portlet.id=xyz0987654321abc&asa.portlet.title=Example&asa.visitor//=1234567890ABCDEFG&asa.url=/wps/myportal/Home/asa&asa.page.title=Samp//le_Page&asa.page.id=XYZ123ABC4567890&asa.page.breadcrumb=/Home/ASA" //width="1" height="1"> // // 5. Check that the query information contained in the requested URL// corresponds to details of the page and portlets. // // For debug purposes, the presence of the tracking picture in the DOM// can be checked with dojo.query("[alt *= 'ASA']") on the client// side, e.g. in FireBug.

var trackingImg = "http://example.com/tracking.gif";

// Subscribe to search events dojo.subscribe("com.ibm.portal.search.RESULTS_ANALYTICS", function(e){ // enable this statement for debugging //console.log( "*** search term was '" + e.query // + "' with " + e.results // + " results in scope '" + e.scope.label + "' (" + // e.scope.id + ")" // ); });

// Only submit to analytics server once the page is complete dojo.subscribe("com.ibm.portal.theme.portlet_ready", function(){ var imgURL = trackingImg + "?" + retrieveData(dojo.query("[class *= 'asa']")); dojo.create("img", {src:imgURL, alt:"ASA Tracking Image"}, dojo.body()); // console.log("*** page initially loaded. Reported URL is: " +

11

// imgURL); // enable this line for debugging });

// Helper - extract all instances of asa microformats and return them // as parameter string function retrieveData(nodes){ var result = new String();

nodes.forEach(function(node, index, arr) { var key = dojo.attr(node, "class").match(/asa\.[^\s]*/); var val = cleanup(dojox.xml.parser.textContent(node));

if (0 < result.length) result += "&"

result += escape(key) + "=" + escape(val); });

return result; }

// Helper - remove clutter from values function cleanup(node_value){ return dojo.map(node_value.split('/'), function(part) { return dojo.trim(part).replace('\n', '') }).join('/'); }

Aggregator Sample WalkthroughThe sample aggregator does not immediately invoke any Javascript code when included in a portal page. Instead, it starts to listen to a Dojo topic and attaches an event handler that is invoked when an event is published to that topic. The reason is that not all microformat instances are present in the page markup when it is delivered to the browser. Instead, some instances are only injected into the page (using Javascript) after the browser rendered the markup. A similar approach is used for search results.

Once all data is present in the page and the Dojo event is published to the topic, the image URL is appended with a query string containing analytics information (page name, portlet titles, etc). The sample code uses the helper function retrieveData, which iterates over all nodes that were passed when calling the function. For each node, the name of the node's CSS class and the text content of the node are formatted as a key=value pair and appended to the return value of the helper function.

Another helper function called cleanup is used to remove undesired whitespace from a node value.

The getElementHTMLByClassName Function If Dojo is not available or desired for an aggregator, the following code can be used to achieve similar functionality.

function getElementHTMLByClassName(strTagName, strClassName, arrReturnElements){ var arrElements = document.getElementsByTagName(strTagName); strClassName = strClassName.replace(/\-/g, "\\-"); var regex = new RegExp("(^|\\s)" + strClassName + "(\\s|$)"); var elem; for(var i = 0; i < arrElements.length; i++){ elem = arrElements[i]; if(regex.test(elem.className)){ arrReturnElements.push(elem.innerHTML); } } return (arrReturnElements)}

This function scans the entire html document for all elements with a tag as defined in “strTagname” and a class name defined in “strClassName”. It returns all these elements in the array defined by “arrReturnElements”.

For example:

If an html document had the following line:

<span class="test" style="display:none;">abcdef></span>

then the following function call:

getElementHTMLByClassName(“span”, “test”, testArray)

would insert abcdef into the array testArray.

Retrieving Portlet Titles

The first section of code uses the getElementHTMLByClassName function to get all the titles of portlets on the current page. The results are loaded into the array portletTitles.

// retrieve portlet titles tagged with // <spanclass="asa.portlet.title"/>var portletTitles = new Array();

getElementHTMLByClassName("span", "asa.portlet.title", portletTitles);

13

Retrieving Portlet IDs

The next section calls getElementHTMLByClassName to get the IDs of all portlets on the current page.

// retrieve portlet IDs tagged with <span class="asa.portlet.id"/>var portletIDs = new Array();

getElementHTMLByClassName("span", "asa.portlet.id", portletIDs);

Retrieve The Page Title And URL

The page HTML will contain a link node with “rel=”bookmark”. This node contains attributes that contain the page title and the page url :<link rel="bookmark" title='Test' href='/wps/mypoc/!ut/p/nm/oid:6_4OPHUKG1089IE0IIST7F831000' hreflang="en"/>

The following code retrieves that link node and stores for use in the next section. (Note: alternatively the asa.page.title tag could have been used to locate the page title).

// retrieve the page's link node containing the bookmarkvar bookmarkElement = null;var links = document.getElementsByTagName("link");for (var index = 0; index < links.length; index++) { var elem = links[index]; if (elem.getAttribute("rel") == "bookmark") { bookmarkElement = elem; break; }}

Create the Image and URL

The last section of code sets up the URL to send the analytics data as part of the src attribute of the image element. Notice that the bookmark element retrieved above is used to get the title and href attributes. The last line writes the img tag out to the html document as a 1x1 pixel image.

var imgURL = "http://example.com/analytics/tracking.png" + "?" + "page_title=" + escape(bookmarkElement.getAttribute("title")) + "&" + "page_url=" + escape(bookmarkElement.getAttribute("href")) + "&" + "portlets=" + escape(portletTitles.join(',')) + "&" + "portlet_ids=" + escape(portletIDs.join(','))//alert("imgURL = " + imgURL);

document.write("<img alt='' src='" + imgURL + "' width='1' height='1'>");

Additional tips for writing aggregators

To retrieve the relevant information from the DOM tree of the HTML document, the aggregator can iterate over all instances of metadata. Within the iteration loop, the aggregator collects and formats individual data according to the requirements of the external analytics service.

After all instances of metadata are collected, the aggregator adds a new element to the HTML DOM tree of the current page, for example, an img element. Adding this element with the source URL that points to the external analytics service triggers a browser request from that remote location. As the element URL contains the collected metadata as HTTP GET parameters, the browser submits this data to the external analytics service.

15

Adding an ASA aggregator to a portal page

Portal administrators can manage the aggregators. they can assign an aggregator to one or more portal labels or pages.

A portal administrator assigns an aggregator to a page by editing the page properties and adding a new parameter. To do this using the portal administration portlets, proceed by following these steps:

1. Click Administration > Manage Pages.2. Locate the page to which you want to assign the aggregator. Use the Manage

Pages portlet to locate the page.3. Click the Edit Page Properties button for the page that you selected.

4. To expand the available choices, click the plus sign ( +) icon next to Advanced Options.

5. Click the I want to set parameters link.

17

6. In the field New parameter, type asa.aggregator.7. In the field New value field, type the name of the aggregator script file. (In this

case asa_sample.js)8. Click Add.

9. Verify that the new parameter has been added to the list.

19

10. All page metadata starting with “asa_js” gets added to a JS object called ibm_page_metadata which can be used in the aggregator. Meta data values are treated as strings, unless the value starts with “=”, in which case they will be treated as JS code or a formula (i.e. a formulaformula‘)Click OK to return to the main Page Properties screen.

11. Click OK to save your changes and return to the Manage Pages screen.

12. Place the aggregator script in in the js directory of the installed theme so that the theme finds it. For the PageBuilder theme, WebDAV can be used to update the theme in the running portal. Custom themes can be edited in the same way, or they can be updated with an aggregator before deploying them

Note: Children pages inherit the script that has been set on the parent page. If you want to use a different aggregator on a child page, follow the same procedure above for the child page to make the appropriate assignment.

21

Instrumenting a theme for Active Site AnalyticsWebSphere Portal provides the Tab Menu - Page Builder theme that is prepared and suitable for use with Active Site Analytics. You can also enhance the other portal themes or crea te your own custom theme to utilize the Active Site Analytics functionality.In order make Active Site Analytics work in a custom theme, you need to do the following:

1. Add metadata to the page.2. Include an aggregator with the page.3. Include microformats of interest, that is related to the statistical data that you want

to collect.

Adding metadata to the pageAn aggregator picks up information stored in the DOM (document object model) tree of a page. This information must be present in the HTML source of the page. The recommended approach to add this information to a page is to implement the theme or skin so that they write all the necessary information into the DOM tree of the page. Examples:

1. You can add the identifier of a portlet to the DOM tree by adding the following line to the Control.jsp of the skin:<span class="asa.portlet.id" style="display:none;"><%= myPortletID %></span>The aggregator can then fetch the identifiers of all portlets on a page by iterating over all occurrences of "span" elements with a class attribute of "asa.portlet.id".Note: This line depends on the definition of myPortletID earlier in the Control.jsp. By default, it is defined by using the following JSP code:<portal-skin:portletID var="myPortletID"/><jsp:useBean id="myPortletID" class="java.lang.String" scope="page"/>

2. You can add the portlet title to the DOM tree by wrapping the <portal-skin:portletTitle> statement in the Control.jsp with a span element that has a class attribute of "asa.portlet.title":<span class="asa.portlet.title"><portal-skin:portletTitle></portal-skin:portletTitle</span>

Including aggregators with the page

The portal ships a default implementation of the theme extension point com.ibm.portal.theme.plugin.ActiveSiteAnalyticsItems. It locates and includes a JavaScript with a name that corresponds to the value specified by the key asa_aggregator given with the page metadata. Add the following code somewhere near the closing body tag ( </body> ) of the Default.jsp file of the theme:

<portal-theme-ext:themeExtension id="com.ibm.portal.theme.plugin.ActiveSiteAnalyticsItems"> <portal-theme-ext:themeExtensionLoop> <portal-theme-ext:themeExtensionItemText /> </portal-theme-ext:themeExtensionLoop></portal-theme-ext:themeExtension>

This code loops over every implementation of the theme extension point and executes it. The default implementation shipped with the portal follows the approach outlined above.

Including microformats of interest

You might need to modify the skins used by a theme so that all of the microformat information that you want to be captured is present when the page is rendered. For example, you can add this code to the Control.jsp file of a skin to ensure the portlet ID of all portlets on the page is available for the aggregator script to discover:<span class="asa.portlet.id" style="display:none;"><%= myPortletID %></span>

In this case the file asa_sample.js looks for elements with the class asa.portlet.id to find the portlet ID of all portlets present on the page. Portlet titles can be rendered by using code similar to this:<portal-skin:portletTitle />To capture such portlet titles, you can add an additional span element with a class defined for titles that the analytics JavaScript file will look for:<span class="asa.portlet.title"><portal-skin:portletTitle /></span>In this case, asa.portlet.title is recognized as the class for all span elements that encompass the rendered text of the portlet titles.

23

Troubleshooting

If Active Site Analytics is not working correctly, perform the following checks:

• Verify that the HTML markup of the portal page contains the metadata for which the aggregator is looking.

• Make sure that the tagging method that is used in the themes, skins, and portlets matches the expected behavior of the aggregator. For example, the Tab Menu - Page Builder theme shipped with the portal tags all metadata with CSS classes whose names start with asa. The sample aggregators look for those CSS classes in order to retrieve portlet identifiers and portlet titles.

• Aggregators are regular JavaScript files. Therefore all tools and helpers that are applicable to generic JavaScript debugging also apply to developing and debugging an aggregator.

About the author

Mike Taylor's current role as part of the Business Partner Technical Enablement team is to collaborate with business partners and customers, developing solutions that leverage products such as WebSphere Portal and Lotus Forms. He provides education to partners and customers on the architecture and best practices for using these products. Mike also has many years of experience in the product development and system integrator roles. You can reach him at taylormi@us.ibm.com

Trademarks

• developerWorks, Domino, IBM, Lotus, Notes, Quickr, Rational, and WebSphere are trademarks or registered trademarks of IBM Corporation in the United States, other countries, or both.

• Windows and Windows NT are registered trademarks of Microsoft Corporation in the United States, other countries, or both.

• Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

• Other company, product, and service names may be trademarks or service marks of others.

25