how_first_click_free_can_help_in_SEO

Nuera Limited is registered in England number 3849708. Registered Office:

70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S.

International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263

1

Copenhagen, Wednesday, 25 May 2011

FIRST CLICK FREE IMPLEMENTATION

Client: NE.SE

Website: http://www.ne.se

Report date: 26th

April, 2011

Version: 1.0

Author: Mark Edmondson & Kristoffer Ewald (Guava DK)

Introduction

This report outlines a set of recommended updates for the existing website that will enable

Google’s “First Click Free” implementation. (FCF)

Version: 1.0- Created: April, 2011 - Last modified: May, 2011

Amaliegade 3-5, 3. sal

1256 Copenhagen K

Tel: 7027 8089

Mail: [email protected]

http://www.ne.se/




2

SEO recommendations Page

First Click Free Implementation........................................................................................................................................... 1 Introduction ....................................................................................................................................................................... 1

Technical Implementation ...................................................................................................................................................... 3 1. Detecting Referrals .................................................................................................................................................. 4

Verifying Googlebot with reverse DNS lookup .................................................................................................................. 5

2. Meta tag – noarchive................................................................................................................................................ 6 3. Robots.txt ............................................................................................................................................................... 7 4. Cookies for FCF ...................................................................................................................................................... 8

Implementing Flash LSO cookies ........................................................................................................................................ 9 USER ABUSE CONSIDERATIONS .................................................................................................................................... 10 3rd PARTY WEBSITES ....................................................................................................................................................... 12

Other Search Engines ....................................................................................................................................................... 12 Social Media websites ...................................................................................................................................................... 12 SEO considerations with backlinks .................................................................................................................................... 12 Partner Websites / Content Syndication .............................................................................................................................. 12




3

TECHNICAL IMPLEMENTATION

First Click Free for Web Search - http://googlewebmastercentral.blogspot.com/2008/10/first-click-free-for-web-

search.html

This first section of the report focuses on the technical requirements to have Google’s First Click Free (FCF) working on

the website.

Google’s guidelines for First Click Free are:

Google’s crawlers must be able to access the content of the site without the need for cookies or filling out forms

Suggested implementation is to configure webservers to not serve the registration pages to the crawler

Allow crawler in robots.txt

Multi-paged articles must all be accessible for the user

Page displayed must be identical to users to that which Google sees

This document will also look at what ways to implement enhancements to this package such as:

Implementing five articles per day with cookies

Complementary meta-tags

Using similar techniques for other traffic sources apart from Google

Blocking user’s attempts to abuse FCF

http://googlewebmastercentral.blogspot.com/2008/10/first-click-free-for-web-search.html

http://googlewebmastercentral.blogspot.com/2008/10/first-click-free-for-web-search.html




4

1. Detecting Referrals

The Referrer request-header - RFC: http://tools.ietf.org/html/rfc2616#section-14.36

All implementations of First Click Free need a mechanism to detect where the visitor has arrived from. This is preferably

implemented by using the HTTP referrer in the server headers, rather than on-page scripting such as JavaScript, since

this could be circumnavigated.

For FCF, the referrer is recommended to look for user agents of the form:

*.google.*

Since this will cover the combinations of its search engines around the world (www.google.se , www.google.com, etc.)

A full list of Google User agents is here: http://www.google.com/support/webmasters/bin/answer.py?answer=1061943 –

this may be useful if looking to only give access to Googlebot-Video or Googlebot-Mobile, for example

Using Apache and .htaccess as an example, this would typically be implemented similar to the following:

#Have a 403 error document for Denied users with subscription page

ErrorDocument 403 /subscribe-now.html

#set the first click free variable

SetEnvIfNoCase Referer "*google*" first_click_free=1

#Send first_click_free past registration screen, others get 403

<FilesMatch "(/subscription-content/.*)">

Order Deny, Allow

Allow from env=first_click_free

Deny from all

</FilesMatch>

The above is intended as example only, and it may be preferred to use a server side scripting language such as JSP to

handle the referral requests – however, the principle remains the same: detect if the referral is from *.google.* and

redirect past a subscribe request.

http://tools.ietf.org/html/rfc2616#section-14.36

http://www.google.se/

http://www.google.com/support/webmasters/bin/answer.py?answer=1061943




5

Verifying Googlebot with reverse DNS lookup

Google Help files: http://www.google.com/support/webmasters/bin/answer.py?answer=80553

Users could try to circumnavigate the restrictions by setting their browser’s User-Agent to Googlebot – to prevent this a

reverse DNS lookup can be used to verify that the bot comes from googlebot.com and action taken accordingly.

The example given in the link above is:

> host 66.249.66.1

1.66.249.66.in-addr.arpa domain name pointer

crawl-66-249-66-1.googlebot.com.

> host crawl-66-249-66-1.googlebot.com

crawl-66-249-66-1.googlebot.com has address 66.249.66.1





6

2. Meta tag – noarchive

NoArchive Information: http://noarchive.net/meta/

It is also recommended to place a “noarchive” meta tag on the pages under the FCF regime. This metatag tells Google

not to keep a version of the webpage which could be accessed via a “cache:” command in Google to access page

content:

To prevent this, place this tag in each page to be available under FCF:

<meta name="robots" content="noarchive" />

http://noarchive.net/meta/




7

3. Robots.txt

http://www.robotstxt.org/

Ensure that Googlebot is allowed in the robots.txt file for all FCF content – that is the User-agent in the file found at

ne.se/robots.txt does not block Googlebot – the current robots.txt file is:

User-agent: *

Disallow: /admin/

Disallow: /clusterjsp/

Disallow: /dev/

Disallow: /dns/

Disallow: /error/

Disallow: /modals/

Disallow: /resources/

Disallow: /temp/

Disallow: /xhp

Disallow: /static/xhp

Disallow: /google69eed29ee822c5b1.html

Disallow: /ip.jsp

Disallow: /newnormalpage.jsp

Disallow: /testads.html

Disallow: /testads.jsp

Disallow: /testanalytics.jsp

Disallow: /testlayout.jsp

Disallow: /user/

Disallow: /openschool/

Disallow: /sok/

Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&from=f&to=l

Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&from=l&to=q

Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&from=q

Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&to=f

http://www.robotstxt.org/




8

4. Cookies for FCF

Cookies are not strictly necessary for FCF, however may be needed if:

FCF content is available across paginated articles, and a user needs to be tracked across a session

The five (or more) articles a day per user implementation is required

In such cases, a first party cookie is needed to be set to track users within and between sessions.

For paginated articles a session cookie (expire=0) is all that is necessary

For the Five clicks a day the expiration should be 24 hrs and include a counter on article views.

Cookie Page Request Diagram




9

Implementing Flash LSO cookies

Using cookies for tracking users isn’t ideal as it can be circumnavigated quite easily – a user can use another browser,

flush their browser cookies etc., however if the cookie information can be moved to a Flash LSO the cookie information is

more rarely removed and can be more reliable.

Guava have a solution that can provide this extra robustness in the cookies, which also allows more advanced

implementation of Google Analytics tracking. How these files communicate between the Flash and the client javascript

has been included with this report – please review this code and the examples provided.

The example in the files (ne.se.example.html) shows very simple the methods of setting, retrieving and resetting a

variable in the LSO using Javascript.

Left out is the actual synchronisation with a possible browser cookie, but it should be fairly simple to build this upon the

methods in the example:

1) If a browser cookie exists: Store the cookie value in the LSO

2) If a browser cookie does not exist but the LSO exists: Create browser cookie with the value in the LSO

3) If neither exists: Create browser cookie and store the value in the LSO




10

USER ABUSE CONSIDERATIONS

Some web-savvy web users may look to work around the measures put in place for First Click Free. An example of

some techniques that are used against the New York Times is shown here

http://www.makeuseof.com/tag/5-ways-york-times-paywall/

In this section we look at common methods and what technical measures can be in place to help prevent them:

Disabling Javascript – if the referral or cookie setting scripts are done via JavaScript, simply turning JavaScript off will

enable users to carry on as if the script wasn’t there.

Likelihood: 5 out of 5

Prevention: Avoid security based on JavaScript.

Faking Referrals – with plugins or other methods, a user can make his browser look like it is a Google crawler when

surfing the site. This way any privileges given to Googlebot will also apply to that user, including access to FCF content.

To prevent this, only techniques such as reverse DNS lookups to verify the User Agent will work.


Prevention: reverse DNS lookup to verify user-agents

Blocking/Flushing Cookies – the most common method, this will allow a user to navigate restrictions based on browser

cookies, such as those tracking how many articles they have seen that session or day. If the Flash cookies LSO are

used this will go a long way to stopping this occurring, since Flash cookies are a lot harder to remove. Also, tracking the

user’s IP address as well and comparing them across cookies should flag up users who are holding multiple sessions.


Prevention: Compare cookie to IP address, LSO flash cookies

Using another Browser – this is related to the cookie issue above: if a user uses another browser such as Internet

Explorer vs Firefox, then a new set of cookies are needed. The only way to prevent this if they are using the same PC is

to also track the IP addresses and compare them across cookie sessions.


Prevention: Compare cookie to IP address, LSO flash cookies

Using Google search – a common method which prompted the Five Clicks a day policy, users could take advantage of

the first click free from Google domains to do searches in Google that would bring up the content they knew the user

could see. They could then click through and read the article at will. This would need to be prevented by tracking a user

across sessions, either via IP address or cookies.


Prevention: Cookies to limit access

Using Google cache – whilst Google crawls the web, it keeps a copy of every page available in its search results in the

Google cache (sitting in RAM on its numerous servers) – a user could access this information by using a “cache:www.url-

to-look-at.com” command to get access to all content Google can see. The use of the “noarchive” meta-tag is meant to

prevent this.


Prevention: “noarchive” metatag

http://www.makeuseof.com/tag/5-ways-york-times-paywall/




11

Printing out the web article – A user may look to take the article and print out a hard copy using the browsers “Print”

command. If this is not wanted for non-paying subscribers, look at using stylesheets to make the print page blank for

non-logged in users – i.e.

<style type="text/css" media="print">

body {visibility: hidden; display: none;}

</style>


Prevention: CSS Hacks

Copy-Paste – A user may also use their PC to copy-paste article information and place in their own format – an even

worse scenario will be for them to publish this information on the web giving away the content with no restrictions. Whilst

little can be done to prevent this against a determined user, steps such as disabling right click on a page will prevent the

casual user (Although create usability concerns) – these commands are usually still easily circumnavigated by turning

JavaScript off.

Other methods include making it harder by turning off the copy-paste highlighting for your page:

ondragstart="return false" onselectstart="return false"

Another solution is to place a transparent gif across the entire article that is copied instead when users try to.

A final and more marketing focused solution would be to use something like the Tynt service, which inserts links back to

the website when someone copy-pastes – in this way there may at least be a link back to your website.

http://www.tynt.com/publisher-tools


Prevention: Disabling right click?

http://www.tynt.com/publisher-tools




12

3rd PARTY WEBSITES

This section deals with how First Click Free affects other 3rd

party websites that are not Google, and suggested

strategies on how www.ne.se can best deal with them.

Other Search Engines

Whilst Google is the dominate search engine for the moment, other search engines such as Bing and Yahoo are still

used and should be catered for if at all possible. In many cases this will simply mean include those search engine User

agents in the same process as Google, including cookies and referrals.

The most pertinent user agents will be:

Google Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Full list including mobile - http://www.google.com/support/webmasters/bin/answer.py?answer=1061943

Bing msnbot/1.1 (+http://search.msn.com/msnbot.htm)

Yahoo Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Social Media websites

Getting mentions on other social media sites can be good for marketing and page purposes, and it may be considered

worthwhile payment in getting a social mention to give some free access to users who follow the link.

Again, this could be restricted on a per user basis, or per referral by only allowing 10 visits per day from that site etc.

SEO considerations with backlinks

To compete well in SEO, backlinks are part of the essential mix. If the site restricts content then naturally less people will

link to the content as it will be less useful for readers hitting a paywall. A good incentive to increase linking to the site

would be to offer users viewing the content from that page a free view of the page.

Factor in the cost of that link if it had to be bought for marketing and SEO purposes verses the revenue lost from

potential subscribers, and it may look to be worthwhile.

This could be coupled with only allowing users with a paid for subscription to be able to use the links for their own

websites, ensuring that at least one subscriber has paid per link and also providing additional incentives to become a

member of www.ne.se

Partner Websites / Content Syndication

Rather than let any website that links to www.ne.se gain free access, websites that are pre approved partners could

appear on a “white-list” of websites that allow its users to see www.ne.se content. Effectively this is what Google will be

receiving, and other websites will generally give better converting traffic than Google.

http://www.ne.se/

http://www.google.com/bot.html


http://search.msn.com/msnbot.htm

http://help.yahoo.com/help/us/ysearch/slurp

http://www.ne.se/

http://www.ne.se/

http://www.ne.se/

Marketing

how_first_click_free_can_help_in_SEO