The Technical SEO Renaissance

THE TECHNICAL SEO RENAISSANCE

MICHAEL KING

MANAGING DIRECTOR, IPULLRANK

@IPULLRANK

DOWNLOAD THESE SLIDES

http://bit.ly/tech-renaissance

An Abridged History of SEO

From my perspective at least

IPULLRANK.COM @ IPULLRANK

First There Were Webmasters

Before the web world splintered into 50 million different jobs, there was just the Webmaster. In 1995 I was even one of them for Microsoft’s ActiveX team.


Then Came Search Engines

Before Search Engines we just found things on directories or resource pages.


SEO Grew out of Webmaster/Info Retrieval World

Basically, it was just a bunch of nerds trying to get money and succeeding.


For A Long Time, It Was Just About Hacks and Tricks

Remember bolding every single instance of a keyword on the page?


Then Google Showed Up


It Was Still About Hacks, Tricks and Testing Things

Remember page rank sculpting?


Then Google Got Better

That was a bad day for a lot of people.


And Better

Panda ruined a lot more people’s days.


And Better

With Penguin a lot of people figured it was time to finally go white hat.


At the Same Time More SEOs Came from Different Disciplines


And It All Became About Content

Now we are the people that make all of the things…right?

And People Started Saying You Don’t Need to Be Technical

That’s right. As an industry, we finally got on the ‘just make good content’ train.


Yet, the Web is More Technical than Ever Before

We are being dragged into a technical SEO renaissance.


Lucky for You This Is The Point Where I have My Own Agency

Shameless plug.

Changes in Web Technologies

What is pushing this renaissance?


JavaScript is Not Going Away

Nearly every site on the web is using JavaScript to perform some type of transformation on the page. It’s obvious that Google needed to solve for this.


Growth of AngularJS

AngularJS, the single page application, MVW framework continues to grow dramatically in the top million websites.


AngularJS is Popular in Most Verticals

It’s likely that you’ll run into an AngularJS site soon enough in your optimization duties


ReactJS is Growing Too

Similarly, Facebook’s MVW framework is growing as well. Although, it has a built in feature for making sure that it renders proper HTML, but it’s still a framework you should know how to optimize for.

Check out this post https://builtvisible.com/react-js-seo/

View Source is Dead

Inspect Element Only


HTTP/2 – Perhaps the Most Important Thing

All you need to know right now is that it’s fast as hell.

Adoption is Super Low For Now Though

We Need More from Our SEO Tools


What are rankings in 2016?

So, You’re #1? Cool Story, Bro.

Where they at tho?


On What Device, OS and Location?

Different browser and OS configurations will show you different results. So what truly is a mobile ranking?

+ + = Ranking #3+Nexus 5 Android NYCChrome

Safari

+ + = Ranking #1+iPhone 6 iOS NYC

Are we measuring for contextand actionability or just vanity?

Please Give Me Position 0 in My Rankings

Please Give Me My Results in Context of Paid

#1 is really #8 in this SERP

What is cloaking in 2016?


304 Response Codes

The Not-Modified response code tells clients that they do not need to download a page again if it has already been downloaded.


Googlebot Respects the 304

What is cloaking when Google accepts 304 response codes and does not revisit that content after indexation?


What is Cloaking in the Adaptive/Responsive Era?

Many people show less or more content based on the context of the browser/user, so what is cloaking now?

Google accepts directives in HTTP headers.


X-Robots-Tag HTTP Header

Many of the popular crawling tools do not look for this in the response headers.


Hreflang HTTP Header

I have not seen a tool that looks for this HTTP header. However, this implementation is preferred over the many lines of hreflang tags on every page.


Rel-Canonical HTTP Header

Many tools do not look for the rel-canonical HTTP header.


JavaScript Redirects for Mobile

Google allows client side redirects for automation redirection of mobile to desktop and vice versa. Many crawling tools do not account for this.

Many SEO tools do not account for these things.


Chrome DevTools Does

Natively in Chrome’s Developer Tools you can see all of the HTTP Request and Response Headers in the Network tab section.


Remote Debugging

You can also see of this for a specific mobile devices with the remote debugging features of Chrome.

Crawlability



Google is Crawling Headless

Make no mistake that Google has not been crawling with just text driven crawlers for a long time.


Adam Audette Said It So You KNOW It’s True

Adam ran a few tests a little under a year ago further proving out these capabilities


Also an Ex-Googler Confirmed It

A former Google developer said the headless crawling was his primary project from 2006 – 2010.


I Said it in 2011

But Mike, what does it matter?


Log Files Went out of Style?

No, log file analysis has not gone out of style. In fact, it’s more valuable than ever before.

LOOK AT YOUR LOG FILES!


Log File Analysis

We parse the logs and query them in MySQL, but you can use any number of tools for this.


Search Console’s Crawl Stats are Mediocre

I personally find what Google gives us to be pretty much useless. Don’t less this take the place of going through your logs.


Log Files Help You Spot What Really Happens

I had a client come last year that was unsure whether it was Penguin or changes in their offline media buying habits had led to the downturn in traffic. Layering the Googlebot logs made it clear.

0

50000000

100000000

150000000

200000000

250000000

0

500000

1000000

1500000

2000000

2500000

Unknown organic organic home organic dyor

organic er organic home,dyor,er organic cushion organic net of home

organic ritani organic diamonds direct home paid brand sessions

tv spend web sessions (as reported by adaptv) Top 800 Monthly SV impressions

Googlebot Visits

Penguin Update


Social Shares Influence Crawl Patterns

We find that there is a higher correlation with crawling and social shares than crawling and links. Commonly held SEO knowledge would make you believe this is entirely dictated by links.

0

1000000

2000000

3000000

4000000

5000000

6000000

0

100000

200000

300000

400000

500000

600000

700000

Googlebot Visits Links Social Shares FB Twitter G+


Prerendering is the Best Practice

Using a solution like prerender.io is the best practice.


Prerender Can Cause Problems

However, we’ve seen Prerender setups misidentify Googlebot as a human users and serve the wrong the AngularJS versions.

0

20000

40000

60000

80000

100000

120000

Googlebot Perceived as Human


Prerender Setup Serving Non-200

Prepping the prerender cache can cause outages and lead to pages falling in and out of the index.

0

2000

4000

6000

8000

10000

12000

14000

16000

20

6

30

1

30

2

30

4

40

3

40

4

40

5

41

0

50

0

50

3

50

4

20

6

30

1

30

2

30

4

40

3

40

4

41

0

50

3

50

4

20

6

30

1

30

2

30

4

40

4

40

5

41

0

50

0

50

3

50

4

20

6

30

1

30

2

30

4

40

1

40

4

40

5

41

0

50

0

50

3

50

4

20

6

30

1

30

2

30

4

40

3

40

4

40

5

41

0

50

2

50

3

50

4

1 2 3 4 5

-

hit

miss

pass


Also Prerendering is Not a Requirement

We got rid of prerender on 2 sections of the site and saw the traffic from Organic Search increase YoY. Google can index it just fine.


AJAX Crawling Deprecated

Google has gotten rid of the AJAX crawling scheme because they don’t need it anymore. They have perfected their ability to crawl headless at reasonable scale. They say don’t block the CSS or JS because it’s a browser.

They Say Don’t Prerender Just for Google


Prerender Does Speed Up Crawling

There is certainly value in prerendering for speed, but then that makes me wonder, why not just not use Angular?

Scraping

The backbone of a lot of our analysis as SEOs


Most Tools use cURL or Similar

There are many similar libraries, but many SEO tools are built on library called cURL for managing HTTP requests. This is a text based crawler.

This is a fundamental flaw of many SEO tools because they are not constructing the DOM and its CSS/JS transformations. Search Engines are.


PhantomJS – For Scraping the Modern Web

There are many headless browsers out there such as Selenium, but PhantomJS is the go to. http://www.phantomjs.org


HorsemanJS – The Easy PhantomJS

I prefer HorsemanJS which is a NodeJS module that is incredibly simple. http://www.horsemanjs.org


In-Browser Scraping

I wrote a post on how you can scrape using your browser’s Console.


Scrape the Unscrapable

Nothing is in unscrapable when you’re using a browser.


Multi-Page In-browser Scraping

If you want to scrape multiple pages check out ArtooJS https://medialab.github.io/artoo/

Content & Linking

You knew I wasn’t going to get up here and not talk about these things, right?


Google Looks at Entities First

According to Paul Haahr, Google looks at entities first and it appears that we as an industry have talked about this in theory, but not truly worked it into the optimization workflow.


AlchemyAPI

I encourage you to work AlchemyAPI into your keyword research and content optimization process.


Term Relevance

There’s also been a lot of talk about co-relevance, topic modeling and TF*IDF. This is a great opportunity for technical elements to influence content creation.


In Fact Just Do Everything Cyrus Says

https://moz.com/blog/7-advanced-seo-concepts


Moz’s New Tool Will Help

Moz has made the related keywords concept more accessible by adding it to Moz Pro. Searchmetrics also has a similar solution.


John Mueller Says 302s Pass PageRank

Don’t let Google mislead you about 302s vs. 301s.


These Rankings Say He’s Wrong

We had a client with millions for 302s and all we did was change them to 301s

June August September

HyperTraffic (1-3) 1272 1,283 1,473

Traffic (4-10) 979 983 837

Striking Distance (11-20) 167 161 110

Emerging (21-40) 106 103 98

Developmental (41-100) 156 90 87

Unranked 0 60 75

0

200

400

600

800

1000

1200

1400

1600301 redirects


This Traffic Improvement Says He’s Wrong

Traffic went up pretty dramatically despite seasonality.

August 27th301 redirects

11,039,18810,455,517

11,348,902

12,049,481

11,250,148

12,902,847

11,040,192

13,158,296

11,287,790 12,203,99812,508,164

13,158,296

14,041,313

15,609,390

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

14,000,000

16,000,000

18,000,000

April May Jun Jul Aug Sep Oct

Organic Visits

2014 2015


Internal Linking Structures

Internal Linking Structures are still an incredibly powerful asset. I’d like to see more tools like what Portent has built to visualize them and understand where there is opportunity. https://www.portent.com/blog/seo/seo-force-directed-diagrams.htm

Google is becoming the presentation layer of the web. Help them do it.


Schema.org / JSON-LD

Schema.org is a lot more reasonable to implement now that they are allowing JSON-LD.

Page Speed

Google’s next big play


Google Expects Ludicrously Speedy

Google has an expectation that the above the fold content on the page will load within 1 second. You basically get 400 ms to make that happen.


The Critical Rendering Path


DevTools Is Your Friend

In the Timeline section, DevTools gives you a point by point break down of each operation and how it impacts the site’s load time. This is how you can determine what exactly is slowing down your pages.


How Your Code Is Laid Out Matters A LOT


This Is what PageSpeed Insights Helps With


External Resources Also Slow Down the Page

Chartbeat is typically a huge culprit and using rel-dns-prefetch can speed this up considerably.


AMP is a Spec to Enforce the Critical Rendering Path

Sound familiar?


Responsive Sites are SLOOOOOOOW

One of the key things you can do, aside from optimizing for the critical rendering path, is conditional loading.


PreBrowsing Directives

Browsers have preloading directives which allow the page to download elements in the background for later viewing or within the same session.


Not to Be Confused with Prerender.io


Google Uses It in the SERPs


I Sent Thousands of Headless Visits


Rel-Prerender’d Pages Generally Perform Better


Use it With the GA API

Get the pageviews of the ga:nextPagePath based on the ga:previousPagePath set to the current page that you’re on.

You’ll get a list of the pages your users are most likely to visit next. Set one of these as the rel-prerender.


Code.

Inject rel-prerender into the page programmatically.

You could also session the URLs the user is going to toensure you are always prerendering a new page.


Speed Improvements

…

Don’t use it on mobile.

Analytics packages aside from Google Analytics may show fake sessions.


Defer JS Load with Page Visibility API

…


Rel-Preload for Same Page Preloading

I haven’t played with it yet, but Ian says they’re using it on the Portent site. https://www.smashingmagazine.com/2016/02/preload-what-is-it-good-for/

How to Go Forward

Things you should know and do to be prepared for the technical renaissance


Understand the Document Object Model

https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction


Critical Rendering Path

https://developers.google.com/web/fundamentals/performance/critical-rendering-path/analyzing-crp?hl=en


Page Speed

https://www.portent.com/blog/design-dev/ultimate-site-speed-guide-what-impacts-site-speed.htm


Log File Analysis

https://builtvisible.com/log-file-analysis/


Understand SEO for JavaScript Frameworks

https://builtvisible.com/javascript-framework-seo/


Learn DevTools

http://ipullrank.com/6-things-you-should-know-about-in-chrome-devtools/


Accelerated Mobile & Instant Pages

Even if you’re not a publisher, I encourage you to watch these projects as I expect that Google and Facebook will grow the specsdramatically once they’re adopted.


Read Up On HTTP/2

https://moz.com/blog/http2-a-fast-secure-bedrock-for-the-future-of-seo


Get Back to Testing All The Things

Let’s stop chasing the content train and get back to making experiences that perform.


I Just Feel Like Michelangelo Has a Place In This Deck

That’s All I’ve Got

THANK YOU

Michael King

Managing Director, iPullRank

[email protected] http://ipullrank.com

@iPullRank

Marketing

The Technical SEO Renaissance