Why and How SmartNews uses SaaS?

Why and How SmartNews uses SaaS?

@takus 2015.06.26

SmartNews Official Character Chikyu-Kun

Takumi Sakamoto (@takus)

SmartNews Crosses 10 Million Downloads in Global

Source: App Annie

SmartNews has No.1 MAU in News Apps, Japan

Comparison of the other News Apps

4.3M MAU

Source: Nielsen Japan

Ranking of Active Users

 * 2 5 1 1 199     -‐‑‒ E .

1

2

3

4

5

6

7

8

9

10

1 B 12 A 0 1 5

Long Time Spent

Facebook messenger

LINE Facebook Gmail Youtube COOKPAD News App A

SmartNews News App B

News App C

※ Spending minutes/person, month

Source: Nielsen Japan

Why SaaS?

Use cases

Relationship

Summary

We want to focus on OUR PRODUCT

We must move fast

Engineering resources are

limited

Efficient Ads

System

Document Classification

Site Reliability

Engineering

High Performance

API Server

News Crawler

Realtime Ranking

Algorithm

Data Management

Platform

UI/UX

Can we assign excellent engineers to every positions?

No, probably

The answer is

Why SaaS?

• Solves specific issues for us

• monitoring, alerting, visualising etc...

• Less bootstrap process

• tiny setup process (a java agent, a chef recipe etc...)

• built-in user friendly web Interface

• No dedicated person is needed to develop/operate

• experts in an area do it well

• easy to manage with programable API

Why SaaS?

Use cases

Relationship

Summary

NewRelic

• (Application) Monitoring as a Service

• easy setup for popular language (e.g. Java)

• add method traces by annotation

• trace SQL queries automatically

• records deployments

Setup for Tomcat

• deploy newrelic archives to $CATALINA_HOME

• edit newrelic.yml

• application name

• add -javaagent

• $CATALINA_HOME/newrelic/newrelic.jar

• add environment if necessary

• -Dnewrelic.environment=xxx

Customize for each Apps

# add method tracer @Trace(metricName="API/yourMethod", dispatcher=true) public Response yourMethod() { ... ... ... }

# add custom metric MetricAggregator aggregator = NewRelic.getAgent().getMetricAggregator(); aggregator.recordMetric(key, value);

# record exception NewRelic.noticeError(ex);

Trace methods & queries

Record Deployments

from fabric.api run from fabric.decorators import runs_once, parallel

@parallel @task def deploy(): do_your_task() notify()

@runs_once def notify(): ver = release_version() run("java -jar newrelic.jar deployment --appname=xxxx --revision=%s" % ver)

Record deployment by fabric

https://docs.newrelic.com/docs/agents/java-agent/instrumentation/recording-deployments-java-agent

NewRelic Insights

Query to custom metrics via NRQL SmartNews seems top 10 user of Insights

What's great?

• Developer can monitor their own app by themselves

• adding custom tracer, custom metrics

• Don't have to prepare profiler for each language

• Java, Scala, Ruby, (Golang), etc...

• suit for Microservices

Datadog

• (System) Monitoring as a Service

• easy to setup with Chef

• 100+ built in integration (AWS, MySQL, Docker...)

• correlate metrics with events

• tag-based flexible visualization

• build your own interactive dashboards

Setup dd-agent with Chef

# Install cookbook

knife cookbook site install datadog

# Set Datadog-specific attributes

node.default['datadog']['api_key'] = "xxxxxxxxxx"

# Add a recipe in your run list

"run_list": ["recipe[datadog::dd-agent]"]

BTW, What is dd-agent?

http://help.datadoghq.com/hc/en-us/articles/203034929-What-is-the-Datadog-Agent-What-resources-does-it-consume-

Built-in Integrations

Setup MySQL Monitor

# Edit conf.d/mysql.yaml

instances:

- server: localhost

user: datadog

pass: xxxxxxxx

tags:

- dbtype:master

options:

replication: 0

Correlate metrics w/ deployment

New Relic Integration

sum:apache.status_2xx_count{role:web} by {env}

EC2 Classic

VPC

Tag-based flexible visualization

http://qiita.com/takus/items/c1d71bfcc231d0c24e0a

Migrate from EC2 Classic to VPC with Route53

role = web env = classic

role = web env = vpc

Provision clusters with Chef

role = web env = classic

role = web env = vpc

Aggregate metrics with 'env' tag

Check clusters by Host Maps

https://www.datadoghq.com/blog/2015/03/introducing-host-maps-know-thy-infrastructure/

Build your own dashboards

PagerDuty Integration

Only pager for related teams (members)

What's great?

• No dedicated Ops required for system monitoring

• chef cookbook & built-in integration

• customisable dashboard for each developers

• Find the root cause quickly

• gather all of metrics/events/alerting at one place

• tag-based flexible visualization

• correlating metrics & events

Chartio

• Dashboard as a Service

• create beautiful dashboards

• provide connectors to various data sources

• MySQL, Redshift, Google Analytics, BigQuery, etc...

• Presto with Prestogres

• non-engineer can create dashboard w/o SQL

How to create charts?1. Build a query to data source (drag & drop / SQL)

2. Modify data (filter、sort、add column)

3. Visualise it (table, pie chart, bar chart)

Connecting to Presto

https://github.com/treasure-data/prestogreshttp://docs.treasuredata.com/articles/chartio

Join BigQuery & MySQL

Layer1 BigQuery Result

article_id country pv

1 JP 1000000

2 US 2000000

3 GB 3000000

4 JP 4000000

article_id url title

1 http://aaa Article A

2 http://bbb Article B

3 http://ccc Article C

4 http://ddd Article D

Layer2 MySQL Result

Join

https://support.chartio.com/docs/charts/#merging-data-joins

Join BigQuery & MySQL

article_id country pv url title

1 JP 1000000 http://aaa Article A

2 US 2000000 http://bbb Article B

3 GB 3000000 http://ccc Article C

4 JP 4000000 http://ddd Article D

Joined Layer : BigQuery + MySQL

https://support.chartio.com/docs/charts/#merging-data-joins

CSV

Graph

Mail

Categorical Drop down

Could you give me the ios's article ranking of bbc.co.uk in UK?

e.g. : A request from BizDev team

How is the result of a A/B test in California? It seems that iPhone 6's user complaints about new features. Can I compare with the result of each devices?

e.g. : A request from client developers

Categorical Drop down

What's great?

• No dedicated SQL Monkey required for visualising

• everyone can be SQL Monkey

• non-engineer can do by WebUI

• Easy to drill down from overview to each users

• overall summaries

• each OS, country, region, language, etc...

c.f. About Our Data Platform

https://goo.gl/Rc1uuO http://goo.gl/KL1fpL

OneLogin

• Identity as a Service

• provides single sign on to various SaaS

• SAML / Auto complete form with browser extension

• non-frustrated 2FA with mobile app

Why SaaS?

Use cases

Relationship

Summary

Send a Pull Request

Send a Pull Request

• Wrote a fluentd - datadog integration

• only aiming for meeting our requirements

• It became an official integration

• https://www.datadoghq.com/2015/02/monitor-

fluentd-datadog/

Hold an User Meetup

http://blog.takus.me/2015/02/26/dd-sushi/

Gifts from SaaS

Discuss products in conferences

Discuss products in conferences

• Fortunately, I had some chances to attend them

• Velocity Conference 2013, 2015

• Amazon re:Invent 2014

• Great time to talk with core developers

• learn new services & new features

• explain our feature requests by myself

• ask their product roadmap

New Relic's Service Map

Anomaly detection by SumoLogic

Future of SaaS

• SaaS is becoming hot area in startup

• 2014/12/11: NewRelic raised $115M in IPO

• 2015/01/29: Datadog raised $31M in Funding

• 2015/06/01: SumoLogic raised $80M in Funding

• SaaS will evolve rapidly with a large amount of funds?

• probably in my opinion

Why SaaS?

Use cases

Relationship

Summary

SaaS makes life easier

• The real world is complex. We have a lot of tasks to do

• Professionals solve specific issues in behalf of us

• No dedicated engineer is needed for it

• We can focus on our own problems

We’re hiring!

Site reliability engineer iOS/Android engineer

Web application engineer Productivity engineer

ML/NLP engineer Growth hack engineer

Ad engineer

http://about.smartnews.com/en/careers/

One more thing...

Join Our Free Lunch!

We love talking about Technologies, Products, Medias, etc...