• Home

  • Documents

  • Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create...
26
Confidential Customized for Lorem Ipsum LLC Version 1.0 Spark na Google Cloud Friends don't let friends build data centers

Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Confidential Customized for Lorem Ipsum LLC Version 1.0

Spark na Google CloudFriends don't let friends build data centers

Page 2: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Data Scientist @Avenue Code

Evandro CaldeiraCientista de dados na Avenue Code. Formado em Engenharia da Computação e louco por café

E-mail: [email protected]

mailto:[email protected]
Page 3: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

BrasilBelo HorizonteSão PauloPorto Alegre

EUA

Canadá

Page 4: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Confidential Customized for Lorem Ipsum LLC Version 1.0

TOC

Overview

On premise vs cloud

Como migrar

Demo

#
Page 5: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Por que Spark?

Page 6: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 7: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 8: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

On premise1 Equipamentos

2 Gerenciamento

3 Picos de uso

4 $$$

Page 9: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Migragação para GCP:Descomissionamento de um datacenter em 2018

Spotify

Page 10: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Primeiros passos para GCP

Page 11: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 12: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 13: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

O que fazer1 Mova os dados

2 Experimente

Page 14: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

O que fazer3 Use clusters efêmeros

5 Delete o cluster ao finalizar

4 Workers preemptivos

https://cloud.google.com/dataproc/docs/concepts/compute/preemptible-vms?hl=pt-br
Page 15: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Hands on

Page 16: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 17: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Créditos grátis!

https://cloud.google.com/free/?hl=pt-br
Page 18: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Instalação1 Google SDK

2 Spark standalone

https://spark.apache.org/downloads.html
Page 19: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Criação do cluster

Page 20: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
https://cloud.google.com/dataproc/docs/guides/create-cluster
Page 21: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 22: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Execução de job

Page 23: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
https://cloud.google.com/dataproc/docs/guides/submit-job?hl=pt-br
Page 24: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 25: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Sourcehttps://github.com/evandroc/tdc-spark

https://github.com/evandroc/tdc-spark
Page 26: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Obrigado.


MySQL Cluster 7 - O'Reilly Mediaassets.en.oreilly.com/1/event/36/Discover What_s New in... · 2010-04-15 · MySQL Cluster 7.1 Monitor Reliability Manage Access Platform Faster restarts

MySQL Cluster 7 - O'Reilly Mediaassets.en.oreilly.com/1/event/36/Discover What_s New in... · 2010-04-15 · MySQL Cluster 7.1 Monitor Reliability Manage Access Platform Faster restarts

Documents
CMA-ES with Restarts for Solving CEC 2013 Benchmark ...CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems Ilya Loshchilov Laboratory of Intelligent Systems Ecole Polytechnique

CMA-ES with Restarts for Solving CEC 2013 Benchmark ...CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems Ilya Loshchilov Laboratory of Intelligent Systems Ecole Polytechnique

Documents
ST. NICHOLAS NATIONAL SHRINE AUG RESTARTS CONSTRUCTION

ST. NICHOLAS NATIONAL SHRINE AUG RESTARTS CONSTRUCTION

Documents
LEGAL LABELS · 2020-01-21 · EXHIBIT LABELS MAILING LABELS OFFICE USE, DOCUMENT & INVENTORY LABELS WATER BOTTLE LABELS ANNIVERSARY SEALS PROMOTIONAL LABELS LEGAL LABELS The verdict

LEGAL LABELS · 2020-01-21 · EXHIBIT LABELS MAILING LABELS OFFICE USE, DOCUMENT & INVENTORY LABELS WATER BOTTLE LABELS ANNIVERSARY SEALS PROMOTIONAL LABELS LEGAL LABELS The verdict

Documents
Kids Labels - Name Tags & Labels for kids

Kids Labels - Name Tags & Labels for kids

Presentations & Public Speaking
Parent Club Start Up or Restarts

Parent Club Start Up or Restarts

Documents
Fast Database Restarts at Facebook

Fast Database Restarts at Facebook

Documents
Airbus restarts work on final assembly line for A320 in

Airbus restarts work on final assembly line for A320 in

Documents
Sustainable Labels Labels + Packaging Product Guide for

Sustainable Labels Labels + Packaging Product Guide for

Documents
Independent Record Labels Update die Labels

Independent Record Labels Update die Labels

Documents
BootJacker : Compromising Computers using Forced Restarts

BootJacker : Compromising Computers using Forced Restarts

Documents
Referee Training Course - Restarts June 17, 2003 P. 1 Richard Baker - 2003 Restarts This material was developed for the Entry Level of becoming a Referee,

Referee Training Course - Restarts June 17, 2003 P. 1 Richard Baker - 2003 Restarts This material was developed for the Entry Level of becoming a Referee,

Documents
Easy labels - Personalized fabric labels

Easy labels - Personalized fabric labels

Documents
Rapid Randomized Restarts for Multi-Agent Path Finding: … · 2018-06-14 · Rapid Randomized Restarts for Multi-Agent Path Finding: Preliminary Results Extended Abstract Liron Cohen

Rapid Randomized Restarts for Multi-Agent Path Finding: … · 2018-06-14 · Rapid Randomized Restarts for Multi-Agent Path Finding: Preliminary Results Extended Abstract Liron Cohen

Documents
Quicklabel System - Candy Labels and Chocolate Labels

Quicklabel System - Candy Labels and Chocolate Labels

Business
Labels by Accurate Labels Private Limited, Noida

Labels by Accurate Labels Private Limited, Noida

Business
REStARTS: Das Experimentierbuch · REStARTS: Das Experimentierbuch A u t o r en: VKI - Patricia Corieri – Philippe Léonard – Michel Riethmuller – Mario Carbonaro CIRA – Marika

REStARTS: Das Experimentierbuch · REStARTS: Das Experimentierbuch A u t o r en: VKI - Patricia Corieri – Philippe Léonard – Michel Riethmuller – Mario Carbonaro CIRA – Marika

Documents
Risk-based restarts of untreated subsea oil and gas ... · Risk-Based Restarts of Untreated Subsea Oil and Gas Flowlines in the GoM 6 required to achieve successful understanding

Risk-based restarts of untreated subsea oil and gas ... · Risk-Based Restarts of Untreated Subsea Oil and Gas Flowlines in the GoM 6 required to achieve successful understanding

Documents
A & A Labels · Pesticide labels, Self Adhesive Labels, Printed Labels, PE Label, PP labels & FMCG Labels. About Us Founded in the year 1995, the pioneering Manufacturer and Supplier

A & A Labels · Pesticide labels, Self Adhesive Labels, Printed Labels, PE Label, PP labels & FMCG Labels. About Us Founded in the year 1995, the pioneering Manufacturer and Supplier

Documents
2013 Section Meeting Coaching Workshop Restarts/Set Pieces For the Intermediate Coach

2013 Section Meeting Coaching Workshop Restarts/Set Pieces For the Intermediate Coach

Documents
Mindware · Manufacturer of Labels, Product labels,clear to clear labels,mark sensing labels , shrink labels, movie tickets, taffeta suppliers, thermal ribbon convertors, labels manaufatcurer

Mindware · Manufacturer of Labels, Product labels,clear to clear labels,mark sensing labels , shrink labels, movie tickets, taffeta suppliers, thermal ribbon convertors, labels manaufatcurer

Documents
Adcraft Labels | Specialty Labels and Flexible Packaging€¦ · Tamper evident labels Self destruct labels Authenticity labels Track & Trace labels Flexible Packaging & garner Films

Adcraft Labels | Specialty Labels and Flexible Packaging€¦ · Tamper evident labels Self destruct labels Authenticity labels Track & Trace labels Flexible Packaging & garner Films

Documents
Handling Screen Rotations and Other App Restarts - Custom

Handling Screen Rotations and Other App Restarts - Custom

Documents
Restarts 1. Topics Kick-Off Throw-In Goal Kick Corner Kick Dropped Ball Free Kick – Direct – Indirect – Penalty 2 Restarts when ball is out of play Restarts

Restarts 1. Topics Kick-Off Throw-In Goal Kick Corner Kick Dropped Ball Free Kick – Direct – Indirect – Penalty 2 Restarts when ball is out of play Restarts

Documents
Welcome to Robos. · face stickers Security labels Gluing dots Kanban- cards Booklet labels PIN covering Type labels Decoration labels Hologram labels Stamped parts Safemark labels

Welcome to Robos. · face stickers Security labels Gluing dots Kanban- cards Booklet labels PIN covering Type labels Decoration labels Hologram labels Stamped parts Safemark labels

Documents
Online Job Scheduling Mechanisms: How Bad are Restarts?

Online Job Scheduling Mechanisms: How Bad are Restarts?

Documents
TABLE OF CONTENTS— PLANT LABELS & TAGS · 2018-07-16 · TABLE OF CONTENTS— PLANT LABELS & TAGS PLANT LABELS & TAGS Thin Plastic Labels Heavy Plastic Labels Plastic Computer Labels

TABLE OF CONTENTS— PLANT LABELS & TAGS · 2018-07-16 · TABLE OF CONTENTS— PLANT LABELS & TAGS PLANT LABELS & TAGS Thin Plastic Labels Heavy Plastic Labels Plastic Computer Labels

Documents
Nutrition Labels. 0H1XWA 0H1XWA Brian Regan - Labels

Nutrition Labels. 0H1XWA 0H1XWA Brian Regan - Labels

Documents
Restarts - Montana Ref · Restarts Restart Chart Name of Restart: Abbreviation: Cause of Restart: 2-Touch? Who Performs? Score Directly? Where Located? When in Play? Restrictions

Restarts - Montana Ref · Restarts Restart Chart Name of Restart: Abbreviation: Cause of Restart: 2-Touch? Who Performs? Score Directly? Where Located? When in Play? Restrictions

Documents
Branching Strategies and Restarts in SAT Solvers Ashish Sabharwal

Branching Strategies and Restarts in SAT Solvers Ashish Sabharwal

Documents