55
1 Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Building Scalable Data Applications with Spring and GemFire Luke Shannon , Field Engineer - GemFire , Pivotal

Building Scalable Web Applications with Spring Data Gemfire

Embed Size (px)

Citation preview

1 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Building Scalable Data Applications with Spring and GemFire

Luke Shannon, Field Engineer - GemFire, Pivotal

2 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

High Level Session Topics

Short GemFire Overview

GemFire Demo

Developing Server Side Code

Building Test Cases

Developing Client Side Code

3 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

What Is GemFire?

4 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

What is GemFire?

In-Memory Data Grid

• Uses JVM Heap

• Options for disk storage

Key/Value Store

• Objects or JSON

• Query Language

• Parallel Function Execution

Shared Nothing Architecture

• Data in each node independent from other members

Multiple Site Distribution

• Clusters distribute data across a WAN

5 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Why Use It?

Highly Scalability

• Partitioned data

• Light weight process

High Read/Write Through Put

• Asynchronous replication of data

• Multiple copies of data across partitions

• Single hop access to data

Rich Event Framework

• Includes push notifications

Group Membership

• Maintains consistency of data across members

• Maintains membership

6 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Why Use It? - continued

Great integration with Spring

• Spring Data (Spring Data Gemfire)

• Spring Boot

• Spring XD

Our focus for today!

Spring

Data

Spring

XDSpring

Boot

7 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Who is Using It?

8 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Who is Using It?

“This isn’t the online mass-scale case study that Pivotal has

for GemFire – GE Research is using GemFire to handle 17

billion individual records in-memory. Another railway

organization, China Railways, has 3TB of operational data

in-memory and a further 400 TB archived. China Rail’s size

challenges even India Rail’s – they process 4.6 million

transactions per day and up to 40,000 per second. Finally

GIRE, a payment processing organization is running 19

million payment transactions a month with over 4000

concurrent clients connecting via intermittent internet

connections.”

http://www.forbes.com/sites/benkepes/2015/01/05/pivotal-helps-india-railways-keep-on-moving-and-selling-tickets/

9 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Where to get GemFire?

https://network.pivotal.io/products/pivotal-gemfire

10 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

What do you get?

11 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

What do you get?

12 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

What do you get?

13 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Working with GemFire

GFSH

• GemFire Shell

• Uses the GemFire API

• Start, Stop, Manage Cluster and Members

14 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

GemFire Simplified

JVM

Cache

Region(s)

Logic Data

15 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

GemFire Terminology

Cache

• Single Instance across the Cluster

Region

• Implementation of the java.util.HashMap

• Two Storage Patterns

• Partitioned

• Replicated

• Lots of setting and configurations

• Eviction

• Expiration

• Read Through

• Write Through

• Disk Persistence

• etc

16 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

A Common GemFire Deployment Cluster

Locators

Servers

Clients

Cache

Cache

Cache

TCP/IP

Clients connect

directly to data

servers

Optional

Back End

Apps contains

GF Caches and

Regions

Read Through,

Write Through,

Persistence

17 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Demo

Unless otherwise indicated, these sl ides are

© 2013-2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense:

http://creativecommons.org/l icenses/by -nc/3.0/

Short Gemfire Overview

18 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Webinar Work Flow

Configure the Cluster

Code the Model

Code the Repositories

Create Test Cases

Create Client Application

Review a Function

19 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Today’s Lab Environment - Cluster

6 Gigebyte Brix

• Ubuntu Server 14.04.1 LTS 64 bit

• 8 GB of RAM

• Oracle JDK 7

• GemFire 8.0

• Cisco Router

• Cisco Switch

20 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Today’s Lab Environment – Developer + Client

Mac OSX

• Oracle JDK 8

• STS 3.6.3

• Ansible 1.8

• Cluster Related Tasks:

• Distribute files

• Collect files

• Execute gfsh commands

• Maven

21 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Configure The Server

22 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Start Up Lifecycle For Spring Configured GF Server

GemFireProcess

• Started with gfsh

• Creates Cache

• Bootstraps Spring

Spring Context

• Create Regions

• Create server side logic

• Add to the Cache

23 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Future Start Up Process

Spring Context

• Started with gfsh

• Creates Cache

• Create Cache Components

24 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Work Flow Today

Create GemFire cache.xml

• It references the Spring cache config

Create Spring Data GemFire applicationContext.xml

• Why not use annotations?

• Does not have full support for all GemFire options

• Editor in STS provides assistance

25 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Demo

Unless otherwise indicated, these sl ides are

© 2013-2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense:

http://creativecommons.org/l icenses/by -nc/3.0/

Configure Server and Deploy Code

26 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Coding The Model

27 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Basic Model

Customers

• Name

• DOB

• ID (this will also act as the key)

Orders

• NameOfProduct (just a String value to keep it simple)

• customerId

• ID (this will also act as the key)

28 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

A Word On Keys

Can be any class

• Strings and Integers are fine

Remain in memory

• Eviction/Expiration affect values only

Inexpensive to iterate keys

• Key can wrap meta data

• Processing can occur at key without object

29 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

A Word On Object Models

GemFire supports object graphs

• Can make queries more complex

• Memory efficiency

• Advantages to referencing ID(s)

Data stored in serialized form

• Default Serialization

• Extension Point for Custom Serialization

• Out of scope for today

Objects can be co-located in partitions

• Out of scope for today Sizing Guidelines:

1. Want redundant copies (1+)

2. Room for GC

3. Room for indexes

30 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Getting Gemfire using Maven

31 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Demo

Unless otherwise indicated, these sl ides are

© 2013-2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense:

http://creativecommons.org/l icenses/by -nc/3.0/

Creating the Object Model

32 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Adding Repositories

33 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Why Repositories

Spring Data Repository

• “The goal of Spring Data repository abstraction is to significantly reduce the

amount of boilerplate code required to implement data access layers for various

persistence stores”

Spring Data GemFire

• “Spring Data GemFire provides support to use the Spring Data repository

abstraction to easily persist entities into GemFire and execute queries”

Gemfire template is also an option

• Provides access to several options:

• CRUD

• Calling Functions

• Query Service

• More knowledge of Gemfire required to use

34 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

What is OQL?

Object Query Language

• GemFire Query Language

• Query any object

• Navigate object collections

• Invoke methods of object attributes

Examples:

• SELECT * FROM /myRegion WHERE sp.containsAll(set('20', '21', '22'))

• SELECT DISTINCT * FROM /myRegion p WHERE p.name.startsWith('Bo’)

• SELECT * FROM /myRegion.keys k WHERE k.ID = 1

• SELECT COUNT(*) FROM /myRegion WHERE ID > 0 LIMIT 50

35 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Registering Repositories

Register the package that contains the Repo(s)

• <gfe-data:repositories base-package="com.acme.repository" />

Extend GemfireRepository <Key Class, Value Class>

36 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Demo

Unless otherwise indicated, these sl ides are

© 2013-2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense:

http://creativecommons.org/l icenses/by -nc/3.0/

Coding and Deploying the Repositories

37 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Creating Test Cases

38 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Testing GemFire

True distributed testing is difficult

• Not covered in this session

Logic can be tested in a single Gemfire node cluster

39 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Creating the Test Cache – Using API

• Good is working directly with the Cache or Regions

40 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Creating the Test Cache – Using Annotations

1. Add the following Annotations to the test case:

2. Add Spring Data Gemfire cache definition to test classpath

41 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Demo

Unless otherwise indicated, these sl ides are

© 2013-2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense:

http://creativecommons.org/l icenses/by -nc/3.0/

Creating Test Cases

42 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Creating a Simple Function

43 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

GemFire Functions

Execute in parallel across the grid

• Many configurations

• Beyond the scope today

Client aggregates results

Spring Data Gemfire provides:

• Simple POJO based function logic

• Annotation support for:

• Registering

• Executing

44 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Registering Functions with Gemfire

Enable Annotation Support

• <gfe:annotation driven/>

Load the bean:

• Register it as a bean

• Context:Component-scan in xml

45 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Function Invocation

Interface supported by annotations

• Spring creates a dynamic proxy

• Creates Gemfire FunctionService to execute the function

46 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Demo

Unless otherwise indicated, these sl ides are

© 2013-2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense:

http://creativecommons.org/l icenses/by -nc/3.0/

Creating and Deploying A Function

47 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Creating a Client

48 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Spring Boot to the Rescue

Spring Boot makes it easy to:

• Create a new Gemfire based project

• Manages all the dependencies

Application class is easy configuration point to wire in client

49 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Client Cache

50 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Demo

Unless otherwise indicated, these sl ides are

© 2013-2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense:

http://creativecommons.org/l icenses/by -nc/3.0/

Create a Simple Web Client

51 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Spring XD

52 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Spring XD and Gemfire

Sources

• Gemfire Source

• Uses CacheListener to get entries from Gemfire

• Gemfire Continuous Query

• Register an OQL to determine which objects are sent by Gemfire

Sink

• Allows for a Stream to write to a Region

• No support for Locators

• Writes directly to a cache server

53 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Wrapping Up

This has just scratched the surface

• Integration with Spring Transactions

• Read Through

• Write Through

• Client Notifications

• And more…

54 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Further Reading

Gemfire Docs:

• http://gemfire.docs.pivotal.io/latest/userguide/index.html#about_users_guide.html

Spring Data Gemfire Docs:

• http://docs.spring.io/spring-data-gemfire/docs/current/reference/html/

Spring XD Doc:

• http://docs.spring.io/spring-xd/docs/1.0.3.RELEASE/reference/html/#gemfire-

source

55 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/

Thank You!