Upload
spring-io
View
1.600
Download
0
Embed Size (px)
Citation preview
1 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Building Scalable Data Applications with Spring and GemFire
Luke Shannon, Field Engineer - GemFire, Pivotal
2 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
High Level Session Topics
Short GemFire Overview
GemFire Demo
Developing Server Side Code
Building Test Cases
Developing Client Side Code
3 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
What Is GemFire?
4 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
What is GemFire?
In-Memory Data Grid
• Uses JVM Heap
• Options for disk storage
Key/Value Store
• Objects or JSON
• Query Language
• Parallel Function Execution
Shared Nothing Architecture
• Data in each node independent from other members
Multiple Site Distribution
• Clusters distribute data across a WAN
5 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Why Use It?
Highly Scalability
• Partitioned data
• Light weight process
High Read/Write Through Put
• Asynchronous replication of data
• Multiple copies of data across partitions
• Single hop access to data
Rich Event Framework
• Includes push notifications
Group Membership
• Maintains consistency of data across members
• Maintains membership
6 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Why Use It? - continued
Great integration with Spring
• Spring Data (Spring Data Gemfire)
• Spring Boot
• Spring XD
Our focus for today!
Spring
Data
Spring
XDSpring
Boot
7 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Who is Using It?
8 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Who is Using It?
“This isn’t the online mass-scale case study that Pivotal has
for GemFire – GE Research is using GemFire to handle 17
billion individual records in-memory. Another railway
organization, China Railways, has 3TB of operational data
in-memory and a further 400 TB archived. China Rail’s size
challenges even India Rail’s – they process 4.6 million
transactions per day and up to 40,000 per second. Finally
GIRE, a payment processing organization is running 19
million payment transactions a month with over 4000
concurrent clients connecting via intermittent internet
connections.”
http://www.forbes.com/sites/benkepes/2015/01/05/pivotal-helps-india-railways-keep-on-moving-and-selling-tickets/
9 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Where to get GemFire?
https://network.pivotal.io/products/pivotal-gemfire
10 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
What do you get?
11 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
What do you get?
12 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
What do you get?
13 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Working with GemFire
GFSH
• GemFire Shell
• Uses the GemFire API
• Start, Stop, Manage Cluster and Members
14 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
GemFire Simplified
JVM
Cache
Region(s)
Logic Data
15 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
GemFire Terminology
Cache
• Single Instance across the Cluster
Region
• Implementation of the java.util.HashMap
• Two Storage Patterns
• Partitioned
• Replicated
• Lots of setting and configurations
• Eviction
• Expiration
• Read Through
• Write Through
• Disk Persistence
• etc
16 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
A Common GemFire Deployment Cluster
Locators
Servers
Clients
Cache
Cache
Cache
TCP/IP
Clients connect
directly to data
servers
Optional
Back End
Apps contains
GF Caches and
Regions
Read Through,
Write Through,
Persistence
17 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Demo
Unless otherwise indicated, these sl ides are
© 2013-2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense:
http://creativecommons.org/l icenses/by -nc/3.0/
Short Gemfire Overview
18 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Webinar Work Flow
Configure the Cluster
Code the Model
Code the Repositories
Create Test Cases
Create Client Application
Review a Function
19 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Today’s Lab Environment - Cluster
6 Gigebyte Brix
• Ubuntu Server 14.04.1 LTS 64 bit
• 8 GB of RAM
• Oracle JDK 7
• GemFire 8.0
• Cisco Router
• Cisco Switch
20 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Today’s Lab Environment – Developer + Client
Mac OSX
• Oracle JDK 8
• STS 3.6.3
• Ansible 1.8
• Cluster Related Tasks:
• Distribute files
• Collect files
• Execute gfsh commands
• Maven
21 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Configure The Server
22 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Start Up Lifecycle For Spring Configured GF Server
GemFireProcess
• Started with gfsh
• Creates Cache
• Bootstraps Spring
Spring Context
• Create Regions
• Create server side logic
• Add to the Cache
23 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Future Start Up Process
Spring Context
• Started with gfsh
• Creates Cache
• Create Cache Components
24 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Work Flow Today
Create GemFire cache.xml
• It references the Spring cache config
Create Spring Data GemFire applicationContext.xml
• Why not use annotations?
• Does not have full support for all GemFire options
• Editor in STS provides assistance
25 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Demo
Unless otherwise indicated, these sl ides are
© 2013-2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense:
http://creativecommons.org/l icenses/by -nc/3.0/
Configure Server and Deploy Code
26 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Coding The Model
27 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Basic Model
Customers
• Name
• DOB
• ID (this will also act as the key)
Orders
• NameOfProduct (just a String value to keep it simple)
• customerId
• ID (this will also act as the key)
28 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
A Word On Keys
Can be any class
• Strings and Integers are fine
Remain in memory
• Eviction/Expiration affect values only
Inexpensive to iterate keys
• Key can wrap meta data
• Processing can occur at key without object
29 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
A Word On Object Models
GemFire supports object graphs
• Can make queries more complex
• Memory efficiency
• Advantages to referencing ID(s)
Data stored in serialized form
• Default Serialization
• Extension Point for Custom Serialization
• Out of scope for today
Objects can be co-located in partitions
• Out of scope for today Sizing Guidelines:
1. Want redundant copies (1+)
2. Room for GC
3. Room for indexes
30 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Getting Gemfire using Maven
31 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Demo
Unless otherwise indicated, these sl ides are
© 2013-2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense:
http://creativecommons.org/l icenses/by -nc/3.0/
Creating the Object Model
32 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Adding Repositories
33 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Why Repositories
Spring Data Repository
• “The goal of Spring Data repository abstraction is to significantly reduce the
amount of boilerplate code required to implement data access layers for various
persistence stores”
Spring Data GemFire
• “Spring Data GemFire provides support to use the Spring Data repository
abstraction to easily persist entities into GemFire and execute queries”
Gemfire template is also an option
• Provides access to several options:
• CRUD
• Calling Functions
• Query Service
• More knowledge of Gemfire required to use
34 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
What is OQL?
Object Query Language
• GemFire Query Language
• Query any object
• Navigate object collections
• Invoke methods of object attributes
Examples:
• SELECT * FROM /myRegion WHERE sp.containsAll(set('20', '21', '22'))
• SELECT DISTINCT * FROM /myRegion p WHERE p.name.startsWith('Bo’)
• SELECT * FROM /myRegion.keys k WHERE k.ID = 1
• SELECT COUNT(*) FROM /myRegion WHERE ID > 0 LIMIT 50
35 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Registering Repositories
Register the package that contains the Repo(s)
• <gfe-data:repositories base-package="com.acme.repository" />
Extend GemfireRepository <Key Class, Value Class>
36 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Demo
Unless otherwise indicated, these sl ides are
© 2013-2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense:
http://creativecommons.org/l icenses/by -nc/3.0/
Coding and Deploying the Repositories
37 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Creating Test Cases
38 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Testing GemFire
True distributed testing is difficult
• Not covered in this session
Logic can be tested in a single Gemfire node cluster
39 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Creating the Test Cache – Using API
• Good is working directly with the Cache or Regions
40 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Creating the Test Cache – Using Annotations
1. Add the following Annotations to the test case:
2. Add Spring Data Gemfire cache definition to test classpath
41 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Demo
Unless otherwise indicated, these sl ides are
© 2013-2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense:
http://creativecommons.org/l icenses/by -nc/3.0/
Creating Test Cases
42 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Creating a Simple Function
43 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
GemFire Functions
Execute in parallel across the grid
• Many configurations
• Beyond the scope today
Client aggregates results
Spring Data Gemfire provides:
• Simple POJO based function logic
• Annotation support for:
• Registering
• Executing
44 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Registering Functions with Gemfire
Enable Annotation Support
• <gfe:annotation driven/>
Load the bean:
• Register it as a bean
• Context:Component-scan in xml
45 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Function Invocation
Interface supported by annotations
• Spring creates a dynamic proxy
• Creates Gemfire FunctionService to execute the function
46 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Demo
Unless otherwise indicated, these sl ides are
© 2013-2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense:
http://creativecommons.org/l icenses/by -nc/3.0/
Creating and Deploying A Function
47 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Creating a Client
48 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Spring Boot to the Rescue
Spring Boot makes it easy to:
• Create a new Gemfire based project
• Manages all the dependencies
Application class is easy configuration point to wire in client
49 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Client Cache
50 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Demo
Unless otherwise indicated, these sl ides are
© 2013-2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense:
http://creativecommons.org/l icenses/by -nc/3.0/
Create a Simple Web Client
51 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Spring XD
52 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Spring XD and Gemfire
Sources
• Gemfire Source
• Uses CacheListener to get entries from Gemfire
• Gemfire Continuous Query
• Register an OQL to determine which objects are sent by Gemfire
Sink
• Allows for a Stream to write to a Region
• No support for Locators
• Writes directly to a cache server
53 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Wrapping Up
This has just scratched the surface
• Integration with Spring Transactions
• Read Through
• Write Through
• Client Notifications
• And more…
54 Unless otherwise indicated, these sl ides are © 2013 -2014 Pivotal Software, Inc. and l icensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/l icenses/by -nc/3.0/
Further Reading
Gemfire Docs:
• http://gemfire.docs.pivotal.io/latest/userguide/index.html#about_users_guide.html
Spring Data Gemfire Docs:
• http://docs.spring.io/spring-data-gemfire/docs/current/reference/html/
Spring XD Doc:
• http://docs.spring.io/spring-xd/docs/1.0.3.RELEASE/reference/html/#gemfire-
source