Upload
horacio-gonzalez
View
243
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Our presentation at BreizhCamp 2013
Citation preview
Le développeur de l'Est le plus à l'Ouest !Java & C++ Developer, coding addict,
continuous intégration ayatollah
● Senior Software Engineer currently, Crédit Mutuel Arkea
● FinistJug actif member
BreizhBeans community and commiter
[email protected] +sebastien.lambour
@FinistSeb
Sebastien Lambour
Horacio GonzalezSpaniard lost in Brittany, Java developer,
dreamer and all-around geek
● Senior Software Engineer at Crédit Mutuel Arkea○ Currently...
● Leader of the FinistJUG and the BreizhBeans community
http://lostinbrittany.org/ +Horacio.Gonzalez @LostInBrittany
BreizhBeans
Developer community at the far end of French Brittany
http://breizhbeans.org/ @BreizhBeans
I help to create a startup on social translation, would
you code with me ?
Great, I do the backend! How many members?
Some 30.000 - 50.000 will be fine!
Ouch, is it not too small? I like big toys!!!!
Me too!Let's think big then
OK, it will rock !
Nexus 4
Why ?
Backends They often look like that !
Transforming it is HARD !
Next challenge :Granite aircraft !!
A sexy backend?Let's groom it!
OK, go around...
Light weight Test driven
ScalableCloud Ready
Java
Data Driven
Following the beaten trackBrainless route
The usual suspects...
JSF 2.0Hibernate
JPACXF
SQL
Spring CRUD(Create, Read, Update,Delete)
Booooooring...And ineffective
Let's use it!
Design choicesBecause we need some solid roots
Core principlesWhat do we want our backend to be?
● Scalability ○ Because traffic growth should be a blessing, not a problem!
● Adaptability ○ Because your business evolves, and your backend need to evolve
with it!
● Testability○ Because you want to be sure that your code works!
● NoSQL (meaning Not Only SQL)○ Because you need not only tables and foreign keys, but non-SQL
stores, search indexes, caches...
● Productivity○ Because the other 4 doesn't matter if you need to spend 4
working days to modify a service...
A first choice : Apache Thrift
« Software framework for scalable and performant cross-language services development »
● Interface description language● Multi-language code generation● Serialisation/deserialisation● RPC Framework● High performance
○ On transport layer○ On data manipulation
Thrift, it's cool Thrift boots productivity!
A second choice : MongoDB
● We needed a NO-SQL store for documents
● We wanted to use Cassandra○ Cassandra & Thrift, a love story
● We wanted a PaaS-ready backend○ Neither Cloudbees nor Cloudfoundry supported Cassandra○ But they supported MongoDB
● We loved MongoDB○ Document DB are great for document oriented applications
ThriftMongoBridgeBecause we wanted both
Thrift and MongoDB
Thrift rocks and works
But...
How to put Thrift objects into MongoDB ?
First bad idea!
Use Spring Data or Morphia
Thrift object mapping is bad !
Damned!
First bad idea!
Second bad idea !
Use the Mongo Driver JSON parsing
Deserialisation will be a bit hard...
Damned !
Second bad idea!
Thrift isn't cool anymore ?
We need a new idea !
Thrift, it's cool !But we had work to do...
Write a TBSONProtocol !
The ThriftMongoBridge is born !
What does it do ?
struct AnotherThrift {1:string anotherString,2:i32
anotherInteger,}
Thrift IDL
POJO generated by the Thrift compiler
public class AnotherThrift implements org.apache.thrift.TBase... { ... public String anotherString; public int anotherInteger;
... public static final Map<_Fields, org.apache.thrift.meta_data.FieldMetaData> metaDataMap; .... public void read(org.apache.thrift.protocol.TProtocol iprot)... ... public void write(org.apache.thrift.protocol.TProtocol oprot) ......;}
Compile to
{anotherString:"value", anotherInteger:69}
Write To BSON Read from BSON
TBSonSerializerTBSonProtocol TBSonDeSerializer
Really simple to use Too simple ?
AnotherThrift anotherThrift = new AnotherThrift();
anotherThrift.setAnotherString("value");
anotherThrift.setAnotherInteger(69);
// serialize into DBObject
DBObject dbObject = ThriftMongoHelper.thrift2DBObject(anotherThrift);
{anotherString:"value", anotherInteger:69}
How it works
DBObject (BSON)
TBSONProtocol
TBaseTBSONSerializer
write(protocol)
getResult
Context Stack
sentense directives (begin / end)
Data (read / write)
Struct contextDBObject
push pop
Thrift object generated
MongoDB object
The BreizhCamp demoBecause you want to see some code,
don't you?
The BreizhCamp demoGiven this Thrift model & service
struct BreizhCampEvent { 1:i64 date, 2:string startTime, 3:string endTime, 4:com.breizhbeans.backend.thrift.enums.model.EventTypes type, 5:string room, 6:string resume, 7:string details, 8:list<string> speakers 9:list<com.breizhbeans.backend.thrift.enums.model.Track> tracks,}
service ProgramService {
com.....GetProgramResponse getProgram(1:com.....GetProgramRequest request)
throws (1:com....TechnicalException texp, 2:com...FunctionalException fexp)
}
struct GetProgramRequest {1:com.breizhbeans.backend.thrift.enums.model.Track track,2:com.breizhbeans.backend.thrift.enums.model.EventTypes type,3:string room,4:list<string> speakers,
}struct GetProgramResponse {
1:list<com.breizhbeans.backend.thrift.model.BreizhCampEvent> events,}
The BreizhCamp demoFirst : Write a test
● One loader by collection● One goal : Have Human readable test ;-)● Manage automatically data insertion before test and
clean up after test @Test
public void testGetEvents() throws Exception {
List<BreizhCampEvent> actualEvents = programDao.getEvent();
List<BreizhCampEvent> expectedEvents = new ArrayList<BreizhCampEvent>();
expectedEvents.add(getEventPimpMyBackend());
Assert.assertEquals(expectedEvents, actualEvents);
}
Application code to test
Standard JUnit assertion
Expected data building(used by the test and the data loader)
The BreizhCamp demoWrite a DAO
public List<BreizhCampEvent> getEvent() throws TechnicalException {
DBCursor cursor = null;
try {
List<BreizhCampEvent> events = new ArrayList<BreizhCampEvent>();
DBCollection collection = db.getCollection(collectionName);
DBObject query = new BasicDBObject();
// execute the query
cursor = collection.find(query);
while(cursor.hasNext()) {
events.add( (BreizhCampEvent) ThriftMongoHelper.DBObject2Thrift(cursor.next()));
}
return events;
} catch (Exception e) {
throw new H4TTechnicalException("Unexpected error", e, null);
} finally {
cursor.close();
}
}
retrieve DB collection
Build and execute the Query
Deserialize Thrift objects
Error handling
Never forget it ;-)
@Override
public BreizhCampEvent updateRoom(final BreizhCampEvent event) throws TechnicalException {
try {
BasicDBObject updateQuery = new BasicDBObject();
updateQuery.append("$set", new BasicDBObject()
.append(getFields(BreizhCampEvent._Fields.ROOM), event.getRoom()));
db.getCollection(collectionName)
.update(getPrimaryKeyRequest(event), updateQuery)
.getLastError()
.throwOnError();
return event;
} catch (Exception e) {
throw new H4TTechnicalException("Unexpected error", e, ObjectUtils.toString(event));
}
}
The BreizhCamp demoCan I update thrift Object by code?
Mongo Update Query
Thrift structure mapping
Query execution
Error handling
The BreizhCamp demoCan I update thrift Object by script?
Mongo DB Bson writed : > db.program.find();
{ "_id" : "id=10", "date" : NumberLong("-56433978000000"), "startTime" : "14:45", "endTime" : "15:40", "type" : 4, "room" : "Bréhat", "resume" : "Pimp my backend!", "details" : "Le frontend, le frontend...... ...", "speakers" : [ "Sébastien Lambour", "Horacio Gonzalez" ], "tracks" : [ 7 ], "id" : 10, "classname" : "com.breizhbeans.backend.thrift.model.BreizhCampEvent" }
Update the field room :> db.program.update();
{ "id": 10}, { $set: { "room": "Ouessant" } }
And the perfs?Because speed matters!
Talking about perfsThe candidates are ...
Morphia
Spring Data
And... ThriftMongoBridge, of course
Spring Data vsThriftMongoBridge
● Write and Read 500 objects into MongoDB.
● Each document is composed with a List<> and a Set<> of 500 objects.
● The document have a JSON size of 91613 bytes
Who will win ?
CodeThriftMongoBridge
First// Get a collection
DBCollection collection = db.getCollection("dummyColl");
Write// Serialize the Thrift object
DBObject dbObject = ThriftMongoHelper.thrift2DBObject(thriftObject);
// Put the document
collection.insert(dbObject);
ReadDBCursor cursorDoc = collection.find();
while (cursorDoc.hasNext()) {
DBObject dbObject = cursorDoc.next();
ThriftObject thirftObject = (ThriftObject)ThriftMongoHelper.DBObject2Thrift(dbObject);
}
CodeSpring Data
First// Retrieve the MongoOperations bean
MongoOperations mongoOperation =
(MongoOperations) ctx.getBean("mongoTemplate");
Write// Serialize the Thrift object
mongoOperation.save(springObject, "springObject");
ReadList<SpringObject> listUser =
mongoOperation.findAll(SpringObject.class, "springObject");
OK. Spring Data is (a little bit) more compact...
Time to run !
Thrift over MongoDB rocks!
We need a logo for our OpenSource library...
Available now :
2.0
https://github.com/BreizhBeans/ThriftTools
A storage library a backend is not,young padawan
Because we need more things
At the beginning...Pretty layers, neatly isolated, like a Care Bears' birthday cake
Later in the projectLike a cake after an earthquake
And in the end...As pretty as a British pudding... and almost as tasty
Be Layer lessNow build the Backend !
● Each package have a goal
● NO package assembly rules
● Improve the complexity detection by dependencies analysis
Goal : Fight against the "God classes" anti-pattern
unreachable dream
Prevent protocol rupture, have a stable services signature.
com.h4t.CheckAccessResponse checkAccess (1:com.h4t.CheckAccessRequest request) throws (1:com.h4t.TechnicalException texp, 2:com.h4t.FunctionalException fexp)
stable method signature
Thrift services
Integrate this into the backend
Thrift IFace implementation
Request / Response decapsulation + log & monitoring
Core Business code
MongoOpérations
Thrift object
ThriftMongoHelper
Mongo Driver
DBObject
write read
Query
DBObject
Inversion of control
Keep simple too !
Prefer static dependencies injection.
Time to do the choice
or
Inversion of control
Why ?
Because and
○ binding of the MongoDB service by the cloud○ no production configuration required○ only one war
TestabilityBecause without tests
you need faith
Test strategy
No compromise!The rules are simple... no exception!
TDD is my religion1. Execute test without setup2. Execute test with your build tool3. Execute test into your IDE4. No injection into test framework5. Reuse only application injection6. Keep it simple (repeat it 7 times)7. Have a loose coupling (test the smallest parts)8. Test must be deterministic and reproducible9. Do pair review with the tests
10. If the complexity is growing up refactor now...
11. If a rule goes wrong, rewrite it !
From concept to reality
ThriftMongoTestToolsThrift DataSet
Collection Loader
Load thrift object with preset datasinto MongoDB
Get expected Thift object with preset datas
Get a Thrift object from MongoDB related to a unique key
DAO : Validate queries and set up the test data loaders
IFACE : Functional tests
SERVICE : Test a business centric service
TU : Unit test not related to the business (test first approach)
Tests have goals
@Before
public void setUp() throws Exception {
eventLoader.load(getEventPimpMyBackend());
}
private BreizhCampEvent getEventPimpMyBackend() throws Exception {
String details = "Le frontend, le frontend... .....";
BreizhCampEvent event = eventLoader.getExpected(10, "14/06/2013", "14:45", "15:40", EventTypes.CONF, "Bréhat", "Pimp my backend!", details);
event = eventLoader.addSpeakers(event, "Sébastien Lambour", "Horacio Gonzalez");
event = eventLoader.addTracks(event, Track.CLOUD);
return event;
}
Tests are Data Driven
Build test Data with java code, files, BDDor the MongoThrift Test tools (not opensourced yet ;-) )
Inject Data into MongoDB
What's next?Because our work is not completed... yet!
Why integrate our lib ?
#Pragmatism● Productivity oriented● Fully testable ● Container less (no servlet dependencies)● Open Source : Apache 2.0
The ThriftMongoBackend Core model will born soon !
With a name: TINAF (ThriftMongoBackend Is Not A Framework)
What's next ?
● monitoring Thrift services in the cloud
● load tests with vmware & 10gen
● contribute TINAF
● Develop a thrift service valve to ensure brandwitdth for critical services
... do a talk at Devoxx BE
Thank you !