If you can't read please download the document
Upload
couchbase
View
181
Download
0
Embed Size (px)
Citation preview
Rapid Data Modeling and TestingAARON BENTON
bit.ly/cbcommunity
Access to Dev Team and other Experts and Champions Brant Burnett (CenterEdge) / Dave Starling (SeenIt)Early Access to BuildsLaura Czajkowski
About MeWho we areWhat we do
+
We use Couchbase and Sync Gateway for a variety of solutionsIncluding: Several mobile applications, media, short urls, session management, looking for more and more use casesWhile Couchbase is a fantastic database, choosing NoSQL presents a new set of challenges
We choose Couchbase for many reasons: Performance, Scalability, all of the reasons that many of you chose CouchbaseIn particular Mobile was a big use caseBut this introduced a whole other set of issues, how do we represent data?
OUR CAPABILITIES
Make fun of different workflows, regardless of flow we still have to deal with data
When you get a new project, what is the first thing your business owners want you to do?
Given ProjectStart DevelopmentWrite Code to Fit DataGather RequirementsRestart Development
EnthusiasmDisillusionmentPanicSearch for the GuiltyPunishment of the Innocent Praise for Non-Participants
Lighten mood with The Six Phases of a ProjectAnyway, moving on data modeling is a critical piece of any project
First, solve the problem. Then, write the code.
John Johnson
Data modeling is a problem that needs to be solved
Data Modeling
Who is data modeling now? What phases? What tools? How do you go about it? Where is it in your process?Checkout the session: Agile document models & data structures with Clarence and Tyler from Couchbase
PhasesConceptual Logical Physical
Conceptual Data ModelingEntity NamesEntity Relationships
Users
Products
Line Items
Orders
Entity NamesEntity RelationshipsDatabase Agnostic
N1PersonBirthplaceLocation
PersonLocation
Born InBirthplace of
PersonLocation
(1, 1) Born InBirthplace of (0, N)
>Person
0. .N1>Location>Born in >< Birthplace ofChen
PersonLocation
IDEF1x
PersonLocation
Born InBirthplace ofBachmanCrow's FootISOUML
Modeling Notations
Logical Data ModelingEntity NamesEntity AttributesEntity RelationshipsUsersUser ID (PK)UsernamePasswordFirst NameLast NameAddressCityStateZip CodeCountryEmailPhone
OrdersOrder ID (PK)Order NumberUser ID (FK)Order DateAddressCityStateZip CodeCountryPhoneEmailCredit CardExpiration Date
Order Line ItemsOrder ID (FK)Product ID (FK)QuantityTotal
ProductsProduct ID (PK)Product NameDescriptionPrice
AttributesPrimary / Foreign Keys: Database Agnostic
Physical Data ModelingEntity Names -> Table NamesEntity Attributes -> Field NamesEntity Relationships -> KeysPrimary / Foreign KeysData Types
NoSql ModelingHow do you represent a model?What tools do you use?How do you show relationships?Does code define the model?What key patterns should be used?
JSON
Our physical models initially were JSONThis worked but was lackingNo ContextNo DescriptionNo Rules
JSON Schema
We also explored JSON Schema
YAML
We moved on to YAMLThis provided:clearer structureless verbosedocumentationdata typingWe stored these models with the code in the repository
Now What?
You've defined your physical NoSQL models, now what?
Starting Development! KiddingWe need data to work withHow many of you have spent time building throw away code, just to have data to work with your applications?
To begin development often times we need a data set to work withWe need pretend or fake data to work with
Manually creating documents is very time consuming and not practical
Fake Data GeneratorsFakerJSChanceJSjson-generator.comobjgen.com/jsonmockaroo.comjsonschema.net
There are a lot of Fake data generators available, some are actually very flexible, but they all only deal with a single model.
Often times we need to relate models
Relationships are Hard
All of these tools only deal with a single a model Or than can only generate data and you still have to write code to use themRelationships are hard
FakeIT
Just FakeIt, now all your problems are solved.
Any fancy logo makers?
FakeitCLI Tool inspired by SwaggerModels defined in YAML Generates JSON, CSON, YAML, CSVExports to Couchbase, Sync Gateway, Zip, Console
Like many of you I've spent countless hours writing code to generate fake dataI'm lazyI wanted to learn N1QL
Fakeit DataExposes FakerJS and ChanceJSCustom Build Functions w/ Pre and PostMultiple ModelsDependenciesDefinitions*fakeit is for structure, cbworkloadgen is for sizing
npm install fakeit -g
If you have NodeJS install it now
Original YAML Model
Our original YAML model
Simple Model
Data Attribute
A "data" attribute is added to each property"description" has been removed just to fit code on the screen
Console Output
- From the directory of the YAML file we run the fakeit command- Which by default will generate 1 document from the model- We can use the -n parameter to specify the number of documents to generate from the model
File Output
We can output each document to an individual file, where the file name is the keyIf the directory does not exist it is created
Output Formats
Using the -o parameter we can specify the file format to use, the default is JSONYAML and CSON are also supported
CSV Output
Another output format that can be used is csvEven if you were using a SQL backend, you can still generate random data and import into that system
Zip Archive
We can take any of these output formats and flush them into a Zip archive using the -a parameterThis Zip file could be imported into Couchbase through cbdocloaderThis is using the default output format of JSON
. . . ?
What about Couchbase?Of course there is Couchbase support, would I be up hear talking about this if there wasn't? :)
Couchbase
We could take the exported Zip archive and import it into Couchbase using cbdocloaderBut wouldn't it be better to stream the generated documents directly into Couchbase?
What about Mobile?
Sync Gateway
We can stream generate documents through the Sync GatewayThis is using Sync Gateway without any authentication
Sync Gateway with Auth
In this example we are using a Sync Gateway that requires authWe specify the username using the -u parameter and the password using the -p parameterA sync session is created prior to the documents being loaded
How does it Work?
Property Data:
valueA static value to be usedExecuted at build time
fakeA FakerJS template stringExecuted at build time
pre_buildFunction body used to initialize a propertyProperty is set to the returned value
buildFunction body used to set a propertyProperty is set to the returned value
post_buildFunction body used to manipulate a property after it has been setProperty is set to the returned value
pre_runFunction body that is executed before any models are generated
post_runFunction body that is executed before after all models have been generated
Data can be generated using any of these methods
You are defining a function bodyDescribe what each of the available variables are
Flow
Model Pre_RUN
Model PRE_BUILD
Initialize Property Value(EACH)
Start Document Run
Property PRE_BUILD + BUILD(EACH)
PROPERTY POST_BUILD (EACH)
MODEL POST_BUILD
MODEL POST_RUN
Define pre / post run
Flow
Build order example using console.logs to show the flow
Flow
Build order example using console.logs to show the flow
Multiple Models
Multiple models in the same directory can be executed together
Multiple Models
Dependencies
UsersProductsReviewsOrders
More often than not our models have dependencies on other models
Dependencies
We specify dependencies on Models
FakeIt resolves the dependencies prior to generation, making the data from previous models available to the current one being generated
Dependencies
By default fakeit looks at the CWD, however you can specify the model directory by using --models
Definitions
Models can get very large, Model definitionsDefinitions can have definitions
Definitions
Often times our models cannot be 100% random data, we have to rely on existing data structures from various systems
continents.json
countries.csvInput
We can provide input to fakeit models using the -i parameter
continents.json
countries.csv
Input
How we use FakeItYAML models store in the repositoryMake commands to generate dataDocker Compose: NginxCouchbaseSync GatewayTesting / QA
Small footprint that can generate a large amount of datacompletely scriptedDevelopers can use and try out various models quickly
Demos
If time permits
Fakeit ExamplesContactsEcommerceFlatFlight DataMusicSimpleSocialgithub.com/bentonam/fakeit-examples
Several Examples also come with multiple N1QL queries
Questions?
HOW TO CONTACT MEDont hesitate to contact with me. I will be happy to answer your questions.
TWITTER: @bentonam
github.com/bentonam
EMAIL: [email protected]
THANK YOU