Developing and Testing a MongoDB and Node.js REST API

Valeri KarpovNode.js Engineer, MongoDBwww.thecodebarbarian.com

github.com/vkarpov15@code_barbarian

*

What is this talk about?

•Briefly introduce MongoDB and Node.js

•MongoDB is great for storing web/mobile app data

•So let’s build a REST API using Node.js!

•+ learn a bit about test-driven dev with Node.js

•+ learn two MongoDB schema design principles

•Server-side only

•Upcoming EdX course

*

What is MongoDB?

•Document database - store objects, not columns

•Often better* performance• Better data locality than RDBMS for 1 query vs 5 JOINs

*

What is MongoDB?

•Easier ops• No need to set up databases, tables, schemas, etc.

• Run one executable, you now have a working db

• MongoDB Cloud Manager and Ops Manager

•Developer sanity• In my experience, easiest database to get working

• Minimal data transformation between DB and server

• Company has dedicated “Developer Experience” team

*

Overview

•Part 1: Shopping Cart Application• Search for products

• Add them to your cart

• Check out with Stripe

•Part 2: Using Node.js and the Mongoose ODM

•Part 3: Schema Design

•Part 4: Building an API with the Express framework

•Part 5: Testing with Mocha + Superagent

*

Part 1: What Does the App Do?

*

What Does the App Do?

*

What Does the App Do?

*

App Structure

•"Bad programmers worry about the code. Good programmers worry about data structures and their relationships." - Linus Torvalds

•3 schemas for 3 collections:

•Products

•Categories

•Users

*

Schema Relationships

•Product belongs to one or more categories

•Users can have multiple products in their cart

•Representing relationships in MongoDB is tricky• MongoDB doesn’t have a JOIN equivalent

•But that’s what mongoose is for• Mongoose provides pseudo-JOINs (separate queries)

*

Part 2: Using the Mongoose ODM

•“Object document mapper” (like ORM, but for MongoDB)

•“MongoDB object modeling designed to work in an asynchronous environment”

•Written for Node.js

•Provides schema validation, pseudo-JOINs, etc.

*

Brief Overview of Node.js

•require()

● Async I/O

*

What Does Async Mean?

Register eventhandler

In node.js, you don’t execute I/O imperatively.

You register a callback to execute when the I/O is done

Single thread - handler can’t be interrupted

Prints before“done reading”

*

Why Async?

•Most web app/mobile app backends are I/O bound• Multi-threading doesn’t help if CPU is just waiting for I/O

• Mostly just waiting for database to respond

•Nowadays there’s a REST API for everything• Analytics and tracking: Segment

• Transactional email: Vero

• Mailing lists: MailChimp

•Threads are cumbersome for I/O bound programs

*

Your First Mongoose Schema

matches X@Y.Z

*

Using Your Schema

Create User

Save user to MongoDB

Load user from MongoDB

Print user to stdout

*

Part 2 Takeaways

•Mongoose provides several neat features• Model part of MVC

• Default values

• Schema validation and declarative schema design

*

Part 3: Schema Design

•3 schemas:• Product

• Category

• User

•Going to use mongoose to define schemas

•Will use a couple key schema design principles

*

Product Schema

*

Product Schema in Action

*

Category Schema

*

Category Schema Queries

•What categories are descendants of “Electronics”?

•

•

•What categories are children of “Non-Fiction”?

•

•What categories are ancestors of “Phones”?

*

Product + Category Schemas

*

Category Schema Takeaways

•Queries in MongoDB should be simple

•Strive for minimal data transformation by server

•“Store what you query for”

•“If you need [the aggregation framework in a heavily used API endpoint], you're screwed anyway, and should fix your program.” - Linus Torvalds

•Good for performance and developer sanity

*

User Schema

User’s cart as an arrayof ObjectIds...

*

Principle of Least Cardinality

•Product and user = many-to-many relationship

•Don’t necessarily need a mapping table

•User won’t have 1000s of products in cart

•Can represent relationship as array in user since one side is small

•If one side of many-to-many is bounded and/or small, it is a good candidate for embedding

•Arrays that grow without bound are an antipattern!• 16mb document size limit

• network overhead

*

Part 4: The Express Framework

•Most popular Node.js web framework

•Simple, pluggable, and fast

•Great tool for building REST APIs

*

Your First Express App

Route Parameter

*

What is REST?

•Representational State Transfer

•HTTP request -> JSON HTTP response

•Business logic on top of MongoDB schemas• Access control, emails, analytics, etc.

*

Structuring Your REST API

*

GET /category/id/:id

Find Category

Error handling

Output JSON

*

GET /category/parent/:id

*

GET /product/category/:id

*

Adding Products to User’s Cart

•Recall cart is an array of products

*

Adding Products to User’s Cart

Get cart from HTTP request

Overwrite user’s cart

Let mongoose handlecasting and validating data

*

PUT /me/cart Takeaways

•Mongoose lets you be lazy

•Access control using subdocs

*

Bonus: Stripe Checkout

Create a Stripe charge with the npm ‘stripe’ module

*

Bonus: Stripe Checkout

Error handling

Empty user carton success

*

Part 4 Takeaways

•Express REST API on top of mongoose• Access control

• Business logic

• Define what operations user can take on database

•Mongoose casting and validation for APIs

*

Part 5: Test-Driven Development

•Building an API is tricky

•Lots of different error conditions

•Express has a lot of magic under the hood

*

NodeJS Concurrency and Testing

•Node.js runs in an event loop

•Single threaded

•Can run client and server on same thread!• Client sends HTTP request

• Client registers a callback awaiting the result

• Server’s “on HTTP request” event handler is triggered

• Server sends response, continues waiting for events

• Client’s callback gets fired

•Test server end-to-end

*

Superagent

•NodeJS HTTP client

•Isomorphic: runs in both browser and NodeJS

•Same author as Express

*

Mocha

•Testing Framework for NodeJS

•Same author as Express

•BDD-style syntax• describe() -> test suite

• it() -> individual test

*

Setting Up Category API Tests

*

Testing GET /category/id/:id

*

Part 5 Takeaways

•NodeJS concurrency makes testing easy

•Not just unit tests - full E2E for your REST API

•Can manipulate database and make arbitrary HTTP requests

*

•Upcoming EdX Video Course

•Slides on http://www.slideshare.net/vkarpov15

•Looking for beta testers! Sign up for notifications• http://goo.gl/forms/0ckaJ4YvJN

•Interested in learning about AngularJS?• Professional AngularJS on Amazon

•More NodeJS+MongoDB content at:• www.thecodebarbarian.com

• Twitter: @code_barbarian

Thanks for Listening!