Building OpenStreetMap.org
Tom Hughes
[email protected]@thughes
Session all about the code behind osm.org and how we develop it
and how you can helpI'm Tom Hughes and I've been helping run the
servers since 2007I work on everything but in particular on the
main web site
Web Site Architecture
DatabaseServerWeb/RailsServerWeb/RailsServerWeb/RailsServerRailsServerRailsServerRailsServer
FileServer
Quick overview of architecture**Three frontends using round
robin DNSRun apache and passengerHandle most requestsHeavy requests
proxied to shared backend pool**Three backendsRun apache, passenger
and cgimapSome API requests sent to cgimapEverything else handled
by passenger (currently)
Database Schema
Usersusers / user_tokens / user_preferences / user_roles /
user_blocks
Geodatachangesets / changeset_tags / changeset_comments /
...
current_nodes / current_node_tags / nodes / node_tags
current_ways / current_way_tags / current_way_nodes /
current_relations / current_relation_tags /
current_relation_members /
GPS Datagpx_files / gpx_file_tags / gps_points
Notesnotes / note_comments
Overview of databaseUsers table, with various auxiliary
tables**Geodata, everything grouped by changesets**Nodes current
tables with latest version, history tables with all versions yes
duplicated...**Repeated for ways and relations but with tables to
link to child objects**Imported GPS data yes the names are not
consistent**Notes with commentsVarious others with diaries,
friends, messages,etc
The Rails Port
Main web site is Ruby on Rails known as the rails port for
historical reasonsFind it on github issues welcome pull requests
even more welcomeIncludes an implementation of the APIAlternate
implementation of part of the API is what is used in production
though more later
The Rails Port
Getting StartedINSTALL.md
CONFIGURE.md
CONTRIBUTING.md
AlternativelyVAGRANT.md
We have issues...
Get started checkout the code!Do a (small) bit of
reading**INSTALL.md tells you how to do the basic
setup**CONFIGURE.md covers a few things you may want to do once you
have the basics working**CONTRIBUTING.md explains how we work **Or
you can use vagrant to do much of the basic setup read VAGRANT.md
for more information**If you're looking for things to do then we
have open issues!
The Rails Port
Models app/models
Views app/views
Controllers app/controllers
Javascript app/assets/javascripts
Stylesheets app/assets/stylesheets
Strings config/locales/en.yml
Structure is a basic MVC architectureModels represent the data
with a class for each type of data**Views are what the user sees we
use ERB templates which are a mix of HTML and Ruby**Controllers
handle user requests fetching the needed data and then rendering a
view**We also have client side javascript code using jQuery**And
stylesheets to control client side rendering of the views**All text
needs to be translatable add strings to english and Translatewiki
does the rest
The Rails Port
ModelsNode class is current_nodes
OldNode class is nodes
Trace class is gpx_files
Tracetag class is gpx_file_tags
Tracepoint class is gps_points
Models have one class per database tableMostly the names
matchExcept when they don't...**Big difference is that current
tables lose their prefix**and history tables gain an old
prefixApplies to ways and relations as well**GPS tables also
differentThis is largely a question of history and the rails code
being written around an existing schema
The Rails Port
Stylerubocop for ruby
rake jshint for javascript
Testsrake test for ruby
javascript coming soon?
Travis will run on pull requests
Coveralls will check coverage
Please run style checkers we have rubocop setup for ruby and
jshint for rails**Even more important please run the testsWe have
good tests for the ruby codeBut not (yet) for javascript**Style
checks and tests will be run by travis when you open a
pull**Coverage will also be checked by coveralls only the ruby code
at the moment but coverage is good and we'd like to keep it that
waySo add tests for any new ruby code!
The Rails Port
MergingPull requests notified to rails-dev list
Review of behaviour and UI changes
Review of code changes
Larger changes may get a test instance
When your change is ready, open a PRThe rails-dev list will be
notified, as it is for all issues**First thing we will usually do
is look at the change as a user what are the UI changes etc**When
we're happy with that I will usually look at the actual code for
any implementation issuesNot hard and fast though, things may
happen in a different order, and small fixes are often just merged
straight off**Larger changes may get a test instance, with people
invited to try it out
CGImap
Second part of our story is CGImapAlso on githubC++
implementation of (part of) the APIIncreased performance by
carefully tailored SQL queriesBetter memory management for
potentially large data sets
CGImap
Currently handles a number of API callsThe main map call
Single node, way and relation fetches
Multiple node, way and relation fetches
The full way and relation calls
Want to extend it to cover more API callsHistory calls
Changeset calls
Currently handlesThe main map call what editors use to get
dataFetching individual map objectsFetching multiple map
objectsFetching map objects with their child objects**Want to
extend itCalls which include object historyChangeset callsAnd so
on
CGImap
Currently runs as a separate daemonAccessed via FastCGI
Core code has now been split into a libraryWill allow creation
of a ruby gem around the library
Which then allows direct integration with rails code
JSON output supportCan be compiled with JSON support
Not currently enabled in production
Currently runs as a separate daemon called via FastCGI**Core
code has been split into a libraryWill allow a gem wrapper to be
written for integration with rails codeAvoids current duplication
of API code between rails and cgimap**Supports JSON but not
currently enabled as only calls handled by cgimap would be
supportedMoving rails code to call cgimap via gem wrapper will
allow this (and other changes) to be implemented onceNeed to reach
reasonable API completeness in cgimap first though help please!
CGImap
StyleFormatted in LLVM style with clang-format
Configure --with-clang-format
Run make clang-format to reformat
TestsRun tests with make check
Needs postgresql running
Travis coming soon?
Formatted in LLVM style with clang-formatNeed to configure
with-clang-format then you can reformat with make clang-formatNeeds
clang 3.6**Run the tests with make checkNeed to have postgres
runningTravis config merged but not enabled yet
Get Involved
GitHubhttps://github.com/openstreetmap/openstreetmap-websitehttps://github.com/zerebubuth/openstreetmap-cgimapMailing
Listshttps://lists.openstreetmap.org/listinfo/rails-devhttps://lists.openstreetmap.org/listinfo/devIRC#osm-devEngineering
Working
Grouphttps://wiki.osmfoundation.org/wiki/[email protected]
None of this happens by magic, and many of the people working on
this are also trying to keep the servers runningSo please, get
involved if you're planning something big, or just want advice,
then please contact us and we'll be glad to helpI can take some
questions now, or find me later I'm around all weekend and for the
hackday on MondayAnd if you want to know more about how we actually
turn all this code into a running service then stay around for my
colleague Grant Slater's talk which is up next in this room
