Embed Size (px)
Citation preview
Python, Geodata and Maps
PyDX Conference, Oct 11th 2015
Hannes Hapke @hanneshapke
Hello, my name is Hannes.
Why could this presentation be interesting to you?
Location data is key today!
What will you know in 30 min?
• Geospatial basics
• Geocode locations and other sources
• How to make geospatial queries
• How to serve geospatial data via APIs
Broad Field …
SRID, PostGIS, GDAL, tiles, Open Street Map, Spatial queries, Mapnik, Shapely, ogr2ogr, kml, GIS, Proj , TIGER, SpatiaLite, Mapbox, leaflet, raster, shapefiles, geometries, EPSG, WGS 84, layers, ArcGIS, bbox, distance loopup, datum, GEOS, geocode, WKT, WKB, GeoDjango, GPS, projections, tiles
Broad Field …
SRID, PostGIS, GDAL, tiles, Open Street Map, Spatial queries, Mapnik, Shapely, ogr2ogr , kml , GIS, Proj, TIGER, SpatiaLite, Mapbox, leaflet, raster, shapefiles, geometries, EPSG, WGS 84, layers, ArcGIS, bbox, distance loopup, datum, GEOS, geocode, WKT, WKB, GeoDjango, GPS, projections, tiles
Geospatial Basics
Just a very few …
Coordinate Systems
Source: https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/LocationAwarenessPG/Art/sphere_to_cylinder.jpg
projectedvs.
unprojected coordinate systems
Remember
4326
Projections
Source: http://gdb.voanews.com/9E85F1AA-B4C0-49FA-ADC0-27BA1FD1C426_mw1024_s_n.jpg
Projections
WGS-84 GCJ-02
Same, same, but different …
GIS Data
raster formatvector formatother formats
Storing Geospatial Data
SourcesSQLite: https://upload.wikimedia.org/wikipedia/commons/thumb/3/38/SQLite370.svg/640px-SQLite370.svg.png Postgres: http://www.tivix.com/uploads/blog_pics/postgresql_1.png
Postgres is a defacto standard
Source PostGIS: http://www.moderncourse.com/wp-content/uploads/2015/03/postgis.png
Thanks to PostGIS
PostGIS, what?
Adds geospatial functionalityto Postgres
Source PostGIS: https://en.wikipedia.org/wiki/PostGIS
Useful tools …
QGIS
Source QGIS: https://underdark.files.wordpress.com/2012/07/stamen_watercolor1.png?w=700
CartoDB
Source CartoDB: http://dmdphilly.org/wp-content/uploads/2015/07/cartodb-editor.2876823e.png
Enough basics!
Let’s get started …
Wait, one last comment …
Consider the user’s privacy …
Sources for Geodata
Geocode Addresses
Geocoder• pip install geocoder
• Package by Denis Carriere
• Supports 20 geocoding services
• Multiple export data structures
• Command line interface
Code Example#!/usr/bin/python
import geocoder """Sample code to demonstrate geocoding"""
address = "70 NW Couch St, Portland, OR 97209" result = geocoder.google(address) print ("Your address: %s" % address) print ("%s, %s" % (result.lng, result.lat)) print ("Location type: %s" % result.accuracy)
Geolocating IP Addresses
Code Example#!/usr/bin/python
import geocoder """Sample code to demonstrate the ip geocoder IP address obtained via http://fakeip.org/"""
ip_address = "192.169.226.73" result = geocoder.ip(ip_address) print ("Your ip address: %s" % ip_address) print ("Location address: %s" % result.address) print ("%s, %s" % (result.lng, result.lat))
Images asgeodata sources
Code Example#!/usr/bin/python
from pyexif import pyexif """Sample code to demonstrate pyexif"""
file_name = “your_image.JPG" result = pyexif.Exif(file_name) print ("Your location: %s %s" % result.lat, result.lon)
GPS Latitude 44.100339 degrees GPS Longitude 121.772294 degrees GPS Altitude 3109.044444 m GPS Time Stamp 21:49:20.09 GPS Speed 0.26 GPS Img Direction Ref True North GPS Img Direction 233.1646707 GPS Dest Bearing Ref True North GPS Dest Bearing 233.1646707 GPS Date Stamp 2015:08:15
Shapefiles
What are Shapefiles?
Source: http://store.mapsofworld.com/image/cache/data/GIS/Oregon%20County-900x700.jpg
Obtain Shapefiles# Install the gis stack (geos, proj4, postgres, postgis, gdal) $ brew install postgis gdal
# Clone the Portland Atlas $ git clone https://github.com/Caged/portland-atlas.git
$ cd portland-atlas/ $ make shp/neighborhoods.shp
# Check structure $ ogrinfo -al -geom=OFF shp/neighborhoods.shp # Check projection $ ogrinfo -so shp/neighborhoods.shp -sql "SELECT * FROM neighborhoods"
Generate Model from Shapefile$ python manage.py ogrinspect \ sample_geodjango/data/neighborhoods.shp \ PortlandNeighborhood --srid=4326 --mapping
# This is an auto-generated Django model module created by ogrinspect. from django.contrib.gis.db import models
class PortlandNeighborhood(models.Model): objectid = models.FloatField() name = models.CharField(max_length=50) shape_leng = models.FloatField()
… geom = models.PolygonField(srid=4326) objects = models.GeoManager() # Auto-generated `LayerMapping` dictionary for PortlandNeighborhood model portlandneighborhood_mapping = { 'objectid' : 'OBJECTID', 'name' : 'NAME', 'shape_leng' : 'SHAPE_LENG', … 'geom' : 'POLYGON', }
Load Shapefile into your DBimport os from django.contrib.gis.utils import LayerMapping from .models import Neighborhood
neighboorhood_mapping = { 'name': 'NAME', 'poly': 'POLYGON', }
neighboorhood_shp = os.path.abspath( os.path.join( os.path.dirname(__file__), '../../data', 'neighborhoods.shp'))
def run(verbose=True): lm = LayerMapping( Neighborhood, neighboorhood_shp, neighboorhood_mapping, transform=False, encoding='iso-8859-1') lm.save(strict=True, verbose=verbose)
Now, what to do with the data?
Generate KML files with Python
How to create kml data structures?
• Keyhole Markup Languange (KML)
• XML based
• Developed by Google for Google Earth
• Package SimpleKMLhttps://pypi.python.org/pypi/simplekml/1.2.8
Code Example""" Example code which will generate a kml file of Portland's neighborhoods
Execute this script with the runscript command of the Django extensions $ python manage.py runscript simplekml_example -v3 """
import simplekml from project.models import Neighborhood
def run(): kml = simplekml.Kml() neighborhoods = Neighborhood.objects.all()
for neighborhood in neighborhoods: # this is just bare example # normally use neighborhood.poly.kml to the kml data kml.newpolygon( name=neighborhood.name, outerboundaryis=list( neighborhood.poly.boundary.coords[0]), innerboundaryis=list( neighborhood.poly.boundary.coords[0]) ) kml.save("portland_neighborhoods.kml")
Google Earth
Generate Static Maps
Motionless• pip install motionless
• Package by Ryan Cox
• Supports street, satellite, and terrain maps
• Based on Google Static Map API
• Watch out for the rate limit
Code Example#!/usr/bin/python """Sample code to demonstrate motionless"""
import requests from StringIO import StringIO from PIL import Image from slugify import slugify from motionless import DecoratedMap, AddressMarker
address = "70 NW Couch St, Portland, OR 97209" # generate static map object dmap = DecoratedMap(maptype='satellite', zoom=18) dmap.add_marker(AddressMarker(address))
# download the static image response = requests.get(dmap.generate_url()) if response.status_code == 200: image = Image.open(StringIO(response.content)) image.save('.'.join([slugify(address), 'png'])) else: print ("Download error with status code %s", response.status_code)
Code Example
But what about database queries?
GeoDjango
GeoDjango
• Created by Justin Bronn
• Part of the Django core
• Supports various spatial databases and libraries
• Let’s you define geospatial attributes in your models
GeoDjango• PointField
• LineStringField
• PolygonField
• MultiPointField
• MultiLineStringField
• MultiPolygonField
GeoDjango Queries
GeoDjangoSpatial Lookups
bbcontains, bboverlaps, contained, contains, contains_properly, coveredby, covers, crosses, disjoint, equals, exact, same_as, intersects, overlaps, relate, touches, within, left, right, overlaps_left, overlaps_right, o v e r l a p s _ a b o v e , o v e r l a p s _ b e l o w, strictly_above, strictly_below
GeoDjangoDistance Lookups
• distance_lt
• distance_lte
• distance_gt
• distance_gte
• dwithin
GeoDjango Example
GeoDjango - Example• Simple GeoModel
• Demonstrate the GeoDjango Admin
• How to save Points
• How to search for Points within a Polygon
• Leaflet Maps
GeoDjango - Example
GeoDjango - Modelclass CallLocation(TimeStampedModel):
address = models.CharField( max_length=255) …
location = models.PointField() objects = models.GeoManager()
def save(self, *args, **kwargs): try: # get the geocoding results result = geocoder.google(self.address) # correct the address spelling self.address = result.address.encode('utf-8') self.location = fromstr( 'POINT(%s %s)' % (result.lng, result.lat), srid=4326) except (AttributeError, Exception): print "Oops! Couldn't geocode address because of %s" \ % sys.exc_info()[0]
super(CallLocation, self).save(*args, **kwargs)
GeoDjango - Viewclass CallLocationCreateView(CreateView):
model = CallLocation form = CallLocationForm success_url = reverse_lazy('home') fields = ['address', 'call_type', 'comment']
def get_context_data(self, *args, **kwargs): context = super( CallLocationCreateView, self ).get_context_data(**kwargs)
context['neighborhoods'] = Neighborhood.objects.all() context['pk'] = int(self.kwargs.get('pk', -1)) if context['pk'] > 0: neighborhood = get_object_or_404(Neighborhood, pk=context['pk']) context['object_list'] = CallLocation.objects.filter( location__within=neighborhood.poly) else: context['object_list'] = CallLocation.objects.all() return context
GeoDjango - Adminfrom django.contrib.gis import admin from .models import CallLocation, CallType, Neighborhood
class GeoLocationAdmin(admin.OSMGeoAdmin): default_zoom = 11 list_display = ( 'call_type', 'address', 'modified',)
class NeighboorhoodAdmin(admin.OSMGeoAdmin): default_zoom = 11 list_display = ('name',)
admin.site.register(CallLocation, GeoLocationAdmin) admin.site.register(Neighborhood, NeighboorhoodAdmin) admin.site.register(CallType)
GeoDjango - Admin
GeoDjango - Template <script type="text/javascript"> function initialize() {
var map = new L.Map( "terrain", { center: new L.LatLng(45.524221, -122.670749), zoom: 16 } ); …
var markers = []; var bbox = [];
{% for call in object_list %} markers.push( L.marker( [ {{ call.location.coords.1 }}, {{ call.location.coords.0 }} ] ).addTo(map)); bbox.push( [ {{ call.location.coords.1 }}, {{ call.location.coords.0 }} ] ); {% endfor %}
map.fitBounds(bbox); }; … </script>
GeoDjango - Example
Django REST Framework
REST+GIS - Example
• Create an API endpoint
• Set up a GeoSerializer
• AngularJS ng-map
• something very practical …
REST+GIS - Example
What needs to change in your API code?
REST+GIS - Model
Nothing needs to change
REST+GIS - Serializer
from .models import CallLocation from rest_framework import serializers from rest_framework_gis import serializers as gis_serializer
class CallSerializer(gis_serializer.GeoModelSerializer):
call_type = serializers.StringRelatedField()
class Meta: model = CallLocation fields = ('status_id', 'address', 'location', 'call_type', 'timestamp') geo_field = 'point'
pip install djangorestframework-gis
REST+GIS - Serializer { "status_id": "650065994463838208", "address": "2200 NE 36th Ave, Portland, OR 97212, USA", "location": "SRID=4326;POINT (-122.6259952999999996 45.5384747000000019)", "call_type": "PROPERTY LOST, FOUND, RECOVERED ", "timestamp": "2015-10-02T21:53:05Z" }
{ "status_id": "650864005439840258", "address": "8900 SW 30th Ave, Portland, OR 97219, USA", "location": { "type": "Point", "coordinates": [ -122.7077201, 45.4608926 ] }, "call_type": "THEFT - COLD ", "timestamp": "2015-10-05T02:44:06Z" }
Without the djangorestframework-gis
With the djangorestframework-gis
REST+GIS - View
# Django Packages # 3rd party from rest_framework import viewsets # project packages from .models import CallLocation from .serializers import CallSerializer
class CallLocationViewSet(viewsets.ModelViewSet): """ API endpoint that allows users to be viewed or edited. """ serializer_class = CallSerializer
No magic here.
REST+GIS - Front end$scope.getMarkers = function () { $scope.state.markers = []; Call.query().$promise.then(function(data) { data.forEach(function(call){ $scope.state.markers.push(( [call.location.coordinates[1], call.location.coordinates[0]])) }); }, function(errResponse) { console.log('REST error'); }); }; ... <div class="container-fluid"> <map zoom="10" center="[45.5200, -122.6819]" on-center-changed="getMarkers()"> <marker ng-repeat="p in state.markers" position="{{ p }}"></marker> </map> </div>
AngularJS with ngmap
How to limit theAPI requests?
Limit requests by theusing bounding box
# views.py from rest_framework import viewsets from rest_framework_gis.filters import InBBOXFilter
class CallLocationViewSet(viewsets.ModelViewSet): """ API endpoint that allows users to be viewed or edited. """ serializer_class = CallSerializer # bbox defines which column is used to select the objects returned # to the frontend bbox_filter_field = 'location' # filter settings filter_backends = (InBBOXFilter, ) # the overlapping allows polygons to be partially be part of the bbox bbox_filter_include_overlapping = True # not needed for PointFields
pip install django-filters
Limit requests by the using bounding box
// Change AngularJS call
_getPolygonStr = function (map){ var bounds = []; bounds[0] = map.getBounds().getSouthWest().lng(); // min lon bounds[1] = map.getBounds().getSouthWest().lat(); // min lat bounds[2] = map.getBounds().getNorthEast().lng(); // max lon bounds[3] = map.getBounds().getNorthEast().lat(); // max lat return bounds.join(","); };
...
if ($scope.map) { bounds = _getPolygonStr($scope.map); };
...
Call.query({in_bbox: bounds}).$promise.then(function(data) { data.forEach(function(call){ … }); }
/api/calls/?in_bbox=-123.093887304,45.327228800,-122.269912695,45.71211299
Recap• SRID, Projections
• How to get geodata (geocoding)
• How to store it
• How to query it
• How to create an API endpoint
Useful Resources
Useful Resources
• Python Geospatial DevelopmentErik Westra, O’Reilly
Useful Resources• GeoDjango Tutorial
https://docs.djangoproject.com/en/1.8/ref/contrib/gis/
• Python & GeoData PyCon 2014 (thanks to @pdxmele): https://github.com/pdxmele/python-geodata-bffs
• GeoDjango DjangoCon 2014 (thanks to @aleck_landgraf)https://www.youtube.com/watch?v=mUhinowr3RY
• Creating maps with D3.js (thanks to @oceankidbilly)http://wrobstory.github.io/2013/06/creating-with-d3-dymo.html
Useful Resources
• Fake Addresses (thanks to @alphydan): https://fakena.me/random-real-address/
• Portland Atlas (thanks to @Caged) https://github.com/caged/portland-atlas
• Boston GIS Grouphttp://www.bostongis.com/
Big thanks to
@alphydan @aleck_landgraf @srikanth_chikoo@patrickbeeson
Thank you.
