1. 1. Visualize NGiNX log with GoogleMap
2. 2.
3. 3. purpose ! NGiNX ! !
4. 4.
5. 5. ! Internet ! ! GoogleMap
6. 6. ! ngx_http_geoip_module ! Google BigQuery ! () GeoFUSE
7. 7. GeoFUSE? ! GPL ! mbasa/GeoFuse GitHub : https://github.com/mbasa/geofuse ! GeoFUSE : http://geofuse.georepublic.net/geofuse/ ! ! (GUI)
8. 8.
9. 9.
10. 10. overwiew GeoFUSE Big QueryNGiNX 1.Logging 2.Transfer 3.Query 4.CSV
11. 11. 1.Logging ! NginxGeoIP ! GeoIP C APIGeoIP Datale ! nginx--with-http_geoip_module ! Nginx.confGeoIP Datale ! GeoIPIP http { geoip_country /usr/local/share/GeoIP/GeoIP.dat; geoip_city /usr/local/share/GeoIP/GeoIPCity.dat; real_ip_header X-Forwarded-For; set_real_ip_from 10.0.0.0/8; log_format ltsv "geoip_country_name:" "$geoip_city_country_name" "t" # "geoip_country_code3:" "$geoip_city_country_code3" "t" # JPN/USA "geoip_city:" "$geoip_city" "t" # "geoip_latitude:" "$geoip_latitude" "t" # "geoip_longitude:" "$geoip_longitude" # ; }
12. 12. 2.Transfer ! NginxBigQuery ! uentd ! in_tail ! uent-plugin-google-cloud ! NGiNXltsv ! BigQueryjson ()
13. 13. 3.Query ! textPayload json ! JSON_EXTRACT JSON_EXTRACT_SCALAR ! (SUM,AVG,MIN,MAX) SELECT ROW_NUMBER() OVER (ORDER BY count DESC) AS ID, COUNT (*) AS count, JSON_EXTRACT_SCALAR(textPayload,'$.geoip_longitude') AS lon, JSON_EXTRACT_SCALAR(textPayload,'$.geoip_latitude') AS lat, AVG(JSON_EXTRACT_SCALAR (textPayload,'$.response_time')) AS avelage_time FROM [HOGE.nginx_access_20150402] GROUP BY lat,lon ORDER BY count DESC
14. 14. 4.csv ! BigQuerycsv ! Geo-FUSE ! Geo-FUSE ! ! ID ! lon.lat() ID,count,avelage_time,lon,lat 1,3135686,0.2498652794178637,139.6900,35.6900 2,1596097,0.2536400474791719,139.7514,35.6850 3,182082,0.23747259653015085,135.5200,34.6864 4,117914,0.22794176617882336,121.5250,25.0392
15. 15.
16. 16. ! EC ( IP) ! (Wi-)
17. 17.