Embed Size (px)
Citation preview
PENGUMPULAN DATA ONLINE Arum Handini Primandari
DATA ONLINE
Data tersedia online oleh beberapa lembaga baik independen maupunpemerintahan. Sebagai contoh:
INDONESIA
Badan Pusat Statistik
https://www.bps.go.id/
BMKG: DATA IKLIM
Mengisi kolom komentar
dan saran, kemudian data
dapat terdowload dalam
.xlx
BNPB
http://dibi.bnpb.go.id/
dibi/
OPEN DATA INDONESIA
data.go.id
US https://www.data.gov/
Data.gov is a rich resource for civic hackers, tech entrepreneurs, data
scientists, and developers of all stripes. Here you’ll find information
about APIs, open source projects, and relevant developer resources
across government. You’ll also find updates on the data.gov
infrastructure itself, an open source project managed on GitHub.
US CENSUS BUREAUhttps://www.census.gov/
US NATIONAL LIBRARY OF MEDICINEhttps://www.nlm.nih.gov/databas
es/download/pubmed_medline_
documentation.html
AUSTRALIA
data.gov.au provides an easy way to find, access and reuse
public datasets from Government. The main purpose of the site is
to encourage public access to and reuse of public data. It was
created following the Government’s Declaration of Open
Government and as a response to the Government 2.0 Taskforce
Report.
https://data.gov.au/
KAGGLEhttps://www.kaggle.com/
1. Lakukan sign in (apabila telah
mempunyai akun di kaggle)
2. Buat akun di kaggle, kemudian
lakukan browse data
CONTOH KAGGLE
API
APIs are available for both desktop and mobile operating systems.
The Windows API, for example, provides developers with user interface controls, so that you can do: accessing the file, creating and deleting files, send and receive data over a local network or the Internet, etc.
Mobile APIs, provide commands: for detecting touchscreen input, such as tapping, swiping, and rotating, keyboard, a search bar, and a tab bar, which provides navigation buttons the bottom of the screen, etc.
A website API for web developers allows them to access specific information from the site. A website API may be as simple as a set of XML elements with a few basic commands for retrieving the information.
https://techterms.com/definition/api
Stands for "Application Programming Interface."
An API is a set of commands, functions, protocols, and objects that programmers can use to create software or interact with an external system.
It provides developers with standard commands for performing common operations so they do not have to write the code from scratch.
COMMA-SEPARATED VALUE
In computers, a CSV (comma-separated values) file stores tabular data (numbers and text) in plain text. Each line of the file is a data record.
Each record consists of one or more fields, separated by commas. New lines means new row.
The basic idea of separating fields with a comma is clear, but that idea gets complicated when the field data may also contain commas or even embedded line-breaks.
https://en.wikipedia.org/wiki/Comma-
separated_values
Here's an example:
16611001,Nindya,DS
15611002,Bagas,BS
15611056,Gandi,Aktuaria
Open notepad, type those data, save it in
.csv
DELIMITER-SEPARATED VALUES
Formats that use delimiter-separated values (also DSV) store two-dimensional arrays of data by separating the values in each row with specific delimiter characters (tab, space, comma, colon, vertical bar).
Column headers are sometimes included as the first line, and each subsequent line is a row of data. The lines are separated by newlines.
Here's an example:
Note the use of the double quote to enclose each field
"Date","Pupil","Grade"
"25 May","Bloggs, Fred","C"
"25 May","Doe, Jane","B"
"15 July","Bloggs, Fred","A"
"15 April","Muniz, Alvin ""Hank""","A"
Open notepad, type those data, save it in
.csv
https://en.wikipedia.org/wiki/Delimiter-
separated_values
JSON FILE
JSON is JavaScript Object Notation
A JSON file is a file that stores simple data structures and objects.
It contains data in a standard data interchange format which is lightweight, text-based, and human-readable.
JSON files were originally based on a subset of JavaScript, but is considered a language-independent format, being supported by many different programming APIs.
https://fileinfo.com/extension/json
Contoh file json
READ JSON IN R
Contoh data:
https://catalog.data.gov/dataset/public-data-listing
1. Buka laman tersebut, akan muncul file json seperti gambar di samping
2. Klik “download”
3. Buka Rstudio
4. Install: jsonlite
READ AND WRITE JSON IN R
Read json offline:1. Setelah klik “download”, akan muncul laman
yang memuat file json.2. Sorot semua data (CTRL+A), copy data tersebut
(CTRL+C), kemudian paste (CTRL+V) kenotepad; gunakan notepad++ untukmempermudah
3. Simpan file dalam ekstensi .json
Read json online:1. Setelah klik “download” dan muncul datanya,
maka cukup copy URL dari file tersebut.2. Paste URL ke dalam program R.
library(jsonlite)
ssa <- fromJSON("D:\\Subjects\\TI & Big
Data\\Contoh data\\ssa.json") #OFFLINE
ssa <-
fromJSON("https://www.ssa.gov/data.json")
#ONLINE
ssa$dataset
str(ssa$dataset)
ssa.df <- data.frame(ssa$dataset)
View(ssa.df)
READ AND WRITE JSON IN R
Setelah data dibaca ke dalam R, data dapat disimpan (ditulis) ke dalam file ekstensi .csv dengan sintak:
>write.csv(x, file = “ ”, …)• x merupakan data frame yang ingin
disimpan• file = “ ” isikan direktori tujuan beserta
nama.ekstensi penyimpanan
write.csv(ssa$dataset, file =
"D:\\Subjects\\TI & Big Data\\Contoh
data\\ssa.csv")
Error in if (inherits(X[[j]], "data.frame") && ncol(xj) > 1L) X[[j]] <-as.matrix(X[[j]]) : missing value where TRUE/FALSE needed
(selesaikan error tersebut: penyebaberror, penyelesaian?)
LATIHAN JSON
1. Buka kaggle.com, arahkan ke dataset
2. Filter jenis file .json dan ukuran <10MB
3. Temukan: subtitle untuk game of thrones
4. Read and write file tsb dalam csv
GEOJSON FILE
JSON file created in the GeoJSON format, a format used for geospatial data interchange; supports points, lines, polygons, and other similar primitives; used for defining objects such as roads, bodies of water, buildings, and points of interest.
library(rgdal)
if (!require(geojsonio)) {
install.packages("geojsonio")
library(geojsonio)
}
library(sp)
library(maps)
library(ggmap)
library(maptools)
nycounties <-
geojsonio::geojson_read("http://eric.clst.org/assets/wiki/uplo
ads/Stuff/gz_2010_us_040_00_500k.json", what = "sp")
plot(nycounties)
mapImage <- ggmap(get_googlemap(c(lon = -82.394012,
lat = 34.852619), scale = 1,
zoom = 11), extent = "normal")
nycounties.df <- fortify(nycounties)
print(mapImage + geom_polygon(aes(long, lat, group =
group), data = nycounties.df, colour = "green"))
print(mapImage + geom_polygon(aes(long, lat), data =
nycounties.df, colour = "green"))
CONTOH GEOJSON 1. Buka file contoh file geojson
2. Copy, kemudian paste dalam notepad
3. Simpan dengan ekstensi.geojson
4. Buka laman visualisasigoogle maps API (developers)
5. Drag dan drop file geojsonpada peta di google map
https://developers.google.com/maps/docum
entation/javascript/examples/layer-data-
dragndrop?hl=id#sample-geojson
https://data.gov.au/dataset/ballarat-bike-
racks/resource/eb6351d6-cb22-4df9-9ce0-
b5c586047c03
Contoh file data .geojson
Visualisasi file geojson denganGoogle Map API
RESULT ON GOOGLE MAPS API FOR DEVELOPERS
https://developers.google.com/maps/?hl=id
GEOJSON TO CSV ONLINE
http://www.convertcsv.com/g
eojson-to-csv.htm
READ GEOJSON IN R
library(rgdal)
if (!require(geojsonio)) {
install.packages("geojsonio")
library(geojsonio)
}
library(sp)
library(maps)
library(ggmap)
library(maptools)
bike <- geojson_read("D:\\Subjects\\TI & Big
Data\\Contoh data\\bike.geojson", what = "sp")
plot(bike)
1. Download file geojson2. Simpan file, menggunakan notepad, dalam
format .geojson3. Buka Rstudio4. Install package: rgdal, sp5. Untuk visualisasi map, install package:
maps, ggmap, maptools;
Membaca file geojson:> geojson_read(file = “ “, what = “sp”, …)Keterangan: sp merupakan spatial.
XML READ
XML is a file extension for an Extensible Markup Language (XML) file format used to create common information formats and share both the format and the data on the World Wide Web, intranets, and elsewhere using standard ASCII text.
Example:
https://www.tutorialspoint.com/r/r_xml_files.htm
READ XML in R
LATIHAN
1. Download file dengan ekstensi csv, ukuran >10 MB
2. Download file dengan ekstensi json, ukuran <10 MB. Konversi file tersebut keekstensi csv.
3. Download file dengan ekstensi xml, kemudian bentuklah dalam data frame. Konversikan ke ekstensi .csv.
4. Lakukan deskripsi data
