Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Introducing taocpp/jsonhttps://github.com/taocpp/json
Daniel Frey 2016-11-10
Disclaimer• Opinions expressed are solely my own and do
not express the views or opinions of my employer
• All mistakes are mine, please point them out
• When taking a note for a question, please write down the slide number!
2
Origin• Authors of the PEGTL and taocpp/json:
Dr. Colin Hirsch and Daniel Frey
• Originally a benchmark for the PEGTLhttps://github.com/ColinH/PEGTL
• Benchmarked against 40 other libraries https://github.com/miloyip/nativejson-benchmark
• Inspired by Niels Lohmann’s JSON library https://github.com/nlohmann/json
3
Our use-cases• REST APIs
• Logging: Structured data, supports ELK stack
• Several thousand JSON value constructions and serializations per second
• Main development driven by demand
• Used in production code
4
Type
— tao::json::type
JSO
N v
alue
json::value• Contains a type and a
C++ union to store data
• Query type: v.type()
• Basic JSON types
NULL_
ARRAY
OBJECT
BOOLEAN
STRING
(Number)
5
Type
— tao::json::type
JSO
N v
alue
Type• Contains a type and a
C++ union to store data
• Query type: v.type()
• Basic JSON types
• Numbers come in three flavours
NULL_N
umbe
r
ARRAY
OBJECT
SIGNED
UNSIGNED
DOUBLE
BOOLEAN
STRING
6
Type
— tao::json::type
JSO
N v
alue
Type• First extension:
A plain old raw pointer
• Rarely used…
• …but allows important optimisations!
• Non-owning, dumb. Like… really dumb!
NULL_N
umbe
r
ARRAY
OBJECT
SIGNED
UNSIGNED
DOUBLE
BOOLEAN
STRING
RAW_PTR
7
Type
— tao::json::type
JSO
N v
alue
DataNULL_
Num
ber
ARRAY
OBJECT
SIGNED
UNSIGNED
DOUBLE
BOOLEAN
STRING
RAW_PTR
std::uint64_t
double
std::string
std::vector<value>
std::map<std::string, value>
std::int64_t
bool
const value*
-
8
Type
— tao::json::type
JSO
N v
alue
GetterNULL_
Num
ber
ARRAY
OBJECT
SIGNED
UNSIGNED
DOUBLE
BOOLEAN
STRING
RAW_PTR
v.get_unsigned()
v.get_double()
v.get_string()
v.get_array()
v.get_object()
v.get_signed()
v.get_boolean()
v.get_raw_ptr()
9
Type
— tao::json::type
JSO
N v
alue
Status• Second extension:
Other statuses, part 1
• Default-constructed values do not have a type assigned. The type/status is UNINITIALIZED
NULL_N
umbe
r
ARRAY
OBJECT
SIGNED
UNSIGNED
DOUBLE
BOOLEAN
STRING
UNINITIALIZED
RAW_PTR
10
Type
— tao::json::type
JSO
N v
alue
Status• Second extension:
Other statuses, part 2
• When a value is destructed, moved, … the type/status is set to DISCARDED (sometimes)
NULL_N
umbe
r
ARRAY
OBJECT
SIGNED
UNSIGNED
DOUBLE
BOOLEAN
STRING
UNINITIALIZED
RAW_PTR
11
DISCARDED
LoggingLOG(INFO, 1234, “Hello, world!”);
// LOG(LVL,ID,…) macro expands to: if(tao::log::is_active(LVL, ID)) { tao::json::value v = __VA_ARGS__; tao::log::write(LVL, ID, v); }
12
Value constructionjson::value v = “Hello, world!”;
13
Value constructionjson::value v = “Hello, world!”; json::value v = true; json::value v = 42u; json::value v = -42; json::value v = 1.23;
14
Value constructionjson::value v = “Hello, world!”; json::value v = true; json::value v = 42u; json::value v = -42; json::value v = 1.23; json::value v = json::null; json::value v = json::empty_array; json::value v = json::empty_object;
15
Array constructionjson::value v = json::empty_array; v.emplace_back(true); v.emplace_back(42); v.emplace_back(“Hello, world!”);
16
Array constructionjson::value v = json::array({ true, 42, “Hello, world!” });
• Think std::initializer_list<value>.
• Reality is somewhat more complicated.
17
Object constructionjson::value v = json::empty_object; v.emplace(“foo”, true); v.emplace(“bar”, 42); v.emplace(“baz”, “Hello, world!”);
v[“foo”][“bar”]; // will throw v[“foo”] = 123u; v[“x”][“y”] = “nice”;
18
Object constructionjson::value v = { { “foo”, true }, { “bar”, 42 }, { “baz”, “Hello, world!” } };
• Keys are always strings, values are anything a direct construction of value accepts.
• No duplicate keys are allowed.
19
Object constructionjson::value v = { { “foo”, true }, { “bar”, 42 }, { “baz”, “Hello, world!” } };
• Keys are always strings, values are anything a direct construction of value accepts.
• No duplicate keys are allowed.
20
Object constructionjson::value v = { { “Some dwarfs”, { { “Thorin”, “Leader” }, { “Kili”, “Nephew” }, { “Balin”, “Third-cousin” }, { “Dori”, “Remote kinsman” }, { “Bifur”, “From Moria” } } } };
21
Object constructionjson::value dwarfs = { { “leader”, “Thorin” }, { “nephews”, json::array({ “Kili”, “Fili” })}, { “third_cousins”, json::array({ “Balin”, “Dwalin”, “Oin”, “Gloin” })}, … };
22
LoggingLOG(INFO, 1234, “Hello, world!”);
23
LoggingLOG(INFO, 1234, { { “foo”, true }, { “bar”, 42 }, { “baz”, “Hello, world!” } });
24
Loggingstruct user { bool is_human; std::string name; unsigned age; };
user u = …;
25
LoggingLOG(INFO, 1234, { { “msg”, “add user” }, { “user”, { { “is_human”, u.is_human }, { “name”, u.name }, { “age”, u.age } } } });
26
Loggingtemplate<> struct traits<user> { static void assign( value& v, const user& u) { v = { { “is_human”, u.is_human }, { “name”, u.name }, { “age”, u.age } }; } };
27
LoggingLOG(INFO, 1234, { { “msg”, “add user” }, { “user”, u } });
28
Loggingtemplate<> struct traits<user> { static void assign( value& v, const user& u ); static const char* default_key; };
29
LoggingLOG(INFO, 1234, { { “msg”, “add user” }, u });
30
LoggingLOG(INFO, 1234, { { “msg”, “add user” }, u });
LOG(INFO, 1235, { { “msg”, “assign manager” }, u, { “manager”, u2 } });
31
Traitstemplate<> struct traits<user> { static user as(const value& v); };
auto u = v.as<user>();
32
Traitstemplate<> struct traits<user> { static user as(const value& v) { return user( v.at("is_human").get_boolean(), v.at("name").get_string(), v.at("age").as<unsigned>() ); } }; // other options under consideration
33
Traits
• Construct a value from any T
• Convert a value to any T
• Modify default behaviour
• value is actually basic_value<traits>
34
SAX
… …
SAX producers
sax::from_value
sax::from_string
sax::parse_file
SAX API
null() boolean(v) number(v) string(v)
begin_array() element() end_array()
begin_object() key(v) member()
end_object()
SAX consumers
sax::to_value
sax::to_stream
sax::to_pretty_stream
SAX schema validation
nlohmann::to_value nlohmann::from_value
sax::tee
35
SAX producertemplate<typename Consumer> void from_value(const value& v, Consumer& c) { switch(v.type()) { case type::UNINITIALIZED: throw …; case type::DISCARDED: throw …; case type::NULL_: c.null(); return; case type::BOOLEAN: c.boolean(v.get_boolean()); return; case type::SIGNED: c.number(v.get_signed()); return; case type::UNSIGNED: c.number(v.get_unsigned()); return; case type::DOUBLE: c.number(v.get_double()); return; case type::STRING: c.string(v.get_string()); return; … } }
36
SAX producertemplate<typename Consumer> void from_value(const value& v, Consumer& c) { switch(v.type()) { … case type::ARRAY: c.begin_array(); for(const auto& e : v.get_array()) { from_value(e, c); c.element(); } c.end_array(); return; … } }
37
SAX producertemplate<typename Consumer> void from_value(const value& v, Consumer& c) { switch(v.type()) { … case type::OBJECT: c.begin_object(); for(const auto& e : v.get_object()) { c.key(e.first); from_value(e.second, c); c.member(); } c.end_object(); return; … } }
38
SAX producertemplate<typename Consumer> void from_value(const value& v, Consumer& c) { switch(v.type()) { … case type::RAW_PTR: if(const auto* p = v.get_raw_ptr()) { from_value(*p, c); } else { c.null(); } return; } throw …; }
39
SAX
… …
SAX producers
sax::from_value
sax::from_string
sax::parse_file
SAX API
null() boolean(v) number(v) string(v)
begin_array() element() end_array()
begin_object() key(v) member()
end_object()
SAX consumers
sax::to_value
sax::to_stream
sax::to_pretty_stream
SAX schema validation
nlohmann::to_value nlohmann::from_value
sax::tee
40
SAX consumerclass to_stream { private: std::ostream& os; bool first = true; void next() { if(!first) os << ’,’; } public: explicit to_stream(std::ostream& os) noexcept : os(os) {} … };
41
SAX consumerclass to_stream { … void null() { next(); os << “null”; } void boolean(const bool v) { next(); os << (v ? “true” : “false”); } void number(const std::int64_t v) { next(); os << v; } void number(const std::uint64_t v) { next(); os << v; } void number(const double v) { next(); os << v; } void string(const std::string& v) { next(); os << ’”’ << escape(v) << ’”’; } … };
42
SAX consumerclass to_stream { … void begin_array() { next(); os << ’[’; first = true; } void element() { first = false; } void end_array() { os << ’]’; } void begin_object() { next(); os << ’{’; first = true; } void key(const std::string& v) { string(v); os << ’:’; first = true; } void member() { first = false; } void end_object() { os << ’}’; } };
43
SAX
… …
SAX producers
sax::from_value
sax::from_string
sax::parse_file
SAX API
null() boolean(v) number(v) string(v)
begin_array() element() end_array()
begin_object() key(v) member()
end_object()
SAX consumers
sax::to_value
sax::to_stream
sax::to_pretty_stream
SAX schema validation
nlohmann::to_value nlohmann::from_value
sax::tee
44
SAX decoupling• DOM-less usage, e.g., parse and pretty print a
file via SAX for extremely large JSON files
• Parse and generate binary formats (BSON, BJSON, UBJSON, …)
• Combine with other JSON libraries
• With tee, feed events to multiple consumers
45
46
0.Overall
20% 40% 60% 80% 100%
ArduinoJson(C++)
CAJUN(C++)
ccan/json(C)
cJSON(C)
Configuru(C++11)
dropbox/json11(C++11)
gason(C++11)
hjiang/JSON++(C++)
Jansson(C)
JeayeSON(C++14)
jsmn(C)
JSONSpirit(C++)
JSONVoorhees(C++)
json-c(C)
JsonBox(C++)
jsoncons(C++)
JsonCpp(C++)
JVar(C++)
Jzon(C++)
mikeando/FastJson(C++)
nbsdx_SimpleJSON(C++11)
Nlohmann(C++11)
Parson(C)
PicoJSON(C++)
RapidJSON(C++)
RapidJSON_AutoUTF(C++)
RapidJSON_FullPrec(C++)
RapidJSON_Insitu(C++)
sajson(C++)
Sheredomjson.h(C)
SimpleJSON(C++)
taocpp/json&Nlohmann(C++11)
taocpp/json(C++11)
tunnuz/JSON++(C++)
ujson(C++)
ujson4c(C)
Vinenthz/libjson(C)
YAJL(C)
69%69%
82%82%
89%89%
69%69%
99%99%
87%87%
55%55%
55%55%
87%87%
52%52%
58%58%
69%69%
88%88%
88%88%
65%65%
95%95%
90%90%
88%88%
74%74%
90%90%
34%34%
96%96%
83%83%
87%87%
93%93%
93%93%
100%100%
93%93%
86%86%
82%82%
67%67%
100%100%
100%100%
74%74%
94%94%
63%63%
88%88%
86%86%
Result
47
1.Parse
0 150 300 450 600
ArduinoJson(C++)
CAJUN(C++)
ccan/json(C)
cJSON(C)
Configuru(C++11)
dropbox/json11(C++11)
gason(C++11)
hjiang/JSON++(C++)
Jansson(C)
JeayeSON(C++14)
jsmn(C)
JSONSpirit(C++)
JSONVoorhees(C++)
json-c(C)
JsonBox(C++)
jsoncons(C++)
JsonCpp(C++)
JVar(C++)
Jzon(C++)
mikeando/FastJson(C++)
nbsdx_SimpleJSON(C++11)
Nlohmann(C++11)
Parson(C)
PicoJSON(C++)
RapidJSON(C++)
RapidJSON_AutoUTF(C++)
RapidJSON_FullPrec(C++)
RapidJSON_Insitu(C++)
sajson(C++)
Sheredomjson.h(C)
SimpleJSON(C++)
strdup(C)
taocpp/json&Nlohmann(C++11)
taocpp/json(C++11)
tunnuz/JSON++(C++)
ujson(C++)
ujson4c(C)
Vinenthz/libjson(C)
YAJL(C)
555555
588588
3030
2727
4949
4848
66
143143
6565
8181
343343
8383
253253
7272
269269
5353
7171
6868
159159
2020
155155
4141
4545
100100
77
1717
1313
77
77
1919
5050
00
5353
3030
174174
2929
88
5656
5757
Time(ms)
48
1.Parse
0 2,000,000 4,000,000 6,000,000 8,000,000
ArduinoJson(C++)
CAJUN(C++)
ccan/json(C)
cJSON(C)
Configuru(C++11)
dropbox/json11(C++11)
gason(C++11)
hjiang/JSON++(C++)
Jansson(C)
JeayeSON(C++14)
jsmn(C)
JSONSpirit(C++)
JSONVoorhees(C++)
json-c(C)
JsonBox(C++)
jsoncons(C++)
JsonCpp(C++)
JVar(C++)
Jzon(C++)
mikeando/FastJson(C++)
nbsdx_SimpleJSON(C++11)
Nlohmann(C++11)
Parson(C)
PicoJSON(C++)
RapidJSON(C++)
RapidJSON_AutoUTF(C++)
RapidJSON_FullPrec(C++)
RapidJSON_Insitu(C++)
sajson(C++)
Sheredomjson.h(C)
SimpleJSON(C++)
strdup(C)
taocpp/json&Nlohmann(C++11)
taocpp/json(C++11)
tunnuz/JSON++(C++)
ujson(C++)
ujson4c(C)
Vinenthz/libjson(C)
YAJL(C)
4545
6,248,3786,248,378
279,061279,061
263,589263,589
285,815285,815
579,229579,229
943943
877,851877,851
379,444379,444
974,012974,012
99
831,566831,566
1,477,9801,477,980
571,890571,890
1,427,2861,427,286
311,812311,812
425,949425,949
85,10485,104
1,719,4361,719,436
6868
2,208,2012,208,201
290,371290,371
594,122594,122
1,648,3761,648,376
115115
115115
115115
104104
1212
66
805,471805,471
66
825,699825,699
206,072206,072
1,729,5621,729,562
285,984285,984
3131
560,910560,910
768,299768,299
AllocCount
49
2.Stringify
0 70 140 210 280
ArduinoJson(C++)
CAJUN(C++)
ccan/json(C)
cJSON(C)
Configuru(C++11)
dropbox/json11(C++11)
gason(C++11)
hjiang/JSON++(C++)
Jansson(C)
JeayeSON(C++14)
JSONSpirit(C++)
JSONVoorhees(C++)
json-c(C)
JsonBox(C++)
jsoncons(C++)
JsonCpp(C++)
JVar(C++)
Jzon(C++)
mikeando/FastJson(C++)
nbsdx_SimpleJSON(C++11)
Nlohmann(C++11)
Parson(C)
PicoJSON(C++)
RapidJSON(C++)
RapidJSON_AutoUTF(C++)
RapidJSON_FullPrec(C++)
RapidJSON_Insitu(C++)
Sheredomjson.h(C)
SimpleJSON(C++)
strdup(C)
taocpp/json&Nlohmann(C++11)
taocpp/json(C++11)
tunnuz/JSON++(C++)
ujson(C++)
Vinenthz/libjson(C)
YAJL(C)
5050
161161
8585
5757
204204
8686
7272
279279
8787
7272
115115
8585
2121
207207
9595
135135
5757
2020
7777
149149
9696
164164
105105
88
1616
99
1010
6161
224224
00
2222
1919
172172
2121
8686
9797
Time(ms)
50
3.Prettify
0 60 120 180 240
ArduinoJson(C++)
ccan/json(C)
cJSON(C)
Configuru(C++11)
gason(C++11)
Jansson(C)
JSONSpirit(C++)
JSONVoorhees(C++)
json-c(C)
JsonBox(C++)
jsoncons(C++)
Jzon(C++)
nbsdx_SimpleJSON(C++11)
Nlohmann(C++11)
Parson(C)
RapidJSON(C++)
RapidJSON_AutoUTF(C++)
RapidJSON_FullPrec(C++)
RapidJSON_Insitu(C++)
Sheredomjson.h(C)
taocpp/json&Nlohmann(C++11)
taocpp/json(C++11)
ujson(C++)
Vinenthz/libjson(C)
YAJL(C)
127127
9898
5858
210210
9292
9595
149149
182182
3131
194194
9696
3333
147147
114114
166166
1010
3030
1010
1212
6767
3030
2727
2424
8686
113113
Time(ms)
Thank you!https://github.com/taocpp/json
Questions?