Erlang Data Storage Modules Tutorial

by Flavio Ishii

Erlang Data Storage Modules

•Modules:•Erlang Terms Storage (ETS)•Disk Erlang Term Storage (DETS)•Mnesia Distributed DBMS

• Data Storage Design

Flavio Ishii, Sept. 17, 2009

Tuples

•A Tuple groups items into one entity/term.

•Used in Erlang data storage modules.

Flavio Ishii, Sept. 17, 2009

Tuples

> N = { “Flavio Ishii” }.

> S = [“cycling”,"ultimate frisbee" ].

> P = { person, { 1, N, S } }.

> { person, Properties } = P.

> Properties.

> { Id, Fullname, Sports } = Properties.

> X = lists:append( Sports, [“basketball”] ).

> X.

Flavio Ishii, Sept. 17, 2009

ETS

• Erlang Term Storage

• Constant access time (excl. ordered_set)• Table Options:

• access rights: private | public | *protected• types: *set | ordered_set | bag | duplicate_bag• transfer ownership: {heir, Pid, HeirData}, or

give_away/3

• Various methods to query the table: mnesia function (pattern matching) or qlc

* defaultFlavio Ishii, Sept. 17, 2009

ETS Table Access Rights

• private - Only owner process can read/write

• public - All processes with table id can read/write

• protected - Only owner process can write and any process with table id can read.

Flavio Ishii, Sept. 17, 2009

ETS Table Types

• set - unique {key, value}, inserts may overwrite. • [ {b,2}, {a,5} ]

• ordered_set - key ordered and unique tuple• [ {a,5}, {b,2} ]

• bag - duplicate key allowed• [ {a,5}, {b,2}, {b,4} ]

• duplicate_bag - duplicate tuple allowed• [ {a,5}, {b,2}, {b,2} ]

• hash tables: set, bag, and duplicate_bag• balanced binary tree: ordered_set

Flavio Ishii, Sept. 17, 2009

ETS

> Tab = ets:new(user, []).

> ets:insert(Tab, [{1,flavio},{2,ralph},{3,melissa},{4,bob}]).

> ets:info(Tab).

> ets:match(Tab,'$1').

> ets:match(Tab,{'$1',bob}).

> ets:match_object(Tab,{'$1',bob}).

> ets:select(Tab,[{{'$1',bob},[],['$$']}]).

> ets:select(Tab,[{{'$1',bob},[],['$_']}]).

> qlc:eval( qlc:q([{Y} || {X,Y} <- ets:table(Tab), (X > 2) and (X < 4)])).

Flavio Ishii, Sept. 17, 2009

DETS

• Stores persistent data in disk/file (disk seeks)

• Process needs to open the file.• Table may be shared by local processes.• No ordered_set table type.

• Must be properly closed when process is terminated.

• Checks for consistency on startup after crash.

Flavio Ishii, Sept. 17, 2009

DETS> dets:open_file(user, []).

> dets:insert(user, [{1,flavio},{2,ralph},{3,melissa},{4,bob}]).

> dets:info(user).

> dets:match(user,'$1').

> dets:match(user,{'$1',bob}).

> dets:match_object(user,{'$1',bob}).

> dets:select(user,[{{'$1',bob},[],['$$']}]).

> dets:select(user,[{{'$1',bob},[],['$_']}]).

> qlc:eval( qlc:q([{Y} || {X,Y} <- dets:table(user), (X > 2) and (X < 4)])).

Flavio Ishii, Sept. 17, 2009

Mnesia

• Erlang’s distributed DBMS

• ETS & DETS core• Location Transparency• Fault Tolerance via Table Replication &/or

Fragmentation across nodes

• Transactions, Locking, & Dirty Operations• Schema Manipulation at runtime

• ACID Properties

Flavio Ishii, Sept. 17, 2009

Mnesia ACID Properties

• Atomicity - succeed on all or no nodes• Consistency - ensure consistent state

after crash• Isolation - isolate manipulations on

same record• Durability - changes are committed to

disk.

Flavio Ishii, Sept. 17, 2009

Mnesia Table Options

• {type, Type} % set, ordered_set, bag• {record_name, Name}• {ram_copies, NodeList} % fastest• {disc_copies, NodeList} % RAM and disc copies• {disc_only_copies, NodeList} % slowest• {attributes, AtomList}• {index, IndexAtomList}

Flavio Ishii, Sept. 17, 2009

Terminal Time!

• Create record structure• Create schema and tables• Insert records• Query records

Flavio Ishii, Sept. 17, 2009

api.hrl

% this file defines the records

-record(counter,{id_name,value}).

-record(user_details,{password,firstname,lastname,sports=[]}).

-record(user,{id,username,user_details}).

Flavio Ishii, Sept. 17, 2009

Part 1/4 of api_db.erl

% this file defines the records

-module(api_db).

-export([init_db/0, add_sport/2, mne_fun_query/1, qlc_query/1]).

-include("api.hrl").

-include_lib("stdlib/include/qlc.hrl").

init_db() ->

mnesia:create_schema([node()]),

io:format("Mnesia schema created~n"),

mnesia:start(),

mnesia:change_table_copy_type(schema, node(), disc_copies),

mnesia:create_table(counter, [{disc_copies,[node()]}

, {attributes, record_info(fields, counter)}]),

mnesia:create_table(user, [{disc_copies,[node()]}, {index, [username]}

, {attributes, record_info(fields, user)}]),

io:format("Mnesia tables created~n"),

add_sample_data().

Flavio Ishii, Sept. 17, 2009

Part 2/4 of api_db.erl

insert_user(UserRecord) ->

case UserRecord#user.id =:= undefined of

true -> NewUser = UserRecord#user{ id = mnesia:dirty_update_counter(counter, user_id, 1) };

false -> NewUser = UserRecord

end,

% mnesia:dirty_write(NewUser). or use a transaction...

Fun = fun() ->

mnesia:write(NewUser)

end,

mnesia:transaction(Fun).

add_sample_data() ->

Flavio = #user{username="flavio"

, user_details=#user_details{ password="mypassword", firstname="Flavio", lastname="Ishii", sports=["biking","basketball"]}},

insert_user(Flavio),

Bob = #user{username="bob", user_details=#user_details{ password="hispassword", firstname="Bob", lastname="The Builder", sports=["soccer"]}},

insert_user(Bob),

io:format("Users added to table.~n").

Flavio Ishii, Sept. 17, 2009

Part 3/4 of api_db.erl

% > api_db:add_sport("flavio","football").

add_sport(Un,NewSport) ->

[User] = mne_fun_query({username,Un}),

UserDetails = User#user.user_details,

Sports = UserDetails#user_details.sports,

NewList = lists:append(Sports,[NewSport]),

NewUser = User#user{user_details=#user_details{sports=NewList}},

insert_user(NewUser).

% This is one method of querying a user.

% > api_db:qlc_query({username,"flavio"}).

qlc_query({username,Username}) ->

F = fun() ->

qlc:e(qlc:q([U#user.user_details || U <- mnesia:table(user)

, U#user.username =:= Username

]))

end,

mnesia:transaction(F).

Flavio Ishii, Sept. 17, 2009

Part 4/4 of api_db.erl

% This may not be ideal but it demonstrates the use of Pattern Matching

% > api_db:mne_fun_query({username,"flavio"}).

mne_fun_query({username,Un}) ->

MatchHead = #user{username='$1', _='_'},

Guard = [{'=:=','$1',Un}],

Result = ['$_'],

MatchSpec = [{MatchHead, Guard, Result}],

mnesia:dirty_select(user, MatchSpec);

% > api_db:mne_fun_query({sport,"soccer"}).

mne_fun_query({sport,Sport}) ->

UserDetails = #user_details{_='_',sports='$1'},

MatchHead = #user{user_details=UserDetails,_='_'},

Guard = [{'=:=','$1',[Sport]}],

Result = ['$_'],

MatchSpec = [{MatchHead, Guard, Result}],

F = fun() -> mnesia:select(user, MatchSpec) end,

mnesia:transaction(F);

mne_fun_query(_) -> io:format("No match~n").

Flavio Ishii, Sept. 17, 2009

ETS DETS MnesiaPersistent storageComplex search queriesDistributed Replicated data storageTable FragmentationFault tolerance via replicationTight Erlang couplingComplex objects & relationshipsDynamic reconfigurationTable indexingDistributed Transactions

X XX X XX X X

XX X XX X X

XX X X

XX

Head to Head Feature List

Flavio Ishii, Sept. 17, 2009

Possible Design Permutations

Flavio Ishii, Sept. 17, 2009

More Permutations

• Table access (i.e. public, protected)

• Table type (i.e. set, bag...)

• Table locks

• Distribution via replication and/or table fragmentation

Flavio Ishii, Sept. 17, 2009

Useful Mnesia Functions

• change_table_access_mode/2

• change_table_copy_type/3

• backup/2

• install_fallback/2

• restore/2

• dump_tables/1

• lock/2

• move_table_copy/3

• async_dirty/2

• sync_dirty/2

• add_table_copy/3

• transform_table/3

• change_table_frag/2

• activity/4Flavio Ishii, Sept. 17, 2009

Resources

• Programming Erlang, Joe Armstrong

• ets, dets, qlc, mnesia manuals: • http://www.erlang.org/doc/man/

• Mnesia User’s Guide: • http://www.erlang.org/doc/apps/mnesia

• Mnesia - A Distributed Robust DBMS for Telecommunications Applications, Håkan Mattsson, Hans Nilsson and Claes Wikström

Flavio Ishii, Sept. 17, 2009

Contact Info

USASK - MADMUC Lab

http://flavioishii.com

@flavioishii

Flavio Ishii, Sept. 17, 2009