Embed Size (px)
DESCRIPTION
Citation preview
MongoDB World 2014
by S.D.O.C. Ltd.Billing on Top of MongoDBOfer Cohen
Who Am I ?
● Open source evangelist
● Former Board member of OpenSourceMatters
(Non-profit-org behind Joomla)
● S.D.O.C. Ltd. Co-Founder
What are we doing?
● We increase business success through great
open source technologies and services○ Open & Transparent
○ Don't reinvent the wheel
○ High Quality
○ Lean & Effective
How did we get to billing (history)● Client: Golan Telecom (Israel)
How did we get to billing (history)● Client: Golan Telecom
○ New & lean player in the market
○ 0=>~1M subscribers in 4 years
○ Limited resources & love open source
How did we get to billing (history)● Client: Golan Telecom
○ Start environment from Day 1
○ Short and aggressive time to market
○ Unlimited plan for ~25$
○ Customer can do almost everything in the website
■ Obviously requires 24/7 uptime
How did we get to billing (history)● Start with Anti-Fraud solution
● 2 Different data structure, 2 separated tables○ Outgoing calls (MOC)
○ incoming calls (MTC)
○ SMS (duration=0 means SMS)
Anti-Fraud in RDBMS How was it look like? Example code... $base_query = "SELECT imsi FROM moc WHERE callEventStartTimeStamp >=" . $start
. " UNION SELECT imsi FROM mtc WHERE callEventStartTimeStamp >=" . $start
$base_query = "SELECT imsi FROM (" . $base_query . ") AS qry ";
if (isset($args['imsi']))
$base_query .= "WHERE imsi = '" . $this->_connection->real_escape_string($args['imsi']) . "'";
$base_query .= "GROUP BY imsi ";
$mtc_join_query = "SELECT 'mtc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(callingNumber)<=10, callEventDuration, 0))) AS israel_duration "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(callingNumber)<=10, CEILING(callEventDuration/60)*60, 0))) AS
israel_duration_round "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "
. "FROM mtc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";
Anti-Fraud in RDBMS How was it look like? Example code...
$moc_join_query = "SELECT 'moc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(connectedNumber)<=10, callEventDuration, 0))) AS israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(connectedNumber)<=10, CEILING(callEventDuration/60)*60, 0))) AS israel_duration_round "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "
. "FROM moc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";
Anti-Fraud in RDBMS How was it look like? Example code... $group_query = "SELECT base.imsi, moc.duration AS moc_duration, moc.charge AS moc_charge, "
. "mtc.duration AS mtc_duration, mtc.charge AS mtc_charge, "
. "mtc.duration_round AS mtc_duration_round, moc.duration_round AS moc_duration_round, "
. "moc.israel_duration AS moc_israel_duration, moc.non_israel_duration AS moc_non_israel_duration, "
. "moc.israel_duration_round AS moc_israel_duration_round, moc.non_israel_duration_round AS moc_non_israel_duration_round, "
. "mtc.israel_duration AS mtc_israel_duration, mtc.non_israel_duration AS mtc_non_israel_duration, "
. "mtc.israel_duration_round AS mtc_israel_duration_round, mtc.non_israel_duration_round AS mtc_non_israel_duration_round, "
. "mtc.sms_count AS mtc_sms_count, moc.sms_count AS moc_sms_count "
. "FROM "
. "( " . $base_query . " ) AS base "
. " LEFT JOIN (" . $mtc_join_query . " ) AS mtc ON base.imsi = mtc.imsi "
. " LEFT JOIN (" . $moc_join_query . " ) AS moc ON base.imsi = moc.imsi " ;
if (isset($args['limit'])) {
$limit = (int) $args['limit'];
} else {
$limit = 100000;
}
Anti-Fraud in RDBMS How was it feel…?
After moving to Mongo
● One main collection for 2 types
● Aggregate much more simple
After moving to Mongo$base_match = array(
'$match' => array('source' => 'nrtrde','unified_record_time' => array('$gte' => new MongoDate($charge_time)),
));$where = array(
'$match' => array('record_type' => 'MOC','connectedNumber' => array('$regex' => '^972'),'event_stamp' => array('$exists' => false),'deposit_stamp' => array('$exists' => false),'callEventDurationRound' => array('$gt' => 0), // not sms
),);$group = array(
'$group' => array("_id" => '$imsi',"moc_israel" => array('$sum' => '$callEventDurationRound'),'lines_stamps' => array('$addToSet' => '$stamp'),
),);$project = array(
'$project' => array('imsi' => '$_id','_id' => 0,'moc_israel' => 1,'lines_stamps' => 1,
),);
After moving to Mongo$having = array(
'$match' => array(
'moc_israel' => array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.moc.israel'))
),
);
$moc_israel = $lines->aggregate($base_match, $where, $group, $project, $having);//sms out to all numbers
$where['$match']['record_type'] = 'MOC';
$where['$match']['callEventDurationRound'] = 0;
$group['$group']['sms_out'] = $group['$group']['mtc_all'];
unset($group['$group']['mtc_all']);
unset($having['$match']['mtc_all']);
$group['$group']['sms_out'] = array('$sum' => 1);
$having['$match']['sms_out'] = array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.smsout'));
$project['$project']['sms_out'] = 1;
unset($project['$project']['mtc_all']);
$sms_out = $lines->aggregate($base_match, $where, $group, $project, $having);
What is billing?
● Group of processes of communications service providers
● responsible to collect consumption data● calculate charging and billing information● produce bills to customers● process their payments and manage debt
collectionWikipedia
What is billing?
● This is how we see it
● The KISS way
Telecom Today - RDBMS challenges
● Telecom situation world wide today:
○ Unlimited packages, extend 3g usage, with high
competition
○ High-volume - 4g, LTE
○ Different Events - different data structure
5 *main* data-structures from different sources● NSN - Calls● SMSC - SMS● MMSC - MMS● GGSN - Data● TAP3 - International usage
Billing and MongoDB - Pros
NSN Record> db.lines.findOne({"usaget" : "call", aid:XXXXXXX})
{
"_id" : ObjectId("52bafd818f7ac3943a8b96dc"),
"stamp" : "87cea5dec484c8f6a19e44e77a2e15b4",
"record_type" : "01",
"record_number" : "13857000",
"record_status" : 0,
"exchange_id" : "000006270000",
"call_reference" : "700227d9a3",
"urt" : ISODate("2013-12-25T15:09:15Z"),
"charging_start_time" : "20131225170915",
"charging_end_time" : "20131225171517",
"call_reference_time" : "20131225170914",
"duration" : 362,
"calling_number" : "972546918666",
"called_number" : "26366667",
"call_type" : "3",
"chrg_type" : "0",
"called_imsi" : "000030000000000",
"imsi" : "000089000101076",
"in_circuit_group_name" : "",
"in_circuit_group" : "",
"out_circuit_group_name" : "NBZQZA8",
"out_circuit_group" : "0503",
"tariff_class" : "000000",
"called_number_ton" : "6",
"org_dur" : 362,
"type" : "nsn",
"source" : "binary",
"file" : "CF0322.DAT",
"log_stamp" : "0a3ffdb7c9ccc175e38be2fcc00f8c28",
"process_time" : "2013-12-25 17:35:22",
"usaget" : "call","usagev" : 362,
"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f0001c9")),
"aid" : XXXXXXX,
"sid" : YYYYY,
"plan" : "LARGE",
"aprice" : 0,
}
SMS Record> db.lines.findOne({"usaget" : "sms", aid:XXXXXXX})
{
"_id" : ObjectId("52e52ff1d88db071648b4ad7"),
"stamp" : "9328f3aaa114aaba910910053a11b3e8",
"record_type" : "1",
"calling_number" : "000972546918666",
"calling_imsi" : "000089200000000",
"calling_msc" : "000972000000000",
"billable" : "000000000000000",
"called_number" : "000972547655380",
"called_imsi" : "425089109386379",
"called_msc" : "000972586279101",
"message_submition_time" : "140125192517",
"time_offest" : "02",
"message_delivery_time" : "140125192519",
"time_offest1" : "02",
"cause_of_terminition" : "100",
"call_reference" : "5137864939035049",
"message_length" : "050",
"concatenated" : "1",
"concatenated_from" : "09",
"source" : "separator_field_lines",
"type" : "smsc",
"log_stamp" : "a5950686e364d1400c13dd1857c3340e",
"file" : "140125192403_5735golan.cdr",
"process_time" : "2014-01-26 17:53:21",
"urt" : ISODate("2014-01-25T17:25:17Z"),
"usaget" : "sms","usagev" : 1,
"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f0001db")),
"aid" : XXXXXXX,
"sid" : YYYYY,
"plan" : "LARGE",
"aprice" : 0,
"usagesb" : 4,
"billrun" : "201402"
}
Data Record> db.lines.findOne({"usaget" : "data", aid:XXXXXXX})
{
"_id" : ObjectId("539076678f7ac34a1d8b9367"),
"stamp" : "8a9f891ec85c5294c974a34653356055",
"imsi" : "400009209100000",
"served_imsi" : "400009209100000",
"ggsn_address" : "XX.XX.144.18",
"charging_id" : "2814645234",
"sgsn_address" : "XX.XX.145.9",
"served_pdp_address" : "XX.XX.237.95",
"urt" : ISODate("2014-06-05T09:34:47Z"),
"record_opening_time" : "20140605123447",
"ms_timezone" : "+03:00",
"node_id" : "GLTNGPT",
"served_msisdn" : "00002546918666",
"fbc_uplink_volume" : 61298,
"fbc_downlink_volume" : 217304,
"rating_group" : 0,
"type" : "ggsn",
"source" : "binary",
"file" : "GLTNGPT_-_0000056580.20140605_-_1251+0300",
"log_stamp" : "45a227ced1098bc76a44774eae04eb67",
"process_time" : "2014-06-05 16:39:40",
"usaget" : "data","usagev" : 278602,
"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f000200")),
"apr" : 0.020264916503503202,
"aid" : XXXXXXX,
"sid" : YYYYY,
"plan" : "LARGE",
"aprice" : 0,
"usagesb" : 478682116,
"billrun" : "201406"
}
TAP3 Record (intl roaming)> db.lines.findOne({type:"tap3", aid:9073496}){
"_id" : ObjectId("538d9ac98f7ac3e17d8b4fd6"),"stamp" : "8f6cdc8662307ee2ed951ce640a585b5","basicCallInformation" : {
"GprsChargeableSubscriber" : {"chargeableSubscriber" : {
"simChargeableSubscriber" : {"imsi" : "400009209100000"
}},"pdpAddress" : "XX.XX.227.158"
},"GprsDestination" : {
"AccessPointNameNI" : "internet.golantelecom.net.il"},"CallEventStartTimeStamp" : {
"localTimeStamp" : "20140529205131","TimeOffsetCode" : 0
},"TotalCallEventDuration" : 163
},"LocationInformation" : {
"gprsNetworkLocation" : {"RecEntityCodeList" : {
"RecEntityCode" : ["","\u0000"
]},"LocationArea" : 00001,"CellId" : 0001
},"GeographicalLocation" : {
"ServingNetwork" : "MMMM"}
},"ImeiOrEsn" : false,
"GprsServiceUsed" : {"DataVolumeIncoming" : 195120,"DataVolumeOutgoing" : 48600,"ChargeInformationList" : {
"ChargeInformation" : { "ChargedItem" : "X", "ExchangeRateCode" : 0, "ChargeDetailList" : { "ChargeDetail" : {
"ChargeType" : "00", "Charge" : 001, "ChargeableUnits" : 100000, "ChargedUnits" : 100000, "ChargeDetailTimeStamp" : { "localTimeStamp" : "20140529205131", "TimeOffset" : 0 } }
}}
}},"OperatorSpecInfoList" : {
"OperatorSpecInformation" : ["00000000.0000","00000000.0000","00000000.0000"
]},"record_type" : "e","urt" : ISODate("2014-05-29T18:51:31Z"),"tzoffset" : "+0200","imsi" : "400009209100000","serving_network" : "DEUE2","sdr" : 0.0001,"exchange_rate" : 1.12078,"type" : "tap3","file" : "CDBELHBISRGT02253",
"log_stamp" : "a0ad109c6e795f6c1feeef9ef649d937","process_time" : "2014-06-03 12:50:08",
"usaget" : "data","usagev" : 243720,"arate" : DBRef("rates", ObjectId
("521e07fed88db0e73f000219")),"apr" : 0.46640616,"aid" : 9073496,"sid" : 78288,"plan" : "LARGE","out_plan" : 243720,"aprice" : 0.46640616,"usagesb" : 39139746,"billrun" : "201406"
}
Billing and MongoDB - ProsLoose coupling and loose traditional billing components
Oldw/o MongoDb New
with MongoDb
Billing and MongoDB - Pros
● Call can be separated CDRs● BillRun unify records only on export or
presentation layer you are aggregate not in the DB.
● No need for mediation● Can monitor CDR level and use billing in one
system
Billing and MongoDB - Pros
Sophisticated rating module
● Rating module depends on CDR structure
● Easy to implement recurring rating
MongoDB advantages with Billing
Invoice JSON2PDF process● Use json as metadata for the PDF
● Easy to export
● Fast processing
MongoDB advantages with BillingInvoice metadata
> db.billrun.findOne({billrun_key:"201405", aid:9073496}){ "aid" : NumberLong(9073496), "subs" : [ { "sid" : NumberLong(78288), "subscriber_status" : "open", "current_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3dd")), "next_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3dd")), "kosher" : false, "breakdown" : { "in_plan" : { "base" : { "INTERNET_BILL_BY_VOLUME" : { "totals" : { "data" : { "usagev" : NumberLong(1975547725), "cost" : NumberLong(0), "count" : NumberLong(1352) } }, "vat" : 0.18 }, "IL_MOBILE" : { "totals" : { "sms" : { "usagev" : NumberLong(159), "cost" : NumberLong(0), "count" : NumberLong(159) }, "call" : { "usagev" : NumberLong(20788), "cost" : NumberLong(0), "count" : NumberLong(83) } }, "vat" : 0.18 },
"IL_FIX" : { "totals" : { "call" : { "usagev" : NumberLong(1217), "cost" : NumberLong(0), "count" : NumberLong(16) } }, "vat" : 0.18 }, "INTERNAL_VOICE_MAIL_CALL" : { "totals" : { "call" : { "usagev" : NumberLong(60), "cost" : NumberLong(0), "count" : NumberLong(2) } }, "vat" : 0.18 }, "service" : { "cost" : 83.898305085, "vat" : 0.18 } }, "intl" : { "KT_USA_NEW" : { "totals" : { "call" : { "usagev" : NumberLong(149), "cost" : NumberLong(0), "count" : NumberLong(2) } }, "vat" : 0.18 } } },
MongoDB advantages with BillingInvoice metadata
"credit" : { "refund_vatable" : { "CRM-REFUND_PROMOTION_1024-BILLRUN_201405" : -33.898305084746 } } }, "lines" : { "data" : { "counters" : { "20140425" : { "usagev" : NumberLong(11991615), "aprice" : NumberLong(0), "plan_flag" : "in" }, …. /* data by date really long, so let’s cut it from this demonstration */ } } }, "totals" : { "vatable" : 50.000000000254005, "before_vat" : 50.000000000254005, "after_vat" : 59.00000000029973 }, "costs" : { "credit" : { "refund" : { "vatable" : -33.898305084746 } }, "flat" : { "vatable" : 83.898305085 } } }, { "sid" : NumberLong(354961), "subscriber_status" : "open", "current_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3de")), "next_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3de")),
"kosher" : false, "breakdown" : { "in_plan" : { "base" : { "service" : { "cost" : 8.466101695, "vat" : 0.18 } } } }, "totals" : { "vatable" : 8.466101695, "before_vat" : 8.466101695, "after_vat" : 9.9900000001 }, "costs" : { "flat" : { "vatable" : 8.466101695 } } } ], "vat" : 0.18, "billrun_key" : "201405", "totals" : { "before_vat" : 58.466101695254004, "after_vat" : 68.99000000039973, "vatable" : 58.466101695254004 }, "_id" : ObjectId("5382fd3cd88db0c31a8b74cc"), "invoice_id" : NumberLong(14738102), "invoice_file" : "201405_009073496_00014738102.xml"}
Infrastructure
BETAFirst Production
Today
Data center 1
Data center 2
Data center 1
Data center 2 Data center 1
Data center 2BillRun application
BillRun core
PHP Yaf framework
Mongodloid, Zend and more
libraries
MongoDB
App stackWeb service Cli/Cron services
Pay Attention! What to keep in mind
Transactional (tx) processes● write concern - acknowledged (default in 2.4)● findAndModify (A.K.A FAM) - document tx● value=oldValue● 2 phase commit - app side● Transaction lock by design
High performance tricks● With SSD you get x20 more performance● MongoDB loves RAM● Follow MongoDB production notes
○ Readahead low as possible○ THP - transparent hugepages
Pay Attention! What to keep in mind
TCO - MongoDB based solution
● 3 Months dev
● 3 Months QA
● Few days of maintenance infra and app
● Easy to scale
● Easy to add features
