1

(1)2013 Fall11

2006~ 2008~ 1999~2009100 MVP2020122Copyright Proprietary and Confidential. All rights reserved.2 3Copyright Proprietary and Confidential. All rights reserved.Trend of Big DataBig Data IBM90%2GoogleFacebook Big Data capture storage search analytics

4Copyright Proprietary and Confidential. All rights reserved."Data Scientist : The sexist job of the 21st century", Harvard Business Review, Oct 2012 5

Copyright Proprietary and Confidential. All rights reserved.Obama Administration : Big Data is a Big Deal6

201232 Big DataBig Data( 2013/5/12)Copyright Proprietary and Confidential. All rights reserved.Big Data

Source : IBM 2012CEO7User data, Transaction data, Social data, Machine dataCopyright Proprietary and Confidential. All rights reserved.User data, transaction data, social data, machine data7Big Data

Source : IBM Big Data Hub8Copyright Proprietary and Confidential. All rights reserved.Big Data

Source : IBM 2012CEO9Copyright Proprietary and Confidential. All rights reserved.Business IntelligenceCopyright Proprietary and Confidential. All rights reserved.:Copyright Proprietary and Confidential. All rights reserved.Marketing and CRM Cycle

Data WarehousingData MiningE-MarketingCopyright Proprietary and Confidential. All rights reserved. 13Copyright Proprietary and Confidential. All rights reserved.Big data 14

Copyright Proprietary and Confidential. All rights reserved.Twitter200 million tweets per dayPeak 10,000 per secondHow to analyze the data ?

Zynga"Analytics company, not a gaming company230 million players per monthHarvest 15TB data per daytest new featurestarget advertising15

4U box = 40 TB1PB = 25 boxesCopyright Proprietary and Confidential. All rights reserved.Facebook6 billion messages per day2 PB (compressed) online6 PB replication250 TB growth per monthCassandra / HBase architecture

Copyright Proprietary and Confidential. All rights reserved.eBay

Analyze & ReportDiscover & Explore

Copyright Proprietary and Confidential. All rights reserved.Big data Data Mining Text Mining 18Copyright Proprietary and Confidential. All rights reserved.(1) Data MiningCopyright Proprietary and Confidential. All rights reserved. Data Mining Clustering Classification Association rules Sequential pattern analysis20Copyright Proprietary and Confidential. All rights reserved.

{2,5} {2,3,5} Copyright Proprietary and Confidential. All rights reserved.1. Clustering

Copyright Proprietary and Confidential. All rights reserved. K-means (K=2)23Pick seedsReassign clustersCompute centroidsxxReassign clustersxxxxCompute centroidsReassign clustersConverged! (similarity)34Copyright Proprietary and Confidential. All rights reserved.2. Classification

Copyright Proprietary and Confidential. All rights reserved. :

Copyright Proprietary and Confidential. All rights reserved. : ()

Copyright Proprietary and Confidential. All rights reserved. Weather Data: Play tennis or not ?27

Copyright Proprietary and Confidential. All rights reserved.Which attribute to choose ?28

Copyright Proprietary and Confidential. All rights reserved.Which attribute to choose ?choose the attribute that produces the "purest" nodesand more informative Information gain (ID3, C4.5, C5)29ig(outlook) = average(3/5, 4/4, 3/5) = 0.73ig(humidity) = average(4/7, 6/7) = 0.71ig(windy) = average(6/8, 3/6) = 0.63ig(temperature) = average(2/4, 4/6, 3/4) = 0.64

Copyright Proprietary and Confidential. All rights reserved.30

outlookCopyright Proprietary and Confidential. All rights reserved.3. Association rulesBuy (milk) Buy (bread) 80 %Nokia N95 80-120 60 %Support and Confidence

Copyright Proprietary and Confidential. All rights reserved. : AprioriFP growth32

Copyright Proprietary and Confidential. All rights reserved.4. Sequential patternStar War Empire Strikes Back Return of the JediCopyright Proprietary and Confidential. All rights reserved. Sequential pattern ()

Copyright Proprietary and Confidential. All rights reserved. Sequential pattern ()Jurassic Park Toy Story , Jurassic Park 2 : Lost WorldJurassic Park Terminator 2 : Judgment DayCopyright Proprietary and Confidential. All rights reserved. (1) 1 : 2 : 201036Copyright Proprietary and Confidential. All rights reserved. () 1 (Association) , , DVD : 37Copyright Proprietary and Confidential. All rights reserved. () 2 (Clustering) (Female, 30~40) (Employee, 500K~800K yearly)Direct Marketing : DM 38Copyright Proprietary and Confidential. All rights reserved. (), DM 100 DM 10-12 2 % DM 20 8-10 %39Copyright Proprietary and Confidential. All rights reserved.

ConversionFunnelCopyright Proprietary and Confidential. All rights reserved. (2) ()350 1 : 2 : 35025 ID, account, points, email, sex, nickname, birthday, join_date, job 41Copyright Proprietary and Confidential. All rights reserved. () 1 (Association) AddressTaipei (Confidence 63.5%)

42Copyright Proprietary and Confidential. All rights reserved. () 2 (Clustering)350Group 155.78: Group 219.53: Group 37.05: 80%43Copyright Proprietary and Confidential. All rights reserved. (3) : Peak alarm () Region alarm () Copyright Proprietary and Confidential. All rights reserved. 45Copyright Proprietary and Confidential. All rights reserved.46Copyright Proprietary and Confidential. All rights reserved. Excel Access 47Copyright Proprietary and Confidential. All rights reserved.1. 48Copyright Proprietary and Confidential. All rights reserved.

49Copyright Proprietary and Confidential. All rights reserved.

50Copyright Proprietary and Confidential. All rights reserved.1. 2. 3. 4.

51Copyright Proprietary and Confidential. All rights reserved.1. WebMobile2. 3.

52Copyright Proprietary and Confidential. All rights reserved.2. 53Copyright Proprietary and Confidential. All rights reserved. 1 : 2 : (, 300-400) (: M, : F )() (: F, 30-40, : F, 0-10 )() ()54Copyright Proprietary and Confidential. All rights reserved.(, 300-400) (: M, : F )() (: F, 30-40, : F, 0-10 ) : () ()55Copyright Proprietary and Confidential. All rights reserved.3. 56Copyright Proprietary and Confidential. All rights reserved.100057Copyright Proprietary and Confidential. All rights reserved.

58Copyright Proprietary and Confidential. All rights reserved.52% : : , 30-40, 100-12028% : : , > 213% : , , 1. 2. 3.

( : , 30-40, 100-120) (1000) 59Copyright Proprietary and Confidential. All rights reserved. 60Copyright Proprietary and Confidential. All rights reserved. 1 STEP 1 : STEP 2 : - , , 3C - , , - , , - and 74.3 % 42.8 % PN 65.2 %61Copyright Proprietary and Confidential. All rights reserved. 1 ()STEP 3 : : () : and () : PN ()

62Copyright Proprietary and Confidential. All rights reserved. 2 STEP 1 : STEP 2 : , 25-35, 63.2 % , 25-40 74.1 % or 59.8 % 45.3 %63Copyright Proprietary and Confidential. All rights reserved. 2 ()STEP 3 : , 25-35 63.2 % , 25-40, 74.1 % or 59.8 % 45.3 %DIY

64Copyright Proprietary and Confidential. All rights reserved. 6565Sheet1

Sheet2User_IDIncomeStudentCredit-ratingClass : Buy-computer1HighNoFairNo2HighNoExcellentNo8MediumNoFairNo9LowYesFairYes11MediumYesExcellentYesUser_IDIncomeStudentCredit-ratingClass : Buy-computer3HighNoFairYes7LowYesExcellentYes12MediumNoExcellentYes13HighYesFairYesUser_IDIncomeStudentCredit-ratingClass : Buy-computer4MediumNoFairYes5LowYesFairYes6LowYesExcellentNo10MediumYesFairYes14MediumNoExcellentNo

Sheet3NameAgeIncomeCredit-ratingSandy Jones40MediumFairAndre Beau31-40HighExcellent....NameAgeIncomeCredit-ratingFrank Jones>40HighFairSylvia Crest40HighFairSylvia Crest