AT!-WAND Presentation FARCON v1 8-24-16 v2

AT!-WAND Challenge 1

Leveraging Predictive Analytics toIncrease Fast-food Sales

WAND Corporation Analytics Challenge Jan-Apr, 2016

Kevin Church, Co-founder Chuck Gehman, VP of Product Management


Topics

• Introduction to Analyze This!• Introduction to WAND Corporation

and the Analytics Challenge• Analytic Approach• Model Results for 1 Restaurant• Actionable Intelligence• WAND Learnings & Operational Plans


• Twin Cities based Meetup group founded in 2015. Now have 550 members

• Mission: Raise the level/awareness of local Data Science expertise

• Monthly meetings to share best practices in Data Science

• Ongoing quarterly data challenges– Winter challenge with WAND Corporation– Summer challenge with Science Museum of

Minnesota. Grand Finale this afternoon at FARCON!– Fall challenge with _____ kickoff Sept. 28th


Introduction to WAND

• “Highly evolved” 100 person tech company in Eden Prairie

• Building state of the art Point of Sale and Digital Menu Board solutions

• Started working on Big Data platform last year

• Jumped at the opportunity to work with AT on the challenge


AT!-WAND Analytics Challenge• January 27 – April 27, 2016• Predict Daily Net Sale for 50 fast-food

restaurants based on ZIP Code, Hours of Operation– Training dataset 2013-15– Test dataset January 2016

• Teams of up to 5 members, tool agnostic• Six teams competed, three crossed finish line• Panel of judges rates accuracy, creativity,

business intelligence and storytelling


Training Team* Approach• Data cleaning• Cluster analysis• Brainstorm 3rd party data• Assemble analytics dataset• Generate models• Sanity Check• Actionable intelligence

* Scott Sutherland, Will Craft, Jeffrey Van Voorst, Linda Ruetz, Patrick Mobley


Data-Cleaning: Missing Data

Jump to Low Outliers


Cluster Analysis: Closely Correlated Stores

533

507

525

519

506

524

522

504

470

486

483

482

485

480

473

477

479

471

476

468

474

467

518

581

513

809

210

527

464

462

2318528

516

1871

1863

1866512

531

515

1872

1869510

509

461

1868

1865459

458

201

135

32.88

55.26

77.63

100.00

Variables

Simila

rity

Dendrogram, 7 ClustersComplete Linkage, Correlation Coefficient Distance

Jump to PCA Jump to Matrix Plot


Brainstorm 3rd party dataWhich factors are likely to impact Net Sales within a Store?

• Business date extraction• Holidays• Weather (temp, humid, severe)• Local Events (sports, celebrations)• Consumer/Business prices• Little league season

Jump to Dataset Details


Analytic Approach

• Forward Selection• Test residuals for curvature• Plot model residuals vs. time to identify

“shifts”• Research shifts & outliers on Google• Test new features, add to Master dataset• Reduce model via backwards elimination• Delete/Investigate outliers dates


Model Results: Cluster 7, Store 506Johns Creek, GA

Jump to Review Data

Model Summary (44 terms)Sy.x R-sq R-sq(adj)

357.62 90.0% 89.56%

CoefficientsTerm Coef SECoef T-Value P-Val VIFConstant 3358 488 6.88 0DayNo -23.16 5.39 -4.30 0.000 18.4DayNo^2 0.80 0.17 4.73 0.000 19.1DOW_Tue 202.10 41.70 4.85 0.000 1.8DOW_Wed 414.50 41.90 9.88 0.000 1.8DOW_Thu 683.20 42.30 16.14 0.000 1.8DOW_Fri 1011.0 42.20 23.95 0.000 1.8DOW_Sat -1336.8 42.10 -31.74 0.000 1.8DOW_Sun -1666.2 43.50 -38.29 0.000 1.9Month_Jan -512.50 50.30 -10.19 0.000 1.7Month_Feb -138.70 49.80 -2.78 0.005 1.5Month_Mar -107.30 49.20 -2.18 0.030 1.5Month_Apr -166.50 53.80 -3.10 0.002 1.5Month_May 245.80 50.30 4.89 0.000 1.7Month_Jul -377.70 57.00 -6.63 0.000 2.1Month_Aug -323.60 47.60 -6.80 0.000 1.5Month_Sep -347.80 48.40 -7.18 0.000 1.5Month_Oct -138.60 48.40 -2.87 0.004 1.6Month_Nov -223.30 52.40 -4.26 0.000 1.7Year_2014 -75.10 25.50 -2.95 0.003 1.2

CoefficientsTerm Coef SECoef T-Value P-Val VIFNewYr -3244.0 213.00 -15.22 0.000 1.15D_B4PalmFri 267.00 119.00 2.25 0.025 1.1Easter -679.00 259.00 -2.62 0.009 1.1MomDay -1010.00 213.00 -4.74 0.000 1.1DadDay -975.00 212.00 -4.61 0.000 1.1MemDay_SaSu -524.0 153.00 -3.41 0.001 1.1MemDay -3259.0 213.00 -15.31 0.000 1.1Ramadan 213.40 55.70 3.83 0.000 1.84Jul_13-14 -3752.0 258.00 -14.56 0.000 1.04JulWkEnd -1363.0 185.00 -7.36 0.000 1.1LaborDay -2744.0 213.00 -12.88 0.000 1.1Halloween -543.00 213.00 -2.55 0.011 1.1MTW_B4Thx 380.00 129.00 2.95 0.003 1.2Blk_F -3291.0 214.00 -15.34 0.000 1.1Blk_Sa -1022.00 215.00 -4.75 0.000 1.1XmasEve -2821.0 210.00 -13.44 0.000 1.0Dec26 -1885.0 210.00 -8.97 0.000 1.0Dec27-31 -846.00 112.00 -7.59 0.000 1.2MeanVisMiles 20.20 5.48 3.68 0.000 1.1Event2_Snow -810.00 187.00 -4.33 0.000 1.1Event2_ThndrStrm -239.00 101.00 -2.36 0.019 1.0AvgTemp<32 -200.80 94.80 -2.12 0.034 1.2Petrol$ 1190.00 332.00 3.58 0.000 263.8Petrol$^2 -207.80 54.90 -3.78 0.000 263.15Sun -411.00 118.00 -3.49 0.000 1.2


Predict Net Sale for Store

506: SanityCheck

Jump to Date Effects

Jump to Petrol$ Effect


Predict Net Sale for Store 506: - Factors NOT in Model

- Deleted Rows- Is the Model Statistically Valid?

Factors tested but NOT significant …1. Holidays (MLK, Valentines, Presidents, Lent, Columbus, Veterans) 2. CPI$ (Beef, Chicken, Bread, Coffee, Nat.Gas, Electric)3. Weather (Temp, Press, Humid, DewPt, WindSpeed, Precipitation)

No Dates were deleted in creating this model

Residuals are …* Not normally distributed* Not constant w.r.t. time* Constant with respect to X* Constant w.r.t. Ypredict

Jump to Residuals Charts


Analytic Conclusions• Learn a lot from very little by leveraging dates and

ZIP code• Cluster analysis helpful to develop features• Study model residuals to identify previously

unknown influences• R-sqrd not always helpful when comparing models• Linear regression superior to time series analysis if

you want BI• Leverage power of technology to simplify analytics• SME critical to achieving a valid model


Actionable Intelligence1. Southerners lose their appetites when it snows!2. Speaking of snow, take a few more holidays.3. Do Holiday dips suggest competitive sales (e.g.,

Mother’s Day specials/discount)?4. Temp/Humid effects suggest an A/C upgrade is

called for (Store 482 losing $60/d for each 10% increase in humidity)?

5. Could coupons be cannibalizing Net Sale?6. Not all Store Re-Openings are the same!


Suggested WAND Next Steps1. Partner with Store Managers to improve models.

a. Add features to test impact of National sporting events & Fantasy sports,b. Test interaction of wind speed x direction

2. Learn how to monitor model residuals to detects changes in Net Sales.

3. Use models for Fraud/Error detection. Errors? Who can void/comp meals? Free drinks? Make-up orders? Theft when Manager offsite? X-check against employee work schedules? X-check against inventory errors.

4. Offer a consultancy to improve restaurant performance (e.g., dynamic scheduling, dynamic pricing, fact-based purchasing, Holiday “gimmicks”). Target stores with flat or decreasing Net Sales or in States with increasing minimum wages.

5. Run an experiment to see if there is value in restaurants capturing local weather data.

6. Study sales within a day for dynamic staffing.


Q&A


Support Slides


Data-Cleaning: Low Outliers

Jump Back


Principle Components AnalysisInforms the Cluster Analysis

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

% o

f Tot

al V

aria

nce

in N

et S

ales

Number of Principle Components

WAND Stores Net SalesPrinciple Components vs. %Variance

Jump Back


Cluster Analysis: Correlation Distance?

5000

2500

0

4000

2000

0

1000050000

4000

2000

0500025000 400020000

459

522

458

524

459 522

Stores 458, 459 Similarity = 88.58

Stores 522, 524 Similarity = 95.18

WAND Data Challenge, Daily Net Sales from 1/1/13-12/30/14Matrix Plot of Stores 458, 459, 522, 524

Jump Back


Store & Cluster Count Map

Jump Back


Analytics Dataset• Dependent Variable: Daily Net Sales for a Store• Features

– Date related (Day of Week, Day of Week No., Day of Month, Month, Year)

– Holidays (NewYr, MLK_SaSu, MLK, Feb1-7, ValDay, ValDay_WkEnd, PresDay_SaSu, PresDay, Lent, 5D_B4PalmFri , PalmFri, Easter, Passover, MomDay, MemDay_SaSu, MemDay, DadDay, Ramadan, WkEnd>Jul4, Jul4, 4JulWkEnd, LtJlyErlyAug,LbrDay_SaSu, LaborDay, ColumDay, Halloween, VetDay, MTW_B4Thx, Blk_F, Blk_Sa, 10daysB4Xmas, XmasEve, Dec27-31

– Weather (Temp, Humid, DewPt, Wind Speed, Pressure, Precipitation, Visibility, Cloud Cover, Weather Events)

– Events (Local Sports, Homecoming, National Sports)– Consumer Price Index (Beef, Chicken, Bread, Coffee, Petrol,

Natural Gas, Electricity)Jump Back


Predict Net Sale for Store 506: Date Effects

Jump Back


Predict Net Sale for Store 506: Petro Dollars

Jump Back


Predict Net Sale for Store 506: Residuals 4-in-1 Plot

43210-1-2-3-4

99.99

9990

50

10

1

0.01

N 1051AD 4.158P-Value <0.005

Standardized Residual

Perc

ent

60005000400030002000

2.5

0.0

-2.5

-5.0

Fitted Value

Stan

dard

ized

Resid

ual

3.752.501.250.00-1.25-2.50-3.75

150

100

50

0


Freq

uenc

y

1093

100291

182

072

963

854

745

636

527

418

3921

2.5

0.0

-2.5

-5.0

Observation Order

Stan

dard

ized

Resid

ual

Normal Probability Plot Versus Fits

Histogram Versus Order

Residual Plots for NetSales_506P17c, Final Model

Jump Back


Predict Net Sale for Store 506: Residuals Control Chart

Jump Back


Predict Net Sale: Date Extraction DetailsCluster 1 2 3 4 5 6 7Store 459 1863 2318 462 479 482 506Avg Net Sale 4,144$ 3,048$ 5,867$ 4,661$ 3,859$ 4,451$ 4,669$ R-Sqrd 81.1% 70.0% 87.5% 84.7% 69.8% 65.6% 89.6%Sy.x 278$ 255$ 520$ 402$ 307$ 437$ 358$

Day of Month

Day of Week (M-Su)Mon $0 $494 $0 $0 $550 $0 $0Tue $82 $360 $94 $220 $463 $0 $202Wed $297 $277 $332 $0 $309 $0 $415Thu $610 $120 $851 $496 $145 $355 $683Fri $1,205 $191 $1,597 $1,981 $293 $1,011 $1,011Sat $579 $496 $1,084 $1,119 $0 $0 $1,337Sun $213 $0 $0 $1,046 $905 $394 $1,666

Jan $568 $401 $726 $732 $653 $563 $513Feb $230 $0 $0 $0 $0 $258 $139Mar $0 $0 $0 $346 $112 $256 $107Apr $164 $0 $0 $0 $0 $0 $167May $140 $166 $307 $0 $0 $173 $246Jun $0 $106 $245 $480 $0 $191 $0Jul $99 $0 $179 $400 $0 $469 $378Aug $543 $142 $614 $509 $268 $0 $324Sep $417 $250 $753 $674 $364 $0 $348Oct $296 $0 $347 $130 $91 $350 $139Nov $228 $0 $496 $0 $182 $0 $223Dec $0 $206 $0 $0 $0 $192 $0

2013 $0 $197 $1,685 $151 $0 $0 $02014 $122 $0 $0 $387 $95 $0 $752015 $309 $261 $246 $0 $0 $0 $0

Jump Back


Predict Net Sale: Jan-Jun EffectsCluster 1 2 3 4 5 6 7Store 459 1863 2318 462 479 482 506Avg Net Sale 4,144$ 3,048$ 5,867$ 4,661$ 3,859$ 4,451$ 4,669$ R-Sqrd 81.1% 70.0% 87.5% 84.7% 69.8% 65.6% 89.6%Sy.x 278$ 255$ 520$ 402$ 307$ 437$ 358$

NewYr $1,876 $929 $1,404 $655 $1,988 $1,473 $3,244Jan1-7 $696MLK_SaSu $404MLK $1,014Feb1-7 $381 $425 $397 $390 $4031stSuFeb $958ValDayValDay_WkEndPresDay_SaSu $685PresDay $1,066Lent $105 $374 $1965D_B4PalmFri $419 $267PalmFri $830 $637 $707Easter $912 $1,129 $1,042 $1,112 $1,981 $679PassoverApr11-13_2015 $1,061SprngSlmp $238MomDay $735 $930 $672 $806 $1,010MemDay_SaSu $454 $827 $418 $524MemDay $942 $1,077 $838 $1,462 $763 $3,259DadDay $739 $946 $975Ramadan $202 $213


Predict Net Sale: July-Dec EffectsCluster 1 2 3 4 5 6 7Store 459 1863 2318 462 479 482 506Avg Net Sale 4,144$ 3,048$ 5,867$ 4,661$ 3,859$ 4,451$ 4,669$ R-Sqrd 81.1% 70.0% 87.5% 84.7% 69.8% 65.6% 89.6%Sy.x 278$ 255$ 520$ 402$ 307$ 437$ 358$

4dB4Jul4 $6583rdJuly $2,1074thJuly $820 $1,225 $2,531 $1,311 $1,4634Jul_13-14 $3,7525thJuly $1,487WkEnd>Jul4 $751 $742 $1,251 $546 $1,363LtJlyErlyAug $663 $421 $741 $398LbrDay_SaSu $372 $675 $410LaborDay $572 $670 $1,520 $333 $892 $2,744ColumDay $490Halloween $625 $543VetDayMT_B4Thx $570TuB4Thx $502WB4Thx $1,407 $1,446MTW_B4Thx $672 $609 $880 $380TuWThxSaSu $2,513Blk_F $3,291Blk_FSa $923 $916 $389 $1,414Blk_Sa $1,022Dec14-23 $463Dec18-24 $433 $511 $1,162Dec23 $1,298Dec24 $4,340 $2,274 $2,821Dec26 $1,885Dec26-29 $246Dec26-31 $614 $434 $1,474 $364 $472Dec27-31 $846

Jump Back


Predict Net Sale: CPI$ & Weather

Jump Back


Predict Net Sale: Miscellaneous


Coupons Cannibalize Net Sale?

8/26/158/19/158/12/158/5/157/29/157/22/157/15/157/8/157/1/15

1500

1000

500

0

-500

-1000

-1500

BusinessDate

Indi

vidua

l Valu

e

_X=0

UCL=1314

LCL=-1314

+2SL=876

-2SL=-876

+1SL=438

-1SL=-438

1

Results include specified rows: 912:973

I Chart of Store 482 Net Sale RESI29

Jump Back


Monitor Model Residuals: Early Warning System

1/29/1

5

1/22/1

5

1/15/1

51/8

/151/1

/15

1/29/1

5

1/22/1

5

1/15/1

51/8

/151/1

/15

2000

1000

0

-1000

-2000

BusinessDate

Indi

vidua

l Valu

e

_X=0

_X=0

UCL=1581

UCL=574

LCL=-1581

LCL=-574

+2SL=1054

+2SL=383

-2SL=-1054

-2SL=-383

+1SL=527+1SL=191

-1SL=-527-1SL=-191

Net Sale (Actual - Average) Net Sale (Actual - Predicted)I Chart of Store 509 Net Sale (Data vs. Model Variation)

Jump Back


Error/Fraud Detection:

Model Residuals

OK 420-2-4

99.999

90

50

10

10.1

N 91AD 0.153P-Value 0.957

Standardized ResidualPe

rcen

t

50004500400035003000

3.0

1.5

0.0

-1.5

-3.0

Fitted Value

Stan

dard

ized

Resid

ual

210-1-2-3

16

12

8

4

0


Freq

uenc

y

1201101009080706050403020101

3.0

1.5

0.0

-1.5

-3.0

Observation Order

Stan

dard

ized

Resid

ual

Normal Probability Plot Versus Fits

Histogram Versus Order

Residual Plots for NetSales_509P3

1/31/131/28/131/25/131/22/131/19/131/16/131/13/131/10/131/7/131/4/131/1/13

1000

0

-1000

1/31/141/28/141/25/141/22/141/19/141/16/141/13/141/10/141/7/141/4/141/1/14

1000

0

-1000

1/31/151/28/151/25/151/22/151/19/151/16/151/13/151/10/151/7/151/4/151/1/15

1000

0

-1000

BusinessDate

Indi

vidua

l Valu

e

_X=0

UCL=789

LCL=-789

+2SL=526

-2SL=-526

+1SL=263-1SL=-263

2013

_X=0

UCL=789

LCL=-789

+2SL=526

-2SL=-526

+1SL=263-1SL=-263

2014

_X=0

UCL=789

LCL=-789

+2SL=526

-2SL=-526

+1SL=263-1SL=-263

2015

Results exclude specified rows: 94:124

I Chart of Store 509 Net Sale RESI3 by Year


Error Detection: Correction?


Error/Fraud Detection: 4/5 < -$240

1/31/131/28/131/25/131/22/131/19/131/16/131/13/131/10/131/7/131/4/131/1/13

500

0

-500

1/31/141/28/141/25/141/22/141/19/141/16/141/13/141/10/141/7/141/4/141/1/14

500

0

-500

1/31/151/28/151/25/151/22/151/19/151/16/151/13/151/10/151/7/151/4/151/1/15

500

0

-500

BusinessDate

Resid

ual N

et S

ale, A

ct-P

red

$

_X=0

UCL=720

LCL=-720

+2SL=480

-2SL=-480

+1SL=240

-1SL=-240

2013

_X=0

UCL=720

LCL=-720

+2SL=480

-2SL=-480

+1SL=240

-1SL=-240

2014

_X=0

UCL=720

LCL=-720

+2SL=480

-2SL=-480

+1SL=240

-1SL=-240

2015

6

6




Error/Fraud Detection: 8 consecutive < $0

1/31/131/28/131/25/131/22/131/19/131/16/131/13/131/10/131/7/131/4/131/1/13

500

0

-500

1/31/141/28/141/25/141/22/141/19/141/16/141/13/141/10/141/7/141/4/141/1/14

500

0

-500

1/31/151/28/151/25/151/22/151/19/151/16/151/13/151/10/151/7/151/4/151/1/15

500

0

-500

BusinessDate

Indi

vidua

l Valu

e

_X=0

UCL=729

LCL=-729

+2SL=486

-2SL=-486

+1SL=243

-1SL=-243

2013

_X=0

UCL=729

LCL=-729

+2SL=486

-2SL=-486

+1SL=243

-1SL=-243

2014

_X=0

UCL=729

LCL=-729

+2SL=486

-2SL=-486

+1SL=243

-1SL=-243

2015

222



Jump Back


Data Review: Store 506

1st Quartile 3705.1Median 4950.33rd Quartile 5451.7Maximum 7528.6

4601.8 4735.7

4886.2 5002.5

1063.9 1158.7

A-Squared 22.45P-Value <0.005Mean 4668.8StDev 1109.3Variance 1230558.6Skewness -0.505620Kurtosis -0.611348N 1057Minimum 1526.6

Anderson-Darling Normality Test

95% Confidence Interval for Mean

95% Confidence Interval for Median

95% Confidence Interval for StDev

700060005000400030002000

Median

Mean

50004900480047004600

95% Confidence Intervals

Summary Report for NetSales_506

8000

7000

6000

5000

4000

3000

2000

1000

NetSa

les_506 5217

3185

2013201420152016

YearTime Series Plot of NetSales_506

Jump Back


Store Re-Openings

12/31/

13

12/27/

13

11/27/

13

10/28/

13

9/28/1

3

8/29/1

3

7/30/1

3

6/30/1

3

5/31/1

35/1

/134/1

/133/2

/13

1/31/1

31/1

/13

6000

5000

4000

3000

2000

1000

0

BusinessDate

522

Time Series Plot of Store 522 Net Sale

12/29/

159/2

9/15

6/30/1

53/3

1/15

12/30/

149/3

0/14

7/1/14

4/1/14

12/31/

1310/

1/13

7/2/13

4/2/13

1/1/13

18000

16000

14000

12000

10000

8000

6000

4000

2000

BusDate

2318

_Net

$4349

201320142015

Year

Time Series Plot of 2318_Net$

Jump Back


Store Location & Clusters Map

1-red, 2-blue, 3-green, 4-cyan, 5-brown, 6-orange, 7-purple

Jump to Map 2


Predict Net Sale: 1 Model per ClusterDate Extraction Effects

Jump to Details


Predict Net Sale: 1 Model per ClusterHoliday Effects

Jump to Details


Actionable Intelligence1. Southerners lose their appetites when it snows!2. Speaking of snow, take a few more holidays.3. Do Holiday dips suggest competitive sales (e.g.,

Mother’s Day specials/discount)?4. Temp/Humid effects suggest an A/C upgrade is

called for (Store 482 losing $60/d for each 10% increase in humidity)?

5. Could coupons be cannibalizing Net Sale?6. Not all Store Re-Openings are the same!


Suggested WAND Next Steps1. Partner with Store Managers to improve models.

a. Add features to test impact of National sporting events & Fantasy sports,b. Test interaction of wind speed x direction

2. Learn how to monitor model residuals to detects changes in Net Sales.

3. Use models for Fraud/Error detection. Errors? Who can void/comp meals? Free drinks? Make-up orders? Theft when Manager offsite? X-check against employee work schedules? X-check against inventory errors.

4. Offer a consultancy to improve restaurant performance (e.g., dynamic scheduling, dynamic pricing, fact-based purchasing, Holiday “gimmicks”). Target stores with flat or decreasing Net Sales or in States with increasing minimum wages.

5. Run an experiment to see if there is value in restaurants capturing local weather data.

6. Study sales within a day for dynamic staffing.

Documents

AT!-WAND Presentation FARCON v1 8-24-16 v2