Download pptx - Possible Visions for Mahout 1.0

© 2014 MapR Technologies 1

What’s Coming in Mahout 1.0?

Ted Dunning, Chief Application ArchitectMapR Technologies




A typical encounter with a potential Mahout

user


Which leads us to

the Mahout 1.0 vision





Example: Cooccurrence Analysis


How often do items co-occur?// load distributed matrixval A = drmFromHDFS(...)

// compute co-occurrencesval C = A.t %*% A


How often do items co-occur?// load distributed matrixval A = drmFromHDFS(...)

// compute co-occurrencesval C = A.t %*% A

Under the covers:

Optimizer rewrites the matrix multiplication and transpose operations to a TransposeSelf operator

Optimizer chooses from two physical operators for TransposeSelf


Which items co-occur anomalously?

// compute & broadcast number // of interactions per itemval numInteractions =

drmBroadcast(A.colSums)

// create indicator matrixval I = C.mapBlock() { case (keys, block) =>

// allocate sparse block of indicator matrix val indicatorBlock = sparse(block.nrow, block.ncol) // compute indicators with loglikelihood ratio test for (row <- block)

indicatorBlock(row.index,::) = computeLLR(row,numInteractions) keys -> indicatorBlock

}


Runtime

• prototype on Apache Spark– fast and expressive cluster

computing system– general computation graphs, in-memory primitives, rich API, interactive

shell

• future: add Stratosphere– project proposed to

Apache Incubator recently– similar to Apache Spark, adds data flow optimization and efficient out-

of-core execution




How Does This Apply?


How Can I Start?


Q & A

@ted_dunning @mapr maprtech

[email protected]

Engage with us!

MapR

maprtech

mapr-technologies
