Embed Size (px)
Citation preview
ShareInsights - An Unified Approach to Full-stack DataProcessing
Mukund DeshpandePersistent Systems Ltd.
’Bhageerath’, Senapati BapatRoad,
Pune 411005, INDIAmukund_deshpande@
persistent.com
Dhruva RayPersistent Systems Ltd.
’Bhageerath’, Senapati BapatRoad,
Pune 411005, [email protected]
Sameer DixitPersistent Systems Ltd.
’Bhageerath’, Senapati BapatRoad,
Pune 411005, [email protected]
Avadhoot AgastiPersistent Systems Ltd.
’Bhageerath’, Senapati BapatRoad,
Pune 411005, INDIAavadhoot_agasti@
persistent.com
ABSTRACTThe field of data analysis seeks to extract value from data foreither business or scientific benefit. This field has seen a re-newed interest with the advent of big data technologies anda new organizational role called data scientist. Even withthe new found focus, the task of analyzing large amounts ofdata is still challenging and time-consuming.
The essence of data analysis involves setting up data pipe-lines which consists of several operations that are chainedtogether - starting from data collection, data quality checks,data integration, data analysis and data visualization (in-cluding the setting up of interaction paths in that visualiza-tion).
In our opinion, the challenges stem from from the tech-nology diversity at each stage of the data pipeline as well asthe lack of process around the analysis.
In this paper we present a platform that aims to signifi-cantly reduce the time it takes to build data pipelines. Theplatform attempts to achieve this in following ways.
1. Allow the user to describe the entire data pipeline witha single language and idioms - all the way from dataingestion to insight expression (via visualization andend-user interaction).
2. Provide a rich library of parts that allow users to quicklyassemble a data analysis pipeline in the language.
3. Allow for a collaboration model that allows multipleusers to work together on a data analysis pipeline as
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected]’15, May 31–June 4, 2015, Melbourne, Victoria, Australia.Copyright is held by the owner/author(s). Publication rights licensed to ACM.ACM 978-1-4503-2758-9/15/05 ...$15.00.http://dx.doi.org/10.1145/2723372.2742800.
well as leverage and extend prior work with minimaleffort.
We studied the efficacy of the platform for a data hackathoncompetition conducted in our organization. The hackathonprovided us with a way to study the impact of the approach.Rich data pipelines which traditionally took weeks to buildwere constructed and deployed in hours. Consequently, webelieve that the complexity of designing and running thedata analysis pipeline can be significantly reduced; leadingto a marked improvement in the productivity of data ana-lysts/data scientists.
Categories and Subject DescriptorsH.4 [Information Systems Applications]: Miscellaneous;D.3.2 [Language Classifications]: [Data-flow languages]
General TermsManagement, Design, Experimentation, Human Factors, Lan-guages.
KeywordsBig data; Data analysis; Data visualization; Data pipeline
1. INTRODUCTIONThe field of data analysis has seen dramatic changes in
the last few years. These changes are seen primarily onthe dimensions of data, evolving user needs and technologychanges.
On the data side of the equation, the volume of data thatneeds to be analyzed has become extremely large. It isnow routine to analyze terabytes of data and in some cases,petabytes as well. The nature of the data is also multi-structured with text, audio as well as visual data being ana-lyzed side-by-side with structured tabular data. In line withan increased amount of public data being made available,
the need for integrating it with private organization datahas never been higher.
On the consumption side of the equation, users are makingincreasing demands on the data visualization component ofdata analysis. With the advent of multiple devices and formfactors, data needs to be visualized and analyzed at multipleresolutions, possibly in environments with varying networkconnectivity. Also, the bar for aesthetics and ease of use fordata visualization has gone up significantly. Users are de-manding richer and more interesting widgets for displayingdata and intuitive ways of interacting with them. Finally,with users being increasingly connected on networks (enter-prise and social), the ability to share and collaborate withtheir peers for building and running the data pipeline is alsobecoming crucial.
On the technology side of the equation, the last few yearshave seen several technologies reaching mainstream - bothbig data technologies as well as advanced use of JavaScriptfor data visualization. Though big data technologies al-low analyzing large data-sets, they have also brought anadditional complexity of multiple technologies and differentprogramming paradigms. Additionally, at every boundary,there remain integration challenges. Concomitantly, datacrunching libraries in Python and R are seeing a revived in-terest and there appears to be a renewed interest in usingthese libraries in the data pipeline.
In our opinion, all of these challenges have made the taskof doing data analysis increasingly difficult and time con-suming.
The essence of data analysis involves setting up data pipe-lines which consists of several operations that are chainedtogether - starting from data collection, data quality checks,data integration, data analysis and data visualization. (in-cludes setting up interaction paths in that visualization)
Since, data pipelines operates on electronic media, thetask of running and changing a data analysis pipeline is rel-atively easy; Consequently, in most cases, the data analysispipeline is built and stabilized by incrementally building thepipeline. This iterative process mandates that it be veryeasy to change and edit an existing data pipeline. Sadly,that is not the state of the affairs[4].
In this paper, we present a platform which attempts to sig-nificantly ease the process of building complete data pipelinesfor businesses and data scientists. The platform and thesupposed benefits were tested by running a data hackathon.Data pipelines and dashboards which took weeks to imple-ment were completed in under six hours.
The paper is organized as follows. In section 2, we explaincurrent approaches and the challenges faced in building datapipelines. In section 3, we illustrate key concepts of the uni-fied representation provided on the ShareInsights platformby building complex data pipelines. Section 4 describes thekey building blocks and features provided by the platform.Section 5 reports the findings of the hackathon used to ver-ify the utility of the platform, Finally we conclude the paperwith observations and future areas of work.
2. BUILDING A DATA ANALYSIS PIPELINEIn this section, we outline the current approaches to build-
ing data pipelines and the challenges faced by businesses anddata scientists.
2.1 Current Approaches
2.1.1 Data Warehouse stackThis stack uses an extract-transform-load (ETL) mecha-
nism to ingest, clean, transform and ultimately load pro-cessed data into the data warehouse[8]. Reporting toolsquery the warehouse to give reports to the end-user. Thetypical architecture is shown in figure 1.
Figure 1: BI Architecture for data pipelines
This configuration is widely used and there is a relativematurity in the tooling and understanding of the data pipelineprocess. Almost all available tools provide a user interfacefor each step of the ETL process including reporting. Evenso, building the pipeline requires investment in tool knowl-edge and relational data theory. Due to it’s relational lin-eage, this stack was traditionally not adept at handling un-structured data.
2.1.2 Big Data stackThis stack leverages the Hadoop platform to do the tradi-
tional ETL processing described earlier. ETL is done with amap-reduce paradigm (as well as other programming styles)and uses a cluster of commodity hardware to scale[6]. Sup-port for handling unstructured data is more natural on thisplatform.
With the advent of powerful client machines, fast JavaScriptengines and rich widget libraries, JavaScript has become thede facto choice for creating interactive data visualizations.
The canonical architecture for this stack is shown in fig-ure 2.
Figure 2: Big Data Architecture for data pipelines
2.2 ChallengesBoth these approaches have serious challenges that limit
the efficacy of developers building the data pipeline in orga-nizations[14].
1. Multiple Technologies:Data visualization technologies/tooling and data trans-formation technologies/tooling are inherently different.Apart from requiring different mind-sets, multiple tech-nology stacks bring their attendant problems of dataserialization, interface design and the like. This prob-lem is more acute for the Big Data stack where engi-neers need to glue reporting to the transformed data.
2. Unavailability of Skills:Given the diverse technology pieces present in the datapipeline, lack of skills is an acute problem that theindustry faces. This translates to a lower team velocityto build data pipelines. The problem is made moreacute as there is an explosion of technology options ateach phase of the data pipeline.
3. Lack of Collaboration Model:Although most technologies (whether it is ETL or re-porting) provide lot of libraries to work with, the ab-straction model is not necessarily enforced. It is quitecommon for the complexity to spread both in the li-brary as well as the glue code that puts systems to-gether. This problem is more acute for the Big Datastack - that uses JavaScript based interactive visualiza-tion - where the user needs to resort to significant cus-tom programming. This hampers collaboration and itis very hard to leverage prior work. More importantly,since there is no systemic enforcement of abstractions,existing code needs to be understood before re-use.This leads to significant overheads.
3. SHAREINSIGHTS PLATFORMThe platform’s unique contribution is to present an uni-
fied representation across the data stack so that commonidioms can be used for the entire data pipeline. Since datapipeline architects are dealing with one logical model forthe whole pipeline, we believe that it would translate into afaster turnaround.
In the subsequent sections, we explain the key abstractionsand techniques provided by the platform using examples.
Use Case : Analysis of Apache Open Source Projects.The Apache activity analysis dashboard analyzes open
source projects on http://www.apache.org. It computes aproject activity index to allow for relative comparison ofproject activity in the same (and/or complementary) tech-nology category . This dashboard is shown in figure 3.
The raw data for this dashboard comes from bug tickets,project commit history, stack overflow traffic and projectcollaborators information. We allow dashboard users totweak the weightage given to each of the four parameters- check-ins, bug issues, contributors and releases (the foursliders at the top of the dashboard) to determine the over-all project activity (and thereby relevance) in the projectcloud. Once the weight factors have been decided, users canchange the date range of their analysis using the provideddate slider. Finally, clicking on a project bubble will revealstatistics about the selected project.(figure 13).
In the following sections, we demonstrate the key abstrac-tions of our system using this example.
Figure 3: Apache Open Source Project Analysis
3.1 Flow FileAll data pipelines are expressed in a flow file. Each flow
file has a name and contains the data pipelines associatedwith it. Each flow file has a Data (D) configuration section, aWidget (W) configuration section, a Task (T) configurationsection, a Flow (F) configuration section - to process dataobjects through tasks (T) - and a Layout (L) configurationsection to position widgets (W) in the dashboard.
These sections are elaborated below.
3.2 Data SectionThis section contains the data objects (sources and sinks)
used by the data pipeline. Every dashboard needs data ob-jects to work off from. The data section allows the dash-board to configure these data objects. Data object con-figuration involves specifying the protocol and protocol pa-rameters used to fetch the data as well as parameters tounderstand the payload.
The platform provides popular protocol connectors - suchas File (local, remote), HTTP/S, FTP, JDBC, ad-hoc queriesover JDBC - and recognizes popular data payload formatssuch as CSV, AVRO, XML and JSON documents.
A sample data object configuration described in figure 4indicates that the data source is a CSV file. (present in thedata directory of the dashboard)
D. stack summary :s epara to r : ’ , ’source : ’ s t a ckove r f l ow . csv ’format : ’ csv ’
Figure 4: Data source configuration
Additionally, users need to explicitly call out the schemaof the payload as shown in figure 5.
stack summary :[ p ro j e c t , quest ion , answer , tags ]
Figure 5: Data source schema
These column names may be used to configure tasks whichtransform data sources into other forms. In this example,the data object columns are questions, answers and tagsrelated to the projects being analyzed by this data pipeline.Even though, this example shows data read from a localfile, the data object can be configured to directly talk to theprovider APIs. This is shown in figure 6. The => notationmaps JSON paths in the payload to column names.
D:s t a c k q u e s t i o n s : [
ques t i on => t i t l e ,tags => tags ,
]D. s t a c k q u e s t i o n s :
source : https : // api . stackexchange . com/2 .2/ que s t i on s ? order=desc&s o r t=a c t i v i t y&s i t e=s tackove r f l ow
pro to co l : httpformat : j sonr eque s t type : getht tp header s :
X−Access−Key : XXX
Figure 6: Configure data source with provider APIs
3.3 Task SectionThe task section contains all the task configurations used
by the data pipeline.Tasks operate on data objects and are consumed in data
flows (shown in section 3.4). The platform comes pre-loadedwith a set of useful transformations. The input data to atask is determined contextually; the preceding data object ortask (tasks also output temporary data objects). Tasks aregenerally configured by specifying column names to operateon.
For example a filter task would use a column name ina filter expression as shown in figure 7. Note that the taskconfiguration assumes that it will be used in a context wherethe data source has a rating column.
c l a s s i f i c a t i o n :type : f i l t e r b yf i l t e r e x p r e s s i o n : r a t i n g < 3
Figure 7: Filter task
Tasks can be reused in multiple flows as long as the pre-ceding data source has the column the task consumes.
Task transformations can add columns (e.g. join oper-ation) or reduce columns (e.g. group operation) or pre-serve columns (e.g. filter operation) to the transformed datasource.
3.4 Flow SectionThe flow section contains a collection of flows. Each flow
allows data-sets to be transformed into other forms. Weleverage the Unix pipe notation to express data transforma-tion semantics.
A sample flow is shown in the F section of figure 8. Theflow transforms svn jira summary to checkin jira emailsdata sink with a groupby task (using a composite key).
D:. . .
svn j ira summary : [ p ro j e c t , year , noOfBugsnoOfCheckins , noOfEmailsTotal . . . ]
c h e c k i n j i r a e m a i l s : [ p ro j e c t , year ,t o t a l c h e c k i n s , t o t a l j i r a ,t o t a l e m a i l s ]
F :. . .
D. c h e c k i n j i r a e m a i l s :D. svn j ira summary | T. g e t s v n j i r a c o u n t
T:. . .g e t s v n j i r a c o u n t :
type : groupbygroupby : [ p ro j e c t , year ]aggregate s :
− operator : sumapply on : noOfCheckinso u t f i e l d : t o t a l c h e c k i n s
− operator : sumapply on : noOfBugso u t f i e l d : t o t a l j i r a
− operator : sumapply on : noOfEmailsTotalo u t f i e l d : t o t a l e m a i l s
Figure 8: A data transformation flow
We constrain each flow to start with (optionally) multipledata sources (i.e fan-in) and end with (optionally) multipledata outputs (fan-out). The input data objects are trans-formed to output data objects via transform (or task) opera-tors configured in the task section. Data sinks are automati-cally configured by the system. (As outlined in section 3.2).Note that data-sink is purely a concept for clarity of ex-planation. The system internally makes no differentiationbetween a data source and a data sink.
3.4.1 Sharing processed data objectsData objects (sources or sinks) can be shared in the fol-
lowing ways.
Enable Remote Access.To make the data object available to the dashboard (over
a network connection), specify endpoint to true as shown infigure 9. Data objects which hold aggregated or summarizeddata for visualization purposes would set this property totrue.
Enable Group Access.To make the data object available to other dashboards,
specify a name by which this data object will be referencedby other dashboards. In figure 10, checkin jira emails ispublished as a named data source - project chatter. Otherdashboards can use this data object by name without hav-ing to configure it in their own dashboards. (The platform
D. c h e c k i n j i r a e m a i l s :endpoint : t rue
OR#+ i s an a l i a s f o r endpoint : t rue+D. c h e c k i n j i r a e m a i l s :
D. svn j ira summary |T. g e t s v n j i r a c o u n t
Figure 9: A data sink configured as an endpoint
searches for this data object - in the shared objects list -when referenced in another dashboard)
This concept allows for creation of dashboards with thesole intention of building flows to transform raw data sourcesinto a shared set of data entities on the platform.
D. c h e c k i n j i r a e m a i l s :pub l i sh : p r o j e c t c h a t t e rendpoint : t rue
Figure 10: A published data sink
Data sources and sinks can be configured with both pub-lish and endpoint values. If none of these keywords are used,it is assumed to be a throw-away data source/sink.
3.4.2 Linear flow specificationData sinks can also be input to other flows as demon-
strated in figure 11.
D. t emp re l ea se count : D. r e l e a s e s| T. c a l c u l a t e t o t a l r e l e a s e
D. r e l q a t a g s : (D. temp re lease count ,D. stack summary
) | T. combine stack summary
Figure 11: A flow using intermediate data object
Note that the user can only specify simple (as in linear)flows. Since data sinks can become data sources for otherflows, it is possible to build up arbitrarily complicated trans-formation paths.
On submission, the platform internally builds a directedacyclic graph (DAG) from the collection of flows specifiedby the user.
3.5 Widget SectionThe widget section contains a collection of widget config-
urations. Widgets show transformed (endpoint) data andneeds to be minimally configured with a data source andthe type of widget (Line, Pie, Bubble etc). The platformcomes pre-loaded with a set of commonly used widgets. Ev-ery widget has a set of attributes which associate (or bind)with data source columns. These attributes are called dataattributes or widget columns[9]. The remaining attributes ofa widget are visual attributes. Visual attributes are avail-able in various categories such as legend, axis, dimensionconfigurations and the like.
D:. . .
p r o j e c t d a t a : [ p ro j e c t , year , t o t a l wt . . . ]
W:. . .
p ro j e c t t e chno l ogy bubb l e :# Data Att r ibute stype : BubbleChartsource : D. p r o j e c t d a t a | T. ge t date
| T. a g g r e g a t e p r o j e c t b u b b l e st ext : p r o j e c ts i z e : t o t a l wtl e g e n d t e x t : technology# Visua l At t r ibute sd e f a u l t s e l e c t i o n : Trued e f a u l t s e l e c t i o n k e y : t ex td e f a u l t s e l e c t i o n v a l u e : ’ pig ’. . .l egend :
show legends : t rue. . .
Figure 12: Widget configuration
Figure 12 shows how the project bubble widget is config-ured using project data (endpoint data) as a source. Thewidget columns are text and size and are bound to theproject and total wt columns of the data schema respec-tively.
Note that multiple widgets can be configured with thesame data source.
Most of the widgets used by the Apache dashboard areprovided by the platform. The widget which lets the userdecide the relative weight of each parameter to computeproject activity index is a custom widget - written usingthe platform extension APIs.(Section 4.2)
3.5.1 Widget to Widget InteractionDashboards allow users to interact with widgets to explore
data patterns. Common examples include filter and drill-down operations. For example, a pie wedge selection maybe used to filter rows in a data grid. In general, the visualinformation-seeking mantra: ”overview first, zoom and filter,then details on demand” [13] has never been more relevant.
In the Apache dashboard, selection of a project in thebubble widget reflects the project statistics at the right asshown in figure 13.
Figure 14 demonstrates how this interaction is representedas a flow in the widget source.
Notice that the interaction specification is represented as adata transformation flow - identical in all respects to flows inthe Flow (F) section. These transformations also use tasksdefined in the task (T) section. Popular widget flow tasksprovided by the platform are group, filter and map.
The task filter projects configuration is shown in fig-ure 15. This task filters by the project column of the datasource with values retrieved from widget project cate-gory bubble’s text widget column property. Note that we
Figure 13: Project selection updates project details
W:
. . .
project name :# Data Att r ibute stype : HTMLtag : s e c t i o nsource : D. p r o j e c t l a t e s t r e l e a s e
| T. f i l t e r p r o j e c t s
Figure 14: Widget interaction modeling
treat widgets as data objects and widget columns as datacolumns(attributes).
Note the absence of event handlers and traditional im-perative programming constructs to handle widget - widgetinteraction.
T:
. . .
f i l t e r p r o j e c t s :type : f i l t e r b yf i l t e r b y : [ p r o j e c t ]
#widget data sourcef i l t e r s o u r c e :W. p r o j e c t c a t e g o r y bu b b l e
#widget columnf i l t e r v a l : [ t ex t ]
Figure 15: A filter task configured for use in inter-action flows
3.6 Layout SectionThe layout section arranges the widgets on the dashboard
screen. The platform models any dashboard as a grid ofwidgets. Every cell in the grid holds a reference to a widgetname or can itself be a layout. Every row in the grid is bro-ken into twelve columns (arbitrary) of equal width. Each cellspecifies how many columns it will span. A sample layout isspecified in figure 16.
To explain the key concepts, we have only used few con-figuration items from the complete Apache project analysis
L :d e s c r i p t i o n : Apache Pro j e c t Ana lys i srows :#Row 1 has a c e l l spans 12 columns− [ span12 : W. apache custom widget ]
#Row 2 has two c e l l s#Ce l l 1 spans 4 c o l s#Ce l l 2 spans 8 c o l s− [ span4 :W. y e a r s l i d e r l a y o u t ,
span8 :W. r i g h t p r o j e c t i n f o l a y o u t ]
. . .
− [ span5 :W. pro j e c t ca t ego ry bubb l e ,span7 :W. r i g h t s l i d e r s l a y o u t ]
Figure 16: Apache dashboard layout
dashboard. The complete listing of this dashboard can beavailed on request from any of the authors.
The flow file grammar (simplified) is presented in ap-pendix B for reference.
3.7 Data Sharing
Use Case : Tweet Analysis .In this section, we demonstrate data sharing by creat-
ing two data pipelines. The first data pipeline transformssource tweet data into shared data objects. The second datapipeline consumes these shared data objects to create an in-teractive dashboard. We also illustrate the ability to han-dle unstructured data and the notion of custom tasks. Thedashboard we want to build is shown in figure 17.
Figure 17: Tweet analysis for IPL
The raw data for this dashboard comes from the tweetscollected during the Indian Premier League (IPL) competi-tion. The dashboard allows users to select teams and a daterange. The remaining widgets are filtered for the selectedteam and date range. The streamgraph [3] depicts the rel-ative team tweet volumes, the word clouds show popularwords(players, team names) and the map shows the favorite
team for major cities in India. (The diameter of the markeris proportional to total tweets for that team and location)
3.7.1 Data-Processing ModeThe platform allows dashboards to be created in a data-
processing mode only. In this mode, the flow file has onlya data section (D), flow section (F) and a task section (T).The (transformed) aggregated data from the flows are made’available’ (as defined in section 3.4.1) for consumption inother dashboards. We demonstrate using one representativeflow. The data section is shown in figure 18.
D:i p l t w e e t s : [
postedTime => c r ea ted at ,body => text ,l o c a t i o n => user . l o c a t i o n
]p l a y e r s t w e e t s : [
date ,p layer ,# normal ized name o f p laye rcount #count o f tweets f o r a p layer
]
Figure 18: IPL Data Section
In this case, source data are tweets retrieved from Gnipprovided APIs [7]. Each tweet is a document with hierarchi-cal data items. Users need to map paths in this document(in a similar fashion to XPath or JSONPath queries) to col-umn names. In figure 18, user.location path is mapped toa schema attribute - location. The flow in figure 19 takesraw tweets (from ipltweets) and transforms it to a schemarepresented by players tweets.
F :. . .
D. p l a y e r s t w e e t s : D. i p l t w e e t s |T. p l a y e r s p i p e l i n e |T. p laye r s count
D. p l a y e r s t w e e t s :endpoint : t ruepub l i sh : p l a y e r s t w e e t s
Figure 19: Transform raw tweet data
The first step transforms (in parallel) the date to a stan-dard format and extracts player names from the tweet bylooking at an user provided dictionary (which maps the mul-titude of player names - abbreviations, nick names etc - toa standardized player name). This is shown (and explainedwith comments) in figure 20 and figure 21.
p l a y e r s p i p e l i n e :p a r a l l e l : [
T. norm ipldate ,T. e x t r a c t p l a y e r s]
Figure 20: Transform 1 : Normalize date and ex-tract player in parallel
norm ipldate :type : mapoperator : datetrans form : postedTime #a t t r i b u t e to mapinput format : ’E MMM dd HH:mm: s s Z yyyy ’output format : yyyy−MM−ddoutput : date # output column
e x t r a c t p l a y e r s :type : mapoperator : e x t r a c ttrans form : body #a t t r i b u t e to trans formd i c t : p l aye r s . txt #standard names mapoutput : p laye r #output column
Figure 21: Transform 1 : Normalize date and ex-tract player tasks
The first step of the transformation thus produces theschema shown in figure 22.
p l a y e r s t w e e t s : [postedTime , body , l o c a t i o n # i p l t w e e t sdate , #normal ized datep layer #normal ized p layer name
]
Figure 22: Intermediate schema
The second step of the transformation takes the inter-mediate schema and groups by date and player to give thecount of tweets for the composite key (date + player). Thisis shown in figure 23. This produces data in the schemedescribed in figure 18.
p l aye r s count :type : groupbygroupby : [ date , p laye r ]
Figure 23: Transform 2 : Group by date and player
Some of the important transformation tasks are providedby the platform. Users however can extend the capability ofthe platform by providing their own transformation tasks.User defined tasks are treated on par with system providedtasks and are represented in the flow file in an identical fash-ion. The complete listing for this dashboard is available inappendix A.1.
3.7.2 Data-consumption ModeThe shared data object (player tweets) is now available
to other consumers (dashboards) on the platform. The con-sumption dashboard typically has a layout section (L), wid-get section (W) and a task section (T) for modeling widgetinteraction. When widgets use the shared data object astheir source, the platform automatically searches the shareddata objects on the platform for matches. Breaking up thedata processing and data consumption activities results inbenefits which are outlined in greater detail in section 4.5.3.The complete listing for this dashboard is available in ap-pendix A.2.
Figure 24: ShareInsights Platform Blocks
4. THE SHAREINSIGHTS PLATFORMIn this section, we provide details on the building blocks
of the ShareInsights platform (See figure 24).
4.1 Flow File Compilation ServicesThe flow file compilation module is the heart of the plat-
form. The compilation module builds an abstract syntaxtree (AST) from the flows specified in the flow and wid-get section. The AST eventually gets converted into eithera Pig[12]/Spark[15] job - for data processing - and a datacube (in JavaScript) - for ad-hoc widget interaction (group,filter etc). This is shown diagrammatically in figure 25.
The AST provides opportunities to optimize the completeflow. For example, tasks can be re-arranged to minimizedata transfers to the browser.
Additionally, the generated output needs to be cognizantof the operating environment settings (constraints) such asscreen resolution and client computing resources.
Screen Resolution : At one end of the spectrum, mobiledevices have limited screen space - further limiting interac-tion methods - and at the other end, power analysts havelarge monitors that affords richer visualization and interac-tion methods.
Client Computing Resources : It is not guaranteed thatthe user will have a powerful device for accessing the visu-alization. Additionally, JavaScript support may be variedacross devices and in the worst case even turned off.
These constraints influence what analysis can be displayedmeaningfully and the platform needs to choose the appro-priate representation and execution engine.
4.2 ShareInsights Extension ServicesThe ShareInsights platform comes with a library of data
connectors, data formats, tasks and widgets. Through theplatform extension services, users can bring their own ex-tensions to the platform. The section lists the possible waysto extend the platform.
Figure 25: Flow file compilation
Connectors : The Connectors API allows to connectdata sources via new protocols to the platform.
Data formats : The Data formats API allows for recog-nizing and processing new data formats.
Tasks : The Tasks API allows for introducing new kindsof tasks on the platform. Tasks can be broadly categorizedas
1. Transforming a column value into another value. Thesekinds of tasks are called operators and work off a givendata point.
2. Transforming a bag of values into a point value. Thesekinds of tasks are called user defined aggregates.
3. Transforming a data object via the underlying engine(e.g. Spark) APIs. For example, the platform groupoperator generates calls to the engine group operator.
4. Transforming a data object via a native map reducejob. This is very useful as many organizations haveexisting map reduce jobs and they can be part of theplatform through this route.
Since, we support multiple execution contexts, each task(potentially) has an implementation for each run-time en-vironment - Hadoop and JavaScript (browser) run-times.To encourage adoption and lower barriers of entry, taskscan be written in either Java, JavaScript, Python or R andcan use existing libraries on those environments. For ex-ample Python natural language processing libraries can bewrapped with our APIs and used in the flow file.
Widgets : The Widgets API allows new widgets to beintroduced to the platform. Commercial and open sourcewidgets can easily be made part of the platform by imple-menting this interface.
Styling : The dashboard look and feel can be changedor enhanced using Cascading Style Sheets (CSS). Stylesheetauthors can use widget names specified in the flow file asstyle targets in the CSS file.
4.3 ShareInsights Development ServicesShareInsights uses regular desktop software for contrib-
utors to be productive and requires no additional installs.Data pipeline analysts interact with ShareInsights via thedevelopment interface or the extension services interface.These interfaces are described in the subsequent sections.
4.3.1 Development interfaceShareInsights uses the browser exclusively for data-pipeline
development. All ShareInsights dashboard operations (cre-ate, edit, run, save etc) as well as data operations can beaccessed via intuitive REST APIs.
For example, to create a dashboard from scratch, usersneed to navigate to the following URL : https://<server-base-path>/dashboards/<dashboard-name>/create.
Running this command would take the user to a web ed-itor to edit the dashboard as shown in figure 26. The keycomponents of the editor are also highlighted in the samefigure. The data explorer will be covered in section 4.4.
Figure 26: Dashboard Editor
4.3.2 Extension Services interfaceThe ShareInsights platform provides a secure file transfer
protocol (SFTP) interface to upload the various types ofextensions - widgets, connectors, tasks and stylesheets. Theinterface is file based and each dashboard has appropriatelynamed folders for task, widgets etc. Additionally, users canupload dashboard data to a ’data’ folder. All data files inthis folder can be referred in the data object configuration(using relative paths from this data folder).
4.4 ShareInsights Data APIThe generated interactive dashboard code from the flow
file follows a Single Page Architecture style [10].Dashboard data is retrieved via a REST API provided by
the platform(for endpoint data - see section 3.4.1). This isdescribed in the following sections.
Dashboard endpoint data.To see all endpoint data associated with a dashboard, the
following URL (/<dashboard-name>/ds) can be accessedvia the browser or curl - as shown in figure 27.
Once the endpoint data names are known, they can beused to further browse data and perform ad-hoc queries asdemonstrated in the following sections.
Figure 27: Endpoint data for dashboard
Browsing dashboard endpoint data.Data browsing can be achieved either from the raw REST
APIs as shown in figure 28 or from the platform provideddata explorer.
Figure 28: Query endpoint data
The data explorer effectively runs the dashboard in aheadless mode and displays the data in a tabular format.The data explorer for the Apache dashboard is shown infigure 29.
Figure 29: Data Explorer (UI for endpoint data)
Ad-hoc queries on dashboard endpoint data.The platform also exposes a simplified query language for
ad-hoc queries. In figure 30, the count of items in each cate-gory is returned by querying the data endpoint - projects.(/ds/<dataset-name>/groupby/<column>/<aggregate-function>/<column>)
Figure 30: Ad-hoc query on endpoint data
4.5 Collaboration ServicesShareInsights encourages collaboration in the data analy-
sis ecosystem via the following mechanisms.
4.5.1 Branch and Merge ModelThe ShareInsights platform leverages the collaboration
model found in distributed version control systems (DVCS),like Git. Since the entire data pipeline is represented as asingle text file, it makes it very amenable to manage via asource control system. CRUD operations on flow files mapto source commits. A DVCS encourages collaboration viabranching and repeated merging. This model is mature andis the preferred way to manage code assets [2]. ShareInsightsleverages this model for flow files.
Additionally, since the flow file has clearly demarcatedsections, the anxieties with merging and repeated branchingshould be significantly lower.
4.5.2 Configuration Driven DevelopmentThe flow file has been designed to let users configure tasks,
data, widgets and even flows. Configuration is the activityof setting property values.(as seen from earlier examples)
The only active elements in the flow file then are theflows in the flow(F) section and widget sources (for widgetto widget interaction). Here too, we have restricted the rep-resentation to linear flows. There are no control structuresother than the notion of chaining via pipes. Consequently,it is easier to pick up paths which matter when users modifyexisting flows (users ’fork’ existing dashboards to modify).We believe that this will reduce barriers to experimentation.
4.5.3 Flow file GroupsThe canonical form of the flow file is to have all sections.
However, this is optional. The system allows for creation offlow files which have only the data, flow and task sections.Processed data objects can be shared - either by publishingand/or making it a visualization endpoint - for consumptionby other dashboards. The set of dashboards which sharedata objects and the set of dashboards which consumes themform a natural flow file group.
This technique has the following benefits
1. It addresses the problem of allowing other users to ben-efit and leverage prior work.(an important aspect ofcollaboration). It is a common experience that datacleaning takes a significant percentage of the total time[5]. The platform ecosystem encourages that this effortis not repeated across data pipelines.
2. It allows for teams to be deployed in their area ofstrength. To glean insights effectively, it is a com-mon practice to divide and conquer; each dashboardattempts to answer specific questions or guide users to-wards a certain dimension of the problem space. Con-
sequently, there could be multiple non-intersecting dash-boards in the problem space - each consuming only asubset of data flows. The team which designs dash-boards have possibly a different mindset from the teamwhich focuses on publishing processed data for theproblem space. This technique then allows each sub-team to focus on their strengths and work fairly inde-pendently.
3. It allows for efficient processing of raw data sources.In this configuration, long running data flows are ex-ecuted only by the dashboard which shares the dataobjects.
4. As a corollary to the above point, teams building inter-active dashboards on processed data can get extremelyquick feedback to changes in the flow file.(as long run-ning data pipelines will not be executed when the flowfile is saved) This is vital to encourage rapid changesduring dashboard design.
5. RACE2INSIGHTSTo test the value provided by the unified representation,
we organized an internal competition called Race2Insightswhere untrained users were challenged to create interactivedashboards within six hours.
5.1 Competition setupThis section describes the format and the setup of the
competition.
Data-sets:We identified seven interesting data-sets that containedboth public and enterprise data. Each data-set hadmultiple files that contained both transaction as wellas reference data about business entities. Teams wererandomly allocated a data-set and were encouraged toaugment with other interesting data-sets.
Teams and Team Composition:Fifty two teams participated in the competition. Eachteam had five members. Participants were encouragedto form teams with known colleagues. Teams had vary-ing skill level ranging from zero to little programmingbackground at one end of the spectrum to significantskills in data processing at the other end.
Training:Participants were exposed to the platform five daysprior to the competition. Training videos and hands-on lab sessions were conducted for participants to fa-miliarize them with the nuances of flow file creation.A day before the competition, participants were as-signed the data-set (chosen by a lottery system) andonly schema details were shared. Teams were encour-aged to paper-prototype and design the data pipelines.Access to systems were switched off.
Competition day:On the day of the competition, the actual data-setswere published and teams were given six hours to buildinteractive dashboards to demonstrate business insightsfor their problem space.
Judging:Two rounds of panel-style judging were performed.Teams had to present their dashboard to an internaland an external committee. Teams had to present theflow file to the internal committee for review as well ashighlight any extensions they may have written or ad-ditional data-sets used. The external committee purelyevaluated the dashboard in terms of business value forthat domain.
5.2 LearningsThe section describes some of the findings from the hackathon.
5.2.1 Dashboards from competition dataThe data generated during the competition as well as the
practice sessions - application logs, flow file growth, errormessages, execution logs - were used to build dashboards(using the platform) to illustrate usage of the platform dur-ing the competition hours.
The dashboard shown in figure 31 highlights the popularoperators and widgets.
Figure 31: Platform usage
Another interesting plot shows the correlation betweenpractice and success. Practice runs versus competition runs(a run = a dashboard execution) for the various teams areshown in figure 35. The finalists are teams{5,9,12,18,33,35,41}.The winning teams are teams{12, 18, 33}.
Figure 32: Does practice matter?
5.2.2 Observations
1. Teams produced extremely rich dashboards in six hours.Prior to building this platform, equivalent dashboardstook four to six weeks to develop. A few dashboardsare shown in figure 33 and 34.
2. Some of the winning teams wrote custom tasks to pro-cess the data. For example, one team wrote a task topredict resolution dates of service tickets based on key-words present in the ticket. The custom task looks no
Figure 33: Service Desk Ticket Analysis
Figure 34: ’Branderstanding’
different from a platform provided task and was usedby other team members as a black box.
The extension points provided by the platform are cru-cial as it is not possible to anticipate all possibilities.The ability to not differentiate between a platform taskand a user defined one is also critical for a consistentconceptual model.
3. Teams ’forked’ off existing (help or sample) dashboardsto get started. Figure 35 shows the flow file size foreach team at the start of the competition.
4. Data cleaning is non trivial task. Teams prepared syn-thetic data for practice runs. During the actual com-petition, the real data provided forced teams to definemore elaborate pipelines to cleanse the data.
5. More training was required to grasp widget to widgetinteraction specification.
6. Browser based development and deployment - a zeroinstall footprint - helped the competition logistics dur-ing the actual competition hours as well as duringtraining sessions by allowing teams to participate re-motely.
Figure 35: Fork to go (Size in bytes)
7. Error reporting (during dashboard execution) leakedthe abstraction. The most popular debugging strategywas to go to a stable version and then incrementallyadd till the error resurfaced.
6. FUTURE DIRECTIONSThis section outlines some of the key enhancements to the
platform.Future platform versions will focus on exploiting the AST
generated from the flow file to do execution optimization.For example one can conceive of rearranging user flows tominimize data transfers to the client. The uniform interfacefor widget and flow tasks will facilitate this feature.
Also, since the flow file is an abstraction layer, more workneeds to be done to enable users to pin-point errors quickly.(Without leaking the underlying engine errors or debug logs).Also, tools to identify performance bottlenecks need to beprovided.
We want to auto-construct meta-dashboards which pro-vide statistics and analysis of all the data columns used inthe data pipeline. Since, data cleaning is a non-trivial ac-tivity[5], we believe this feature would be of immense helpfor huge data sizes.
Since data is published on the platform, it potentially al-lows for discovery of data-sets to enrich an existing datapipeline. This is an important feature [11] [1].
7. CONCLUSIONHaving an unified representation to model all stages of the
data pipeline from data ingestion to interactive visualizationdesign has significant benefits in terms of reduced time tobuild. Configuration driven development reduces the com-plexities associated with control structures and cleanly seg-regated sections allow for reuse and easy modification. Theabstraction hides technology differences at every layer of the
data stack and empowers the data analyst/data scientist tofocus on building value for end-users. The notion of allow-ing transformed data to be a shared entity allows for re-use(by other data pipelines) of the non trivial effort which goesbehind curating and cleansing data.
8. ACKNOWLEDGMENTSWe thank the following individuals for their involvement
with the ShareInsights platform. Reshma Godse, KalyaniZomte and Leena Rajendran for their involvement as thelead engineers of the platform, Vinayak Datar and KartikVyas for their customer facing roles and Nilesh Joshi forsystems infrastructure support.
9. REFERENCES[1] C. Bizer, P. Boncz, M. L. Brodie, and O. Erling. The
meaningful use of big data: Four perspectives – fourchallenges. SIGMOD Rec., 40(4):56–60, Jan. 2012.
[2] C. Brindescu, M. Codoban, S. Shmarkatiuk, andD. Dig. How do centralized and distributed versioncontrol systems impact software changes? InProceedings of the 36th International Conference onSoftware Engineering, ICSE 2014, pages 322–333, NewYork, NY, USA, 2014. ACM.
[3] L. Byron and M. Wattenberg. Stacked graphs ;geometry & aesthetics. IEEE Transactions onVisualization and Computer Graphics,14(6):1245–1252, Nov. 2008.
[4] S. Chaudhuri. What next?: A half-dozen datamanagement research goals for big data and the cloud.In Proceedings of the 31st Symposium on Principles ofDatabase Systems, PODS ’12, pages 1–4, New York,NY, USA, 2012. ACM.
[5] T. Dasu and T. Johnson. Exploratory Data Miningand Data Cleaning. John Wiley & Sons, Inc., NewYork, NY, USA, 1 edition, 2003.
[6] J. Dean and S. Ghemawat. Mapreduce: Simplifieddata processing on large clusters. Commun. ACM,51(1):107–113, Jan. 2008.
[7] Gnip.com. Twitter data - gnip :https://gnip.com/sources/twitter/, 2015.
[8] R. Kimball and M. Ross. The Data WarehouseToolkit: The Complete Guide to DimensionalModeling. John Wiley & Sons, Inc., New York, NY,USA, 2nd edition, 2002.
[9] R. Krishnamurthy and M. M. Zloof. RBE: renderingby example. In Proceedings of the EleventhInternational Conference on Data Engineering, March6-10, 1995, Taipei, Taiwan, pages 288–297, 1995.
[10] M. Mikowski and J. Powell. Single Page WebApplications: JavaScript End-to-end. ManningPublications Co., Greenwich, CT, USA, 1st edition,2013.
[11] K. Morton, M. Balazinska, D. Grossman, and J. D.Mackinlay. Support the data enthusiast: Challengesfor next-generation data-analysis systems. PVLDB,7(6):453–456, 2014.
[12] C. Olston, B. Reed, U. Srivastava, R. Kumar, andA. Tomkins. Pig latin: A not-so-foreign language fordata processing. In Proceedings of the 2008 ACMSIGMOD International Conference on Management of
Data, SIGMOD ’08, pages 1099–1110, New York, NY,USA, 2008. ACM.
[13] B. Shneiderman. The eyes have it: A task by datatype taxonomy for information visualizations. InProceedings of the 1996 IEEE Symposium on VisualLanguages, VL ’96, pages 336–, Washington, DC,USA, 1996. IEEE Computer Society.
[14] M. Winslett. Databases in virtual organizations: Acollective interview and call for researchers. SIGMODRec., 34(1):86–89, Mar. 2005.
[15] M. Zaharia, M. Chowdhury, M. J. Franklin,S. Shenker, and I. Stoica. Spark: Cluster computingwith working sets. In Proceedings of the 2Nd USENIXConference on Hot Topics in Cloud Computing,HotCloud’10, pages 10–10, Berkeley, CA, USA, 2010.USENIX Association.
APPENDIXA. IPL FLOW GROUP
A.1 Data Processing Dashboard
D:i p l t w e e t s : [
postedTime => c r ea ted at ,body => text ,displayName => user . l o c a t i o n
]p l a y e r s t w e e t s : [
date , p layer , count]teams tweets : [
date , team , count]dim teams : [
team number , team ,team fullName , s o r t o rde r ,co lo r , noOfTweets
]team players : [
p layer , team fullName ,team , p l aye r id , noOfTweets
]l a t l o n g : [
s ta te , point one , point two ,p o i n t t h r e e
]p l aye r twee t s : [ p layer ,
team , date , p l aye r id ,team fullName , noOfTweets
]team tweets : [
s o r t o rde r , date , co lo r ,team , team fullName , noOfTweets
]tm rgn raw cnt : [
date , team , s tate , count]tm rgn tm dtls : [
s o r t o rde r , noOfTweets , co lo r ,s ta te , team , date , team fullName
]
team reg ion tweet s : [po int one , point two ,po in t th ree , s ta te ,team fullName , team ,co lo r , s o r t o rde r ,date , noOfTweets
]tagc loud tweet s raw : [
date , word , count]tagc l oud twee t s : [
date , word , count]
# −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−F:
D. p l a y e r s t w e e t s : D. i p l t w e e t s |T. p l a y e r s p i p e l i n e |T. p laye r s count
D. p l aye r twee t s : (D. p l aye r s twee t s ,D. team players
) | T. j o in p l aye r t eam
D. teams tweets : D. i p l t w e e t s |T. t eams p ipe l i n e |T. teams count
D. team tweets : (D. teams tweets ,D. dim teams
) | T. jo in dim teams
D. tm rgn raw cnt : D. i p l t w e e t s |T. t e a m s p i p e l i n e r e g i o n |T. teams reg ions count
D. tm rgn tm dtls : (D. tm rgn raw cnt ,D. dim teams
) | T. jo in dim teams two
D. team reg ion tweet s : (D. tm rgn tm dtls ,D. l a t l o n g
) | T. j o i n l a t l o n g
D. tagc loud tweet s raw :D. i p l t w e e t s |T. word date ext rac t i on |T. words count
D. tagc l oud twee t s :D. tagc loud tweet s raw |T. topwords
# −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−T:
p l a y e r s p i p e l i n e :p a r a l l e l : [
T. norm ipldate ,T. e x t r a c t p l a y e r s
]
t eams p ipe l i n e :p a r a l l e l : [
T. norm ipldate ,T. ext ract teams
]t e a m s p i p e l i n e r e g i o n :
p a r a l l e l : [T. norm ipldate ,T. e x t r a c t l o c a t i o n ,T. ext ract teams
]word date ext rac t i on :
p a r a l l e l : [T. norm ipldate ,T. ext ract words
]norm ipldate :
type : mapoperator : datetrans form : postedTimeinput format : ’E MMM dd HH:mm: s s Z yyyy ’output format : yyyy−MM−ddoutput : date
e x t r a c t p l a y e r s :type : mapoperator : e x t r a c ttrans form : bodyd i c t : p l aye r s . txtoutput : p laye r
ext ract teams :type : mapoperator : e x t r a c ttrans form : bodyd i c t : teams . csvoutput : team
e x t r a c t l o c a t i o n :type : mapoperator : e x t r a c t l o c a t i o ntrans form : displayNamematch : c i t ycountry : INDoutput : s t a t e
ext ract words :type : mapoperator : ext ract wordstrans form : bodyoutput : word
j o in p l aye r t eam :type : j o i nl e f t : p l a y e r s t w e e t s by p layerr i g h t : team players by p laye rj o i n c o n d i t i o n : l e f t outerp r o j e c t :
p l a y e r s t w e e t s d a t e : datep l a y e r s t w e e t s p l a y e r : p laye rp l a y e r s t w e e t s s t a t e : s t a t ep l aye r s twee t s count : noOfTweetsteam players team : teamteam players team ful lName :
team fullNamet e a m p l a y e r s p l a y e r i d :
p l a y e r i d
jo in dim teams :type : j o i nl e f t : teams tweets by teamr i g h t : dim teams by team fullNamej o i n c o n d i t i o n : l e f t outerp r o j e c t :
teams tweets date : dateteams tweets team : team fullNameteams tweets count : noOfTweetsdim teams team : teamdim teams sor t order : s o r t o r d e rd im teams co lor : c o l o r
jo in dim teams two :type : j o i nl e f t : tm rgn raw cnt by teamr i g h t : dim teams by team fullNamej o i n c o n d i t i o n : l e f t outerp r o j e c t :
tm rgn raw cnt date : datetm rgn raw cnt team : team fullNametm rgn raw cnt s tate : s t a t etm rgn raw cnt count : noOfTweetsdim teams Team : teamdim teams sor t order : s o r t o r d e rd im teams co lor : c o l o r
j o i n l a t l o n g :type : j o i nl e f t : tm rgn tm dtls by s t a t er i g h t : l a t l o n g by s t a t ej o i n c o n d i t i o n : LEFT OUTERp r o j e c t :
tm rgn tm dtls team ful lName :team fullName
tm rgn tm dt l s s ta t e : s t a t etm rgn tm dt l s date : datetm rgn tm dtls noOfTweets : noOfTweetstm rgn tm dtls team : teamtm rgn tm dt l s s o r t o rde r : s o r t o r d e rtm rgn tm dt l s co l o r : c o l o rl a t l o n g p o i n t o n e : po int onel a t l o n g p o i n t t w o : point twol a t l o n g p o i n t t h r e e : p o i n t t h r e e
p l aye r s count :type : groupbygroupby : [ date , p laye r ]
teams count :type : groupbygroupby : [ date , team ]
teams reg ions count :type : groupbygroupby : [ date , team , s t a t e ]
words count :type : groupbygroupby : [ date , word ]
topwords :type : topngroupby : [ date ]orderby column : [ count DESC]l i m i t : 20
A.2 Data consumption DashboardFor brevity of the source listing, it is assumed that all
the data objects used by widgets in this listing have beenpublished (with identical names) and end-pointed in listingA.1.
# −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−L :
d e s c r i p t i o n : Clash o f Titansrows :− [ span12 : W. teams ]− [ span11 : W. i p l d u r a t i o n ]− [ span11 : W. r e l a t i v e t eamtwee t s ]− [ span6 : W. word team player tweets ,
span5 : W. r eg i on twee t s ]
# −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−W:
i p l d u r a t i o n :type : S l i d e rsource : [ ’2013−05−02 ’ , ’2013−05−27 ’]s t a t i c : t ruerange : t rues l i d e r t y p e : date
r e l a t i v e t eamtwee t s :type : Streamgraphsource : D. team tweets |
T. f i l t e r b y d a t e |T. f i l t e r b y t e a m
#widget columnsx : datey : noOfTweetsc o l o r : c o l o rs e r i e : teamxAxis :
type : ’ datetime ’yAxis :
a l lowDecimals : f a l s emin : 0max : 25000
teams :type : L i s tsource : D. dim teams#widget columnstext : team#Visua l a t t r i b u t e simage pos i t i on : r i g h t
p laye r twee t s :type : WordCloudsource : D. p l aye r twee t s |
T. f i l t e r b y d a t e |T. f i l t e r b y t e a m |T. aggrega te by p laye r
#widget columnstext : p laye rs i z e : noOfTweets
#v i s u a l a t t r i b u t e sshow too l t i p : t ruet o o l t i p t e x t : [ p layer , noOfTweets ]
teamtweets :type : WordCloudsource : D. team tweets |
T. f i l t e r b y d a t e |T. aggregate by team
#widget columnstext : teams i z e : noOfTweets
#v i s u a l a t t r i b u t e sshow too l t i p : t ruet o o l t i p t e x t : [ team , noOfTweets ]
wordtweets :type : WordCloudsource : D. tagc l oud twee t s |
T. f i l t e r b y d a t e |T. aggregate by word
#widget columnstext : words i z e : count
#v i s u a l a t t r i b u t e sshow too l t i p : t ruet o o l t i p t e x t : [ word , count ]
r eg i on twee t s :type : MapMarkersource : D. team reg ion tweet s |
T. f i l t e r b y d a t e |T. f i l t e r b y t e a m |T. aggregate by team reg ion
country : INDmarkers :− marker1 :
type : c i r c l e m a r k e r#widget columnsl a t l o n g v a l u e : po int onemarkers i ze : noOfTweetsf i l l c o l o r : c o l o rt o o l t i p t e x t : [
s ta te ,team ,noOfTweets
]
#sub layout s
teamtweetstab :type : Layoutrows :− [ span11 : W. teamtweets ]
p laye r tweet s tab :type : Layoutrows :− [ span11 : W. p laye r twee t s ]
wordtweetstab :type : Layoutrows :− [ span11 : W. wordtweets ]
word team player tweets :type : TabLayouttabs :− name : ’ Player ’
body : W. p laye r tweet s tab− name : ’Word ’
body : W. wordtweetstab− name : ’Team’
body : W. teamtweetstab
# −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
T:
aggr ega te by p laye r :type : groupbygroupby : [ p laye r ]aggregate s :− operator : sum
apply on : noOfTweetso u t f i e l d : noOfTweets
aggregate by team :type : groupbygroupby : [ team ]aggregate s :− operator : sum
apply on : noOfTweetso u t f i e l d : noOfTweets
aggregate by word :type : groupbygroupby : [ word ]aggregate s :− operator : sum
apply on : counto u t f i e l d : countorde rby aggregate s : t rue
f i l t e r b y d a t e :type : f i l t e r b yf i l t e r b y : [ date ]f i l t e r s o u r c e : W. i p l d u r a t i o n
f i l t e r b y t e a m :type : f i l t e r b yf i l t e r b y : [ team ]f i l t e r s o u r c e : W. teamsf i l t e r v a l : [ t ex t ]
aggregate by team reg ion :type : groupbygroupby : [ team , point one , s ta te , c o l o r ]aggregate s :− operator : sum
apply on : noOfTweetso u t f i e l d : noOfTweets
B. FLOW FILE GRAMMAR
/∗ −−−−−−−− Parser r u l e s −−−−−−−−−−− ∗/f l o w F i l e :
dataSect ion ? f l owSec t i on ?ta skSec t i on ?da ta D e t a i l s S e c t i on ? EOF ;
/∗ −−−−−−−−−− data s e c t i o n −−−−−−−− ∗/dataSect ion : ’D’ ’ : ’ dataItem+ ;dataItem : dataItemName ’ : ’ ’ [ ’ hdrs ’ ] ’ ;dataItemName : IDENTIFIER ;hdrs : hdrFie ld ( ’ , ’ hdrFie ld )∗ ;hdrFie ld : IDENTIFIER ;
/∗ −−−−−−−−−− f low s e c t i o n −−−−−−−− ∗/f l owSec t i on : ’F ’ ’ : ’ f low+ ;f low : ’D. ’ outputData ’ : ’ mult iDataStart ∗
’D. ’ inputDataInFlow( ’ , ’ ’D. ’ inputDataInFlow )∗multiDataEnd( ’ | ’ ’T. ’ taskInDataFlow )+;
mult iDataStart : ROUND BRACKET OPEN ;multiDataEnd : ROUND BRACKET CLOSE ;outputData : IDENTIFIER ;inputDataInFlow : IDENTIFIER ;taskInDataFlow : IDENTIFIER ;
/∗ −−−−−−−−−− task s e c t i o n −−−−−−−− ∗/ta skSec t i on : ’T’ ’ : ’ task+ ;task
:taskName ’ : ’’ type ’ ’ : ’ taskType(paramName ’ : ’
’ [ ’ ∗’ ( ’∗ paramValue ( ’ , ’ paramValue )∗ ’ ) ’∗’ ] ’ ∗ )+
;taskName : IDENTIFIER ;taskType : IDENTIFIER ;paramName : IDENTIFIER ;paramValue : IDENTIFIER ;
/∗ −−−− data d e t a i l s s e c t i o n −−−− ∗/da ta D et a i l s S e c t i on : dataI temInDeta i l+ ;dataItemInDeta i l :
’D. ’ dataItemNameInDetail ’ : ’ prop+ ;prop :
(dd paramName ’ : ’ dd paramValue )+;dataItemNameInDetail : IDENTIFIER ;dd paramName : IDENTIFIER ;dd paramValue : IDENTIFIER ;
/∗ −−−−−−− Lexer r u l e s −−−−−−−−− ∗/ROUND BRACKET OPEN : ’ ( ’ ;ROUND BRACKET CLOSE : ’ ) ’ ;IDENTIFIER : [ a−zA−Z ]+[ a−zA−Z0−9 ]∗ ;WS : [ \ t \ r \n\u000C]+ −> sk ip ;LINE COMMENT : ’# ’ ˜ [\ r \n ]∗ −> sk ip ;
