Temporally Coherent Interpretations for Long Videos Using Pattern Theory

Fillipe Souza, Sudeep SarkarUniversity of South Florida

Tampa, FL USAfillipe@mail.usf.edu,sarkar@cse.usf.edu

Anuj SrivastavaFlorida State University

Tallahassee, FL USAanuj@stat.fsu.edu

Jingyong SuTexas Tech University

Lubbock, TX USAjingyong.su@ttu.edu

Abstract

Graph-theoretical methods have successfully providedsemantic and structural interpretations of images andvideos. A recent paper introduced a pattern-theoretic ap-proach that allows construction of flexible graphs for rep-resenting interactions of actors with objects and inferenceis accomplished by an efficient annealing algorithm. Ac-tions and objects are termed generators and their interac-tions are termed bonds; together they form high-probabilityconfigurations, or interpretations, of observed scenes. Thiswork and other structural methods have generally been lim-ited to analyzing short videos involving isolated actions.Here we provide an extension that uses additional temporalbonds across individual actions to enable semantic inter-pretations of longer videos. Longer temporal connectionsimprove scene interpretations as they help discard (tem-porally) local solutions in favor of globally superior ones.Using this extension, we demonstrate improvements in un-derstanding longer videos, compared to individual inter-pretations of non-overlapping time segments. We verifiedthe success of our approach by generating interpretationsfor more than 700 video segments from the YouCook dataset, with intricate videos that exhibit cluttered background,scenarios of occlusion, viewpoint variations and changingconditions of illumination. Interpretations for long videosegments were able to yield performance increases of about70% and, in addition, proved to be more robust to differentsevere scenarios of classification errors.

1. IntroductionThe problem of understanding activities in video data

and providing meaningful semantic interpretations is veryimportant. In recent years, a variety of solutions have beenproposed and, among other ideas, the techniques based onencoding scene structure using graphs have shown promisein this problem area. These approaches represent items ofinterest – objects, actors, actions, etc. – as nodes in graphsand ascertain their interactions through graph edges. The

main advantage of this framework is that one can naturallyassociate probability models with such graphs, thus provid-ing statistical interpretations to solutions. Also, one can useboth prior knowledge and the current data to deduce opti-mal interpretations in a coherent way. The main limitationin the current graph-theoretical solutions has been the rigid-ity of graph structures. In most cases, the graph geometries(connectivities, neighborhoods, etc) are pre-determined andonly the node values are allowed to be variable. Even whenthe edges are allowed to change, they are usually based on asimple thresholding, or decisions that are spatio-temporallylocal, i.e. isolated from other nodes.

(a) (b)

Figure 1: Overview of the pattern theoretic framework pro-posed in [1]. (a) shows basic elements of this framework:a generator space containing basic ontological elements ofrepresentation called generators, machine learning-basedconcept classifiers and prior knowledge in terms of fre-quency tables of concept co-occurrences. (b) shows a pat-tern theoretic video interpretation that is a combination ofgenerators. Connections between generators that representontological concepts indicate occurrence of certain interac-tions. Features are connected to ontological generators tosupport their semantic value in the interpretation.

Souza et al. [1] recently introduced a flexible graph-theoretical approach that is based on Grenander’s patterntheory [2]. Here the flexibility comes from the fact that both

1

nodes and edges are allowed to be variables and are inferredfrom available knowledge. There are two dominant sourcesof knowledge: (1) the prior in form of frequency tablesof concept co-occurrences, contextual knowledge about ac-tions represented by the underlying ontology extracted fromprevious annotated videos, and (2) objects and actions de-tected using machine learning techniques, and their detec-tion scores, in the current video. In [1], the authors stud-ied short videos containing individual actions (pick up, putdown, pour, stir, etc) and demonstrated the strength of thispattern theoretic approach and the flexibility of its repre-sentation. Here one does not need to explicitly model eachof the variants for an action. It is capable of discoveringhidden events or events not previously considered duringannotation phase. An illustration of this pattern theoreticframework is shown in Figure 1.

The main limitation of [1]’s work is that it cannot per-form well with large videos containing multiple actions andcomplex interactions between actions. If one splits largevideos into smaller, disjoint time windows and performs in-dividual inferences, then the overall interpretation can beboth inconsistent and sub-optimal. In this paper we pur-sue a more comprehensive approach by introducing tempo-ral bonds across sub-configurations that represent individ-ual interactions (actions performed on/with objects). Thisadditional structure enables us to discard (temporally) localsolutions in favor of globally optimal and temporally con-sistent configurations, as illustrated in Figure 2.

We demonstrate these ideas on a recent challenging dataset of cooking scenarios, the YouCook datasets. Its videosdepict high-level activities in unconstrained scenarios, withcluttered background, clutter of objects, variable conditionsof illumination, different viewpoints and camera motion.

2. Related WorkImplicit and explicit structural models have been pro-

posed for tackling the problem of generating semanticallycomplex video interpretations. The main difference be-tween these methods is in the way contextual, logical andtemporal dependencies are encoded. Implicit structuralmodels [3] [4] [5] attempt to capture these information im-plicitly through some general form of data representation[6], such as the BoVW framework [7] and Linear Dynam-ical Systems (LDS) [8], or a set of coefficients learned us-ing the max-margin framework [9]. Others compute dis-tributions or some statistical summaries, such as the co-occurrence of concepts, which serve to indicate or corrobo-rate with certain probability the existence of more complexones and also used to derive semantic description throughtext [10, 11, 12]. These approaches do not offer flexibilityin representation because if new concepts are later addedto redefine the characterization of a single complex event,the models have to be reconstructed. Moreover, it is not

(a) (b) (c) (d)

(e) (f)

Figure 2: Illustration of advantage in using temporal bonds.Top rows shows frames from two consecutive segments ofa video. The first segment depicts the interaction put bowldown (the small one with the left hand) and second segmentdepicts stir ingredients in a bowl using spatula. (e) shows[1]’s interpretations for both segments. (f) shows our ap-proach’s interpretation for both segments. Shaded circlesdenote correctly identified generators.

clear how scalable these methods are for when the struc-ture variability and semantic complexity of the target eventsincrease.

The traditional explicit structural models are mostclosely related to our approach. These models are typ-ically hand-crafted or algorithmically learned from thetraining data. They can be generated in terms of dy-namic Bayesian Networks (DBN) [13][14], Sum ProductNetworks (SPN) [15, 16], Stochastic Context-free Gram-mars (SCFG) [17][18][19][20], AND-OR graphs [21][22],Petri Nets [23], or general hierarchical graphical models[24] [25]. Complex events are usually sought to be com-posed by a set of temporally ordered sequence of sub-events [18], which can also suffer certain order variations orhave optional steps [19]. These works typically consist ofa low-level layer in which feature observations provide ev-idence for concepts from the top layers, such as sub-eventsor composite events. For example, Hilde et al. [19] useHMMs to learn models for sub-events such as pour cof-fee and take cup and model more semantically complexevents such as preparing coffee using a SCFG that describestheir syntax in the form of occurrence of sub-events. Otherworks use the SVM framework to provide confidence val-ues as evidence for the occurrence of sub-events. Contextand temporal dependence constraints are mostly suppliedby coefficients learned with the max-margin framework orco-occurrence statistics of the target concepts.

Unlike the representation proposed by [1], these meth-ods face limitations in representation caused by increasein structural complexity and non-linearity with respect tothe variations in order, presence or absence of certain sub-

events, specially when structural learning is required [14].Such models impose a typical order or a limited number ofvariations [19] in which a complex events can occur givena set of related sub-events. Automatic learning of structuresdoes not seem easily scalable for when updates on the mod-els are needed to include new knowledge and can requirelarge amounts of data to be available for learning all possi-ble structural variations. Additionally, it does not providea mechanism to discover new optional sub-events. Follow-ing [1], our approach overcomes these limitations by puttingforward a representation model based on rules of domainknowledge ontology that is formed mostly by simple con-cepts, each having a combinatorial signature. These combi-natorial signatures span a space of structures that may repre-sent either interactions between concepts or complex eventsof interest. By simple combinatorial rules, our method is ca-pable of accounting for a larger space of structures, whichincludes identifying different structural variations and un-seen structures that represent one same event.

3. Pattern Theoretic FormulationKnowledge representation through Pattern Theory con-

sists of using concepts and concept combination rules de-scribed by some domain-specific ontology. In Pattern The-ory, these concepts become generators and combinationrules are transformed into bond structures of generators.Based on the target ontology, generators are organized intolevels of abstraction and at each level they are further classi-fied into different modality groups, each group establishingdifferent properties of similarity (which can be structural orsemantic). A collection of generators organized in this fash-ion forms a domain-specific generator space. Such genera-tor space forms the basis for generating complex structuresby the combinations of generators. Formally,

Definition 1 Generators are building blocks used to con-struct graph structures that represent patterns of interest.They will be denoted by gi ∈ G, where G is a chosen gener-ator space.

Remark 1 Each generator gi has a bond structureB(gi) = (Bs(gi), Bv(gi)) that is defined by a structuralarrangement Bs(g) of in-bonds and out-bonds with coor-dinates j = 1, 2, ..., w(g). Each bond has a bond valuesβj(g) ∈ Bv(g). Bs(g) accounts for the set of bonds of agenerator g and Bv(g) is the set of bond values of bonds ing.

Definition 2 A modality setM is a partition of G such thatany pair of generators in a modalityMk ∈M holds similarproperties.

Remark 2 A modality Mk ∈ M pertaining to G is in-duced by a similarity s ∈ S, defining in which terms the

generators in that modality are similar to each other. Anexample is the modality utensils = { bowl, cup, pan, spoon}; the modality name is suggestive of the common propertyamong its generators.

The concept of modality adds flexibility to the construc-tion of structures by allowing generators of same modalityto be exchangeable and take on different roles in one sameconfiguration. Generators combine to each other throughtheir bond structures to form configurations that representcomplex patterns of interest. Bond structures account forontological constraints stemming from logical, contextual,and temporal dependence, which ensure construction ofconfigurations with coherent pattern structures. Formally,

Definition 3 A configuration c = σ(g1, g2, ..., gn) is a con-nected graph structure composed by n generators gi ∈ Gthat respect the bond relation ρ.

Remark 3 For each closed bond gi′′ ↓ gi′′ in a configu-ration, ρ(βj′ (gi′ ), βj′′ (gi′′ )) returns TRUE to indicate thatthe out-bond βj′ (gi′ ) and in-bond βj′′ (gi′′ ) are compatible.

A configuration is a graph structure whose connectionsare identified as closed bonds by pairs of generators. Bondsfrom a configuration carry energy values that collectivelymeasure the quality of the pattern structure they compose.Such energy value is formulated as the response of an ac-ceptor function A(βj′ (gi′ ), βj′′ (gi′′ )).

Definition 4 The acceptor function A(.) measures theworth value of a closed bond gi′ ↓ gi′′ in terms someontological constraints to form a certain pattern structure.A(.)’s computation varies with the type of compatibility be-tween generators.

The probability of a configuration c = σ(g1, . . . , gn) isexpressed as a product of terms associated with its genera-tors gi ∈ G and closed bonds gi′′ ↓ gi′′ .

p(σ(.)) =

∏n(k,k′ )∈σ A1/T (βj(gi), βj′ (gi′ ))

Z(T ), (1)

where k = βj(gi) and k′= βj′ (gi′ ) denote bonds of

generators, Z(T ) is the partition function, T is set to 1,and n denotes the number of generators that form an in-terpretation. Thus, its energy equivalent form is E(σ(.)) =− log p(σ(.))Z(T ), which results in

E(σ(.)) = −∑

(k,k′ )∈σ

logA(βj(gi), βj′ (gi′ )) (2)

Finding a certain pattern structure using some chosengenerator space as basis means to look into a combinato-rial search space. Since it is computationally prohibitive toenumerate all possible configurations and measure the qual-ity of each one, it is typical to consider stochastic processesfor the purpose, which consists of minimizing E(σ(.)).

(a)

(b)

(c)

Figure 3: This illustration shows a sample of generatorsform the chosen generator space. They are shown withthe names of ontological concepts they represent and theirbond structures. (a), (b), and (c) depict generators of ac-tions, objects and features, respectively. In-bond are shownin shaded semi-circles and out-bonds in white-filled semi-and blanket bonds are out-bonds. The bond values of thesebonds are the names of modality groups.

3.1. Generator Space

The chosen generator space G is organized into 3 levelsof abstraction. In each level resides elements with similarcharacteristics. Level 1 consists of generators to representmotion and shape features. Level 2 is formed by generatorsthat represent object labels. Level 3 contains generators torepresent actions. In each level generators can be furtherclustered into modality groups. An illustration of the gen-erator space is shown in Figure 3 and a through descriptionfollows below.

In particular, motion features histograms of visual words(BoVW) constructed using a visual dictionary of his-tograms of optic flow (HOF). Shape features are given byhistograms of oriented gradients. For each video segmentthere is a single HOF-based BoVW, formulated as a motionfeature generator. HOGs are computed for each bound-ing box track available across the video segment sequence.Thus, each video segment will be associated with multipleshape feature generators, each corresponding to a boundingbox track of some detected object.

Feature generators provide support for concepts repre-sented by ontological generators that convey he seman-tics of a video interpretation.These connections are closedbonds named support bonds. Ontological generators ofaction concepts bond to ontological generators of object

concepts to indicate interactions (e.g., generator stir bondto generator spatula) and ontological generators of objectsconnect to other generators of objects to also indicate inter-action between the involved objects; these connections arecalled semantic bonds.

Generators of actions bond to other generators of actionsto sustain temporal coherence between actions in interpre-tations of consecutive video segments.These are called tem-poral bonds and is the main contribution of this work. Bothsemantic and temporal bonds are considered ontologicalbonds for forming connections between ontological gener-ators.

3.2. Bonds: Combinations of Generators

Two arbitrary generators gi ∈ G and gj ∈ G connect toeach other through a single bond whose bond values con-form to the bond relation ρ(.). Bond values of out-bondsare names of modality groups. For example, the generatorstir has an out-bond with bond value stirrer, indicatingthat it can connect to any of the generators in the modal-ity group stirrer = {spatula, whisk}. For being in themodality group stirrer, spatula has a variable number ofin-bonds of bond value stirrer, which means that it can beconnected to any other generator with an out-bond of bondvalue stirrer.

Support, semantic and temporal bonds carry bondenergies that quantify how valuable for the interpreta-tion a combination between two generators is. Thebond energies are computed using the acceptor functionA(βj′ (gi′ ), βj′′ (gi′′ )) = exp(tanh(kf(gi, gj))). For sup-port bonds, f(gi, gj) is the classification score output bythe gi’s classifier that determine how likely is for the fea-ture generator gj to belong to the concept represented by gi.Here concept classifiers are multi-class linear-SVM classi-fication models, for both actions and objects. For seman-tic and temporal bonds, f(gi, gj) come from concept co-occurrence tables.

3.3. Pattern Theoretic Video Interpretations

A long video consists of a sequence of video segmentswith variable lengths. Each video segment depicts a basicinteraction depicting a steps of cooking a recipe. A videointerpretation is a sequence of interpretations for a tempo-ral window containing a single video segment or multipleconsecutive segments. A unit of interpretation is definedby the limits of a single temporal window. Each tempo-ral window is associated with a set of feature generators.An interpretation is generated for each temporal window bysolving the optimization problem indicated in Equation 2.To this end, we implement a MCMC-based simulated an-nealing algorithm. This inference algorithm makes use ofa local proposal function that makes small changes to varythe interpretation structure and a global proposal function

that samples interpretations from a global jump configura-tion built with the k best generators for each feature. Gen-erating an interpretation is a combinatorial search problemwhose elements of comibations are generators. The searchspace size varies with the number of features available. If atemporal window consists of one motion feature generatorand two shape feature generators, then there are more than1900 possible interpretations (6 actions × 18 objects × 18objects); the search space size grows exponentially with thenumber of features.

4. ResultsIn this section, we analyzed the numerical performance

and qualitative advantages of using the pattern theoretic ap-proach with temporal bonds. First, we evaluated the qualityof output interpretations by analyzing samples taken fromthe experiments. We discussed the effects of adding tempo-ral bonds to the bond structure of generators and in whichscenarios temporal bonds can lead to more interesting (de-sirable) interpretations. We also analyzed how critical theinclusion of temporal bonds to the model is when interpre-tations are based on multiple segments of videos.Then, weevaluated the performance in controlled scenarios of clas-sification errors stemming from synthetic concept classi-fiers. We finalized our discussion with a comparative per-formance analysis on the YouCook data set when using realmachine learning based concept classifiers.

For comparative analysis, we contrasted the performanceprofile of the proposed approach with [1]’s and a base-line algorithm that generates interpretations exclusivelybased on the best classification scores using linear-SVMclassification models (i.e. a purely machine learning-basedmethod). The performance metric consisted of counting thenumber of correct ontological generators found in the inter-pretation given the ground-truth’s. The highest performancerate is 1 and lowest is 0. For example, the performance rateof the interpretation in Figure 6j is 0.86.

4.1. Interpretations with Temporal Bonds

Temporal bonds allow the pattern theoretic process totake into account temporal dependence information be-tween consecutive actions, accordingly, interpretations. Wefound several cases in which temporal bonds helped identi-fying the correct actions across multiple consecutive videosegment interpretations. Four of these cases of success areillustrated in Figures 4, 5 and 6.

Figures 4e and 4f show two interpretations generatedby [1], each for one of two consecutive video segments. Re-call that each video segment depicts some action that resultsin an interaction with one or multiple objects. These inter-pretations were generated by optimizing the energy func-tion for each video segment, separately. [1]’s method failsto find the correct action for the first video segment, in-

terpreting it as pick up instead of put down. Contrarily,the pattern theoretic interpretation using temporal bonds,shown in Figure 4k, successfully identifies the action putdown, while maintaining the good semantic bonds capturedby [1]. Unlike our approach, applying [1]’s method to gen-erate an unified interpretation for the two segments at oncedoes not produce a better interpretation than the ones gen-erated based on separate inference of each segment. In fact,this interpretation, illustrated in Figure 4i, introduced moreerrors, confusing pick up by put down in both segments.

Another example of success by our approach taken fromthe experiments is depicted in Figure 4l, where the actionstir was correctly inferred for describing the interaction oc-curring in the second segment. The approach in [1] failed todetermine the action stir, instead inferring pick up in the sin-gle segment scenario and season in multiple segment one.

In both illustrated cases, not only was our approach ableto generate improved interpretations through the addition oftemporal bonds but it also preserved relevant generators ofobjects and bonds correctly identified in the single-segmentbased inference from [1]’s approach. This same effecthas been observed for larger set of segments, as illustratedin Figures 5 and 6. We discussed these cases in the nextsection to point the benefits of generating interpretationsfor larger temporal windows containing multiple video seg-ments. Since more degrees of freedom are available whenconsidering temporal bonds, this permitted our approach toexplore other possibilities of interpretations with more con-fidence than when they were not present.

4.2. Interpretations for Multiple Segments

We identified several cases in which interpretations gen-erated for multiple-segment temporal windows using ourapproach helped determine the correct interpretations of ac-tions for video segments that are misinterpreted by [1]’s ap-proach. Temporal bonds allow our approach to not onlysearch for coherent local interactions but also naturally fo-cus on identifying the correct temporal ordering of actionsin adjacent video segments.

For instance in Figures 5 and 6, our approach’s interpre-tation (Figures 5j and 6j) was able to preserve the correctlydetected objects found in interpretations generated by [1](Figures 5e-5i and Figures 6e-6i) while fixing the action in-terpretations of consecutive video segments. More inter-estingly, the case depicted in Figure 5 shows that our ap-proach’s interpretation leveraged the confidence of the ac-tion interpretation in the third segment, put down, to prop-agate multiple corrections in the two past segments and thevideo segment ahead; the action interpreted sequence wasput down → pick up → put down → pick up (Figure 5j).Nonetheless, the same effect was not observed when using[1]’s method, which produced the sequence pick up→ putdown→ put down→ put down (Figure 5i).

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j)

(k) (l)

Figure 4: (a) and (b) illustrate two consecutive video seg-ments describing steps for making french toast. (c) and(d) illustrate two consecutive video segments describingsteps for making dough. The pairs (e)-(f) and (g)-(h) showthe corresponding interpretations by [1] based on single-segment windows, while (i) and (j) are derived from two-segment windows. (k) and (l) present corresponding inter-pretations generated by our approach.

In this work, temporal bonds were only explored at thelevel of actions under the assumption that coherence in de-termining the participating objects in interactions is mostlydependent on correctly identifying the action being per-formed. The focus then revolves around finding tempo-ral coherent interpretations in terms of actions. Identifyingthe correct sequence of actions indirectly influences on thequality of the overall sequence of interpretations.

4.3. Experiments with Synthetic Classifiers

Classification scores help measure the quality of an inter-pretation; therefore, they are essential for ascertaining theglobal optimal interpretation. We set up two controlled sce-

(a) (b) (c) (d)

(e) (f) (g) (h)

(i)

(j)

Figure 5: (a)-(d) illustrate four consecutive video segmentsdescribing steps for making salad. (e)-(h) show interpreta-tions by [1] based on single-segment windows, and (i) forthe four segments at once. (j) depicts the interpretation byour approach.

narios of degradation stemming from concept classifiers inorder to evaluate our approach’s tolerance to classificationerrors. First, we varied the classification error rates of theclassifiers from 10% to 60%. In these cases, the classifica-tion score ranks of the correct labels for the affected featuresare the second best. Then, we fixed the classification errorrate to 50% and vary the score rank of the feature’s correctlabels from 2 to 5.

Figure 7 shows the performance profile of the ap-proaches for increasing rates of classification error. In Fig-ure 7a, where interpretations are generated for each individ-ual video segment, our approach and [1]’s were superior tothe baseline’s, but only for high rates of classification er-ror (>20%). In this same case, our approach and [1]’s hadcomparable performance rates, since no temporal data couldbe explored. Our approach produced performance improve-ment increase of more than 7% over [1]’s for larger tempo-ral windows, multiple video segments at once (Figure 7b).

(a) (b) (c) (d)

(e) (f) (g) (h)

(i)

(j)

Figure 6: (a)-(d) illustrate four consecutive video segmentsdescribing steps for making sandwich. (e)-(h) show inter-pretations by [1] based on single-segment windows, and (i)for the four segments at once. (j) present the interpretationby our approach.

In summary, if the concept classifiers are not sufficientlygood to be used alone, these results indicate that ontology-based approaches like ours and [1]’s are imperative in orderto achieve reasonably sufficient performance.

Figure 8 shows the performance profiles in which ap-proximately 50% of the features had their correct labels’sclassification scores as the kth best classification scores,where k varies from 2 to 5. Overall, Figures 8a-8b showthat our approach was consistently capable of correctingthe feature labeling, even when the feature correct label hadthe fifth best classification score. Figure 8b shows that formultiple-segment based inference our approach is consis-tently superior to [1]’s, with up to 12% increase. This sug-gests that under uncertain scenarios, our approach wouldbe more advantageous because it improves performanceby generating video interpretations based on multiple seg-ments.

4.4. Experiments with Real Classifiers

Overall our approach improved the total average inter-pretation performance by approximately 5 times the base-

(a) 1-segment temporal window (b) 4-segment temporal window

Figure 7: Interpretation performance profiles for varyingscenarios of classification error rates, ranging from 10% to60%.

(a) 1-segment temporal window (b) 4-segment temporal window

Figure 8: Interpretation performance profiles showing thetolerance of comparative approaches for decreasing rank ofclassification scores of the correct label for about 50% ofthe video features.

line’s and by ∼30% over [1]’s when no bond types weregiven preference over others (Table 1). Three types ofbonds contribute to measure the quality of an interpreta-tion (see Section 3.2). Figure 9 shows how the overall aver-age video interpretation performance varied as certain typesof bonds were given more participation weight than others.In all cases, overweighting support bonds dropped the in-terpretation performance rate to the baseline’s, which waslow because of the weak concept classifiers. More weighton the participation of temporal bonds was sufficient toachieve higher performance rates for all multiple-segmentcases (Figure 9b). This emphasizes our assumption that cor-rect action interpretations should naturally lead to correctidentification of the true involved objects. Overweightingsemantic bonds was influential mostly in the single-segmentcase (Figure 9a), where temporal bonds were not relevant.

We also observed the average performance when con-sidering the top k interpretations for describing each video.Figure 10a shows that our approach and [1]’s had com-parable performance for the single-segment case when notemporal information is available. When generating multi-ple interpretations for temporal windows containing multi-ple consecutive segments, our approach provided the bestoverall performance. In comparison to [1]’s original idea of

Table 1: Comparison of overall average performance ofvideo interpretations for increasing temporal window sizes(# video segments).

Method 1 2 3 4Ours 0.40 0.52 0.50 0.52

Souza et al. 0.41 0.44 0.41 0.46Baseline 0.11 0.12 0.13 0.13

analyzing individual segments, it nearly doubling the per-formance, with improvements ranging from about 67% to73%, depending on the number k (compare Figure 10b withFigure 10b).

(a) 1-segment temporal window (b) 4-segment temporal window

Figure 9: Video interpretations performance profile of ourapproach when different bond types have more participationthan others.

(a) 1-segment temporal window (b) 4-segment temporal window

Figure 10: Video performance interpretation when con-sidering the top k best interpretations for describing eachvideo.

5. Time Complexity and Running TimeOur global and local proposal functions explore the

bond-constrained space of feasible solutions in an efficientmanner. The time computational complexity was workedout to be O(k ∗ mc ∗ mo + k(nf + mo ∗ mo)), where kis the total number of sampling iterations, mc is the numberbonds from a candidate generator for replacement,mo is thetotal number of open bonds in a current interpretation andnf is the number of feature generators. For all the experi-ments we fixed k = 3000. The running time grows linearly

Table 2: Average CPU+I/O time of videos per number offeature generators nf . Machine spec: 4 16-core 2.3 GHzCPUs (AMD Opteron 6376), 16 16GB RAM units.

nf 30 33 34 36 37 44 46 48sec 279 342 382 399 375 477 513 550

with the number of feature generators nf , consistent withour analysis (see Table 2).

6. Conclusion

In this paper we advanced the adaption of Grenander’spattern theory originally proposed in [1] by introducing abond structure that captures temporal information. This al-lowed us to generate temporally coherent semantic inter-pretations of videos. Similar to [1], the basic units of inter-est (i.e., actions and objects) are denoted by generators thatcombine to each other to form graphical structures, whichrepresent video interpretations. The quality of an inter-pretation is governed by the energies of its bonds. Thesebond energies are defined using classification scores andfrequency tables of concept co-occurrences, which help de-fine and seek optimal configurations. While previous ap-plications have been restricted to analyzing short videoscontaining isolated actions, we have extended this idea tolonger videos using additional bond structures, which al-low interactions between actions that are adjacent in time.The aforementioned experiments, involving more than 700video segments from the YouCook data set, demonstratedthe power of adding action temporal bonds in the configu-rations. Not only did we improve the performance in detec-tion of generators but we also improved the overall sceneinterpretations. In addition, the our approach was morerobust to degradation in feature-level classification perfor-mance than its counterparts.

The use of additional bond structures clearly helped in-terpret more complex scenes and allowed for enhanced in-ferences. In view of the flexibility of this pattern theo-retic framework in representing complex systems, in futurework, we plan to include additional (types of) generatorsand bond structures. In this future extension, configurationsthat represent interpretations of small video segments canbe turned into composite generators, naturally augmentingthe representation hierarchical system, which can be used tohelp understand even longer videos depicting more complexactivities.

Acknowledgment

This research was supported in part by NSF grants1217515 and 1217676.

References[1] F. D. M. de Souza, S. Sarkar, A. Srivastava, and J. Su,

“Pattern theory-based interpretation of activities,” in Inter-national Conference on Pattern Recognition (ICPR), 2014.1, 2, 3, 5, 6, 7, 8

[2] U. Grenander and M. I. Miller, Pattern theory: from repre-sentation to inference, 2007, vol. 1. 1

[3] H. Pirsiavash and D. Ramanan, “Detecting activities of dailyliving in first-person camera views,” in IEEE Conference onComputer Vision and Pattern Recognition (CVPR), 2012, pp.2847–2854. 2

[4] A. Fathi and J. M. Rehg, “Modeling actions through statechanges,” in IEEE Conference on Computer Vision and Pat-tern Recognition (CVPR), 2013, pp. 2579–2586. 2

[5] P. Bojanowski, R. Lajugie, F. Bach, I. Laptev, J. Ponce,C. Schmid, and J. Sivic, “Weakly supervised action labelingin videos under ordering constraints,” in European Confer-ence on Computer Vision (ECCV), 2014, pp. 628–643. 2

[6] C. C. Tan, Y.-G. Jiang, and C.-W. Ngo, “Towards textuallydescribing complex video contents with audio-visual conceptclassifiers,” in ACM International Conference on Multimedia(MM), 2011, pp. 655–658. 2

[7] I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld,“Learning realistic human actions from movies,” in IEEEConference on Computer Vision and Pattern Recognition(CVPR), 2008, pp. 1–8. 2

[8] S. Bhattacharya, M. M. Kalayeh, R. Sukthankar, andM. Shah, “Recognition of complex events: Exploiting tem-poral dynamics between underlying concepts,” in IEEEConference on Computer Vision and Pattern Recognition(CVPR), 2014. 2

[9] K. Tang, L. Fei-Fei, and D. Koller, “Learning latent temporalstructure for complex event detection,” in IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), 2012,pp. 1250–1257. 2

[10] M. U. G. Khan, L. Zhang, and Y. Gotoh, “Towards coher-ent natural language description of video streams,” in IEEEInternational Conference on Computer Vision Workshops(ICCV Workshops), 2011, pp. 664–671. 2

[11] N. Krishnamoorthy, G. Malkarnenkar, R. Mooney,K. Saenko, U. Lowell, and S. Guadarrama, “Generat-ing natural-language video descriptions using text-minedknowledge,” Conference of the North American Chapterof the Association for Computational Linguistics: HumanLanguage Technologies, p. 10, 2013. 2

[12] P. Das, C. Xu, R. F. Doell, and J. J. Corso, “A thousandframes in just a few words: Lingual description of videosthrough latent topics and sparse object stitching,” in IEEEConference on Computer Vision and Pattern Recognition(CVPR), 2013, pp. 2634–2641. 2

[13] B. Laxton, J. Lim, and D. Kriegman, “Leveraging temporal,contextual and ordering constraints for recognizing complexactivities in video,” in IEEE Conference on Computer Visionand Pattern Recognition (CVPR), 2007, pp. 1–8. 2

[14] E. Swears, A. Hoogs, Q. Ji, and K. Boyer, “Complex activityrecognition using granger constrained dbn (gcdbn) in sportsand surveillance video,” in IEEE Conference on ComputerVision and Pattern Recognition (CVPR), 2014. 2, 3

[15] M. R. Amer and S. Todorovic, “Sum-product networksfor modeling activities with stochastic structure,” in IEEEConference on Computer Vision and Pattern Recognition(CVPR), 2012, pp. 1314–1321. 2

[16] M. R. Amer, D. Xie, M. Zhao, S. Todorovic, and S.-C. Zhu,“Cost-sensitive top-down/bottom-up inference for multiscaleactivity recognition,” in European Conference on ComputerVision (ECCV), 2012, pp. 187–200. 2

[17] M. S. Ryoo and J. K. Aggarwal, “Semantic representationand recognition of continued and recursive human activities,”International Journal of Computer Vision (IJCV), vol. 82,no. 1, pp. 1–24, 2009. 2

[18] N. Vo and A. Bobick, “From stochastic grammar to bayesnetwork: Probabilistic parsing of complex activity,” in IEEEConference on Computer Vision and Pattern Recognition(CVPR), 2014. 2

[19] K. Hilde, A. Arslan, and T. Serre, “The language of actions:Recovering the syntax and semantics of goal-directed humanactivities,” in IEEE Conference on Computer Vision and Pat-tern Recognition (CVPR), 2014. 2, 3

[20] H. Pirsiavash and D. Ramanan, “Parsing videos of actionswith segmental grammars,” 2014. 2

[21] P. Wei, Y. Zhao, N. Zheng, and S.-C. Zhu, “Modeling4d human-object interactions for event and object recogni-tion,” in IEEE International Conference on Computer Vision(ICCV), 2013, pp. 3272–3279. 2

[22] M. R. Amer, S. Todorovic, A. Fern, and S.-C. Zhu, “Montecarlo tree search for scheduling activity recognition,” inIEEE International Conference on Computer Vision (ICCV),2013, pp. 1353–1360. 2

[23] M. Albanese, R. Chellappa, V. Moscato, A. Picariello,V. Subrahmanian, P. Turaga, and O. Udrea, “A constrainedprobabilistic petri net framework for human activity detec-tion in video,” IEEE Transactions on Multimedia, vol. 10,no. 6, pp. 982–996, 2008. 2

[24] T. Lan, L. Sigal, and G. Mori, “Social roles in hierarchicalmodels for human activity recognition,” in IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), 2012,pp. 1354–1361. 2

[25] A. Fathi, J. K. Hodgins, and J. M. Rehg, “Social interac-tions: A first-person perspective,” in IEEE Conference onComputer Vision and Pattern Recognition (CVPR), 2012, pp.1226–1233. 2