Embed Size (px)
Citation preview
Noisy Submodular Maximization via Adaptive Samplingwith Applications to Crowdsourced Image Collection Summarization
Adish SinglaETH Zurich
Sebastian TschiatschekETH Zurich
Andreas KrauseETH Zurich
Abstract
We address the problem of maximizing an unknownsubmodular function that can only be accessed via noisyevaluations. Our work is motivated by the task of sum-marizing content, e.g., image collections, by leveragingusers’ feedback in form of clicks or ratings. For sum-marization tasks with the goal of maximizing cover-age and diversity, submodular set functions are a nat-ural choice. When the underlying submodular functionis unknown, users’ feedback can provide noisy evalua-tions of the function that we seek to maximize. We pro-vide a generic algorithm – EXPGREEDY – for maximiz-ing an unknown submodular function under cardinalityconstraints. This algorithm makes use of a novel explo-ration module – TOPX – that proposes good elementsbased on adaptively sampling noisy function evalua-tions. TOPX is able to accommodate different kinds ofobservation models such as value queries and pairwisecomparisons. We provide PAC-style guarantees on thequality and sampling cost of the solution obtained byEXPGREEDY. We demonstrate the effectiveness of ourapproach in an interactive, crowdsourced image collec-tion summarization application.
IntroductionMany applications involve the selection of a subset of items,e.g., summarization of content on the web. Typically, thetask is to select a subset of items of limited cardinality withthe goal of maximizing their utility. This utility is often mea-sured via properties like diversity, information, relevance orcoverage. Submodular set functions naturally capture thefore-mentioned notions of utility. Intuitively, submodularfunctions are set functions that satisfy a natural diminish-ing returns property. They have been widely used in diverseapplications, including content summarization and recom-mendations, sensor placement, viral marketing, and numer-ous machine learning and computer vision tasks (Krause andGuestrin 2008; Bilmes 2015).
Summarization of image collections. One of the moti-vating applications, which is the subject of our experimentalevaluation, is summarization of image collections. Given acollection of images, say, from Venice on Flickr, we would
Copyright © 2016, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.
like to select a small subset of these images that summa-rizes the theme Cathedrals in Venice, cf., Figure 1. We castthis summarization task as the problem of maximizing a sub-modular set function f under cardinality constraints.
Submodular maximization. Usually, it is assumed thatf is known, i.e., f can be evaluated exactly by querying anoracle. In this case, a greedy algorithm is typically used formaximization. The greedy algorithm builds up a set of itemsby picking those of highest marginal utility in every itera-tion, given the items selected that far. Despite its greedy na-ture, this algorithm provides the best constant factor approx-imation to the optimal solution computable in polynomialtime (Nemhauser, Wolsey, and Fisher 1978).
Maximization under noise. However, in many realisticapplications, the function f is not known and can only beevaluated up to (additive) noise. For instance, for the im-age summarization task, (repeatedly) querying users’ feed-back in form of clicks or ratings on the individual imagesor image-sets can provide such noisy evaluations. There areother settings for which marginal gains are hard to com-pute exactly, e.g., computing marginal gains of nodes in vi-ral marketing applications (Kempe, Kleinberg, and Tardos2003) or conditional information gains in feature selectiontasks (Krause and Guestrin 2005). In such cases, one canapply a naive uniform sampling approach to estimate allmarginal gains up to some error ε and apply the standardgreedy algorithm. While simple, this uniform sampling ap-proach could have high sample complexity, rendering it im-practical for real-world applications. In this work, we pro-pose to use adaptive sampling strategies to reduce samplecomplexity while maintaining high quality of the solutions.
Our approach for adaptive sampling. The first key in-sight for efficient adaptive sampling is that we need to esti-mate the marginal gains only up to the necessary confidenceto decide for the best item to select next. However, if the dif-ference in the marginal gains of the best item and the secondbest item is small, this approach also suffers from high sam-ple complexity. To overcome this, we exploit our second keyinsight that instead of focusing on selecting the single bestitem, it is sufficient to select an item from a small subset ofany size l ∈ {1, . . . , k} of high quality items, where k is thecardinality constraint. Based on these insights, we proposea novel exploration module – TOPX – that proposes goodelements by adaptively sampling noisy function evaluations.
(a) Image collection to be summarized
EXPGREEDY : ”Venice” using pairwise comparisons via crowdsourcing
EXPGREEDY : ”Venice Carnival” using pairwise comparisons via crowdsourcing
EXPGREEDY : ”Venice Cathedrals” using pairwise comparisons via crowdsourcing
1 2 3 4 5 6
1 2 3 4 5 6
1 2 3 4 5 6
Inferred Borda score of selected item
Inferred Borda score of selected item
Inferred Borda score of selected item
0.833 0.806 0.816 0.772 0.786 0.80
0.848 0.888 0.877 0.852 0.884 0.819
0.850 0.864 0.890 0.862 0.880 0.812
(b) Results for three summarization tasks
Figure 1: (a) Image collection V to be summarized; (b) Summaries obtained using pairwise comparisons via crowdsourcing for the themes(i) Venice, (ii) Venice Carnival, and (iii) Venice Cathedrals. Images are numbered from 1 to 6 in the order of selection.
Our contributions. Our main contributions are:• We provide a greedy algorithm EXPGREEDY for sub-
modular maximization of a function via noisy evalua-tions. The core part of this algorithm, the explorationmodule TOPX, is invoked at every iteration and imple-ments a novel adaptive sampling scheme for efficientlyselecting a small set of items with high marginal utilities.
• Our theoretical analysis and experimental evaluation pro-vide insights of how to trade-off the quality of subsets se-lected by EXPGREEDY against the number of evaluationqueries performed by TOPX.
• We demonstrate the applicability of our algorithms in areal-world application of crowdsourcing the summariza-tion of image collections via eliciting crowd preferencesbased on (noisy) pairwise comparisons, cf., Figure 1.
Related WorkSubmodular function maximization (offline). Submodu-lar set functions f(S) arise in many applications and, there-fore, their optimization has been studied extensively. For ex-ample, a celebrated result of Nemhauser, Wolsey, and Fisher(1978) shows that non-negative monotone submodular func-tions under cardinality constraints can be maximized upto a constant factor of (1 − 1/e) by a simple greedy al-gorithm. Submodular maximization has furthermore beenstudied for a variety of different constraints on S, e.g., ma-troid constraints or graph constraints (Krause et al. 2006;Singh, Krause, and Kaiser 2009), and in different settings,e.g., distributed optimization (Mirzasoleiman et al. 2013).When the function f can only be evaluated up to (addi-tive) noise, a naive uniform sampling approach has been em-ployed to estimate all the marginal gains (Kempe, Kleinberg,and Tardos 2003; Krause and Guestrin 2005), an approachthat could have high sampling complexity.
Learning submodular functions. One could approachthe maximization of an unknown submodular function byfirst learning the function of interest and subsequently opti-mizing it. Tschiatschek et al. (2014) present an approach forlearning linear mixtures of known submodular componentfunctions for image collection summarization. Our work iscomplimentary to that approach wherein we directly targetthe subset selection problem without learning the underly-ing function. In general, learning submodular functions from
data is a difficult task — Balcan and Harvey (2011) provideseveral negative results in a PAC-style setting.
Submodular function maximization (online). Ourwork is also related to online submodular maximiza-tion with (opaque) bandit feedback. Streeter and Golovin(2008) present approaches for maximizing a sequence ofsubmodular functions in an online setting. Their adversar-ial setting forces them to use conservative algorithms withslow convergence. Yue and Guestrin (2011) study a morerestricted setting, where the objective is an (unknown) linearcombination of known submodular functions, under stochas-tic noise. While related in spirit, these approaches aim tominimize cumulative regret. In contrast, we aim to identifya single good solution performing as few queries as possible— the above mentioned results do not apply to our setting.
Best identification (pure exploration bandits). In ex-ploratory bandits, the learner first explores a set of actionsunder time / budget constraints and then exploits the gath-ered information by choosing the estimated best action (top1
identification problem) (Even-Dar, Mannor, and Mansour2006; Bubeck, Munos, and Stoltz 2009). Beyond best indi-vidual actions, Zhou, Chen, and Li (2014) design an (ε, δ)-PAC algorithm for the topm identification problem wherethe goal is to return a subset of size m whose aggregate util-ity is within ε compared to the aggregate utility of them bestactions. Chen et al. (2014) generalize the problem by con-sidering combinatorial constraints on the subsets that can beselected, e.g., subsets must be size m, represent matchings,etc. They present general learning algorithms for all decisionclasses that admit offline maximization oracles. Duelingbandits are variants of the bandit problem where feedback islimited to relative preferences between pairs of actions. Thebest-identification problem is studied in this weaker infor-mation model for various notions of ranking models (e.g.,Borda winner), cf., Busa-Fekete and Hullermeier (2014).However, in contrast to our work, the reward functions con-sidered by Chen et al. (2014) and other existing algorithmsare modular, thus limiting the applicability of these algo-rithms. Our work is also related to contemporary work byHassidim and Singer (2015), who treat submodular maxi-mization under noise. While their algorithms apply to per-sistent noise, their technique is computationally demanding,and does not enable one to use noisy preference queries.
Problem Statement
Utility model. Let V = {1, 2, . . . , N} be a set of N items.We assume a utility function f : 2V → R over subsets of V .Given a set of items S ⊆ V , the utility of this set is f(S).Furthermore, we assume that f is non-negative, monotoneand submodular. Monotone set functions satisfy f(S) ≤f(S′) for all S ⊆ S′ ⊆ V; and submodular functionssatisfy the following diminishing returns condition: for allS ⊆ S′ ⊆ V\{a}, it holds that f(S∪{a})−f(S) ≥ f(S′∪{a}) − f(S′). These conditions are satisfied by many real-istic, complex utility functions (Krause and Guestrin 2011;Krause and Golovin 2012). Concretely, in our image col-lection summarization example, V is a collection of images,and f is a function that assigns every summary S ⊆ V ascore, preferring relevant and diverse summaries.
Observation model. In classical submodular optimiza-tion, f is assumed to be known, i.e., f can be evaluated ex-actly by an oracle. In contrast, we only assume that noisyevaluations of f can be obtained. For instance, in our sum-marization example, one way to evaluate f is to query usersto rate a summary, or to elicit user preferences via pairwisecomparisons of different summaries. In the following, weformally describe these two types of queries in more detail:
(1) Value queries. In this variant, we query the value forf(a|S) = f({a} ∪ S) − f(S) for some S ⊆ V and a ∈ V .We model the noisy evaluation or response to this query bya random variable Xa|S with unknown sub-Gaussian distri-bution. We assume that Xa|S has mean f(a|S) and that re-peated queries for f(a|S) return samples drawn i.i.d. fromthe unknown distribution.
(2) Preference queries. Let a, b ∈ V be two items andS ⊆ V . The preference query aims at determining whetheran item a is preferred over item b in the context of S (i.e.,item a has larger marginal utility than another item b). Wemodel the noisy response of this pairwise comparison bythe random variable Xa>b|S that takes values in {0, 1}.We assume that Xa>b|S follows some unknown distribu-tion and satisfies the following two properties: (i) Xa>b|Shas mean larger than 0.5 iff f(a|S) > f(b|S); (ii) the map-ping from utilities to probabilities is monotone in the sense,that if given some set S, and given that the gaps in utili-ties satisfy f(a|S)− f(b|S) ≥ f(a′|S)− f(b′|S) for itemsa, b, a′, b′ ∈ V , then the mean of Xa>b|S is greater or equalto the mean of Xa′>b′|S . For instance, the distribution in-duced by the commonly used Bradley-Terry-Luce prefer-ence model (Bradley and Terry 1952; Luce 1959) satisfiesthese conditions. We again assume that repeated queries re-turn samples drawn i.i.d. from the unknown distribution.
Value queries are natural, if f is approximated viastochastic simulations (e.g., as in Kempe, Kleinberg, andTardos (2003)). On the other hand, preference queries maybe a more natural way to learn what is relevant / interest-ing to users compared to asking them to assign numericalscores, which are difficult to calibrate.
Objective. Our goal is to select a set of the items S ⊆ Vwith |S| ≤ k that maximizes the utility f(S). The optimal
solution to this problem is given bySopt = arg max
S⊆V,|S|≤kf(S). (1)
Note that obtaining optimal solutions to problem (1) isintractable (Feige 1998). However, a greedy optimizationscheme based on the marginal utilities of the items can pro-vide a solution Sgreedy such that f(Sgreedy) ≥ (1 − 1
e ) ·f(Sopt), i.e., a solution that is within a constant factor of theoptimal solution can be efficiently determined.
In our setting, we can only evaluate the unknown utilityfunction f via noisy queries and thus cannot hope to achievethe same guarantees. The key idea is that in a stochastic set-ting, our algorithms can make repeated queries and aggre-gate noisy evaluations to obtain sufficiently accurate esti-mates of the marginal gains of items. We study our proposedalgorithms in a PAC setting, i.e., we aim to design algorithmsthat, given positive constants (ε, δ), determine a set Sthat isε-competitive relative to a reference solution with probabil-ity of at least 1− δ. One natural baseline is a constant factorapproximation to the optimal solution, i.e., we aim to deter-mine a set S such that with probability at least 1− δ,
f(S) ≥ (1− 1
e) · f(Sopt)− ε. (2)
Our objective is to achieve the desired (ε, δ)-PAC guaran-tee while minimizing sample complexity (i.e., the numberof evaluation queries performed).
Submodular Maximization Under NoiseWe now present our algorithm EXPGREEDY for maximiz-ing submodular functions under noise. Intuitively, it aims tomimic the greedy algorithms in noise-free settings, ensuringthat it selects a good element in each iteration. Since we can-not evaluate the marginal gains exactly, we must experimentwith different items, and use statistical inference to selectitems of high value. This experimentation, the core part ofour algorithm, is implemented via a novel exploration mod-ule called TOPX.
The algorithm EXPGREEDY, cf., Algorithm 1, iterativelybuilds up a set S ⊆ V by invoking TOPX(ε′, δ′, k′, S) at ev-ery iteration to select the next item. TOPX returns candidateitems that could potentially be included in S to maximize itsutility. The simplest adaptive sampling strategy that TOPXcould implement is to estimate the marginal gains up to thenecessary confidence to decide for the next best item to se-lect (top1 identification problem). However, if the differencein the marginal gains of the best and the second best item issmall, this approach could suffer from high sample complex-ity. Extending ideas from the randomized greedy algorithm(Buchbinder et al. 2014), we show in Theorem 1 that in ev-ery iteration, instead of focusing on top1, it is sufficient forEXPGREEDY to select an item from a small subset of itemswith high utility. This corresponds to the following two con-ditions on TOPX:
1. TOPX returns a subset A 6= ∅ of size at most k′.2. With probability at least 1− δ′, the items in A satisfy
1
|A|∑a∈A
f(a|S) ≥ maxB⊆V|B|=|A|
[1
|B|∑b∈B
f(b|S)
]− ε′. (3)
Algorithm 1: EXPGREEDY
1 Input: Ground set V; No. of items to pick k; ε, δ > 0;2 Output: Set of items S ⊆ V : |S| ≤ k, such that S isε-competitive with probability at least (1− δ);
3 Initialize: S = ∅foreach j = 1, . . . , k do
4 A = TOPX(ε′, δ′, k′, S)5 Sample s uniformly at random from A6 S = S ∪ {s}7 return S
At any iteration, satisfying these two conditions is equiv-alent to solving a topl identification problem for l = |A|with PAC-parameters (ε′, δ′). TOPX essentially returns a setof size l containing items of largest marginal utilities givenS. Let us consider two special cases, (i) l = 1 and (ii) l = kfor all j ∈ {1, . . . , k} iterations. Then, in the noise free set-ting, EXPGREEDY for case (i) mimics the classical greedyalgorithm (Nemhauser, Wolsey, and Fisher 1978) and forcase (ii) the randomized greedy algorithm (Buchbinder etal. 2014), respectively. As discussed in the next section, thesample complexity of these two cases can be very high. Thekey insight we use in EXPGREEDY is that if we could effi-ciently solve the topl problem for any size l ∈ {1, . . . , k}(i.e., |A| is neither necessarily 1 or k), then the solution tothe submodular maximization problem is guaranteed to be ofhigh quality. This is summarized in the following theorem:Theorem 1. Let ε > 0, δ ∈ (0, 1). Using ε′ = ε
k , δ′ = δ
k andk′ = k for invoking TOPX, Algorithm 1 returns a set S thatsatisfies E[f(S)] ≥ (1 − 1
e ) · f(Sopt) − ε with probabilityat least 1 − δ. For the case that k′ = 1, the guarantee isf(S) ≥ (1− 1
e ) · f(Sopt)− ε with probability at least 1− δ.The proof is provided in Appendix A of the extended ver-
sion of paper (Singla, Tschiatschek, and Krause 2016).As it turns out, solutions of the greedy algorithm in the
noiseless setting Sgreedy often have utility larger than (1 −1e ) · f(Sopt). Therefore, we also seek algorithms that withprobability at least 1− δ identify solutions satisfying
f(S) ≥ f(Sgreedy)− ε. (4)This can be achieved according to the following theorem:Theorem 2. Let δ ∈ (0, 1). Using ε′ = 0, δ′ = δ
k and k′ = 1for invoking TOPX, Algorithm 1 returns a set S that satisfiesf(S) = f(Sgreedy) with probability at least 1− δ.
If we set k′ = 1 and ε = 0, then condition (3) is actuallyequivalent to requiring that TOPX, with high probability,identifies the element with largest marginal utility. The prooffollows by application of the union bound. This theorem en-sures that if we can construct a corresponding explorationmodule, we can successfully compete with the greedy algo-rithm that has access to f . However, this can be prohibitivelyexpensive in terms of the required number of queries per-formed by TOPX, cf., Appendix B of the extended versionof this paper (Singla, Tschiatschek, and Krause 2016).
Exploration Module TOPXIn this section, we describe the design of our explorationmodule TOPX used by EXPGREEDY.
items (sorted by utility)
0.0
0.2
0.4
0.6
0.8
1.0
uti
lity
average utilities
items (sorted by utility)
0.0
0.2
0.4
0.6
0.8
1.0Selecting top 2
items (sorted by utility)
0.0
0.2
0.4
0.6
0.8
1.0Selecting top 4
items (sorted by utility)
0.0
0.2
0.4
0.6
0.8
1.0Selecting best
Figure 2: An example to illustrate the idea of topl item se-lection with N = 5 items and parameter k′ = 4. While boththe top1 identification (l = 1) and the top4 identification(l = k′) have high sample complexity, the top2 identifica-tion (l = 2 ∈ {1, . . . , k′}) is relatively easy.
TOPX with Value QueriesWe begin with the observation that for fixed l ∈ {1, . . . , k′}(for instance l = 1 or l = k′), an exploration module TOPXsatisfying condition (3) can be implemented by solving atopl identification problem. This can be seen as follows.
Given input parameter S, for each a ∈ V \ S define itsvalue va = f(a|S), and define va = 0 for each a ∈ S.Then, the value of any set A (in the context of S) is v(A) :=∑a∈A va, which is a modular (additive) set function. Thus,
in this setting (fixed l), meeting condition (3) requires iden-tifying a setAmaximizing a modular set function under car-dinality constraints from noisy queries. This corresponds tothe topl best-arm identification problem. In particular, Chenet al. (2014) proposed an algorithm – CLUCB – for identify-ing a best subset for modular functions under combinatorialconstraints. At a high level, CLUCB maintains upper andlower confidence bounds on the item values, and adaptivelysamples noisy function evaluations until the pessimistic esti-mate (lower confidence bound) for the current best topl sub-set exceeds the optimistic estimate (upper confidence bound)of any other subset of size l. The worst case sample com-plexity of this problem is characterized by the gap betweenthe values of l-th item and (l+1)-th item (with items indexedin descending order according to the values va).
As mentioned, the sample complexity of the topl problemcan vary by orders of magnitude for different values of l, cf.,Figure 2. Unfortunately, we do not know the value of l withthe lowest sample complexity in advance. However, we canmodify CLUCB to jointly estimate the marginal gains andsolve the topl problem with lowest sample complexity.
The proposed algorithm implementing this idea is pre-sented in Algorithm 2. It maintains confidence intervals formarginal item values, with the confidence radius of an item i
computed as radt(i) = R√
2 log(
4Nt3
δ′
)/Tt(i), where the
corresponding random variables Xi|S are assumed to havean R-sub-Gaussian tail and Tt(i) is the number of observa-tions made for item i by time t. If the exact value of R isnot known, it can be upper bounded by the range (as longas the variables have bounded centered range). Upon ter-mination, the algorithm returns a set A that satisfies con-dition (3). Extending results from Chen et al. (2014), we canbound the sample complexity of our algorithm as follows.Consider the gaps ∆l = f(π(l)|S)− f(π(l + 1)|S), whereπ : V → {1, . . . , N} is a permutation of the items such that
Algorithm 2: TOPX1 Input: Ground set V; Set S; integer k′; ε′, δ′ > 0;2 Output: Set of items A ⊆ V : |A| ≤ k′, such that A
satisfies (3) with probability at least 1− δ′;3 Initialize: For all i = 1, . . . , |V|: observe Xi|S and setvi to that value, set T1(i) = 1;for t = 1, . . . do
4 Compute confidence radius radt(i) for all i;5 Initialize list of items to query Q = [];
for l ∈ 1, . . . , k′ do6 Mt = arg maxB⊆V,|B|=l
∑i∈B vi;
7 foreach i = 1, . . . , |V| do8 If i ∈Mt set vi = vi − radt(i), otherwise
set vi = vi + radt(i)
9 Mt = arg maxB⊆V,|B|=l∑i∈B vi;
10 if [∑i∈Mt
vi −∑i∈Mt
vi] ≤ l · ε′ then11 Set A = Mt and return A12 Set q = arg max
i∈(M\M)∪(M\M)radt(i);
13 Update list of items to query:Q = Q.append(q)
foreach q ∈ Q do14 Query and observe output Xq|S ;15 Update empirical means vt+1 using the output;16 Update observation counts Tt+1(q) = Tt(q) + 1
and Tt+1(j) = Tt(j) for all j 6= q;
f(π(1)|S) ≥ f(π(2)|S) ≥ . . . ≥ f(π(N)|S). For everyfixed l, the sample complexity of identifying a set of top litems is characterized by this gap ∆l. The reason is that if∆l is small, many samples are needed to ensure that the con-fidence bounds radt are small enough to distinguish the topl elements from the runner-up. Our key insight is that we canbe adaptive to the largest ∆l, for l ∈ {1, . . . , k′}. That is, aslong as there is some value of l with large ∆l, we will beable to enjoy low sample complexity (cf., Figure 2):Theorem 3. Given ε′ > 0, δ′ ∈ (0, 1), S ⊆ V and k′, Algo-rithm 2 returns a set A ⊆ V, |A| ≤ k′ that with probabilityat least 1− δ′ satisfies condition (3) using at most
T ≤ O(k′ minl=1,...,k′
[R2H(l,ε′) log
(R2
δ′H(l,ε′)
)])samples, where H(l,ε′) = N min{ 4
∆2l, 1ε′2 }.
A proof sketch is given in Appendix C of the extended ver-sion of this paper (Singla, Tschiatschek, and Krause 2016).
TOPX with Preference QueriesWe now show how noisy preference queries can be used.As introduced previously, we assume that there exists an un-derlying preference model (unknown to the algorithm) thatinduces probabilities Pi>j|S for item i to be preferred overitem j given the values f(i|S) and f(j|S). In this work,we focus on identifying the Borda winner, i.e., the itemi maximizing the Borda score P (i|S), formally given as
1(N−1) ·
∑j∈V\{i} Pi>j|S . The Borda score measures the
probability that item i is preferred to another item cho-sen uniformly at random. Furthermore, in our model where
we assume that an increasing gap in the utilities leads tomonotonic increase in the induced probabilities, it holdsthat the top l items in terms of marginal gains are thetop l items in terms of Borda scores. We now make useof a result called Borda reduction (Jamieson et al. 2015;Busa-Fekete and Hullermeier 2014), a technique that allowsus to reduce preference queries to value queries, and to con-sequently invoke Algorithm 2 with small modifications.
Defining values vi via Borda score. For each itemi ∈ V , Algorithm 2 (step 3, 15) tracks and updatesmean estimates of the values vi. For preference queries,these values are replaced with the Borda scores. The sam-ple complexity in Theorem 3 is then given in terms ofthese Borda scores. For instance, for the Bradley-Terry-Luce preference model (Bradley and Terry 1952; Luce1959), the Borda score for item i is given by 1
(N−1) ·∑j∈V\{i}
11+exp (−β(f(i|S)−f(j|S))) . Here, β captures the
problem difficulty: β →∞ corresponds to the case of noise-free responses, and β → 0 corresponds to uniformly randombinary responses. The effect of β is further illustrated in thesynthetic experiments.
Incorporating noisy responses. Observing the value foritem i in the preference query model corresponds to pairing iwith an item in the set V \{i} selected uniformly at random.The observed preference response provides an unbiased es-timate of the Borda score for item i. More generally, we canpick a small set Zi of fixed size τ selected uniformly at ran-dom with replacement from V \ {i}, and compare i againsteach member of Zi. Then, the observed Borda score for itemi is calculated as 1
τ ·∑j∈Zi
Xi>j|S . Here, τ is a parameterof the algorithm. One can observe that the cost of one prefer-ence query is τ times the cost of one value query. The effectof τ will be further illustrated in the experiments.
The subtle point of terminating Algorithm 2 (step 10) isdiscussed in Appendix D of the extended version of this pa-per (Singla, Tschiatschek, and Krause 2016).
Experimental EvaluationWe now report on the results of our synthetic experiments.
Experimental SetupUtility function f and set of items V . In the synthetic ex-periments we assume that there is an underlying submodularutility function f which we aim to maximize. The explo-ration module TOPX performs value or preference queries,and receives noisy responses based on model parametersand the marginal gains of the items for this function. For ourexperiments, we constructed a realistic probabilistic cover-age utility function following the ideas from El-Arini et al.(2009) over a ground set of N = 60 items. Details of thisconstruction are not important for the results stated belowand can be found in Appendix E of the extended version ofthis paper (Singla, Tschiatschek, and Krause 2016).
Benchmarks and Metrics. We compare several variantsof our algorithm, referred to by EXPGREEDYθ, where θrefers to the parameters used to invoke TOPX. In particu-lar, θ = O means k′ = 1 and ε′ = ε/k (i.e., competingwith (1 − 1
e ) of f(Sopt)); θ = G means k′ = 1 but ε′ = 0
0
200
400
600
800
1000
1200
1400
0 0.5 1 1.5 2 2.5
Totalnum
bero
fque
ries
Varianceσ2inthefeedbackvalues
GREEDY
x103
EXPGREEDY[#queries=360]
EXPGREEDYO
EXPGREEDYG
UNIFORM
(a) Sample complexity with varying noise
0
50
100
150
200
250
300
350
400
0 5 10 15 20 25 30 35 40 45 50 55
#Que
riesp
eritem
Itemindices(orderedby#queries)
EXPGREEDY
EXPGREEDYO
EXPGREEDYG
UNIFORM
(b) Distribution of queries across items
0
0.4
0.8
1.2
1.6
2
2.4
2.8
1 2 3 4 5 6
Marginalgainsinu4lity
Itera4on/Sizeofselectedset|S|
GREEDY
topl=2topl=4
topl=3
topl=3
topl=1topl=1
[totalu4lity=10.66]
[totalu4lity=10.70]
RANDOM[totalu4lity=3.62]
EXPGREEDY3
EXPGREEDYO
EXPGREEDYG
UNIFORM
(c) Execution instance showing topl size
3
5
7
9
11
0 1 3 5 7 10
15
20
25
30
35
40
Totalu0lity
Average#queriesperiteminoneitera0on
σ=0σ=2.5σ=10σ=10()σ=∞
2
2
2
2
2UNIFORM
(d) Varying σ2 for value queries
3
5
7
9
11
0 1 3 5 7 10
15
20
25
30
35
40
Totalu0lity
Average#queriesperiteminoneitera0on
β=∞,τ=Nβ=2,τ=1β=0.5,τ=1β=0.5,τ=1()β=0
UNIFORM
(e) Varying β for preference queries
3
5
7
9
11
0 1 3 5 7 10
15
20
25
30
35
40
Totalu0lity
Average#queriesperiteminoneitera0on
β=∞,τ=Nβ=0.5,τ=10β=0.5,τ=3β=0.5,τ=1β=0.5,τ=1()UNIFORM
(f) Varying τ for preference queries
Figure 3: Experimental results using synthetic function f and simulated query responses: (a)–(d) results are for value queries and (e)–(f)results are for preference queries. (a) EXPGREEDY dramatically reduces the sample complexity compared to UNIFORM and the other adaptivebaselines. (b) EXPGREEDY adaptively allocates queries to identify largest gap ∆l. (c) An execution instance showing the marginal gains andsizes of topl solutions returned by TOPX. (d)–(f) EXPGREEDY is robust to noise and outperforms UNIFORM baseline.
(i.e., competing with f(Sgreedy)); omitting θ means k′ = kand ε = ε/k. As benchmarks, we compare the performancewith the deterministic greedy algorithm GREEDY (with ac-cess to noise-free evaluations), as well as with random se-lection RANDOM. As a natural and competitive baseline, wecompare our algorithms against UNIFORM (Kempe, Klein-berg, and Tardos 2003; Krause and Guestrin 2005) — re-placing our TOPX module by a naive exploration modulethat uniformly samples all the items for best item identi-fication. For all the experiments, we used PAC parameters(ε = 0.1, δ = 0.05) and a cardinality constraint of k = 6.
ResultsSample Complexity. In Figure 3(a), we consider valuequeries, and compare the number of queries performed bydifferent algorithms under varying noise-levels until con-vergence to the solution with the desired guarantees. Forvariance σ2, we generated the query responses by samplinguniformly from the interval [µ − σ2, µ + σ2], where µ isthe expected value of that query. For reference, the querycost of GREEDY with access to the unknown function f ismarked (which equals N · k). The sample complexity dif-fers by orders of magnitude, i.e., the number of queries per-formed by EXPGREEDY grows much slower than that ofEXPGREEDYG and EXPGREEDYO. The sample complexityof UNIFORM is worse by further orders of magnitude com-pared to any of the variants of our algorithm.
Varying σ2 for value queries and (β, τ ) for preferencequeries. Next, we investigate the quality of obtainedsolutions for a limited budget on the total number of queriesthat can be performed. Although convergence may be slow,one may still get good solutions early in many cases. In Fig-ures 3(d)–3(f) we vary the available average budget per itemper iteration on the x-axis — the total budget available in
terms of queries that can be performed is equivalent to N · ktimes the average budget shown on x-axis. In particular,we compare the quality of the solutions obtained by EXP-GREEDY for different σ2 in Figure 3(d) and for different(β, τ) in Figure 3(e)–3(f), averaged over 50 runs. The pa-rameter β controls the noise-level in responses to the prefer-ence queries, whereas τ is the algorithm’s parameter indicat-ing how many pairwise comparisons are done in one queryto reduce variance. For reference, we show the extreme caseof σ2 =∞ and β = 0 which is equivalent to RANDOM; andthe case of σ2 = 0 and (β = ∞, τ = N) which is equiv-alent to GREEDY. In general, for higher σ in value queries,and for lower β or smaller τ in preference queries, morebudget must be spent to achieve solutions with high utility.
Comparison with UNIFORM exploration. In Fig-ure 3(d) and Figure 3(e)–3(f), we also report the quality ofsolutions obtained by UNIFORM for the case of σ2 = 10and (β = 0.5, τ = 1) respectively. As we can see in theresults, in order to achieve a desired value of total utility,UNIFORM may require up to 3 times more budget in compar-ison to that required by EXPGREEDY. Figure 3(b) compareshow different algorithms allocate budget across items (i.e.,the distribution of queries), in one of the iterations. We ob-serve that EXPGREEDY does more exploration across differ-ent items compared to EXPGREEDYG and EXPGREEDYO.However, the exploration is heavily skewed in comparison toUNIFORM because of the adaptive sampling by TOPX. Fig-ure 3(c) shows the marginal gain in utility of the consideredalgorithms at different iterations for a particular executioninstance (no averaging over multiple executions of the algo-rithms was performed). For EXPGREEDY, the size of toplsolutions returned by TOPX in every iteration is indicatedin the Figure, demonstrating that TOPX adaptively allocatesqueries to efficiently identify the largest gap ∆l.
Image Collection SummarizationWe now present results on a crowdsourced image collectionsummarization application, performed on Amazon’s Me-chanical Turk platform. As our image set V , we retrieved60 images in some way related to the city of Venice fromFlickr, cf., Figure 1(a). A total of over 100 distinct workersparticipated per summarization task.
The workers were queried for pairwise preferences as fol-lows. They were told that our goal is to summarize imagesfrom Venice, motivated by the application of selecting asmall set of pictures to send to friends after returning froma trip to Venice. The detailed instructions can be found inAppendix F of the extended version of this paper (Singla,Tschiatschek, and Krause 2016). We ran three instances ofthe algorithm for three distinct summarization tasks for thethemes (i) Venice, (ii) Venice Carnival, and (iii) VeniceCathedrals. The workers were told the particular theme forsummarization. Additionally, the set of images already se-lected and two proposal images a and b were shown. Thenthey were asked which of the two images would improve thesummary more if added to the already selected images.
For running EXPGREEDY, we used an average budgetof 25 queries per item per iteration and τ = 3, cf., Fig-ure 3(f). Note that we did not make use of the functionf constructed for the synthetic experiments at all, i.e., thefunction we maximized was not known to us. The results ofthis experiment are shown in Figure 1(b), demonstrating thatour methodology works in real-word settings, produces highquality summaries and captures the semantics of the task.
ConclusionsWe considered the problem of cardinality constrained sub-modular function maximization under noise, i.e., the func-tion to be optimized can only be evaluated via noisy queries.We proposed algorithms based on novel adaptive samplingstrategies to achieve high quality solutions with low sam-ple complexity. Our theoretical analysis and experimen-tal evaluation provide insights into the trade-offs betweensolution quality and sample complexity. Furthermore, wedemonstrated the practical applicability of our approach on acrowdsourced image collection summarization application.Acknowledgments. We would like to thank Besmira Nushi forhelpful discussions. This research is supported in part by SNSFgrant 200021 137971 and the Nano-Tera.ch program as part of theOpensense II project.
ReferencesBalcan, M., and Harvey, N. J. A. 2011. Learning submodularfunctions. In STOC.Bilmes, J. 2015. Submodularity in machine learning applica-tions. AAAI, Tutorial.Bradley, R. A., and Terry, M. E. 1952. Rank analysis of in-complete block designs: I. The method of paired comparisons.Biometrika 39(3/4):324–345.Bubeck, S.; Munos, R.; and Stoltz, G. 2009. Pure explorationin multi-armed bandits problems. In ALT, 23–37.Buchbinder, N.; Feldman, M.; Naor, J.; and Schwartz, R. 2014.Submodular maximization with cardinality constraints. InSODA, 1433–1452.
Busa-Fekete, R., and Hullermeier, E. 2014. A survey ofpreference-based online learning with bandit algorithms. In Al-gorithmic Learning Theory, 18–39. Springer.Chen, S.; Lin, T.; King, I.; Lyu, M. R.; and Chen, W. 2014.Combinatorial pure exploration of multi-armed bandits. InNIPS, 379–387.El-Arini, K.; Veda, G.; Shahaf, D.; and Guestrin, C. 2009. Turn-ing down the noise in the blogosphere. In KDD.Even-Dar, E.; Mannor, S.; and Mansour, Y. 2006. Action elim-ination and stopping conditions for the multi-armed bandit andreinforcement learning problems. JMLR 7:1079–1105.Feige, U. 1998. A threshold of ln n for approximating set cover.Journal of the ACM 45:314–318.Hassidim, A., and Singer, Y. 2015. Submodular optimizationunder noise. Working paper.Jamieson, K. G.; Katariya, S.; Deshpande, A.; and Nowak,R. D. 2015. Sparse dueling bandits. In AISTATS.
Kempe, D.; Kleinberg, J.; and Tardos, E. 2003. Maximizing thespread of influence through a social network. In KDD.Krause, A., and Golovin, D. 2012. Submodular function maxi-mization. Tractability: Practical Approaches to Hard Problems3:19.Krause, A., and Guestrin, C. 2005. Near-optimal nonmyopicvalue of information in graphical models. In UAI.Krause, A., and Guestrin, C. 2008. Beyond convexity: Sub-modularity in machine learning. ICML, Tutorial.Krause, A., and Guestrin, C. 2011. Submodularity and its ap-plications in optimized information gathering. TIST 2(4):32.Krause, A.; Guestrin, C.; Gupta, A.; and Kleinberg, J.2006. Near-optimal sensor placements: Maximizing informa-tion while minimizing communication cost. In IPSN, 2–10.Luce, R. D. 1959. Individual Choice Behavior: A theoreticalanalysis. Wiley.Mirzasoleiman, B.; Karbasi, A.; Sarkar, R.; and Krause, A.2013. Distributed submodular maximization: Identifying rep-resentative elements in massive data. In NIPS.Nemhauser, G.; Wolsey, L.; and Fisher, M. 1978. An analysis ofthe approximations for maximizing submodular set functions.Math. Prog. 14:265–294.Singh, A.; Krause, A.; and Kaiser, W. J. 2009. Nonmyopicadaptive informative path planning for multiple robots. In IJ-CAI, 1843–1850.Singla, A.; Tschiatschek, S.; and Krause, A. 2016. Noisy sub-modular maximization via adaptive sampling with applicationsto crowdsourced image collection summarization (extendedversion). http://arxiv.org/abs/1511.07211.Streeter, M., and Golovin, D. 2008. An online algorithm formaximizing submodular functions. In NIPS, 1577–1584.Tschiatschek, S.; Iyer, R.; Wei, H.; and Bilmes, J. 2014. Learn-ing mixtures of submodular functions for image collection sum-marization. In NIPS.Yue, Y., and Guestrin, C. 2011. Linear submodular bandits andtheir application to diversified retrieval. In NIPS.Zhou, Y.; Chen, X.; and Li, J. 2014. Optimal PAC multiple armidentification with applications to crowdsourcing. In ICML.
