Published as a conference paper at WACV 2015

How to Transfer? Zero-Shot Object Recognition viaHierarchical Transfer of Semantic Attributes

Ziad Al-Halah Rainer StiefelhagenInstitute for Anthropomatics and Robotics

Karlsruhe Institute of Technology{ziad.al-halah, rainer.stiefelhagen}@kit.edu

Abstract

Attribute based knowledge transfer has proven very suc-cessful in visual object analysis and learning previously un-seen classes. However, the common approach learns andtransfers attributes without taking into consideration theembedded structure between the categories in the sourceset. Such information provides important cues on the intra-attribute variations. We propose to capture these variationsin a hierarchical model that expands the knowledge sourcewith additional abstraction levels of attributes. We alsoprovide a novel transfer approach that can choose the ap-propriate attributes to be shared with an unseen class. Weevaluate our approach on three public datasets: aPascal,Animals with Attributes and CUB-200-2011 Birds. The ex-periments demonstrate the effectiveness of our model withsignificant improvement over state-of-the-art.

1. IntroductionSemantic attributes describe the object’s shape, texture

and parts. They have the unique property of being both ma-chine detectable and human understandable. By changingthe recognition task from labeling to describing, attributesrepresent an adequate knowledge that can be easily trans-ferred and shared with new visual concepts. Thus, they canbe used to recognize unseen classes with no training sam-ples (i.e. zero-shot classification).

In the prevailing approach, attributes are learned fromall seen classes and then reused to describe or classify anunseen one. However, this doesn’t account for the highintra-attribute variance. Using all the seen classes helps inlearning visual semantics in a very abstract manner. Hence,subsets of classes that share similar attributes cannot be dis-tinguished easily. Eventually, the fine properties of the at-tribute that help in discriminating a group from another arelost when it is learned from all the classes. Consider forexample the attribute beak in Figure 1. The global attributemodel would learn that a beak is an elongated extension ata certain position relative to the head; i.e. ignoring the dis-

Red-Headed- Pileated- Red-Belied-

Wo

od

pecker

Green- Florida- Blue-

Rufous- Ruby-Throated- Anna-

Laysan- Black-Footed- Sooty-

California-Gull

Beak?

Jay H

um

min

gbird

A

lbatro

ss

Figure 1: The high intra-attribute variance is better repre-sented at different semantic levels of abstraction. This helpsin directing the transfer process to identify the most suitablesource of knowledge to share with a novel class.

tinctive long thin beak shape of the hummingbird speciesor the wide curved-end of the albatross species. In otherwords, the global model does not take advantage of the richinformation already available in the source dataset. Thisresults in transferring less discriminative attributes to thenovel class. On the other hand, capturing these specificproperties of beak relative to each subgroup of birds is ben-eficial. It gives us the option to select the most proper typeof beak to share with the unseen class. Accordingly, know-ing that both Gull and Albatross are Seabirds, it is intuitiveand probably more discriminative to describe the beak ofthe California-Gull as an albatross-like-beak.

In this work we study the benefit of learning attributesat different levels of abstraction, from the most specific thatdistinguish one class from another to the most general thatare learned over all categories. We propose a novel hierar-chical transfer model that can find the best type of attributesto be shared with an unseen class. We evaluate the proposedmodel on three challenging datasets each with a differentgranularity of object categories. The evaluation shows thatsignificant gain can be achieved with our guided transfermodel with improvements from 26% and up to 35% overstate-of-the-art in zero-shot classification.

arX

iv:1

604.

0032

6v1

[cs

.CV

] 1

Apr

201

6

2. Related workOur work relates to two fields in computer vision litera-

ture; the attribute-based recognition and hierarchical trans-fer learning.

Since the introduction of semantic attributes [10, 9, 15],they have gained increasing attention from the computer vi-sion community. They represent an intermediate layer ofsemantics and facilitate various tasks in visual recognitionlike zero-shot learning [9, 8, 15], aiding object classificationand localization [26], relative attribute comparison [19] anddetection of unfamiliar classes [24]. However, in the pre-vailing direction the attributes are learned in a global man-ner from all classes available in the source [15, 19, 9, 17].Such methods cannot cope with the high variations in eachof the attributes. Some approaches jointly model objects, at-tributes and their correlations [26, 17] to handle such varia-tions. Nonetheless, these correlations cannot be learned forunseen classes since there is no training data.

Our approach is related to the work on learning class-specific attributes. In [9] a set of attributes are learned perclass as an intermediate step for feature selection in orderto reduce attribute correlations. Yu et al. [27] propose tolearn data-driven attributes at the category-level to betterdiscriminate the classes. However, data-driven attributesusually carry no semantic meaning; thus, their approachrequires user interaction when performing zero-shot learn-ing. In [29] the concepts in ImageNet are augmented witha set of semantic and data-driven attributes. These are usedalong with the hierarchy to learn a better similarity met-ric for content-based image retrieval. Correspondingly, wepropose to explicitly model the intra-attribute variations atdifferent abstraction levels. That is, rather than just usingclass-specific attributes, we expand the notion of the at-tribute from the most abstract to the most specific drivenby the embedded relations between the categories.

Additionally, hierarchies represent an attractive structurefor knowledge transfer and they have been exploited in var-ious ways: parameter transfer [22, 21], representation trans-fer [2] and bounding box annotations propagation [12]. Ofparticular relevance to our work is the joint modeling of hi-erarchy and attributes. In [1], a ranking classifier is trainedusing attributes for label embedding; showing that using hi-erarchical labels along global attributes as side informationimproved the zero-shot performance. In the recent work of[5], a hierarchy and exclusion graph is learned over the var-ious object categories. The graph models binary relationsamong the classes like mutual exclusion and overlap. Theyalso model similar relations between objects and global at-tributes and use it for zero-shot classification.

We exploit the hierarchical structure of the categories intwo aspects. We leverage the hierarchy to automaticallypropagate annotations and learn attributes at different lev-els of abstraction. Then we use it in guiding the transfer

process to select the most promising knowledge source ofattributes to share with novel classes. To the best of ourknowledge, this has not been done before.

3. ApproachHierarchical representation of concepts and objects is

part of the human understanding of the surrounding world.We usually try to combine objects into certain groups basedon a common criteria like functionality or visual similar-ity. This helps us to better learn the commonality as wellas the differences in and across groups. The key idea of ourapproach is to take advantage of the embedded structure inthe object category space and extend the notion of global at-tributes to include different levels of abstraction. The objecthierarchy groups the classes based on their overall visualsimilarity; thus provides a natural way to guide the transferprocess to share information from the knowledge sourcesthat will most likely contain relative information.

In the following, we describe the three main steps of ourmethod. Starting with a set of classes, global attributes anda hierarchy in the category space: (1) we automatically pop-ulate the hierarchy with additional attributes; (2) learn theseattributes to capture subtle differences between similar cat-egories; and finally, (3) use a hierarchy-guided transfer toselect the proper attributes to share with a novel class. Westart by defining the notation used throughout the paper.

Notation. Let C = {ck}Kk=1 be a set of categories, wherea subset of these categoriesQ = {qt}Tt=1 have training sam-ples while the rest Z = {zl}Ll=1 have none, and C = Q∪Z .A set of semantic attributes A = {am}Mm=1 describe allclasses in C. A directed acyclic graph H = (N , E) definesa hierarchy over the classes, with nodes N = {ni}Ii=1 andedges E = {eij : ni, nj ∈ N}. An attribute am at node ni

is referred to as anim .

3.1. Populating the hierarchy with attributes

To get the attribute description of the inner nodes weexploit the hierarchy H by transferring the attributes an-notation in a bottom-up approach from the seen classes Q(leaf nodes) to the root. For example, the inner node Dog(Figure 2a) has the attributes patches leveraged from thechild nodes Collie and Dalmatian, which in turn propagatespatches along other attributes up to node Carnivore. Then,the active attributes of node nj are defined as

anjm = 1 if ∃ani

m = 1 and ni ∈ child(nj), (1)

where child(n) is the set of nodes of the subtree rooted withn. Consequently, the root node of H will be described withall attributes of Q. It’s important to note that the attributelabel propagation is used to find the active attributes at acertain node and to guide the transfer process afterwards.This does not change the underlying ground truth attribute

Polar-Bear

Grizzly-Bear

German-Shepherd

Collie

Chihuahua

Fox

Wolf

Bobcat

Siamese-Cat

Unseen Class

Persian-Cat

Tiger Lion

Carnivore

Bear

Canine Feline

Cat

Big-Cat

Dog

Attribute Propagation Attribute Transfer

patches, small … furry

Dalmatian

paws-Bear

(a) Propagating (b) Learning (c) Transfer

patches, small …

Figure 2: Illustrative figure of our hierarchical transfer model. We leverage the hierarchy in many aspects: (a) attributelabel propagation, (b) modeling the intra-attribute variations and (c) guiding the transfer process to select and share the mostrelevant knowledge source for a novel class. (Best viewed in color)

labels of samples whether it is class-based or image-basedannotation.

3.2. Learning at different levels of abstraction

To learn the various attributes classifiers, we first definethe support set of an attribute am, i.e. the set of samples thatprovide evidence of am. An attribute a

njm in the hierarchy

has the support set supp(anjm ). The set contains samples

labeled with the attribute of that class (lbl(anjm ) : nj ∈ Q),

and additionally the samples of its children which share thesame attribute with nj , i.e.

supp(anjm ) =

⋃ni∈child(nj)

supp(anim ) ∪ lbl(anj

m ). (2)

To capture the fine differences that characterize an at-tribute at node n, we use a child-vs-parent learning scheme[18]. The attribute anc

m is learned with the following positive(TP ) and negative (TN ) sets

TP = supp(ancm ) TN = supp(anp

m )− supp(ancm ), (3)

where np is the parent node of nc. For example, the attributepaws of node Bear (Figure 2b) is supported by paws sam-ples from the classes Polar-Bear and Grizzly-Bear. Then,aBearpaws is learned to capture the differences against the other

paws samples from parent Carnivore. The root nr ofH hasno parent; hence {anr

m } are learned in the standard 1-vs-allscheme. In other words, the root attributes naturally map tothe commonly used global attributes in the literature.

3.3. Hierarchical transfer

Having the attributes learned at different levels of ab-straction, we then leverage the hierarchy to guide the knowl-edge transfer process and find the proper attributes to trans-fer to novel classes.

Analyzing the recall and precision properties of the at-tribute classifiers inH, we generally notice that the attributepredictions towards the leaf level ofH have higher precisionand lower recall than the ones towards the root of the hier-archy, which are characterized with low precision and highrecall. This is expected from a learning scheme as the onewe use. While the lower levels capture the discriminativesmall differences that distinguish a small group of classesagainst another regarding an attribute (high precision), inthe higher levels of H, the common visual properties of theattribute across a larger set of categories is learned (highrecall). Similar to [30], we find that combining classifierswith such an opposite recall and precision primacy results inan improved performance when compared to the constituentclassifiers. Furthermore, the combination of classifiers fromdifferent levels in the hierarchy increases the robustness ofthe final classifier against noise that might be produced bythe constituent classifiers.

Accordingly, for a novel class zl in H, we transfer theattributes of its ancestors across the different levels of ab-straction (e.g. aPersianCat

furry in Figure 2c), such that:

szl(azlm|x) =

∑ni∈anc(zl)

[[azlm = anim ]] sni

(anim |x)∑

ni∈anc(zl)

[[azlm = anim ]]

, (4)

where [[·]] is the Iverson bracket (i.e. [[P ]] = 1 if conditionP is true and 0 otherwise), sn(anm|x) is the score of the at-tribute am for node n given sample x, and anc(n) is the setof ancestor nodes of n. Once the attributes are transferred tozl, the final prediction score s(zl|x) of the zl category canbe defined by averaging over the attributes of that class as:

s(zl|x) =

M∑m=1

[[azlm = 1]] s(azlm|x)

M∑m=1

[[azlm = 1]]

. (5)

In the following, we refer to our Hierarchical AttributeTransfer model as HAT.

4. EvaluationWe evaluate our model on three datasets. Each provides

different characteristics regarding the granularity of classes.This give us the chance to see how the performance of theproposed HAT model varies with regards to the complexityof the embedded knowledge in the source set. Next, wepresent the three datasets, the hierarchy and the features weextract from images to train the different attribute models.

Datasets. (1) The aPascal/aYahoo (aPaY) [9] containstwo subsets. The first (aP) uses 12,695 images and 20 cat-egories from PASCAL VOC 2008 [6]. The second (aY)has 12 disjoint classes and 2,644 images collected from theYahoo image search engine. Per-image labels of 64 binaryattributes are provided for both subsets. In the predefinedzero-shot split, aP is used for training and aY for testing.

(2) The Animals with Attributes (AwA) [15] consists of30,475 images of 50 classes of animals. They are describedwith 85 semantic attributes on the class level. The authorsprovide a fixed split for zero-shot experiments. They select40 classes for training and 10 for testing.

(3) The CUB-200-2011 Birds (CUB) [25] has 200 birdclasses and 11,788 images. Each image is labeled with 312attributes. Unlike the previous two, there is no predefinedsplit for this dataset. For our experiments, we randomlyselect 150 classes for training and 50 for testing.

Hierarchy. We learn the object hierarchies using theWordNet ontology. By querying the ontology with the cat-egory labels we extract a tree that brings the classes into ahierarchical ordering. Subsequently, we prune the hierarchyto remove intermediate nodes that have a single child.

Deep Features.1 Motivated by the impressive successof deep Convolutional Neural Networks (CNN), we en-code the images using a CNN-based deep representationto train the attribute classifiers. We use the CNN modelCNN-M2K provided by [4] which has a structure similar toAlexNet [14]. We also use the BVLC implementation [13]

1The deep features used in this work are available on our website:https://cvhci.anthropomatik.kit.edu/˜zalhalah/

Features aPaP aPaY AwA CUB

Shallow 84.12 70.91 71.16 60.78CNN-M2K 92.82 80.73 78.64 76.03GoogLeNet 93.63 80.03 79.47 76.59

Table 1: Attribute prediction performance (mean AUC) us-ing deep and shallow representations.

of GoogLeNet [23] which has a much deeper architecture.We follow the best practice found in [4] to extract the deeprepresentation. The image is resized to 256 x 256. Then5 image crops obtained from center and corners with theirflipped versions are fed to the CNN. The output of the lasthidden layer is extracted, sum-pooled and L2-normalizedto be used as our deep-features to train the different mod-els. For all our attribute classifiers, we use linear SVMs [7]with regularized logistic regression. The cost parameter Cis estimated using 5-folds cross validation.

4.1. Attribute Prediction with Deep Features

We first evaluate the performance of the deep featuresthat we use in learning attribute classifiers. We comparetheir performance with attributes learned using “shallow”features. We use the precomputed features provided by [9]and [15] as the “shallow” representation for aPaY and AwArespectively. For CUB, we encode the images with Fishervectors based on SIFT and Color descriptors and use thatas the shallow representation. In aPaY, we consider twosetups: (i) within-category attribute prediction (aPaP), i.e.the evaluation is done on the aPascal testing set; (ii) across-category prediction (aPaY), i.e. we evaluate on the disjointaYahoo set. In both cases, the attributes are learned usingthe aPascal training set. For AwA and CUB, we use thezero-shot testing setups defined in the previous section.

In Table 1, we show the performance of the three repre-sentations in terms of mean AUC under ROC of the attributepredictions. The deep-feature models constantly outper-form thier counterpart across all datasets with an increasebetween 7% to 15%. This high performance of the deepfeatures in attribute prediction is expected. CNNs automat-ically learn to capture features with varying complexity ateach layer. While the lower layers learn features like edgesand color patches, the higher layers learn much complexstructures of the object like parts [28]. Many of these corre-spond directly to semantic attributes which explains the im-pressive performance of the deep representation using oursimple linear classifiers.

4.2. Zero-Shot Classification

To populate the hierarchy with attributes (Section 3.1),our model requires class-based attribute descriptions.Hence, for aPaY and CUB we average all image-based at-

Model Features aPaY AwA CUB

DAP [16] Shallow 19.1 41.4 -IAP [16] Shallow 16.9 42.2 -AHLE [1] Shallow - 43.5 [17.0] 2

HEX [5] DECAF - 44.2 -TMV-BLP [11] Shallow - 47.1 -HAT (ours) DECAF - 48.9 -

DAP CNN-M2k 31.9 54.0 33.7ENS CNN-M2k 43.1 57.7 37.3HAT (ours) [3] CNN-M2k 46.3 68.8 48.6

DAP GoogLeNet 35.5 59.9 36.7ENS GoogLeNet 42.8 63.5 39.4HAT (ours) GoogLeNet 45.4 74.9 51.8

Table 2: Zero-shot multi-class accuracy on the threedatasets.

Model Features aPaY AwA CUB

DAP CNN-M2k 87.3 88.5 82.2ENS CNN-M2k 85.6 88.5 89.5HAT (ours) CNN-M2k 87.1 92.0 94.9

DAP GoogLeNet 88.0 89.8 83.9ENS GoogLeNet 84.7 90.1 90.7HAT (ours) GoogLeNet 86.7 94.1 95.5

Table 3: Zero-shot mean AUC under ROC curve for the testclasses.

tribute vectors of each class to calculate the class-attributeoccurrence matrices. Then, the binary class-attribute nota-tions are created by thresholding the resulting occurrencematrix at its overall mean value. Along with our Hierarchi-cal Attribute Transfer model (HAT), we also train two com-mon baselines for global attributes using our deep features:(i) The Direct Attribute Prediction model (DAP), where theclass prediction is formulated as a MAP estimation [15]; (ii)The Ensemble model (ENS), that combines the predictionsof the attributes using a sum formulation similar to the onewe use in Eq. 5 but based on global attributes [20]. For ENSand HAT, we normalize the prediction scores of the classesto have zero mean and unit standard deviation.

HAT vs. state-of-the-art. In Table 2, we report the nor-malized multi-class accuracy on the three test sets. Ourmodel outperforms the state-of-the-art on the three datasetswith a wide margin. In [1], a model that uses object hier-archy as side information is used to learn attributes basedon Fisher vectors. In the recent work of [5], a hierarchicalmodel of objects is learned using deep features similar tothe one we use. Finally, [11] a transductive multi-view em-bedding is learned using global attributes, word space andlow-level features. Nonetheless, our model improves over

25 50 75 100 125 150 1750

10

20

30

40

50

60

70

DAP

HAT

# Classes in Source

Acc

ura

cy

(a)25 50 75 100 125 150 175

50

55

60

65

70

75

80

85

90

95

100

DAP

HAT

# Classes in Source

mA

UC

(b)

Figure 3: The performance of DAP and HAT in CUB withvarying number of classes in the source as demonstratedwith (a) multi-class accuracy and (b) mean AUC. (Features:GoogLeNet)

the best state-of-the-art results by 26% (aPaY), 28% (AwA)and 35% (CUB).

HAT vs. deep-feature baseline. Compared to ourstrong baseline with deep features (DAP and ENS), HATstill performs the best in terms of both multi-class accu-racy and mean AUC (Table 3). Figure 4 shows the high-est ranking results obtained by the three models for eachtest class in the AwA dataset. While distinctive classes likeChimpanzee and Humpback-Whale are correctly classifiedby all models, both DAP and ENS confuse visually simi-lar classes that share many global attributes like Leopard& Persian-Cat and Rat & Raccoon, and more samples arewrongly ranked high by these models. To the contrary, HATlearns the fine differences among the shared attributes ofthese classes which helps in discriminating them efficiently.For example, HAT learns the differences between the at-tributes of Big-Cat and Cat (Figure 2c) which facilitates theseparation among the novel classes Leopard and Persian-Cat (Figure 4). Furthermore, we find that normalizing theprediction scores of the novel classes (Eq. 5) to have a zeromean and unit standard deviation makes the scores morecomparable. This improves the accuracy of both the base-line (ENS) and our model (HAT) with the latter surpassingthe former. However, this requires that the test data is avail-able as a batch at the test time.

The relative improvement in accuracy of our HAT modelover the baseline is 6% (aPaY), 18% (AwA) and 31%(CUB). This trend nicely follows the level of granularityof the objects in the data sets. Our model takes advantageof the underlying structure and the commonality among theclasses and it is able to distinguish between fine grainedclasses more efficiently. On the other hand, it is harder forthe baseline models (DAP & ENS) to distinguish such finegrained objects using the abstract global attributes.

In the rest of the experiments, we report the results whenusing the GoogLeNet features for our model and the base-lines.

Per-image vs. per-class attributes. aPaY and CUB pro-

2Although we followed a similar split on CUB as in [1], the actualclasses may differ and the results may not be directly comparable.

vide attribute annotation at the image level which we usedto train the models in the previous experiments. We evalu-ate on these two datasets using class-based attributes simi-lar to those in AwA. We notice that the accuracy of both thebaseline and HAT decreases in these settings. Where ENShas 38.3% and 34.6%, the HAT model achieves 40.3% and48.2% on aPaY and CUB. This seems to differ from thefindings in [16]. The reason could be related to the typeof features used. In [16] a set of shallow features are usedwhich require a relatively larger number of samples to traingood attribute classifiers. This in turn results in noisy at-tribute predictions if there are few image-based annotationsof the attribute. In comparison, using the deep features wecan learn better attribute classifiers even if the training datais relatively sparse.

Unknown attributes of the novel class. Although thisevaluation setup is not possible with the global attributemodel, HAT enables us to carry out zero-shot recognitioneven if the attribute description of the novel class is un-known. To do that, we again leverage the hierarchy andtransfer the attribute description of the parent node to thenovel class. Using this setup, HAT achieves an accuracy of31.1% (aPaY), 59.7% (AwA) and 32.6% (CUB). This dropin performance is reasonable since we are transferring themore generic attributes of the parent. Hence, confusion canarise when multiple test classes share the same parent in thehierarchy. Nonetheless, HAT makes it possible to performattribute-based zero-shot classification when only the labelof the novel class is available.

Source set complexity. In the following experiment,we vary the complexity of the knowledge contained in thesource (the number of seen classes) compared to the target(the unseen classes). This helps to have a better understand-ing of the characteristics of the different models as the rich-ness of the embedded information in the source changes.We use the CUB dataset and start with a random set of 25classes to be in the source. We gradually increase the sourceset with additional 25 random classes. At each step, the restof the 200 classes is used as the target set to conduct zero-shot classification.

In Figure 3 we see that when the source is relatively poorand contains less structured knowledge, both DAP and HATperforms at the same level. However, as the source get big-ger and more complex HAT consistently outperforms DAPwith an increasingly wider margin. Unlike DAP that usesa single layer of global attributes, HAT is able to take ad-vantage of the complexity of information available in thesource. HAT captures the commonality among the cate-gories and exploits it to learn and transfer more discrimi-native attributes to distinguish the unseen categories.

5. ConclusionIn this paper, we present a simple yet very effective

model for zero-shot object recognition. Our model takesadvantage of the embedded structure in the category spaceto learn attributes at different levels of abstraction. Further-more, it exploits inter-class relations to provide a guidedknowledge transfer approach that can select and transfer theexpected relevant attributes to a novel class. The evalua-tion on three challenging datasets shows the superior per-formance of the proposed model over the state-of-the-art.

We are considering to extend our approach in differentdirections. In the current model, we assume similar impor-tance for the transferred attributes from the different layersin the hierarchy. However, the confidence in the attributespredictions and their relative relatedness to the novel classdiffer through the different abstraction levels. Using adap-tive weights when transferring attributes could improve themodel performance. Moreover, we considered only the an-cestor nodes as a knowledge source. The siblings of a novelclass represent another promising option. Including them inthe transfer process may help in sharing more discrimina-tive attributes.

References[1] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid. Label-

Embedding for Attribute-Based Classification. In CVPR,2013.

[2] Z. Al-Halah, L. Rybok, and R. Stiefelhagen. What to Trans-fer? High-Level Semantics in Transfer Metric Learning forAction Similarity. In ICPR, 2014.

[3] Z. Al-Halah and R. Stiefelhagen. How to Transfer? Zero-ShotObject Recognition via Hierarchical Transfer of Semantic At-tributes. In WACV, 2015.

[4] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Re-turn of the Devil in the Details: Delving Deep into Convolu-tional Nets. In BMVC, 2014.

[5] J. Deng, N. Ding, Y. Jia, A. Frome, K. Murphy, S. Bengio,Y. Li, H. Neven, and H. Adam. Large-Scale Object Classifi-cation using Label Relation Graphs. In ECCV, 2014.

[6] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, andA. Zisserman. The PASCAL Visual Object Classes Challenge2008 (VOC2008) Results, 2008.

[7] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. LIBLINEAR: A library for large linear classification.Journal of Machine Learning Research, 9:1871–1874, 2008.

[8] A. Farhadi, I. Endres, and D. Hoiem. Attribute-Centric Recog-nition for Cross-category Generalization. In CVPR, 2010.

[9] A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. DescribingObjects by their Attributes. In CVPR, 2009.

[10] V. Ferrari and A. Zisserman. Learning Visual Attributes. InNIPS, 2008.

[11] Y. Fu, T. M. Hospedales, T. Xiang, Z. Fu, and S. Gong. Trans-ductive Multi-view Embedding for Zero-Shot Recognition andAnnotation. In ECCV, 2014.

[12] M. Guillaumin and V. Ferrari. Large-scale Knowledge Trans-fer for Object Localization in ImageNet. In CVPR, 2012.

chimpanzee giant-panda

leopard persian-cat

pig hippo-potamus

humpback-whale

raccoon rat seal

(a) DAP

(b) ENS

(c) HAT (ours)

Figure 4: The highest scoring results of DAP, ENS and HAT (CNN-M2K) for each test class in AwA. (Best viewed in color)

[13] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Gir-shick, S. Guadarrama, and T. Darrell. Caffe: ConvolutionalArchitecture for Fast Feature Embedding. arXiv, 2014.

[14] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet Clas-sification with Deep Convolutional Neural Networks. In NIPS,2012.

[15] C. Lampert, H. Nickisch, and S. Harmeling. Learning to detectunseen object classes by between-class attribute transfer. InCVPR, 2009.

[16] C. Lampert, H. Nickisch, and S. Harmeling. Attribute-basedclassification for zero-shot visual object categorization. T-PAMI, 2013.

[17] J. Liu, B. Kuipers, and S. Savarese. Recognizing Human Ac-tions by Attributes. In CVPR, 2011.

[18] M. Marszalek and C. Schmid. Semantic Hierarchies for VisualObject Recognition. In CVPR, 2007.

[19] D. Parikh and K. Grauman. Relative Attributes. In ICCV,2011.

[20] M. Rohrbach, M. Stark, and B. Schiele. Evaluating Knowl-edge Transfer and Zero-Shot Learning in a Large-Scale Set-ting. In CVPR, 2011.

[21] R. Salakhutdinov, J. Tenenbaum, and A. Torralba. Learning toShare Visual Appearance for Multiclass Object Detection. InCVPR, 2010.

[22] B. Shahbaba and R. M. Neal. Improving Classification Whena Class Hierarchy is Available Using a Hierarchy-Based Prior.Bayesian Analysis, 2:221–238, 2007.

[23] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov,D. Erhan, V. Vanhoucke, and A. Rabinovich. Going Deeperwith Convolutions. arXiv, 2014.

[24] C. Wah and S. Belongie. Attribute-Based Detection of Unfa-miliar Classes with Humans in the Loop. In CVPR, 2013.

[25] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.The Caltech-UCSD Birds-200-2011 Dataset. Technical report,2011.

[26] Y. Wang and G. Mori. A Discriminative Latent Model of Ob-ject Classes and Attributes. In ECCV, 2010.

[27] F. X. Yu, L. Cao, R. S. Feris, J. R. Smith, and S.-f. Chang. De-signing Category-Level Attributes for Discriminative VisualRecognition . In CVPR, 2013.

[28] M. D. Zeiler and R. Fergus. Visualizing and UnderstandingConvolutional Networks. In ECCV, 2014.

[29] H. Zhang, Z.-j. Zha, Y. Yang, S. Yan, Y. Gao, and T.-S. Chua.Attribute-augmented Semantic Hierarchy. In ACM MM, 2013.

[30] A. Zweig and D. Weinshall. Exploiting Object Hierarchy:Combining Models from Different Category Levels. In ICCV,2007.