An Input-aware Factorization Machine for Sparse Prediction

Yantao Yu , Zhen Wang , Bo Yuan*Graduate School at Shenzhen, Tsinghua University

{yyt17, z-wang16}@mails.tsinghua.edu.cn, yuanb@sz.tsinghua.edu.cn

AbstractFactorization machines (FMs) are a class of gen-eral predictors working effectively with sparse data,which represent features using factorized parame-ters and weights. However, the accuracy of FMscan be adversely affected by the fixed representa-tion trained for each feature, as the same feature isusually not equally predictive and useful in differ-ent instances. In fact, the inaccurate representationof features may even introduce noise and degradethe overall performance. In this work, we improveFMs by explicitly considering the impact of eachindividual input upon the representation of features.We propose a novel model named Input-aware Fac-torization Machine (IFM), which learns a uniqueinput-aware factor for the same feature in differ-ent instances via a neural network. Comprehensiveexperiments on three real-world recommendationdatasets are used to demonstrate the effectivenessand mechanism of IFM. Empirical results indicatethat IFM is significantly better than the standardFM model and consistently outperforms four state-of-the-art deep learning based methods.

1 IntroductionPrediction now plays a crucial role in many personalizedsystems, such as online advertising [McMahan et al., 2013;Juan et al., 2016] and recommendation [Koren et al., 2009;Cheng et al., 2014]. Typically, the recommendation task isformulated as estimating a function that maps categorical pre-dictor variables (a.k.a. features) to a target. For example, weneed to predict the click probability (target) that a user (firstpredictor variables) of a particular occupation will click onan item (second predictor variables). The first and secondpredictor variables are usually combined in the form of aninstance, e.g., {young, female, student, pink, skirt}.

To build predictive models with these categorical predictorvariables, it is indispensable to accurately represent them inmachine identifiable forms. A common solution is to con-vert them to a set of binary features (a.k.a. feature vector)via one-hot encoding [Cheng et al., 2016]. Depending on the

*Corresponding author

number of possible values of categorical predictor variables,the generated feature vector can be very high dimensional andsparse. To build an effective model with such sparse data,factorization machines (FMs) were proposed [Rendle, 2010],which learn a one-dimensional weight and a k-dimensionalembedding vector as the representation of each feature fromsparse data. Owing to its efficient linear training time andhigh prediction accuracy, FMs have been successfully appliedto various applications, from recommendation systems [Ren-dle et al., 2011] to natural language processing [Petroni etal., 2015]. Despite great promise, FMs produce a single rep-resentation for each feature and the same representation of agiven feature is shared in different instances to compute itspredictive power, which may lead to inferior performance. Inthis paper, we argue that the impact of each individual inputshould be given full consideration when creating the repre-sentation for each feature.

Many existing studies attempt to improve the predictionaccuracy of FMs by focusing on feature interactions. For ex-ample, DeepFM [Guo et al., 2017] models high-order featureinteractions through a neural network, while AFM [Xiao etal., 2017] enhances FMs by learning the importance of eachfeature interaction from data via a neural attention network.Nevertheless, these improvements are limited as the unique-ness of each instance is not exploited.

In our work, we propose to improve FMs from a new per-spective that tries to refine the representation of features ac-cording to different instances. In real-world applications, afeature usually has dissimilar levels of predictive power indifferent situations. For example, the feature female is ap-parently crucial for click probability in an instance: {young,female, student, pink, skirt}. However, in another instance:{young, female, student, blue, notebook}, the feature femaleis relatively less crucial. As such, the same feature on differ-ent instances should be assigned different levels of predictivepower to better reflect its specific contribution.

In this paper, we present a novel model for predictiontasks under sparsity named Input-aware Factorization Ma-chine (IFM), which enhances FMs by explicitly consideringthe impact of each individual input on the representation offeatures. It refines the weight and embedding vector of eachfeature with regard to different instances and adds nonlinear-ity to the model simultaneously. Specifically, we adopt theidea of end to end memory network [Sukhbaatar et al., 2015]

to enable each feature to contribute dissimilarly in differentinstances. In this way, the representation of each feature is notonly related to itself, but also to the instances containing it.In contrast to other deep learning based methods that mainlyfocus on feature interactions, our use of neural networks formore informative representations of features greatly enhancesthe expressiveness and interpretability of FMs. Comprehen-sive experiments show that our IFM features two major ad-vantages: i). it produces better prediction results comparedto existing techniques; ii). it provides deeper insights into therole that each feature plays in the prediction task.

2 PreliminariesFactorization machines are proposed to learn feature inter-actions for sparse data, which combine the advantages ofSupport Vector Machines (SVMs) with factorization models.FMs enhance linear regression (LR) using the second-orderfactorized interactions between features. Given a real valuedfeature vector x ∈ Rn where n denotes the number of fea-tures and most of the elements xi in a vector x are zero, theFM model estimates the target by modelling the interactionsvia factorized interaction parameters:

yFM (x) = w0 +

n∑i=1

wixi +

n∑i=1

n∑j=i+1

⟨vi,vj⟩xixj (1)

where x is the feature vector of the instance and xi denotesthe i-th dimension of the feature vector while y represents thevalue predicted by the FM model (e.g., the estimated proba-bility of click). w0 is the global bias and wi models the weightof the i-th variable. vi is a k-dimensional embedding vectorof the i-th variable and ⟨vi,vj⟩ is the dot product of two vec-tors of size k, which models the interaction between the i-thand j-th variables. Here, k is the dimension of embeddingvectors, which controls the complexity of FMs. Note that thefeature interactions can be reformulated [Rendle, 2010] as:

n∑i=1

n∑j=i+1

⟨vi,vj⟩xixj =1

2

k∑f=1

n∑

j=1

vj,fxj

2

−n∑

j=1

v2j,fx2j

(2)

where vj,f denotes the f -th element in vj . The time complex-ity of Equation 1 is O(kn2), but with reformulating it dropsto linear time complexity O(kn).

It is worth noticing that FMs produce a fixed representationfor each feature: the same weight wi and embedding vectorvi are shared across all different instances that involve thei-th feature. However, it is not unusual that a certain fea-ture is not equally predictive and useful across different in-stances. Consequently, due to the lack of flexibility in featurerepresentation, FMs may suffer from the insufficient abilityfor modelling complex data, which may adversely affect theirperformance in general.

3 Input-aware Factorization MachineIn this section, we present the details of the proposed IFMmodel to show how to boost the performance of FMs via theinput-aware strategy.

…0 1 0 1 0 0 1 00

…2v

4v 7v…2w

4w 7w

Input-aware Factor Estimating

Representation Refining

Factorization Machines

y x

Sparse Input x

Embedding Layer

Factor Estimating Net

Reweighting Layer

FM Prediction Layer

Output

0

Figure 1: The network architecture of our proposedInput-aware Factorization Machines model

3.1 The IFM ModelSimilar to factorization machines, given a sparse feature vec-tor x ∈ Rn as input, where a feature value xi = 0 means thatthe i-th feature does not exist in the instance, IFM predictsthe target as:

yIFM (x) = w0 +

n∑i=1

wx,ixi +

n∑i=1

n∑j=i+1

⟨vx,i,vx,j⟩xixj

(3)where the first and second terms model the global bias of dataand weights of features respectively, and the third term cap-tures the feature interaction. Note that, in our method, the sec-ond and third terms model each feature’s effect on the labelin a nonlinear way, as the weight wx,i and embedding vectorvx,i of each feature not only correlate with the i-th feature butalso are related to the input vector x. Figure 1 illustrates thenetwork architecture of our proposed IFM model, where theextra components in addition to traditional FMs are markedby dark background.

Embedding LayerAs in FMs, a weight and an embedding vector are randomlyinitialized for each feature as its representation. Formally,let vi ∈ Rk be the embedding vector of the i-th feature,where k is the embedding size. Due to the sparsity of x,we only need to include the embedding vectors of non-zerofeatures, i.e., Vx = {vi} where xi = 0. Finally, we stackall vectors in Vx into a single k × h dimensional vector:Vx =

[vT1 ,v

T2 ....,v

Th

], where h denotes the number1 of non-

zero elements per instance.

Factor Estimating NetworkThe input-aware strategy is inspired by memory networks[Weston et al., 2014], which are a class of deep learning basedmodels mainly used in question answering (QA) tasks. Oneof their key ideas is to allow different memory records to con-

1In this paper, we assume that h is a fixed value for each dataset.

...

...

...

Fully Connected Layers

Input XV

Softmax

...

Input-aware Factor

2v 4v 7v

,2mx ,4m

x ,7mx

Figure 2: The neural network architecture of FactorEstimating Network

tribute differently to each question. Motivated by this inher-ent flexibility of memory networks, the input-aware strategyis proposed to extend the functionality of FMs by introduc-ing an input-aware factor mx,i into the feature weight wi andembedding vector vi, which allows each feature to contributedissimilarly in different inputs.

To estimate the input-aware factor mx,i, we first feed thestacked vector Vx into the Factor Estimating Network (FEN)to learn a unique vector Ux for the given input x. Figure 2shows the architecture of FEN. The network begins with astack of fully connected layers that are capable of learningunique information from x. Formally, the definition of thefully connected layers is as follows:

a1 = σ1 (W1Vx + b1) ,...

Ux = aL = σL (WLaL−1 + bL)(4)

where Wl, bl, σl and al are the weight matrix, bias vector,activation function and the output of the l-th layer, respec-tively. Next, Ux, which is the output vector of the last hiddenlayer aL, is used to learn an input-aware factor mx,i for eachfeature in the input x:

m′x = UxP,P ∈ Rt×h,

mx,i = h× exp(m′x,d)∑h

j=1 exp(m′x,j)

, xi = 0(5)

where P denotes the weight matrix that transforms Ux to a h-dimensional vector m′

x, and t denotes the number of neuronsin the last hidden layer. Finally, the elements m′

x,d, d ∈ [1, h]in m′

x are normalized through a softmax function, so that thesum of all elements is equal to h, representing the input-awarefactor mx,i of each non-zero feature in x.

Reweighting LayerOnce the outputs from FEN are obtained, they are used torefine the feature weight wi and embedding vector vi withregard to the current input. The inputs of this layer are thewi and vi of the given input x and the input-aware factormx,i from the previous layer. Formally, the definition of the

reweighting layer is as follows:

wx,i = mx,iwi

vx,i = mx,ivi(6)

where wx,i, vx,i are the refined representations of features forthe specific input x.

FM Prediction LayerIn this layer, the input-aware weight wx,i and embedding vec-tor vx,i are fed into factorization machines to predict the tar-get as Equation 3. It is worth mentioning that, similar toEquation 2, we can reformulate Equation 3 to reduce the run-time of this layer to linear time complexity O(kn):

n∑i=1

n∑j=i+1

⟨vx,i,vx,j⟩xixj =12

k∑f=1

( n∑j=1

vx,j,fxj

)2

−n∑

j=1

v2x,j,fx2j

(7)

In summary, in IFM, the model parameters are Θ ={w0, wi,vi,Wl,bl,P}. Compared to the FM model, the ad-ditional model parameters of IFM are {Wl,bl,P}, which aremainly used for capturing the input-aware factors.

3.2 Relationship with FM and AFMIt is easy to see that, in the IFM model, by fixing all mx,i to1, wx,i and vx,i only depend on the i-th feature, and IFM re-duces to the original FM model. On the other hand, if we ne-glect the second term of Equation 1, AFM [Xiao et al., 2017]can be seen as a special case of IFM.

To show this, we compare the formulation of feature inter-action in AFM with ours:

n∑i=1

n∑j=i+1

⟨vx,i,vx,j⟩xixj =

n∑i=1

n∑j=i+1

mx,imx,j ⟨vi,vj⟩xixj

=pTn∑

i=1

n∑j=i+1

ai,j ⟨vi ⊙ vj⟩xixj

(8)

As can be seen from Equation 8, by replacing mx,imx,j

with ai,j and fixing pT to a constant vector of [1, ..., 1], wecan exactly recover the AFM model.

3.3 LearningAs IFM directly enhances FMs from the perspective of datamodelling, it can also be applied to a variety of predictiontasks, including regression, classification and ranking. Differ-ent objective functions should be used to customize the IFMmodel for different tasks. For binary classifications, the lossfunction is log loss. For regression, a commonly adopted lossfunction is the squared loss:

Lreg =∑x∈χ

(yIFM (x)− y (x))2 (9)

where χ denotes the training data, and y (x), yIFM (x) de-note the label and target of the input x, respectively. In thiswork, we mainly focus on the regression task and optimizethe squared loss of Equation 3. The optimization for rankingand classification tasks can be done in the same way. To opti-mize the loss function, we employ adaptive gradient descent(AdaGrad), a universal solver for neural network models. The

key to implementing the AdaGrad algorithm is to obtain thederivative of the squared loss Lreg w.r.t. each parameter withan adaptive learning rate. In our model, not only can we trainall parameters directly through backpropagation, but also wecan pretrain the feature weight wi and embedding vector vi

through a FM.Overfitting is a perpetual issue in model training and we

employ dropout [Srivastava et al., 2014] and L2 regulariza-tion to control the overfitting of FEN. In doing so, the actualobjective function to be optimized is:

Lreg=

∑x∈χ

(yIFM (x)− y (x))2+λ∥Φ∥2 (10)

where λ controls the regularization strength and Φ denotesthe weight matrix {W,P} in FEN.

4 Related WorkAs extensions of FMs, GBFM [Cheng et al., 2014] selectsgood features using gradient boosting and models only theinteractions between good features. Field-aware FM (FFM)[Juan et al., 2016] enables each variable to have multiple em-bedding vectors for feature interactions, but its large numberof parameters may result in undesirable computational cost.In our work, we aim to boost the performance of FMs byexplicitly considering the impact of each individual input onthe representation of features, which refines the weight andembedding vector of each feature with regard to different in-stances.

Meanwhile, an increasing number of researchers are in-terested in employing deep learning for prediction. Specif-ically, some models use DNN to explore high-order featureinteractions: Wide&Deep [Cheng et al., 2016], employs aMultilayer Perceptron(MLP) on the concatenation of featureembedding vectors to learn feature interactions; DeepCross[Shan et al., 2016] applies a deep residual MLP to learn crossfeatures; PNN [Qu et al., 2016] introduces a product layerbetween the embedding layer and DNN layers to learn fea-ture interaction; DeepFM [Guo et al., 2017] uses the FMcomponent and the deep component to get 2-order and high-order interaction information; NFM [He and Chua, 2017] ex-tends FMs by using DNN to explore high-order nonlinear fea-ture interactions; xDeepFM [Lian et al., 2018] learns certainbounded-degree feature interactions explicitly and arbitrarylow/high-order feature interactions implicitly. Some othermodels extend FMs by discriminating the importance of fea-ture interactions. For example, AFM [Xiao et al., 2017] usesattention mechanism to weight different feature interactions,and FwFM [Pan et al., 2018] uses mutual information to dis-tinguish the strength of feature interactions.

In all the above models, each feature holds a fixed repre-sentation for different instances, instead of an adaptive rep-resentation according to each unique instance. Consequently,the performance of these methods may be limited, as the im-pact of each instance on the representation of features is ig-nored. By extending FMs with a more accurate and flexiblerepresentation of features, IFM is expected to produce supe-rior performance compared to existing methods.

5 ExperimentsIn this section, we conduct extensive experiments to answerthe following questions:

• (Q1) How do the key hyper-parameters of IFM (e.g., thedropout ratio and the number of hidden layers) impactits performance?

• (Q2) Can the factor estimating network effectively refinethe representation of features for different instances?

• (Q3) How does IFM perform compared to state-of-the-art methods for sparse prediction?

5.1 Experimental SettingsDatasets. For regression tasks, we evaluate various pre-diction models on two public datasets: Frappe2 [Baltrunaset al., 2015] and MovieLens [Harper and Konstan, 2016].The Frappe dataset has been used for context-aware mobileapp recommendation, which contains 96,202 records with957 users and 4,082 apps. The MovieLens dataset has beenused for personalized tag recommendation, which contains668,953 tag applications of 17,045 users on 23,743 items with49,657 distinct tags. We follow the data processing details ofNFM [He and Chua, 2017] and randomly split instances by8:1:1 for training, validation and test. For binary classifica-tion, we use the Avazu3 dataset, which was published in thecontest of Avazu Click-Through Rate Prediction in 2014. Thepublic dataset is randomly split into training and test sets by4:1. Meanwhile, we remove the features appearing less than20 times to reduce dimensionality.Evaluation Metrics. We use MAE (mean absolute error)and RMSE (root mean square error) for evaluating regres-sion tasks such as recommendation and use AUC (Area Un-der ROC) and log loss (cross entropy) for binary classificationtasks.Baselines. We compare IFM with the following competitivemethods, some of which are state-of-the-art models for sparseprediction.

- LibFM [Rendle, 2012]: This is the official implementa-tion4 for factorization machines that features stochasticgradient descent.

- Wide&Deep [Cheng et al., 2016]: The wide part is lin-ear regression and the deep part is a three-layer MLPswith layer sizes 1024, 512 and 256.

- DeepFM [Guo et al., 2017]: The FM part is a factor-ization machine, and the deep part is a three-layer MLPwith layer sizes 200, 200 and 200.

- NFM [He and Chua, 2017]: We take the implementationof NeuralFM and set the number of layers in the hiddenlayer to 1 with 512 neurons as the original paper.

- AFM [Xiao et al., 2017]: We use the implementationfor AttentionFM, where the attention factor is set to 16and the L2 regularization of attention net is set to 2 asrecommended.

2http://baltrunas.info/research-menu/frappe3http://www.kaggle.com/c/avazu-ctr-prediction4http://www.libfm.org/

Parameter Settings. We implement our method using Ten-sorflow5. To enable a fair comparison, all methods are learnedby optimizing the squared loss (Equation 9) using the Ada-grad (Learning rate: 0.01). The batch sizes for Frappe andMovieLens are set to 2048 and 4096, respectively, while theembedding size is set to 256 for all methods. The batch sizesfor Avazu is set to 2048 and the embedding size is set to40. We use a L2 regularization with λ = 0.00001 for NFM,Wide&Deep, DeepFM and IFM. The default setting for thenumber of neurons per layer in IFM is 256. For Wide&Deep,AFM, DeepFM and IFM, we find that pre-training their fea-ture embeddings with FM leads to better performance thanusing random initialization, and so we present their perfor-mance with pre-training.

5.2 Hyper-parameter Study (Q1)We study the impact of hyper-parameters on IFM in this sec-tion, including the number of hidden layers, the dropout ratioand activation functions. We conduct experiments on the vali-dation data by holding the settings for the FM prediction layerwhile varying the settings for FEN.

1 2 3 4Number of layers

0.286

0.288

0.290

0.292

0.294

RMSE

on

Frap

pe Frappe

0.425

0.426

0.427

0.428

0.429

0.430

0.431

RMSE

on

Mov

iele

ns

Movielens

(a) Number of Layers.

0.00 0.25 0.50 0.75Dropout Ratio

0.28

0.29

0.30

0.31

0.32

RMSE

on

Frap

pe Frappe

0.420

0.425

0.430

0.435

0.440

0.445

0.450RM

SE o

n M

ovie

lens

Movielens

(b) Dropout Ratio

Figure 3: Impact of network hyper-parameters(RMSE)

1 2 3 4Number of layers

0.06

0.07

0.08

0.09

0.10

MAE

on

Frap

pe Frappe

0.13

0.14

0.15

0.16

0.17

0.18

MAE

on

Mov

iele

ns

Movielens

(a) Number of Layers.

0.00 0.25 0.50 0.75Dropout Ratio

0.06

0.07

0.08

0.09

0.10

0.11

MAE

on

Frap

pe Frappe

0.14

0.16

0.18

0.20

MAE

on

Mov

iele

ns

Movielens

(b) Dropout Ratio

Figure 4: Impact of network hyper-parameters(MAE)

Depth of Network. Figure 3(a) and Figure 4(a) show theimpact of the number of hidden layers in FEN. We can ob-serve that the performance of IFM increases with the increas-ing of networks at the beginning. However, the model perfor-mance starts to degrade when the depth of networks is greaterthan 2(Frappe) or 3(Movielens). This is caused by overfittingevidenced by the observation that the training error still keepsdecreasing.

Dropout ratio. Dropout is an effective regularization tech-nique for neural networks to prevent overfitting. Figure 3(b)

5Codes are available at https://github.com/gulyfish/Input-aware-Factorization-Machine

and Figure 4(b) show the impact of dropout ratio on FEN. Bysetting the dropout ratio to a proper value, the performance ofIFM on both datasets can be significantly improved. Specifi-cally, the optimal dropout ratios for Frappe and MovieLensare 0.3 and 0.4, respectively. This verifies the usefulnessof dropout on the factor estimating network, which also im-proves the generalization of IFM.

Activation Function. A common practice in deep learningliterature is to employ non-linear activation functions on hid-den neurons. We thus compare the performance of differentactivation functions on IFM. As shown in Figure 5, ReLU ismore appropriate than others for the two datasets.

sigmoid tanh relu identityactivation functions

0.285

0.290

0.295

0.300

RMSE

on

Frap

pe Frappe

0.420

0.425

0.430

0.435

0.440

RMSE

on

Mov

iele

nsMovielens

sigmoid tanh relu identityactivation functions

0.065

0.070

0.075

0.080

MAE

on

Frap

pe Frappe

0.145

0.150

0.155

0.160

0.165

0.170

MAE

on

Mov

iele

ns

Movielens

Figure 5: Impact of activation functions on IFM per-formance

0 25 50 75 100Epoch

0.0

0.1

0.2

0.3

0.4

0.5

RMSE

FrappeIFM(train)IFM(valid)FM(train)FM(valid)

(a) Frappe

0 25 50 75 100Epoch

0.0

0.1

0.2

0.3

0.4

0.5

RMSE

MoivelensIFM(train)IFM(valid)FM(train)FM(valid)

(b) Movielens

Figure 6: The comparison of convergence behaviorsof IFM and FM

5.3 Impact of the Factor Estimating Network (Q2)The factor estimating network in IFM plays a pivotal rolein refining the representations of features for different in-stances. We first compare IFM with FM to demonstrate theimportance of the factor estimating network. Figure 6 showsthat IFM generally converges faster than FM. On Frappe andMovieLens, both the training and validation errors of IFM aremuch lower than those of FM, indicating that IFM can betterfit the data and lead to more accurate prediction. This justifiesthe rationality of IFM’s design of learning a unique weight forthe same feature in different instances via a neural network,which is the key contribution of this work.

Mechanism AnalysisTo better demonstrate the effect of FEN, we select three testinstances from Frappe. Table 1 shows the difference in fea-ture representation with respect to different instances. Wecan see that, in IFM, the same predictor (e.g., x3 and x9) mayhave significantly different degrees of importance in differ-ent instances, as evidenced by the diverse input-aware factor

values. By contrast, FM assigns each feature a fixed levelof predictive power in three instances, resulting in systemati-cally larger prediction errors.

Table 1: The changes of representation w.r.t. thesame features in different instances from FrappeN Method x3 x9 · · · y − y (x)

1 FM w3,v3 w9,v9 · · · 0.22IFM 3.12∗ (w3,v3) 0.15∗ (w9,v9) · · · 0.09

2 FM w3,v3 w9,v9 · · · 0.34IFM 0.67∗ (w3,v3) 0.05∗ (w9,v9) · · · 0.16

3 FM w3,v3 w9,v9 · · · 0.14IFM 2.29∗ (w3,v3) 1.19∗ (w9,v9) · · · 0.07

Performance Analysis of Model ComponentsNote that, in IFM and other baseline models, each extramodel component has its own unique functionality: DNNmodels high-order feature interactions; attention tries to pro-duce a weight for each feature interaction; FEN focuses onthe adaptive representation of features. To provide more in-sights into the effect of each type of component, we first trainall models without using the extra components (i.e., simu-lating the standard FM model). We then freeze feature em-beddings, and train the extra components only. To enablea fair comparison, the hyper-parameters of each model arecarefully tuned for best possible performance, and upon con-vergence, the relative improvements (RI) of each model onFrappe and MovieLens are listed in Table 2. It is clear thatwith the help of FEN, IFM achieves 35% and 22% improve-ment on the two datasets, respectively. Although the DNNand attention components also bring noticeable benefits to theFM model, the importance of incorporating adaptive featurerepresentation is clearly highlighted.

Table 2: Relative improvements (RI) of extra compo-nents on Frappe and MovieLens

Method Component Frappe MovieLensRI RI

FM - - -DeepFM DNN 20% 12%

NFM DNN 21% 10%AFM Attention 12% 9%IFM FEN 35% 22%

5.4 Performance Comparison (Q3)In this section, we compare IFM with several strong base-lines, and the results are summarized in Table 3 and Table 4,from which we have the following key observations:

• First, IFM consistently achieves the best results (mini-mum prediction errors) on three datasets with large per-formance margins over state-of-the-art methods. Thisdemonstrates overall the effectiveness of IFM and fur-ther justifies the importance of input-aware feature rep-resentation.

• Second, Wide&Deep, DeepFM and NFM are consis-tently better than the FM model, which is attributed tothe feature extraction by DNN for modelling higher-order feature interactions. However, the superiority ofIFM demonstrates that a shallow model with accuraterepresentation of features can achieve even better per-formance than deep learning methods. This points toa promising direction for future research: developingmore effective methods for learning accurate feature rep-resentations.

Table 3: Performance comparison for regression taskon Frappe and MovieLens (embedding size: 256)

Method Frappe MovieLensMAE RMSE MAE RMSE

LibFM 0.1611 0.3352 0.2604 0.4706Wide&Deep 0.1304 0.3222 0.2385 0.4502

DeepFM 0.0891 0.3149 0.1856 0.4485NFM 0.0912 0.3074 0.2141 0.4413AFM 0.1280 0.3085 0.2220 0.4379IFM 0.0625* 0.2872* 0.1522* 0.4273*

* indicates that the improvement of IFM over all other methodsis statistically significant (α=0.05).

Table 4: Performance comparison for binary classifi-cation task on Avazu (embedding size: 40)

FM AFM NFM DeepFM IFM

AUC(%) 76.20 77.82 77.98 78.22 78.50Logloss 0.3912 0.3821 0.3798 0.3782 0.3771

6 ConclusionIn this paper, we presented an effective predictive modelnamed Input-aware Factorization Machine (IFM) for sparsedatasets. It aims to enhance traditional FMs by purposefullylearning more flexible and accurate representation of featuresfor different instances with the help of a factor estimating net-work. The major advantage is that it provides a principledmechanism to better reflect the predictive power of each fea-ture in specific situation, while keeping the linear time com-plexity as the original FMs. Extensive experiments on threereal-world benchmark datasets show that the accurate repre-sentation of features can yield better performance than captur-ing high-order feature interactions. In terms of four popularperformance metrics, IFM significantly outperforms the clas-sical FM and state-of-the-art deep learning based approachessuch as Wide&Deep, AFM, NFM and DeepFM on regressionand classification tasks.

References[Baltrunas et al., 2015] Linas Baltrunas, Karen Church,

Alexandros Karatzoglou, and Nuria Oliver. Frappe: Un-derstanding the usage and perception of mobile app rec-ommendations in-the-wild. CoRR, abs/1505.03014, 2015.

[Cheng et al., 2014] Chen Cheng, Fen Xia, Tong Zhang, Ir-win King, and Michael R Lyu. Gradient boosting factor-ization machines. In Proceedings of the 8th ACM Con-ference on Recommender systems, pages 265–272. ACM,2014.

[Cheng et al., 2016] Heng Tze Cheng, Levent Koc, JeremiahHarmsen, Tal Shaked, and Hemal Shah. Wide & deeplearning for recommender systems. In Proceedings of the1st Workshop on Deep Learning for Recommender Sys-tems, pages 7–10. ACM, 2016.

[Guo et al., 2017] Huifeng Guo, Ruiming Tang, YunmingYe, Zhenguo Li, and Xiuqiang He. Deepfm: afactorization-machine based neural network for ctr predic-tion. In Proceedings of the 26th International Joint Con-ference on Artificial Intelligence, pages 1725–1731, 2017.

[Harper and Konstan, 2016] F Maxwell Harper andJoseph A Konstan. The movielens datasets: Historyand context. Acm transactions on interactive intelligentsystems (tiis), 5(4):19, 2016.

[He and Chua, 2017] Xiangnan He and Tat-Seng Chua. Neu-ral factorization machines for sparse predictive analytics.In Proceedings of the 40th International ACM SIGIR con-ference on Research and Development in Information Re-trieval, pages 355–364. ACM, 2017.

[Juan et al., 2016] Yuchin Juan, Yong Zhuang, Wei-ShengChin, and Chih-Jen Lin. Field-aware factorization ma-chines for ctr prediction. In Proceedings of the 10thACM Conference on Recommender Systems, pages 43–50.ACM, 2016.

[Koren et al., 2009] Yehuda Koren, Robert Bell, and ChrisVolinsky. Matrix factorization techniques for recom-mender systems. Computer, (8):30–37, 2009.

[Lian et al., 2018] Jianxun Lian, Xiaohuan Zhou, FuzhengZhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun.xdeepfm: Combining explicit and implicit feature inter-actions for recommender systems. In Proceedings of the24th ACM SIGKDD International Conference on Knowl-edge Discovery & Data Mining, pages 1754–1763. ACM,2018.

[McMahan et al., 2013] H Brendan McMahan, Gary Holt,David Sculley, Michael Young, Dietmar Ebner, JulianGrady, Lan Nie, Todd Phillips, Eugene Davydov, DanielGolovin, et al. Ad click prediction: a view from thetrenches. In Proceedings of the 19th ACM SIGKDD in-ternational conference on Knowledge discovery and datamining, pages 1222–1230. ACM, 2013.

[Pan et al., 2018] Junwei Pan, Jian Xu, Alfonso Lobos Ruiz,Wenliang Zhao, Shengjun Pan, Yu Sun, and Quan Lu.Field-weighted factorization machines for click-throughrate prediction in display advertising. In Proceedings ofthe 2018 World Wide Web Conference on World Wide Web,pages 1349–1357. International World Wide Web Confer-ences Steering Committee, 2018.

[Petroni et al., 2015] Fabio Petroni, Luciano Del Corro, andRainer Gemulla. Core: Context-aware open relation ex-traction with factorization machines. In Proceedings of the

2015 Conference on Empirical Methods in Natural Lan-guage Processing, pages 1763–1773, 2015.

[Qu et al., 2016] Yanru Qu, Han Cai, Kan Ren, WeinanZhang, Yong Yu, Ying Wen, and Jun Wang. Product-basedneural networks for user response prediction. In Data Min-ing (ICDM), 2016 IEEE 16th International Conference on,pages 1149–1154. IEEE, 2016.

[Rendle et al., 2011] Steffen Rendle, Zeno Gantner,Christoph Freudenthaler, and Lars Schmidt-Thieme.Fast context-aware recommendations with factorizationmachines. In Proceedings of the 34th internationalACM SIGIR conference on Research and development inInformation Retrieval, pages 635–644. ACM, 2011.

[Rendle, 2010] Steffen Rendle. Factorization machines. InData Mining (ICDM), 2010 IEEE 10th International Con-ference on, pages 995–1000. IEEE, 2010.

[Rendle, 2012] Steffen Rendle. Factorization machines withlibFM. ACM Trans. Intell. Syst. Technol., 3(3):57:1–57:22,May 2012.

[Shan et al., 2016] Ying Shan, T Ryan Hoens, Jian Jiao, Hai-jing Wang, Dong Yu, and JC Mao. Deep crossing: Web-scale modeling without manually crafted combinatorialfeatures. In Proceedings of the 22nd ACM SIGKDD Inter-national Conference on Knowledge Discovery and DataMining, pages 255–262. ACM, 2016.

[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton,Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi-nov. Dropout: a simple way to prevent neural networksfrom overfitting. The Journal of Machine Learning Re-search, 15(1):1929–1958, 2014.

[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Jason We-ston, Rob Fergus, et al. End-to-end memory networks. InAdvances in neural information processing systems, pages2440–2448, 2015.

[Weston et al., 2014] Jason Weston, Sumit Chopra, and An-toine Bordes. Memory networks. arXiv preprintarXiv:1410.3916, 2014.

[Xiao et al., 2017] Jun Xiao, Hao Ye, Xiangnan He, Han-wang Zhang, Fei Wu, and Tat-Seng Chua. Attentional fac-torization machines: learning the weight of feature inter-actions via attention networks. In Proceedings of the 26thInternational Joint Conference on Artificial Intelligence,pages 3119–3125, 2017.