Prediction of Human Personality Traits From Annotation Activities

Nizar Omheni1, Omar Mazhoud1, Anis Kalboussi1 and Ahmed HadjKacem21Higher Institute of Computer and Management, University of Kairouan, ReDCAD Laboratory,

Khmais Alouini Street 3100, Kairouan, Tunisia2Faculty of Economics and Management, University of Sfax, ReDCAD Laboratory,

Road of the Airport Km4 3018, Sfax, Tunisiafomheninizar, omarmazhoud, kalboussianisg@gmail.com, ahmed.hadjkacem@fsegs.rnu.tn

Keywords: Annotation, Personality, Big Five Personality Model.

Abstract: We show how reader’s annotation activity captured during an active reading session relates to their personal-ity, as measured by the standard Five Factor Model. For 120 volunteers having usually the habit of reading,we gather personality data and annotation practices. We examine correlations between readers personalityand such features of their annotative activities such as the total number of annotation acts, average number ofannotation acts, number of textual annotation acts, number of graphical annotation acts, number of referentialannotation acts and number of compounding annotation acts. Our results show significant relationships be-tween personality traits and such features of annotation practices. Then we show how multivariate regressionallows prediction of the readers personalities traits given their annotation activities.

1 INTRODUCTION

Studying human activity and interaction with technol-ogy has grown dramatically over the last decade. Yetstudying reading poses particular challenges. (Mar-shall,2010) reported the citation of Tzvetan Todorov,quoted by Nicholas Howe in Jonathan Boyarins com-pilation, the Ethnography of Reading: ”Nothing ismore commonplace than the reading experience, andyet nothing is more unknown. Reading is such a mat-ter of course that at first glance it seems there is noth-ing to say about it”. Although details of reading ac-tivity (moving eyes, writing annotation...) may tell ussomething about the reader. When people read andinteract actively with their reading materials they dounselfconscious activities which can be keys featuresto their personalities.

For decades, the psychologists search to under-stand the human personality and to find a systematicway to measure it. After several researches they showa relation of dependence between human personalitytraits and different behaviors. (Ryckman,2010) re-ported the Allport’s 1 definition of personality: ”per-sonality is the dynamic organization within the in-

1Gordon Willard Allport (November 11, 1897 October9, 1967) was an American psychologist. He was one of thefirst psychologists to focus on the study of the personality,and is often referred to as one of the founding figures ofpersonality psychology.

dividual of those psychophysical systems that deter-mine his characteristic behavior and thought”. Thus,in Allports view human behavior is really controlledby internal forces known as the personality traits.

This paper attemps to bridge the gap betweenreading activity research and personality researchthrough reader’s annotation practices. Our core re-search question asks whether annotation activity canpredict personality traits. If so, then there is an oppor-tunity to use a natural human practice as a new sourceto better understand the reader personality.

Several works has shown the opportunity of pre-dicting user personality using the information peo-ple reveal in their online social profile (Twitter, Face-Book) (Bachrach et al,2012) (Golbeck et al,2011).They refer to what people share, self-description, sta-tus updates, photos, tags, etc. We pretend that annota-tive activity is more spontaneously and natural prac-tice and it can reveal something about human person-ality.

Personalization attracted increased attention inmany areas. So the need to predict personality traitsincreases over time mainly when several research hasshown the link between personality traits and suc-cess in human relationship and practices (Barrick andMount,1991) (Eswaran et al,2011). By nature an in-troverted person is not interested to make so much re-lation with other while an extravert person do. Actu-ally, certain developed recommendation systems con-

263

sider the personality traits as key feature of recom-mendation (Nunes et al,2008).

The paper is structured as follows: In Section 2,we present background on the Big Five personalityindex. Then we present our experimental setup andmethods. In the third section we present the results oncorrelation between each annotative activity featureand personality factor. Next, we show how multivari-ate regression allows prediction of annotators person-alities traits given their annotation activities. We con-clude with a discussion of the possible implicationsthat this work has for such domains of application.

2 BACKGROUND AND RELATEDWORK

2.1 The Big Five Personality Model

The big five personality traits are the best ac-cepted and most commonly used scientific measureof personality and have been extensively researched(Peabody and De Raad,2002). That personality iswell described as five traits was discovered throughthe study of the adjectives from natural langage thatpeople used to describe themselves and then analyz-ing the data with a statistical procedure known as fac-tor analysis that is used to reduce lots of informationdown to its most important parts. In the following wecite a brief explanation of the five personality traits.

2.1.1 Openness to Experience

Openness includes traits like imagination, apprecia-tion for art, depth of emotions, adventure, unusualideas, intellectual curiosity, and willingness to experi-ment. People who score high in openness like usuallyto learn new things and enjoy new experiences.

2.1.2 Conscientiousness

Conscientiousness includes traits like orderliness,selfdiscipline, deliberateness, and striving theachievement. People that have a high degree ofconscientiousness are planned, have the tendencyto act dutifully, have the sense of responsibility andcompetence.

2.1.3 Extraversion

Extraversion includes traits like energy, positiveemotions, surgency, assertiveness, sociability andtalkativeness. Extraverts people get their energy from

interacting with others, while introverts get their en-ergy from within themselves.

2.1.4 Agreeableness

Agreeableness includes traits like trust in others, sin-cerity, altruism, compliance, modesty and sympathy.People that have high degree of agreeableness arefriendly, cooperative, and compassionate, while peo-ple with low agreeableness may be more distant.

2.1.5 Neuroticism

Neuroticism relates to ones emotional stability anddegree of negative emotions. This dimension mea-sures the people degree of anxiety, angry, moodiness,and the sensitivity to stress. People that score highon neuroticism often experience emotional instabilityand negative emotions.

2.2 Related Work

In (Burger,2011) view the personality is a ”consistentbehavior patterns and intrapersonal processes orig-inating within the individual”. Trait psychologistsassume that personality is relatively stable and pre-dictable (Burger,2011). So, several research work hasbeen done with personality traits as it influences hu-man decision making process and interests. (Nuneset al,2008) pionneered the model and implement ofpersonality traits in computers. Indeed, (Nunes etal,2008) propose to model the user’s traits in a pro-file which they called User Psychological Profile -UPP. In order to fill in the profile UPP the authorsutilised an online tool called the NEO-IPIP2 inven-tory based on 300 items. Through user’s answers toNEO-IPIP inventory the authors are able to predictthe user personality. Through their experimentation(Nunes et al,2008) try to prove that RecommenderSystems can be more efficient if they use the UserPsychological Profile (UPP). Although the authorsfollow an explicit way to predict the user traits, the re-sults presented in (Nunes,2008) are fruitful. (Tkalcicet al,2009) propose a personality-based approach thatis based on the big five model for collaborative fil-tering Recommender Systems. In fact, the authorscalculate the user personality scores by means of aquestionnaire. Then they measure the user similar-ity, that is based on personality, that yields a list ofclose neighbours. This list is used after as a databaseto compile a personalized list of recommended items.

2The NEO-IPIP is a computer based Personality Inven-tory, able to measure people Personality Traits created byJohn Johnson (Johnson.).

WEBIST�2014�-�International�Conference�on�Web�Information�Systems�and�Technologies

264

(Roshchina et al,2011) propose personality-based rec-ommender system which the aim is to select for theuser reviews that have been written by like-minded in-dividuals. The user similarity is calculated based onthe presonality traits according to the Big Five model.The authors predict the presonality traits based on lin-guistic cues collected from the user-generated text.(Bachrach et al,2012) and (Golbeck et al,2011) showthe relationship between personality traits and variousfeatures of social media (FaceBook, Twitter). Theirfindings prove the possibility to predict accurately theuser’s personality through the publicly avaible infor-mation on their social network profile.

As it is mentioned above most of works has beendone with the ”Big Five” model of personality dimen-sions which has emerged as the most well-researchedand regarded measures of personality in last decades.The best of our knowledge, our work is among thefirst to look at the relationship between annotation ac-tivity and personaltiy traits. Much works try to pre-dict personality from what the user offer consciously(answers to a questionnaire, informations available onpublic social profile...). Despite the fact that the find-ings of these researchs are fruitful we believe that pre-dicting personality from annotative activity is morecredible as the annotation is defined as ”a basic andoften unselfconscious way in which readers interactwith texts” (Marshall,2010).

Due to the spontaneously and unselfconscious as-pects of annotation we are interested to predict per-sonality from this potential source of knowledge.

3 DATA COLLECTION

We consider group of 120 volunteers. The subjects se-lected were recruited with respect to certain criterias.Infact, the age of our volunteers should be between 18and more and they should be academic people. In oursample we have the two sex (44 women and 76 men).Another criteria for selection , we asked if the volun-teer has the habit of reading and does he annotated hisdocuments frequently. If all these conditions exist thesubject can be selected to our experimentation.Each subject was instructed to answer a standard FiveFactor Model questionnaire (the NEO-IPIP Inven-tory). Then, he obtained a feedback regarding his per-sonality based on his responses. This step gives usthe personality scores based on the Big Five Modelfor each volunteer. To associate personality scoresto subjects annotative activities, we gather annotationpractices for each people. Here, we collect documentsannotated in a spontaneous and natural way. So weasked, first of all, if the subject had a document an-

notated previously (academic course, book,...). If notwe asked him what topics interest him, then we givehim an article with few pages to not weary him.We are very careful to the comfortability of the vol-unteers during the experience to guarantee their spon-taneous and natural reactions. Thus they are free tochoose places and conditions to read and annotate thedocuments and they have enough time to do. Thestrategy followed give us fruitful results. Infact, thedifferent subjects (who have not a document anno-tated previously) interact actively with the readingmaterials in view of the feel of comfortableness andthe interest to the document read.

Figure 1: Annotation practices of a reader.

3.1 User Annotation Activity

(Marshall,2010) defines the annotation as ”a basic andoften unselfconscious way in which readers interactwith texts”. We mean by annotation the act to adda critical or explanatory notes to a written work, tohighlight a passage, to write down, and so on marksthe reader makes on a page during his reading activity.To fulfil our experimentation, we ask each subject togive us an annotated paper document. Then, we anal-yse the readers annotations to extract some features.We started by classifying annotations in the threegeneral categories cited by (Agosti and Ferro,2003) :graphical annotation acts, textual annotation acts, andreference annotation act all depends to the material-ization sign of the semantics of the annotation addedto the annotated document. Then, for each reader, wecollect a simple set of statistics about their annotativeactivity. These included the following:

1. Total Number of Annotation Act (TNAA)

2. Average Number of Annotation Act (num-ber of annotation acts per a single annotatedpage)(ANAN)

3. Number of Graphical Annotation Act (NGAA)

4. Number of Textual Annotation Act (NTAA)

Prediction�of�Human�Personality�Traits�From�Annotation�Activities

265

5. Number of Reference Annotation Act (NRAA)

6. Number of compounding Annotation Act (textualsign, graphic sign and reference sign of annotationact can be compounded together in order to ex-press complex meanings of annotation)(NCAA).

This set of statistics tends to characterize quantita-tively the reader’s annotation practices. Next we runa Pearson correlation3 analysis between subjects’ per-sonality scores and each of the features obtained fromanalyzing their annotative activities.

4 PERSONALITY ANDANNOTATION FEATURESCORRELATION

We study the Pearson correlation between subjects’personality scores and each of the features obtainedfrom analyzing their annotative activities. We reportthe correlation values in table I. Those that were sta-tistically significant for p < 0:05 are bolded.

Table 1: Pearson correlation values between annotation fea-tures scores and presonality scores.

Open. Consc. Extra. Agree. Neuro.TNAA -0,059 0,128 -0,138 0,089 -0,287ANAA 0,003 0,080 -0,210 0,163 -0,183NGAA -0,067 0,040 -0,130 0,105 -0,207NTAA 0,001 0,182 0,040 0,085 -0,211NRAA -0,075 0,045 -0,122 0,077 -0,207NCAA -0,059 -0,012 -0,147 0,014 -0,219

We found in our analysis fewer significant corre-lations, but we believe a larger sample size would pro-duce much better results. However, the results we ob-tained even with a small sample show promise that theannotative activity can be useful for computing suchpersonality traits. In fact, table I shows significantcorrelations for Neuroticism, Conscientiousness, andExtraversion traits. We need larger sample to verifythe inference of the other traits from peoples annota-tions.

Next, we present the scatter plots for the most sig-nificant correlations between annotation practices fea-tures and personality traits. These plots presentingthe relationship between annotative activity featuresand human traits, where horizontal axis represents theaverage personality trait scores and the vertical axisrepresnts the annotative activity feature values.

3Pearson’s correlation r 2 [�1;1] measures the linear re-lationship between two random variables.

4.1 Conscientiousness

As presented in table I Conscientiousness is positivelyrelated to the number of textual annotation act (fig.2).The rest of the correlation values are not consideratedbecause of p-value > 0:05. But this is not a reason toreject definitively the rest of annotation features as alarger sample size may produces other significant cor-relations.The considered correlation may indicates that consci-entious people are interested to use textual annota-tion acts. Infact, conscientious individuals are prudentwhich means both wise and cautious, better organizedand they avoid acting spontaneously and impulsively.Thus, it may be the case that people who have highdegree of conscientiousness are interested to use tex-tual annotation more than other annotation acts as itdemands more reflexion, reasoning and cognitive ef-fort.

Figure 2: Scatter Plot showing Number of Textual Annota-tion Act against Conscientiousness scores.

4.2 Extraversion

According to results shown in table I, Extraversion isnegatively correlated with the average number of an-notation act (fig.3). The rest of the correlation valuescan be probably significant with a larger sample size.We can interpret the regression fit shown in figure 3as follow: The fit is correlated negatively which is notsurprising as extraversion is marked by pronouncedengagement with the external world where extravertstend to be energetic and talkative while introverts aremore likely to be solitary and reserved. Thus, it maybe the case that reading and annotation is an intimateactivities, we do it in private, so people who are so-cially active are less willing to practice annotation.

4.3 Neuroticism

Table I shows that neuroticism is negatively corre-lated with all the features of annotation activity. Here,

WEBIST�2014�-�International�Conference�on�Web�Information�Systems�and�Technologies

266

Figure 3: Scatter Plot showing Average Number of Annota-tion Act against Extraversion scores.

the chosen sample size is sufficient to have significantcorrelations for all the annotation features. The differ-ent correlation values are very significant which canshow the sensitivity of annotation practices to the neu-roticism trait.In other hand, one possible explanation for these cor-relations is that more Neurotic people are emotion-ally reactive and they experience negative emotionsfor unusually long periods of time which can dimin-ish the neurotics ability to think clearly and make de-cisions. Thus those who score high on Neuroticismare less eager to use annotation act as they can notactively and critically engaging with the content for along periods of time.

Figure 4: Scatter Plot showing Total Number of AnnotationAct against Neuroticism scores.

4.4 Openness and Agreeableness

Unfortunately, the correlation values related to theOpenness and Agreeableness traits are very low. Butwe can not reject definetly the hypothese of predictionof these traits from annotation activity. We may ob-tain significant values if we increase the sample size.As an exemple, the p-value of the regression fit ofthe Average Number of Annotation Act against theAgreeableness traits is p = 0:076. This value can beameliorated with a larger sample size.

Figure 5: Scatter Plot showing Number of compoundingAnnotation Act against Neuroticism scores.

5 PREDICTING PERSONALITY

Previously we examined the correlations betweeneach of Big Five personality dimension and anno-tative activity features. Now, we are interested tomake predictions about a subject’s personality basedon multiple annotation features. First of all, we usedthe multivariate linear regression to predict each per-sonality trait using the annotation features. Next, weused the coefficient of multiple determination R2 tomeasure the strength of fit. Also, we measure theF-test to verify the statistical significance of the col-lective influence that have the annotation features onthe personality traits. Thus, larger values of the F-teststatistic provide stronger evidence against H04. To re-ject H0 the value of the F-test should exceeds a criticalvalue calculated as follow:

F =R2=k

(1�R2)=[n� (K +1)]Where k is the number of explanatory variables in ourmodel which corresponds to the number of annotationactivity features(K=6)and n represents the sample size(n=120). So the Fobserved is compared against a criticalF with 6 degree of freedom in the numerator and n-7degrees of freedom for error in denominator. In ourcase Fcritical=2.18 for alpha level5 of 0.05. Resultsshown in table II indicate that the null hypothesis isrejected for two cases. In fact, Neuroticism and Con-scientiousness can be predicted with reasonable ac-curacy using features of annotation activity, whereasother traits are more difficult to be predicted using an-notations. Prediction regarding Conscientiousness isreasonably accurate, with R2 value of 0.12, Fobserved

4The null hypothesis states that there is no relationshipbetween annotation activity features and personality traits.

5The alpha level is defined as the probability of what iscalled a Type I error in statistics. That is the probability ofrejecting H0 when in fact it was true.

Prediction�of�Human�Personality�Traits�From�Annotation�Activities

267

value of 2.52 which exceeds the Fcritical value and P-value of 0.03 which is lower than the a value whereP-value is the probability the F-test statistic is largerthan the observed F-value. For Neuroticism we ob-tained the model with the best fit, with an R2 valueof 0.14, Fobserved value of 3.11 and P-value of 0.01,indicating quite accurate a prediction. The model forExtraversion has a lower fit and the model for Agree-ableness is even less accurate. It seems that Opennessis the hardest trait to predict using annotation activityfeatures.Table 2: Predicting personality traits using annotation ac-tivity features through multivariate linear regression.

Trait R2 F test P-valueOpenness 0.03 0.57 0.76

Conscientiousness 0.12 2.52 0.03Extraversion 0.07 1.32 0.25

Agreeableness 0.05 1.03 0.41Neuroticism 0.14 3.11 0.01

6 DISCUSSION

In this study we show that Neuroticism and Conscien-tiousness traits are correlated with annotation activityfeatures. We expect a larger sample size can be help-ful to verify the correlation of the other human traitsto annotation practices.

Our findings are based on pen-and-paper approachwhich is qualified by its relative ease with which thereader may interact with a document in an intuitiveand familiar manner.

Recent researchs endeavor to replace the ”pen-and paper” pardigm for the annotating needs. Dif-ferent systems and tools of annotation are developedsuch as: iAnnotate (Plimmer et al,2010), u-Annotate(Chatti et al,2006), YAWAS6, iMarkup (2013), etc.Such tools enable readers to annotate their digital doc-uments with free form annotations similarly to ”pen-and paper” case. iAnnotate for example is an anno-tation tool for android system and it enables users toadd annotations with the pencil, highlighter, and notetools.

Recently we intend talking about new products forreading such as the tablet. With the aid of such de-vices, the user may interact easily with a digital docu-ment and enter her annotations as he do in the case of”pen-and-paper”. Thus, our findings is promising andoriginal to be applied in such digital areas.

6Yawas is a free web annotation tool for Firefox andChrome built on top of Google Bookmarks. Yawas enablesyou to highlight text on any webpage and save it in yourGoogle Bookmarks account.

We believe it’s the occasion to develop a systemto attract those that have a curiosity to use anno-tation platforms and practices - from end users in-cluding scholars, scientists, journalists, public ser-vants...etc. We expect our system enables readers tointeract actively with their reading materials via an-notation practices. Our goal is to use the traced anno-tations on the digital document to infer the annotatorpersonality traits. Thus the expected system shouldcontain free form annotation tool easy to be used, beable to infer user personality traits from captured an-notation activity and refines the user traits profile byreference to new captured annotations. Based on themodelled profile we expect our system be able to offerservices to users such as friendship recommendationbased on similarity of users personality, customizinguser interface using such predefined personas, sharingannotated documents...etc.

To achieve the personalization process we need toknow certain user’s features. Several research worksprove that prediction of personality traits reveals alot about a user’s features. These findings was ap-plied in several domains such as recommender sys-tems (Nunes et al,2008) (Roshchina et al,2011) andthe results is interesting.

The annotation as an unselfconscious practiceconstitutes a credible source of knowledge. In(Kalboussi et al,2013) the annotation activity is usedto invoke the appropriate Web services to users. Thisproves that annotation is rich enough to be used dif-ferently.

In this paper we use annotation to infer suchreader traits. That is a promising work and representsa new tendency to model user personality from humanbehaviour.

In other hand, let’s be objective, there are somelimitations to this work. The most important issue isthe sample size as we expect more significant resultswith a larger sample. This limitation may be due tothe dependency to ”pen-and-paper” approach whichprevents us to benefit from the population of readersover the web. In addition, people are not interestedto participate in our experimentation unless they aremotivated. We expect resolving these issues in futureworks.

Finally, our work is the first step to study the rela-tion of reader annotation practices to human person-ality traits. So much perspectives can be subjects offuture works such as studying factors which are likelyto influence annotating behaviour such as familiaritywith annotation tools and interest in the content topic.Also our research can be extended to study the pos-sibility to predict human traits through social annota-tions. These avenues and others are very interesting

WEBIST�2014�-�International�Conference�on�Web�Information�Systems�and�Technologies

268

and represent an opened future directions which needsmore investigations.

7 CONCLUSION

In this paper, we have shown that such users’ per-sonalities traits can be predicted from their annota-tion practices. With this ability of prediction manyopportunities are opened which suggests future direc-tions in variety of areas such as user modeling, recom-mender systems, user interface design and so on areasrelative to personalization research domain. Further-more, this work bridges the gap between the readingand the personality research domains and it remainsan open research questions to see whether personalitycan also be predicted using other potentially featuresof reading activity as well as the influence of such en-vironmental factors on human annotation behaviour.

REFERENCES

Agosti,M. Ferro,N., 2003. Annotations: Enriching a Digi-tal Library. In: Proceedings of the 7th European Con-ference (ECDL) Trondheim, Norway. Springer Pages88-100.

Bachrach,Y. Kosinski,M. Graepel,T. Kohli,P. Still-well,D.,2012. Personality and Patterns of FacebookUsage. In: Web Science12 Proceedings of the 3rdAnnual ACM Web Science Conference. ACM NewYork, NY, USA, Pages 24-32.

Burger, J. M., 2011. Personality. Editor Wadsworth, USA.Barrick, M. R. Mount, M.K., 1991. The big five personal-

ity dimensions and job performance: a meta-analysis.Personnel psychology, Volume 44, Issue 1, Pages 126.

Chatti, M. A. Sodhi,T. Specht, M. Klamma,R. Klemke,R.,2006. u-Annotate: An Application for User DrivenFreeform Digital Ink Annotation of E-Learning Con-tent. In: Proceedings of the Sixth IEEE Interna-tional Conference on Advanced Learning Technolo-gies (ICALT), Washington, DC, USA. Pages 1039-1043.

Eswaran,S. Aminul Islam,Md. Dayang Hasliza,M.Y., 2011.A Study of the Relationship between the Big Five Per-sonality Dimensions and Job Involvement in a ForeignBased Financial Institution in Penang. In: Interna-tional Business Research. Vol. 4, No. 4.

Johnson, J.A. The IPIP-NEO: InternationalPersonality Item Pool Representationof the NEOPI-R, viewed 8 June 2013,http://www.personal.psu.edu/ j5j/IPIP/ipipneo300.htm

Golbeck, J. Robles,C. Edmondson, M. Turner,K., 2011.Predicting Personality from Twitter. In: Privacy, Se-curity, Risk and Trust PASSAT, 2011 IEEE Third In-ternational Conference on Privacy, Security, Risk, and

Trust, and 2011 IEEE Third International Conferneceon Social Computing.

Kalboussi, A. Mazhoud, O. Hadj Kacem,A.,2013. Annota-tive Activity as a Potential Source of Web Service Invo-cation. In: Proceedings of the 9th International Con-ference on Web Information Systems and Technolo-gies (WEBIST), Aachen, Germany. SciTePress Pages288-292.

Nunes, M.A. Cerri,S.A. Blanc,N., 2008. Improving Rec-ommendations by Using Personality Traits in UserProfiles. In: International Conferences on KnowledgeManagement and New Media Technology, Graz, Aus-tria.

Nunes, M.A. Cerri,S.A. Blanc,N., 2008. Towards User Psy-chological Profile. In: IHC ’08 Proceedings of theVIII Brazilian Symposium on Human Factors in Com-puting Systems. ACM Pages 196-203.

Nunes,M.A.,2008. Recommender Systems based on Person-ality Traits. PhD Thesis, University of MONTPEL-LIER 2.

Marshall, C., 2010. Reading and Writing the ElectronicBook.. Editor Gary Marchionini, University of NorthCarolina, Chapel Hill.

Plimmer, B. Hsiao-Heng Chang, S. Doshi, M. Laycock,L.Seneviratne,N., 2010. iAnnotate: Exploring Multi-User Ink Annotation in Web Browsers. In: Proceed-ings of the 11th Australasian User Interface Confer-ence (AUIC). ACM Pages 52-60.

Peabody, D. De Raad, B., 2002. The Substantive Natureof Psycholexical Personality Factors: A ComparisonAcross Languages. Journal of Personality and SocialPsychology, vol 83, issu 10; Pages 983-997.

Roshchina, A. Cardiff, J. Rosso, P., 2011. User ProfileConstruction in the TWIN Personality-based Recom-mender System.. In: Proceedings of the Workshopon Sentiment Analysis where AI meets Psychology(SAAIP), IJCNLP. Pages 7379.

Ryckman,R. M., 2008. Theories of Personality. EditorThomson Wadsworth USA.

Schmitt, D. Allik, J. McCrae, R. Benet-Martinez,V., 2007.The geographic distribution of Big Five personalitytraits: Patterns and profiles of human self-descriptionacross 56 nations. Journal of Cross-Cultural Psychol-ogy, vol 38, Pages 173-212.

Tkalcic, M. Kunaver, M. Tasic, J. Košir,A., 2009. Per-sonality based user similarity measure for a col-laborative recommender system. In: Proceedings ofthe 5th Workshop on Emotion in Human-ComputerInteraction-Real world challenges, Cambridge, UK.Pages 30-37.

The iMarkup Annotation Tool: a Commer-cial tool, viewed 13 September 2013,http://www.bplogix.com/support/imarkup-client.aspx

Prediction�of�Human�Personality�Traits�From�Annotation�Activities

269