Workforce Language Assessment in the 21st Century Knowledge Economy: A Quality Management Perspective

Presenters/Authors & DiscussantNick Saville and Mike Milanovic

University of Cambridge ESOL ExaminationsRyan Damerow, Courtney Pahl, & Kathi Bailey

Monterey Institute of International Studies & TIRFMary Ann Christison

University of UtahDiscussant: Jean Turner

Monterey Institute of International Studies

Surveying English Language Assessment Practices in

International Plurilingual Organizations

Ryan Damerow (TIRF)Courtney Pahl

Monterey Institute of International StudiesKathi Bailey (MIIS & TIRF)

TIRF

The International Research Foundation for English Language Education

Promoting research and best practices to improve the use of English in the 21st century

global knowledge economy

TIRF

Survey based in part on research commissioned and published by TIRF in 2009 – available as a free downloadable PDF at TIRF’s website: www.tirfonline.org

Sampling and Respondents

Questionnaire designed and distributed withSurveyMonkey (convenience sampling)

Monterey Institute alumni Anaheim University alumni and studentsInternational Chambers of CommerceGlobal Finance Association

English Career magazine (Taiwan) Individuals working in international organizationsN = 243 responses to date

Organizational Sectors Represented

Indicate the sector that best describes your organization. Please check only one.

Organizational Sectors Represented

The “ Other” category Included the following:International NGOs, government & diplomacy,

law and legal practice, technology, arms control, grant making, maritime activities, translation & interpretation

And some responses that we thought were already covered by our existing categories: automobile manufacturing, accounting, and telecommunications

In what countries does your organization currently operate?

Afghanistan, Algeria, Angola, Argentina, Australia, Austria, Azerbaijan, Bahrain, Bangladesh, Belgium, Belize, Benin, Bolivia, Brazil, Burkina Faso, Burundi, Camaroon, Cambodia, Canada, Central African Republic, Chile, China, Colombia, Costa Rica, Czech Republic, Denmark, Dominican Republic, Ecuador, Egypt, El Salvador, Estonia, Ethiopia, Finland, France, Gabon, Ghana, Germany, Georgia, Guatemala, Haiti, Holland, Honduras, India, Indonesia, Iran, Iraq, Ireland, Israel, Italy, Japan, Jordan, Kazakhstan,

In what countries does your organization currently operate?

Kenya, Korea, Kosovo, Kyrgystan, Latvia, Lebanon, Liberia, Libya, Lithuania, Madagascar, Malaysia, Mali, Mexico, Morocco, Mozambique, Namibia, Nepal, New Zealand, Nicaragua, Nigeria, Norway, Oman, Panama, Peru, Philippines, Poland, Portugal, Russia, Rwanda, Saudi Arabia, Senegal, Serbia, Singapore, South Africa, Spain, Sri Lanka, Sudan, Sweden, Taiwan, Tanzania, Thailand, Togo, Turkmenistan, UAE, Uganda, UK, Ukraine, USA, Venezuela, Vietnam, Yemen, & Zambia.

In what countries does your organization currently operate?

“Too many to name”“Worldwide”“Globally”“Every country of the Caribbean”“In 32/ 36/ over 40/ over 60/ 72/ 80/ 87/ 90/

120/ over 160/ 167/ over 190 countries”

Languages used to communicate within your organization

Arabic Cantonese Dutch English French German Italian Japanese Korean Mandarin Portuguese Russian Spanish Other (please specify)

What is the primary language used by your employees in those locations to communicate

within your international organization?

Other Languages Reported

Albanian, Amharic, Armenian, Azerbaijani, BahasaMelu, Bangla, Catalan, Creole, Czech, Danish, Dari, Estonian, Farsi, Finnish, Georgian, Gujarati,Hebrew, Hindi, Hokkien, Khmer, Kirundi,Kinyarwanda, Macedonian, Malay, Marahti,Melanesian Pidgin, Mongolian, Norwegian,Pashtu, Polish, Serbo-Croation, Slovak, Slovenian,Swahili, Swedish, Tamil, Turkish, Ukrainian, Urdu, Vietnamese, & African and Indian languages

Languages used to communicate with clients Arabic Cantonese Dutch English French German Italian Japanese Korean Mandarin Portuguese Russian Spanish Other (please specify)

What is the primary language used by your employees in those locations to communicate

with clients of your international organization?

Other Languages Used with Clients

All of the “other” languages listed above and also Bulgarian, Hungarian, Malagasy,

Malayam, Kannada, Kazakh, Kyrgyz, Lozi, Quechua, Tajik, Telegu, and “too many to name.”

Research Question #1

1. What are the necessary English language skills individuals must possess to be successful as competent employees and leaders in international plurilingual organizations in the 21st century knowledge-based economy?

English Speaking & Listening Skills

Interacting in face-to-face meetings 88.5% Telephone conversations 88.1% Listening during face-to-face meetings 87.2%Listening to presentations 79.8% Giving presentations 78.2% Listening to teleconferences 73.3% Running face-to-face meetings 72.4% International travel 67.9%

English Speaking & Listening Skills

Presenting during teleconferences 66.3% Negotiations (for example, a contract) 65.8% Socializing 63.8% Talking with people at exhibitions & tradeshows 53.9%

Conducting interviews 48.1% Being interviewed 47.7% Other (please specify) 15.2%

English Writing Skills

Writing e-mails 95.9% Writing correspondence 81.5% Writing reports 75.3% Writing proposals 69.1% Writing memos 60.9% Writing newsletters 37.0% Writing manuals 35.8% Other (please specify) 14.4%

English Reading Skills Reading e-mails 97.1% Reading correspondence 86.4% Reading reports 86.4% Reading proposals 73.7% Reading memos 73.3% Reading manuals 65.8% Reading newsletters 60.5% Other 9.5%

Research Question #2

What assessment procedures and instruments are used to determine whether such individuals possess

the necessary skills to accomplish those roles successfully?

Do you have English language requirements for any of your

employees?

• No 9.5% • Yes 81.0% • Don't know 9.5%

If you have English language requirements, do they vary by

position?

No 25.3% Yes 59.1% Don't know 15.6%

Are bilingual/plurilingual applicants hired over monolingual applicants

with equal skill sets (other than their language abilities)?

• No 7.8% • Yes 70.0% • Don't know 22.2%

Are bilingual/plurilingual employees paid more than monolingual

employees in the same positions?

• No 41.2% • Yes 17.3% • Don't know 41.6%

Do you test the English proficiency of any of your employees using any

commercially developed test(s)?

• No 60.9% • Yes 13.2% (n = 32)• Don't know 26.3% If yes, what tests…? (TOEIC was identified 6 times

and TOEFL four times; UN Proficiency Exam twice; government OPI, and UniCredit Group were each mentioned once.)

Do you test the English proficiency of your employees using any in-house procedure(s)?

• No 37.4% • Yes 36.6% (n = 89)• Don't know 27.6% If yes, what procedures are used? In-house interviews, some writing samples, speech recognition tool (1), & translating documents (1)

Do you test the English proficiency of your employees using any in-house procedure(s)?

“There is no formal procedure, but the application and interview process can weed out potential contractors whose English abilities are weak.”

“We don't use any official test, but through the interview process and writing samples we are able to assess language ability.”

“Financial test in English.”

Do you test the English proficiency of your employees using any in-house procedure(s)?

“The HR department develops language tests for English (or the relevant language for the project) depending on the position.”

“No specific procedures except conversation.”“We talk to them. If they speak English well enough to communicate with management, they pass.”

If you use any in-house testing procedure(s), what training is given to the test

administrators?

There were 75 responses to this item. Among them

26 said “None” 14 said they didn’t know or weren’t sure. 5 said “Not applicable.”

If you use any in-house testing procedure(s), what training is given to the

test administrators? “None; testing would be very simple.”“The interviewer is the experienced one.”“The administrators do the tests based on their

experience in the area tested.” “The interviewer must speak English fluently.” “The Japanese manager simply uses his own personal

criteria of what defines ‘good English.’”“Most are PhDs in linguistics.”

Think about some employees in your organization who are bilingual or plurilingual. Have the bilingual or

plurilingual skills of those employees had a positive impact on your organization?

•No 4.5% •Yes 82.3% •Don't know 13.2%

Think about some employees in your organization who are NOT bilingual or plurilingual. Has the lack of bilingual or

plurilingual skills of those employees had a negative impact on your organization?

• No 40.7%

• Yes 27.2%

• Don't know 32.1%

Does your organization support employees in the improvement of their

second or foreign language skills?

• No 28.0% • Yes 59.3% • Don't know 14.0%

Does your organization calculate return on investment for language

development?

• No 54.5% • Yes 7.1% • Don't know 39.6%

If you answered YES to the previous question, please check all the methods for calculating

return on investment that apply: Change in quality of work 71.4% Change in time needed to complete tasks 71.4%Change in quantity or output of work 42.9% Change documented by experts (e.g., supervisors, personnel managers, etc.) 42.9%Change in costs 35.7% Other (please specify) 21.4%

Closing CommentsIn Summary70% of the respondents said that bilingual or

plurilingual applicants get hiring preference.

81% said there are English requirements for employees.

Only 13.2 % of the respondents report using commercially developed language tests.

Closing Comments

59.3% of the respondents reported that their organization does provide language development support for employees

BUTOnly 7.1% said that their organization calculates the Return on

Investment for such training.

Closing Comments

About 95% of the respondents report thatEnglish is usedas a language of communication within the

organizationas a language of communication with clients

Closing Comments

Over one-third report that their organization uses some in-house procedure(s) to assess employees’ English proficiency.

Of the 75 respondents who report using some kind of in-house procedure, at least one-third (n = 26) reported no training is provided for test administrators.

Closing Comments

Over 75% say English is the primary languageused to communicate within the organization andabout two-thirds say English is the primarylanguage used to communicate with clients.

Relatively few organizations in this small data set report using TOEIC or other professionally developed language tests.

Closing Comments

Following Nick’s and Mike’s presentation about Quality Management Systems (QMS) in language assessment, it seems that little attention is paid to the quality of language assessment by many of these international organizations.

Many business schools use the QMS framework – which may be a reason for language testers to use it as a talking point in business contexts.

Closing CommentsLimitations to the Study: 1. Qualitative data not yet fully analyzed2. More data forthcoming3. Convenience sampling 4. Regional and sector distribution uneven5. What to do with the “don’t know” answers?6. Exclude responses from language-based groups?7. Individual respondents vs. organizations as the

source of data -- What to do with multiple responses from the same organization?

Closing Comments

For reference lists on various languageassessment topics and one on language inbusiness, industry and the professions,please visit www.tirfonline.org

Click on “Access to Resources” and then on “References” (downloadable Worddocs in APA format – please share with others).