1

5 May 2009

Azzman MohamedLeeds Business SchoolLeeds Metropolitan University

Dear Sir,

Doctoral Symposium Submission for AHRD Conference 2009

Paper Title: SOFT SKILLS EVALUATION: CONTRIBUTING FACTORS INMALAYSIA

Name of author: Azzman Mohamed

Affiliation: PhD Candidate at Leeds Metropolitan Business School (2nd Year)

Address: Leeds Metropolitan UniversityFaculty of Business and LawBronte HallHeadingly CampusLeeds, LS6 3QS

Supervisor: Professor Jim Stewart and Dr. Rick Holden

Email Address: A.Mohamed9443@student.leedsmet.ac.uk

Telephone number: 0113 8120000

Conference Stream: Doctoral Symposium

Submission type: Refereed paper

Keywords: Evaluation, Soft skills, Training, Return on investment, Malaysia

Thank you.

Kind regards,

Azzman Mohamed

2

Soft skills evaluation: contributing factors in Malaysia

Azzman MohamedLeeds Metropolitan University, UK

Abstract

In increasing markets and economy, human capital investments have evolved as aresult of intense competition, rapid growth and change, colliding cultures, technologywaves and more, that occurs at unprecedented speeds. The interest and purpose ofthis research is due to Malaysian Government worries on Government LinkedCompanies (GLCs) and to assist the Malaysian Government as in the 9th MalaysianPlan, to focus on the National Mission to achieve Vision 2020 through one of thepriority areas, human capital development.

This paper draws on the training evaluation literature and the return-on-investment(ROI) model of Philips (1994) about the factors contributing to the implementation ofevaluation on business impacts (BI) and ROI for soft skills training in Malaysian GLCs.It is also to investigate if there are relationships between evaluation contributingfactors with the decision to evaluate higher level evaluation and to find which of theseevaluation factors contribute the most. The proposed contribution factors to bediscussed are organisational policy, organisational resources and organisationalculture.

Both quantitative and qualitative methodology will be adopted, using e-mailquestionnaire and semi-structured interviews. Respondents will consist of HRmanagers and executives of the Malaysian GLCs across the country.

It is hoped that the findings of the contribution factors can perhaps to be used asguidelines and allow the GLCs to make necessary preparations for future higher levelevaluation.

Keyword(s): Evaluation, Soft skills, Training, Return on investment, Malaysia

Introduction

In increasing markets and growth of the economy, human resource (HR) practitionersand HR personnel are putting more emphasis on the development of human capitalinvestments in their organisations. Indeed, towards the extent of this, AmericanSociety for Training and Development (ASTD) (2004) stated that workplace learningprofessionals must “articulate the business value” of their initiatives and demonstratea payback for their efforts in the form of improved organisational performance andmeasurable results. It is seen the importance of showing the bottom line impact oftraining programmes not only proves that the programme is beneficial for theorganisation, but also provides a platform for deciding which programme is to becontinued or needs discontinuation (Burkett, 2005). According to Bartel (2000), it isimportant for companies to have accurate measures of the rate of return on

3

investments (ROI) in employee training, as this is what guides their human capitalinvestment decisions.

The most recent survey by the Chartered Institute of Personnel and Development(CIPD) (2008) in its Learning and Development Annual Survey Report 2008, statedthat compared with two years ago, organisations are now requiring a broader range(61%) and a higher level of skills (40%). The report also mentioned that the key skillsthat employers class as very important include interpersonal (79%) andcommunication skills (68%). However, 66% of organisations feel that new employeescurrently lack both communication/ interpersonal skills and management/leadershipskills. The report also mentioned that organisations are now looking for more andbetter skills among their staff, with interpersonal and communications skillsconsidered as very important. While these, along with management skills, areabilities that organisations consider essential to their success. Echol (2005) statedthat over 60 per cent of all jobs require skills and competencies of educatedknowledge workers. This demand resulted in an outlook for the supply of workers tobe poor. Echols (2005) also stated that by year 2008 25 per cent of seniormanagement positions would be vacant and the supply of qualified candidates to fillin these vacancies is unclear. In general, organisations implement trainingprogrammes with the idea that this increases employee’s efficiency and productivity.

All this emphasis on training employees has resulted in issues of successful trainingconcentrating on evaluation as the bottom line approach for monetary returns.According to Burkett (2005), a systematic approach to measuring training impactbegins with an evaluation framework. Kirkpatrick (1998) created a four-level modeland later Phillips (1994) expanded upon this framework to incorporate a fifth level,ROI. The five-level evaluation framework is as follows:

1. Level 1: Reaction and planned action. Measures reaction and the actionplanned following a training or learning initiative.

2. Level 2: Learning. Measures learning and knowledge gained from the trainingprogramme.

3. Level 3: Application. Measures the extent to which trainees apply learned skilland knowledge back on the job.

4. Level 4: Business impact. Measures changes in business results relateddirectly to the training project.

5. Level 5: ROI. Measures ROI or how the monetary benefits of the programmecompare with its costs.

To date, the training evaluation literature has focused on methods used in trainingevaluation (Rowden, 2005), on incidents, determinants and impacts of training(Bishop, 1997) and on aligning evaluation methods with strategic organisationalneeds (Potter et al., 2003). The current evaluation practices during 2003 stated that asurvey conducted by LIMRA (a consulting company based in the USA for over 850financial services companies in more than 70 countries) Field Development andPerformance Committee (FDPC) of 81 companies (26 responded to the survey),found that only about half measure training at levels 3 and 4 and fewer than 10%measure level 5 evaluation (Johnson et al., 2005). Dixon (1996) stated that “all thebest practice companies were conducting Level 3 and 4 evaluations selectively ratherthan consistently. Echols (2005) stated that the financial impact of strategicinvestment in training and education is well documented. Measuring business impact

4

of the learning function, high performance companies cite that sales per employeeare 27 per cent greater, revenue growth is 40 per cent greater and income growth is50 per cent greater. Hence the need for rigorous analytical tools for human capitalinvestment decisions.

Furthermore, the literature on training evaluation (at the time of writing) has notexamined the relationship of training implementation factors with decisions toevaluate training programmes at higher levels such as business impacts (BI) and ROI.These training contributing factors, their contribution made and the exploration ofpotential gaps are discussed in the literature review section.

The aim of this research is to investigate the extent to which Malaysian GLCsevaluate higher level evaluation. GLCs are defined as companies with a primarycommercial objective and in which the Malaysian Government has a direct controllingstake which refers to the Government’s ability to appoint Board of Directors (BOD)members and senior management and make major decisions for GLCs. The categoryof GLCs comprises companies controlled by the respective State Governments andState-level agencies (Khazanah Nasional Berhad, 2008). The interest in selectingthese organisations for this study is due to the Malaysian government worries on theGLCs’ sustainability in the market; to cope with these challenges and requirements,GLCs require a distinct set of quality manpower by equipping their employees withthe right training.

It also aims to investigate if there are relationships between training contributingfactors with the decision to evaluate higher level evaluation. Furthermore, thisresearch seeks to find which of these training contributing factors contribute to higherlevel evaluation. As outlined above, this topic has not yet been sufficiently addressedin the literature and comprises the testing of contributing factors of trainingimplementation and its relationships with higher level evaluation and, moreimportantly, finding which of these factors contribute strongly towards higher levelevaluation (contribution to knowledge at PhD level). It will provide the trainingliterature with an implementation model related to factors contributing to investmentsin higher level evaluation.

Literature Review

Definition of Evaluation

Weiss (1972, p.4) in McCoy and Hargie (2001) defines evaluation as “to measure theeffects of a program against the goals it set out to accomplish as a means ofcontributing to subsequent decision making about the program and improving futureprogramming”. Remenyi and Sherwood-Smith (1999) define evaluation as “an activitythat is central to and inextricably linked with decision making and consequently withthe management and control of rational business activity”, while other authors suchas Buckley and Caple (1990) define evaluation as “a process of attempting to assessthe total value of training: that is the cost benefits and general outcomes, whichbenefit the organisation as well as the value of the improved performance of thosewho have undertaken training”

5

Evaluation is important to measure any programme that is going to be run or ongoing,to measure how well the programme is functioning, the impact of the programme and,most importantly, the analysis of the programme benefits relative to the costs (Berkand Rossi, 1990). Remenyi and Sherwood (1999) suggest that evaluation supportsthe decision making of the whole process of development of a programme, fromjustifying design decisions to justifying decisions about the design process. However,in order to carry out evaluation, organisations must be ready and have to considerthe objectives and outcomes of the evaluation. Tyler (1942) in Stufflebeam andShinklefield (1985) suggested that goals and objectives must be defined beforeevaluation is to be carried out and then evaluation will measure whether the goalsand objectives have been met. The suggestion of defining the goals and objectives inorder to evaluate is also agreed by other authors such as Weiss (1972) in McCoy andHargie (2001). He suggests that in order for evaluation to be successful, it requiresthe use of research methodologies to measure the outcome of the programmeagainst established criteria.

Definition of Soft Skills

As the name suggests, soft skills are the skills of relationship management andcreativity and ability to influence when interacting with people (Guisti, 2007). Softskills are intangible, elusive and therefore not easily taught. Goleman (1995) definessoft skills as “emotional intelligence”. These refers to the ability to manage conflicts,motivate a team, communicate with other people and many more, while others suchas Halfhill and Nielsen (2007) define soft skills as the interpersonal skills used inone-to-one and one-to group settings. Guisti (2007) divided soft skills into two:interpersonal skills and advanced soft skills. Examples of interpersonal skills areeffective communications, leadership, team building and listening skills, whileexample of advanced soft skills are career planning, marketing and sales pitches,project management and delegation, public speaking and time management.

Today, organisations are equipping their employees with soft skills, especially seniormanagers and the top management group, because they realise the intense need forsoft skills. An individual who has the knowledge and ability to use soft skills in his/herwork is more likely to be successful (Goleman, 1995; Bolton, 1986). A study of 968major firms in the USA by Huselid (1995), and Pfeffer and Veiga (1999) in Halfhill andNielsen (2007) found that lower turnover, increased profits, higher sales peremployee and increased stock market value are the results of effective managementskills. Thus those skills are vital in order to ensure the development of organisations(Guisti, 2007).

Training Evaluation Practices

In today’s rapid moving environment and competitive business challenges, traininghuman capital has become an important asset in organisations and businesses onwhich they spend a huge amount of money every year. Bachler (1997, p. 98)reported that American business spent an estimated US$52 billion in training in 1996.Carrigan (2001) stated that about US$750 billion is spent annually around the world,representing an average of almost two per cent of the payroll of many organisations.Organisations are increasingly working towards developing the human capital

6

towards more effectiveness in its training programmes. We have heard trainingproviders all around the globe working towards developing training programmes tomeet performance improvement initiatives.

Training is only a method of human capital intervention, however, evaluation allowsfor justification to meet organisations objectives, needs and performance. It is utterlyimportant for us to concentrate on the impact of the training process itself. There area few methods to measure or calculate training benefits used by organisations andthey vary from lower to higher level evaluation. Tamkin et al. (2002) referred toseveral available models of evaluation such as the following:

Four-level approach: Kirkpatrick (1959) from Kirkpatrick (1996) Kirkpatrick plus

o Five-level approach: Hamblin (1974)o Organisation Elements Model (OEM): Kaufman, Keller and Watkins

(1995)o Indiana University approach: Molenda, Pershing and Reigheluth (1996)o Carousel of Development: Industrial Society (2000)o Five-level ROI Framework: Phillips (1994), Phillips and Holton (1995)o KPMT model: Kearns and Miller (1997)o CIRO (Context, Input, Reaction, Outcome) Approach: Warr, Bird and

Rackham (1970)

Even though there are many evaluation models available and training function isbelieved to have become an important agenda for business performance today, westill see many organisations not yet very subtle about the importance of conductingevaluation nowadays. Very minimal attention is still paid towards the returns that theirtraining programmes are achieving. Rowden (2005) stated that HR managers areoften plagued by the lack of simple and effective methods for showing the benefits ofthe training provided by their departments, often resorting to ‘smiley sheets’ or justassuming that if training is based on a needs analysis, it is probably effective.

Furthermore, Bartel (2000) stated that data from ASTD’s Benchmarking Forumindicate that the number of companies performing some type of results-orientedevaluation grew from 27 per cent of Forum companies in 1994 to 40 per cent in 1995,but the number of these firms which convert the results from training into monetaryterms in order to calculate an ROI on the training investment is not clear.

For the purpose of this research, the researcher will study the evaluation approachwith relation to the Five-level ROI Framework by Phillips (1994).

Argument for Measuring HRD Costs and Benefits

Developing human capital does have demonstrable costs. However, it is more than acost, indeed it is an investment. According to Smith (2004), it is an investment in thedevelopment of individual, organisational and ultimately societal capability. As in anyinvestment, the return on the outlay should be demonstrable. Reasons advanced inopposition to measure HRD effectiveness frequently revolve around a view thatbenefits are too difficult to measure. A variation on this assessment will be toosubjective and requiring too many assumptions to be made (Campbell, 1994;

7

Edwards et al., 2003). Somewhat cynically, it has also been suggested that theaversion to outcomes measurement on the part of some HRD managers may be assimple as fear of collecting data which may show their programmes to be ineffective,the possibility being that HRD budgets may be cut as a result (Kraiger, 2002). Otherthan this fear of potentially adverse budget impacts, the arguments againstmeasuring ROI or non-return on soft skills training investment essentially distils downto a view that isolating the outcomes from HRD and tying these back to initial outlayis just too complex and too difficult a task to be undertaken. This is an argument toohard to maintain.

If training employees is important to be done, if an investment is being made in thedevelopment of human capital, then it is reasonable that the business impact andROI should be able to be demonstrated and measured. Asserting that soft skills’training is intrinsically a good thing and therefore its effectiveness needs to bemeasured is not a sustainable argument. The relationship between soft skills trainingprogrammes and the resulting improvements, the broad benefits of soft skills traininginvestments, can and should be actively identified and measured. Just as HRD itselfis not an optional extra but a strategic imperative, so too is the effectivemeasurement and evaluation of training outcomes.

Factors Contributing To Training Evaluation

The systematic approach to training requires elements of evaluation. Manyorganisations venture into training without taking prior consideration of factors whichpromote its application. For example, Cohen (2005) indicates that there are too manyreasons why organisations are shying away from taking on evaluation. It wasmentioned in the 2004 ASTD survey that a cross-section of benchmarking servicesorganisations concluded that the majority of these organisations embark on lowerlevel evaluation (Level 1) because an evaluation programme is not an easy task.Very many factors have been considered as important for successful evaluation. Forthis study, the following discussion will work through some of the key related factorsto be used in this research for the decision to evaluate a soft skills trainingprogramme and its success at the higher levels of evaluation, BI and ROI. Thecontributing factors considered in this paper are:

top management support, government involvement, shareholder value, knowledge and skills, time, cost and budgets, reliable data availability, effective training, internal and external evaluator, performance transparency, and economic conditions.

Having the top management involved in the evaluation programme may influence thedecision to evaluate training and its success. Phillips and Phillips (2002) discussedthe importance of top executives’commitment and involvement in order to carry out atraining evaluation programme. Others, such as Elenkov and Manev (2005), West et

8

al. (2003), Henry (2001), Sastry (1999), Den Hartog and Verburg (1997), Howell andAvolio (1993) and Howell and Higgins (1990), discussed the idea of top managementsupport’s positively influencing outcomes of other innovation processes inorganisations. The second factor influencing the decision to evaluate training at thehigher level is government involvement. Bussmann (2008), Reeve and Peerbhoy(2007) and Hashim (2001) suggests the importance of government involvement toensure evaluation is being carried out, while van Brakel (2002) mentioned that labourlaw is one of the factors influencing HRM practices and policies. Other authorsdiscuss shareholder value as one of the contributing factors. Shenkman and Gobarty(2007), Kearns (2005), May et al. (2003), van Brakel (2002) and Hillman and Keim(2001), for example, discuss a lot the power of shareholders and how shareholdervalue can influence decision making for higher evaluation of training and its success.

For an organisation to evaluate soft skills training, it requires knowledge and skillswithin its staff. Johnson et al. (2005) and Wright and Belcourt (1995) mention thatorganisations are shying away from evaluating training programmes due to lack ofknowledge and skills. Others such as Parker-Wilkins (2006), Burkett (2005), Rowden(2005), IOMA (2004), Potter et al. (2003), Williams et al. (2002), Tucker et al. (1999),Bishop (1997) and Phillips (1997) discuss and suggest the steps to be taken in theevaluation process.

The fifth factor is time, cost and budget. Burkett (2005), Johnson et al. (2005) andParry (1996) discuss the time factor that holdsback the decision to evaluate a trainingprogramme and its success, while Bamberger et al. (2004) explore the combinationof time and budget constraints. Phillips and Phillips (2002), Tucker et al. (1999) andLewis and Thornhill (1994) on the other hand discuss the cost of evaluation side.

The sixth element is reliable data availability which is also important for organisationsto consider. Johnson et al. (2005), van Brakel (2002), Bartel (2000), Tucker et al.(1999) and Phillips (1997) mention that without reliable data, organisations fail toevaluate training programmes and calculate ROI. Later, they and with some otherauthors such as Campbell (2006), Burkett (2005), Rowden, (2005), and Smith (2004)discuss a lot guidelines to manage data. Effective and relevant training given to staffis a seventh factor that determines the decision to evaluate training and to ensure itssuccess (Berge, 2008; van Brakel, 2002; Tucker et al.,1999). The eighth factor isinternal and external evaluators. Posavac and Carey (2003) discuss the advantagesand disadvantages of both while others such as Wang and Wang (2005), Odusami etal. (2003) and Elaine (1998) explain the importance of external evaluators to carryout the job.

Performance transparency is also one of the important elements that influence thedecision to evaluate a training programme and to ensure a meaningful result isachieved (van Brakel, 2002). Later, Johnson et al. (2005), Phillips and Phillips (2002),Bramley (1998) and Campbell (1994) discuss the reasons for this. Last but not least,economic conditions is one of the factors organisations need to consider in order tomake a decision to evaluate a training programme (Kirwan and Birchall, 2006;Rowden, 2005; Hashim, 2002; Holton, 1995). Later, Longenecker et al. (1998)explain the relationship between economic conditions and the need to evaluateeducation programmes.

9

All of these factors are important for organisations to consider in order to evaluatesoft skills training programmes. The researcher categorised these factors into threemajor internal factor groups: organisational policy, resources and culture, and oneexternal factor which becomes a proposed conceptual framework for the researchshown in Figure 1.

Figure 1: Proposed Conceptual Framework

Contributing Factors (Independent Variables)

Evaluating (Dependent Variables)

Research Questions

Intense competition, rapid growth and change, colliding cultures, technology wavesand more that occur at unprecedented speeds are the multiple challengesconfronting the world economy today (Kirwan and Birchall, 2006; Dunn and Pope,2001; Longenecker et al., 1998). In order to cope with these challenges andrequirements, the organisation requires a distinct set of quality manpower (Kirwanand Birchall, 2006; Dunn and Pope, 2001). The realisation of the Ninth MalaysianPlan for example is heavily dependent upon on how efficient the organisationmanages its human resources functions. Leaders and executives have been askedto take on new roles requiring an array of new, often unfamiliar, management andinterpersonal skills. Associates are expected to demonstrate ‘high performance’ skills(collaboration, teamwork and innovation) that allow them to work more efficiently andeffectively.

This research draws upon the training evaluation literature, the implementation of BIand ROI for measuring the monetary returns of the Soft Skills Training Programme in

Organisational Policy Top Management Support Government Involvement Shareholder Value

Organisational Resources Knowledge and Skills Time, Cost and Budget Reliable Data Availability

Economic Conditions

Business ImpactsReturn On Investments

10

Malaysian GLCs by Philips (1994). Evaluation of training programmes in Malaysiahas its limitations within the context of soft skills in business impacts and monetaryreturns to the organisation.

While all organisations realise the importance of putting the most talented leaders inthe right spots at the right time, responding to business opportunities, mostcompanies however are finding it difficult to attract and retain good people. Axelrod etal. (2001) found a research by McKinsey in 2000 that 89 per cent of respondentssurveyed thought it is more difficult to attract talented people now than it was threeyears ago, and 90 per cent thought it is now more difficult to retain them. Just 5 percent of the respondents strongly agreed that their companies had enough talentedmanagers to pursue all or most promising business opportunities. Realising thisintense, most GLCs in Malaysia have to manage their manpower more efficiently.One of the approaches is to increase the enrolment in soft skills training for middleand top-level management.

Thus, in the light of these circumstances, a profound knowledge about trainingevaluation will be necessary in addressing human capital investments fororganisation productivity. The overall aim of this study is to develop a model ofcontributing factors for decisions to undertake higher level evaluation for soft skillstraining in Malaysian GLCs. The research questions that guided this study are:

1. To what extent do Malaysian GLCs evaluate the business impact and ROI of softskills training?

2. What are the contributing factors affecting the decision to evaluate the businessimpact and ROI of soft skills training?

3. Which of these contributing factors affect the perceived success of evaluatingbusiness impact and ROI on soft skills training?

Hypotheses

Starting from the above mentioned key research questions and research aims, thefollowing hypotheses are introduced, arising from a review of contemporary literatureon the training evaluation process.

The basis for this work is due to the Malaysian Government worries on the GLCssustainability in the market. A paper by the Putrajaya Committee (2005), CatalysingGLC Transformation to Advance Malaysia’s Development, mentioned that some ofthe GLCs are less profitable than before. The paper suggested three key successfactors as guidelines to ensure success and sustainability of Malaysian GLCs. Thethree key success factors are:

a. Strong leadership – intense personal conviction, commitment and involvementfrom corporations’ senior management, which typically requires a change inmindset and behaviour at that level first.

b. Large investments in human capital - required as multiple aspects of thetransformation need efforts going beyond ‘business as usual’ resources andcapabilities.

11

c. Steadfastness and political will – the learnings from successfultransformations all have the common characteristics of making and stickingwith many tough decisions. This includes divestments or closure of structurallyunderperforming businesses, changes of Chairman, CEO, Board membersand senior executives, as well as large-scale redundancies.

Indeed, one of these key success factors is investment in human capital. Thereforemaking it is important for GLCs to sustain and, most importantly, to compete in themarket. Training costs a lot of money and should be monitored closely by thecompanies in order to get the most returns instead of just spending. Inability ornegligence in monitoring or evaluating training at the business impact and ROI levelsdue to certain factors or weaknesses will drift the companies into more difficulties.The knowledge of promoting or encouraging companies into evaluating their traininginvestments allows the identification of areas for improvement. This research looksspecifically into the understanding of the factors contributing to their decisions toinvest in higher level evaluations such as BI and ROI. Within this context, it hasencouraged various hypotheses to be developed and tested. The following are twosamples of possible hypotheses developed to answer research question (3).

Probability of Hypothesis

The contributing factors contribution towards the decision to evaluate businessimpact and ROI.

eg. The more the support from top management, the more likely the decision

to evaluate at the business impact and ROI levels. The more reliable the data available with regard to training resources and

benefits, the more likely the decision to evaluate at the business impactand ROI levels.

Methodology and methods of data collection

There are various research methodologies in the social sciences providingresearchers with a choice of the most appropriate methodology to fit the researchsituation. The strengths and weaknesses of each methodology are clearly defined.The reality of the research objectives and questions becomes the prime reason forsuch a methodology to be chosen. Remenyi et al. (1998) indicated that the topic andspecific research questions become the main drivers for such a chosen methodology.The concept of business impact and ROI in training evaluation implementation is anew approach to understanding which requires a grasp of the factors contributing tomore decisions in evaluating ROI and business impacts of soft skills training. In thisparticular study, the unit of analysis is proposed to be GLCs in Malaysia whichcomprises small and large GLCs.

In this study, the researcher will use multi-methods in collecting data - questionnaireand case studies. This refers to e-mail questionnaire and semi-structured interview.The reason for the researcher’s using multi-methods is to capture the most accuratemeaning of the data collected. According to Wright and Crimp (2000), a mixture of

12

qualitative and quantitative methods will lead to more significant and more reliableconclusions. It is intended that triangulation will be adopted through use of an e-mailquestionnaire survey and case studies to provide additional insights and to confirmsome of the issues in the quantitative analysis. SPSS will be used in the quantitativesection of the research while data gathered from the qualitative section are to addmore richness and understanding to the quantitative data and to validate results aftermultiple regression analysis. The use of case studies is targeted at 10% of the wholepopulation of GLCs. The case study method will consist of semi-structured interviewsand document analysis.

Stage 1: Literature review (still developing - ongoing)

In the first stage, literature on training evaluation practices is examined. Additionally,literature on training evaluation practices and benefits has to be discussed andevaluated and their contributions as well as their shortcomings have to be explained.For these analyses, the gathering of secondary data was used as a major method ofresearch. This helped to develop initial hypotheses and to establish an initial modelwhich needs to be evaluated by qualitative and quantitative research. This stage ofthe research refers to the first and aim.

Stage 2: Research of training evaluation practices (questionnaire survey)

Quantitative research will be conducted first to help seek the quantitative part of thestudy by aiding conceptual development and instrumentation (Rossman & Wilson,1991; Janesick, 1998). The quantitative research is aimed at using e-mailquestionnaire survey distributed to HR managers and HR practitioners in MalaysiansGLCs.

According to Bell (1993), the aim of a survey is to obtain information which can beanalysed for patterns to be extracted and comparisons made. A survey is said to aimto obtain information from a representative selection of the population and from thesample it will then be possible to present the findings as being representative of thepopulation as a whole.

The use of questionnaire survey design enables the researcher to find information toanswer all three research questions in a speedier and cheaper way. The design isaimed to minimise non-response bias and increase validity of response by containinga limited number of open-ended questions and a majority of questions requiringsimply the ticking of a box or circling of a number.

In order to ensure that respondents easily understood the questionnaire, a pilot studyusing questionnaire will be made with a group of practitioners and academics in thepreliminary stage to provide suggestions for improving questionnaire clarity. Includedin the questionnaire is a covering letter stating the purpose of the research and itsimplications and advantages for the organisation for future innovations in training.

However, it should be accepted that no questionnaire survey is perfect. According toBryman (1995) and Remenyi et al. (1998), the set of questionnaire data allows datato be gathered about aspects of ‘what’, ‘when’, ‘how much’, or ‘how long’. Generally,surveys are not suitable to answer research questions relating to ‘who’, ‘how’ or ‘why’problems. This is when validation using qualitative method takes place.

13

Stage 3: Validation Using Qualitative Method

The use of a case study approach is very familiar in qualitative research. Yin (1994,p.13) defined the case study as an empirical inquiry where 1) it investigates acontemporary phenomenon within its real life context, 2) when the boundaries withinphenomenon and context are not clearly evident, and in which 3) multiple sourcesare used. According to Yin (1994, p.3), in many situations, the use of case studiesarises out of the desire to understand complex social phenomena. Yin states,

“The case study allows an investigation to retain the holistic and meaningfulcharacteristics of real-life events – such as individual life cycles, organisationaland managerial processes, neighbourhood change, international relations,and the maturation of industries.”

Following the analysis of the questionnaire, an interview and document analysisschedule will be developed focusing on the issues arising from it. The format willinclude some general questions which will add to the information gathered in thesurvey regarding training evaluation practices and implementation factors. The issuesof the roles of top management, government and all other levels in the organisationswill be discussed. This will include a detailed discussion of the organisations’experiences and perceptions of BI and ROI practices. Issues related to the latter twobeing used for decision making in training, improvement in monetary result areas,whether it is felt that the evaluation is worth the effort, its impact on the organisationand an overall evaluation of BI and ROI will be discussed. This will also includereports on training and training evaluation and company reports on BI and ROI, whichwill all provide additional data.

The aim of the interviews allows validation of the second stage of quantitative dataanalysis using the questionnaire survey. In this case, the strength of the qualitativeapproach will allow the researcher a more clear and concise situation arising from thequestionnaires. Furthermore, the outcome of the interviews will allow for triangulation,thus strengthening the findings of this research. According to Johnson andOnwuegbuzie (2004), quantitative research allows testing and validating ofquantitative data, in this case interviews will balance the results found.

For a provisional sample structure for in-depth interviews, relevant criteria are definedin order to build a reasonable sample.

Although an interview schedule will be used, the interviews will be allowed to flow in afairly unstructured manner in order to build confidence, encourage honest responsesand reduce social desirability bias. All interviews will be tape-recorded with thepermission of respondents and transcribed to improve reliability.

Data Analysis - Quantitative and Qualitative

The following analysis methods are applied in each of the research questions:

1. Research Question 1 (To what extent do Malaysian GLCs evaluate the businessimpact and ROI of soft skills training?)

14

Descriptive statistics will be employed for this question: mean, median,mode and percentages

Develop two section questions on Business Impact (BI) and ROI.

2. Research Question 2 (What are the contributing factors affecting the decision toevaluate the business impact and ROI of soft skills training?)

Find the contributing factors (CF) through a thorough literature review. Develop or adapt questions on contributing factors (CF) affecting the

decision to evaluate the business impact (BI) and ROI. Use descriptive data on RQ1

o Correlate CF with BI and ROI

3. Research Question 3 (Which of these factors affect the perceived success ofevaluating business impact and ROI on soft skills training?)

Apply multiple regression analysis to look at which of these factorsaffect the perceived success of evaluating business impact and ROIon soft skills training?

Create two Multivariate modelso Model 1 (Business Impact)

Dependent variable = BI Independent variable = CF

o Model 2 (ROI) Dependent variable = ROI Independent variable = CF

** Factor analysis will be conducted for the contributing factors, BI and ROI, forconstruct validity.

Conclusion

The study is to provide a timely and comprehensive investigation of contributingfactors in decision making for effective implementation of soft skills trainingevaluation in Malaysian GLCs. Given the wealth of theoretical literature relative totraining evaluation implementation, and the absence of information describingcontributing factors of training evaluation for soft skills at the business impact andROI levels for successful evaluation practices, the study helps to bridge the gapbetween theory and practice. Despite the possible potential limitations, the followingcontributions are hoped to be achieved.

1. This is the first comprehensive study of contributing factors in soft skillstraining evaluation in Malaysian GLCs (and there is no similar study in anASEAN country to the researcher’s knowledge, at the time of writing).

2. Development of an integrated conceptual model of contributing factors in softskills training evaluation for business impact and ROI within the Malaysiancontext and the identification of contributing factors in seeking soft skillstraining evaluation practices can be used by managers and executives asuseful guidelines in adopting and implementing soft skills training evaluation.

3. It will offer substantial descriptive findings on the current contributing factorsfor effective training evaluation practices and reasons for not adopting them inthe Malaysian context. These findings may encourage executives and

15

managers to re-evaluate and improve their soft skills training implementationin the light of the revealed practices and problems of others.

4. Previously unavailable information on the association between the contributingfactors of soft skills training and its practices and adoptions will be analysed. Inparticular, this study will initiate an empirical link between contributing factorsof soft skills training and training evaluation practices.

5. It will construct a model for contributing factors of soft skills training evaluationin a proper organisational perspective. Therefore an organisation would beable to assess the various significant variables identified and, based on thisassessment, decide whether the benefits it could gain by redesigning itstraining evaluation to adopt these evaluation techniques would bring monetaryreturns and business impacts to the company.

Overall, the study will identify and examine some clear disparities betweencontributing factors in soft skills training implementation theory and practice, andprovided insights into the efforts needed to bridge the gap.

References

ASTD Press (2004). ASTD Competency Study. Alexandria, VA, American Society forTraining and Development.

Axelrod, B., H. Handfield-Jones and T. Welsh. (2001). "The War for Talent, Part Two".McKinsey Quarterly Report. 2: 44-57.

Bachler, C. (1997). “The trainer’s role is turning upside down”. Workplace 76(6): 94-95, in Rowden, R. W. (2005). "Exploring Methods to Evaluate the Return-on-Investment from Training." Business Forum 27(1): 31-37.

Bamberger, M., J. Rugh, M. Church and L. Fort(2004). "Shoestring Evaluation:Designing Impact Evaluations Under Budget, Time and Data Constraints." AmericanJournal of Evaluation 25(1): 5-37

Bartel, A. P. (2000). "Measuring the Employer's Return on Investments in Training:Evidence from the Literature." Industrial Relations 39(3): 502.

Bell, J. (1993). Doing Your Research Project: A Guide for First-Time Researchers inEducation and Social Science. Buckingham, St Edmundsbury Press Ltd.

Berge, Z. L. (2008). "Why it is so hard to evaluate training in the workplace."Industrial and Commercial Training 40(7): 390-395.

Bishop, J. H. (1997). "What We Know About Employer Provided Training: A Reviewof the Literature." Research in Labor Economics 16(96-09): 19-87.

Bramley, P. (1999). "Evaluating effective management learning." Journal of EuropeanIndustrial Training 23(3): 145-153.

Bryman, A. (1995). Quantity and Quality in Social Research . London. Unwin Hyman.

16

Burkett, H. (2005). "ROI on a shoestring: strategies for a resource-constrainedenvironments: Measuring more with less (Part 1)". Industrial and CommercialTraining 37(1): 10-17.

Buckley, R., and J. Caple (1990). The Theory and Practice of Training London,Kogan Page.

Bussmann, W. (2008). "The emergence of evaluation in Switzerland." Evaluation14(4): 499-506.

Campbell, C.P. (1994). "A primer on determining the cost effectiveness of training:Part 2", Industrial and Commercial Training. 27(11): 17-25

Carrigan, S. (2001), "Training: Investment in the future". Canadian HR Reporter.14(11): 2

CIPD (2008). Learning and Development Annual Survey Report, April 2008. [online]London, CIPD. Available from www.cipd.co.uk/NR/rdonlyres/3A3AD4D6-F818-4231-863B-4848CE383B46/0/learningdevelopmentsurvey.pdf [accessed20/06/08]

Cohen, S. L. (2005). "Controlling program evaluation." Performance Improvement44(8): 23-26.

Den Hartog, D. N., and R. M. Verburg (1997). "Charisma and rhetoric:Communicative techniques of international business leaders." Leadership Quarterly8(4): 355-391, in Elenkov, D. S., and I. M. Manev (2005). "Top managementleadership and influence on innovation: the role of sociocultural context." Journal ofManagement 31(3): 381-402.

Dixon, N. (1996). "New routes to evaluation." Training and Development 50(5): 82-8

Dunn, A. G. and S. Pope (2001). "Leadership development which delivers results."Industrial & Commercial Training 33(6): 220-224.

Echols, M. E. (2005). "Invest in People." Leadership Excellence 22(6): 20-21.

Edwards, J. E.,J. Scott, J. C. Scott and N. S. Raju (2003). The Human ResourcesProgram Evaluation Handbook. Thousand Oaks, CA. Sage.

Elaine, B. (1999). The Business of Consulting: The Basics and Beyond. SanFrancisco, CA. Jossey Bass/Pfeiffer.

Elenkov, D. S., and I. M. Manev (2005). "Top management leadership and influenceon innovation: the role of sociocultural context." Journal of Management 31(3): 381-402.

Hashim, J. (2005). Training Evaluation: Clients' Roles. Journal of European IndustrialTraining. 25(7): 374-379

Henry, J. P. (2001). Creativity and perception in management. London, Sage, inElenkov, D. S. and I. M. Manev (2005). "Top management leadership and influence

17

on innovation: the role of sociocultural context." Journal of Management 31(3): 381-402.

Hillman, A. J. and G. D. Keim (2001). "Shareholder value, stakeholder management,and sosial issues: what's the bottom line?" Strategic Management Journal 22: 125-139.

Holden, D. J. and M. A. Zimmerman (2009). A Practical Guide to Program EvaluationPlanning. California, Sage.

Holton, E. (1995). In search of an integrative model for human resource developmentevaluation. Academy of Human Resource Development, St. Louis, MO , in Rowden,R. W. (2005). "Exploring Methods to Evaluate the Return-on-Investment fromTraining". Business Forum 27(1): 31-37.

Howell, J. M. and B. J. Avolio (1993). "Transformational leadership, transactionalleadership, locus of control, and support for innovations: Key predictors ofconsolidated-business-unit performance." Journal of Applied Psychology 78(6): 891-903, in Elenkov, D. S. and I. M. Manev (2005). "Top management leadership andinfluence on innovation: the role of sociocultural context." Journal of Management31(3): 381-402.

Howell, J. M. and C. A. Higgins (1990). "Champions of technological innovation."Administrative Science Quarterly 35(2): 317-341, in Elenkov, D. S. and I. M. Manev(2005). "Top management leadership and influence on innovation: the role ofsociocultural context." Journal of Management 31(3): 381-402.

IOMA (2004). Training ROI: Sovereign Bank's 8-step process for predicting the ROIof training. Managing Training Development: 1, 7, 10-11.

Janesick, V. (1998) "Stretching" exercises for qualitative researchers. ThousandOaks, CA. Sage.

Johnson, J. M. Catania, D. Freitag, M. Gmach, B. Hamstra and F. Williamson (2005)."Evaluating Field Development Activities: A Practical Road Map to ROI". LIMRA'sMarketfacts Quarterly 24 (1): 30-33.

Johnson, B. and A. Onwuegbuzie (2004) “Mixed Methods Research: A ResearchParadigm Whose Time Has Come”. Educational Researcher, 33(7): 14-26.

Kearns, P. (2005). Evaluating the ROI from Learning. How to Develop Value-basedTraining. London, Chartered Institute of Personnel and Development.

Kearns, P. and T. Miller (1997). Measuring the impact of training and development onthe bottom line. FT Management Briefings. Pitman Publishing, London, in Tamkin, P.,J. Yarnall and M. Kerrin (2002). Kirkpatrick and Beyond: A Review of Models ofTraining Evaluation. Brighton, The Institute for Employment Studies.

Kaufman, R., J. M. Keller and R. Watkins (1995). "What works and what doesn't:evaluation beyond Kirkpatrick." Performance and Instruction 35(2): 8-12, in Tamkin,

18

P., J. Yarnall and M. Kerrin (2002). Kirkpatrick and Beyond: A Review of Models ofTraining Evaluation. Brighton, The Institute for Employment Studies.

Khazanah Nasional Berhad (2008). "Khazanah Nasional Berhad." [online] KualaLumpur, Khazanah Nasional Berhad. Available from www.khazanah.com.my/faq.htm[Accessed 27/06/08].

Kirkpatrick, D. L. (1996). "Great ideas revisited: revisiting Kirkpatrick's four-levelmodel." Training and Development 50(1): 54-57, in Tamkin, P., J. Yarnall and M.Kerrin (2002). Kirkpatrick and Beyond: A Review of Models of Training Evaluation.Brighton, The Institute for Employment Studies.

Kirkpatrick, D. L. (1998). Evaluating Training Programs. San Francisco, CA, Berrett-Koehler, in Burkett, H. (2005). "ROI on a shoestring: strategies for a resource-constrained environments: Measuring more with less (Part 1)". Industrial andCommercial Training 37(1): 10-17.

Kirwan, C. and D. Birchall (2006). "Transfer of learning from managementdevelopment programmes: testing the Holton model." International Journal ofTraining and Development 10(4): 252-268.

Kraiger, K. (2002). "Decision-based Evaluation", in Kraieger, K. Creating,Implementing and Managing Effective Training and Development. San Francisco,CA. Jossey-Bass.

Lewis, P. and A. Thornhill. (1994). "The evaluation of training: an organisationalculture approach." Journal of European Industrial Training 18(8): 25-32

Longenecker, C. O., J. L. Simonetti and D. LaHote (1998). "Increasing the ROI onmanagement education efforts." Career Development International 3(4): 154-160.

May, G. L., J. J. Sherlock and C. K. Mabry (2003). "The future: the driver forshareholder value and implications or HRD." Advances in Developing HumanResources 5(3): 321-338.

Molenda, M., J. Pershing, et al. (1996). "Designing instructional systems." in Craig, R.(ed.), The ASTD Training and Development Handbook: a Guide to Human ResourceDevelopment (4th edn), New York, McGraw Hill, in Tamkin, P., J. Yarnall and M.Kerrin (2002). Kirkpatrick and Beyond: A Review of Models of Training Evaluation.Brighton, The Institute for Employment Studies.

Odusami, K.T., R. R. O. Ijagba and M. M. Omirin (2003). The Relationship betweenLeadership, Team Composition and Construction Project Performance in Nigeria".International Journal of Project Management. 21: 519-527.

Parker-Wilkins, V. (2006). "Business impact of executive coaching: demonstratingmonetary value." Industry and Commercial Training 38(3): 122-127.

Parry, S. B. (1996). "Measuring training's ROI." Training and Development 50(5): 72-77.

19

Phillips, J. (1994). ROI: The Search for Best Practice, American Society for Trainingand Development, in Tamkin, P., J. Yarnall and M. Kerrin (2002). Kirkpatrick andBeyond: A Review of Models of Training Evaluation. Brighton, The Institute forEmployment Studies.

Phillips, J. and E. Holton (1995). In Action: Measuring Return on Investment, PLACE,American Society for Training and Development.

Phillips, J. J. (1997). Handbook of Training Evaluation and Measurement Methods.Houston, Gulf Publishing.

Phillips, J. J. and P. P. Phillips (2002). "11 reasons why training & developmentfails...and what you can do about it." Training 39(9): 78-85.

Posavac, E. J. and R. G. Carey (2003). Program Evaluation: Methods and CaseStudies. London, Pearson Education.

Putrajaya Committee (2005). Catalysing GLC Transformation To Advance Malaysia'sDevelopment. Section II - Policy Guideline. [Online] Kuala Lumpur, PutrajayaCommittee. Available from www.pcg.gov.my/pdf/2.%20Section%20II.pdf [ Accessed02/10/08], 2008

Potter, M. A., C. E. Ley, C. I. Fertman, M. M. Eggleston and D. Senoi (2003)."Evaluating Workforce Development: Perspectives, Processes, and LessonsLearned." Journal of Public Health Management Practice 9(6): 489-495.

Reeve, J. and D. Peerbhoy (2007). "Evaluating the evaluation: understanding theutility and limitations of evaluation as a tool for organisational learning." HealthEducational Journal 66(2): 120-131.

Remenyi, D., B. Williams, A. Money and E. Swartz (1998). Doing Research inBusiness and Management: An Introduction to Process and Method. London. Sage

Remenyi, D. and M. Sherwood-Smith (1999). "Maximise information systems valueby continuous participative evaluation." Logistics Information Management 12(1/2):14-31.

Rossman, G.B. and B.L.Wilson (1991) “Numbers and words revisited: beingshamelessly eclectic ", Evaluation Review 9(5): 627-43.

Rowden, R. W. (2005). "Exploring Methods to Evaluate the Return-on-Investmentfrom Training". Business Forum 27(1): 31-37.

Sastry, M. A. (1999). "Managing strategic innovation and change." AdministrativeScience Quarterly 44(2): 420-422, in Elenkov, D. S. and I. M. Manev (2005). "Topmanagement leadership and influence on innovation: the role of socioculturalcontext." Journal of Management 31(3): 381-402.

Shenkman, M. and B. Gorbaty (2007). "Your highest-yielding (under-capitalised)asset: the ROI of leadership." Business Strategy Series 8(4): 277-282.

20

Smith, I. W. (2004). "Continuing Professional Development and Workplace Learning8: Human resource development: the Return-on-investment". Library Management25(3): 232-234.

Stufflebeam, D. L. and A. J. Shinkfields (1985). Systematic Evaluation. New York,KluwerNijhoff Publishing

Tamkin, P., J. Yarnall and M. Kerrin (2002). Kirkpatrick and Beyond: A Review ofModels of Training Evaluation. Brighton, The Institute for Employment Studies.

Tucker, R. L., R. W. Glover, D. W. Long, C. T. Haas and C. Alemany (1999). Return-on-Investment (ROI) analysis of education and training in the construction industry.Center for Construction Industry Studies. Texas, University of Texas Austin. 6.

Tyler, R. W. (1942). "General statement of evaluation." Journal of EducationResearch 35: 492-501, in Stufflebeam, D. L. and A. J. Shinkfields (1985). SystematicEvaluation. New York, KluwerNijhoff Publishing

van Brakel, R. (2002). "Why ROI isn't enough." T+D 56(6): 72.

Wang, G. G. and J. Wang (2005). "Human resource development evaluation:emerging market, barriers, and theory building." Advances in Developing HumanResources 7(1): 22-36.

Warr, P. B. and N. Rackham (1970). The Evaluation of Management Training,Gower, in Tamkin, P., J. Yarnall and M. Kerrin (2002). Kirkpatrick and Beyond: AReview of Models of Training Evaluation. Brighton, The Institute for EmploymentStudies.

Weiss, C. H. (1972). Evaluation Research. New Jersey, Prentice-Hall, in McCoy, M.and O. D. W. Hargie (2001). "Evaluating evaluation: implications for assessingquality." International Journal of Health Care Quality Assurance 14(7): 317-327.

West, M. A., C. Borrill, J. Dawson, F. Brodbeck, D. Shapiro and B. Haward (2003)."Leadership clarity and team innovation in health care." Leadership Quarterly 14:393-410, in Elenkov, D. S., and I. M. Manev (2005). "Top management leadershipand influence on innovation: the role of sociocultural context." Journal ofManagement 31(3): 381-402.

Williams, S. D., T. S. Graham and B. Baker (2003). "Evaluating outdoor experientialtraining for leadership and team building." Journal of Management Development22(1): 45-59.

Wright, L. T. and M.Crimp (2000). The Marketing Research Process, London.Prentice Hall.

Wright, P. C. and M. Belcourt (1995). "Costing training activity - a decision maker'sdilemma." Management Decision 33(2): 5-15.

Yin, R. K. (1994). Case Study Approach: Design and Methods. Thousand Oaks. CA,Sage.

21

.