The Growth of Learning Analyticslearn.trustvip.com › rs › visionary › images › BHG_Growth in Learning … · The Growth of Learning Analytics Introduction We are in the midst

Stacey Harris, VP of Research David Grebow, Principal Learning Analyst July 2013

Research Brief

The Growth of

Learning Analytics

Sponsored By: Meridian Knowledge Solutions, LLC

© 2013 Brandon Hall Group. Licensed for distribution by Meridian Knowledge Solutions, LLC. Page 2

The Growth of Learning Analytics

Research Brief


By:

Stacey Harris, VP of Research David Grebow, Principal Learning AnalystJuly 2013



TABLE OF CONTENTS INTRODUCTION 4

THE CHANGING ROLE OF THE LMS 4

MODERN DEFINITION OF LEARNING ANALYTICS 5

LEARNING DATA 6

REPORTING 7

CONTINUOUS ANALYSIS 8

ASSESSING THE NEED FOR LEARNING ANALYTICS 10

STAGE 1: TMS TO LMS: ADMINISTRATOR FOCUSED 11

STAGE 2: MORE DATA MEANS MORE ANSWERS: LEARNING-MANAGER FOCUSED 11

STAGE 3: GRAPHICAL TREATMENT OF DATA: LEARNER FOCUSED 12

STAGE 4: DYNAMIC REPORTS: LINE MANAGER- AND BUSINESS-MANAGER FOCUSED 13

STAGE 5: TO BIG DATA AND BEYOND: BUSINESS FOCUSED 14

KEY PRACTICES IN IMPLEMENTING LEARNING ANALYTICS 16

SUMMARY 17

ABOUT BRANDON HALL GROUP 18

THE VALUE OF MEMBERSHIP 18

ABOUT MERIDIAN KNOWLEDGE SOLUTIONS 19



Introduction

We are in the midst of a revolution – a data revolution. Since the dawn of the computer era, our businesses, organizations, and personal lives have been increasingly directed by data. The Growth of Learning Analytics is a story about the increasing need to leverage that data to drive improved organizational outcomes.

This data revolution is a result of several forces colliding in today’s business environment:

• The growing interest in leveraging learning as a tool to engage both internal and external audiences.

• The growing demand to leverage learning data to inform organizations’ critical business decisions concerning talent.

• The recent attention given to ensuring that today’s learning functions are so effective that they become a competitive advantage for their organizations.

Brandon Hall Group has recently completed several research efforts on the evolution of learning analytics, and how that growth has been affected by both technology advancements and industry demand for change. In this paper we will look at:

• The changing role of the LMS and technology in learning analytics.

• The modern definition of learning analytics.

• Assessing the need for learning analytics and big data.

• Key practices in implementing learning analytics.

The Changing Role of the LMS

As a business system, the LMS has arrived. It is a necessary tool for any organization that manages the compliance training and development needs of more than a few hundred employees. We are also seeing growth in its reach to extended learning audiences, when education and training are required for partners, vendors, or even direct to consumers. In a recent survey by Brandon Hall Group, 86% of organizations said that they already had an LMS. The remaining 14% responded that they do not have, or are still considering, an LMS. The majority of respondents indicated they are moving toward an enterprise-wide LMS.

Regardless of the growing popularity and business necessity of the LMS, more than 43% of respondents in the same survey who currently have an LMS were also looking to upgrade or replace their current system. The reasons for wanting to replace an LMS are seen in Figure 1 and include reporting, system integration issues, and end-user ease of use.



Figure 1: Three Most Important New LMS Requirements

Source: Brandon Hall Group, 20121

The most frequent complaints organizations convey about their LMS are the availability and usefulness of reports2. Many organizations overlook the importance of reporting features during the selection process, but within a year it becomes a major concern for learning and business leaders alike. Brandon Hall Group’s research found that organizations tend to struggle with reporting features in one of three ways:

• The system only produces canned reports, providing no flexibility for altering fields or changing the layout of the report.

• The system does not allow reporting of data in enough detail to suit learning and business leaders.

• The system does not have analytics features, and therefore does not allow the learning function to easily compare learning data to business data.

Modern Definition of Learning Analytics

Learning analytics is the continuous analysis and reporting of connections and insights that can be gathered from comparing learning, people, and business data for the purpose of making better business decisions. It is a process that begins simply with data – and evolves into a business tool that can be leveraged for critical decisions.

1 Brandon Hall Group, LMS Trends Survey, November 2012 N=3472 The Race to Replace, Results from the 2012 LMS Trends Survey, Brandon Hall Group, David Wentworth, December 2012



Let’s discuss each element of this definition:

Learning Data

Any field captured within a learning technology tool can be considered learning data. We quickly think of things like learning completion records, or skills gaps – but information such as learner demographics, future job roles, content access points, and location of content delivery is also part of the learning data landscape. It is important to expand the definition of learning data, especially if an organization plans to analyze the information at a later date.

Employee profiles within the LMS are generally seen as the central location for learning data, but they may not include everything that an organization needs to answer important questions. Many organizations need to go beyond the employee profile, to data that is captured at the content level or even the learning access point, as seen in Figure 2.

Figure 2: Example of Learning Data on Tools Used to Access Learning

Source: Meridian Knowledge Solutions



No matter the level of data that is required, or accessible, an important principle in all reporting and data management efforts is a simple rule that “garbage in will produce garbage out.” How data is entered into the system is as important as what data is accessible. When documenting important learning data, the organization should also document where the original information comes from and the level of reliability of that information. For example, if the information is coming from multiple ERP or HRIS feeds, an organization needs to make sure that the definition of full-time employee or the role of manager is the same across all the systems that are feeding the data into the learning platform.

Reporting

A reporting tool within a learning platform is really the engine by which an organization can produce various views of the data entered into the system. There are several levels of reporting tools in the market that range from simple tools that produce only canned flat files to those that offer full-blown dynamic dashboard reports.

The simplest tools often only focus on single-dimension reporting – giving only one view of data in a snapshot of time.

Figure 3: Single Dimension Reporting

Source: Brandon Hall Group

Data-driven decisions are really the best way to go – but sometimes a single dimension view can give us a warped perspective on a very interconnected environment. These are reports that are often pulled in response to questions such as:

• How much are we spending on learning?

• Who completed their compliance training?

• What is the most popular course?

An organization may decide to cut expenditures due to this type of reporting, or remove a course due to lack of popularity, without looking at other factors such as the increase in new products and services which may require more learning, or the fact that the least popular course has the greatest business outcomes.



The best reporting tools allow organizations to ask a question and obtain a full spectrum of data points, plus also allow the organization to represent the information in a way that makes the most sense for their stakeholders. For example, a regional delivery location report could be provided as a list of locations with country codes – or it could be represented as a visual map that allows stakeholders to quickly see their delivery regions in a single snapshot.

Figure 4: Example of Visual Options for Reporting


Continuous Analysis

It is important to think as broadly as possible when looking at learning analytics and the data that will drive the reports. The real value of the analytics occurs when you begin to combine data points. Even if they look disparate when viewed individually, taken together and crunched by the algorithms that enable the various requests to be answered, they may add up to beneficial answers for the organization.

For example, an employee’s completion record by itself is valuable data. However, when combined with geographic location or increasing competency and proficiency levels over time, there may be a wider picture.

What learning “analytics” tools can provide when used effectively is a better way to contextualize



the information you are pulling from all of your learning technology and possibly business technologies as well. As seen in Figure 5 below, learning analytics is the continuous analysis of connections and insights that can be gathered from comparing learning, people, and business data for the purposes of making better business decisions. In simplest terms, learning analytics provides a multi-dimensional view of your data. It also provides tools to allow an organization to input business logic and insights, which can be used during data analysis efforts.

Figure 5: What are “Learning Analytics”?


The most important element of this model is the analysis/insights work. Gathering and thinking about more data can seem overwhelming, but by leveraging some pre-defined algorithms, some basic statistics tools and pattern matching, and combining that with business insights, you can create more contextualized business decisions.

Another critical element in this definition is the term “continuous.” Analytics tools should also provide a view of each data point over time – showing how data is trending up or down from a single point.

Figure 6: Example of Analytics with Trending Views


Workforce Data Goals Data

Environment Data

Business Data

Business Outcomes

Business Decisions

Analysis/Insights

Open Requisitions

Performance Goals

Financial Improvements

Engagement Scores



Every time a decision is made, that information needs to feed back into the analysis to allow the tool to adjust for changes in decisions. For example, an organization may see that employee engagement is down in a certain region and also that the three managers in those regions never completed a required coaching and mentoring program. The organization may make an adjustment to provide the program for the managers and then re-assess engagement levels the following year, but it is critical that the new information is now included in that analysis.

Assessing the Need for Learning Analytics

Figure 7: The Evolution of Learning Reporting


Learning analytics is a relatively recent development for the LMS, and the need for more complex reporting tools evolved over time. As mentioned earlier, most organizations need to work their way up to leveraging learning analytics, and beyond that, predictive analytics and big data analysis. The first steps start with ensuring data is clean, the basic reporting needs are met, and audiences are well defined. There are five identifiable stages our research has shown in the learning reporting market. Most organizations evolve through them over time.



Stage 1: TMS to LMS: Administrator Focused

When the Training Management Systems became the early Learning Management Systems, the primary person needing reports was the training administrator. A “flat file” is one of the earliest types of reports the LMS could generate. The flat file was a simple list of names and course completion dates for each individual. It was useful for tracking the people who took and completed a course. There was no need for any analysis other than the number of people who completed a prescribed course. The problem with a simple flat file quickly became apparent. As organizations grew and the numbers of people and courses increased, flat files became too long and unmanageable. Reporting needed to move to the next level.

Stage 2: More Data Means More Answers: Learning-Manager Focused

As the size of the learner base and learning function responsibilities increased, learning managers realized there was a more comprehensive picture of learning that could be developed from the reporting data. Course completions by person no longer provided enough value when it came to planning. Exportable data files replaced flat files and enabled learning managers to query, analyze and answer, for example, the following questions inside third-party analytics tools:

One Company’s Analytics Journey

A large global telecommunications organization started the journey toward advanced learning analytics, like most organizations, with a few critical business questions concerning learning transactions and the implementation of a new LMS from Meridian Knowledge Solutions. Four years later — and a few hundred reports down the road — the organization manages learning data for 100,000 employees and customers on an annual basis.

For this organization, the critical path to success started with leveraging Meridian’s out-of-the-box reports, then advancing to the ad hoc reporting tools, and finally moving into advanced analytics and enterprise dashboards with the support of the Meridian system. “As our learning organization matured, it required more data to drive it, to support goals in both improved quality and efficiencies,” a company learning leader shared. “The ad hoc reporting tools allowed our organization to pull customized fields, troubleshoot data coming from outside the LMS, create new report structures, and provide only the data business leaders were requesting in an easy to understand format.” Now the company is shifting its focus to clean up its data at much deeper levels, and perform deeper learning analytics to compare data to business outcomes and measurement goals.



• How many people out of a total (say 10,000) did not complete a course?

• What was the increase in course completions year over year?

• How do the registration rates compare to the completion rates?

• How many people started and completed the surveys on courses and programs?

Learning managers needed very simple analytics to make their decisions and to support those decisions when presenting them to senior leaders. More data needed to be collected and analyzed, and the exportable data files made that possible. They enabled data to be output into an Excel spreadsheet or similar format and then turned into charts and graphs and printed reports. Organizations often leveraged standard report templates to share information. With regard to learning analytics, this can be a good start for many.

Stage 3: Graphical Treatment of Data: Learner Focused

As the analysis became more complex, interpreting the analysis needed to become easier. No longer could a Learning Manager handle interpreting all the learning data alone; now learners and business divisions themselves needed to be able to access reports and data. The result was the graphical representation of the data in dashboards and other dynamic interfaces.

The dashboard could quickly show personal training records, course completion rates, gaps in personal certificates or certifications, levels of competency, mapping skills levels to job descriptions and more. The deeper levels of data and the resulting analysis now required a faster way of find answers.

Figure 8: Visual Employee Reports and Dashboard Tools

Source: aŜǊƛŘƛŀƴ YƴƻǿƭŜŘƎŜ {ƻƭǳǘƛƻƴǎ



Learning data were now in the hands of everyone with access to the LMS, so the tools could begin to shift from an administrative platform to a career development and business tool.

Stage 4: Dynamic Reports: Line Manager- and Business-Manager Focused

Up to this point, data analysis could usually be done on a monthly or quarterly basis. Updating data on a nightly basis can often suffice for many organizations. But as line managers and business leaders began to use learning data to make daily decisions, they needed to speed up the analysis and reporting provided by the LMS.

The result is more dynamic reports and ad hoc reporting tools that can update, analyze, and report information whenever needed. These dynamic reports allow a manager to request a weekly review of a team’s learning, a tally on a Wednesday about a group- level certification, or a same-day report on an employee for a performance review. Not only has the speed of the requests and reports increased, the level of data that is being analyzed has increased to cover the entire lifecycle of employees’ learning and achievement — from the day they start, through all their promotions, until the day they depart.

Ad hoc reporting tools give line managers, business leaders, and learning leaders the ability to organize the information and compare information in ways that make the most sense for them or their stakeholders.

Figure 9: Example of Ad Hoc Report Builders




Real-time data is more than simply an advance in analytics; it answers a real need. For example, the organizations in certain industries lose millions of dollars every year as a result of inaccurate audits. These companies need up-to-the-minute reporting on compliance data to be able to meet the standards set by the federal, state and industry auditing bodies.

Highly regulated organizations -- especially those where unannounced spot checks take place -- must be able to prove at any point in time that they are in complete compliance. Real-time questions and answers that produce a solid report are the only way the people in charge of a mining operation, manufacturing line, food production plant or a nuclear power generating facility can prove they are in compliance and can continue to operate.

Stage 5: To Big Data and Beyond: Business Focused

Learning has become a mission-critical business metric that can often show how “adaptable” an organization is by looking at the overall level of skills and qualifications. This “big picture” requires the input of big data that is much more informative than the previous levels of analytics.

Big data is a collection of data sets so large and complex that it is difficult to process or analyze using traditional database management tools. It has data characteristics that need to be managed, such as volume, velocity, variety, veracity, variability, and complexity3.

Big data for learning incorporates every data point across the organization including the following:

• Demographics

• Feedback

• Course starts / course completions

• Test results

• Skills levels

• Performance reviews

• Course access points

• Time on system

• Clicks and scrolling

Big data incorporates almost everything that can be considered a data point and can be used as part of an analysis of usage or results or a learning course or program. The challenges with big data often include capture, curation, storage, search, sharing, analysis, and visualization. We begin to see new visualization tools that allow us to make sense of large amounts of data. One example of this is a heat map, a way of actually watching the data change in real time, moving from red through yellow to green.

3 Brandon Hall Group, Learning Meets Big Data, 2013



Figure 10: Big Data Reports: Visualized Heat Maps


Big data analytics provide answers to a new set of learning-related questions that many organizations would have never thought to ask, such as:

• Is there a connection between when and where course content is accessed and the level of employee engagement or performance?

• What courses or certifications are correlated with improved performance in a specific division?

• Which programs resulted in the greatest measured improvements in productivity, by employee role?

• When we see audit support reports on certifications, how much time do we have before we begin to see problems?

• With the number of new employees, what does the compliance gap look like across all the regulatory courses?

These are very different questions and answers that can impact business decisions across the entire enterprise.

The evolution of learning reporting and analytics has moved from the individual student to the entire organization, from questions about a person’s course completion to analyzing the level of readiness of an entire company. Every organization needs to assess their own readiness for diving into deep levels of learning analytics, but a good foundation can be built on solid data capture practices and by working on reporting standards that meet the needs of particular audiences.

Mid-Atlantic Regional

Information Service

Training & Support

IT Systems

Special Functions

ResidentsAdministrative

Non Exempt Employees

Professional Services

Northeast Regional

Technology Support

Restaurants

Client NetworkSecurity ServicesRegion 5:

Mid-Atlantic North America

Southwest Regional

YPG Internal

MembersMechanical

Support Operations

Leadership & Management Corporate

Legal



Key Practices in Implementing Learning Analytics

Learning analytics have been proven to be valuable tools and, like any tool, require a certain level of skill and understanding to be properly used. Here are some of the best practices we discovered during our research:

• Educate and train everyone who needs reports on how to use your reporting capabilities and tools. Organizations too often buy LMS systems with great capabilities and they fail to use them effectively to get the best data in and out.

• Organizations that have the most success with data analytics clearly define data input requirements and parameters for all stakeholders. Data must be the same across the board. Even something as simple as the difference in a single letter of an employee code can result in what is called skewed data.

• Leverage “self-service” segmentations and drill downs. Organizations that enable business leaders’ easy access to their data, especially when it can be shown in visual formats and drilled into for greater detail, find greater buy-in for the learning initiatives.

• Data can be more than names and numbers. Organizations that get the most out of their learning analytics also have the ability to include use-case studies as part of their datasets. The resulting picture includes real stories about usage to support the more objective data.

• The best learning analytics tools have reports and data views that are user-friendly and provide opportunities to share the data for further analysis and insights.

• Anyone using the LMS with the new data analytic capabilities needs to understand the nuances and opportunities posed by data warehousing. Big data means you will be pulling data from the LMS and many of the other enterprise software systems (e.g., HRIS) in the organization. It is up to everyone, not only people focused on learning, to maintain the integrity of the data.

• Scheduling can become complicated with access to real-time data. To keep the cost of operations down, system users need to make a clear distinction between reports that can be scheduled (e.g., weekly or monthly) and reports that need to be available on an ‘as requested’ basis. The question is, “What is the ROI for each report?” The answer is not, “let’s run this on a daily basis now because we can.”



Summary

The desire to embed learning into how our businesses and organizations grow and expand, creating real competitive advantage from our talent, continues to drive the need for more relevant and actionable data. The learning industry as a whole is requiring better reporting and analytical capabilities than ever before from our learning technology. The promise of cleaner, more accurate data – leveraged into useful reports, dashboards, statistics and big data analysis — is a reality. It can happen with a little planning and upfront work, and flexible technology.

Report summary:

• More than 40% of current LMS users are looking to replace their LMS, many due to their poor reporting capabilities.

• Learning analytics are the continuous analysis and reporting of connections and insights that can be gathered from comparing learning, people, and business data for the purpose of making better business decisions.

• Expand the view of learning data and reporting capabilities.

• Garbage in = garbage out; focus on clean data.

• Ensure your organization is ready to leap into full learning analytics; take one stage at a time.

• Trends in reporting and analytics include dynamic tools, analytics capabilities, and big data views.



About Brandon Hall Group

With more than 10,000 clients globally and 20 years of delivering world class research and advisory services, Brandon Hall Group is the most established and well-known research organization in the performance improvement industry. We conduct research that drives performance and provides strategic insight for executives and practitioners responsible for growth and business results.

Brandon Hall Group has an extensive repository of thought leadership research and expertise in our primary research areas — Learning and Development, Talent Management, Sales Effectiveness, Marketing Impact, and Executive Management.

At the core of our offerings is a Membership Program that combines research, benchmarking, and unlimited access to data and analysts. Our members have access to research and connections that help them make the right decisions about people, processes, and systems, complemented with analyst advisory services tailored to help put the research into daily action.

The Value of Membership

The Brandon Hall Group Membership Program encompasses comprehensive research resources and an array of advisory services. Our Membership Program provides:

• Cutting-Edge Information – Our rigorous approach for conducting research is constantly evolving and up-to-date, providing your organization with current and future trends, as well as practical research insights.

• Actionable Research – Your membership includes advisory services and tools that are research-driven and provide you a breakthrough approach to addressing immediate challenges and opportunities inside your organization.

• Customizable Support – Whether you are an executive or entry-level practitioner, our research and analyst insights can be leveraged at an individual level and across the entire organization. We realize that every organization has unique needs, so we provide multiple analyst and research access points.

• Community of Peers – We realize the value of connecting with your peers and being part of a community that is focused on continuous improvement. Your membership provides you with personal connections to fellow professionals.

• Unlimited Access – Every member of your team has the ability to utilize research, best practices, and advisory services when they need it most.

To learn more about Brandon Hall Group, please call us at (561) 865-5017 or email us at [email protected].

mailto:[email protected]



About Meridian Knowledge Solutions

What is your next step with learning? Meridian Knowledge Solutions helps you create measurable value with your learning strategy. Meridian provides technology platforms that empower enterprises, governments, and member-based organizations to develop their people by delivering learning, assessing performance, and fostering collaboration. We’ve been making customers our partners since 1997.

Our suite can be deployed on-premise, hosted, or in a secure cloud. Innovative design promotes configuration to business processes, while reducing costly implementations. Meridian helps extend learning beyond employees to the value chain, including partners, members and customers. Mobile capabilities enable on-the-job-training, while social sharing builds the learning community.

meridianksi.com

http://www.meridianksi.com

Documents

The Growth of Learning Analyticslearn.trustvip.com › rs › visionary › images › BHG_Growth in Learning … · The Growth of Learning Analytics Introduction We are in the midst