IBM and Acrolinx Analytics: Driving Process and Quality Improvements across the Enterprise

Michelle CareySeptember 2014

2

Menu

Challenges to improving quality across a vast enterprise

Corporate goals Current quality metrics Process and quality improvements from analytics Deploying analytics across the enterprise Requirements (of course!)

3

Approximately 1500 professional writers who create product info; also nearly 4000 total Acrolinx users

Publishing billions of words annually

Enabled for translation into dozens of languages

Deliv 1

Topic 1

Topic 4

Deliv 2

Topic 2

Topic 3

Managing more than 2 million English DITA topics and millions more of other types of topics (marketing, support, etc.)

IBM is pretty big…

4

Challenges of the large, distributed enterprise

Driving consistent client information experience across large, far-flung organizations is a monumental task.

IBM is no exception: thousands of products and thousands of content authors

5

Enterprise goals for content

Improve content quality not just for information development (ID)– Marketing– Sales– Support– Learning/training

Prove that quality content brings in revenue– Really tough question to answer– IBM marketing plans to implement a strategy to

measure this

6

Quality metrics we use today

7

Traditional metrics for assessing content quality

Content quality is difficult to assess, but today we do look at these metrics:– Translation– Support calls– Customer satisfaction/sentiment analysis– Acrolinx score cards, readability/voice– Editing for Quality (EFQ)

8

EFQ

We also measure quality against characteristics as defined in the book Developing Quality Technical Information.– Shameless plug: Get the 3rd edition today—new for

2014! Nine characteristics of content quality, such as

accuracy, clarity, retrievability EFQ is a very thorough and time-consuming edit that

we do on selected content.

9

Process and quality improvementsHow analytics reporting can help

10

Content authors

Authoring– Fix the worst problems first– Learn from mistakes– Understand the value of global English, cost savings

Targeted education– Editors and architects can provide the right education at

the right time to address problems

11

Team resources

Managers can see who is or isn’t using the tool– Take actions to discover why folks aren’t using the tool– Encourage folks by showing cost savings

Documentation team leads can target and budget for quality improvement work– This is especially crucial in resource-constrained and

continuous delivery environment

12

Spreading the word

Convincing other domains across IBM that Acrolinx is worth using.– Get more authors on board, achieve greater consistency

and better quality across more domains– Show cost savings

13

Rule and terminology updates

Updating terminology– For example, “Daeja” appears as a spelling error, but

this is a product name. We need to adjust the terminology database.

Updating rules– Analyze reports on rules to find rules that are

overflagging, are too restrictive, and rules that aren’t flagged at all, which could indicate a problem

14

Internal processes

Upgrading servers and lingware– Determine optimal times for doing upgrades,

maintenance, etc.

Ensuring that checking metrics metadata is correct

15

Deploying analytics and reporting across the enterprise

16

Deploying analytics

Phase 1: Configure custom fields and cost savings values

17

Mapping quality metrics to $ value

Most rule and terminology violations have a potential cost savings and avoidance for translation or mechanical editing or both.

We determined our estimates by working with our translation coordinators and content quality experts.

18

Sample of cost savings per rule or term

19

Deploying analytics

Phase 1: Configure Phase 2: Beta testing Phase 3: Announce and delegate Phase 4: Maintain

20

Beta testing results: early feedback

21

Requirements

Need report generator IDs added to checking reports so that we can find them in the history.

Quality improvement by document is nice—but would be nice to have the user ID linked to each doc.

Create Flags by document—is there a way to group the rules? Why do I want to look at one rule at a time? What I really want to know is what are we worst at—so where is the biggest opportunity for improvement.

Average checking score: I don't understand this graph. Average of what? I get 1 number, (331) which must be the average but the Y axis only goes to 335 and there is nothing plotted on it. How are you to derive a measure of quality?

22

More requirements

Under View Scorecards. There are 1185073 items found. The report shows 10 at a time. There is no filter, no way to sort outside of check date. How do I find what I want here?

Export: IBM doesn’t have Excel across the board; should say generic spreadsheet

Generation of user ID lists categorized by user metadata. For example, I need a list of user IDs by departments, or by business unit, or by client type, etc.

Need to create a profile of reports to run such that we can request say, multiple reports at once.

The other huge thing missing is the ability to automatically schedule and send reports to user email IDs.

23

Post-beta

Starting in early 2015, we’ll announce the reporting feature to all users.

We’ll also update our metadata – Problem: keeping the metadata current when so many

users change jobs, products, departments, or divisions– Need an easy way to sync these users with their work

status

24