Upload
others
View
27
Download
0
Embed Size (px)
Citation preview
Qlik Sense Internship Report
i
Qlik Sense Implementation
Juan Manuel López Mora
Dashboard creation and implementation of the test
performance Methodology
Internship report presented as partial requirement for
obtaining the Master’s degree in Information Management
ii
NOVA Information Management School
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa
QLIIK SENSE IMPLEMENTATION
Dashboard creation and implementation of the test performance Methodology
Juan Manuel Lopez Mora
Internship Report presented as the partial requirement for obtaining a Master's degree in
Information Management, Specialization in Knowledge Management and Business Intelligence
Advisor / Co‐supervisor: Miguel de Castro Simões Ferreira Neto
May 2020
Qlik Sense Internship Report
iii
ABSTRACT
Strategy, Decision, and Governance SDG Group is a multinational company found in Italy in 1991,
with offices in Portugal, it has experienced a high and consistent growth based on the performance
and quality of their projects, that tailor the client necessities. It has been award as Qlik partner of the
year for Europe in the last four years.
The project has two parts the first one is to create a Sales Datamart for the managers of the EMEA
Region of a Pharmaceutical company. The dashboard is a web application that uses content from Qlik
Business Intelligence Software and displays it in a graphical web interface.
The second part of the internship is the creation and implementation of the Sense Test Performance
Methodology, Qlik is a software that runs in memory, and the performance is profoundly affected if
proper practices are not applied. In SDG, most experience consultants applied the practices, but
knowledge dissemination is reduced to the teams that work together.
The twenty-five consultants that the company has in Lisbon would be involved in the development of
the Qlik Sense Test Performance Methodology. Also, a report delivers to the client with the resultant
performance of the dashboard.
KEYWORDS
Business Intelligence; DataMart; Qlik Sense; Good Practices; Performance
Qlik Sense Internship Report
iv
INDEX
1. Introduction .................................................................................................................. 1
1.1. Company Description ............................................................................................ 2
1.2. Problem Definition and Motivation ...................................................................... 2
1.3. Role in the Project ................................................................................................. 2
1.4. Study Goals ............................................................................................................ 3
2. Literature review .......................................................................................................... 4
2.1. Business Intelligence Software .............................................................................. 4
2.2. Data Warehouse .................................................................................................... 5
2.3. Data mart ............................................................................................................... 7
2.4. Schemas ................................................................................................................. 7
2.5. Dashboard ............................................................................................................. 8
2.5.1. KPI ................................................................................................................... 8
2.5.2. Types of Performance Dashboards ................................................................ 9
2.5.3. Design of the Dashboard .............................................................................. 10
2.6. Change Management .......................................................................................... 10
2.7. Memory Performance ......................................................................................... 11
3. PROJECT METHODOLOGY ........................................................................................... 12
3.1. Schedule .............................................................................................................. 13
3.1.1. Kick OFF ........................................................................................................ 13
3.1.2. Qlik Training .................................................................................................. 13
3.1.3. Web Dashboard ............................................................................................ 14
3.1.4. Qlik Test Performance Methodology ........................................................... 14
3.1.5. Deliver the project ........................................................................................ 14
4. WEB DASHBOARd ....................................................................................................... 15
4.1. Architecture for the projects and folder Structure ............................................. 15
4.1.1. Extraction...................................................................................................... 16
4.1.2. Transform ..................................................................................................... 17
4.1.3. Load .............................................................................................................. 17
4.1.4. Configuration Files ........................................................................................ 18
Qlik Sense Internship Report
v
4.2. The Data Model ................................................................................................... 19
4.2.1. Facts Data ..................................................................................................... 20
4.2.2. Master Data .................................................................................................. 20
4.3. Application ........................................................................................................... 21
4.3.1. KPI ................................................................................................................. 23
4.3.2. Dashboard .................................................................................................... 24
4.3.3. Geography .................................................................................................... 24
4.3.4. Therapeutic Area .......................................................................................... 25
4.4. Test ...................................................................................................................... 26
5. QLIK TEST Performance Methodology ....................................................................... 28
5.1. Design of the Methodology ................................................................................. 28
5.1.1. Excel Checklist .............................................................................................. 29
5.1.2. Document Analyzer ...................................................................................... 30
5.1.3. Scalability Tools ............................................................................................ 31
5.1.4. Application Performance Presentation ........................................................ 32
5.1.5. Results in an example App ........................................................................... 33
5.1.6. Flags .............................................................................................................. 33
5.1.7. Improvements .............................................................................................. 33
5.1.8. Qlik Sense Performance Methodology Conclusion ...................................... 34
6. Conclusions ................................................................................................................. 36
7. Limitations and recommendations for future works ................................................. 37
8. Bibliografia .................................................................................................................. 38
Appendix A ...................................................................................................................... 40
QLIK SENSE Terminology ............................................................................................ 40
Qlik View Data QVD. .............................................................................................. 40
Set Analysis. ........................................................................................................... 40
Synthetic Keys in the Data Model .......................................................................... 40
A.1.4 Link Tables in the Data Model ...................................................................... 41
A.1.5 Concatenate or No Concatenate in the Data Model .................................... 41
Appendix B ...................................................................................................................... 42
Qlik Main Functions .................................................................................................... 42
Mappings ............................................................................................................... 42
Hierarchy ................................................................................................................ 42
Join 42
Appendix C ...................................................................................................................... 44
Qlik Sense Internship Report
vi
The Qlik Server ............................................................................................................ 44
Qlik Management Console .................................................................................... 44
The Hub .................................................................................................................. 45
AppendiX D ...................................................................................................................... 47
The Memory Load In Qlik ........................................................................................... 47
Symbol Table .......................................................................................................... 47
Data Tables ............................................................................................................ 47
APPENDIX E ..................................................................................................................... 48
Qlik Checks .................................................................................................................. 48
Qlik Sense Internship Report
vii
LIST OF FIGURES
Figure 2.1 – Garner Quadrant, Source (Gartner, 2018). ............................................................ 5
Figure 2.2 – Immon DW model, Source (Panoply, 2019). ......................................................... 6
Figure 2.3 – Kimball, Source (Panoply, 2019) ............................................................................ 7
Figure 2.4 – Types of Dashboard, Source (Eckerson,2009) ........................................................ 9
Figure 2.5 - Change Management Steps Source (Kotter, 2012) .............................................. 11
Figure 3.1 - SCRUM, SOURCE (Sliger, 2011) ............................................................................. 13
Figure 3.2 - Project Schedule .................................................................................................... 13
Figure 4.1 – Data Architecture ................................................................................................. 16
Figure 4.2 - Folder Organization ............................................................................................... 16
Figure 4.3 - Metadata Table ..................................................................................................... 18
Figure 4.4 - Data Model ............................................................................................................ 20
Figure 4.5 - Granularity ............................................................................................................ 21
Figure 4.6 - Landing Page ......................................................................................................... 22
Figure 4.7 - Horizontal Menu ................................................................................................... 22
Figure 4.8 - Vertical Menu ........................................................................................................ 23
Figure 4.9 - KPI example ........................................................................................................... 24
Figure 4.10 - Geography Dashboard ........................................................................................ 25
Figure 4.11 - Therapeutic area ................................................................................................. 26
Figure 4.12 Data Robot Log ...................................................................................................... 27
Figure 5.1 - Check List Panel ..................................................................................................... 29
Figure 5.2 - Document Analyzer ............................................................................................... 30
Figure 5.3 - Optimized variables............................................................................................... 31
Figure A.1 - Synthetic Keys ...................................................................................................... 41
Figure C.1 - Qlik Server QMC. ................................................................................................... 45
Figure C.2 - The hub ................................................................................................................. 46
Figure D.1 - Data tables. Source (Cronström, 2012). ............................................................... 47
Figure E.1 -data check Source Qlik ........................................................................................... 49
Qlik Sense Internship Report
viii
LIST OF TABLES
Table 2.1 – Aspects of a KPI Source (Eckerson, 2009). ............................................................... 9
Table 3.1 - SCRUM parts ........................................................................................................... 12
Table 4.1 - .txt files of a Qlik Sense App .................................................................................. 17
Table 4.2 - Metadata Fields ...................................................................................................... 19
Table 4.3 Benchmark Selection KPI .......................................................................................... 23
Table 4.4 Principal KPI .............................................................................................................. 23
Table 5.1 - Scalability Tools result ............................................................................................ 32
Table 5.2 - Results CIE INDEX App ............................................................................................ 34
Table E.1 - Check Interface Source Qlik.................................................................................... 48
Qlik Sense Internship Report
ix
LIST OF ABBREVIATIONS AND ACRONYMS
Acronym Explanation of acronym
BI Business Intelligence CM Change Management ETL Extraction, Transformation, and Loading KPI Key Performance Indicator QMC Qlik Management Console QS Qlik Sense QVD QlikView Data file SDG Strategy Decision Governance US User Story
Qlik Sense Internship Report
1
1. INTRODUCTION
Business Intelligence (BI) has been a top priority of IT executives for several years, and the market for
related software products continues to proliferate, despite the challenging macroeconomic
conditions (Sommer, 2014). With the incrementation of data and analytics insights, the last phrase
has more relevance. Still, also, the BI software has evolved to improve the performance with big data
and includes analytics.
The Extraction Transform Load ETL process turn raw data into information, Consultant companies
like SDG Group help companies in the whole process through the implementation of Analytic
Software that allows constructing insights from the data, in the process, the consultants give
architecture and analytics solutions.
The projects aim to introduce a Business Intelligence solution by providing a tailored web graphical
interface for a Customer in the Pharmaceutical Sector with Emphasis in the EMEA Region. This
project enables the possibility to examine different support systems for the Organization to manage
data, make decisions, and improve its competitive advantage in the market.
The leading software used during the internship is Qlik that offers governed data discovery and agile
analytics. The main advantage of Qlik is the associative engine that will automatically figure out the
relationship within the data, as it loads all the information in memory, all the information related to
the Qlik Sense software as functions and terminology is explained in the Appendixes.
Besides working on the data model and the dashboards, the Sense Test Performance Methodology
is developed; this is an innovative project for the company; innovation and creativity are depending
on Knowledge Management, which is the master’s degree specialization.
The company have been developed in Qlik for more than ten years and has different Managers and
senior consultants with an extensive Qlik good practice skill, even though this tacit knowledge is not
transmitted to the whole company and is an issue when new consultants start a project, one critical
success factor of any consultant firm is the knowledge transmission between is employees, actually
transforming tacit knowledge into explicit knowledge is the beginning of a knowledge creation
(NONAKA, 2014).
The Sense Test Performance Methodology aims to guarantee the performance of all the Qlik Sense
applications that are developing in-house. The main idea is the creation of a user manual that would
recompile the tacit and non-tacit knowledge of the consultants, includes the most used memory
performance tools on the market, and a final report with the app performance delivers to the client.
As the Sense Test Methodology is related to performance, some results contain RAM and CPU usage,
these concepts are explaining in the literature review chapter. Appendix E would provide all the
information related to Qlik Engine.
Qlik Sense Internship Report
2
1.1. COMPANY DESCRIPTION
SDG Group is a multinational company found in Italy in 1991, with offices in Portugal. The company
has experienced high and consistent growth thanks to the performance and quality of their projects
that adapt to the necessities of the client. It has been awarded as Qlik partner of the year for Europe
in the last four years.
The Client that the author is assigned during the Internship is Janssen that is the pharmaceutical
company of Johnson & Johnson, with over 40,000 employees working on the five continents, around
$4.5 billion in research development annually, and a top company in the pharmaceutical sales.
1.2. PROBLEM DEFINITION AND MOTIVATION
Janssen pharmaceutical is one of the principal clients of the company and needs a web-based
dashboard for the EMEA region. Managers need to assert the right decisions quickly, and for that,
they need clear and accurate information, especially the one related to Sales and Expenses.
Companies need information systems that are easy to use, and they do not have to invest in training
to be used, that the reason to select a web-based dashboard with the necessary KPI, graphics, and
tables. The name of the project is Ensemble and would use Qlik Sense and web front end
technologies like Angular and HTML.
SDG is growing and the sophistication and necessities of the clients that follow the trend of Website
usage. People want websites that load quickly and without any problems. Surveys conducted by
Google indicate that internet users expect websites to load within three to four seconds. If sites take
longer than that, users are more likely to turn towards the competition (Wertz, 2017).
A dashboard that the user must wait for more than five seconds may not be used and would finish in
a waste of resources for the company that is buying the service and to an unsatisfied customer for
SDG. The topic could be solved by implementing good practices and having metrics of the opening
and reload times of the app.
To summarize, the project would have two parts, one to develop a data web-based dashboard and
the second to ensure the performance of any Qlik sense app developed by SDG. Both parts keep the
author highly motivated first because it includes the subject of the master with its specialization, and
second the possibility to be an expert in the development and performance of Qlik Sense software.
1.3. ROLE IN THE PROJECT
Qlik Sense Internship Report
3
The role in the project would be to be part of the Ensemble Team and worked in all the phases of the
projects, that includes the data model construction, The Qlik visualizations, and the Web Site
construction. Otherwise, the focus would be in the Business Intelligence area, which is the data
model construction and the necessary Qlik Sense objects render on the website.
The author constructs the Sense Test Performance Methodology based on the Ensemble work,
research, and capture the knowledge that exists in the company. It is essential to understand the
Social Networks to capture the knowledge and adopt a change management methodology to
succeed.
1.4. STUDY GOALS
The main goals are to create a performance web-based dashboard that allows the EMEA Managers
to make the correct decisions and develop the Sense Test Performance Methodology.
Specific Objectives
▪ Guarantee the veracity of the data extracted and confirmation of the selected metrics.
▪ Facilitate the managerial level of the corporation an effective analysis about the performance
of the company Sales and OPEX operations.
▪ Present a Web based analysis solution that empowers users to access up-to-date data that
include all information needed for business analysis in a clean presentation.
▪ Achieve consultants' participation in the Sense Test performance methodology.
▪ Empower the consultants to implement Qlik Sense good practices.
▪ Change the development process in order to get better results.
Qlik Sense Internship Report
4
2. LITERATURE REVIEW
The main topic during the internship is Business Intelligence (BI), which is a broad category of
applications, technologies, and processes for gathering, storing, accessing, and analyzing data to help
business users make better decisions (Watson, 2009)
The evolution of the BI system is taking the development of new paradigms and challenges. BI
systems differ from earlier forms of Decision Support System in several ways: First, they typically
involve systematic integration, aggregation and management of structured and unstructured data in
increasingly ‘real-time’ data warehouses, which enable new forms of fact-based Second, BI solutions
today deal with very large and increasing amounts of data (‘Big Data’) and can rely on exponentially
increasing processing capacities (incl. in-memory technologies) (Wieder, 2015).
2.1. BUSINESS INTELLIGENCE SOFTWARE
During the internship, Qlik Sense was the most used program, especially for Business Intelligence
development, an application program interface (API) was used to integrate the data to a web-based
report that is named by Qlik as a Mash-Up and use AngularJS, HTML, and CSS. As the web facilitates
the navigation for the final user is in high demand for the client even that needs more time to
develop.
Qlik offers governed data discovery and agile analytic and BI via its lead product Qlik Sense. The Qlik
Analytics Platform supports developers in creating customized applications and for the embedded
use case. QlikView continues to be enhanced and makes up a more substantial portion of the
company’s installed customer base. At the same time, Qlik Sense now accounts for more than 50
percent of license revenue (Gartner, 2018) Figure 2.1.
Qlik Sense Internship Report
5
Figure 2.1 – Garner Quadrant, Source (Gartner, 2018).
For the developer, Qlik has some advantages that are important to be aware. The most significant is
that the tables are linked automatically by the name of the field; consequently, it is not necessary to
define Primary and Foreign Keys. If two tables have more than one field with the same name, it will
generate a Synthetic key, and it is necessary to rename the fields to only have one as key. All the Qlik
functionality is explained in Appendixes A, B, and C.
2.2. DATA WAREHOUSE
A data warehousing is a technique for collecting and managing data from varied sources to provide
meaningful business insights. It is a blend of technologies and components which aids the strategic
use of data. There are two main approaches to construction, Immon, and Kimball:
Immon(Top-Down)
The Data Warehouse (DW) is centralized; it starts with its construction, therefore a normalized data
model is designed first, then the dimensional data marts, which contain data required for specific
business processes or specific departments, are created from the data warehouse. (Sansu, 2012)
Qlik Sense Internship Report
6
The DW contains data at an atomic level, and from here, the data marts are created, as is shown in
Figure 2.2, according to Immon, this are the Datawarehouse characteristics.
▪ Subject-oriented: The data elements are related to a business subject
▪ Time-variant: The data is added over time
▪ Non-volatile: Data in the data warehouse is never overwritten or deleted. Once committed,
the data is static, read-only, and retained for future reporting.
▪ Integrated: The database contains data from all of an organization’s operational applications.
Figure 2.2 – Immon DW model, Source (Panoply, 2019).
Kimball(Bottom-Up )
The model starts the construction from the bottom, in a company the departments or business units,
each one according to its necessities constructed the Data Mart and then if it is necessary as the
company growths a Data Warehouse that integrated all the Data Marts is constructed.
The approach takes less effort, as is shown in Figure 2.3; otherwise, in the end, the same data could
have different values in each department.
Qlik Sense Internship Report
7
Figure 2.3 – Kimball, Source (Panoply, 2019)
2.3. DATA MART
A data mart is a subset of a data warehouse oriented to a specific business line. Data marts contain
repositories of summarized data collected for analysis on a specific section or unit within an
organization, for example, the sales department (Panoply, 2019).
The Data Mart typically have few Data Source Sources that are linked to business need; the size is no
more than 100 Gigabytes. The advantages are that the departments can access quickly to the
information, and the dashboard is constructed according to specific needs.
The data marts are classified as dependent that is the Immon’s model, and it is easier to construct
because the data is clean, and the ETL process is basically to select the information that is needed
from the central DW, and dependent that is Kimball’s approach and must deal with all the aspect of
the ETL process.
2.4. SCHEMAS
The Schema is an essential part of the DW design; according to the information that is needed, a
schema would be constructed and would have an influence on the performance and the queries that
would be necessary to extract the data. The most used are the Start and Snowflake Schema.
Star Schema
In the Star Schema, the data is organized into Facts and Dimensions. A fact is an event that is counted
or measured, such as a sale or log in. A dimension contains reference information about the Fact,
Qlik Sense Internship Report
8
such as date, product, or customer (Rouse, 2019). The structure of the Schema Fact data is like a star
with a central Fact table that is a link through a key to Dimension tables.
Snowflake Schema
The Snowflake schema is like a Start schema, but with the dimension tables normalized, therefore
the schema would have a fact table that is linked to dimensions tables, and this one has links to other
tables.
As the Snowflake schema has more tables than the Star, more complex queries are needed to access
the data, but the system has less redundancy and is easier to maintain. If a dimension table is big in
relatively big in size snowflaking is better as it reduces space (Diffen, 2019).
2.5. DASHBOARD
A data dashboard is an information management tool that visually tracks, analyzes, and displays Key
Performance Indicators, KPI, graphics, or information that is constructed in a visual way that
facilitates the comprehensions for the business user and lets them make the right decisions.
One of the key aspects of a dashboard is to construct the story, “when colleagues and senior
management are overwhelmed by data and facts without context, you lose, You will need the “what”
of your story (the facts and data), but you also need the “who?”, the “how?”, the “why?” and the
often missed “so what?”. It’s these story elements that will make your data relevant and tangible for
your audience.” (Fink, 2007).
2.5.1. KPI
The Key Performance Indicators are the base of the dashboard and are the measures that are
important for the business and are aligned with a strategic objective, The only difference between a
metric and a KPI is that a KPI embodies a strategic objective and measures performance against a goa
(Eckerson, 2009)
A KPI should have the elements that are shown in Table 2.1, to be effective, it is crucial to check that
all of the elements are present, the Strategy is the most subjective and depends on the directives of
the company.
Qlik Sense Internship Report
9
Elements Description
Strategy All KPI should be part of the company strategic
Targets KPI should be performed against a target, according to the company needs
Range Target has a range of Performance
Encodings These are the colors that the KPI should have according to the vale and Range, giving a visual display of the value
Time Frames The target should be in a data frame
Benchmarks The Target should be compared to a baseline like Previous Month or Quarter
Table 2.1 – Aspects of a KPI Source (Eckerson, 2009).
There are two kinds of KPI, the Outcomes and the Drivers, the first ones are related to past
information like Expenses and the drivers are KPI that impact the Outcomes like in expenses the
number of debt negotiations.
2.5.2. Types of Performance Dashboards
One crucial point when a dashboard is designed is to know to whom the dashboard would be
delivered, and the information would be different if it is presented to the executive boards, the
manager, or the operators. Strategic, Tactical, and Operational are the types of performance
dashboards as is shown in Figure 2.4.
Figure 2.4 – Types of Dashboard, Source (Eckerson,2009)
Qlik Sense Internship Report
10
The strategic dashboard, as the name said is focused on strategy and used by the executives of the
company, the information is the data that is summarized, the Outcomes KPI is used to Strategic
decisions, and the periodicity is generally quarterly or monthly.
The Tactical dashboards help managers to optimize the performance of the processes, and this
dashboard is a mix of the Strategic and the operational. Usually, the mid-lever person must follow
the directions of the executives and supervise the persons under its control.
Operational dashboards aim to control processes on an intraday basis and are used for the Staff of
the company, the information is detailed, and the most used KPI are the drivers.
Generally, strategic dashboards comprise outcome KPI, operational dashboards show driver KPI and
operational metrics, and tactical dashboards have a mix of outcome and driver KPI (Eckerson, 2009).
2.5.3. Design of the Dashboard
The design of the dashboards follows the DAR Concept (Dashboard/Analysis/ Reporting) to develop a
clean and somewhat minimal (or necessary) This helps to both increase adoption and understanding
of the application and frequently helps performance. The main idea is to have a story that guides the
user through the data, in this order of ideas, the first page will be a resume with the principals KPI
and primary information, after this detail information of the summary in other sheets and could
navigate throughout the dashboard. (QlikView, 2013).
2.6. CHANGE MANAGEMENT
The Sense Test Performance Methodology requires change management as the objective is to
introduce the change in the organization routine, and a CM methodology should be followed in order
to succeed in the implementation.
Qlik Sense Internship Report
11
Figure 2.5 - Change Management Steps Source (Kotter, 2012)
Change at the organizational level needs a commitment with stakeholders on two phases to
approach, align key stakeholders, and engage organization based on seven steps (Kotter, 2012). The
first step is the one called Understand. In this starting point, it is necessary to talk with the key
stakeholders and share information with them in order to align the different goals. After this phase,
starts the Enlist step, where there are select people who will be in charge to promote the changes,
and that will compose the change team.
The third step is named Envisage, in this case, new visions and missions, but also new strategies are
developed, after this, in the Motivate step the senior level must put pressure on the organization
staff, they should feel a sense of urgency to change. The three last steps are concentrate on
employees; during the fifth one, the Communicate, they are included in the new vision of the
company, and in the sixth one, they will operate and Act in terms of produce a real change inside the
organization. The last one, the Consolidate step, the organization in the whole undertakes to
consolidate gains. All the steps are shown in Figure 2.5
2.7. MEMORY PERFORMANCE
The Qlik performance depends on the RAM. RAM functions as the computer’s short-term or
temporary memory. It stores the data inputted into an application. RAM is only responsible for
holding data while an application is running. Once the application closes, the data is lost and works in
conjunction with the central processing unit (CPU). If RAM is the temporary memory, the CPU is the
brain of the computer. The CPU chip retrieves data from the RAM. The chip takes those instructions
and begins to move and process data through the motherboard in the correct order to its next
designated (Pritchard, 2019).
Qlik Sense Internship Report
12
3. PROJECT METHODOLOGY
According to the Project Management Institute, Project management is the application of
knowledge, skills, tools, and techniques to project activities to meet the project requirements and a
Project is temporary in that it has a defined beginning and end in time, and therefore defined scope
and resources (PMI, 2019).
The SCRUM framework was used for effective collaboration, and Scrum is an agile method of
iterative and incremental product delivery that uses frequent feedback and collaborative decision
making. (Sliger, 2011), all the steps used in the internship are explained in Table 3.1. The principal
tool was the Jira Software that is a powerful work management tool for all kinds of use cases, from
requirements and test case management to agile software development, with the software the user
has access to a Scrum Board (Atlassian, 2019)
SCRUM Description
Planning Meeting It is the first meeting to kick off the sprint, the priorities and goals are established in the meeting
Daily Meeting
The Daily meeting is a 15 minutes reunion to check the work for the next 24 hours, connect the team, analyze the obstacles. It should be quick and informative. During the meeting, the SCRUM Board is analyzed.
The SCRUM Board
These boards have the ToDos, In Progress, Code Review and Test, as is seen in figure 3.1, the priority and owner is showed in the Board
Spring Review Spring Retrospective
In this meeting the DEMO and work of the sprint is done Its a team meeting at the end of the Spring to see the Good, bad and to improve of the meeting
Table 3.1 - SCRUM parts
Scrum focuses on planning, committing, and delivering time-boxed chunks of work called sprints, as
is shown in Figure 3.1. The development of the web-based dashboard is four sprints, and each sprint
is twenty-one days, and five days of User Acceptance Testing UAT. The Senior Consultant does the
Backlog, which is the planning of the sprint with the construction of the User Stories.
Qlik Sense Internship Report
13
Figure 3.1 - SCRUM, SOURCE (Sliger, 2011)
3.1. SCHEDULE
Figure 3.2 - Project Schedule
3.1.1. Kick OFF
It is the start meeting of the internship were the team that the author will work will be presented,
the chronogram as well, and the principal goals.
3.1.2. Qlik Training
Qlik Sense Internship Report
14
In twenty-one days, the author would be familiarized with Qlik, the primary BI tool that is used
during the internship, in Appendix A, B, and C of this document the main topics of the software are
Explain.
3.1.3. Web Dashboard
The construction of the dashboard is divided into four sprints, and each one has fifteen working days
of development and six days of UAT.
▪ Sprint 1: The goal is to develop the data model according to the business requirements.
▪ Sprint 2: The goal is to finish the development of the data model and start to construct the
KPIs and Objects in Qlik Sense and start the web Front End
▪ Sprint 3: The goal is to finish the Qlik Sense Objects
▪ Sprint 4: the goal is to develop the Web part and test all the data
3.1.4. Qlik Test Performance Methodology
With the good practices of the Ensemble Web Dashboard, research, and knowledge, the Qlik Test
performance Methodology will be developed in three sprints.
▪ Sprint 1 and 2: The goal is to document good practices and investigate tools that are useful to
measure the performance of a Qlik application
▪ Sprint 3: The goal is to construct and test a methodology that guarantees good practices
3.1.5. Deliver the project
Deliver the manual with the Qlik Test Performance methodology and do a final presentation to the
whole company and next steps for the implementation
Qlik Sense Internship Report
15
4. WEB DASHBOARD
In this chapter, all the information related to the web dashboard is analyzed; the project needs the
construction of a dependent data mart, the data comes from a DW that is powered by Amazon
Redshift, and the data Extraction is reduced to the selection of the needed data.
The folder structure and architecture would be analyzed in deep because it is going to be an input for
the Qlik Test Sense Methodology, and the company aims to uniform the folder structure of the
projects.
The result of all the processes would be a web dashboard that will render the Qlik Sens objects that
are known as master items. SDG has a web boilerplate that makes the Angularjs connections with
Qlik and brings the master objects with an angular directive. The major part of web development is
HTML and CSS.
The project is a web front end located inside the Qlik Server, and the web communicates with the
Qlik sense app and parses the complete object, the web is an extension of the Qlik app with the
HTML capabilities like responsive menus and easy to use.
4.1. ARCHITECTURE FOR THE PROJECTS AND FOLDER STRUCTURE
A business Intelligence solution came from the Data that could be in databases, ERP, CSV, or other
formats. The Data that is Streams of raw facts representing events such as business transactions is
transformed into information that is meaningful and useful to human beings in the decision process.
This Transformation is done structurally through The Extraction, Transform, Load ETL Process.
After the information is debugged, unify, and consoled in a Data Warehouse, that could be used for
departments to construct different data marts of the company.
In the internship, The Data comes from different sources; the principal is a DW that is powered by
Amazon Redshift, which is a fully managed, petabyte-scale data warehouse service in the cloud, that
lets the user make fast queries (Amazon, 2019).
The whole architecture of the projects is explained in Figure 4.1. From bottom-up, first, it shows the
data extracted, and the Data warehouse is constructed with the Talend software; another team does
this step. Our process consists of the construction of the data mart, from a DW, it follows Immon´s
approach, then the Qlik Sense ETL process and app development, finally, the web front end interface
that is known as a mash-up.
Qlik Sense Internship Report
16
Figure 4.1 – Data Architecture
The Qlik Sense follows the ETL process; mainly, each step of the process creates a Qlik Sense
application. The steps are named Layer, so Layer one is the Extract process, Layer two the Transform,
and Layer three the Load. The scripts of the application are .txt files that are saved on the server and
imported to the Qlik application; it is safer to keep scripts and apply a Git version control system to
follow and share the development process.
All these files are organized inside folders in an architecture way that it is understandable and easy to
follow or find the files, as we see in Figure 4.2.
Figure 4.2 - Folder Organization
4.1.1. Extraction
Qlik Sense Internship Report
17
The extraction process occurs in Layer one. The required fields are extracted with SQL queries and
loaded into the Qlik View Documents QVD. On the Qlik Sense server, the connections to the DW is
configured with the respective credentials. The main.txt is the menu of the .txt files. Table 4.1 shows
the files used in the extraction process. At the end of this process, the QVDs are loaded and saved.
File Function
Extraction 1 – Main.txt The Menu, (Pameça, 1997) included in the Qlik app with an import statement
General - Variables.txt Load the variables of the applications
Extraction 2 - Configuration Tables
Loop in the Metadata File looking for the tables and fields to load
Extraction 3 - Data Extraction
Queries to extract the data and put it in a table.
Extraction 4 - LoadQVD Load all the tables in the QVD’s Folder
Extraction 5- VAR reset Variables that were loaded know are assigned to null
Table 4.1 - .txt files of a Qlik Sense App
4.1.2. Transform
In the transformation process, the data is taken from the QVDs of layer one, and the business
transformations are applied. In the internship dates transformation, to create the Month Year
dimension, and Quarters as numbers in order to compare the values are done in the metadata Excel
file.
The process follows the last step logic with a main.txt menu after the process ends; the data is
loaded into the QVDs of layer two.
4.1.3. Load
The load is the last step and is the app that the client has access to, in this project is a Web that is in
the Qlik Server and connects to a Qlik sense application to get the created objects.
The scripts of the application will construct the data model according to the business requirements.
The schema is a start model with a big fact table.
The Data came from layer two, and the data model is constructed. Finally, the final QVD is stored in
layer three.
Qlik Sense Internship Report
18
4.1.4. Configuration Files
4.1.4.1. The Metadata File.
The Metadata is an Excel file with a dictionary of the Data Warehouse tables used in the project.
The system will read the following information:
▪ Tables to be extracted from the data lake
▪ Source fields need to be loaded;
▪ Source Table
This excel file is also used to configure the connection and extraction of the data sources, to manage
the transformation of data, and to manage metadata. The main sheet in the Metadata file is the
Tables(sys), which has the organization of the source tables to Qlik content, where “Source Table” is
the original table.
Some of the information found in this excel is described in the following explanatory Figure 4.3.
Figure 4.3 - Metadata Table
Some of the information found in this excel is described in the table 4.2:
Qlik Sense Internship Report
19
Field Name Description
Entity Area of analysis.
Sheet Excel sheet name where the table fields are listed.
Source table Name of the Source table in the database.
iConnect Table Name of the QVD to be stored.
Where Clause Where clauses are set here in order to prefilter the data extraction.
QVD Type It differentiates between Master Data and Fact Data.
QVD Path The full path.
Table 4.2 - Metadata Fields
4.1.4.2. Variable Configuration
The configuration file Variable Configuration is where variables and (this includes expressions for the
applications, calculations, and labels for the main metrics and figures) and variables that control
parts of the customized navigation in the dashboard are defined. The system reads the variable
name, the value of the variable, and is active or not.
Critical practice is to follow a naming convention for the variables, the names start with a lower-case
prefix that indicates the type of contents the variable is holding, and then a descriptive noun for the
variable in Camel-Case:
▪ Single values are prefixed with a “v”, for example, vCurrentYear
▪ Expressions are prefixed with an “e”, for example, eSalesAmount
▪ Constants, values that are never meant to be changed, are prefixed with “const”, for example
constYes (Bitmetric, 2016)
4.2. THE DATA MODEL
The design of the data model represents the business needs that the dashboard would require;
according to Qlik, the best approach is a Star Schema and is the one that is advised in all the
developments. In the internship, the model is a Star Schema with a big fact table that has the
majority of data and three Master Tables that are Product, Calendar, and Exchange Rate, as is shown
in Figure 4.4.
Qlik Sense Internship Report
20
Figure 4.4 - Data Model
4.2.1. Facts Data
Since the fact data is coming from different extractions of the view, all facts data will be loaded
independently but concatenated to the Star main table, FACT DATA
4.2.2. Master Data
All master data will be loaded independently but connected to the Star table via a key.
Calendar
The Master Calendar table contains all dates from the natural calendar with different formats
(YYYYMM, MMM-YY, etc...).
Product
The Master Product table contains the JnJ products, harmonized names with the Therapeutic Area
assigned.
Exchange Rate
Qlik Sense Internship Report
21
The Master Contains table the local currency and the exchange rate
Granularity:
Geography granularity is shown in Figure 4.5, start with the region EMEA, then it has the clusters that
include different countries like IBERIA stands for Portugal and Spain, and finally the countries. The
user can filter with the grade of detail that considers necessary.
The top hierarchy in products is the Therapeutic area; Jansen has six priorities therapeutics areas,
that area Cardiovascular Metabolic disease, Immunology, Infectious Diseases Vaccines,
Neuroscience, Oncology, and Pulmonary Hypertension. Then came the Product Family that groups
different products, later the product, and finally, the product SKU that has the specific product name
and specification as is shown in Figure 4.5.
The Dates the lowest level of granularity is Month, then Quarter and finally Year, also a Month-Year
Dimension is created.
Figure 4.5 - Granularity
4.3. APPLICATION
Qlik Sense Internship Report
22
The user enters and authenticates in a landing page, to authenticate un app with the credentials
made the process, this page works for different applications like Brand Cockpit, Ensemble and CIE
Index as is shown in Figure 4.6.
Figure 4.6 - Landing Page
After the user login, it enters the Geography Area site that offers a pleasant, easy-to-use, navigating
menu on the left and a vertical menu on top, then the site with vital statistics to the Ensemble
Application, displaying information on the top 5 Clusters.
The horizontal menu lets the client navigate to the different sheets of the web mashup, that are
Geography, Therapeutic Area, Competitor Analysis and CIE Index, the users can navigate through
them like a standard website, Figure 4.7
Figure 4.7 - Horizontal Menu
The vertical menu has a series of toggles that let the user choose between different granularity and
range measures, as is shown in Table 4.3 and Figure 4.8,also to filter according to its necessities .
Qlik Sense Internship Report
23
Figure 4.8 - Vertical Menu
Menu name Description
PY Previous Year.
BP Business Forecast created at the beginning of the year.
JU June Update. Take into account that this data is available after June. From Jan to June, it is Actual. After June it is forecast.
NU November Update. The data is available after November. From Jan to Nov, it is actual data. After November, it is forecast.
YTD Year to date stands for the months of the year until the current date.
CAL Calendar, it is a Month date Picker.
Table 4.3 Benchmark Selection KPI
4.3.1. KPI
Table 4.4 shows the main KPIs that are used in the project; all of them have a benchmark that the
user can select; the default is the Previous Year. The company defines the target, and according to
them, the encodings are determined, green, and arrow up if it is improving, amber for neutral, and
red and arrow down worst.
KPI Name Description
OPEX Stands for Operating Expenses and shows the expenses according to the selection.
Market Share It is the portion of the market that the product has.
Evolution Index
It is the share of the sales of the products of the current year with market sales of the previous year.
Loyalty It is the current ranking of the product according to the users.
Table 4.4 Principal KPI
Qlik Sense Internship Report
24
Figure 4.9 shows a KPI example, with all the parts that are Strategy Range, Encodings, Time Frames
and Benchmarks,
Figure 4.9 - KPI example
4.3.2. Dashboard
The Dashboard would have information about Sales, Operation Expenses OPEX, and a Loyalty
measure that is a survey of the doctor with the pharmaceutical products. The Net Trade Sales that
are the gross sales minus the sales allowances, sales discounts and sales returns, in the internship the
data was taken from a table a not operation besides aggregating according to the necessities was
done.
The Operational Expenses OPEX stands for the costs a company incurs for running their day-to-day
operations. These expenses must be ordinary and customary costs for the industry in which the
company operates. (ROSS, 2019) The company must monitor and have an acceptable level of
expenses.
The loyalty Measure is a rank that is classified through surveys to the doctors about the Janssen
products, the information goes until level five, being one the best rank and 5 the worst. Managers
monitor any change that represents the acceptance of the products.
The objects were created first in the Qlik Sense App, then are imported trough a configure angular
directive. When the chart does not exist in Qlik, like the line chart with two lines, it is necessary to
parse the data in a hypercube that represents extractions of the data. Then the graphic is done with
a JavaScript graphic library like Picasso. This web development is out of the scope of the report.
4.3.3. Geography
This Site gives information classify by geography, as is shown in Figure 4.10, the web dashboards
start showing the information about the Region EMEA and then the top 5 Clusters. The principal
objects are:
1. Line Chart, in blue, the current selection, and in gray, the selected benchmark. The bottom of
a sales KPI with all the corresponding elements is displayed.
2. Market Share.
3. Loyalty Ranking.
Qlik Sense Internship Report
25
Figure 4.10 - Geography Dashboard
4.3.4. Therapeutic Area
The user must select a therapeutic to access the site, after these three KPI are shown, as is seen in
Figure 4.11. the main objects are the two-line charts, that show the data according to the selected
toggle.
The idea is that the managers can compare the desired Therapeutic are for different countries; the
default measures are NTS and Market Share.
Qlik Sense Internship Report
26
Figure 4.11 - Therapeutic area
4.4. TEST
For a dashboard to be accepted and has utility inside the organization, the data must be accurate;
misleading data leads managers to make wrong decisions that involve a high cost for the company.
The business owner accepts any US after the tests is approved, the test is an excel sheet that
contains the steps to test the data; one example could be:
1. Open the dashboard and select country Spain, Product Darzalex.
2. Check the value of the line chart for the year 2018.
3. Open the Database administrator and write the following query.
4. The Values match
The developer must create a test per US, then do the test and present graphical evidence. The
business developer redoes the test and agrees or rejects the development after he accepts it, the US
is ready for production.
During the internship, the author investigates Robot Framework that is a generic open-source
automation framework for acceptance testing, acceptance test driven development (ATDD), and
robotic process automation (RPA) (robotframework, 2019).
The robot can open a website, navigate through them, check for functionalities, also to see the
display value on the web, and compare it to a database. The data robot uses Python and the
selenium library.
Qlik Sense Internship Report
27
The author develops a small data robot in Python program and Selenium library that compares data
from the web with data of the database. The results are shown in Figure 4.12.
Figure 4.12 Data Robot Log
The development is a small demo that shows the managers the technology to analyze and decide for
future developments. The Data robot has advantages like program the test periodically and gets
alerts in case of a mismatch. Also, it is a guarantee that the data is accurate, converting the tool in a
win-win situation.
Qlik Sense Internship Report
28
5. QLIK TEST PERFORMANCE METHODOLOGY
The second part of the methodology is related to Business Intelligence and Change Management, the
goal of change management is to introduce the change on the organization routine, defying the
organizational changes required to transform project’s product into benefits.
The Qlik Sense Teste performance Methodology brings quality to the projects and improves the
consultant’s skills. One crucial point is that the manager for Janssen is requesting for the Qlik Test
Performance Methodology; this gives the project the support to implement the Kotter steps.
Four CM workshops were organized.
1. Present the project to the seniors, managers, and team leaders, all of them agree to
collaborate and share knowledge.
2. Present the sketch and get feedback.
3. Show the results in an app and see methods to implement.
4. Demo session with all the consultants/
The managers and team leaders start to transmit a sense of urgency, also the slogan “better
developers, the stronger company.”. The idea is that the consultants understand that following the
Qlik Test Performance Methodology would lead them to be better professionals and helps the
company to a position as a leader in quality.
For Business Intelligence, active research in methods and software that measure the performance is
done by the author, the managers and the team leaders want a practical methodology that is easy to
implement. The good practice of the Ensemble, the knowledge of the teams, and workshops are the
input of the Sense Test Performance Methodology.
5.1. DESIGN OF THE METHODOLOGY
The methodology was constructed with the supervision of the Managers and Senior consultants
under the SCRUM methodology. First, a manual is constructed with the Folder Structures, the ETL
process explained in chapter 4, and the Qlik Engine functionality appendix D.
The Qlik Sense Performance Methodology would have four steps; the idea is that before any app
goes to production, the application passed for these steps; if the results are not satisfied, the
developers should correct and test it again.
The Steps are:
Qlik Sense Internship Report
29
• Excel Checklist
• Document Analyzer
• Scalability Tool
• Application Performance Presentation
5.1.1. Excel Checklist
An excel file is constructed and delivered to all the company consultants; the file aims to secure the
performance of the application. The consultant should answer and add graphical evidence of
eighteen items after the questions are answered a table panel, as is shown in Figure 5.1, resume the
findings.
The panel would give the team leader insight into the application, to answer all the checklist
questions the consultant could use the Document Analyzer that would be explained in the next
step. The questions are grouped into three categories.
Figure 5.1 - Check List Panel
▪ 14 Critical issues that profoundly affect the data model like synthetic keys, Tables load
without the QVD optimized, circular references, appropriated granularity, Timestamps
removed.
▪ 5 High issues: Aggregation variables, the keys with strings, and without Autonumber (), String
Comparison, Variable configuration with all the variables.
▪ 8 Medium issues: Time Stamps removed, Set Analysis used where possible, data island avoid.
For a detail explanation of the questions that are on the excel, check Appendix E.
Qlik Sense Internship Report
30
5.1.2. Document Analyzer
The document analyzer is a free application profiling tool for Qlik Sense that can help you identify
items such as unused fields, poorly performing expressions and data model problems (WUNDERLICH,
2017) is created by Rob Wunderlich.
A document analyzer is a valuable tool that highlighted where the main flags of the application are,
as is shown in Figure 5.2. This tool helps the consultant to get the status of the app. It is easy to
install, and as it is a Qlik Sense app is easy to tun also.
Figure 5.2 - Document Analyzer
The calculation times of the document analyzer are not accurate because the app is not running in
the server; for this reason, the times would not be taking in the account except to check if it is an
outlier.
As is shown Figure 5.2 three crucial flags should be considered
1. Three objects take more time than the rest.
2. Seventy-six fields are not used and take 35 percent of the RAM usage.
3. One hundred ninety-nine variables are not loaded in the back end on the variable
configuration file.
Qlik Sense Internship Report
31
As the field removal is a business decision and the time to check is short, the Pareto optimal was
suggested, which is to take the 20 percent of the number fields that more memory space occupies
and check with the business owner if they can be removed.
Most cost resource expressions are related to nested ifs and string comparison issues. The first one is
solved with the implementation of a Pick() function, as is shown in Figure 5.3, and the second one,
the string is changed for a dummy value.
Figure 5.3 - Optimized variables
The developer team should use the last two steps in conjunction in order to analyze the application
and correct possible issues. After this, the team leader should approve, and the app pass to the next
step.
The document analyzer is an early victory for the Sense Performance Methodology, which is
accessible to users and explains in a secure form the flags of the apps.
5.1.3. Scalability Tools
The Qlik Sense Enterprise Scalability Tools are used to create, run, load, and performance tests
towards Qlik Sense Enterprise for Windows, the tool is created by Qlik that offers a guarantee on the
results.
The Qlik Sense Scalability Tools is a suite of tools that simulated virtual users against the Qlik sense
server, making it possible to tailor the performance of the application. With this application, essential
information about the RAM and CPU is retrieved, and compare with the acceptance criteria of the
stakeholder, one example of the results is showed in Table 5.1 - Scalability Tools result.
Qlik Sense Internship Report
32
KPIs Median Average Maximum
Time to open application
2 seconds 2 seconds 2 seconds
Initial RAM cost 116 MB 116 MB 116 MB
Response time first user
3 seconds
29 seconds
857 seconds
Resource consumption;
CPU
2 cpu seconds 2 cpu seconds 2 cpu seconds
Resource consumption;
RAM
116 MB 116 MB 116 MB
Table 5.1 - Scalability Tools result
The Scalability tool can do a stress test; the application generates virtual users that simulate the
behavior of real users and test the performance of the Qlik Sense Object like graphics that are on the
dashboard according to the number of users that the application will have in production.
With this result we can start to analyze which could be the main issues of the application, in Table
5.1, the RAM Consumption is too high, in most cases there is a lot of fields or dimensions that are not
used and are taking memory. The error should appear in the last two steps of the methodology.
As the Scalability tools compare the information against the server is necessary to configure a virtual
proxy to intercept the server and obtain the information, all the installation steps are on the user
manual, even though it is only necessary to install one time per server.
As Qlik creates and certifies the result, the tool could be the guarantee that SDG could offer to the
sophisticated clients that need the times, RAM, and CPU usage of the application according to a
certain number of users as the case of Janssen.
5.1.4. Application Performance Presentation
A Presentation document is delivered to the client, with performance information of the Qlik Sense
Scalability tool. A boilerplate is created, so the consultant only must edit and add the information
related to the app. This document is a guarantee for the customer and innovation process for SDG.
Qlik Sense Internship Report
33
5.1.5. Results in an example App
In the third workshop, the manager decided that the Qlik Sense Test performance methodology
would be tested in the CIE Index Application, this is one of the most important dashboards created in
the company for Janssen Pharmaceutical. Moreover, as the data is growing, the speed of the
dashboard is decreasing.
Having one of the crucial dashboards is also an early victory that helps in the change management
process as most of the consultants know the app.
5.1.6. Flags
From the excel checklist, only one change has to be done and is that the link keys in the data model
are not numbers; with the Qlik autonumber() function, any field could be converter in a unique
number that would optimize the queries.
In the document analyzer, the following flags exist:
▪ Field Count: 157 Unused: 75 that represents 47.8% of the total and 49.7% of RAM
Consumption
▪ Four unused files with more RAM Consumption came from the same table
▪ 498 variables in total, 199 did not load in the variable configuration, and the application must
read from front end causing more load times and RAM consumption
▪ Ten variables with nested ifs.
5.1.7. Improvements
the Scalability tool was applied to the apps before and after the improvements to test the
methodology with the following results as is shown in Table 5.1 - Scalability Tools result.
Qlik Sense Internship Report
34
Graphic Improvement
RAM Consumption:
The app needs 38% less of RAM Consumption
Open the app:
The Corrected application is faster to open in 0.3 seconds faster
Objects Load:
The objects that have the nested if improve the performance around 50%
Chrome Test:
The improve in the mashup time is around 10%
Table 5.2 - Results CIE INDEX App
5.1.8. Qlik Sense Performance Methodology Conclusion
The Qlik Sense Performance Methodology has a massive acceptance in the consultants, also the
author must expose the results to the managers in Spain and to the whole team, which is around one
hundred specialists. In Spain, the Test Performance methodology is apply, making it a success.
Qlik Sense Internship Report
35
One of the key success factors is to follow the Change management steps and involve the significant
part of the teams, besides capturing the knowledge helps that everyone empowered.
The other important point is that it is easy, and the result is a better performance of the app that can
be tested, also the consultants that applied become better developers and gain Qlik knowledge
faster. For junior consultants that are learning, the software becomes a secure form to gain
independence and understand the usage.
Qlik Sense Internship Report
36
6. CONCLUSIONS
The leading software used in the internship is Qlik Sense that with the Data discovery let the user
improve and understand the relationships of the data, and let the software be one of the principal
leaders in the Business Intelligence Software.
The objective of creating a web dashboard for the Janssen managers of the EMEA region was
complete, with all the User Stories approved, the business information that the EMEA managers
need to make decisions displays in graphics and Key Performance Indicators.
To succeed, dashboards must-have KPI with all the aspects that are strategy, targets, range,
encodings, time frames, and Benchmarks as the ones developed in the Web-dashboard. The business
owners give excellent feedback and accept the developed work.
The web-based navigation gives easy to use dashboard, with menus and selections as standard webs,
facilitating the navigations and saves the company to invest in the manager's training, also with the
web there is more freedom in the placement and space of the object.
The Qlik Sense Performance Methodology follows the seven steps of the Kotter methodology for
Change Management; This allows having a strategy to adopt the change, one of the most important
aspects is to have a backup from the directives and that the team understands the sense of urgency.
One of the principal goals is to improve the performance of the apps through innovation in the
development process. As Qlik works in memory, the RAM and CPU reduction determines the
performance, as showed in the CIE index app, the Qlik Sense Methodology allowed through four easy
steps to achieve the goal. The four steps, Excel Check List, Data Analyzer, Scalability tools, and
Guarantee presentation, are easy to use and helps any developer that follows it to understand the
Qlik Software and improve its skills.
To get engagement is also essential to have early victories as the Document Analyzer that is an
unknown free tool with benefits to understand the flags of a Qlik Sense app. The mix of the Change
Management Methodology and the research in performance were the success factors that allow the
Sense Test Performance Methodology to pass the Portugal border and replicates in Barcelona.
Documentation with all the steps is delivered to the company; it includes the Ensemble folder
structure to harmonize the future company developments, installation of the programs, and a
PowerPoint boilerplate to present to the client with all the app performance as a result of the work.
Finally, the decisive point is that all the persons of the company are sensitive and understand the
urgency to deal with the RAM and CPU consumption, and the Test Performance Methodology is
adopted. The author has to apply the master theory, and get strong skills in Qlik Sense, making the
internship a valuable start in the Business Intelligence job market.
Qlik Sense Internship Report
37
7. LIMITATIONS AND RECOMMENDATIONS FOR FUTURE WORKS
One of the most significant barriers is that besides that, the author has knowledge in web
development, especially in HTML, CSS, and React but not in Angular. So, a couple of developments
that could have more impact could not be developed because of the need to bring the paint the
objects with external angular libraries like Picasso.
In the technology and consultant company, the personal rotation is big, in some cases, one
consultant and has good ideas, but the next month was in another company.
The time to investigate, develop, and test is limited, and the knowledge capture was asking all the
team and listen to their impressions. However, with more time, a Social Network study could be
developed and have better results in the group to lead the implementation.
Qlik Sense Internship Report
38
8. BIBLIOGRAFIA
Amazon. (2019). Obtido de https://aws.amazon.com/redshift/
Atlassian. (2019). Atlassia. Obtido de Jira: https://www.atlassian.com/software/jira/guides/use-
cases/what-is-jira-used-for#Jira-for-requirements-&-test-case-management
Cronström, H. (2012). Symbol Tables and Bit Stuffed Pointers. Obtido de Qlik:
https://community.qlik.com/t5/Qlik-Design-Blog/Symbol-Tables-and-Bit-Stuffed-Pointers/ba-
p/1475369
Diffen. (2019). https://www.diffen.com/. Obtido de
https://www.diffen.com/difference/Snowflake_Schema_vs_Star_Schema
Eckerson, W. (2009). Performance Management Strategies, How to create and deploy effective
metrics. The Data Ware House Institute.
FENG, W. (2004). Japan’s Wisdom: the melodies of Japanese People. Hsin Chao She Culture.
Fink, E. M. (2007). Telling the Story—Best practices for using visual analysis to find and tell stories
from data. American Marketing Association presentation.
Gartner. (26 de 02 de 2018). Magic Quadrant for Analytics and Business Intelligence Platforms.
George, S. (2012). computerweekly. Obtido de https://www.computerweekly.com/tip/Inmon-or-
Kimball-Which-approach-is-suitable-for-your-data-warehouse
Kotter, J. P. (2012). Leading Change. Harvard Business Publishing.
Magdalena, R. (2012). Optimizing Time in Business with Business Intelligence Solution. Science Direct.
NONAKA, I. K. (2014). ynamic fractal organizations for promoting knowledge-based transformation–A
new paradigm for organizational theory. European Management Journal, 137-146.
Pameça, P. (1997). The Data Mart: A nee approach to Data Warehousing. International Review of Law
Computers, 251-264.
Panoply. (2019). Panoply. Obtido de https://panoply.io/data-warehouse-guide/data-mart-vs-data-
warehouse/
PMI. (2019). what-is-project-management. Obtido de pmi: https://www.pmi.org/about/learn-about-
pmi/what-is-project-management
Power, D. (2002). A brief History of Decision Support Systems. DSS Resources.
Pritchard, P. (2019). how-does-ram-work-with-the-cpu. Obtido de techwalla:
https://www.techwalla.com/articles/how-does-ram-work-with-the-cpu
Qlik. (2019). diagnostictoolkit. Obtido de https://diagnostictoolkit.qlik-poc.com/
Qlik Sense Internship Report
39
QLik. (2019). work with QVD files. Obtido de Qlik: https://help.qlik.com/en-
US/sense/November2019/Subsystems/Hub/Content/Sense_Hub/Scripting/work-with-QVD-
files.htm
QLikView. (2013). DashBoard, Analysys Reporting(DAR). QlikView Technical Brief.
robotframework. (10 de 12 de 2019). robotframework. Obtido de https://robotframework.org
ROSS, S. (2019). Invistopedia. Obtido de Invistopedia:
https://www.investopedia.com/ask/answers/112814/whats-difference-between-capital-
expenditures-capex-and-operational-expenditures-opex.asp
Rouse, M. (2019). techtarget. Obtido de star schema:
https://searchdatamanagement.techtarget.com/definition/star-schema
sales. (2 de 2019). rrg. Obtido de test2: www.test.com
Sansu, G. (2012). computerweekly. Obtido de https://www.computerweekly.com/tip/Inmon-or-
Kimball-Which-approach-is-suitable-for-your-data-warehouse
Sliger, M. ( 2011). Agile project management with Scrum. PMI® Global Congress. Dallas, TX: PA:
Project Management Institute.
Soejarto. (2003). Tough Times Call for Business Intelligence Services, an Indisputable Area of Growth.
9th Americas Conference on Information Systems. ampa, FL, USA.
Watson H.J, B. W. (2010). The BI-based Organization. International Journal of Business Intelligence
Research, 13-28.
Watson, H. J. (2009). Tutorial: Business Intelligence – Past, Present, and Future. Communications of
the Association for Information Systems, pp. Vol 25, art 39.
Wayne, E. (2009). Performance Management Strategies, How to create and deploy effective metrics.
The Data Ware House Institute.
Wertz, J. (2017). Forbes. Obtido de Why Site Speed & Design Can Make Or Break Your Google
Ranking?: https://www.forbes.com/sites/jiawertz/2017/07/17/why-site-speed-design-can-
make-or-break-your-google-ranking/
Wieder, B. M. (2015). The Impact of Business Intelligence on the Quality of Decision Making – A
Mediation Model. Procedia Computer Science, pp. 1163-1171.
WUNDERLICH, R. (10 de 01 de 2017). the-document-analyzer-compare-tool. Obtido de
qlikviewcookbook: https://qlikviewcookbook.com/2017/01/the-document-analyzer-
compare-tool/
Qlik Sense Internship Report
40
APPENDIX A
QLIK SENSE TERMINOLOGY
Qlik View Data QVD.
QVD is a native Qlik format and can only be written to and read by Qlik Sense or QlikView. The file
format is optimized for speed when reading data from a script, but it is still very compact. Reading
data from a QVD file is typically 10-100 times faster than reading from other data sources Qlik can
compress data down to 10 percent of its original size (QLik, 2019).
The performance of the application depends that all the tables are load from a QVDs and optimized.
Set Analysis.
The set analysis used an expression syntax to aggregate measures based on defined conditions; it is
commonly used in chart objects to exclude or include data, it works in memory after the data is
loaded, a primary example is:
SUM(< Year = ”2019” > Expenses)
This expression summarized the expense of the year 2019, at taking it into account, the current
selections of the user interface. The sum of the expenses, SUM(expenses), but inside the curly
braces, the necessary conditions are added.
There are two general syntax rules for a set expression:
A set expression must be used in an aggregation function. In this example, the aggregation function is
sum (Sales). A set expression is: ”{< Year = 2019 > .” and the braces must enclose
expression.
Synthetic Keys in the Data Model
Qlik references the key of the table by the name of the field. A Synthetic key appears if two tables or
more have more than one field with the same name, these synthetic keys affect the data model, not
only in performance but also Qlik crash when the number increase.
When a model has Synthetics keys, a warning message appears on the script, and, a data model
shows them in a graphic way, the one that we see in Figure A.1. The data could have mismatches
when the model has Synthetic keys.
Qlik Sense Internship Report
41
Figure A.1 - Synthetic Keys
The possible solutions to fix the synthetics keys is to rename the field or create a Link table.
A.1.4 Link Tables in the Data Model
The link table is a table that contains common fields from two or more tables and is created to avoid
synthetic keys.
One of the characteristics of the data model is that it should be subject-oriented, in some cases, the
business requirement needs a Fact Constellation Schema with two fact tables, for example, sales and
budget, that share the same fields and generates a synthetic key.
The Qlik solution is to create a Link Table that” Link” the tables and avoid the synthetic keys, and the
different fact tables have the common fields connected through trough this table.
When the application grows, it is better to concatenate the tables into a big fact table with all the
information. It improves the performance but could be difficult for the business perspective to
understand the data model; the developer should explain this trade-off to the business owner and
guide him to make a decision.
A.1.5 Concatenate or No Concatenate in the Data Model
Qlik concatenates the tables with the same name automatically, so it is vital when the data is load
after the LOAD to put NOCONCATENATE if different tables are needed.
The reason for this is that If the field names and the number of fields of two or more loaded tables
are the same, QlikView will automatically concatenate the content of the different statements into
one table (Qlik, 2019).
This could generate a data model that does not satisfy our needs, and it is one of the mistakes that
are done when someone starts in the Qlik world.
Qlik Sense Internship Report
42
APPENDIX B
QLIK MAIN FUNCTIONS
The Business Intelligence Software has predefined functions that help in the development process;
these functions are a type of procedure or routine that is often used. The developer needs to call
them with the right parameters; in this appendix, the main functions that were used on the
internship are analyzed.
Mappings
Mapping provides an efficient way to substituting field values during script execution; basically, a
mapping is a virtual table that is created with two fields, Field A, Field B, then the value of the Field A
replaces the one of the Field B.
For example, the country Sweden changes to the Country Code SW that is in another table.
Hierarchy
The hierarchy operation is used to transform a parent-child hierarchy table to a table that is useful in
a Qlik Sense data model and is used a lot in the financial dashboards. The operation creates a new
table with Parent-Child Relationships, which usually has the same number of records as the input
table. However, besides, each level in the hierarchy is stored in a separate field. The path field can be
used in a tree structure.
Join
Join is one of the most used operations in SQL and combines two tables into one; There are four types of Join:
▪ Join: the Join is an outer join, creating all possible combinations of values from the two tables.
▪ Inner Join: The result table only contains records existing in both tables. ▪ Left Join: The resulting table only contains combinations between the two tables with a full
data set from the first table Right Join: The resulting table only contains combinations between the two tables with a full data set from the second table
Qlik Sense Internship Report
43
▪ Right Join: The resulting table only contains combinations between the two tables with a full data set from the second table
Qlik Sense Internship Report
44
APPENDIX C
THE QLIK SERVER
During the internship, desktop software was installed to test and construct small applications;
nevertheless, most of the time, the work was done directly on the server.
Janssen has three servers:
▪ Dev: in development, only developers have access; when the Senior developer approves the
work , the app passes to the second server.
▪ QA The business owners test the User Stories here, when they approve, it passes to
production.
▪ PRQA: Production is the one that the final user has access.
The Qlik Server is user-friendly and has two main parts the Qlik Management Console (QMC) where
the apps are, permission data connection are loaded, and the hub where the user can access the
application.
Qlik Management Console
The Qlik Management Console content all the resources; the apps are load, run, and published; also,
where the security rules are created and the data connections are specified, in Figure C.1 the start of
the QMC is shown.
Qlik Sense Internship Report
45
Figure C.1 - Qlik Server QMC.
Different roles are created on the server. The one with more access is the Admin one that can
configure the system, export certificates, or create virtual proxies to connect. In the internship as a
developer, the author only has access to the apps, data connections, Tasks to load the data, and
Extensions.
There is the list of the apps or extensions available, and buttons to edit, delete, export or import, the
only consideration is to have a buck up of the files in case something goes wrong when the app runs.
The Hub
The hub is where the apps can be seen, according to Qlik The hub is where you find all the apps that
you have access rights to use QLIK 2016 if the apps are published, the script could not be edited it;
otherwise, the developer can see the dashboard and edit according to the necessities
As we can see in the. 4 this is the parts found in the hub
Toolbar to search.
Login Information.
Work Area, a zone to create new apps.
Apps that are not published, but the developer has access.
Different streams where the apps are published
Qlik Sense Internship Report
46
A personalized area where the published or not published apps could be added.
Figure C.2 - The hub
Qlik Sense Internship Report
47
APPENDIX D
THE MEMORY LOAD IN QLIK
All the Qlik Data loads in memory; this functionality is the one that allows the data discovery and the
rest of Qlik advantages; nevertheless, to work in memory, Qlik uses the Qlik Engine that improves the
performance.
Symbol Table
Under the hood, the Qlik Engine used the Symbol Tables For each unique field in an associative
database. Qlik creates a two columns table that is referred to as a Symbol table Figure. D.1. The
symbol table contains one record for each unique value loaded through the ETL process. One column
stores the unique, and the other column stores a ‘symbol’ value. The symbol is a ‘bit stuffed pointer
that saves it in memory. (Cronstrom,2016)
The storage need for a bit stuffed pointer value increases the bigger the pointer value (a high row
number bit stuffed pointer value takes up more room). The clearest example to understand this logic
is with the dates, imagine that all the days of the year are on a field, in this case, the symbol table will
have 365 different values, and each value is a stuffed save it in memory. If dates are separated in
months and days, the result symbol tables occupy 30 pointers for the days and 12 for the months
and represent 10 percent of the whole year. Cases like this were fixed during the internship were
even timestamps with seconds were occupying memory without a business need.
Figure D.1 - Data tables. Source (Cronström, 2012).
Data Tables
While creating the symbol tables, Qlik also recreates the actual tables, using the symbol values
in lieu Remember, large bit stuffed pointer values will be repeated in the Data table so three
reasons to reduce the variety of values per field.
Qlik Sense Internship Report
48
APPENDIX E
QLIK CHECKS
The checks that excel has are explained in the next tables
Check Issues
The If function is avoided where possible
If() s suggested to be avoided where possible as if()
operates at the record level. If you have 1k records in an aggregation, it will evaluate that if() condition 1k times
String comparisons avoided where possible:
String comparisons are not nearly as efficient as Set
Analysis. Examples being: Match (), MixMatch(), WildMatch, Pick, etc. Use them only when necessary, for example in the load script as part of a WHERE statement. Avoid them on the front-end where able and instead create flags in the script or use set analysis
Sheets do not have an excessive amount of
objects
calculated. Following that, every time a user selects on that sheet, each object will be recalculated if that current state does not exist in the cache
.
Table E.1 - Check Interface Source Qlik
Qlik Sense Internship Report
49
Figure E.1 -data check Source Qlik
Page | i