Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Enterprise Process Integration Controller
EPIC
Course Content
2
• Overview• File processing concepts• Topology• Entry/Exit points (communication)• Processing• OFTP – overview of other protocols• System Setup/Configuration• Monitoring the System• Common Tasks
EPIC Overview
3
EPIC
4
• Client/Server architecture• Microsoft SQL Server Database• Workflow driven• Multi-threading tasks – vertically scalable• Visibility / Audit controls• XLATE Evolution (XE) translation engines• Multiple communication methods• Automation/Manual
EPIC
5
Input from:
• Directory/share
• User input
• Communication
• Back office system
• Web Upload
What does it do
Files into Epic Decisions made (processing/routing) Files out of Epic
Processing:
• Mapping
• Routing
• Audit
• Control
• Reformatting
Output to:
• Directory/share
• User input
• Communication
• Back office system
• Web Upload
EPIC
6
Directory/share
1. Epic monitors a windows folder
2. Waiting for files to be created by an internal system
Simple Example
Processing
1. Based on a combination of directory and filename, EPIC decides how to process the file
2. Selecting an XE map
3. Selecting a destination
OFTP
1. EPIC initiates an OFTP connection and sends the file
• EPIC picks up files from a directory (created by a backend system)• Converts them from an internal CSV format to a customer’s EDI format• EPIC sends files to the correct customer using OFTP
Ford Motor Company
7
Production Example
Ford Motor Company
8
Supplier
Sending time critical and sensitive information (supply chain logistics)
The Problem
OFTP
Provides a robust connection over ISDN, X.25 or TCP/IP, however security relies on the network, e.g. VPN
Mainframe
ODEX MVS tightly integrated with backend systems using a series of scripts.
Very difficult to replace ODEX MVS
• Production suppliers connect to Ford using OFTP to exchange VDA files (supply chain logistics)• Desire to start using OFTP 2 – supports file encryption and signing – in order to leverage low cost high
speed internet connections• Heavily invested in Mainframe environment – no support for OFTP 2
ODEX MVS (Mainframe) Backend systems
Ford Motor Company
9
Supplier
Can now make use of OFTP2, to give a secure low cost connection.
Plus, no need to invest in a VPN or maintain ISDN or X.25 lines.
The Solution
EPIC
Acts as a bridge between ODEX MVS (OFTP 1) and suppliers (OFTP 2)
Can optionally convert between EBCDIC (mainframe codepage) and ASCII
Mainframe
Continues to see OFTP connections as before
ODEX MVS (Mainframe) Backend systems
OFTPOFTP2
Ford Motor Company
10
• No changes required to complicated backend links• Fast deployment and easy migration path for users• Allows a strategic protocol to be implemented at low cost• Provides a strategic platform capable of supporting new projects/requirements
- e.g. other supplier facing protocols, FTP, AS2, etc.
The Benefits
Scania
11
Production Example
Scania
12
Supplier
Sending time critical and sensitive information (supply chain logistics) using X.25 and ISDN, slow and expensive
The Problem
Amtrix
Exchanges data with internal systems using MQ
Maps data between internal EDIXX3 format and external EDI formats
• Legacy Amtrix system nearing end of life• No support for OFTP 2• New business requirements (web interface and new protocols)• Amtrix configuration evolved over 10 years with limited documentation• Remit to replace it and add new functionality
Amtrix Backend systems
OFTP MQ
Scania
13
Network split into 3 zones, each with their own limitations:
The Environment
CorporateAble to exchange files with internal network shares and FTP servers (for ENGDAT)
DMZ ProductionAble to communicate with the other zones
Public StagingAble to communicate with the outside world (Internet)
Each zone requires two instances of EPIC:• Primary – The default instance of EPIC• Backup – A warm-standby instance of EPIC
Scania
14
The Approach
Phase 1 Epic acting as a front-
end to Amtrix
Suppliers able to start using OFTP 2 quickly
without the need to fully understand Amtrix
No changes required for X.25 and ISDN
suppliers
Phase 2 Epic Starts to Process
ENGDAT data
Passing it to internal systems rather than
through Amtrix
All other data continues to go through Amtrix
Phase 3Adding new functionality
Web Interface for ENGDAT (upload and download CAD/CAM
packages)
New protocols such as FTP
Analysing the data flows through Amtrix and
starting to migrate them
Phase 4Epic able to process all
data
Amtrix no longer required
All data translation taking place within Epic
Scania
15
The Solution – Environment
DMZPublicCorporate
Epic Proxy
Epic Corporate OFTP2
Cisco Router
Bintec Router
X.25
X.25/ISDN
Internal Systems
FTP
DMZ
Epic DMZ
Internal Systems
MQ
Web Server (IIS)
FTP
Scania
16
The Results
• Each phase implemented with no problems• Allowed strategic objectives to be satisfied (implementing OFTP 2 and Web ENGDAT)
without a need to perform full requirements analysis of Amtrix• Allowed data flows to be migrated away from Amtrix one at a time, minimising risk
17
DiNet
DiNet
18
Supplier
Sending time critical and sensitive information (supply chain logistics)
Provided with free DOS software and using a dialup modem (X.28)
Service History
DiNet
Originally running on Tandem Unix using ODEX Plus
Customers
Originally only supported direct X.25 connections
• DiNet has been in operation since 1996: “Data Interchange Network”• Started as a service to allow automotive suppliers to exchange files with larger companies without the
need for expensive private circuits (X.25)• This type of service is referred to as a Clearing Centre or VAN (Value Added Network)
OFTP (X.28) OFTP (X.25)
DiNet
19
Partner 1
Sending messages
Able to use a variety of software or the web interface
Current Solution
DiNet
Multi-site environment with replication of data
• DiNet now runs on Windows using EPIC• Supports a much wider array of communication protocols for users• Allows translation/transcoding of data• Allows more complicated routing/duplicating decisions• Full Web interface
OFTP (X.25, ISDN, TCP/IP)FTP (client or server)SFTP (client of server)AS2
OFTP (X.25, ISDN, TCP/IP)FTP (client or server)SFTP (client of server)AS2
Partner 2
Receiving messages
Able to use a variety of software or the web interface
EPIC File Processing Concepts
“Companies”
21
• EPIC allows both “Internal” and “Trading Partner” companies to be defined.• Files coming into EPIC are matched to an “Originator” company• Files are typically routed to a “Recipient” company• The Originator and Recipient company could be
• “Internal”• “Trading Partners”• One of each (most typical)
Workflows
22
• Every company has 2 default “workflows”• “Originator” – for when the company sends files• “Recipient” – for when the company receives files
• A workflow is a series of one or more “jobs”
• A job performs an action, for example:• Posting a file from an Originator company definition to a Recipient (this doesn’t actually send the
file)• Transmitting a files to a recipient company (e.g. using OFTP)• Translating a file to a different format• Sending an email• Etc. (93 job types currently)
Companies and Workflows
23
• A file received from TP1 is automatically put onto the “Originator” workflow for TP1• This workflow tells EPIC to post the file to the correct destination (TP2) based on routing
information provided during the OFTP connection• The file is then placed on the “Recipient” workflow for TP2 which tells EPIC to send the file
using OFTP 2
Example 2
EPIC receives a file from an external company via OFTP 1 and forwards the file to another external company via OFTP 2 (aka DiNet)
There are two trading partners defined (TP1 and TP2)
TP1 TP2
OFTP 1 OFTP 2
File Processing Overview
24
• The transition from Originator to Recipient
Recipient’sFile
Recipient’sWorkflow
Originator’sFile
Originator Originator’sWorkflow
Post to Recipient
File Received
Recipient File Sent
File Processing Overview
25
• The transition from Originator to Recipient
Recipient’sFile
Recipient’sWorkflow
Originator’sFile
Originator Originator’sWorkflow
Post to Recipient
File Received
Recipient File Sent
Could include• Transformation• Notifications
Should always post to a recipient
File Processing Overview
26
• The transition from Originator to Recipient
Recipient’sFile
Recipient’sWorkflow
Originator’sFile
Originator Originator’sWorkflow
Post to Recipient
File Received
Recipient File Sent
Could include• Transformation• Notifications
Will usually result in files being transferred from Epic to another system, either putting them in a folder or transmitting them in some fashion
File Processing Summary
27
• Each company can have their own specific processing• If one company wants files in EBCDIC and everyone else wants them in ASCII – only their
workflows need to be changed
• Allows reuse of workflows amongst companies who employ similar processes
• Companies typically need different processing based on whether they are sending or receiving files
Why separate originator and recipient?
Topology
Epic Topology
29
BatchAdministrator
SQL
File Directories
Comms links
Administrator
Workstation
ENGDATWorkstation
Comms Monitor
Licensing
Monitor
Clients and Server
30
Epic Applications
31
• Master Server – Performs all decision making and accepts all client connections
• Proxy Server – A server controlled by Epic, could be acting as a communication proxy
• Administrator – used to configure the system
• Workstation – used to view, control and audit files
• Communications Monitor – used to monitor and control connections
• Licensing - Used to license the product
• Epic Monitor – used to monitor the status of the Proxy Server within Epic – not covered in this course
Epic Server
32
• Central Server (Master)
• Able to run as non-interactive system service
• Cluster aware (allows automatic failover to a backup)
• Able to use a Proxy Server for communications
• Uses a scalable SQL Server database (all database access is from the Master server)
Administrator
33
Administrator - Configures the system• General system configuration• Trading partners• Workflows/Automation• Communications• Security• Retention periods• Logging levels
Workstation
34
Workstation• Trace files through Workflows• Shows a detailed audit trail• Provides a complete overview of a file’s process through the system• View comms file status (sent and received)• Re-queue comms files• Reprocess files• Manually submit files for processing• Search for files
Communication Monitor
35
Communications Monitor• Remote monitoring• List of all active and recent communication sessions• Allows calls to be made• Real time system log• Filter based on Trading Partner
Licensing
36
Licensing
• Used to license the product• Used to license product modules
Proxy Server
37
Proxy Server
38
SQL
File Directories
Commslinks
Epic Proxy Server
Epic Master Server
Public Network DMZ Corporate Network
EPIC Installation and Licensing
39
Installation
40
Install from media
• Choose Installation Path• Choose Epic Components• Choose SQL Express (if required)
Run Setup Wizard
• Choose Upgrade or Fresh• Choose Database• Choose Data Path
License• Enter Licensing Details
The Installation Process
41
Installation (1 of 4)
42
• Installation Executable is a self extractor• Select the items to install• After the program installation the Setup
Wizard will automatically run
Installation (2 of 4)
43
• Installation Executable is a self extractor
• Select the items to install• After the program installation the
Setup Wizard will automatically run
Installation (3 of 4)
44
• Installation Executable is a self extractor
• Select the items to install• After the program installation the
Setup Wizard will automatically run
Installation (4 of 4)
45
• Installation Executable is a self extractor
• Select the items to install• After the program installation the
Setup Wizard will automatically run
The Setup Wizard
46
The Setup Wizard
47
• Performs maintenance tasks on the Epic system• Initial Setup mode and Maintenance mode• Upgrade the Epic system• Move the main database• Move the data directories• Setup Server as Service or Application
Functionality
New Setup
48
The Setup Wizard
Setup Wizard - Fresh System (1 of 5)
49
• Select fresh installation• Use ‘Advanced Options’ to select
database if necessary and also the common and archive file directories.• Advanced: Select database server
• Advanced: Select data directory
• Installation summary screen• Installation progress
Fresh System
Setup Wizard - Fresh System (2 of 5)
50
• Select fresh installation• Use ‘Advanced Options’ to select
database if necessary and also the common and archive file directories.• Advanced: Select database server
• Advanced: Select data directory
• Installation summary screen• Installation progress
Fresh System
Setup Wizard - Fresh System (3 of 5)
51
• Select fresh installation• Use ‘Advanced Options’ to select
database if necessary and also the common and archive file directories.• Advanced: Select database server
• Advanced: Select data directory
• Installation summary screen• Installation progress
Fresh System
Setup Wizard - Fresh System (4 of 5)
52
• Select fresh installation• Use ‘Advanced Options’ to select
database if necessary and also the common and archive file directories.• Advanced: Select database server
• Advanced: Select data directory
• Installation summary screen• Installation progress
Fresh System
Setup Wizard - Fresh System (5 of 5)
53
• Select fresh installation• Use ‘Advanced Options’ to select
database if necessary and also the common and archive file directories.• Advanced: Select database server
• Advanced: Select data directory
• Installation summary screen• Installation progress
Fresh System
Upgrading
54
The Setup Wizard
Epic Setup Wizard - Upgrade (1 of 6)
55
• Select Upgrade From A Previous Version
• Use ‘Advanced Options’ to select database if necessary and also the common and archive file directories
• Choose the previous version of Epic to upgrade from
• Upgrade options, new database or in place upgrade• New database – SQL server• New database – data paths
• Upgrade summaryUpgrade
Epic Setup Wizard - Upgrade (2 of 6)
56
• Select Upgrade From A Previous Version
• Use ‘Advanced Options’ to select database if necessary and also the common and archive file directories
• Choose the previous version of Epic to upgrade from
• Upgrade options, new database or in place upgrade• New database – SQL server• New database – data paths
• Upgrade summaryUpgrade
Epic Setup Wizard - Upgrade (3 of 6)
57
• Select Upgrade From A Previous Version
• Use ‘Advanced Options’ to select database if necessary and also the common and archive file directories
• Choose the previous version of Epic to upgrade from
• Upgrade options, new database or in place upgrade• New database – SQL server• New database – data paths
• Upgrade summaryUpgrade
Epic Setup Wizard - Upgrade (4 of 6)
58
• Select Upgrade From A Previous Version
• Use ‘Advanced Options’ to select database if necessary and also the common and archive file directories
• Choose the previous version of Epic to upgrade from• Upgrade options, new database or in place upgrade
• New database – SQL server
• New database – data paths
• Upgrade summary
Epic Setup Wizard - Upgrade (5 of 6)
59
• Select Upgrade From A Previous Version
• Use ‘Advanced Options’ to select database if necessary and also the common and archive file directories
• Choose the previous version of Epic to upgrade from
• Upgrade options, new database or in place upgrade• New database – SQL server• New database – data
paths• Upgrade summaryUpgrade
Epic Setup Wizard - Upgrade (6 of 6)
60
• Select Upgrade From A Previous Version
• Use ‘Advanced Options’ to select database if necessary and also the common and archive file directories
• Choose the previous version of Epic to upgrade from
• Upgrade options, new database or in place upgrade• New database – SQL server• New database – data
paths• Upgrade summaryUpgrade
The Licensing Client
61
Licensing Client
62
• Epic requires a serial number and licence to run the product
• Other optional components require an additional licence
• For most communication types the user will have a unique ID, for which they will also need an authorisation code
• Codes expire after one year, with a one month grace period
EXERCISE
63
64
• Install Epic on your system• Configure to use the local SQL database• Run the licencing client and enter the codes given to you
Exercise
EPIC Configuration
65
Epic Administrator
66
The Epic Administrator is used to:
• Setup system configuration
• Setup trading partners
• Configure workflows
• Setup communications
• Configure security
• Define retention periods
• Manage logging levels
Configure the system
Epic Administrator
67
TaskSimple wizards that allow a user to perform some basic tasks
Connections Used to configure trading partners, subsystems and communication profiles
WorkflowUsed to setup automated file processing in Epic
SystemGeneral system configuration options
Configure the system
Configuring Communications
68
Basic Communications 3 Layer Model
69
• Every connection must have two sides, Ours is the Internal side, and Theirs is the External side• The PHYSICAL link is the connection between the two parties. First we must LOGIN to the
remote and exchange security information such as User IDs and Passwords• The MAILBOX then lets us EXCHANGE FILES for different parties over this PHYSICAL link• If the file exchanged is EDI DATA, it will contain routing information in its INTERCHANGE
header data. This can then be used to define the further handling of that file• If the file exchange is Non-EDI data, then only the top two layers can exist
PHYSICAL
MAILBOX
INTERCHANGE
PHYSICAL
MAILBOX
INTERCHANGE
LOGIN
EXCHANGE FILE
EDI DATA
Internal External
EPIC Protocol Chart
70
OFTP FTP SFTP EMAIL AS2
PHYSICAL SSID Login username and password Certificate Exchange
MAILBOX SFID GET from or PUT to directory Email Address N/A
INTERCHANGE UNB / STX / ISA
Direct Network Connections
71
• ABC LTD has 4 Trading Partners• Each trading partner requires a different file transfer protocol• ABC has therefore to test and maintain each connection individually• Even if two partners use the same protocol they will have different IDs, addresses, passwords
and other setup parameters
AS2
SFTP
OFTP
FTP
Clearing Centres
72
• ABC LTD has just one connection, to DINET VAN, but still has 4 trading partners• ABC can receive files from all 4 Trading Partners and will send back automatic
acknowledgements confirming his receipt. Over just one protocol.• On receipt of ABCs files, Trading Partners will return acknowledgements• ABC receives these acknowledgements when he next connects to DINET
INTERCONNECT
AS2
SFTP
OFTP
EPIC Communication Protocols Overview
73
Comms Architecture
74
All EPIC’s Communications are via a SubsystemThe Subsystem connects the Comms Protocol to the outside world
CommsWorkflows
Alerts
Internal Company NetworkTrading Partner Network
Subsystem
Epic’s Communications Methods
75
• Epic can use a rich variety of different communications protocols to transfer data. These protocols are:
• OFTP, The original OFTP and OFTP2, Encrypted and signed data transfer
• X.25 over ISDN (CAPI)
• TCP/IP (VPN, Internet, ENX etc.)
• X.25 (XOT or X.31)
• AS2 – Applicability Statement version 2• FTP and SFTP (Client and Server) – Internet File Transfer Protocol• Email Networks – Data transfer in email attachment or content
• The course will continue with basic OFTP communications (the most common) but other modules are available.
Back Office
76
• Protocols available against the Epic server • MQ Series – exchange files with IBM MQ Server• SAP R3 – exchange files with SAP using tRFC
• Not associated with a particular company• Nothing to prevent Epic talking to an “external” SAP or MQ system• Likewise, nothing to stop FTP being used to communicate with an “internal” system
Communications
Other
77
Workflow jobs• Write to file – outputs configurable data to a file on disk
• Copy – copies a file to disk
• Email – sends a email (with optional file attachment)
Data Sources• Directory – Epic monitors a local or network directory
• Batch interface – a command line interface is used to “give” Epic a file
Interactive• Workstation application – allows new files to be submitted to Epic and existing files to be downloaded
• Web “Dashboard” Interface – allows new files to be uploaded and downloaded
• ENGDAT Web Interface – allows ENGDAT packages to be created and uploaded or downloaded and extracted
Communications
Basic OFTP Concepts
78
OFTP Communications 3 Layer Model
79
• PHYSICAL = SSID Node (Start Session IDentifier) - Log on to the remote and agree to talk. Exchange SSID Codes and passwords
• MAILBOX = SFID Node (Start File IDentifier) – Transfer Files across the established Network Connection
• INTERCHANGE = EDI Node (UNB / ISA / STX) The Sender and Receiver in the EDI envelope segments
Internal Network
Internal Mailbox
Internal Interchange
Trading Partner/VAN Network
Trading Partner Mailbox
Trading Partner Interchange
SSID
SFID
EDI Code in UNB
Internal External
OFTP example via the DINET VAN
80
• PHYSICAL = SSID Node - Log on to the remote DINET VAN and agree to exchange files• MAILBOX = SFID Node – Transfer Files to the remote Trading Partner’s Mailbox through
DINET (DINET will re-send the file to the remote party)• INTERCHANGE = EDI Node (UNB / ISA / STX) The Sender and Receiver in the EDI envelope
segments
O09321234567
O09321234567
09321234567
DINET
O09327654321
09327654321
SSID
SFID
EDI Code in UNB
Internal External
Setting up OFTP Communications in EPIC
81
Subsystems and Listeners
82
• Each communications protocol needs a subsystem to connect it to the outside world
• Our OFTP example will use a TCP/IP subsystem, but could use several others
• To accept incoming calls the subsystem must have an active Listener
Setting up the Internal SSID
83
• The Internal Node requires an Authorisation Key for its SSID code calculated against the Epic Serial Number – This code is obtained from Data Interchange
• The Internal Node can be entered either from the Epic Administrator or from the Licencing client
• In the Administrator it is done from:Internal Companies -> <My Company> -> Networks -> Add -> New OFTP Network
Adding a Trading Partner
84
• Customer’s and Supplier’s direct connections are profiled under Trading Partners
• The top level of Trading Partner contains a list of that Partner’s codes under the EDI codes tab (UNB)
• A Trading Partner can have several types of communication (Network nodes)
• Any new SSIDs will also be automatically be profiled at SFID (Mailbox) level
Adding a Trading Partner
85
• Mandatory fields are marked in light blue.
• Epic will force you to enter an Originator Workflow and a Recipient Workflow, before you can save the trading partner.
• Only once the trading partner has been saved can you setup a Network for them.
Adding a Network To A Trading Partner
86
• Select the Networks Tab, click Add and select ‘New OFTP Network’
Adding an SSID
87
• OFTP Revision 1 or 2 (2 supports encryption)
• Passwords• Our Sent Password is the
remote’s Received and vice versa.
• Subsystem selection
• Host Address
• Network Connection
• SSL/TLS used?
Profiling Clearing Centres
88
• Partners communicating via a VAN (Value Added Network = Clearing Centre) should be profiled under the Clearing Centre Networks
• Otherwise VAN SSID setup is the same as Trading Partner SSID setup
• A VAN acts as a hub for many companies communications, so is not linked to a specific Trading Partner
Adding More SFIDs and EDI Codes (1 of 2)
89
• SFID (Mailboxes) can be maintained under the network’s Mailboxes tab
• EDI Codes can be maintained under the EDI Codes tab
• Each EDI Code (UNB) entry will have a selected SFID and SSID (File and Network) to communicate through.
• Each EDI Code may be profiled as in the UNB:
• EDICode – up to 35 characters• Qualifier – up to 4 characters• Routing Address – up to 14
characters
Adding More SFIDs and EDI Codes (2 of 2)
90
• Each EDI Code (UNB) entry will have an associated SFID and SSID (File and Network) to communicate through.
• Each EDI Code may be profiled as in the UNB:
• EDICode – up to 35 characters• Qualifier – up to 4 characters• Routing Address – up to 14
characters
Making a Test Call (1 of 2)
91
• From the Network Node Status tab select the Trace Call button
Making a Test Call (2 of 2)
92
• Set the tracing level to see the required comms detail
• Selecting Call makes a full call to the remote allowing data to be sent and or received
• Selecting Test Call makes a simple handshake with the remote over which no data passes in either direction, even if some has been scheduled
• Colour use• Black – Info• Blue - Protocol Trace• Red - Errors
Communication Monitor Client
93
Epic Communication Monitor
94
• Provides a live communications log
• Useful for problem solving• Top pane shows recent sessions
• Session views can be configured as in the Workstation
• Bottom pane shows the session in real time
• Sessions tabs shows past sessions
• Double click to see session log• Right mouse and Trace Call to
further diagnose problems• Comms sessions can be easily
exported for analysis
Tracing Calls
Epic Communication Monitor
95
• Tracing Calls
• Right mouse on existing session, Trace Call OR
• Use the Trace Call button
• Once the trace dialog is started, a different Network or additional tracing may be selected
• Colour Coding
• Red – Error messages
• Orange – Warning messages
• Blue – Trace detail messages
• Black – Normal information messages
EXERCISE
96
97
• Setup Internal and External SSIDs that allow an exchange of data between your Epic and another student
• Use the trace call function to make a call to your comms partner• Change the logging level to show more logging• Use the Communication Monitor to call your comms partner
Exercise – Communicate with your partner
Advanced Communication Features in EPIC
98
Multiple Connections
99
• Backup connections can be profiled in the Connections tab
• Epic tries backup connections by Priority. Use up and down arrows to change this
• Retries can be defined for each method
• Network Overview tab always shows primary connection method
Connection Retry Profiles
100
• Individual connection profiles have retry profiles if a connection fails
• Epic automatically retries a connection if an outbound call fails
• By default, Epic stops attempting to make calls if a connection reaches its retry limit (see next slide for advanced options)
Network Retry Profiles
101
• The network retry profiles are triggered when all of the Connections have reached their retry limit.
• Global Retry Profiles can be setup under the Global Connection Settings
Workflows
102
EPIC Workflow Architecture
103
CommsWorkflow
Alerts
Data Source
Data Definition
Workflow Selector Workflow
Workflow Overview (1 of 3)
104
• The core of the Epic system• Workflows control the whole file manipulation and forwarding process• A Workflow contains the Jobs that process the file• Each Job feeds its output file to the next Job
WorkflowJob
Copy Run Application
Map Email
Workflow Overview (2 of 3)
105
To process a file through Epic, generally it requires a minimum of:
• A Data Source – To match a file• A Workflow– To define the Jobs needed to process the file• A Workflow Selector – To choose the Workflow
Workflow Selector
DataSource
Workflow Overview (3 of 3)
106
• Files always go to the Workflow selected against the Company first (originator/recipient)
• If necessary, from the default workflow against an Origin/Destination, a file can be moved to a different Workflow using a Workflow Selector
Originator and Recipient workflows
Data Sources
107
Data Source
108
To process a file through Epic it requires a minimum of:
• A Data Source – To pickup the file• A Workflow – To define the Jobs needed to process the file• A Workflow Selector – To choose the Workflow
Workflows
Workflow Selector
DataSource
Data Source
109
Data Sources identify the origin of the incoming file
Several Data Source types:
• Directory Data Source • Which directory has the file been picked up from?
• Comms Data Source • Which trading partner has sent us the file?
• MQ Data Source • Which MQ queue has sent us the file
• Command Data Source• A method of manually sending a file to Epic (there are many other ways to do this now).
Workflows
Data Source
110
• The files picked up from the Source will be copied locally to Epic and then deleted
• Be careful what you select, especially if you monitor sub-directories
• Monitoring causes immediate pickup of files
• When running as System Service you MUST use a UNC path if the directory is on a network drive (\\<server>\<Resource>\....)
Data Sources
111
• Picks up files from a specified directory path
• Can be a static filename, or a filename mask
• DOS wildcards can be used in the name • * - Any number of any characters• ? - A single character
• Epic starts monitoring the directory as soon as it is saved
• Is linked to an Originator’s Workflow to determine which Workflow to process the file with in the first instance
Directory Data Source
Data Source
112
Files picked up from Directory Data Sources can be placed on either:• A company’s originating workflow• A specific workflow
The Directory Data Source
Data Sources
113
• Used to process files received via a Commsprotocol (OFTP, AS2, FTP)
• Optionally filters based on protocol origin/destination
• Optionally filters based on VFN/Filename Mask (OFTP/FTP)
Comms Data Source
Data Sources
114
• Used to process files received from an IBM WebSphere system (MQ Series)
• Reads files from a defined MQ (System -> Back Office Systems)
• Posts the file into an Originator Workflow
MQ Data Source
Data Definitions
115
Data Definitions
116
• EDI Data Definition• EDIFACT, ANSI X12, UNGTDI, VDA (Including
EBCDIC files)• Combines EDI standard and Trading Partner
• Non-EDI Data Definition• User defined Flat file• User defined CSV/delimited file• User defined XML file• Able to pick out standard information
(origin/destination/message type) and also any user defined fields
Workflows
117
Workflow
118
To process a file through Epic it requires a minimum of:• A Data Source – To pickup the file• A Workflow – To define the Jobs needed to process the file• A Workflow Selector – To choose the Workflow
Workflow Selector
DataSource
The Workflow
119
• Overview shows the jobs in this workflow
• Select Edit to change• Jobs tab allows detailed Job
maintenance• Each job performs a separate
process on the file, copy, schedule, convert file encoding, etc
• Each job sends its output to the next job
Selecting the Correct Workflow
120
Workflow Selection
121
• Data Sources – Where has the document come from?• Directory Data Source – Which directory was this file picked up from?• Communication Data Source – Which comms source was the file received over?
• Data Definition ties together, the Document Definition, the Trading Partner, the Document Encoding (ASCII, EBCDIC)• Document Definition specifies an EDI or Non-EDI File type, Invoice, ASN, etc. - What type of document
are we processing?• A Trading Partner contains one or more EDI Codes. - Who is this document from or to?
• File Type – Originator or Recipient File Type
EDI Codes
Originator/Recipient
Data Source
Data Definition
Document Definition
Trading Partner
Message /Record Type
WorkflowSelector
Workflow
The Workflow Selector (1 of 2)
122
• Selectors tie together Data Sources and Data Definitions then select which Workflow the file will pass through.
• The sequence of Selectors is critical – starting at the top of the list, the first Selector that fits will be the only one used
• More specific Selectors must be at the top and the less specific, “catch all” selectors at the bottom of the list
• Once a selector has been created, selecting Workflow Selectors/ (Ungrouped) allows you to re-sequence the selected item with the up and down arrows
The Workflow Selector (2 of 2)
123
• Workflow grouping is available at a global level
• (Ungrouped) selectors normally used initially
• A Match Workflow job can match to a further workflow, and can specify which Selector Group to use
• Selector Groups are used to create different order Selector lists for the Match to Workflow job
• If no matching selector is found in the specified Selector Group, (Ungrouped) is tested
Creating a Workflow Selector
124
• Right Mouse click Workflow Selectors / (Ungrouped) and select New Workflow Selector
• Give this Selector a meaningful name
• Select the correct Data Source (and Data Definition if necessary)
• Select the Workflow to be carried out
• An Optional Error Workflow can be set if needed
• This could typically send an email if problems occur whilst processing the file
125
• Create a directory on your C: drive with 2 subdirectories “Pickup” and “Drop”• Create a Data Source for any file in “Pickup”• Create a Workflow that has a single job that copies a file to the “Drop” directory to an output file
called “TEST.txt”• Create a Workflow Selector to tie the two together• Test the process by placing a file in the “Pickup” directory
Exercise – Pickup and Copy Files
Pickup Drop
Placeholders
126
Introduction to Placeholders
127
• A Placeholder is used to provide system variable information into workflows, for items like Filenames, Paths, Virtual Filenames and Email content
• Placeholders start and end with the percent % character
• There are hundreds of Placeholders in Epic, they are grouped into categories such as Comms, Workflow, Date/Time, etc.
Introduction to Placeholders
128
• To access Placeholders in Epic, either right mouse click or use the Insert button, then select the Category and then the actual Placeholder
• A common sequencing placeholder for the Copy workflow job is %FID% (File ID) which may be used to give a unique sequence number to output files
• A number can be added at the end of some placeholders to specify the placeholder length, so that %FID6% would be zero padded to 6 digits e.g. 000456
• Placeholders may be mixed with preset text to provide the required output e.g. C:\TEST\TEST%FID6%.txt
129
• Change your workflow Copy job to write out:• the workflow ID as a sequence number• a Hyphen (-)• the Original Filename with Extension
• Save and test your changes
Exercise – Amend and test your workflow Copy job
File Routing Jobs
130
Workflows
131
Post to Recipient • Used on an originator workflows (the
company from whom the file originated), this “posts” the file to the intended recipient (the file then gets processed on their “recipient” workflow)
• One the file has been moved to the ‘Recipient’ workflow it is marked as a recipient file, prior to this it is marked as an ‘Originator’ file.
Moving Files Between Origin and Recipient Workflows
Originator Workflow
WorkflowSelector
Workflow
Recipient Workflow
WorkflowSelector
WorkflowPost to Recipient
Job
Workflows
132
Move Company Workflow • Used on an originator or recipient
workflow. The Job moves the file from one company’s workflows to another company’s workflow.
• Depending the file type i.e. originator or recipient, the file is placed on the originating or recipient workflow respectively.
Moving Files Between Companies Originator Workflow
WorkflowSelector
Workflow
Recipient Workflow
WorkflowSelector
Workflow
Move Company Workflow
Job
Originator Workflow
WorkflowWorkflowSelector
Workflows
133
Schedule File• used on a recipient
workflow to “send” the file. This job tells Epic which network definition should be used and other salient information (eg the filename, priority etc)
Making Files Available
Originator Workflow
WorkflowSelector
Workflow
Recipient Workflow
WorkflowSelector Workflow
Schedule FileJob
134
• Create a directory on your C: drive called “Pickup”• Create a Global Data Source for any file in “Pickup”• Edit the originator workflow and add a Match Workflow job• Create a Workflow that has a single job that post the file to the recipient workflow. • Create a Workflow Selector to tie the two together
• Create a Workflow for the recipient that has a Schedule job• Edit the recipient workflow and add a Match Workflow job• Create a Workflow Selector for the workflow
Exercise – Pickup and Schedule Files
Workflow Jobs
135
Workflow Jobs
136
• Acknowledge• Forces Epic to send an acknowledgement for a received file
• Acknowledge Mailbag• Specific to Sterling Mailbag format files
• Add to Batch• Pauses processing of files (adding 1 or more to a “batch”), processing is triggered by a
scheduled event
• Analyse• Determines the file type, and possibly the origin/destination and other information if the
file type is recognised (preconfigured or user defined)
Workflow Jobs
137
• Call Network• Epic will make a call to the specified network
• Change State• Allows a file to be marked as “finished” without actually being fully processed
• Check File Exists• Checks a specified filename, the “return code action” is used to determine subsequent
processing
• Compress• Compresses a file using a ZIP archive format.
Workflow Jobs
138
• Condition• Checks a condition eg, %VFN%==TEST The return code action (true/false) controls
subsequent processing
• Convert Encoding• Performs tasks including Codepage conversion, insert/remove delimiters etc.
• Copy• Copies the current file to a location on disk (placeholders can be used to populate the
filename)
• Copy (with XML)• Only used when integrating with Darwin
• Create Functional Acknowledgement• Creates functional acknowledgement(s) based on the analysis of the input file
Workflow Jobs
139
• Create Mailbag• Only used with Sterling Mailbag format files
• Decompress• Extracts files from ZIP, RAR or TAR archives
• Edit Interchange• Modifies the interchange segment of the current file, including sender and recipient codes
(supports Edifact, VDA and X12)
• Extract Data• Extracts a subset of the current file (specified by a start and end position) and continues
processing it as a “new” file
Workflow Jobs
140
• Find & Replace• Can find data – using the return code actions to control logic flow based on the result• Can replace data either swapping strings or using regular expressions
• Insert Data• Inserts a string at a specified position
• Map• Invokes an Xe map to transform files between formats• The map can be set explicitly, taken from a relationship or derived from the Xe Index (a list
of applicable maps)
• Match Relationship• Allows a file to be “tested” to see whether there is a matching relationship for it. The
relationship can specific the origin/destination/message type and can return information to be used in subsequent processing (eg Xe map)
• The job return code can be used to control process flow (match found or not)
Workflow Jobs
141
• Match Workflow • Epic attempts to find a matching Workflow Selector for the current file• A workflow selector specifies a workflow and entry criteria (eg origin/destination)
• Move Company Workflow• Attempts to move a file to a different originator/recipient in the same fashion as Posting,
but not changing the file type (from originator to recipient)
• Post To Recipient Company Workflow• Changes the file type from an originator file to a recipient file• Assigns the file to a recipient company, either with a preset company specified or derived
automatically from information which Epic has available
Workflow Jobs
142
• Print Report• Sends a report file (created as a result of an Xe map) to a specified printer
• Process Batch• Processes a previously created batch if it matches specified criteria (minimum and
maximum file count)• Works with the Add to Batch job
• Process Functional Acknowledgement• Processes received functional acknowledgements and attempts to match them to existing
tracked documents.
• Process Mailbag• Only used for Sterling Mailbag format files
Workflow Jobs
143
• Run Application• Used to run an external application• Can pass the current file as an argument and pick up the output of the application as a
replacement
• Run Workflow• Epic will put the current file onto a new workflow either synchronously (will suspend
current processing) or asynchronously
• SAP (Associate)• When sending a file (outgoing) associating it with SAP means that SAP status messages
will be generated when the file is sent/acknowledged
• SAP (Export)• Causes a file to be sent to SAP
• Schedule ENGDAT Folder• Used to schedule ENGDAT folder files
Workflow Jobs
144
• Schedule File• Used to send files to a trading partner or clearing centre network• Sets the priority, destination, name etc• Can auto-detect the destination or have it set explicitly
• Schedule SFTP file• Used to schedule files to an external SFTP client (unlike FTP, this allows you to chose a
folder during scheduling)
• Set Address• Override the origin and destination details derived by the Analyse job• Also allows the document type to be set
• Set Document Function• Sets the document function (indication as to its purpose). This is visible in the
Workstation and can be used as a filtering criteria for jobs, workflow selectors and retention period profiles
Workflow Jobs
145
• Set Placeholder• Allows an arbitrary string to be associated with a file, workflow or globally• This can then be referenced and checked in any subsequent jobs
• Sign EDI• Sign the current EDI file with a security signature (EDIFACT v4 specific) – doesn’t relate to
the signing performed by OFTP 2 and AS2
• Split• Splits files containing multiple interchanges. Useful if the files contain interchanges
destined for multiple destinations
• Track Document• Tracks the documents within an EDI file so associated functional acknowledgments may
be used
• Update ENGDAT Folder• Add a file to an ENGDAT folder – ENGDAT is not covered on this course
Workflow Jobs
146
• Validate XML• Validates the current XML workflow file against a specified XML schema.
• Verify Signed EDI• See signed EDI
• Verify Signed XML• Validates the digital signature of the current XML file on the workflow
• Wait / Pause• Delays processing for a number of seconds. Can be used as part of a retry loop (e.g. if a
copy job fails, wait 60 seconds before retrying)
• Wait For ENGDAT Folder• Wait for an event to occur on an ENGDAT folder
Workflow Jobs
147
• Wait for Functional Acknowledgement• Pauses the processing of the current file on the workflow until a functional acknowledgement
has been received by the system
• Wait for Protocol Acknowledgement• This job suspend processing of the current file until it has be acknowledged, at which point
processing continues. By setting a timeout value, the return code action of “timeout” could trigger the file to be treated as in error (ie if it was not acknowledged in a certain time)
• Wait for Transmission• As above but waits for the file to be transmitted. Typically follows a Schedule File job.
• Wait For User• Causes a file to be suspended until a user checks it and choses one of the defined choices set
against the job• For example choices could be specified as retry and stop, if the user specifies continue the return
code action could jump to a previous job. If they select stop, the return code action could stop processing
Workflow Jobs
148
• Windows Application Log• Writes a string to the Windows application log, typically in the event of a file ending up in
error
• Write to File• Writes a string to a file on disk, could be part of an audit trail
• Write to Log• Writes a message to the Epic log. This allows the user to add their own business logic
logging to the Epic log
• Write to MQ Message Queue• Writes a file to a specified MQ Queue.
Job Selection Criteria
149
Job Conditions
150
• Return Code Actions• All jobs have multiple possible return codes, either a simple OK/Error or a more
complicated range (eg the Condition job has True/False, the Find/replace job has Found/not).
• Return code actions can continue, stop, change workflow, skip to a different job etc
• Data Definition Conditions• Runs this job only if the file being processed matches a specific Data Definition (usually
specifying file type/origin/destination etc)
• Data Source Conditions• Runs this job only if a more general workflow is processing a specific Data Source
(origin/destination, comms type)
System Settings
Retention Periods
151
Retention Periods
152
• How long does Epic keep data files?• Clean up sweep occurs very 15 minutes• Can be suspended for times when Backups may be running or processing is heavy• Controls:
• How long the file is in the database (details of when received, audit trail, etc.)• How long the Data file exists on the system (how long does Epic keep the actual data)
Retention Periods Tab
153
• Overview tab shows the current state of checking (Enabled or Disabled)
• On busy PCs a suspend period may be defined to avoid times when the server will be very busy
• Retention periods tab shows sequence of retention periods (see following slides)
Retention Periods - Workflow file status
154
• Status for workflow files:
• Finished – the file has been manually marked as finished – no further processing can be done
• Processed and Sent – The file has completed its workflows and for outbound the file has been sent
• Inactive – The file is still in a workflow but nothing has happened for n days
Retention Periods
155
• Enabled flag
• Move files to Archive after n days
• Remove from disk after n days
• Delete files from Database and disk after n days
Retention Periods - Criteria
156
• The Criteria tab selects what conditions apply to this Retention Period
• Like the Workflow Selectors, the first in the list that fits is the one that is used.
• Therefore the most specific must be at the top of the list
Retention Periods – Advanced Settings
157
• Number of days to keep log files
• Number of days to keep session Files
• Session files are cleaned up when the exchanged files expire on retention period
• This setting allows empty sessions to be archived then deleted
User Security
158
User Security – System Settings
159
• To stop unauthorised access to Epic clients
• To add detail to the audit trail of who performed certain actions
• Epic Users can be (encrypted details on the Database), or on Active Directory, or Both
User Security – Defining a user
160
• Whether Integrated or built in security is used, there must always be an Admin account with total rights
• Users or Groups of users may be defined
User Security – Granting Access
161
• Each User/Group has specified permissions on each Client Application
• Top pane shows max permission on client
• Lower pane shows permissions for each client function
User Security – Business Filters
162
• Business filters allow restrictions to certain business areas of Epic, for example:
• Company• Document Function• Message Type• ENGDAT Contact
Data File Encryption
163
Data File Encryption
164
• Epic supports the encryption of all data files stored by the Epic server on disk
• File encryption requires a component licence
• File encryption uses the Advanced Encryption Standard (AES) with a key size of 256 bits
• When enabled, any data files entering the Epic system, whether encrypted or not, are stored to disk in an encrypted form.
• The system decrypts files automatically as and when required, for example when copying a file to a directory on disk using a workflow
• If the component licence expires, Epic will still be able to decrypt existing files on disk, but will stop encrypting new files
• When enabled, File Auditing can track user’s access to files
Data File Encryption
165
Showing both Encryption and File Auditing set
Data File Auditing
166
Audit log shows user “Kev” has extracted and opened the file for testing
User now prompted for comment
Schedules and Actions
167
Alerts Architecture
168
Alerts
Schedule EventAction
CommsWorkflows
Schedules - Overview
169
• Schedules are just timers. Alarm clocks that trigger Event Actions (next slide)
• Some Schedules already defined, others may be added
Schedules – The Timers
170
• Styles allow the number of occurrences to be defined
• Schedule allows both a start date/time and also Server start-up criteria to be set
• The Reference Schedule section details the timer intervals
Event Actions
171
• Timed events - linked to Schedules
• System Events - Triggered by Epic Events
• Email Failed• File Not Sent• Unhandled Workflow Error• Comms – Network Retry
Limit Reached
• Actions• Call Network• Send Email• Poll Monitored Directory
The Log File
172
The System Log
173
• Logs are held in text files in:•
C:\ProgramData\DIP\Epic\<version>\Application Data\Log
• If log file becomes larger than 10Mb, the old one remains at that size and a new one is started
• Logs are subject to the Retention Periods
• Monitor tab shows the live rolling log
System Log - Archive
174
• Archive allows you to pull back log entries for selected dates/times
• This extraction may then be saved as another file or emailed
System Log - Control
175
• System Logging tab allows us to control the level of specific types of logging
• Do not tick too many boxes, especially in the Global section, as the log will quickly become huge
• Target the tracing to the category of logging that you need to solve the problem
• Too much logging is as bad as too little
System Log - Communications
176
• Comms Logging – Control the level of specific communications logging
• Low Level (Base protocol) –CAPI2, HTTP, TCP, XOT
• Higher Level (File Transfer Protocol) – AS2, FTP Client, FTP Server, SFTP Server, OFTP
Server Status
177
Server Status - Overview
178
• Shows status of each function of Epic
• Allows elements of the system to be stopped for maintenance
Server Status – Startup Procedure
179
• Elements may be permanently disabled in every startup, or just in the next startup
Server Status
180
• This tab selects the shutdown procedures that occur when “Suspend” is selected on the Overview tab• “Controlled” means complete any
current workflows or comms sessions, etc but don’t start any more
• Forced means stop Now. Workflows will resume from last job when Epic is restarted
Miscellaneous Items
181
Client Server Communications
182
• System Settings, Server tab
• Default TCP port is 3547• May be changed here if it conflicts
• SSL (encrypted) port is 3549• Used when the clients are
connecting over a public network but without a VPN
• Requires a certificate on the Epic Server
• Client Authentication can be set up for additional security
• These ports must be allowed through any firewalls between client and server
183
• System, E-mail – Allows setup of the SMTP connection to the mail server
• SMTP Server must allow relay
• Test email should be sent before saving settings
• Emails can then be used in Workflow jobs and Event Actions
Monitoring the System
184
Communication Monitor
185
Communications Monitor
186
• Shows sessions (filtered and searchable)• Current (active)• Finished sessions• Archived sessions
• Real time protocol tracing
• Calls can be initiated from this client• Can start a call with extra tracing
Communications Monitor
187
Communications Monitor
188
Workstation
189
Workstation
190
Trace files through the system
All events performed against a file are logged
Originators Files Tab• Shows actions performed by an Originator’s Workflow
Recipients Files Tab• Shows actions performed by Recipient’s Workflow
Consolidated Files Tab• Shows all actions performed on a file by both the
Originator’s Workflow and Recipient’s Workflow
Error Files Tab• Shows files that have been marked as in error
Views
Workstation
191
File ViewEvery file processed by Epic is record in the File View
Workflow File View
File Audit ViewThe audit trail is populated whenever you click on a single file in the upper section. The audit contains details of the comms session as well as details of the workflow processing
• Click on a file record to view its audit trail
Workstation
192
Trace files through the system
All events performed against a file are logged
Received Files Tab• Shows files that have been received by Epic via a
communication protocol
Scheduled Files Tab• Shows files that have scheduled by Epic using the
Make Available job
Sent Files Tab• Shows files that were on the Scheduled Files tab, but
have now been sent
Session Tab• Shows communication sessions that Epic has made
or has received
Views
Workstation
193
• File received and Matched to Originator Company• Appears on Originators Files view
• Files received by Epic • Audit of actions performed by the originator’s workflow
• File is then passed to a Recipient’s Company• Appears on Recipients Files view
• Files sent by Epic• Audit of actions performed by the recipient’s workflow
• All files should appear on both the Originators and Recipient views• Consolidated files view shows a file going through the system (effectively a merged view
of the Originator and Recipient files)• Selecting “Archive” view shows old files
Workstation
194
• Refresh This View
• Filter the view origin/destination, date, status, VFN etc.
• Search for specific files based on VFN, file ID, EDI code etc.
• Switch between ‘Live’ and ‘Archive’ view
• Manual file submit
• File details – communication, basic analysis, size etc.
• Call a network
• Print a report for individual or groups of files
• SFTP server interface
• Help
• Exit
Buttons
Workstation
195
Filtering the view• Select the filter• Change the origin• Select a date range• Choose File Status
Tracing Files
Origin/Destination
Date
Processing State
Workstation
196
Advanced Filter Settings• Time setting• Error status• File status• Maximum returned results
Tracing Files
Error Status
File Status
Maximum Results
Date
Workstation
197
• Searching for files• From any view:
• Select the search option• Enter a Unique File ID• Or Enter other information
• Name (VFN)• Interchange Control ref• Origin UNB code• Destination UNB code• Message type • Document number (BGM)
Tracing Files
Workstation
198
• Click on a file record to view its audit trail• Sample file on the Scania Epic system (received from Amtrix, destined for a supplier)
Tracing Files – EDI file Received via OFTP
File Record
Audit Trail
Tracing Files
199
• All files from Trading Partner A are placed on their originating workflow• Then posts the file to a specific recipient using a fixed destination• The Audit trail shows the destination company and the File ID (which can be used to
search)
EDI file Received via OFTP
Record of comms details
Posts file to
RecipientFile ID of
Recipient’s File
Tracing Files
200
• File processed by My Company’s Recipient workflow• Workflow selector used to match to a specific ‘Copy Inbound Data’ workflow• File extracted using a copy job.
EDI file Received via OFTP
Match Workflow Job
File Copied To Disk
General
201
• Remember the right mouse click!• Most entries in list views allow actions to be performed by right-clicking• Allows copy/paste of most list view entries in the Administrator
• Context sensitive help on every screen
The End
202