Embed Size (px)
Citation preview
ASPERA HIGH-SPEED TRANSFER SOFTWARE Moving the world’s data at maximum speed
PRESENTERS AND AGENDA
PRESENTER
Michelle Munson Aspera President and CEO [email protected]
• Aspera Overview
• Aspera Technology
• Use Cases in Life Sciences
• Aspera + Intel
• Product Overview and Demos
AGENDA
2
WHO IS ASPERA?
Software technology company innovating new data transfer solutions
Based in Emeryville California
Founded in 2004, privately held, now with 145 employees
Creators of the fasp™ protocol
• Innovative, patented, highly efficient bulk data transport technology
• Unique and core to Aspera’s high-performance file transfer software suite
• Outperforms software and hardware WAN acceleration solutions
• Ranked first in every WAN transfer throughput benchmark
Patents: fasp Bulk Data and Dynamic Bandwidth Control issued in USA and 30 other countries, others pending in over
32 countries
Growth: over 2,100 customers, over 18,000 licenses, 50% year-over-year growth
Markets Served: Media and Entertainment, Federal Government, Life Sciences, Healthcare, Cloud Computing,
Software and Gaming, Financial Services, Legal, eDiscovery, Engineering, Technology, Telecommunications, Service
Providers, Architecture and Design, Enterprise IT
Global 24x7 Support: Support and sales offices in Sophia-Antipolis, Singapore, Virginia US, and Direct Sales and Sales
Engineering throughout globe
3
ASPERA’S MISSION
Creating next-generation transport technologies
that move the world’s digital assets at maximum speed,
regardless of file size, transfer distance and network conditions.
TRENDS IN TECHNOLOGY
Big Data Explosion
• 90% of data today file-based or unstructured
• Mix of file sizes—but larger and larger files the norm
Diversity of IP Networks—Media, Bandwidth Rates, and Conditions
• Variable bandwidth rates (slow to super-fast)
• Bandwidth rates increasing—costs decreasing
• Network media remains diverse (terrestrial, satellite, wireless)
• Conditions vary—all networks prone to degradation over distance
Global Workflows—moving Big Data over WANs
• Teams are geographically dispersed
• Over distance, network conditions degrade
• Contemporary TCP acceleration solutions not designed for big data transfer and replication
Cloud Computing Grows Up
• Amazon Web Services (AWS) S3 cloud storage – 2010: 262 billion objects, 2012: 1.3 trillion objects
• More choices: Microsoft Azure, OpenStack, HP Cloud
• No longer a niche – Netflix (transcoding), MTV (global video distribution), BGI (genomic sequencing)
5
WHAT CONSTITUTES “BIG DATA”
6
SAMPLE LIFE SCIENCE CUSTOMERS
Transcoding
Public and Private Research Institutions
Pharmaceutical Companies Public Data Sources/Repositories
BIG DATA CLOUD – WHAT’S THE PROBLEM?
8
CHALLENGES WITH TCP AND ALTERNATIVE TECHNOLOGIES
Distance degrades conditions on all networks
• Latency (or Round Trip Times) increase
• Packet losses increase
• Fast networks just as prone to degradation
TCP performance degrades with distance
• Throughput bottleneck becomes more severe with
increased latency and packet loss
TCP does not scale with bandwidth
• TCP designed for low bandwidth
• Adding more bandwidth does not improve throughput
Alternative Technologies
• TCP-based - Network latency and packet loss must be low
• Modified TCP – Improves TCP performance but insufficient for fast networks
• UDP traffic blasters - Inefficient and waste bandwidth
• Data caching - Inappropriate for many large file transfer workflows
• Data compression - Time consuming and impractical for certain file types
• CDNs & co-lo build outs - High overhead and expensive to scale
9
FASP™ — HIGH-PERFORMANCE DATA TRANSPORT
Maximum line-rate WAN transfer speed
• Transfer performance scales with bandwidth independent
of transfer distance and resilient to packet loss
• Optimal end-to-end throughput efficiency
Congestion Avoidance and Policy Control
• Automatic, full utilization of available bandwidth
• On-the-fly prioritization and bandwidth allocation
Uncompromising security and reliability
• Secure, user/endpoint authentication
• AES-128 cryptography in transit & at-rest
Scalable management, monitoring and control
• Real-time progress, performance and bandwidth utilization
• Detailed transfer history, logging, and manifest
Enterprise-Class File Delivery
• Transfers up to thousands of times faster than FTP/HTTP(S)
• Precise and predictable transfer times
• Extreme scalability (concurrency and throughput) 10
FASP 3 – UNIVERSAL HIGH-SPEED DATA TRANSPORT
CLOUD HYBRID ON PREMISE
fasp 3 DISTANCE INDEPENDENT
INFRASTRUCTURE AGNOSTIC
Locate your big data anywhere and move it
seamlessly with high-speed, secure transfers
FILE SIZE INDEPENDENT
Maximum transfer speed, optimal
bandwidth utilization and maximum I/O
throughput on any storage platform
regardless of number or size of files ALL PARADIGMS & DEPLOYMENT MODELS
Transfer, replicate or synchronize your data in all
types of scenarios: one-to-one, hub and spoke,
unidirectional and multi-directional
FASPTM – PERFORMANCE BREAKTHROUGH
FTP Across US US – EU US – ASIA Satellite
1 GB 1 – 2 hrs 2 – 4 hrs 4 – 20 hrs 8 – 20 hrs
10 GB 15 – 20 hrs 20 – 40 hrs Impractical Impractical
100 GB Impractical Impractical Impractical Impractical
fasp™ 2 Mbps 10 Mbps 45 Mbps 100 Mbps 200 Mbps 1 Gbps
1 GB 70 min. 14 min. 3.2 min. 1.4 min. 42 sec. 8.4 sec.
10 GB 11.7 hrs 140 min. 32 min. 14 min. 7 min. 1.4 min.
100 GB 23.3 hrs 5.3 hrs 2.3 hrs 1.2 hrs 14 min.
Aspera transfer times shorten linearly with bandwidth
Independent of packet loss, delay (network distance) Cross US – Add 1% to 5% Intercontinental – Add 1% to 10% Satellite – Add 1% to 10%
TCP transfer times limited by packet loss, delay (network distance) NOT BANDWIDTH
12
HIGH SPEED TRANSFER TO CLOUD STORAGE WITH DIRECT-TO-CLOUD
Cloud
13
CUSTOMER CASE STUDIES
Challenge
• Moving and sharing genomic research data to investigators and research scientists worldwide
Solution
• Aspera Connect Server embedded in “EasyGenomics™” to make NGS analysis easier and faster
Results
• Transferred 24 Gigabytes of genomic data from Beijing to California in under 30 seconds
• Eliminated the costs and risks of shipping hard drives around the world
Challenge
• Moving large genomics data from clinical providers to GenoSpace’s cloud-based analytics platform running on
Amazon Web Services
Solution
• Aspera On Demand integrated with the GenoSpace portal, moves genomic data to the cloud at high speed,
maintaining patient privacy and security with encryption in transit
Results
• Cloud-ready solution saves the cost and implementation effort of infrastructure build-outs.
• Aspera will scale to accommodate the large and ever-growing genomics data that GenoSpace receives
14
High speed connection
Web portal, Simple UI
Database Computational resources
Algorithms workflows
EasyGenomics™ is a cloud-based software as a service (SaaS) platform for bioinformatics data analyses
Cloud-based solution for data analysis
CUSTOMER CASE STUDY: BGI
Challenge • Sharing large volumes of data between internationally
dispersed collectors, data analysers and researchers
• BGI developed cloud-based bioinformatics platform
“EasyGenomics™ to provide scientists with high-speed
data exchange, point-to-click data analysis workflows
including whole genome resequencing, targeted
resequencing, RNA-Seq, small RNA and Denovo assembly
• BGI needed a high-speed file transfer solution that would
allow users to rapidly upload sequencing data to the cloud
for processing and then quickly download completed
projects, speeding up the data analysis process for
customers
High speed connection
Web portal, Simple UI
Database Computational resources
Algorithms workflows
EasyGenomics™ is a cloud-based software as a service (SaaS) platform for bioinformatics data analyses
Cloud-based solution for data analysis
CUSTOMER CASE STUDY: BGI
Solution • BGI deployed the Aspera Connect Server to expand
BGI’s high-speed sequencing data delivery service
• Integrated Aspera fasp™ high-speed file transfer
technology with EasyGenomics using the rich APIs
available in the Software Development Kit (SDK) and the
Aspera Developer Network (ADN)
• Provided a web-based file transfer framework that
served as the foundation for the EasyGenomics website,
delivering maximum throughput data movement over any
distance with end-to-end security and exceptional
bandwidth control, regardless of fluctuating network
conditions
June 2012: BGI Demonstrated Genomic Data Transfer at Nearly 10 Gigabits per Second Between US and China
ACHIEVING 10GBPS+ TRANSFERS OVER GLOBAL WAN
Joint research project between Aspera and Intel Corporation
• Investigate high-speed (10Gbps and beyond) data transfer solutions
• Leverage Intel® Xeon® processor E5-2600 product systems and Aspera’s fasp transport
Hardware Platform
• Intel® Data Direct I/O Technology (Intel® DDIO) which allows Intel® Ethernet controllers to route I/O traffic directly to the
processor cache
• Built-in support for Single-Root I/O Virtualization (SR-IOV) which allows virtual machine platforms to bypass the hypervisor
in order to directly access resources on the physical network interface
Results
• 300 percent throughput improvement versus a baseline system that did not contain support for Intel® DDIO and SR-IOV,
showing the clear advantages of Intel’s innovative Intel® Xeon® processor E5-2600 product family.
• Similar results across both LAN and WAN transfers confirming that Aspera fasp transfer performance is independent of
network latency and robust to packet loss on the network
• Approximately the same throughput for both physical and virtualized computing environments
0
2
4
6
8
10
2 3 4Th
rough
put(Gbps)
NumberofSessions
AsperaTransferThroughput--Mul pleSessions
DDIO(LAN)
DDIO+SR-IOV(LAN)
DDIO+SR-IOV(50ms-0.1%PacketLoss)
DDIO+SR-IOV(100ms-1%PacketLoss)
DDIO+SR-IOV(500ms-5%PacketLoss)
0
2
4
6
8
10
Baseline DDIO(LAN) DDIO+SR-IOV(LAN)
DDIO+SR-IOV(50ms-0.1%PacketLoss)
DDIO+SR-IOV(100ms-1%PacketLoss)
DDIO+SR-IOV(500ms-5%PacketLoss)
Throughput(Gbps)
AsperaTransferThroughput--SingleSession
17
Results available: asperasoft.com
Hardware
• Intel Xeon processor E5-2600 family (dual
2.70GHz CPU, 2x16 cores with Intel®
HyperThreading Technology)
• NUMA and Intel DDIO enabled as default
• Intel Corporation 82599EB 10-
Gigabit SFI/SFP+ Network Connection
HIGH-SPEED TRANSFER SOFTWARE Moving the world’s data at maximum speed
ASPERA PRODUCT PORTFOLIO
High-speed transfers for web, desktop and mobile
High-speed file transfer servers for on premise, private, public, and hybrid cloud deployments
File sharing, collaboration and exchange applications
Transfer management, monitoring and automation
Scalable, multi-directional, multi-node synchronization
TRANSFER
CLIENTS
WEB
APPLICATIONS
MANAGEMENT &
AUTOMATION
SYNCHRONIZATION
TRANSFER SERVERS
Innovative, patented, highly efficient bulk data transport technology, unique and core to all Aspera products
FASP™ TRANSPORT
19
FASP™ SOFTWARE ENVIRONMENT
20
ASPERA DEVELOPER NETWORK
A complete set of SDKs with guides, reference information, and sample code to assist them with integrating Aspera technology into their own applications. Aspera fasp™ technology can be used in desktop, network-based, and web applications in place of FTP, HTTP, or custom TCP-based copy protocols.
ASPERA MOBILE APIs
Android SDK Aspera Android SDK provides a Java API to transfer files using fasp-
AIR™.
iPhone SDK Aspera iPhone SDK with Objective C API to transfer files using fasp-AIR.
ASPERA APPLICATION APIs
faspex™ Web API The Aspera faspex Web API provides a set of services that enables users
to create and receive digital deliveries via a Web interface, while taking
advantage of fasp high-speed transfer technology.
ASPERA TRANSFER APIs
Aspera Web Services A SOAP based web service API that allows initiation,
monitoring and controlling of fasp based file transfers.
Aspera Web Javascript API exposed by Aspera Connect client. It allows
integration of fasp based file transfers into web
applications.
Connect 3.0 – New, Enhanced Javascript API Introducing the new Connect 2.8 developer preview!
Integrate the functionality of Aspera Connect 2.8, a fasp-
based file transfer client, into your own web applications,
while customizing it to your unique brand.
fasp Manager A class library that allows intiations, monitoring and
controlling of fasp based file transfers.
Aspera Multicast SDK A Java class library that allows initiation and management
of IP multicast based data transmissions using Aspera
fasp-MC™.
Console API The Aspera Console API provides full programmatic management of
transfer sessions including initiation, queuing, management and control
through a RESTful API.
Shares API The Aspera Shares API provides full programmatic control over browsing
Shares, transfer authorization, and upload / download
ASPERA ENTERPRISE SERVER UNIVERSAL FILE TRANSFER SERVER
Key Features and Benefits
• Universal base transfer server
• High-speed fasp™ transport of large data sets
• Enterprise-wide high-volume content ingest and distribution
• Comprehensive automation and scheduling
• Advanced user and server management
• Complete monitoring and reporting
• High-availability configuration (active/active or active/passive)
• Deployable on premise or in the cloud (public, private, hybrid)
Applications and Use Cases
• Enterprise-wide file movement
• High-volume content ingest
• High-performance content distribution
Licensing and deployment
• Multi-tiered licensing based on bandwidth capacity
• Usage-based licensing with Aspera On Demand
• High-availability configuration (active/active or active/passive)
• Comprehensive platform support (Unix, Linux, Windows, Mac, Solaris, Isilon OneFS)
• Supports connections with Aspera Client, Point-to-Point and other Enterprise Servers
22
ASPERA CONNECT SERVER APPLICATION PLATFORM WITH WEB INTERFACE
Key Features and Benefits
• Built on top of the Enterprise Server
• Application platform with Web interface
• Simple web portal exposes root directory for each user
• Transfers via Aspera Connect web browser plug-in
• Powers fasp™ transfers for Shares and faspex web applications
• Rich APIs enable integration into any custom web application
Applications and Use Cases
• Global browser-based file exchange
• Field and third-party content or data gathering
• Integration with custom web applications
Licensing and deployment
• Multi-tiered licensing based on bandwidth capacity
• Usage-based licensing with Aspera On Demand
• Comprehensive platform support
• High-availability configuration (active/active or active/passive)
• Supports connections with Aspera Connect browser plug-in client
• Automatic installation of Aspera Connect browser plug-in on the client side
23
ASPERA CONNECT BROWSER PLUG-IN
Key Features
• High-performance fasp™ transport
• Supports for all major browsers including Chrome, Internet Explorer,
Firefox, Safari on Windows, Mac, and Linux (native 64 and 32 bit)
• Upload/download client transparently launched by browser
• File, directory, and multi-item transfers with queuing
• Automatic retry and resume of partial or failed transfers
• Built-in transfer monitor for visual, on-the-fly transfer rate control and
monitoring
• Comprehensive transport and content security, integrated with at-rest
encryption, easy one-click decryption of files and directories
• Automatic inline installation from the web browser with no administrative
rights required
• Rich Javascript API for fully-integrated branded experience, persistence
across host reboots, and in browser control
• HTTP fallback mode for highly-restrictive networks with support for
authenticated proxies
• Language localized for Japanese, Chinese, French and Spanish
Application examples
• Upload / download client for Connect Server and Aspera faspex™
Server
• Browser based, self installable plug-in enables 3rd party access to
content 24
MULTI-NODE FILE SHARING Aspera Shares
ASPERA SHARES – MULTI-NODE FILE SHARING
Key Features
• Web application provides secure access (at global distances) to a
consolidated view of all shared content
• Fully integrated Aspera Connect provides maximum speed, secure
upload and download of files of any size, at any distance in a
lightweight, simple browser plug-in
• Content can be deployed across enterprise servers, private, public
and hybrid clouds
• Powerful security model administered through a single
management point combining authorization, user management,
and access control
• Search, filtering, and sorting capabilities makes it easy to find
individual files or folders in a very large content store
• NEW: Shares are fully integrated as secure, access controlled
receiving and publishing points
• Files and directories can be published as faspex packages
from any Aspera “Share” to which the user has access.
• faspex packages can be delivered directly to any “Share”
Application examples
• Internal distribution of digital assets within the enterprise
• Collaboration enablement for geographically-distributed teams
• Project team and third-party content or data gathering
• Global browser-based distribution of data across external teams 26
ASPERA SHARES FEATURES AND BENEFITS
Powerful security and access model
• Secure access administered through a single management
point combining authorization, user management, and access
control
• Add and manage users and groups locally, as well as through
Active Directory and LDAP
• Complete control over access, such as which nodes are
visible, and which nodes, directories and files are accessible
• Granular control over all end-user operations such as
browsing, uploading, downloading, making new directories,
renaming or deleting files and directories
Easy to use Web interface
• Users can easily navigate across files and folders to locate
and initiate a high-speed file upload or download
• A single view consolidates all shared content making it easy
to search and navigate across multiple nodes
• Powerful search, filtering, and sorting capabilities simplify
finding files or folders in a very large content store
• Operations can be performed on a single file or folder, or
directly on the search results
• Easy sharing via simple email
27
ASPERA SHARES TECHNOLOGY ADVANTAGES
Extreme Scalability
• Web application is decoupled from the content stores and underlying transfer servers or nodes
• Application can be installed on the same node, or on a separate server supporting multiple nodes with a single view of the
shared content across those nodes
• Supports transfers of a singe file, individual directories, or batches of files and directories of any size over any distance
Location independence
• Content and transfer nodes can be geographically dispersed across regions, countries, or continents
• Nodes can be deployed in private data centers, public or private clouds, or any combination, with a single Web interface
consolidating all content
Flexible Deployment Options
• On-premise, with traditional perpetual software license model
• In the cloud, with Aspera On Demand, as part of a subscription service with a usage based model
• Both software deployments can share content from transfer nodes that are deployed on-premise and in the cloud (hybrid
Easy User Management and Powerful Access Control
• Users and groups can be in AD, LDAP or local (and directories can be “anywhere”)
• Users can have any number of virtual directory access points with individual access control (upload, download, etc.)
28
fasp
Shares
Node
Node DMZ
fasp
Datacenter, Emeryville, CA
Client, NY, NY
THE SOLUTION
• Shares Web app transparently communicates
with Aspera server Nodes and displays
content in a single user interface
• User browses authorized content across
multiple shares
• Independent high-speed data transfers to/from
Datacenter, AWS S3, and Windows Azure
BLOB, transparent to user
USE CASE: SHARING AND COLLABORATION HYBRID ACROSS PUBLIC & PRIVATE CLOUDS
29
fasp
ASPERA SHARES - POWERFUL ACCESS CONTROL
30
ASPERA SHARES - GROUP MANAGEMENT
31
SAML FEDERATED IDENTITY MANAGEMENT
32
ASPERA CONSOLE Network-Wide Transfer Management, Monitoring, and Control
ASPERA CONSOLE INTRODUCTION
Challenges
• Effectively manage multi-node Aspera transfer environment
• Provide different system administrators with different access
rights to nodes and transfers
• Automate routine repeating transfers
• Transfer reporting and statistics for billing or auditing
Aspera Console
• Web-based application accessible via any standard browser
• Consolidated dashboard and detailed drill-downs to monitor
and control transfers in real time
• Automated one-time or recurring Aspera transfers including
multi-point "Smart Transfers”
• Remote configuration of nodes’ properties and users
• Custom profiles for each Console user
• Comprehensive, centralized transfer history database
customizable reports
• Manage any Aspera
o Enterprise Server
o Connect Server
o Point-to-Point Client
• Monitor and control Aspera Sync transfer sessions
34
ASPERA CONSOLE NEW FEATURES AND CAPABILITIES
Aspera Console 2.0
• New look and feel with sortable columns,
searchable pages and improved navigation.
• New Transfer Queuing service with on-the-fly
reordering and reprioritization control for
transfers on managed nodes via Smart Transfers
and new Console API.
• New job-level API for comprehensive transfer
initiation, queuing, management and control
through a RESTful API.
• Fully integrated Aspera Sync reporting including
synchronization progress status, current session
and cumulative statistics on sync’d files and
bytes.
• Integrated reporting (session views, email
notification, reports) of on-the-fly md5sums for
delivery confirmation.
• Integrated AD/LDAP support and event-based
audit logging.
• Enhanced performance specifically for workflows
with hot folder transfers.
• Full support for monitoring and managing iTunes
Transporter transfers.
35
ASPERA CONSOLE MONITOR WITH THE DASHBOARD
36
Dashboard
• Web-based console application
• Centralized view across all managed nodes
• Secure SSH tunnels for automatically connecting to
nodes
• Map view showing location and transfers between
managed nodes
• Graphical view of inbound and outbound bandwidth
usage
• Tabular view of all transfer activity
Activity
• Active, scheduled, recent, and problem transfer
activity
• Real time data for active transfers
• Track individual site transfers
• One click access to detailed information about each
transfer activity and statistics
ASPERA CONSOLE TRACK INDIVIDUAL TRANSFERS
37
Activity view
• See each transfer’s status and performance
• Sort by name, node or time started/completed
• Filter by time completed, time scheduled or by status
• Search for transfers by transfer name, file name or contact
• Drill into each entry for detailed transfer information
ASPERA CONSOLE ADVANCED AND CUSTOM REPORTING
Project-specific reporting and transfer history
• Choose from a large library of simple and advanced built-in
reports
• Reports can be organized, sorted and filtered by virtually any
transfer attribute
• Report templates may be scheduled for automatic runs and
email delivery
Flexible scheduling
• Run immediately, or schedule reports to run at a later date
Extraction options
• Distribute reports via email
• Export to CSV or XML
Custom reporting fields with custom tags
• Custom categories can be derived from metadata assigned to
transfers within Aspera applications
Automatic backup and data purge
• Create scheduled backups of console configuration and history
data
• Clean up Console data by status and age from the web UI
38
ASPERA CONSOLE ADVANCED USER AND NODE MANAGEMENT
39
Manage policies by groups and users
• Define transfer settings and authorization policies by user,
including bandwidth caps and priorities, encryption and
security settings
• Manage policies by user group, including Active Directory,
local Windows and Unix groups
• Filter Aspera Console capabilities by user, group and transfer
path, including transfer activity monitoring, bandwidth control,
and email notifications
Advanced node configuration
• Configure all node properties including bandwidth controls,
encryption settings, directory creation masks, and email
notifications
• Configure manifest files, exclusion rules and partial file
extension
• Configure security requirements and settings
ASPERA CONSOLE EMAIL NOTIFICATIONS
40
Define email templates
• Use default templates
• Completely change the look and feel with
custom templates
• Plain text or HTML
• Custom fields help create useful,
descriptive notifications
Associate with activities
• Trigger email notifications from various
events
o Transfer starts
o Successful transfer completions
o Error conditions
Associate by nodes or by transfer paths
• Specific sources
• Specific destinations
• Activity types (start, completion, error)
ASPERA CONSOLE ADVANCED BANDWIDTH MONITORING & MANAGEMENT
41
Real time control over active transfers
• Monitor active transfers
• Adjust bandwidth utilization and transfer priority on
the fly in real time
Server-side settings
• Bandwidth preferences and limits
• Override client policies
• Aggregate per-node or per-user groups using
VLinks
Configuring VLinks
• Aggregate bandwidth targets
• Including static and scheduled (time-varying) values
• Using differential bandwidth priorities within a VLink
ASPERA CONSOLE TRANSFER CONTROL & AUTOMATION
Ad-hoc transfers
• Remotely browse nodes
• Initiate ad-hoc transfer jobs from any source to any
destination
Automate distribution with "Smart Transfers”
• Completely control all aspects of the transfer
o Source, connection, transfer rate, security, file handling
and notification
• Support single and multi-point distribution
• Fine-grained, to-the-minute scheduling of Smart
Transfers with a repeating schedule
• Keep Smart Transfers private or share with other Console
users
Transfer Processing
• Automatic forwarding
• Custom pre- and post-processing functionality on
managed nodes
• NEW QUEIING SERVICE
42
ASPERA CONSOLE DEPLOYMENT
43
Platform Support
• On-premise installations available for Windows and
Linux
• Included with Aspera On Demand packages for AWS
o Base Transfer Server
o Application Platform packages
Supports management of
• Aspera Enterprise Server
• Aspera Connect Server
• Aspera Point-to-Point Client
• Aspera Sync
o Monitor status and performance statistics of sync
sessions
o Per file/directory synchronization status, conflicts,
errors and pending transfers
aspera sync multi-directional synchronization
aspera faspex server collaborative file transfer
aspera console network-wide transfer
management
aspera enterprise server universal file transfer server
aspera connect serveruniversal file transfer server with web interface
aspera point- to- point desktop client
HIGH-PERFORMANCE FILE SYNCHRONIZATION AND REPLICATION Aspera Sync
PROBLEMS OF TRADITIONAL SYNCHRONIZATION
Typical Situation
Large numbers of files (1 Million++) to be ‘Syncd’ over the WAN, with time-critical deadlines
Traditional ‘rsync’ - Finding “Changes” is highly costly
• Each file is compared over the WAN with costly “chatty” messaging: 10 million files x 100 milliseconds RTT
means comparison time is more than 24 hours!
Traditional ‘rsync’ – Uses TCP to transport new data, very slow over the WAN
• On global WANs (100 millisecond RTT / 1% packet loss and worse) standard TCP rates are <<10Mbps
• On difficult global WANs (300 millisecond RTT / 3 % packet loss and worse) standard TCP rates are <<1 Mbps
• Transferring each gigabyte of data takes hours!
• Opening concurrent rsync jobs has diminishing improvement and is hard to manage
Traditional ‘rsync’ – Moving or renaming directories results in data copies over the WAN
• E.g. a new “build” directory with the very same files gets copied over the network again. If the directory is large,
can be many hours.
Traditional ‘rsync’ does not have tracking, visibility, bandwidth management
• The progress is at best to command line standard output on the host where sync’ing occurs
Proprietary synchronization and replication solutions have one or more of the above problems, and
may only work between identical proprietary systems – e.g. Vendor A and Vendor B synchronization
don’t interoperate.
45
A FUNDAMENTAL SOLUTION
Aspera Sync Solves these Problems
Finding Changes Requires No Remote Communication
• Changes are computed against a local snapshot with fastest possible local comparison
Uses Aspera FASP for line speed transfers over the WAN with a single transport stream
Moving or renaming directories is replicated as-is at the target
Files with the same checksums are de-duplicated saving network and storage bandwidth
Sync progress, status, and throughput are globally visible and controllable through Aspera Console
Fully interoperable between operating systems and storage systems
46
INTRODUCING ASPERA SYNC HIGH-SPEED SYNCHRONIZATION AND REPLICATION
Cross platform software application designed for high performance
synchronization and replication of large file stores over the WAN
Application and use cases
• Offsite synchronization and replication for storage migration, disaster
recovery and business continuity.
• Bi-directional system mirroring for alternate access to digital content.
• Hub and spoke sync for high-speed content collection or distribution.
Features and Benefits
• High speed, multi-directional synchronization of remote files and directories
• Highly scalable – designed for today’s extremely large data sets
• Highly efficient - designed for long distance WANs
• Secure – matches security standards set by government, FIPS 140-2 compliant
• Platform agnostic – runs on industry standard Linux and Windows
• Storage agnostic – compatible with any standard file or block storage system
• Familiar rsync-like interface shrinks the learning curve for IT professionals
• One-to-one, one-to-many, and full mesh synchronization
• Unidirectional, bi-directional; One time, scheduled, or continuous
Scalable, multi-directional, multi-node synchronization
SYNCHRONIZATION
ASPERA SYNC FEATURES OVERVIEW
High speed, multi-directional synchronization of remote files and directories, solving the fundamental
problems of rsync
• Built on fasp™ - overcomes TCP bottlenecks for maximum transfer speed regardless of file size, network or
distance
• Fastest possible resolution of changes - Compares changes to local state (file snapshot), saving costly WAN
chattiness
• File system notification where available; Superfast scan to detect changes in scan-mode for large incremental
data sets
• Quick restart after system down
• Replicates file moves and file renames on the source as a file move or rename on the target; Huge savings not
re-transferring
• Robust - waits for the system(s) to become stable (i.e., detects whether or not files are still being modified) before
syncing
Robust, Built-in Security
• User/endpoint authentication, AES-128 cryptography in transit and at-rest data integrity verification and automatic
resume of partial or failed transfers
Platform Support and deployment
• Command line binary and admin tool for Linux, Windows, and now OS X and Solaris
• Available as an add-on license option to Aspera servers and Aspera Point-to-point
48
ASPERA SYNC NEW FEATURES AVAILABLE IN 1.5
Integrated Reporting and Management in Console
• Full integration with Aspera Console 2.0 for global visibility
into sync sessions and activity
• Reports synchronization progress, per file status, throughput
• Enables pause, resume and on-the-fly bandwidth priority
and speed control for Sync sessions
Advanced Features
• Rich include and exclude filters, full support for soft and hard
links, configurable user and group ID
• Preserves access time, user and group IDs
• File-level de-duplication eliminates unnecessary transfers,
reduces sync times and reduces storage use
• Secure token authorization and document roots as well as
API for embedding Sync.
Expanded platform support
• OS X, Solaris and BSD in addition to Windows and Linux
Introduction of distributed file system watcher
• Fastest possible capture of file system changes on NAS
(NFS, CIFS) deployments, clusters of ingest servers and
very large numbers of files – clusterable and scalable
49
Disaster Recovery / Business Continuity
• Replicate mission critical data from a primary site to one
or more alternate sites
• Systems remain available after a system outage or site
loss - critical data is preserved
SYNC USE CASES
Content Distribution / System Mirroring
• Automatically replicate digital content from one system to
another
• Content is always available, offering high availability of the
service
Hub and Spoke for Collection or Distribution
• Daily collection or distribution of data across a large
network of end points
• Continuous update of online content, software and
media
Data Migration for Diverse Storage Environments
• Replicate entire file system, or portions of the file
system
• Preserve file attributes such as permissions, access
times, ownership, etc.
Primary Site Alternate Site Primary Site Alternate Site
Web Application
50
Synchronization time after adding 31,056 files to 1 million small
files (100 KB each) over WAN of 100ms/1%
ASPERA SYNC – PERFORMANCE BENCHMARKS
Performance comparison synchronizing many small files
(average size 100 KB) over WAN of 100ms/1%
Small files
performance
Number
of files
Data set size Sync time Throughput
Async 978,944 93.3 GB 9,968 sec
(2.8 hours)
80.4 Mbps
rsync 978,944 93.3 GB 814,500 sec
(9.4 days)
0.99 Mbps
Speed up difference 81x
Large file
performance
Number of
files
Data set
size (GB)
Sync time Throughput
Async 5,194 500.1 GB 4,664 sec
(1.3 hours)
921 Mbps
rsync 5,194 500.1 GB 4,320,000 sec
(50 days)
0.98 Mbps
Speed up difference 940x
Performance comparison synchronizing many large files
(average size 100 MB) over WAN of 100ms/1%
Change file
performanc
e
Number
of
existing
files
Number
of files
added
Total
size
Sync
time
Throughpu
t
Async 978,944 31,056 2.97
GB
947 sec
(16 min)
26.9 Mbps
rsync 978,944 31,056 2.97
GB
37,076
sec
(10.3 hrs)
0.68 Mbps
Speed up difference 39x
Change file
performance
Number
of
existing
files
Number
of files
added
Total
size
Sync
time
Throughput
Async 5,194 54 5.49 GB 54 sec 871 Mbps
rsync 5,194 54 5.49 GB 54,573
sec
(15 hrs)
0.86 Mbps
Speed up difference 1000x
Synchronization time after adding new files to set of large files (100 MB) over
WAN of 100ms/1%
First Run
Second Run
51
ASPERA DRIVE AND FASPEX 4.0 Next generation file sharing, sync and exchange platform
PROBLEMS WITH CURRENT FILE SHARING SOLUTIONS
Performance
• Today’s “Drive,” “File Sync” and “File Sharing” services are impractical for Big Data
• On global WANs (100 millisecond RTT / 1% packet loss+) standard TCP rates are <<10Mbps
• On difficult global WANs (300 millisecond RTT / 3 % packet loss+) standard TCP rates are <<1 Mbps
• Transferring or synchronizing each gigabyte of data takes hours!
Data Size
• Large data sets (large files or large collections of files) cannot move
• Transfers and sync sessions are extremely slow and most often do not complete at all
Security
• Security for hosted solutions is dependent on security of the provider
• Huge concerns over privacy, contractual obligations
Limited deployment options
• Typical SaaS offerings don’t support on premise or hybrid infrastructure
Limited sharing paradigm
• Users are typically locked into a single delivery option (Sync, Share, or Person-to-person delivery)
NEW ASPERA SOLUTION
A new file sharing experience
• Brings together the best of Aspera faspex, Shares, and Sync technologies into a single unified
platform for file-based collaboration
• Includes full access control, privacy and security of the Aspera fasp technology
• Single file access and control sharing platform across distributed sharing points which may be on
premise or in cloud
• Builds on Aspera Connect, Cargo, Sync and Mobile and unifies Shares and faspex paradigms on
the backend to achieve a revolutionary file sharing experience
Based on fasp
• Maximum transfer speed fully utilizing the available bandwidth
• File size and number independent and distance agnostic
• Top to bottom secure access control, privacy of data and integrity verification across the entire
platform, built into fasp including encryption over the wire and at rest
• Direct high-speed transfer with cloud and traditional storage
Aspera Drive and faspex 4.0
• Brings Remote file browsing, transfer, synchronization, and package sending and receiving to the
Desktop, Browser and Mobile platforms
• Natural project based collaboration model on the application backend
• Easy to use applications based on familiar models of desktop and web sharing
• Innovative architecture allows for deploying on premise, in the cloud, or in any hybrid combination
you choose
ASPERA DRIVE AND FASPEX 4.0
Aspera Drive
• Browsing of remote Projects and authorized shares fully
integrated in to the desktop explorer
• Drag and drop files or folders to initiate high-speed transfers to
and from remote Shares
• Background file synchronization between desktop and cloud
• Integrated faspex package sending and receiving
Next Generation Aspera faspex
• Web application tier and user interface.
• Combines all of the file sharing and package sending
capabilities of faspex and Shares to power the Drive file-
sharing ecosystem.
• Deploys in conjunction with Aspera Enterprise Server(s) in the
Enterprise or in the cloud
• Combines users and groups with file and directory resources
into project-based collaboration spaces
• Comprehensive and fine grained access control combining
shares with users and projects
Mobile Aspera Drive and faspex Apps
• Extends the experience to mobile platforms
• Browse Projects and Shares and upload and download from
iOS and Android devices using fasp. 55
ASPERA DRIVE AND FASPEX 4.0 INTEGRATED WITH DESKTOP EXPLORER
• Browse remote projects and authorized Shares directly from desktop
explorer
• Drag and drop files or folders to initiate high-speed transfers to and
from desktop and remote Shares (cloud or on premise)
• Background sync large file sets at high-speed between desktop and
remote nodes (cloud or on premise)
• Right click to send faspex Packages directly from desktop explorer
56
