Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
1
Cloud Native at the Edge: High-performance Edge Inference with RedisAI
White PaperA Project Cassini Reference Implementation
Real-time analytics and data insights at EdgeAnalysis of data collected from devices and resources such as sensors and edge devices
is growing rapidly among industrial customers. Image processing, anomaly detection for
predictive maintenance, and log analysis are some of the most popular use cases for
edge analytics. Several reasons make running data analytics on edge devices attractive:
1. Analyzingdataonedgedevicescansavemoneyandresourcesbyoffloading
the computations from cloud.
2. The latency of sending data to the cloud for analytics could be high, particularly when
processing real-time events. Performing inference on real-time data may require the
inference engine to be as close to the data source as possible.
3. Reliability of network connections can be an issue for IoT platforms, hence making
on-device inference and decision making important in critical scenarios.
ChallengesIndustrial IoT market is expected to experience the growth of 15.2% per year for the
forecast period of 2020-2027 (Data Bridge Market Research, 2020), which is mostly driven
by machine learning, AI, and data analytics. However, still most data in IIoT environments are
not currently exploited for analytics, and huge number of devices are not even linked up to
a sever in the cloud.
While many of applications such as assembly line redesign will take place in the cloud,
a substantial number of tasks—predictive maintenance based on real time data, product
consistencybasedoncomputervision,orenergyefficiency—willbeperformedlocally
at the edge. The money that industry can save by edge analytics such as predictive
maintenance could be huge. Some mining companies have reported an estimated loss
of $30k per hour in downtime. This is worse in automakers industry, which is $20k-$30k
per minute (Altman, 2019). With more than 180 billion chips, Arm will play a big part in
integration of the IoT functionality into devices for edge analytics.
Index
Real-time analytics and data
insights at Edge
Challenges
RedisAI, a solution for edge
inference
RedisAI on Arm
Use case: Finger follower
inspection
Edge inference with
Project Cassini
Take the Next Step
2
Running analytics on edge cuts cost and latency while increasing reliability. Still, owners
will have to balance their analytics needs with the core hardware and software capabilities.
In general, IoT devices have limited computational power with lower resources, and many
of the on-the-shelf analytic tools are not designed with an eye on running on small devices.
Therefore, finding right tools and techniques to run applications on the edge could
be challenging.
RedisAI, a solution for edge inferenceRedis is a lightweight in-memory database used as a high-speed local key-value cache. Redis
has a modular structure, which allows adding custom functionalities to run over stored data.
RedisAI is a module written by RedisLabs for the purpose of running deep learning models
over stored data. Running as a Redis module, RedisAI increases the throughput and
reduces latency.
RedisAIontheedgecanbringthefollowingbenefits:
Latency reduction and throughput increase achieved through performing inference
on the data directly from database’s shared memory.
Zero downtime on new model deployment as models can be updated transparently
resulting in no operational downtime.
Supportformajorbackends,includingTensorflow,TensorflowLite,Pytorch and ONNXRuntime.
With low footprint, RedisAI is highly scalable and increases throughput
and available memory.
Figure 1. Redis AI components
3
Redis, RedisAI and Redis Streams work together
The RedisAI solution consists of several components:
RedisAI: The module RedisAI is designed to apply inference on input tensors (multi-
dimensional arrays storing data values) and return the result as output tensors stored
in Redis database (A Tensor data structure is another data type that Redis supports).
RedisAI allows models to be stored into Redis and run against input data.
Redis Streams: Edge devices receive information as streams, a sequence of data
values arrivingfromperipheraldevices,sensors,orothersystems.Datastreams
inRedis ishandlesbyRedisStreamsmodule,whichallowsstreamingdatatobe
ingested into the database and allows consumers to wait on the new data added
to the stream to consume. Consumers can act as one or more consumer groups,
which allowsthemto workinadistributedfashionforparallelprocessingwhen
the rate of data ingestion is high.
RedisGears: A component is required to glue Redis Streams and RedisAI, where it
canreadandfilterstreamingrecords,transformthemintotensors,runinferenceon
them, and store the results as another data stream or send it to an external storage.
Another Redis module that is implemented to achieve such a dynamic programmable
mechanism isRedisGears.Itisaserverlessengine,designedforbatchandeventdriven
data processing. RedisGears functions starts with a reader, which reads from a source
and creates data records, and the next steps will perform custom operations on the
incoming data. A function registers itself as event handler (like data stream events)
so it runsonthenewrecordswhenevertheyareaddedtothestream.
RedisGearsallowsuserstoconnectRedisAIandRedisStreamsinaprogrammaticway byprovidingPythonandCAPIs.ThecomponentsofRedisGearsareasthefigure below:
Figure 2. RedisGears
components and internal
design
4
RedisAI on ArmRedisAI runs on Arm-based processors such as AWS Graviton2, Nvidia Jetson Nano,
and Jetson TX2 out of the box. In addition, RedisAI can be containerized to run on Arm
systems, which allows container orchestration platforms like Kubernetes to automate
the deployment and updated of RedisAI clusters on edge devices or on the cloud.
RedisAI and Redis Streams in production
OrobixisoneofArmecosystempartnersthatprovidesartificialintelligencesolutions
on Arm platforms to manufacturers and other businesses. It has developed an AI lifecycle
andgovernanceplatform(namedInvariant.ai)formanagingartificialintelligenceincritical
contexts. By leveraging Redis Streams and Redis AI, it ensures observability of AI-governed
processes and traceability of production models. Their solutions perform real-time analytics
foranomalydetection,driftdetection,similarityfinding,andpredictivemaintenance.
Use case: Finger follower inspection in automotive industry Orobixhasdesignedanautomateddefectdetectionforfingerfollowerpartsproducedby
a motorcycle parts manufacturer. Finger followers are used in motorcycle engine to transfer
the rotary motion from camshaft to valves. The system had to differentiate between actual
defects and dust or small particles from industrial machinery. Before Orobix, the defect
detection process was handled by human inspection. Orobix used RedisAI cluster for storing
imagesandfindingdefects(anomalies)inproductionparts.
System components
Thesystemdesignforautomatedclassificationofdefectivefingerfollowersisshown
in Figure 3. It uses
1. Armprocessorsfortherestofthecomponents,specificallyNVIDIAJetsonTX2
(VisionBoxDaytona),withQuad-CoreArmCortex-A57CPUand256-coreNVIDIA
Pascal GPU architecture. This allowed the customer to run ML/DL inferences on GPU
cores, increasing the throughput dramatically.
2. CameraswithGigEVisionstandardsupport(TeleDyneDalsaorBasler).
3. General purpose industrial PC for HMI (Human Machine Interface) agent.
5
Invariant.ai platform, which the system is design on top of it, is composed of agents
connecting through communication channels. An agent is a software component
implementing a certain functionality, and explicitly declaring the set of input and output ports.
Each port is mapped to a Redis stream on a certain Redis instance and can either act as an
input (or output) source of messages. Figure 3 shows a Process Graph, which is a concept
usedtodescribetheconnectionsbetweenagentportsandisusedtoconfigurehowdata
flowsacrossthedifferentsoftwarecomponentsthatimplementtherequiredfunctionin
a distributed manner. That provides
fullobservabilityofthedataflowingacrossthedifferentsoftwarecomponents
ease of distribution of software components on different physical machines,
decoupling the execution of software running on them
The process graph of the system is composed of 7 Invariant.ai agents in addition to RedisAI.
Two agents perform the acquisition from two Industrial cameras (Teledyne DALSA)
implementingtheGigEVisionProtocol.Onecameracapturesimagesofthefrontandthe
other one of the back of parts. Each image is written once to a Redis stream, and multiple
agents can consume the same image. In fact, not only the HMI (Human Machine Interface)
agent receives the produced images to show them as a preview to the end-user, but also the
FrontClassificationandBackClassificationagentsarereceivingacopyofthem.TheFrontand
Backclassificationagentsperformpre-processingofthereceivedimagesandthenleverage
RedisAIfortheinference(DeepLearninginthisspecificscenario).EachClassificationagent
produces an inference (defective/not defective), that is then processed by the Modbus agent.
This last agent leverages the Modbus protocol to write to the PLC of the machine the
finaloutcome.
6
Figure 3. Process graph
of defect detection system
All the agents run in Docker containers. Each node capturing images from the cameras run
two containers, one for the camera agent, and one running Redis. Similarly, the node running
classification agents runs two containers, one for front image classification, and one for
the back.
The modular system design helps in customizing hardware components based on resource
requirements. For instance, larger models may require RedisAI to run on a machine with more
memory, while the rest of the system remains intact. In addition, containerized architecture
allows running software components on customized architecture with the least possible
modification,andondifferentmanufacturingenvironments.
Edge inference with Project CassiniProject Cassini is a collaborative initiative covering standards, platform security, and
reference implementations with cloud-native standards at edge. That allows users to
run their software stack on edge devices and connect them to their cloud environment
in a secure and scalable manner.
With more partners joining Project Cassini’s edge ecosystem, users will have a broader
choiceondifferentplatformswithoutexperiencingdifficulties.Thisisimportantfor
edge applications, where the same software may have to run on different hardware
andfirmware.ProjectCassiniallowsfrictionlessorchestrationofapplicationsand
connect them to the different cloud providers.
7
Cloud-native experience through Project Cassini helps edge analytics applications,
where a single software may run on hundreds or thousands of devices, with simple
and seamless management of application lifecycle, scalability, fault tolerance, and
updates are necessary. Redis and RedisAI can run inside containers on Arm devices,
hencebenefitfromcloud-nativesoftwarestackfororchestrationandmanagement.
In addition, SystemReady compliance allows applications to run on different Arm
platformswithnomodificationrequired.
Take the Next StepWe invite partners to engage on Project Cassini and explore the applicability of the RedisAI
for their edge inference solutions. To get involved, contact us at [email protected]
or visit us at arm.com/project-cassini.
All brand names or product names are the property of their respective holders. Neither the whole nor any part of the information contained in, or the product described in, this document may be adapted or reproduced in any material form except with the prior written permission of the copyright holder. The product described in this document is subject to continuous developments and improvements. All particulars of the product and its use contained in this document are given in good faith. All warranties implied orexpressed,includingbutnotlimitedtoimpliedwarrantiesofsatisfactoryqualityorfitnessforpurposeareexcluded.Thisdocument is intended only to provide information to the reader about the product. To the extent permitted by local laws Arm shall not be liable for any loss or damage arising from the use of any information in this document or any error or omission in such information.
© Arm Ltd. 2021