Apache Igniteand Apache Spark - events.static.linuxfound.org · Apache®, Apache Ignite, Ignite®,...

Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSoftwareFoundationintheUnitedStatesand/orothercountries.

DenisMagdaGridGain ProductManager

ApacheIgnitePMC

ApacheIgnite andApacheSparkWhereFastDataMeetstheIoT

http://ignite.apache.org #apacheignite #denismagda

Agenda• IoT DemandstoSoftware• IoT SoftwareStack• DeviceOS/RTOS• DataCollectionandEnrichment• HTAPPlatform• ApplicationAPIs

• Demo

IoT DemandstoSoftware

IoT DemandstoSoftware• Real-timeingestion• Real-timeprocessing• Time-seriessupport• FlexibleQueryingAPIs– SQL– Full-textsearch– Geo-spatial

• Analytics– BI– MachineLearning

• High-availability• Simplescalability

IoT SoftwareStack

HTAP Platform

Data Collection and Enrichment

Device OS / Real-Time OS

Application APIs

IoT SoftwareStack

HTAP Platform

Data Collection and Enrichment

Device OS / Real-Time OS

Application APIs

ApacheIoT SoftwareStack

DeviceOS/RTOS

ApacheMyNewt• OpenSourceRTOS– CortexM0-M4– MIPS&RISC-V

• Networking– BluetoothLowEnergy– Wi-Fi– TCP/IP&UPD

• Build&PackageManagement• Securebootloaderandsigned

images• RemoteFirmwareUpgrade

DataCollectionandEnrichment

DataCollectionandEnrichment• SparkStreaming– Fault-TolerantStreams

Processing• DataCollection– Sockets– Kafka– Flume

• DataEnrichment– SparkAPI

• DataStorage– ApacheIgnite

HTAPPlatform

UseCase: Smart Metering and Utilities – delivers a comprehensive IOT platform

50+ Million Meters

• SilverSpringRequirements– Migratetoin-memoryprocessing– Addscalability&elasticity– Useopensourcetechnologies

• SilverSpring Solution– SaaSArchitecture

• Multi-Tenancy• AdvancedSecurity

– Strongcomputecapabilities• Co-locatedin-memoryprocessing

– Demonstratedbest• On-demandelasticity&scalability• ANSI-99SQLSupport• Transactionalconsistency

GridGainSecurity

SilverSpring IoTPlatform

ApacheIgniteIn-MemoryDataFabric

• Supports Applications of various types and languages

• Open Source – Apache 2.0• Simple Java APIs• 1 JAR Dependency• High Performance & Scale• Automatic Fault Tolerance• Management/Monitoring• Runs on Commodity Hardware

• Supports existing & new data sources• No need to rip & replace

In-MemoryDataGrid• DistributedKey-ValueDataStore

• DataReliability• High-Availability– Activereplicas,automaticfailover

• DataConsistency– ACIDdistributedtransactions

In-MemoryDataGrid:Partitioning

PartitionedCache ReplicatedCache

StreamingtoIgnite• IgniteDataStreamer– FastestDataInjection– Automaticdatapartitioning

• StreamReceivers– Customlogicexecution– Additionaldatatransformation

• StreamAdapter– Socket– Kafka– Flink– RocketMQ– etc.

ApplicationAPIs

ApacheIgniteSQLGrid• ANSI-99SQL• Geo-spatialQueries• Full-textSearch• AlwaysConsistent• FaultTolerant• Cross-Platform– JDBCandODBCdrivers– DML(INSERT,UPDATE,etc.)– DDL(CREATE,DROP,etc.)

ApacheIgniteSQLGrid:Queries• DistributedJoins• AutomaticGroupBy,Aggregations,Sorting

• Cross-CacheJoins,Unions,etc.• Ad-HocSQLSupport

ApacheIgniteComputeGrid• DirectAPIfor

MapReduce• DirectAPIforForkJoin• ZeroDeployment• StateCheckpoints• LoadBalancing• AutomaticFailover

SparkSharedRDDs• IgniteRDD– ShareRDDacrossjobson

thehost– ShareRDDacrossjobsin

theapplication– ShareRDDglobally

• FasterSQL– In-MemoryIndexes– SQLontopofSharedRDD

MachineLearningGrid

Resources• Documentation:– ApacheIgnite:https://apacheignite.readme.io/docs– ApacheSparkStreaming:http://spark.apache.org/streaming/– IgniteandSparkIntegration:https://apacheignite-

fs.readme.io/docs/ignite-for-spark– ApacheMyNewt:http://mynewt.apache.org

• DemoSourceCode:– https://github.com/dmagda/IgniteSparkIoT

ANYQUESTIONS?Thankyouforjoiningus.Followtheconversation.

http://ignite.apache.org

#apacheignite #denismagda

Apache Igniteand Apache Spark - events.static.linuxfound.org · Apache®, Apache Ignite, Ignite®,...

Documents

Better together: Fast Data with Apache Spark™ and Apache Ignite™ by Mike Griggs

Apache&Ignite:& - GridGain Systems Hazelcast! Open Source! Point Solution - IMDG! In6Memory&Data&Fabric:&& Strategic&Approach&to&IMC&

TensorFlow with Apache Ignite and Machine and Deep ... · Achieving Continuous Machine and Deep Learning with Apache Ignite and TensorFlow Yuriy Babak 2019 © GridGain Systems

Apache’IgniteTM (Incubating))A)In:Memory’Data’Fabric’gotocon.com/dl/goto-chicago-2015/slides/DmitriySetraky… · · 2015-05-11meetup/) Apache®,)Apache)Ignite,)Ignite®,)and)the)Apache)Ignite)logo)are)either)registered)trademarks)or)

Elegantes In-Memory Computing mit Apache Ignite und Kubernetes. @data2day

Overview SCALE14x 2016. Agenda/Schedule -Apache Bigtop Overview -Apache Spark Overview/Getting Started -Lunch Break -Apache Ignite -Workshop, tutorial,

HowtoChoosetheBest …...©2015"GridGain"Systems,"Inc." DMITRIY’SETRAKYAN’ GridGain"Founder" Apache"Ignite"PMC"Chair" HowtoChoosetheBest In7Memory’Solu=ons’for’Your’Apps

In Kooperation mit PC DVD-ROM - The Apache Software Foundationpeople.apache.org/~pescetti/tmp/2014-05-trademarks/4076... · 2014-05-24 · Apache OpenOffice 4 ist vollständig kompatibel

Apache ignite Datagrid

Be#er%Together–)Apache%Ignite%&%Apache%Spark% · Apache®,)Apache)Ignite,)Ignite®,)and)the)Apache)Ignite)logo)are)either)registered)trademarks)or)trademarks)of)the)Apache)So8ware)Foundaon)in)the)United)States)and

Apache Ignite for Node.js Developers - GridGain Systems · Apache Ignite Support –Faster Time to Reliable Ignite •Get up and running faster with 2 hours initial consultation •Ensure

Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

In-Memory Computing Platform - GridGain Systems...Apache Ignite Project • 2007: First version of GridGain • Oct. 2014: GridGain contributes Ignite to ASF • Aug. 2015: Ignite

Security Guide Apache Ignite and GridGain...2003/04/20 · Mainframe Ignite Persistence NoSQL Hadoop Persistent Layer RDBMS Machine and Deep Learning Messaging Streaming Events Transaction

IMC Summit 2016 Breakout - Matt Coventon - Test Driving Streaming and CEP on Apache Ignite

Getting Started With Apache Ignite SQL · •Ignite SQL Basics: DML, DDL, connectivity, configuration •Affinity Co-Location and Distributed JOINs •Beyond Memory Capacity: Disk

IMC Summit 2016 Breakout - Roman Shtykh - Apache Ignite as a Data Processing Hub

IMCSummite 2016 Breakout - Nikita Ivanov - Apache Ignite 2.0 Towards a Converged Data Platform

Архитектура Apache Ignite .NET

An Introduction to Apache Ignite - Mandhir Gidda - Codemotion Rome 2017