Counters and AsyncInfrastructures
Proposal for Code ContributionGuy Sela, Senior Engineer, HPE
Who am I ? Senior Engineer at HPE Served as a developer and team leader at the Israeli NSA Developing software since 2004
E-mail: [email protected]: http://www.linkedin.com/pub/guy-sela/70/17/9aa
AgendaCounters Infrastructure Why do we need Counters? Proposed Implementation
Capabilities Examples Under the Hood
Third-Party Alternatives
Async Infrastructure Scope Overview Under the Hood
Why do we need Counters?
High-Frequency events will swamp the Log. Good alternative for Log.debug() and Log.trace() Useful information for troubleshooting. Powerful tool for analyzing bottlenecks. What events happened in my system until now? Analytics
Proposed Implementation
Light-weight infrastructure that is very easy to integrate
into your code
Supports the common use-cases for counters
Found to be extremely useful for monitoring and
troubleshooting production systems
Capabilities
Incremental & State Counters
Configurable interval for print counters delta to the log
Runtime counters silencing based on the users log library
Almost no overhead on the system
Counter querying API based on regular expressions
REST API
Use Case Incremental Counterpublic class UserService {
// High-Scale Eventvoid userEvent() {
log.info(Event Called); // Will swamp the Log // Business Logic
}}
Use Case Incremental Counterpublic class UserService {
// High-Scale Eventvoid userEvent() {
UserServiceCounters.user_event_called.inc(); // Business Logic
}}
Log Output Example 1 Minute Interval[2016-02-16 14:18:27,626] INFO : (com.hpe.test.TestClassA:252) Did Something[2016-02-16 14:18:27,967] INFO : (com.hpe.test.TestClassB:254) Did Something Else[2016-02-16 14:18:28,028] CNT : user_event_called : +37824[2016-02-16 14:18:45,125] INFO : (com.hpe.test.TestClassC:63) Yet Another Log Line[2016-02-16 14:19:28,028] CNT : user_event_called : +41215[2016-02-16 14:19:50,026] INFO : (com.hpe.test.TestClassC:72) I Love Log Lines[2016-02-16 14:20:30,028] CNT : user_event_called : +51621[2016-02-16 14:20:50,026] INFO : (com.hpe.test.TestClassA:11) Log Lines Rock!
10
Log Output Example Multiple Counters[2016-02-16 15:18:27,626] INFO : (com.hpe.test.TestClassA:252) Did Something[2016-02-16 15:18:27,967] INFO : (com.hpe.test.TestClassB:254) Did Something Else[2016-02-16 15:18:28,028] CNT : user_event_called : +37824, other_counter: +72[2016-02-16 15:18:45,125] INFO : (com.hpe.test.TestClassC:63) Yet Another Log Line[2016-02-16 15:19:28,028] CNT : user_event_called : +41215, foo_counter: +17
11
public class UserService {// Boilerplate Code to define a new Counters Groupenum UserServiceCounters {
user_event_called(), other_user_counter(), third_counter();
private OccurenceCounter counter; public void inc() {
counter.inc();}
UserServiceCounters() {counter = new OccurenceCounter(UserServiceCounters, name());
}}
}12
import com.hpe.counter.OccurrenceCounter;
Counter Types
Incremental Counter Value is manipulated using inc(). The log prints show the delta from the last log print. No delta No log print.
State Counter Value is manipulated using set(). The log prints show the current value of the counter in case it changed. Useful for detecting queue build-up.
13
Counters SilencingLog4J Config File // Standard mechanism to change Log Severity
log4j.logger.UserService.user_event_called=WARN
14
Use Case - Using Counters to Identify Bottlenecksvoid myBusinessLogic {
String value = DB.getKey(key);// Socket.sendOnSocket(new Packet(value.toBytes()));// invokeIntensiveMethod(value);
}Scenario: 5000 Events per second which invoke myBusinessLogic();Symptom:We only manage to handle 2000 Events per second
15
Use Case - Using Counters to Identify Bottlenecksvoid myBusinessLogic {
BLCounters.before_db.inc();String value = DB.getKey(key);
BLCounters.after_db.inc();// BLCounters.before_socket.inc();Socket.sendOnSocket(new Packet(value.toBytes()));BLCounters.after_socket.inc();// BLCounters.before_intensive.inc();invokeIntensiveMethod(value);BLCounters.after_intensive.inc();
}16
Log Output will tell us where is the Bottleneck[2016-02-16 15:18:28,028] CNT : before_db: +5000,
after_db: +2000, before_socket: +2000, after_socket:
+2000, before_intensive: +2000, after_intensive: +2000
The DB is Slowing Us!
17
Log Output will tell us where is the Bottleneck[2016-02-16 15:18:28,028] CNT : before_db: +5000,
after_db: +5000, before_socket: +5000, after_socket:
+5000, before_intensive: +5000, after_intensive: +2000
The Intensive Method is Slowing Us!
18
What events happened in my System until Now?
CountersDumperService APIString getCounters(String regexp); Call using REST API / Other Management Console
19
Analytics
20
Under the Hood
Counters based on AtomicLong & AtomicInteger causes
insignificant overhead.
Using enum semantics instead of register new counter
implementation
Background Thread that writes the delta of the counters to
the Log in each configurable interval.
21
Third-Party Alternatives
Known third-party implementations with similar goals:
Metrics, Parfait, JAMon, Java Simon, Perf4J.
We havent made a deep comparison between the
proposed solution to these.
22
Questions?
23
And Now to a Completely Different Subject
24
Async Infrastructure
25
Introduction
In ODLs Hydrogen version, every OSGI Bundle was activated using the BundleActivator pattern.
Every OSGI Service declared its implementations and dependencies in its Activator.
After the transformation to MD-SAL, Plugins/Applications define their dependencies, implementations and API using YANG Model.
Applications that reside in the Controller communicate with each other using MD-SAL: RPCs and Notifications.
26
Introduction Application can be implemented by a number of OSGI Services
and Bundles. The presented Async Infrastructure facilitates communication
between OSGI services, and is agnostic to YANG, RPCs, etc. The infrastructure is currently only relevant for Bundles that still
use the BundleActivator pattern. Specifically for Activators that extend ComponentActivatorAbstractBase.
If the presented capabilities interest the ODL community, we can discuss extension of the infrastructure to support the new OSGI Bundle Activation patterns, that came along with MD-SAL
27
Scope
28
Async Infra (Unwanted?)
Application A
OSGI Service OSGI Service
MD-SAL
Async Infra
Application B
OSGI Service OSGI ServiceAsync Infra
RPC / NotificationsRPC / Notifications
Overview
29
ServiceA implements PacketProcessingListener
Service A Service B Service CdoSomething() doSomethingElse()
onPacketReceived() Notification
MD-SAL
Application A
public class ServiceA implements IServiceA, PacketProcessingListener {private IServiceB serviceB;
public void onPacketReceived(PacketReceived notification) {serviceB.doSomething();
}}
30
public class ServiceB implements IServiceB {private IServiceC serviceC;
public void doSomething() {serviceC.doSomethingElse();
}}
31
32
Service A Service B Service CdoSomething() doSomethingElse()
The presented infrastructure enables you to convert these Java calls to Asynchronous
Multi-Threaded calls just by changing a configuration file!
Example of the config-file/pools/OurThreadPool/poolSize=8
/handlers/com.hpe.ServiceB/PoolName=OurThreadPool
/handlers/com.hpe.ServiceB/WorkerCount=5
/handlers/com.hpe.ServiceC/PoolName=OurThreadPool
/handlers/com.hpe.ServiceC/WorkerCount=6
We create a Thread Pool with 8 Threads, and define ServiceB and ServiceC to use it.
ServiceB will use up to 5 concurrent Threads while ServiceC will use up to 6 concurrent Threads.
33
Glimpse Under the Hood
34
ServiceAdoSomething() ServiceB Proxy
ServiceB Proxy ServiceBDedicated
Queue
Worker Thread
Worker Thread
Worker Thread
ServiceBdoSomething()
Benefits The developers can focus in coding the business logic of the Service. The API remains exactly the same with the help of Proxies and Reflection. Dynamically decide which services will be Async and control Thread Pools. No need to write bug-prone Boilerplate Code for Async method invocation. This
includes BlockingQueues, Producer Threads, Consumer Threads, Executors, Thread Pools, synchronized blocks. Caveat: The code itself obviously must support multi-threaded execution, i.e. Concurrent data structures, etc.
The Queue can be capped, and messages that exceed the cap will be dropped. Important for prevention of memory leaks.
35
What if I want only part of the API to be ASYNC?
public class MyService implements IMyService {public void methodWeWantAsync() {}
public void anotherMethodWeWantAsync() {}
@SyncMethodpublic void methodWeWantSync() {}
}
@SyncMethod - Will tell the infrastructure
