Click here to load reader

Counters and Async Infrastructures · PDF fileCounters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register

  • View
    236

  • Download
    0

Embed Size (px)

Text of Counters and Async Infrastructures · PDF fileCounters and Async Infrastructures ... Multiple...

  • Counters and AsyncInfrastructures

    Proposal for Code ContributionGuy Sela, Senior Engineer, HPE

  • Who am I ? Senior Engineer at HPE Served as a developer and team leader at the Israeli NSA Developing software since 2004

    E-mail: [email protected]: http://www.linkedin.com/pub/guy-sela/70/17/9aa

  • AgendaCounters Infrastructure Why do we need Counters? Proposed Implementation

    Capabilities Examples Under the Hood

    Third-Party Alternatives

    Async Infrastructure Scope Overview Under the Hood

  • Why do we need Counters?

    High-Frequency events will swamp the Log. Good alternative for Log.debug() and Log.trace() Useful information for troubleshooting. Powerful tool for analyzing bottlenecks. What events happened in my system until now? Analytics

  • Proposed Implementation

    Light-weight infrastructure that is very easy to integrate

    into your code

    Supports the common use-cases for counters

    Found to be extremely useful for monitoring and

    troubleshooting production systems

  • Capabilities

    Incremental & State Counters

    Configurable interval for print counters delta to the log

    Runtime counters silencing based on the users log library

    Almost no overhead on the system

    Counter querying API based on regular expressions

    REST API

  • Use Case Incremental Counterpublic class UserService {

    // High-Scale Eventvoid userEvent() {

    log.info(Event Called); // Will swamp the Log // Business Logic

    }}

  • Use Case Incremental Counterpublic class UserService {

    // High-Scale Eventvoid userEvent() {

    UserServiceCounters.user_event_called.inc(); // Business Logic

    }}

  • Log Output Example 1 Minute Interval[2016-02-16 14:18:27,626] INFO : (com.hpe.test.TestClassA:252) Did Something[2016-02-16 14:18:27,967] INFO : (com.hpe.test.TestClassB:254) Did Something Else[2016-02-16 14:18:28,028] CNT : user_event_called : +37824[2016-02-16 14:18:45,125] INFO : (com.hpe.test.TestClassC:63) Yet Another Log Line[2016-02-16 14:19:28,028] CNT : user_event_called : +41215[2016-02-16 14:19:50,026] INFO : (com.hpe.test.TestClassC:72) I Love Log Lines[2016-02-16 14:20:30,028] CNT : user_event_called : +51621[2016-02-16 14:20:50,026] INFO : (com.hpe.test.TestClassA:11) Log Lines Rock!

    10

  • Log Output Example Multiple Counters[2016-02-16 15:18:27,626] INFO : (com.hpe.test.TestClassA:252) Did Something[2016-02-16 15:18:27,967] INFO : (com.hpe.test.TestClassB:254) Did Something Else[2016-02-16 15:18:28,028] CNT : user_event_called : +37824, other_counter: +72[2016-02-16 15:18:45,125] INFO : (com.hpe.test.TestClassC:63) Yet Another Log Line[2016-02-16 15:19:28,028] CNT : user_event_called : +41215, foo_counter: +17

    11

  • public class UserService {// Boilerplate Code to define a new Counters Groupenum UserServiceCounters {

    user_event_called(), other_user_counter(), third_counter();

    private OccurenceCounter counter; public void inc() {

    counter.inc();}

    UserServiceCounters() {counter = new OccurenceCounter(UserServiceCounters, name());

    }}

    }12

    import com.hpe.counter.OccurrenceCounter;

  • Counter Types

    Incremental Counter Value is manipulated using inc(). The log prints show the delta from the last log print. No delta No log print.

    State Counter Value is manipulated using set(). The log prints show the current value of the counter in case it changed. Useful for detecting queue build-up.

    13

  • Counters SilencingLog4J Config File // Standard mechanism to change Log Severity

    log4j.logger.UserService.user_event_called=WARN

    14

  • Use Case - Using Counters to Identify Bottlenecksvoid myBusinessLogic {

    String value = DB.getKey(key);// Socket.sendOnSocket(new Packet(value.toBytes()));// invokeIntensiveMethod(value);

    }Scenario: 5000 Events per second which invoke myBusinessLogic();Symptom:We only manage to handle 2000 Events per second

    15

  • Use Case - Using Counters to Identify Bottlenecksvoid myBusinessLogic {

    BLCounters.before_db.inc();String value = DB.getKey(key);

    BLCounters.after_db.inc();// BLCounters.before_socket.inc();Socket.sendOnSocket(new Packet(value.toBytes()));BLCounters.after_socket.inc();// BLCounters.before_intensive.inc();invokeIntensiveMethod(value);BLCounters.after_intensive.inc();

    }16

  • Log Output will tell us where is the Bottleneck[2016-02-16 15:18:28,028] CNT : before_db: +5000,

    after_db: +2000, before_socket: +2000, after_socket:

    +2000, before_intensive: +2000, after_intensive: +2000

    The DB is Slowing Us!

    17

  • Log Output will tell us where is the Bottleneck[2016-02-16 15:18:28,028] CNT : before_db: +5000,

    after_db: +5000, before_socket: +5000, after_socket:

    +5000, before_intensive: +5000, after_intensive: +2000

    The Intensive Method is Slowing Us!

    18

  • What events happened in my System until Now?

    CountersDumperService APIString getCounters(String regexp); Call using REST API / Other Management Console

    19

  • Analytics

    20

  • Under the Hood

    Counters based on AtomicLong & AtomicInteger causes

    insignificant overhead.

    Using enum semantics instead of register new counter

    implementation

    Background Thread that writes the delta of the counters to

    the Log in each configurable interval.

    21

  • Third-Party Alternatives

    Known third-party implementations with similar goals:

    Metrics, Parfait, JAMon, Java Simon, Perf4J.

    We havent made a deep comparison between the

    proposed solution to these.

    22

  • Questions?

    23

  • And Now to a Completely Different Subject

    24

  • Async Infrastructure

    25

  • Introduction

    In ODLs Hydrogen version, every OSGI Bundle was activated using the BundleActivator pattern.

    Every OSGI Service declared its implementations and dependencies in its Activator.

    After the transformation to MD-SAL, Plugins/Applications define their dependencies, implementations and API using YANG Model.

    Applications that reside in the Controller communicate with each other using MD-SAL: RPCs and Notifications.

    26

  • Introduction Application can be implemented by a number of OSGI Services

    and Bundles. The presented Async Infrastructure facilitates communication

    between OSGI services, and is agnostic to YANG, RPCs, etc. The infrastructure is currently only relevant for Bundles that still

    use the BundleActivator pattern. Specifically for Activators that extend ComponentActivatorAbstractBase.

    If the presented capabilities interest the ODL community, we can discuss extension of the infrastructure to support the new OSGI Bundle Activation patterns, that came along with MD-SAL

    27

  • Scope

    28

    Async Infra (Unwanted?)

    Application A

    OSGI Service OSGI Service

    MD-SAL

    Async Infra

    Application B

    OSGI Service OSGI ServiceAsync Infra

    RPC / NotificationsRPC / Notifications

  • Overview

    29

    ServiceA implements PacketProcessingListener

    Service A Service B Service CdoSomething() doSomethingElse()

    onPacketReceived() Notification

    MD-SAL

    Application A

  • public class ServiceA implements IServiceA, PacketProcessingListener {private IServiceB serviceB;

    public void onPacketReceived(PacketReceived notification) {serviceB.doSomething();

    }}

    30

  • public class ServiceB implements IServiceB {private IServiceC serviceC;

    public void doSomething() {serviceC.doSomethingElse();

    }}

    31

  • 32

    Service A Service B Service CdoSomething() doSomethingElse()

    The presented infrastructure enables you to convert these Java calls to Asynchronous

    Multi-Threaded calls just by changing a configuration file!

  • Example of the config-file/pools/OurThreadPool/poolSize=8

    /handlers/com.hpe.ServiceB/PoolName=OurThreadPool

    /handlers/com.hpe.ServiceB/WorkerCount=5

    /handlers/com.hpe.ServiceC/PoolName=OurThreadPool

    /handlers/com.hpe.ServiceC/WorkerCount=6

    We create a Thread Pool with 8 Threads, and define ServiceB and ServiceC to use it.

    ServiceB will use up to 5 concurrent Threads while ServiceC will use up to 6 concurrent Threads.

    33

  • Glimpse Under the Hood

    34

    ServiceAdoSomething() ServiceB Proxy

    ServiceB Proxy ServiceBDedicated

    Queue

    Worker Thread

    Worker Thread

    Worker Thread

    ServiceBdoSomething()

  • Benefits The developers can focus in coding the business logic of the Service. The API remains exactly the same with the help of Proxies and Reflection. Dynamically decide which services will be Async and control Thread Pools. No need to write bug-prone Boilerplate Code for Async method invocation. This

    includes BlockingQueues, Producer Threads, Consumer Threads, Executors, Thread Pools, synchronized blocks. Caveat: The code itself obviously must support multi-threaded execution, i.e. Concurrent data structures, etc.

    The Queue can be capped, and messages that exceed the cap will be dropped. Important for prevention of memory leaks.

    35

  • What if I want only part of the API to be ASYNC?

    public class MyService implements IMyService {public void methodWeWantAsync() {}

    public void anotherMethodWeWantAsync() {}

    @SyncMethodpublic void methodWeWantSync() {}

    }

    @SyncMethod - Will tell the infrastructure

Search related