Mentat System - Introduction for developersAgenda 1 Introduction 2 Overview 3 Design Technologies Architecture PyZenKitframework Mentatframework 4 Creatingdaemonmodule Overview DemoPiperDaemon.py

Mentat SystemIntroduction for developers

Jan Mach

v1.1, 2018-10-03

Attribution 3.0 Unported (CC BY 3.0)

Jan Mach (CESNET, a.l.e.) Mentat System v1.1, 2018-10-03 1 / 56

Agenda1 Introduction2 Overview3 Design

TechnologiesArchitecturePyZenKit frameworkMentat framework

4 Creating daemon moduleOverviewDemoPiperDaemon.py

5 Hawat: Web interface6 Resources







MotivationKey ideas

Resource consolidationAid for CESNET-CERTS security teamAid for network administrators

Main featuresGathering/receiving information from varioussourcesLong term searchable persistent information storageReal-time and back information processing withvarious methodsFully automatic processing, enable performing ofautomatic actions on specific conditions


Latest news

Released version 2.1.x (Thu Sep 27 2018)Migrated completely to Python3Migrated database to PostgreSQLAutomated build system AlchemistAutogenerated documentation

migration from 0.4.20upgrading from 2.0.x

Public Git code repository and issue tracker


https://homeproj.cesnet.cz/versions/81

https://www.python.org/

https://www.postgresql.org/

https://alchemist.cesnet.cz/

https://alchemist.cesnet.cz/mentat/doc/production/html/manual.html

https://alchemist.cesnet.cz/mentat/doc/production/html/migration.html

https://alchemist.cesnet.cz/mentat/doc/production/html/upgrading.html

https://alchemist.cesnet.cz/mentat/repo.git

https://homeproj.cesnet.cz/projects/mentat

Alchemist build systemhttps://alchemist.cesnet.cz/

Automated build system for Mentat and relatedlibrariesContents:

General informationBuild environment settingsTesting, linting, benchmarkingAutogenerated documentationGit repositoriesDebian packagesPython wheels

Possible improvements:Installation tests, functional testsDocumentation historyAutomated changelogs, repository stats


https://alchemist.cesnet.cz/






System overview

Implementation language: Python3Data model: IDEAData storage: PostgreSQLNetwork communication protocol: Warden


https://www.python.org/

https://idea.cesnet.cz/en/index


https://warden.cesnet.cz/en/index










Jan Mach (CESNET, a.l.e.) Mentat System v1.1, 2018-10-03 10 /56

Wardenhttps://warden.cesnet.cz/en/index

A system for efficient sharing information aboutdetected events (threats)Simple client-server architectureSending and receiving clientsBased on HTTPS protocol with bidirectionalcertificate authenticationCommunication possible with any HTTPS capablelibraryPython client library and simple filer daemon indistributionCommunity approach in data sharing



Data model: IDEAhttps://idea.cesnet.cz/en/index

Intrusion Detection Extensible AlertJSON based format (NoSQL friendly)Shallow structure, strong typed (SQL friendly)Easily extendable and customizablePossibility to mark anonymised, inaccurate,incomplete or forged dataSupport for aggregated, correlated eventsSupport for various data attachmentsDictionaries for description of various eventattributes (Category, Source/Target type, etc.)



IDEA: Example message

Example Botnet C&C report event

{"Format": "IDEA0","ID": "cca3325c-a989-4f8c-998f-5b0e971f6ef0","DetectTime": "2014-03-05T15:52:22Z","Category": ["Intrusion.Botnet"],"Description": "Botnet Command and Control","Source": [

{"Type": ["Botnet", "CC"],"IP4": ["93.184.216.119"],"Proto": ["tcp", "ircu"],"Port": [6667]

}]

}


Libraries

Utils: geoip2, ply, rrdtool, psycopg2Web: Flask, Jinja2, Babel, WTForms, SQLAlchemyidea-format: Library for working with IDEA messagespynspect: Data filtering librarypyzenkit: Application development framework


https://pypi.org/project/geoip2/

https://pypi.org/project/ply/

https://pypi.org/project/geoip2/

https://pypi.org/project/psycopg2/

https://pypi.org/project/Flask/

https://pypi.org/project/Jinja2/

https://pypi.org/project/Babel/

https://pypi.org/project/WTForms/

https://pypi.org/project/SQLAlchemy/

https://pypi.org/project/idea-format/

https://pypi.org/project/pynspect/

https://pypi.org/project/pyzenkit/






System architecture

statistician reporter backup

hawat

controller

enricher storage

...

inspector


System modules

Real-time event processing modulesmentat-inspector (classification and validation)mentat-enricher (whois, geoip)mentat-storage

Event post processing modules (via database)mentat-reportermentat-statisticianmentat-informant(management scripts)

Control modules and user interfacesmentat-controllerHawat


https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-inspector.html

https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-enricher.html

https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-storage.html

https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-reporter.html

https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-statistician.html

https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-informant.html

https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-controller.html

https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/hawat.html

Module designDesign inspired by Postfix MTA

Hierarchical structure of many small one taskdaemonsFilesystem directory message queues (aka. filerprotocol)

Process-level paralelization support, more instancescan work with the same queuePyZenKit as common framework for moduledevelopment

Application life cycleConfiguration loading, validation and merging (JSON)Deamonisation, logging setupDatabase abstract layerIDEA message abstract layerFiltering library, statistical data processing libraryWHOIS library, DNS resolving library, reportinglibrary


http://www.postfix.org/

https://alchemist.cesnet.cz/pyzenkit/doc/production/html/manual.html

Message exchange queue (1)

aka. filer protocolsimple filesystem directory with substructure:

incoming: input queue, only complete messagespending: daemon work directory, messages inprogresstmp: work directoryerrors: messages causing problems duringprocessing

key requirement: atomic move


Message exchange queue (2)

incoming

pending

errors

tmp

DAEMON QUEUE

A

incoming

pending

errors

tmp

DAEMON QUEUE

B

incoming

pending

errors

tmp

DAEMON QUEUE

C

incoming

pending

errors

tmp

DAEMON QUEUE

D

Inserting message into queue:create new file in tmp subdirectoryfilename is arbitrary, but must be unique within allsubdirectorieswhen done writing, move/rename the file to incomingmove must be atomic, so all subdirectories must beon same partition







Design goals

provide feature rich application out of the boxenable customizability and extendability

built-in features are configurable by text files, and/orcommand line argumentscallback hooks for subclassesprepared for inheritance and method overloading


pyzenkit.jsonconf

reading and writing of JSON configuration filesmerging multiple JSON configuration filessupport for configuration directoriessupport for single line comments in JSON files


pyzenkit.daemonizer

setup directories and limitssetup user and group permissionsdouble fork and split sessionsetup signal handlersclose all open file descriptors (except for possiblelog files)redirect stdin, stdout, stderr to /dev/nulldetect current PID and store it to appropriate PID fileat exit remove PID file


pyzenkit.baseapp (1)

base implementation for generic console applicationFeatures:

application life-cycle managementapplication configuration servicecommand line argument parsing servicelogging servicepersistent state serviceapplication runlog serviceplugin system (experimental)application actions



Application usage modes:runplugin

Application life cycle:initsetupprocessevaluateteardown



Built-in actions:config-viewrunlog-dumprunlog-viewrunlogs-dumprunlogs-listrunlogs-evaluate



example implementation can be found in modulesource codedocumentation: https://alchemist.cesnet.cz/

# On Debian Jessie try following (as root):cd /usr/local/lib/python3.4/dist-packagespython3 pyzenkit/baseapp.py --helppython3 pyzenkit/baseapp.pypython3 pyzenkit/baseapp.py --action runlogs-evaluate


https://alchemist.cesnet.cz/pyzenkit/doc/production/html/doc/_doclib/api_pyzenkit.baseapp.html

pyzenkit.zenscript (1)

base implementation for generic console scriptapplicationbased on pyzenkit.baseappAdditional features:

support for executing multiple different commandsexecution modes: default, regular, shellsupport for executions in regular time intervals


pyzenkit.zenscript (2)


# On Debian Jessie try following (as root):cd /usr/local/lib/python3.4/dist-packagespython3 pyzenkit/zenscript.py --helppython3 pyzenkit/zenscript.pypython3 pyzenkit/zenscript.py --command alternativepython3 pyzenkit/zenscript.py --action runlogs-evaluate


https://alchemist.cesnet.cz/pyzenkit/doc/production/html/doc/_doclib/api_pyzenkit.zenscript.html

pyzenkit.zendaemon (1)

base implementation for generic daemonapplicationbased on pyzenkit.baseappAdditional features:

fully automated daemonization processevent driven designsupport for handling arbitrary signalssupport for modularity with daemon components



Event driven design:infinite event loop and event schedulerevents are being emited in different parts ofapplicationevent callbacks must be registered to handle eventsmultiple event callback may handle single event(pipeline)

Event scheduling:scheduleschedule_nextschedule_afterschedule_at



Signal handling:SIGINTSIGUSR1SIGUSR2

Sending signals:

# On Debian Jessie try following (as root):cd /usr/local/lib/python3.4/dist-packagespython3 pyzenkit/zendaemon.py --no-daemonpython3 pyzenkit/zendaemon.py --action signal-usr1python3 pyzenkit/zendaemon.py --action=signal-usr2



COMPONENT 1 COMPONENT 2 COMPONENT 3

HANDLER EVENT A HANDLER EVENT A

HANDLER EVENT B HANDLER EVENT B

HANDLER EVENT CHANDLER EVENT C

SCHEDULER

QUEUE E-A E-B E-A E-C E-C E-B

Real-time message processing module

Daemon components:actual workers in the designthe daemon object is in fact only a container forcomponentscomponents must be registered into the daemonobjectgreat for code reusability



example implementations can be found in modulesource codedocumentation: https://alchemist.cesnet.cz/

# On Debian Jessie try following (as root):cd /usr/local/lib/python3.4/dist-packagespython3 pyzenkit/zendaemon.py --helppython3 pyzenkit/zendaemon.py --no-daemonpython3 pyzenkit/zendaemon.py --action runlogs-evaluate


https://alchemist.cesnet.cz/pyzenkit/doc/production/html/doc/_doclib/api_pyzenkit.zendaemon.html






mentat.daemon.piper (1)

base implementation pipe-like message processingdaemonbased on pyzenkit.zendaemonAdditional features:

preconfigured message queue features:automated inclusion and bootstrapping ofmentat.daemon.component.filer daemon componentadditional configurations and command linearguments related to filer protocol.


mentat.daemon.piper (2)


# On Debian Jessie try following (as root):cd /usr/lib/python3/dist-packagespython3 mentat/daemon/piper.py --helppython3 mentat/daemon/piper.py --no-daemonpython3 mentat/daemon/piper.py --action runlogs-evaluate


https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/api_mentat.daemon.piper.html

Remarks

project is still evolvingthere are many examples directly in the modulesource codeuse existing modules as templates for creating newoneslocal Makefile may be usefull:

make pyflakesmake pylintmake testmake docs


Mentat repository structure

/bin: executables (simple)/conf: configuration files and cron scripts/lib: Python libraries/submodules: local copies of some custom libraries












Overview

Option 1: real-time processing module can beanything that can work according to the filerprotocolOption 2: use pyzenkit and mentat frameworks







DemoPiperDaemon (2)

import pyzenkitimport mentat . constimport mentat .daemon. piper

class DemoPrintComponent( pyzenkit .zendaemon.ZenDaemonComponent) :

def get_events ( se l f ) :return [

{’event ’ : ’message_process ’ ,’ callback ’ : se l f . cbk_event_message_process ,’prepend ’ : False

}]

def cbk_event_message_process( self , daemon, args ) :daemon. logger . info (

”Processing␣message:␣ ’{} ’:␣’{} ’” .format(args [ ’ id ’ ] , str (args [ ’data ’ ] ) . s t r ip ( )

))daemon.queue. schedule ( ’message_commit ’ , args )se l f . inc_s tat i s t i c ( ’ cnt_printed ’ )return (daemon.FLAG_CONTINUE, None)


DemoPiperDaemon (2)class DemoPiperDaemon(mentat .daemon. piper .PiperDaemon) :

def __ in i t __ ( se l f ) :super ( ) . __ in i t __ (

name = ’mentat−demopiper .py ’ ,description = ’DemoPiperDaemon␣−␣Demonstration␣daemon’ ,path_bin = ’ / usr / local / bin ’ ,path_cfg = ’ /tmp’ ,path_log = ’ / var /mentat / log ’ ,path_run = ’ / var /mentat / run ’ ,path_tmp = ’ /tmp’ ,

default_config_dir = None,default_queue_in_dir = ’ / var /mentat / spool /mentat−demopiper .py ’ ,default_queue_out_dir = None,

schedule = [( ’message_enqueue ’ , { ’data ’ : ’{”testA1”:␣1,␣”testA2”:␣2} ’ }) ,( ’message_enqueue ’ , { ’data ’ : ’{”testB1”:␣1,␣”testB2”:␣2} ’ }) ,(mentat . const .DFLT_EVENT_START, )

] ,schedule_after = [

(mentat . const .DFLT_INTERVAL_STATISTICS , mentat . const .DFLT_EVENT_LOG_STATISTICS)] ,

components = [DemoPrintComponent( )

])


DemoPiperDaemon (3)

i f __name__ == ”__main__” :DemoPiperDaemon( ) . run ( )


DemoPiperDaemon (4)save previous code to file:/etc/mentat/examples/mentat-demopiper.pycreate configuration file:/tmp/mentat-demopiper.pyadd module pipeline in:/etc/mentat/conf/mentat-storage.py.confadd module to/etc/mentat/conf/mentat-controller.py.conf

# Create symlink to example:ln −s / etc /mentat /examples/mentat−demopiper .py / usr / local /bin /mentat−demopiper .py# Stop a l l currently running componentsmentat−control ler .py −−command stop# Start a l l currently componentsmentat−control ler .py −−command start# Generate test messagesmentat−ideagen .py −−count 10# View log f i l et a i l −f / var /mentat / log /mentat−demopiper .py. log


DemoPiperDaemon (5)

adding more command line arguments:

# Add to DemoPiperDaemon classdef _init_argparser ( self , ∗∗kwargs ) :

”””:param kwargs : Various additional parameters passed down from object constructor .: return : Valid argument parser object .: rtype : argparse .ArgumentParser”””argparser = super ( ) . _init_argparser(∗∗kwargs)

arggroup_daemon = argparser .add_argument_group( ’custom␣daemon␣arguments ’ )arggroup_daemon.add_argument(

’−−reload−interval ’ ,type = int ,default = None,help = ’time␣interval␣for␣reloading␣internal␣plugins␣in␣seconds ’ )

return argparser


DemoPiperDaemon (6)

default values for configurations

# Add to DemoPiperDaemon classdef _ in i t_conf ig ( self , cfgs , ∗∗kwargs ) :

”””:param l i s t cfgs : Additional set of configurations .:param kwargs : Various additional parameters passed down from constructor .: return : Default configuration structure .: rtype : dict”””cfgs = (

( ’something ’ , None) ,( ’ reload_interval ’ , 300)

) + cfgsreturn super ( ) . _ in i t_conf ig ( cfgs , ∗∗kwargs)







Hawat

Topic for another dayImplemented using Flask, Jinja2, Babel, SQLAlchemyand Mentat frameworksModularization using Flask blueprintsCustomized Flask classes for deeper integration

View classes for common tasks (item management,searching, ...)Application menu, item context menus, ...

Read Flask‘s documentation!


http://flask.pocoo.org/docs/1.0/

http://jinja.pocoo.org/docs/2.10/

http://babel.pocoo.org/en/latest/

http://docs.sqlalchemy.org/en/latest/

http://flask.pocoo.org/docs/1.0/blueprints/

http://flask.pocoo.org/docs/1.0/patterns/subclassing/

http://flask.pocoo.org/docs/1.0/views/

http://flask.pocoo.org/docs/1.0/






Essential resources

Homeproj: Project issue trackerPrimary code repositoryOfficial documentationAlchemist: automated build system


https://homeproj.cesnet.cz/projects/mentat

https://homeproj.cesnet.cz/git/mentat-ng.git/

https://alchemist.cesnet.cz/mentat/doc/production/html/manual.html

https://alchemist.cesnet.cz

Additional resources

Project Mentat: official websiteProject Warden: official websiteIDEA: official websitePostgreSQL: official websiteSphinx: official website


https://mentat.cesnet.cz/en/index




http://www.sphinx-doc.org/en/stable/

Thank you for your attention

Jan [email protected]


mailto:[email protected]

Documents

Mentat System - Introduction for developersAgenda 1 Introduction 2 Overview 3 Design Technologies Architecture PyZenKitframework Mentatframework 4 Creatingdaemonmodule Overview DemoPiperDaemon.py