Administration Guide - support-public.cfm.quest.com · Introduction to syslog-ng 32 What syslog-ng is 32 ... The syslog-ng OSE quick-start guide 77 Configuring syslog-ng on client

syslog-ngOpenSourceEdition3.18

AdministrationGuide

Copyright 2018 One Identity LLC.

ALL RIGHTS RESERVED.

Thisguidecontainsproprietaryinformationprotectedbycopyright.Thesoftwaredescribedinthisguideisfurnishedunderasoftwarelicenseornondisclosureagreement.Thissoftwaremaybeusedorcopiedonlyinaccordancewiththetermsoftheapplicableagreement.Nopartofthisguidemaybereproducedortransmittedinanyformorbyanymeans,electronicormechanical,includingphotocopyingandrecordingforanypurposeotherthanthepurchaserspersonalusewithoutthewrittenpermissionofOneIdentityLLC.TheinformationinthisdocumentisprovidedinconnectionwithOneIdentityproducts.Nolicense,expressorimplied,byestoppelorotherwise,toanyintellectualpropertyrightisgrantedbythisdocumentorinconnectionwiththesaleofOneIdentityLLCproducts.EXCEPTASSETFORTHINTHETERMSANDCONDITIONSASSPECIFIEDINTHELICENSEAGREEMENTFORTHISPRODUCT,ONEIDENTITYASSUMESNOLIABILITYWHATSOEVERANDDISCLAIMSANYEXPRESS,IMPLIEDORSTATUTORYWARRANTYRELATINGTOITSPRODUCTSINCLUDING,BUTNOTLIMITEDTO,THEIMPLIEDWARRANTYOFMERCHANTABILITY,FITNESSFORAPARTICULARPURPOSE,ORNON-INFRINGEMENT.INNOEVENTSHALLONEIDENTITYBELIABLEFORANYDIRECT,INDIRECT,CONSEQUENTIAL,PUNITIVE,SPECIALORINCIDENTALDAMAGES(INCLUDING,WITHOUTLIMITATION,DAMAGESFORLOSSOFPROFITS,BUSINESSINTERRUPTIONORLOSSOFINFORMATION)ARISINGOUTOFTHEUSEORINABILITYTOUSETHISDOCUMENT,EVENIFONEIDENTITYHASBEENADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGES.OneIdentitymakesnorepresentationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthisdocumentandreservestherighttomakechangestospecificationsandproductdescriptionsatanytimewithoutnotice.OneIdentitydoesnotmakeanycommitmenttoupdatetheinformationcontainedinthisdocument.Ifyouhaveanyquestionsregardingyourpotentialuseofthismaterial,contact:

OneIdentityLLC.Attn:LEGALDept4PolarisWayAlisoViejo,CA92656

RefertoourWebsite(http://www.OneIdentity.com)forregionalandinternationalofficeinformation.

Patents

OneIdentityisproudofouradvancedtechnology.Patentsandpendingpatentsmayapplytothisproduct.Forthemostcurrentinformationaboutapplicablepatentsforthisproduct,pleasevisitourwebsiteathttp://www.OneIdentity.com/legal/patents.aspx.

Trademarks

OneIdentityandtheOneIdentitylogoaretrademarksandregisteredtrademarksofOneIdentityLLC.intheU.S.A.andothercountries.ForacompletelistofOneIdentitytrademarks,pleasevisitourwebsiteatwww.OneIdentity.com/legal.Allothertrademarksarethepropertyoftheirrespectiveowners.

Legend

WARNING: A WARNING icon indicates a potential for property damage, personal injury, or death.

CAUTION: A CAUTION icon indicates potential damage to hardware or loss of data if instructions are not followed.

IMPORTANT,NOTE,TIP,MOBILE,orVIDEO:Aninformationiconindicatessupportinginformation.

syslog-ngOSEAdministrationGuideUpdated-November2018Version-3.18

http://www.oneidentity.com/http://www.oneidentity.com/legal/patents.aspxhttp://www.oneidentity.com/legal

Contents

Preface 19

Summaryofcontents 19

Targetaudienceandprerequisites 20

Productscoveredinthisguide 20

Summaryofchanges 21

Version3.17-3.18 21

Version3.16-3.17 22

Version3.15-3.16 22

Version3.14-3.15 23

Version3.13-3.14 23

Version3.12-3.13 24

Version3.11-3.12 25

Version3.10-3.11 26

Version3.9-3.10 26

Version3.8-3.9 27

Version3.7-3.8 28

Version3.6-3.7 29

Version3.5-3.6 30

Acknowledgments 31

Introduction to syslog-ng 32

Whatsyslog-ngis 32

Secureandreliablelogtransfer 32

Flexibledataextractionandprocessing 33

Bigdataclusters 33

Messagequeuesupport 33

SQL,NoSQL,andmonitoring 34

Wideprotocolandplatformsupport 34

Whatsyslog-ngisnot 34

Whyissyslog-ngneeded? 34

Whatisnewinsyslog-ngOpenSourceEdition3.18? 35

Whousessyslog-ng? 36

syslog-ng OSE 3.18 Administration Guide 3

Supportedplatforms 36

The concepts of syslog-ng 38

Thephilosophyofsyslog-ng 38

Loggingwithsyslog-ng 38

Therouteofalogmessageinsyslog-ng 39

Modesofoperation 40

Clientmode 40

Relaymode 41

Servermode 41

Globalobjects 42

Timezonesanddaylightsaving 43

Howsyslog-ngOSEassignstimezonetothemessage 44

Anoteontimezonesandtimestamps 45

Productlicensing 45

Highavailabilitysupport 45

Thestructureofalogmessage 45

BSD-syslogorlegacy-syslogmessages 46

ThePRImessagepart 46

TheHEADERmessagepart 48

TheMSGmessagepart 48

IETF-syslogmessages 48

ThePRImessagepart 49

TheHEADERmessagepart 50

TheSTRUCTURED-DATAmessagepart 51

TheMSGmessagepart 51

Enterprise-widemessagemodel(EWMM) 51

Messagerepresentationinsyslog-ngOSE 52

Structuringmacros,metadata,andothervalue-pairs 54

Specifyingdatatypesinvalue-pairs 55

value-pairs() 56

Thingstoconsiderwhenforwardingmessagesbetweensyslog-ngOSEhosts 61

Commercialversionofsyslog-ng 63

Installing syslog-ng 65

Compilingsyslog-ngfromsource 65


Compilingoptionsofsyslog-ngOSE 67

Uninstallingsyslog-ngOSE 70

ConfiguringMicrosoftSQLServertoacceptlogsfromsyslog-ng 70

The syslog-ng OSE quick-start guide 77

Configuringsyslog-ngonclienthosts 77

Configuringsyslog-ngonserverhosts 80

Configuringsyslog-ngrelays 82

Configuringsyslog-ngonrelayhosts 82

Howrelayinglogmessagesworks 84

The syslog-ng OSE configuration file 86

Locationofthesyslog-ngconfigurationfile 86

Theconfigurationsyntaxindetail 86

Notesabouttheconfigurationsyntax 89

Definingconfigurationobjectsinline 90

Usingchannelsinconfigurationobjects 91

Globalandenvironmentalvariables 93

Modulesinsyslog-ngOSE 94

Loadingmodules 94

Managingcomplexsyslog-ngconfigurations 95

Includingconfigurationfiles 95

Reusingconfigurationblocks 96

Mandatoryparameters 98

Passingargumentstoconfigurationblocks 99

Generatingconfigurationblocksfromascript 100

Pythoncodeinexternalfiles 102

source: Read, receive, and collect log messages 104

Howsourceswork 105

default-network-drivers:Receiveandparsecommonsyslogmessages 108

default-network-drivers()sourceoptions 110

internal:Collectinginternalmessages 113

internal()sourceoptions 114

file:Collectingmessagesfromtextfiles 115

Notesonreadingkernelmessages 116

file()sourceoptions 116


wildcard-file:Collectingmessagesfrommultipletextfiles 127

wildcard-file()sourceoptions 128

linux-audit:CollectingmessagesfromLinuxauditlogs 141

linux-audit()sourceoptions 142

network:CollectingmessagesusingtheRFC3164protocol(network()driver) 143

network()sourceoptions 144

nodejs:ReceivingJSONmessagesfromnodejsapplications 156

nodejs()sourceoptions 157

mbox:Convertinglocale-mailmessagestologmessages 159

mbox()sourceoptions 160

osquery:Collectandparseosqueryresultlogs 161

osquery()sourceoptions 164

pipe:Collectingmessagesfromnamedpipes 167

pipe()sourceoptions 167

pacct:CollectingprocessaccountinglogsonLinux 178

pacct()options 178

program:Receivingmessagesfromexternalapplications 180

program()sourceoptions 181

python:writingserver-stylePythonsources 187

Methodsofthepython()source 190

PythonLogMessageAPI 191

python()andpython-fetcher()sourceoptions 192

python-fetcher:writingfetcher-stylePythonsources 197

Methodsofthepython-fetcher()source 199

snmptrap:ReadNet-SNMPtraps 201

snmptrap()sourceoptions 204

sun-streams:CollectingmessagesonSunSolaris 207

sun-streams()sourceoptions 208

syslog:CollectingmessagesusingtheIETFsyslogprotocol(syslog()driver) 214

syslog()sourceoptions 216

system:Collectingthesystem-specificlogmessagesofaplatform 227

system()sourceoptions 229

systemd-journal:Collectingmessagesfromthesystemd-journalsystemlogstorage 232

systemd-journal()sourceoptions 234

systemd-syslog:Collectingsystemdmessagesusingasocket 238


systemd-syslog()sourceoptions 239

tcp,tcp6,udp,udp6:CollectingmessagesfromremotehostsusingtheBSDsyslogprotocolOBSOLETE 241

tcp(),tcp6(),udp()andudp6()sourceoptions:OBSOLETE 241

Changeanoldsourcedrivertothenetwork()driver 242

unix-stream,unix-dgram:CollectingmessagesfromUNIXdomainsockets 243

UNIXcredentialsandothermetadata 243

unix-stream()andunix-dgram()sourceoptions 244

stdin:Collectingmessagesfromthestandardinputstream 253

stdin()sourceoptions 253

destination: Forward, send, and store log messages 264

amqp:PublishingmessagesusingAMQP 266

amqp()destinationoptions 267

elasticsearch:SendingmessagesdirectlytoElasticsearchversion1.x(DEPRECATED) 278

Prerequisites 280

Howsyslog-ngOSEinteractswithElasticsearch 281

Clientmodes 282

Elasticsearchdestinationoptions 282

elasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigher 294

Prerequisites 297

Howsyslog-ngOSEinteractswithElasticsearch 297

Clientmodes 298

SearchGuardandsyslog-ngOSE 299

Elasticsearch2destinationoptions 301

ExampleusecasesofsendinglogstoElasticsearchusingsyslog-ng 321

file:Storingmessagesinplain-textfiles 321

file()destinationoptions 322

graphite:SendingmetricstoGraphite 333

graphite()destinationoptions 334

SendinglogstoGraylog 337

graylog2()destinationoptions 338

hdfs:StoringmessagesontheHadoopDistributedFileSystem(HDFS) 340

Prerequisites 341

Howsyslog-ngOSEinteractswithHDFS 342


StoringmessageswithMapR-FS 343

Kerberosauthenticationwithsyslog-nghdfs()destination 344

HDFSdestinationoptions 345

PostingmessagesoverHTTP 356

HTTPdestinationoptions 357

http:PostingmessagesoverHTTPwithoutJava 361

Batchmode 363

Batchsize 363

Formattingthebatch 363

HTTPdestinationoptions 364

kafka:PublishingmessagestoApacheKafka 379

Prerequisites 381

Howsyslog-ngOSEinteractswithApacheKafka 382

Kafkadestinationoptions 382

loggly:UsingLoggly 388

loggly()destinationoptions 390

logmatic:UsingLogmatic.io 391

logmatic()destinationoptions 393

mongodb:StoringmessagesinaMongoDBdatabase 395

Howsyslog-ngOSEconnectstheMongoDBserver 396

mongodb()destinationoptions 397

network:SendingmessagestoaremotelogserverusingtheRFC3164protocol(network()driver) 406

network()destinationoptions 407

osquery:Sendinglogmessagestoosquery'ssyslogtable 423

osquery()destinationoptions 424

pipe:Sendingmessagestonamedpipes 426

pipe()destinationoptions 427

program:Sendingmessagestoexternalapplications 433

program()destinationoptions 435

pseudofile() 444

pseudofile()destinationoptions 444

python:writingcustomPythondestinations 446

Methodsofthepython()destination 448

Errorhandlinginthepython()destination 449


python()destinationoptions 451

redis:Storingname-valuepairsinRedis 457

redis()destinationoptions 458

riemann:MonitoringyourdatawithRiemann 464

riemann()destinationoptions 465

smtp:GeneratingSMTPmessages(e-mail)fromlogs 476

smtp()destinationoptions 478

Splunk:SendinglogmessagestoSplunk 486

sql:StoringmessagesinanSQLdatabase 486

Usingthesql()driverwithanOracledatabase 488

Usingthesql()driverwithaMicrosoftSQLdatabase 489

Thewaysyslog-nginteractswiththedatabase 491

MySQL-specificinteractionmethods 492

MsSQL-specificinteractionmethods 492

sql()destinationoptions 492

stomp:PublishingmessagesusingSTOMP 504

stomp()destinationoptions 505

syslog:SendingmessagestoaremotelogserverusingtheIETF-syslogprotocol 511

syslog()destinationoptions 512

syslog-ng():Forwardlogstoanothersyslog-ngnode 528

syslog-ng()destinationoptions 529

tcp,tcp6,udp,udp6:SendingmessagestoaremotelogserverusingthelegacyBSD-syslogprotocol(tcp(),udp()drivers) 542

tcp(),tcp6(),udp(),andudp6()destinationoptions 543

Changeanolddestinationdrivertothenetwork()driver 543

Telegram:SendingmessagestoTelegram 544

telegram()destinationoptions 545

unix-stream,unix-dgram:SendingmessagestoUNIXdomainsockets 548

unix-stream()andunix-dgram()destinationoptions 548

usertty:Sendingmessagestoauserterminal:usertty()destination 558

WriteyourowncustomdestinationinJavaorPython 558

Client-sidefailover 558

log: Filter and route log messages using log paths, flags, and filters 561

Logpaths 561

Embeddedlogstatements 562


Usingembeddedlogstatements 564

if-else-elif:Conditionalexpressions 566

Junctionsandchannels 566

Logpathflags 569

Managingincomingandoutgoingmessageswithflow-control 572

Flow-controlandmultipledestinations 576

Configuringflow-control 576

Usingdisk-basedandmemorybuffering 578

Enablingreliabledisk-basedbuffering 580

Enablingnormaldisk-basedbuffering 581

Enablingmemorybuffering 581

Aboutdiskqueuefiles 582

Filters 583

Usingfilters 583

Combiningfilterswithbooleanoperators 584

Comparingmacrovaluesinfilters 585

Usingwildcards,specialcharacters,andregularexpressionsinfilters 586

Taggingmessages 587

Filterfunctions 588

Droppingmessages 593

Global options of syslog-ng OSE 595

Configuringglobalsyslog-ngoptions 595

Globaloptions 595

TLS-encrypted message transfer 613

SecureloggingusingTLS 613

EncryptinglogmessageswithTLS 614

ConfiguringTLSonthesyslog-ngclients 615

ConfiguringTLSonthesyslog-ngserver 616

MutualauthenticationusingTLS 618

ConfiguringTLSonthesyslog-ngclients 619

ConfiguringTLSonthesyslog-ngserver 620

Password-protectedkeys 622

TLSoptions 623

template and rewrite: Format, modify, and manipulate log messages 630


Customizemessageformatusingmacrosandtemplates 630

Formattingmessages,filenames,directories,andtablenames 631

Templatesandmacros 631

Date-relatedmacros 633

Hardvs.softmacros 634

Macrosofsyslog-ngOSE 635

Usingtemplatefunctions 644

Templatefunctionsofsyslog-ngOSE 645

Modifyingtheon-the-wiremessageformat 668

Modifyingmessagesusingrewriterules 669

Replacingmessageparts 669

Settingmessagefieldstospecificvalues 671

Unsettingmessagefields 674

CreatingcustomSDATAfields 675

Settingmultiplemessagefieldstospecificvalues 676

map-value-pairs:Renamevalue-pairstonormalizelogs 677

Conditionalrewrites 677

Howconditionalrewritingworks 678

Addinganddeletingtags 678

Anonymizingcreditcardnumbers 679

Regularexpressions 680

Typesandoptionsofregularexpressions 681

Optimizingregularexpressions 682

parser: Parse and segment structured messages 684

Parsingsyslogmessages 685

Optionsofsyslog-parserparsers 687

Parsingmessageswithcomma-separatedandsimilarvalues 689

OptionsofCSVparsers 692

Parsingkey=valuepairs 696

Optionsofkey=valueparsers 699

The JSON parser 700

TheJSONparserTheJSONparser 700

OptionsofJSONparsers 703

TheXMLparser 705


OptionsofXMLparsers 709

Parsingdatesandtimestamps 711

Optionsofdate-parser()parsers 712

TheApacheAccessLogParser 714

Optionsofapache-accesslog-parser()parsers 715

TheCiscoParser 716

TheLinuxAuditParser 718

Optionsoflinux-audit-parser()parsers 720

ThePythonParser 721

Parsingenterprise-widemessagemodel(EWMM)messages 727

Thesudoparser 727

Theiptablesparser 728

db-parser: Process message content with a pattern database (patterndb) 730

Classifyinglogmessages 730

Thestructureofthepatterndatabase 731

Howpatternmatchingworks 732

Artificialignorance 733

Usingpatterndatabases 734

Usingparserresultsinfiltersandtemplates 735

Downloadingsamplepatterndatabases 737

Correlatinglogmessagesusingpatterndatabases 738

Referencingearliermessagesofthecontext 740

Triggeringactionsforidentifiedmessages 741

Conditionalactions 743

Externalactions 744

Actionsandmessagecorrelation 745

Creatingpatterndatabases 748

Usingpatternparsers 748

Patternparsersofsyslog-ngOSE 750

What'snewinthesyslog-ngpatterndatabaseformatV5 753

Thesyslog-ngpatterndatabaseformat 753

Element:patterndb 755

Element:ruleset 755

Element:patterns 756

Element:rules 757


Element:rule 758

Element:patterns 760

Element:urls 761

Element:values 762

Element:examples 762

Element:example 763

Element:actions 764

Element:action 766

Element:create-context 768

Element:tags 771

Correlating log messages 772

Correlatingmessagesusingthegrouping-by()parser 772

Referencingearliermessagesofthecontext 776

Optionsofgrouping-byparsers 777

Enriching log messages with external data 781

Addingmetadatafromanexternalfile 781

Usingfiltersasselector 783

Optionsadd-contextual-data() 784

LookingupGeoIPdatafromIPaddresses(DEPRECATED) 786

Optionsofgeoipparsers 788

LookingupGeoIP2datafromIPaddresses 789

Referringtopartsofthemessageasamacro 790

UsingtheGeoIP2parser 790

TransferringyourlogstoElasticsearchusingGeoIP2 791

Optionsofgeoip2parsers 792

Statistics of syslog-ng 794

Metricsandcountersofsyslog-ngOSE 794

Logstatisticsfromtheinternal()source 797

Multithreading and scaling in syslog-ng OSE 799

Multithreadingconceptsofsyslog-ngOSE 799

Configuringmultithreading 801

Optimizingmultithreadedperformance 801

Troubleshooting syslog-ng 803


Possiblecausesoflosinglogmessages 804

Creatingsyslog-ngcorefiles 805

Collectingdebugginginformationwithstrace,truss,ortusc 805

Runningafailurescript 806

Stoppingsyslog-ng 807

Reportingbugsandfindinghelp 808

Recoverdatafromorphaneddiskbufferfiles 808

Nolocallogsafterspecifyinganunusualstoragedirectory 808

Nologsafterspecifyinganunusualportnumber 808

Errormessages 809

Best practices and examples 811

Generalrecommendations 811

Handlinglargemessageload 811

Usingnameresolutioninsyslog-ng 812

Resolvinghostnameslocally 813

Collectinglogsfromchroot 813

Configuringlogrotation 814

The syslog-ng manual pages 816

Thedqtooltoolmanualpage 816

Name 816

Synopsis 816

Description 816

Thecatcommand 817

Files 818

Seealso 818

Author 818

Copyright 818

Theloggenmanualpage 818

Name 819

Synopsis 819

Description 819

Options 819

Examples 822

Files 822


Seealso 822

Author 823

Copyright 823

Thepdbtoolmanualpage 823

Name 823

Synopsis 823

Description 824

Thedictionarycommand 824

Thedumpcommand 824

Thematchcommand 825

Themergecommand 827

Thepatternizecommand 828

Thetestcommand 829

Files 829

Seealso 829

Author 830

Copyright 830

Thesyslog-ngcontroltoolmanualpage 830

Name 830

Synopsis 830

Description 831

Enablingtroubleshootingmessages 831

syslog-ng-ctlquery 832

Thestatscommand 834

Handlingpassword-protectedprivatekeys 835

Displayingtheconfiguration 836

Reloadingtheconfiguration 836

Files 837

Seealso 837

Author 837

Copyright 837

Thesyslog-ng-debunmanualpage 837

Name 838

Synopsis 838

Description 838


GeneralOptions 838

Debugmodeoptions 839

Systemcalltracing 839

Packetcaptureoptions 839

Examples 840

Files 841

Seealso 841

Author 841

Copyright 841

Thesyslog-ngmanualpage 841

Name 842

Synopsis 842

Description 842

Options 842

Files 845

Seealso 845

Author 845

Copyright 845

Thesyslog-ng.confmanualpage 846

Name 846

Synopsis 846

Description 846

Basicconceptsofsyslog-ngOSE 846

Configuringsyslog-ng 847

Files 851

Seealso 852

Author 852

Copyright 852

Third-party contributions 853

GNUGeneralPublicLicense 853

Preamble 853

TERMSANDCONDITIONSFORCOPYING,DISTRIBUTIONANDMODIFICATION 854

Section0 854

Section1 855

Section2 855


Section3 856

Section4 856

Section5 856

Section6 857

Section7 857

Section8 857

Section9 858

Section10 858

NOWARRANTYSection11 858

Section12 858

HowtoApplyTheseTermstoYourNewPrograms 859

GNULesserGeneralPublicLicense 860

Preamble 860

TERMSANDCONDITIONSFORCOPYING,DISTRIBUTIONANDMODIFICATION 862

Section0 862

Section1 862

Section2 863

Section3 863

Section4 864

Section5 864

Section6 865

Section7 866

Section8 866

Section9 866

Section10 866

Section11 867

Section12 867

Section13 867

Section14 868



HowtoApplyTheseTermstoYourNewLibraries 868

Licenseattributions 869

Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License 870


About us 876

Contactingus 876

Technicalsupportresources 876

Glossary 877


Preface

Welcometothesyslog-ngOpenSourceEdition3.18AdministratorGuide!

Thisdocumentdescribeshowtoconfigureandmanagesyslog-ng.Backgroundinformationforthetechnologyandconceptsusedbytheproductisalsodiscussed.

Summary of contents

Introductiontosyslog-ngdescribesthemainfunctionalityandpurposeofsyslog-ngOSE.

Theconceptsofsyslog-ngdiscussesthetechnicalconceptsandphilosophiesbehindsyslog-ngOSE.

Installingsyslog-ngdescribeshowtoinstallsyslog-ngOSEonvariousUNIX-basedplatformsusingtheprecompiledbinaries.

Thesyslog-ngOSEquick-startguideprovidesabrieflyexplainshowtoperformthemostcommonlogcollectingtaskswithsyslog-ngOSE.

Thesyslog-ngOSEconfigurationfilediscussestheconfigurationfileformatandsyntaxindetail,andexplainshowtomanagelarge-scaleconfigurationsusingincludedfilesandreusableconfigurationsnippets.

source:Read,receive,andcollectlogmessagesexplainshowtocollectandreceivelogmessagesfromvarioussources.

destination:Forward,send,andstorelogmessagesdescribesthedifferentmethodstostoreandforwardlogmessages.

log:Filterandroutelogmessagesusinglogpaths,flags,andfiltersexplainshowtorouteandsortlogmessages,andhowtousefilterstoselectspecificmessages.

Globaloptionsofsyslog-ngOSEliststheglobaloptionsofsyslog-ngOSEandexplainshowtousethem.

TLS-encryptedmessagetransfershowshowtosecureandauthenticatelogtransportusingTLSencryption.

templateandrewrite:Format,modify,andmanipulatelogmessagesdescribeshowtocustomizemessageformatusingtemplatesandmacros,howtorewriteandmodifymessages,andhowtouseregularexpressions.

parser:Parseandsegmentstructuredmessagesdescribeshowtosegmentandprocessstructuredmessageslikecomma-separatedvalues.

db-parser:Processmessagecontentwithapatterndatabase(patterndb)explainshowtoidentifyandprocesslogmessagesusingapatterndatabase.

Correlatinglogmessagesexplainshowtocorrelatelogmessagesthatmatchasetoffiltersorthatareidentifiedusingapatterndatabase.

syslog-ng OSE 3.18 Administration Guide

Preface19

Enrichinglogmessageswithexternaldataexplainshowtoimportdatafromexternalsourcestoincludeinthelogmessages,thusextending,enriching,andcomplementingthedatafoundinthelogmessage.

Statisticsofsyslog-ngdetailstheavailablestatisticsthatsyslog-ngOSEcollectsabouttheprocessedlogmessages.

Multithreadingandscalinginsyslog-ngOSEdescribeshowtoconfiguresyslog-ngOSEtousemultipleprocessors,andhowtooptimizeitsperformance.

Troubleshootingsyslog-ngofferstipstosolvingproblems.

Bestpracticesandexamplesgivesrecommendationstoconfigurespecialfeaturesofsyslog-ngOSE.

Thesyslog-ngmanualpagescontainsthemanualpagesofthesyslog-ngOSEapplication.

Third-partycontributionsincludesthetextofthelicensesapplicabletosyslog-ngOpenSourceEdition.

CreativeCommonsAttributionNon-commercialNoDerivatives(by-nc-nd)LicenseincludesthetextoftheCreativeCommonsAttributionNon-commercialNoDerivatives(by-nc-nd)LicenseapplicabletoThesyslog-ngOpenSourceEdition3.18AdministratorGuide.

Target audience and prerequisites

Thisguideisintendedforsystemadministratorsandconsultantsresponsiblefordesigningandmaintainingloggingsolutionsandlogcenters.ItisalsousefulforITdecisionmakerslookingforatooltoimplementcentralizedlogginginheterogeneousenvironments.

Thefollowingskillsandknowledgearenecessaryforasuccessfulsyslog-ngadministrator:

l Atleastbasicsystemadministrationknowledge.

l Anunderstandingofnetworks,TCP/IPprotocols,andgeneralnetworkterminology.

l WorkingknowledgeoftheUNIXorLinuxoperatingsystem.

l In-depthknowledgeoftheloggingprocessofvariousplatformsandapplications.

l Anunderstandingofthelegacysyslog(BSD-syslog)protocolandthenewsyslog(IETF-syslog)protocolstandard.

Products covered in this guide

Thisguidedescribestheuseofthefollowingproducts:

l syslog-ngOpenSourceEdition(syslog-ngOSE)3.18.1andlater


Preface20

https://www.ietf.org/rfc/rfc3164.txthttps://tools.ietf.org/html/rfc5424https://tools.ietf.org/html/rfc5424

Summary of changes

ThissectionliststhechangesofThesyslog-ngOpenSourceEditionAdministratorGuide.

Version 3.17 - 3.18

Changes in product:

l Startingwithsyslog-ngOSEversion3.18,youcanwritecustommessagesourcesinPython.Bothserver-styleandfetcher-stylesourcesaresupported.Formoredetails,see"python:writingserver-stylePythonsources"intheAdministrationGuideand"python-fetcher:writingfetcher-stylePythonsources"intheAdministrationGuide.

l Thehttp()destinationcannowsendabatchoflogmessagesinasingleHTTPrequest,greatlyimprovingtheperformance.Inaddition,thisfeaturealsoallowsyoutopostproperJSON-encodedarraysasPOSTpayloads,whichisrequiredbyseveralRESTAPIs.Fordetails,seeAdministrationGuide.

l Whenhdfs-append-enabledissettotrue,syslog-ngOSEwillappendnewdatatotheendofanalreadyexistingHDFSfile.Notethatinthiscase,archivingisautomaticallydisabled,andsyslog-ngOSEwillignorethehdfs-archive-diroption.

l Thehdfsdestinationnowsupportsthetime-reap()option.Fordetails,see"HDFSdestinationoptions"intheAdministrationGuide.

l Newtemplatefunctionsareavailable:url-decode()andbase64-encode().Fordetails,see"Templatefunctionsofsyslog-ngOSE"intheAdministrationGuide.

l Thesyslog-ng-ctl configcommandcandisplaythecontentsoftheconfigurationfilethatsyslog-ngOSEiscurrentlyrunning.

l Therekeyoptionofvalue-pairs()nowsupportsanewtransformation:shift-levels.Itcutsdot-delimited"levels"inthename(includingtheinitialdot).Forexample,--shift-levels 2deletestheprefixuptotheseconddotinthenameofthekey:.iptables.SRCbecomesSRC

Fordetails,see"value-pairs()"intheAdministrationGuide.

l Thevalue-pairs()optionnowhasanewscope:none.Thisscoperesetspreviouslyaddedscopes,makingitpossibletogetremoveautomaticallyaddedname-valuepairsfromthescope.


l Themax-channelandframe-sizeoptionshavebeenaddedtotheamqp()destination.

Changes in documentation:

l Extendingsyslog-ngOSEinPythonhasbeensupportedforseveralreleases,butsofarthisfeaturewasmostlyundocumented.Nowyoucanfindmoredetailsaboutthis


Preface21

https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-writing-server-style-python-sources/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-fetcher-writing-fetcher-style-python-sources/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide//https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/hdfs-storing-messages-on-the-hadoop-distributed-file-system-hdfs/hdfs-destination-options/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/hdfs-storing-messages-on-the-hadoop-distributed-file-system-hdfs/hdfs-destination-options/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/template-and-rewrite-format-modify-and-manipulate-log-messages/customize-message-format-using-macros-and-templates/template-functions-of-syslog-ng-ose/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/the-concepts-of-syslog-ng/structuring-macros-metadata-and-other-value-pairs/value-pairs/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/the-concepts-of-syslog-ng/structuring-macros-metadata-and-other-value-pairs/value-pairs/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/amqp-publishing-messages-using-amqp/amqp-destination-options/

featurein"python:writingcustomPythondestinations"intheAdministrationGuide.

Version 3.16 - 3.17

Changes in product:

l Anewsourcedriver,linux-audit(),hasbeenadded.Thelinux-audit()sourcereadsandautomaticallyparsestheLinuxauditlogs.Fordetails,seelinux-audit:CollectingmessagesfromLinuxauditlogs.

l Anewsystemsourceoption,exclude-kmsg()makesitpossibletoavoidduplicatecollectionofkernellogsorerrorsinkernellogcollection(forexample,inscenarioswherethelogmanagementonthehostsystemandthecontainerizedsolutionarecollectingthekernellogssimultaneously).Whensettoyes,syslog-ngOSEwillomitkernellogsonplatformswheretheyareavailableseparately.

l Youcannowrefertoanyadditionalparametersattheendoftheargumentinablockbyaddingthreedotstoit().Ittellssyslog-ngOSEthatthismacroaccepts`__VARARGS__`,thereforeanyname-valuepaircanbepassedwithoutvalidation.Fordetails,seePassingargumentstoconfigurationblocks.

l Youcannowmakeparametersmandatoryinblockdefinitionsbydefiningthemwithemptybrackets().Fordetails,seeMandatoryparameters.

l Thefailover()optionallowsyoutospecifywhathappensaftersyslog-ngOSEfailsovertoasecondaryserver.Additionally,thefailover-servers()optionhasbeendeprecatedandremovedfromthedocument.Formoreinformationaboutthefailover()option,seeClient-sidefailoveronpage558.

l AddedsupportforthetimestampformatusedbyCiscoUnifiedCallManagerintheCiscoParser.Fordetails,seethesourcecodeofthisparseronGitHub.


l AnoteaboutJVMstillrunningafterdeletingallJavadestinationsandreloadingsyslog-nghasbeenaddedtothedescriptionofJavadestinations.

l Thedefaultvalueofthe--skip-tokensparameteroftheloggenapplicationhasbeenchangedto0.Fordetails,seeTheloggenmanualpage.

Version 3.15 - 3.16

Changes in product:

l Anewdestinationdriver,telegram(),hasbeenadded.Thetelegram()destinationsendslogmessagestoTelegram,whichisasecure,cloud-basedmobileanddesktop


Preface22

https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/python-writing-custom-python-destinations/https://github.com/balabit/syslog-ng/blob/master/scl/cisco/plugin.confhttps://core.telegram.org/

messagingapp.Formoreinformation,seeTelegram:SendingmessagestoTelegram.

l Anewtemplatefunction,urlencode,hasbeenadded.Youcanusetheurlencodetemplatefunctiontogetherwiththetelegram()destinationtosendsyslogmessagestoTelegram.Formoreinformation,seeTemplatefunctionsofsyslog-ngOSE.

l Toensurethatamoduleisloaded,[email protected],seeLoadingmodules.

l Theadd-contextual-data()hasbeenextendedwiththeignore-case()option.Formoreinformation,seeOptionsadd-contextual-data().

l Thehook-commands()hasbeenadded,whichmakesitpossibletoexecuteexternalprogramswhentheyareinitializedortorndown.Thehook-commands()canbeusedforbothsourceanddestinationdrivers.Formoreinformation,seehook-commands().

Version 3.14 - 3.15

Changes in product:

l Itisnowpossibletouseif {},elif {},andelse {}blockstoconfigureconditionalexpressions.Fordetails,seeif-else-elif:Conditionalexpressions.

l Anewlogpathflag,drop-unmatched,hasbeenadded.Thenewflagcausesmessagestobedroppedalongalogpathwhentheydonotmatchafilterorarediscardedbyaparser.Fordetails,seeLogpathflags.

l SupportforElasticsearch'sShieldhasbeenremoved.

l SupportforPOSIXregularexpressionshasbeenremoved.

Version 3.13 - 3.14

Changes in product:

l Youcanusepassword-protectedprivatekeysinthenetwork()andsyslog()sourceanddestinationdrivers.Fordetails,seePassword-protectedkeys.

l Tobettercontroltowhichlogmessagesyouaddcontextualdata,youcanusefiltersasselectors.Inthiscase,thefirstcolumnoftheCSVdatabasefilemustcontainthenameofafilter.Foreachmessage,syslog-ngOSEevaluatesthefiltersintheordertheyappearinthedatabasefile.Ifafiltermatchesthemessage,syslog-ngOSEaddsthename-valuepairrelatedtothefilter.Fordetails,seeUsingfiltersasselector.


Preface23

https://core.telegram.org/

Version 3.12 - 3.13

Changes in product:

l Anewsourcedriver,stdin(),hasbeenadded.Thestdin()drivercollectsmessagesfromthestandardinputstream.Formoreinformation,seestdin:Collectingmessagesfromthestandardinputstream.

l Anewdestination,SendinglogstoGraylog,andatemplatetosendsyslogmessagestoGraylog,format-gelf,hasbeenadded.

l Anewtemplatefunction,getent,hasbeenadded.YoucanusethegetenttemplatefunctiontolookupentriesfromtheNameServiceSwitchlibraries.Formoreinformation,seegetent.

l Thedefaultvaluesofthe--enable-json,--enable-mongodb,and--with-libmongo-clientcompileparametershavechanged.Formoreinformation,seeCompilingoptionsofsyslog-ngOSE.

l Anewcompileoption,--with-module-path,hasbeenadded.Thenewoptionspecifiessyslog-ngOSE'smoduleinstallationdirectory.Formoreinformation,seeCompilingoptionsofsyslog-ngOSE.

l Anewdestinationdriver,osquery(),hasbeenadded.Thenewdriversendslogmessagestoosquery'ssyslogtable.Formoreinformation,seeosquery:Sendinglogmessagestoosquery'ssyslogtable.

l ItisnowpossibletospecifyTLSoptionsinatls()block.Formoreinformation,see:

l amqp()destinationoptions

l HTTPdestinationoptions

l riemann()destinationoptions

l SupportformicrosecondsinRiemanndestinationshasbeenintroduced.Formoreinformation,seeevent-time().

l Moduleauto-loadingnowalsoworksforthesystem()source.Formoreinformation,see--default-modules.


l Anewsectiondescribingcommonerrormessageshasbeenaddedtothedocument.Formoreinformation,seeErrormessages.

l Severalcorrectionsandeditorialchanges.


Preface24

Version 3.11 - 3.12

Changes in product:

l Anewsystemd-journal()sourceoption,calledread-old-records(),hasbeenadded.Formoreinformation,seeread-old-records().

l Anoptioncalledjvm-options()hasbeenadded,whichallowsyoutofine-tuneJavaVirtualMachinesettingswhenconfiguringElasticsearch,HDFS,andApacheKafkadestinations,orwebservicestowhichyousendlogmessagesviatheHTTPprotocol.Fordetails,see:

l Elasticsearchdestinationoptions

l Elasticsearch2destinationoptions

l HDFSdestinationoptions

l HTTPdestinationoptions

l Kafkadestinationoptions

l Globaloptions

l AnewHDFSdestinationoption,calledhdfs-append-enabled()hasbeenadded.Forfurtherinformation,seehdfs-append-enabled().

l Macrosarenowsupportedinthehdfs-file()option.Fordetails,seehdfs-file().

l ThefollowingnewTLSoptionshavebeenadded:

l dhparam-file()

l ecdh-curve-list()

l pkcs12-file().

l Anewparser,capableofprocessinginputinXMLformat,hasbeenadded.Formoreinformation,seeTheXMLparser.


l Addedsectionaboutcommercialversionofsyslog-ng.Formoreinformation,seeCommercialversionofsyslog-ng.

l Addedwarningabouttherequirementtodeletethepersistfileoncethedir()optionofdisk-buffer()hasbeenmodifiedoranewonehasbeenadded.Formoreinformation,seedestination:Forward,send,andstorelogmessages.

l ClarifiedinformationaboutthePythonparser'sdeinit()method.Itrunsnotonlyatasyslog-nggracefulstop,butatareloadtoo.Fordetails,seeMethodsofthepython()parser.



Preface25

Version 3.10 - 3.11

Changes in product:

l LookingupGeoIP2datafromIPaddresseshasbeenaddedtothedocument.

l http:PostingmessagesoverHTTPwithoutJavahasbeenupgradedwithnewimprovements.

l Thegeoip()parserisnowdeprecated.LookingupGeoIPdatafromIPaddresses(DEPRECATED).

l Thetemplate()optionhasbeenaddedtotheApacheAccessLogParser.Fordetails,see:TheApacheAccessLogParser.

l SSL-relatedoptionshavebeenaddedtoamqp()destination.Fordetails,see:amqp()destinationoptions.

l Theprefix()optionhasbeenaddedtotheCiscoparser.Fordetails,see:TheCiscoParser.

l Thedrop-unmatched()optionhasbeenaddedtothedb-parser()statement.Fordetails,see:Usingpatterndatabases.

l Theevent-time()optionhasbeenaddedtotheRiemanndestination.Fordetails,see:riemann:MonitoringyourdatawithRiemann.


l Anewexamplehasbeenaddedtotheosquery()source.Fordetails,see:osquery:Collectandparseosqueryresultlogs.


Version 3.9 - 3.10

Changes in product:

l wildcard-file: Collectingmessages frommultiple text files has been added tothe document.

l snmptrap:ReadNet-SNMPtrapshasbeenaddedtothedocument.

l osquery:Collectandparseosqueryresultlogshasbeenaddedtothedocument.

l Theelasticsearch2()destinationnowsupportsHTTPSmode,includingencryption,andalsopassword-andcertificate-basedauthentication.Fordetails,seeelasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigher.

l Thehttp()destinationnowsupportsencryption,andalsopassword-andcertificate-basedauthentication.Fordetails,seeHTTPdestinationoptions.


Preface26

l Thehdfs()destinationnowsupportsKerberosauthentication.Fordetails,seeKerberosauthenticationwithsyslog-nghdfs()destination.

l ThePythonParserhasbeenaddedtothedocument.

l TheCiscoParserhasbeenaddedtothedocument.

l map-value-pairs: Rename value-pairs to normalize logs has been added to thedocument.

l Thelist-*templatefunctionsallowyoutomanipulatecomma-separatedlists.Fordetails,seeListmanipulation.

l Thenewbasename()anddirname()templatefunctionsallowyoutoeasilyseparatethepathandfilenames.Fordetails,seeTemplatefunctionsofsyslog-ngOSE.

l stardatehasbeenaddedtothedocument.

l create-statement-append()hasbeenaddedtothedocument.

l Thedefaultvalueofthelog-msg-size()optionhasbeenincreasedto64k.Thatwaysyslog-ngOSEwillnottruncatelonglogmessages,whicharegettingincreasinglycommon.


l Splunk:SendinglogmessagestoSplunkhasbeenaddedtothedocument.

l Aboutdiskqueuefileshasbeenaddedtothedocument.

l AnexamplefailurescripthasbeenaddedtoRunningafailurescript.


Version 3.8 - 3.9

Changes in product:

l WhenusingTLS-transport,youcannowusecertainfieldsoftheX.509certificatesasmacros.Fordetails,see.TLS.X509.

l Theelastic2()destinationdrivernowsupportsSearchGuard,analternativesecuritysolutionforElasticsearch.Fordetails,seeSearchGuardandsyslog-ngOSE.

l .TLS.X509hasbeenaddedtothedocument.

l Unsettingmessagefieldshasbeenupdatedwithgroupunset().


l Correctionsandeditorialchanges.


Preface27

https://github.com/floragunncom/search-guard

Version 3.7 - 3.8

Changes in product:

l Enrichinglogmessageswithexternaldatahasbeenaddedtothedocument.

l Correlatinglogmessageshasbeenaddedtothedocument.

l elasticsearch2:SendinglogsdirectlytoElasticsearchandKibana2.0orhigherhasbeenaddedtothedocument.

l http:PostingmessagesoverHTTPwithoutJavahasbeenaddedtothedocument.

l logmatic:UsingLogmatic.iohasbeenaddedtothedocument.

l loggly:UsingLogglyhasbeenaddedtothedocument.

l Disk-basedbufferinghasbeenaddedtosyslog-ngOSE.Fordetails,seeUsingdisk-basedandmemorybuffering.

l What'snewinthesyslog-ngpatterndatabaseformatV5,,hasbeenaddedtoElement:create-contexthasbeenaddedtodb-parser:Processmessagecontentwithapatterndatabase(patterndb).

l Parsingdatesandtimestampshasbeenaddedtoparser:Parseandsegmentstructuredmessages.

l TheApacheAccessLogParserhasbeenaddedtoparser:Parseandsegmentstructuredmessages.

l Newoptionsoftheset()rewriteoperatorhavebeenaddedtoSettingmessagefieldstospecificvalues.

l ArewriteoperatortounsetfieldshasbeenaddedtoUnsettingmessagefields.

l Atemplatefunctionthatformatsname-valuepairsasArcSightCommonEventFormatextensionhasbeenaddedtoformat-cef-extension.

l NumericaltemplatefunctionsthatworkonnumericalvaluesofacorrelationcontexthavebeenaddedtoNumericaloperations.

l Theinherit-environment()optionhasbeenaddedtoprogram:Receivingmessagesfromexternalapplicationsandprogram:Sendingmessagestoexternalapplications.

l @NLSTRING@hasbeenaddedtoUsingpatternparsers.


l LookingupGeoIPdatafromIPaddresses(DEPRECATED)hasbeenmovedtoEnrichinglogmessageswithexternaldata.



Preface28

Version 3.6 - 3.7

Changes in product:

l mbox: Converting local e-mailmessages to logmessages has been added tothe document.

l Thekeep-alive()optionhasbeenaddedtotheprogram()destination.

l The Linux Audit Parser has been added to parser: Parse and segmentstructuredmessages.

l pythonhasbeenaddedtoTemplatefunctionsofsyslog-ngOSE.

l PostingmessagesoverHTTPhasbeenaddedtothedocument.

l Write your own custom destination in Java or Python has been added to thedocument.

l Looking up GeoIP data from IP addresses (DEPRECATED) has been added tothe document.

l Elasticsearchdestinationoptionshasbeenaddedtothedocument.

l kafka:PublishingmessagestoApacheKafkahasbeenaddedtothedocument.

l hdfs:StoringmessagesontheHadoopDistributedFileSystem(HDFS)hasbeenaddedtothedocument.

l Parsingkey=valuepairshasbeenaddedtothedocument.

l format-cimhasbeenaddedtothedocument.

l Simpletemplatescanbedefinedwithoutbraces.Templatescanalsoreferenceothertemplates.Fordetails,seeTemplatesandmacros.

l Customtemplatefunctionscanbedefinedinthesyslog-ngOSEconfiguration.Fordetails,seeUsingtemplatefunctions.

l CSV-parserscanusestringsasdelimiters.Fordetails,seedelimiters().

l IPv6addressescanbefilteredusinganewfilter.Fordetails,seenetmask6().

l Theloggenutilitycansendmessagesindefinitelyusingthe--permanentoption.

l Thessl-options()optionhasbeedaddedtoTLSoptions.

l TLS-supporthasbeenaddedtoriemann()destinationoptions.

l Theextract-solaris-msgid()parserhasbeedaddedtosun-streams:CollectingmessagesonSunSolaris.

l Thecontextoptionofinherit-propertieshasbeedaddedtoActionsandmessagecorrelation.

l flush-lines()hasbeenaddedtothedocument.

l Thesanitize-utf8flaghasbeenaddedtothelistofsourceflags.

l Theformat-welffunctionhasbeenaddedtoTemplatefunctionsofsyslog-ngOSE.


Preface29

l The pass-unix-credentials() option has been added to Global options of syslog-ng OSE.

l Theuse-uniqid()optionhasbeenaddedtoGlobaloptionsofsyslog-ngOSE.

l TheUNIQIDmacrohasbeenaddedtoMacrosofsyslog-ngOSE.

l TheJSON-parsernowhandlesspecialcharactersinobjectnames.Fordetails,seeextract-prefix().

l Thesyslog-debuntoolusedtogeneratesyslog-ngOSEdebugbundleshasbeendocumented.Fordetails,seeThesyslog-ng-debunmanualpage.

l The--controloptionhasbeenaddedtotheThesyslog-ngmanualpagemanualpage.

l Version3.7andnewerautomaticallyincludestheplugin.conffilesfromthe/scl/*/directories,makingiteasiertouseanddistributeconfigurationblocks.

l The--enable-all-modulescompileroptionhasbeedaddedtoCompilingoptionsofsyslog-ngOSE.

l Thecreate-dirs()optionhasbeenaddedtounix-stream()andunix-dgram()destinationoptions.


l Generatingconfigurationblocksfromascripthasbeenaddedtothedocument.

l Example:Sendingalertwhenaclientdisappearshasbeenaddedtothedocument.

l Thetcp(),tcp6(),udp(),udp6()sourceanddestinationdrivershavebeendeprecated,asalloftheirfunctionalitycanbeachievedwiththenetwork()driver.Forhelponmigratingtothenetwork()driver,seeChangeanoldsourcedrivertothenetwork()driverandChangeanolddestinationdrivertothenetwork()driver.

l ThebeginningofTroubleshootingsyslog-nghasbeenextendedwithbasictroubleshootinginformation.

l Thedescriptionofthechain-hostnames()globaloptionhasbeenclarifiedandextended.Fordetails,seechain-hostnames().

l Othereditorialcorrections.

Version 3.5 - 3.6

Changes in product:


l riemann:MonitoringyourdatawithRiemannhasbeenaddedtothedocument.

l nodejs:ReceivingJSONmessages fromnodejsapplicationshasbeenadded tothedocument.


Preface30

l systemd-journal:Collectingmessagesfromthesystemd-journalsystemlogstoragehasbeenaddedtothedocument.

l systemd-syslog:Collectingsystemdmessagesusingasockethasbeenaddedtothedocument.

l use-rcptid()hasbeenaddedtothedocument.

l Settingmultiplemessagefieldstospecificvalueshasbeenaddedtothedocument.

l TheretriesandthrottleoptionsareavailablefortheSMTP,MongoDB,AMQP,andRedisdestinations.

l Thedescriptionofthemulti-line-modeoptionhasbeenupdated.

l UNIXcredentialsandothermetadatahasbeenaddedtothedocument.

l RUNIDhasbeenaddedtoMacrosofsyslog-ngOSE.

l Theextract-prefixoptionhasbeenaddedtoTheJSONparserTheJSONparser.

l Thegraphite-output,orandpaddingtemplatefunctionshavebeenaddedtoTemplatefunctionsofsyslog-ngOSE.

l PCREisnowarequireddependencyofsyslog-ngOSE,andbydefault,syslog-ngOSEusesPCRE-styleregularexpressions.Therefore,the--enable-pcrecompliationoptionhasbeenremoved.

l graphite:SendingmetricstoGraphitehasbeenaddedtothedocument.

l pseudofile()hasbeenaddedtothedocument.

l Thecustom-domain()andstats-lifetime()optionshavebeenaddedtoGlobaloptions.

l Theretry_sql_insertsoptionhasbeenrenamedtoretriestoincreaseconsistency.

l on-error()canbesetlocallyforMongoDBdestinationsaswell.Also,MongoDBdestinationssupporttheusernameandpasswordoptions,andconnectingtotheserverusingUNIXdomainsockets.Fordetails,seemongodb:StoringmessagesinaMongoDBdatabase.

l Howsyslog-ngOSEconnectstheMongoDBserverhasbeenaddedtothedocument.

l Severaltyposandsyntaxerrorsinexampleshavebeencorrected.

Acknowledgments

OneIdentitywouldliketoexpressitsgratitudetothesyslog-ngusersandthesyslog-ngcommunityfortheirinvaluablehelpandsupport.


Preface31

3

Introduction to syslog-ng

Thischapterintroducesthesyslog-ngOpenSourceEditionapplicationinanon-technicalmanner,discussinghowandwhyisituseful,andthebenefitsitofferstoanexistingITinfrastructure.

What syslog-ng is

Thesyslog-ngapplicationisaflexibleandhighlyscalablesystemloggingapplicationthatisidealforcreatingcentralizedandtrustedloggingsolutions.Amongothers,syslog-ngOSEallowsyouthefollowing.

Secure and reliable log transfer

Thesyslog-ngOSEapplicationenablesyoutosendthelogmessagesofyourhoststoremoteserversusingthelatestprotocolstandards.Youcancollectandstoreyourlogdatacentrallyondedicatedlogservers.TransferlogmessagesusingtheTCPprotocolensuresthatnomessagesarelost.

Disk-based message buffering

Tominimizetheriskoflosingimportantlogmessages,thesyslog-ngOSEapplicationcanstoremessagesonthelocalharddiskifthecentrallogserverorthenetworkconnectionbecomesunavailable.Thesyslog-ngapplicationautomaticallysendsthestoredmessagestotheserverwhentheconnectionisreestablished,inthesameorderthemessageswerereceived.Thediskbufferispersistentnomessagesarelostevenifsyslog-ngisrestarted.

Secure logging using TLS

Logmessagesmaycontainsensitiveinformationthatshouldnotbeaccessedbythirdparties.Therefore,syslog-ngOSEsupportstheTransportLayerSecurity(TLS)protocolto


Introduction to syslog-ng32

encryptthecommunication.TLSalsoallowsyoutoauthenticateyourclientsandthelogserverusingX.509certificates.

Flexible data extraction and processing

Mostlogmessagesareinherentlyunstructured,whichmakesthemdifficulttoprocess.Toovercomethisproblem,syslog-ngOSEcomeswithasetofbuilt-inparsers,whichyoucancombinetobuildverycomplexthings.

Filter and classify

Thesyslog-ngOSEapplicationcansorttheincominglogmessagesbasedontheircontentandvariousparameterslikethesourcehost,application,andpriority.Youcancreatedirectories,files,anddatabasetablesdynamicallyusingmacros.Complexfilteringusingregularexpressionsandbooleanoperatorsoffersalmostunlimitedflexibilitytoforwardonlytheimportantlogmessagestotheselecteddestinations.

Parse and rewrite

Thesyslog-ngOSEapplicationcansegmentlogmessagestonamedfieldsorcolumns,andalsomodifythevaluesofthesefields.YoucanprocessJSONmessages,key-valuepairs,andmore.

Togetthemostinformationoutofyourlogdata,syslog-ngOSEallowsyoutocorrelatelogmessagesandaggregatetheextractedinformationintoasinglemessage.Youcanalsouseexternalinformationtoenrichyourlogdata.

Big data clusters

Thelogdatathatyourorganizationhastoprocess,store,andreviewincreasesdaily,somanyorganizationsusebigdatasolutionsfortheirlogs.Toaccomodatethishugeamountofdata,syslog-ngOSEnativelysupportsstoringlogmessagesinHDFSfilesandElasticsearchclusters.

Message queue support

Largeorganizationsincreasinglyrelyonqueuinginfrastructuretotransfertheirdata.syslog-ngOSEsupportsApacheKafka,theAdvancedMessageQueuingProtocol(AMQP),andtheSimpleTextOrientedMessagingProtocol(STOMP).



SQL, NoSQL, and monitoring

Storing your log messages in a database allows you to easily search and query themessages and interoperate with log analyzing applications. The syslog-ng applicationsupports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, andSQLite.

syslog-ngOSEalsoallowsyoutoextracttheinformationyouneedfromyourlogdata,anddirectlysendittoyourGraphite,Redis,orRiemannmonitoringsystem.

Wide protocol and platform support

syslog protocol standards

syslog-ngnotonlysupportslegacyBSDsyslog(RFC3164)andtheenhancedRFC5424protocolsbutalsoJavaScriptObjectNotation(JSON)andjournaldmessageformats.

Heterogeneous environments

Thesyslog-ngOSEapplicationistheidealchoicetocollectlogsinmassivelyheterogeneousenvironmentsusingseveraldifferentoperatingsystemsandhardwareplatforms,includingLinux,Unix,BSD,SunSolaris,HP-UX,Tru64,andAIX.

IPv4 and IPv6 support

Thesyslog-ngapplicationcanoperateinbothIPv4andIPv6networkenvironments,andcanreceiveandsendmessagestobothtypesofnetworks.

What syslog-ng is not

Thesyslog-ngapplicationisnotloganalysissoftware.Itcanfilterlogmessagesandselectonlytheonesmatchingcertaincriteria.Itcanevenconvertthemessagesandrestructurethemtoapredefinedformat,orparsethemessagesandsegmentthemintodifferentfields.Butsyslog-ngcannotinterpretandanalyzethemeaningbehindthemessages,orrecognizepatternsintheoccurrenceofdifferentmessages.

Why is syslog-ng needed?

Logmessagescontaininformationabouttheeventshappeningonthehosts.Monitoringsystemeventsisessentialforsecurityandsystemhealthmonitoringreasons.



Theoriginalsyslogprotocolseparatesmessagesbasedonthepriorityofthemessageandthefacilitysendingthemessage.Thesetwoparametersaloneareofteninadequatetoconsistentlyclassifymessages,asmanyapplicationsmightusethesamefacility,andthefacilityitselfisnotevenincludedinthelogmessage.Tomakethingsworse,manylogmessagescontainunimportantinformation.Thesyslog-ngapplicationhelpsyoutoselectonlythereallyinterestingmessages,andforwardthemtoacentralserver.

Companypoliciesorotherregulationsoftenrequirelogmessagestobearchived.Storingtheimportantmessagesinacentrallocationgreatlysimplifiesthisprocess.

What is new in syslog-ng Open Source Edition 3.18?

Version3.18ofsyslog-ngOpenSourceEditionincludesthefollowingmainfeatures.

Batch support in the http() destination driver

Thehttp()destinationcannowsendabatchoflogmessagesinasingleHTTPrequest,greatlyimprovingtheperformance.Inaddition,thisfeaturealsoallowsyoutopostproperJSON-encodedarraysasPOSTpayloads,whichisrequiredbyseveralRESTAPIs.Fordetails,seeAdministrationGuide.

Write your own destination in Python

Extendingsyslog-ngOSEinPythonhasbeensupportedforseveralreleases,butsofarthisfeaturewasmostlyundocumented.Nowyoucanfindmoredetailsaboutthisfeaturein"python:writingcustomPythondestinations"intheAdministrationGuide.

Write your own message source in Python

Startingwithsyslog-ngOSEversion3.18,youcanwritecustommessagesourcesinPython.Bothserver-styleandfetcher-stylesourcesaresupported.Formoredetails,see"python:writingserver-stylePythonsources"intheAdministrationGuideand"python-fetcher:writingfetcher-stylePythonsources"intheAdministrationGuide.

Enhancements

l Whenhdfs-append-enabledissettotrue,syslog-ngOSEwillappendnewdatatotheendofanalreadyexistingHDFSfile.Notethatinthiscase,archivingisautomaticallydisabled,andsyslog-ngOSEwillignorethehdfs-archive-diroption.

l Thehdfsdestinationnowsupportsthetime-reap()option.

l Theurlencode()templatefunctionhasbeenrenamedtourl-encode().Also,thetelegram()destinationnowautomaticallyencodesthemessages.

l Newtemplatefunctionsareavailable:url-decode()andbase64-encode().Fordetails,



https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide//https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/destination-forward-send-and-store-log-messages/python-writing-custom-python-destinations/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-writing-server-style-python-sources/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-fetcher-writing-fetcher-style-python-sources/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-fetcher-writing-fetcher-style-python-sources/

see"Templatefunctionsofsyslog-ngOSE"intheAdministrationGuide.

l Thesyslog-ng-ctl configcommandcandisplaythecontentsoftheconfigurationfilethatsyslog-ngOSEiscurrentlyrunning.

l Therekeyoptionofvalue-pairs()nowsupportsanewtransformation:shift-levels.Itcutsdot-delimited"levels"inthename(includingtheinitialdot).Forexample,--shift-levels 2deletestheprefixuptotheseconddotinthenameofthekey:.iptables.SRCbecomesSRC


l Thevalue-pairs()optionnowhasanewscope:none.Thisscoperesetspreviouslyaddedscopes,makingitpossibletogetremoveautomaticallyaddedname-valuepairsfromthescope.


l Whenreceivingmessageswiththedefault-network-drivers()source,syslog-ngOSEnowautomaticallysetsthe${.app.name}name-valuepairtothenameoftheapplicationthatsentthelogmessage.

Fordetails,see"default-network-drivers:Receiveandparsecommonsyslogmessages"intheAdministrationGuide.

Deprecated features

Theelasticsearch()destinationhasbeendeprecated,becauseitsupportsonlyElasticSearchversion1.x,whichhasbeenEnd-of-LifesinceJanuary,2017.Usetheelasticsearch2()destinationinstead.

Who uses syslog-ng?

Thesyslog-ngapplicationisusedworldwidebycompaniesandinstitutionswhocollectandmanagethelogsofseveralhosts,andwanttostoretheminacentralized,organizedway.Usingsyslog-ngisparticularlyadvantageousfor:

l InternetServiceProviders

l Financialinstitutionsandcompaniesrequiringpolicycompliance

l Server,web,andapplicationhostingcompanies

l Datacenters

l Wideareanetwork(WAN)operators

l Serverfarmadministrators.

Supported platformsThesyslog-ngOpenSourceEditionapplicationishighlyportableandisknowntorunonawiderangeofhardwarearchitectures(x86,x86_64,SUNSparc,PowerPC32and64,



https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/template-and-rewrite-format-modify-and-manipulate-log-messages/customize-message-format-using-macros-and-templates/template-functions-of-syslog-ng-ose/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/the-concepts-of-syslog-ng/structuring-macros-metadata-and-other-value-pairs/value-pairs/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/the-concepts-of-syslog-ng/structuring-macros-metadata-and-other-value-pairs/value-pairs/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/default-network-drivers-receive-and-parse-common-syslog-messages/https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/default-network-drivers-receive-and-parse-common-syslog-messages/

Alpha)andoperatingsystems,includingLinux,BSD,Solaris,IBMAIX,HP-UX,MacOSX,Cygwin,Tru64,andothers.

l Thesourcecodeofsyslog-ngOpenSourceEditionisreleasedundertheGPLv2licenseandisavailableonGitHub.

l SeetheDownloadspageforbinarypackages.



https://github.com/balabit/syslog-nghttps://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx

4

The concepts of syslog-ng

Thischapterdiscussesthetechnicalconceptsofsyslog-ng.

The philosophy of syslog-ng

Typically,syslog-ngisusedtomanagelogmessagesandimplementcentralizedlogging,wheretheaimistocollectthelogmessagesofseveraldevicesonasingle,centrallogserver.Thedifferentdevicescalledsyslog-ngclientsallrunsyslog-ng,andcollectthelogmessagesfromthevariousapplications,files,andothersources.Theclientssendallimportantlogmessagestotheremotesyslog-ngserver,whichsortsandstoresthem.

Logging with syslog-ngThesyslog-ngapplicationreadsincomingmessagesandforwardsthemtotheselecteddestinations.Thesyslog-ngapplicationcanreceivemessagesfromfiles,remotehosts,andothersources.

Logmessagesentersyslog-nginoneofthedefinedsources,andaresenttooneormoredestinations.

Sourcesanddestinationsareindependentobjects,log pathsdefinewhatsyslog-ngdoeswithamessage,connectingthesourcestothedestinations.Alogpathconsistsofoneormoresourcesandoneormoredestinations:messagesarrivingfromasourcearesenttoeverydestinationlistedinthelogpath.Alogpathdefinedinsyslog-ngiscalledalog statement.

Optionally,logpathscanincludefilters.Filtersarerulesthatselectonlycertainmessages,forexample,selectingonlymessagessentbyaspecificapplication.Ifalogpathincludesfilters,syslog-ngsendsonlythemessagessatisfyingthefilterrulestothedestinationssetinthelogpath.

Otheroptionalelements thatcanappear in logstatementsareparsersand rewriting rules.Parserssegmentmessages intodifferent fields tohelpprocessing themessages,while rewrite rulesmodify themessagesbyadding, replacing,or removingpartsofthemessages.


The concepts of syslog-ng38

The route of a log message in syslog-ng

Purpose:

Thefollowingprocedureillustratestherouteofalogmessagefromitssourceonthesyslog-ngclienttoitsfinaldestinationonthecentralsyslog-ngserver.

Figure 1: The route of a log message

Steps:

1. Adeviceorapplicationsendsalogmessagetoasourceonthesyslog-ngclient.Forexample,anApachewebserverrunningonLinuxentersamessageintothe/var/log/apachefile.

2. Thesyslog-ngclientrunningonthewebserverreadsthemessagefromits/var/log/apachesource.

3. Thesyslog-ngclientprocessesthefirstlogstatementthatincludesthe/var/log/apachesource.

4. Thesyslog-ngclientperformsoptionaloperations(messagefiltering,parsing,andrewriting)onthemessage,forexample,itcomparesthemessagetothefiltersofthelogstatement(ifany).Ifthemessagecomplieswithallfilterrules,syslog-ngsendsthemessagetothedestinationssetinthelogstatement,forexample,totheremotesyslog-ngserver.



CAUTION:

Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

NOTE:

Thesyslog-ngclientsendsamessagetoallmatchingdestinationsbydefault.Asaresult,amessagemaybesenttoadestinationmorethanonce,ifthedestinationisusedinmultiplelogstatements.Topreventsuchsituations,usethefinalflaginthedestinationstatements.Fordetails,seeLogstatementflags.

5. Thesyslog-ngclientprocessesthenextlogstatementthatincludesthe/var/log/apachesource,repeatingSteps3-4.

6. Themessagesentby thesyslog-ngclientarrives fromasourceset in thesyslog-ngserver.

7. Thesyslog-ngserverreadsthemessagefromitssourceandprocessesthefirstlogstatementthatincludesthatsource.

8. Thesyslog-ngserverperformsoptionaloperations(messagefiltering,parsing,andrewriting)onthemessage,forexample,itcomparesthemessagetothefiltersofthelogstatement(ifany).Ifthemessagecomplieswithallfilterrules,syslog-ngsendsthemessagetothedestinationssetinthelogstatement.

CAUTION:

Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

9. Thesyslog-ngserverprocessesthenextlogstatement,repeatingSteps7-9.

NOTE:

Thesyslog-ngapplicationcanstopreadingmessagesfromitssourcesifthedestinationscannotprocessthesentmessages.Thisfeatureiscalledflow-controlandisdetailedinManagingincomingandoutgoingmessageswithflow-control.

Modes of operationThesyslog-ngOpenSourceEditionapplicationhasthreetypicaloperationscenarios:Client,Server,andRelay.

Client mode



Figure 2: Client-mode operation

Inclientmode,syslog-ngcollectsthelocallogsgeneratedbythehostandforwardsthemthroughanetworkconnectiontothecentralsyslog-ngserverortoarelay.Clientsoftenalsologthemessageslocallyintofiles.

Relay modeFigure 3: Relay-mode operation

Inrelaymode,syslog-ngreceiveslogsthroughthenetworkfromsyslog-ngclientsandforwardsthemtothecentralsyslog-ngserverusinganetworkconnection.Relaysalsologthemessagesfromtherelayhostintoalocalfile,orforwardthesemessagestothecentralsyslog-ngserver.

Server mode



Figure 4: Server-mode operation

Inservermode,syslog-ngactsasacentrallog-collectingserver.Itreceivesmessagesfromsyslog-ngclientsandrelaysoverthenetwork,andstoresthemlocallyinfiles,orpassesthemtootherapplications,forexampleloganalyzers.

Global objectsThesyslog-ngapplicationusesthefollowingobjects:

l Source driver:Acommunicationmethodusedtoreceivelogmessages.Forexample,syslog-ngcanreceivemessagesfromaremotehostviaTCP/IP,orreadthemessagesofalocalapplicationfromafile.Fordetailsonsourcedrivers,seesource:Read,receive,andcollectlogmessages.

l Source:Anamedcollectionofconfiguredsourcedrivers.

l Destination driver:Acommunicationmethodusedtosendlogmessages.Forexample,syslog-ngcansendmessagestoaremotehostviaTCP/IP,orwritethemessagesintoafileordatabase.Fordetailsondestinationdrivers,seedestination:Forward,send,andstorelogmessages.

l Destination:Anamedcollectionofconfigureddestinationdrivers.

l Filter:Anexpressiontoselectmessages.Forexample,asimplefiltercanselectthe



messagesreceivedfromaspecifichost.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.

l Macro:Anidentifierthatreferstoapartofthelogmessage.Forexample,the${HOST}macroreturnsthenameofthehostthatsentthemessage.Macrosareoftenusedintemplatesandfilenames.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.

l Parser:Parsersareobjectsthatparsetheincomingmessages,orpartsofamessage.Forexample,thecsv-parser()cansegmentmessagesintoseparatecolumnsatapredefinedseparatorcharacter(forexampleacomma).Everycolumnhasauniquenamethatcanbeusedasamacro.Fordetails,seeparser:Parseandsegmentstructuredmessagesanddb-parser:Processmessagecontentwithapatterndatabase(patterndb).

l Rewrite rule:Arulemodifiesapartofthemessage,forexample,replacesastring,orsetsafieldtoaspecifiedvalue.Fordetails,seeModifyingmessagesusingrewriterules.

l Log paths:Acombinationofsources,destinations,andotherobjectslikefilters,parsers,andrewriterules.Thesyslog-ngapplicationsendsmessagesarrivingfromthesourcesofthelogpathstothedefineddestinations,andperformsfiltering,parsing,andrewritingofthemessages.Logpathsarealsocalledlogstatements.Logstatementscanincludeother(embedded)logstatementsandjunctionstocreatecomplexlogpaths.Fordetails,seelog:Filterandroutelogmessagesusinglogpaths,flags,andfilters.

l Template:Atemplateisasetofmacrosthatcanbeusedtorestructurelogmessagesorautomaticallygeneratefilenames.Forexample,atemplatecanaddthehostnameandthedatetothebeginningofeverylogmessage.Fordetails,seeCustomizemessageformatusingmacrosandtemplates.

l Option:Optionssetglobalparametersofsyslog-ng,liketheparametersofnameresolutionandtimezonehandling.Fordetails,seeGlobaloptionsofsyslog-ngOSE.

Fordetailsontheaboveobjects,seeTheconfigurationsyntaxindetail.

Timezones and daylight savingThesyslog-ngapplicationreceivesthetimezoneanddaylightsavinginformationfromtheoperatingsystemitisinstalledon.Iftheoperatingsystemhandlesdaylightsavingcorrectly,sodoessyslog-ng.

Thesyslog-ngapplicationsupportsmessagesoriginatingfromdifferenttimezones.Theoriginalsyslogprotocol(RFC3164)doesnotincludetimezoneinformation,butsyslog-ngprovidesasolutionbyextendingthesyslogprotocoltoincludethetimezoneinthelogmessages.Thesyslog-ngapplicationalsoenablesadministratorstosupplytimezoneinformationforlegacydeviceswhichdonotsupporttheprotocolextension.



How syslog-ng OSE assigns timezone to the message

Whensyslog-ngOSEreceivesamessage,itassignstimezoneinformationtothemessageusingthefollowingalgorithm.

1. Thesenderapplication(forexamplethesyslog-ngclient)orhostspecifiesthetimezoneofthemessages.Iftheincomingmessageincludesatimezoneitisassociatedwiththemessage.Otherwise,thelocaltimezoneisassumed.

2. Specifythetime-zone()parameterforthesourcedriverthatreadsthemessage.Thistimezonewillbeassociatedwiththemessagesonlyifnotimezoneisspecifiedwithinthemessageitself.Eachsourcedefaultstothevalueoftherecv-time-zone()globaloption.Itisnotpossibletooverrideonlythetimezoneinformationoftheincomingmessage,butsettingthekeep-timestamp()optiontonoallowssyslog-ngOSEtoreplacethefulltimestamp(timezoneincluded)withthetimethemessagewasreceived.

NOTE:

Whenprocessingamessagethatdoesnotcontaintimezoneinformation,thesyslog-ngOSEapplicationwillusethetimezoneanddaylight-savingthatwaseffectivewhenthetimestampwasgenerated.Forexample,thecurrenttimeis2011-03-11(March11,2011)intheEU/Budapesttimezone.Whendaylight-savingisactive(summertime),theoffsetis+02:00.Whendaylight-savingisinactive(wintertime)thetimezoneoffsetis+01:00.Ifthetimestampofanincomingmessageis2011-01-01,thetimezoneassociatedwiththemessagewillbe+01:00,butthetimestampwillbeconverted,because2011-01-01meantwintertimewhendaylightsavingisnotactivebutthecurrenttimezoneis+02:00.

3. Specifythetimezoneinthedestinationdriverusingthetime-zone()parameter.Eachdestinationdrivermighthaveanassociatedtimezonevalue:syslog-ngconvertsmessagetimestampstothistimezonebeforesendingthemessagetoitsdestination(fileornetworksocket).Eachdestinationdefaultstothevalueofthesend-time-zone()globaloption.

NOTE:

Amessagecanbesenttomultipledestinationzones.Thesyslog-ngapplicationconvertsthetimezoneinformationproperlyforeveryindividualdestinationzone.

CAUTION:

If syslog-ng OSE sends the message is to the destination using the legacy-syslog protocol (RFC3164) which does not support timezone information in its timestamps, the timezone information cannot be encapsulated into the sent timestamp, so syslog-ng OSE will convert the hour:min values based on the explicitly specified timezone.

4. Ifthetimezoneisnotspecified,localtimezoneisused.



5. Whenmacroexpansionsareusedinthedestinationfilenames,thelocaltimezoneisused.(Also,ifthetimestampofthereceivedmessagedoesnotcontaintheyearofthemessage,syslog-ngOSEusesthelocalyear.)

A note on timezones and timestampsIf the clients run syslog-ng, then use the ISO timestamp, because it includestimezone information. Thatway you do not need to adjust the recv-time-zone()parameter of syslog-ng.

Ifyouwantsyslog-ngtooutputtimestampsinUnix(POSIX)timeformat,usetheS_UNIXTIMEandR_UNIXTIMEmacros.Youdonotneedtochangeanyofthetimezonerelatedparameters,becausethetimestampinformationofincomingmessagesisconvertedtoUnixtimeinternally,andUnixtimeisatimezone-independenttimerepresentation.(Actually,UnixtimemeasuresthenumberofsecondselapsedsincemidnightofCoordinatedUniversalTime(UTC)January1,1970,butdoesnotcountleapseconds.)

Product licensing

Startingwithversion3.2,thesyslog-ngOpenSourceEditionapplicationislicensedunderacombinedLGPL+GPLlicense.Thecoreofsyslog-ngOSEislicensedundertheGNULesserGeneralPublicLicenseVersion2.1license,whiletherestofthecodebaseislicensedundertheGNUGeneralPublicLicenseVersion2license.

NOTE:

Practically,thecodestoredunderthelibdirectoryofthesourcecodepackageisunderLGPL,therestisGPL.

FordetailsabouttheLGPLandGPLlicenses,seeGNULesserGeneralPublicLicenseandGNUGeneralPublicLicense,respectively.

High availability supportMultiplesyslog-ngserverscanberuninfail-overmode.Thesyslog-ngapplicationdoesnotincludeanyinternalsupportforthis,asclusteringsupportmustbeimplementedontheoperatingsystemlevel.AtoolthatcanbeusedtocreateUNIXclustersisHeartbeat(fordetails,seethispage).

The structure of a log messageThefollowingsectionsdescribethestructureoflogmessages.Currentlytherearetwostandardsyslogmessageformats:



http://www.linux-ha.org/wiki/Main_Page/

l TheoldstandarddescribedinRFC3164(alsocalledtheBSD-syslogorthelegacy-syslogprotocol):seeBSD-syslogorlegacy-syslogmessages

l ThenewstandarddescribedinRFC5424(alsocalledtheIETF-syslogprotocol):seeIETF-syslogmessages

l TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesbetweensyslog-ngnodes:seeEnterprise-widemessagemodel(EWMM)

l Howmessagesarerepresentedinsyslog-ngOSE:seeMessagerepresentationinsyslog-ngOSE.

BSD-syslog or legacy-syslog messagesThissectiondescribestheformatofasyslogmessage,accordingtothelegacy-syslogorBSD-syslogprotocol.Asyslogmessageconsistsofthefollowingparts:

l PRI

l HEADER

l MSG

Thetotalmessagecannotbelongerthan1024bytes.

Thefollowingisasamplesyslogmessage:

Feb 25 14:09:07 webserver syslogd: restart

Themessagecorrespondstothefollowingformat:

timestamp hostname application: message

Thedifferentpartsofthemessageareexplainedinthefollowingsections.

NOTE:

Thesyslog-ngapplicationsupportslongermessagesaswell.Fordetails,seethelog-msg-size()optioninGlobaloptions.However,itisnotrecommendedtoenablemessageslargerthanthepacketsizewhenusingUDPdestinations.

The PRI message part

ThePRIpartofthesyslogmessage(knownasPriorityvalue)representstheFacilityandSeverityofthemessage.Facilityrepresentsthepartofthesystemsendingthemessage,whileseveritymarksitsimportance.ThePriorityvalueiscalculatedbyfirstmultiplyingtheFacilitynumberby8andthenaddingthenumericalvalueoftheSeverity.Thepossiblefacilityandseverityvaluesarepresentedbelow.

NOTE:

Facilitycodesmayslightlyvarybetweendifferentplatforms.Thesyslog-ngapplicationacceptsfacilitycodesasnumericalvaluesaswell.



https://tools.ietf.org/search/rfc3164https://tools.ietf.org/search/rfc3164

Numerical Code Facility

0 kernelmessages

1 user-levelmessages

2 mailsystem

3 systemdaemons

4 security/authorizationmessages

5 messagesgeneratedinternallybysyslogd

6 lineprintersubsystem

7 networknewssubsystem

8 UUCPsubsystem

9 clockdaemon


11 FTPdaemon

12 NTPsubsystem

13 logaudit

14 logalert

15 clockdaemon

16-23 locallyusedfacilities(local0-local7)

Table 1: syslog Message Facilities

Thefollowingtableliststheseverityvalues.

Numerical Code Severity

0 Emergency:systemisunusable

1 Alert:actionmustbetakenimmediately

2 Critical:criticalconditions

3 Error:errorconditions

4 Warning:warningconditions

5 Notice:normalbutsignificantcondition

6 Informational:informationalmessages

7 Debug:debug-levelmessages

Table 2: syslog Message Severities



The HEADER message partTheHEADERpartcontainsatimestampandthehostname(withoutthedomainname)ortheIPaddressofthedevice.ThetimestampfieldisthelocaltimeintheMmm dd hh:mm:ssformat,where:

l MmmistheEnglishabbreviationofthemonth:Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec.

l ddisthedayofthemonthontwodigits.Ifthedayofthemonthislessthan10,thefirstdigitisreplacedwithaspace.(ForexampleAug 7.)

l hh:mm:ssisthelocaltime.Thehour(hh)isrepresentedina24-hourformat.Validentriesarebetween00and23,inclusive.Theminute(mm)andsecond(ss)entriesarebetween00and59inclusive.

NOTE:

Thesyslog-ngapplicationsupportsothertimestampformatsaswell,likeISO,orthePIXextendedformat.Fordetails,seethets-format()optioninGlobaloptions.

The MSG message part

TheMSGpartcontainsthenameoftheprogramorprocessthatgeneratedthemessage,andthetextofthemessageitself.TheMSGpartisusuallyinthefollowingformat:program[pid]: message text.

IETF-syslog messagesThissectiondescribestheformatofasyslogmessage,accordingtotheIETF-syslogprotocol.Asyslogmessageconsistsofthefollowingparts:

l HEADER(includesthePRIaswell)

l STRUCTURED-DATA

l MSG

Thefollowingisasamplesyslogmessage(source:https://tools.ietf.org/html/rfc5424):

1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

Themessagecorrespondstothefollowingformat:

VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

Inthisexample,theFacilityhasthevalueof4,severityis2,soPRIis34.TheVERSIONis1.Themessagewascreatedon11October2003at10:14:15pmUTC,3millisecondsintothenextsecond.Themessageoriginatedfromahostthatidentifiesitselfas"mymachine.example.com".TheAPP-NAMEis"su"andthePROCIDisunknown.The



https://tools.ietf.org/html/rfc5424https://tools.ietf.org/html/rfc5424

MSGIDis"ID47".TheMSGis"'suroot'failedforlonvick...",encodedinUTF-8.TheencodingisdefinedbytheBOM:

Thebyteordermark(BOM)isaUnicodecharacterusedtosignalthebyte-orderofthemessagetext.

ThereisnoSTRUCTURED-DATApresentinthemessage,thisisindicatedby"-"intheSTRUCTURED-DATAfield.TheMSGis"'suroot'failedforlonvick...".

TheHEADERpartofthemessagemustbeinplainASCIIformat,theparametervaluesoftheSTRUCTURED-DATApartmustbeinUTF-8,whiletheMSGpartshouldbeinUTF-8.Thedifferentpartsofthemessageareexplainedinthefollowingsections.

The PRI message part

ThePRIpartofthesyslogmessage(knownasPriorityvalue)representstheFacilityandSeverityofthemessage.Facilityrepresentsthepartofthesystemsendingthemessage,whileseveritymarksitsimportance.ThePriorityvalueiscalculatedbyfirstmultiplyingtheFacilitynumberby8andthenaddingthenumericalvalueoftheSeverity.Thepossiblefacilityandseverityvaluesarepresentedbelow.

NOTE:

Facilitycodesmayslightlyvarybetweendifferentplatforms.Thesyslog-ngapplicationacceptsfacilitycodesasnumericalvaluesaswell.


0 kernelmessages

1 user-levelmessages

2 mailsystem

3 systemdaemons


5 messagesgeneratedinternallybysyslogd

6 lineprintersubsystem

7 networknewssubsystem

8 UUCPsubsystem

9 clockdaemon


11 FTPdaemon

Table 3: syslog Message Facilities




12 NTPsubsystem

13 logaudit

14 logalert

15 clockdaemon

16-23 locallyusedfacilities(local0-local7)

Thefollowingtableliststheseverityvalues.

Numerical Code Severity

0 Emergency:systemisunusable

1 Alert:actionmustbetakenimmediately

2 Critical:criticalconditions

3 Error:errorconditions

4 Warning:warningconditions

5 Notice:normalbutsignificantcondition

6 Informational:informationalmessages

7 Debug:debug-levelmessages

Table 4: syslog Message Severities

The HEADER message partTheHEADERpartcontainsthefollowingelements:

l VERSION: Version number of the syslog protocol standard. Currently this canonly be 1.

l ISOTIMESTAMP:ThetimewhenthemessagewasgeneratedintheISO8601compatiblestandardtimestampformat(yyyy-mm-ddThh:mm:ss+-ZONE),forexample:2006-06-13T15:58:00.123+01:00.

l HOSTNAME:Themachinethatoriginallysentthemessage.

l APPLICATION:Thedeviceorapplicationthatgeneratedthemessage

l PID:TheprocessnameorprocessIDofthesyslogapplicationthatsentthemessage.It isnotnecessarilytheprocessIDoftheapplicationthatgeneratedthemessage.

l MESSAGEID:TheIDnumberofthemessage.



NOTE:

Thesyslog-ngapplicationsupportsothertimestampformatsaswell,likeISO,orthePIXextendedformat.ThetimestampusedintheIETF-syslogprotocolisderivedfromRFC3339,whichisbasedonISO8601.Fordetails,seethets-format()optioninGlobaloptions.

Thesyslog-ngOSEapplicationwilltruncatethefollowingfields:

l IfAPP-NAMEislongerthan48charactersitwillbetruncatedto48characters.

l IfPROC-IDislongerthan128charactersitwillbetruncatedto128characters.

l IfMSGIDislongerthan32charactersitwillbetruncatedto32characters.

l IfHOSTNAMEislongerthan255charactersitwillbetruncatedto255characters.

The STRUCTURED-DATA message part

TheSTRUCTURED-DATAmessagepartmaycontainmeta-informationaboutthesyslogmessage,orapplication-specificinformationsuchastrafficcountersorIPaddresses.STRUCTURED-DATAconsistsofdatablocksenclosedinbrackets([]).EveryblockincludestheIDoftheblock,andoneormorename=valuepairs.Thesyslog-ngapplicationautomaticallyparsestheSTRUCTURED-DATApartofsyslogmessages,whichcanbereferencedinmacros(fordetails,seeMacrosofsyslog-ngOSE).AnexampleSTRUCTURED-DATAblocklookslike:

[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]

The MSG message part

TheMSGpartcontainsthetextofthemessageitself.TheencodingofthetextmustbeUTF-8iftheBOM1characterispresentinthemessage.IfthemessagedoesnotcontaintheBOMcharacter,theencodingistreatedasunknown.UsuallymessagesarrivingfromlegacysourcesdonotincludetheBOMcharacter.CRLFcharacterswillnotberemovedfromthemessage.

Enterprise-wide message model (EWMM)ThefollowingsectiondescribesthestructureoflogmessagesusingtheEnterprise-widemessagemodelorEWMMmessageformat.

TheEnterprise-widemessagemodelorEWMMallowsyoutodeliverstructuredmessagesfromtheinitialreceivingsyslog-ngcomponentrightuptothecentrallogserver,through

1Thebyteordermark(BOM)isaUnicodecharacterusedtosignalthebyte-orderofthemessagetext.



anynumberofhops.Itdoesnotmatterifyouparsethemessagesontheclient,onarelay,oronthecentralserver,theirstructuredresultswillbeavailablewhereyoustorethemessages.Optionally,youcanalsoforwardtheoriginalrawmessageasthefirstsyslog-ngcomponentinyourinfrastructurehasreceivedit,whichisimportantifyouwanttoforwardamessageforexampletoaSIEMsystem.Tomakeuseoftheenterprise-widemessagemodel,youhavetousethesyslog-ng()destinationonthesenderside,andthedefault-network-drivers()sourceonthereceiverside.

ThefollowingisasamplelogmessageinEWMMformat.

1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - - {"MESSAGE":"Oct 11 22:14:15 mymachine su: 'su root' failed for username on /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}

Themessagehasthefollowingparts.

l TheheaderofthecomplieswiththeRFC5424messageformat,wherethePROGRAMfieldissetto@syslog-ng,andtheSDATAfieldisempty.

l TheMESSAGEpartisinJSONformat,andcontainstheactualmessage,aswellasanyname-valuepairsthatsyslog-ngOSEhasattachedtoorextractedfromthemessage.The${._TAGS}fieldcontainstheidentifierofthesyslog-ngsourcethathasoriginallyreceivedthemessageonthefirstsyslog-ngnode.

TosendamessageinEWMMformat,youcanusethesyslog-ng()destinationdriver,ortheformat-ewmm()templatefunction.

ToreceiveamessageinEWMMformat,youcanusethedefault-destination-drivers()sourcedriver,ortheewmm-parser()parser.

Message representation in syslog-ng OSEWhenthesyslog-ngOSEapplicationreceivesamessage,itautomaticallyparsesthemessage.Thesyslog-ngOSEapplicationcanautomaticallyparselogmessagesthatconformtotheRFC3164(BSDorlegacy-syslog)ortheRFC5424(IETF-syslog)messageformats.Ifsyslog-ngOSEcannotparseamessage,itresultsinanerror.

TIP:

Incaseyouneedtorelaymessagesthatcannotbeparsedwithoutanymodificationsorchanges,usetheflags(no-parse)optioninthesourcedefinition,andatemplatecontainingonlythe${MESSAGE}macrointhedestinationdefinition.

Toparsenon-syslogmessages,forexample,JSON,CSV,orothermessages,youcanusethebuilt-inparsersofsyslog-ngOSE.Fordetails,seeparser:Parseandsegmentstructuredmessages.

Aparsedsyslogmessagehasthefollowingparts.



l Timestamps

Twotimestampsareassociatedwitheverymessage:oneisthetimestampcontainedwithinthemessage(thatis,whenthesendersentthemessage),theotheristhetimewhensyslog-ngOSEhasactuallyreceivedthemessage.

l Severity

Theseverityofthemessage.

l Facility

Thefacilitythatsentthemessage.

l Tags

Customtextlabelsaddedtothemessagethataremainlyusedforfiltering.Noneofthecurrentmessagetransportprotocolsaddstagstothelogmessages.Tagscanbeaddedtothelogmessageonlywithinsyslog-ngOSE.Thesyslog-ngOSEapplicationautomaticallyaddstheidofthesourceasatagtotheincomingmessages.Othertagscanbeaddedtothemessagebythepatterndatabase,orusingthetags()optionofthesource.

l IP address of the sender

TheIPaddressofthehostthatsentthemessage.NotethattheIPaddressofthesenderisahardmacroandcannotbemodifiedwithinsyslog-ngOSEbuttheassociatedhostnamecanbemodified,forexample,usingrewriterules.

l Hard macrosHardmacroscontaindatathatisdirectlyderivedfromthelogmessage,forexample,the${MONTH}macroderivesitsvaluefromthetimestamp.Themostimportantconsiderationwithhardmacrosisthattheyareread-only,meaningtheycannotbemodifiedusingrewriterulesorothermeans.

l Soft macrosSoftmacros(sometimesalsocalledname-valuepairs)areeitherbuilt-inmacrosautomaticallygeneratedfromthelogmessage(forexample,${HOST}),orcustomuser-createdmacrosgeneratedbyusingthesyslog-ngpatterndatabaseoraCSV-parser.TheSDATAfieldsofRFC5424-formattedlogmessagesbecomesoftmacrosaswell.Incontrastwithhardmacros,softmacrosarewritableandcanbemodifiedwithinsyslog-ngOSE,forexample,usingrewriterules.



NOTE:

Itisalsopossibletosetthevalueofbuilt-insoftmacrosusingparsers,forexample,tosetthe${HOST}macrofromthemessageusingacolumnofaCSV-parser.

Thedataextractedfromthelogmessagesusingnamedpatternparsersinthepatterndatabasearealsosoftmacros.

TIP:

Forthelistofhardandsoftmacros,seeHardvs.softmacros.

Message size and encodingInternally,syslog-ngOSErepresentseverymessageasUTF-8.Themaximallengthofthelogmessagesislimitedbythelog-msg-size()option:ifamessageislongerthanthisvalue,syslog-ngOSEtruncatesthemessageatthelocationitreachesthelog-msg-size()value,anddiscardstherestofthemessage.

Whenencodingissetinasource(usingtheencoding()option)andthemessageislonger(inbytes)thanlog-msg-size()inUTF-8representation,syslog-ngOSEsplitsthemessageatanundefinedlocation(becausetheconversionbetweendifferentencodingsisnottrivial).

Structuring macros, metadata, and other value-pairs

Available in syslog-ng OSE 3.3 and later.

Thesyslog-ngOSEapplicationallowsyoutoselectandconstructname-valuepairsfromanyinformationalreadyavailableaboutthelogmessage,orextractedfromthemessageitself.Youcandirectlyusethisstructuredinformation,forexample,inthefollowingplaces:

l amqp()destination

l format-welf()templatefunction

l mongodb()destination

l stomp()destination

l orinotherdestinationsusingtheformat-json()templatefunction.

Whenusingvalue-pairs,therearethreewaystospecifywhichinformation(thatis,macrosorothername-valuepairs)toincludeintheselection.

l Selectgroupsofmacrosusingthescope()parameter,andoptionallyremovecertainmacrosfromthegroupusingtheexclude()parameter.



l Listspecificmacrostoincludeusingthekey()parameter.

l Definenewname-valuepairstoincludeusingthepair()parameter.

Theseparametersaredetailedinvalue-pairs().

Specifying data types in value-pairsBydefault,syslog-ngOSEhandleseverydataasstrings.However,certaindestinationsanddataformats(forexample,SQL,MongoDB,JSON,AMQP)supportothertypesofdataaswell,forexample,numbersordates.Thesyslog-ngOSEapplicationallowsyoutospecifythedatatypeintemplates(thisisalsocalledtype-hinting).Ifthedestinationdriversupportsdatatypes,itconvertstheincomingdatatothespecifieddatatype.Forexample,thisallowsyoutostoreintegernumbersasnumbersinMongoDB,insteadofstrings.

CAUTION:

Hazard of data loss! If syslog-ng OSE cannot convert the data into the specified type, an error occurs, and syslog-ng OSE drops the message by default. To change how syslog-ng OSE handles data-conversion errors, see on-error().

Tousetype-hinting,enclosethemacroortemplatecontainingthedatawiththetype:(""),forexample:int("$PID").

Currently the mongodb() destination and the format-json template function supportsdata types.

Example: Using type-hintingThefollowingexamplestorestheMESSAGE,PID,DATE,andPROGRAMfieldsofalogmessageinaMongoDBdatabase.TheDATEandPIDpartsarestoredasnumbersinsteadofstrings.

mongodb( value-pairs(pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM")) pair("message", "$MESSAGE")) ) );

ThefollowingexampleformatsthesamefieldsintoJSON.

$(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$M

Documents

Administration Guide - support-public.cfm.quest.com · Introduction to syslog-ng 32 What syslog-ng is 32 ... The syslog-ng OSE quick-start guide 77 Configuring syslog-ng on client