SharePoint Audit Logging with HP ArcSight and LOGbinder SP™

LOGbinder SP™ is the only recognized solution for providing reliable audit information about the security events of SharePoint via HP ArcSight.

Written by Randy Franklin Smith

Chief Technology Officer LOGbinder™

LOGbinder is a division of Monterey Technology Group, Inc.

© 2012 Monterey Technology Group, Inc. ALL RIGHTS RESERVED.

This document contains proprietary information protected by copyright. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose without the written permission of Monterey Technology Group, Inc. (“Monterey”). The information in this document is provided in connection with Monterey products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Monterey products. EXCEPT AS SET FORTH IN MONTEREY'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, MONTEREY ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL MONTEREY BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF MONTEREY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Monterey makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Monterey does not make any commitment to update the information contained in this document.

If you have any questions regarding your potential use of this material, contact:

www.logbinder.com

Email: support@logbinder.com

Contents Background ................................................................................................................................................... 4

SharePoint logs: A Primer for HP ArcSight Integration ................................................................................. 4

Diagnostic Logs (a.k.a. Unified Logging Service) ....................................................................................... 4

IIS Logs ...................................................................................................................................................... 6

Usage Logs ................................................................................................................................................ 6

Audit Logs .................................................................................................................................................. 7

SharePoint Audit Log Storage ............................................................................................................... 8

SharePoint Audit Reports ...................................................................................................................... 8

SharePoint Audit Log Alert Recommendations .................................................................................... 9

Challenges of SharePoint Auditing .............................................................................................................. 10

Bridging the Gap with LOGbinder SP™ ....................................................................................................... 11

Example screen shots of the LOGbinder SP™ content package for HP ArcSight ........................................ 13

Comprehensive Application Security Intelligence ...................................................................................... 14

About LOGbinder™ ..................................................................................................................................... 15

Page 4 of 15

Background SharePoint is the dominant platform for documents and other unstructured data. Companies’ most scrutinized business processes and documentation are often facilitated by SharePoint, since it hosts unstructured data for distributed collaboration and acts as a repository of this data. High-level analysis, reporting and decision-making data stored in these documents is typically of greater value to intruders than the voluminous, cryptic and raw data found in corporate databases. Combine that risk with the fact that unstructured data is growing at an enormous rate and SharePoint becomes a key area of concern for management and information security staff.

It is critical that businesses obtain visibility into the activity associated with such sensitive information. Enterprise risk management demands it. Microsoft SharePoint content databases abound with confidential data and regulation-sensitive processes. Advanced persistent threats frequently target documents for exfiltration. At risk to the company are its compliance standards, reputation and the market penalty associated with breach of sensitive or confidential data.

Without security intelligence from the SharePoint platform itself, organizations can neither detect nor address these threats. Moreover, to see risks across the entire technology stack, application security intelligence should correlate to system security intelligence. Organizations must connect SharePoint audit logs in a meaningful way to a Security Information and Event Management (SIEM) system such as HP ArcSight.

A number of daunting technical challenges make such correlation difficult, and in fact, application event logs are missing from most SIEM implementations. LOGbinder™ bridges this gap by providing meaningful application security intelligence to any SIEM.

This technical brief provides an overview of SharePoint event logs with a spotlight on the gaps between them and SIEM products, specifically HP ArcSight. It then describes how LOGbinder SP™ bridges this gap to save you money and reduce your security risks.

SharePoint logs: A Primer for HP ArcSight Integration SharePoint generates several different logs ranging from true audit logs to diagnostic trace logs and usage analysis. This brief will identify what (if any) security intelligence can be learned from each log. It will then explain which logs are readily available to SIEMs, and which logs are not readily available.

Diagnostic Logs (a.k.a. Unified Logging Service) SharePoint outputs data that can be useful for troubleshooting to 2 different diagnostic logging facilities: the Windows Application event log and “Trace Logs” in text format which are stored by default in \Program Files\Common Files\Microsoft Shared\Web Server Extensions\[version]\LOGS\. The files have a “.log”

Technical challenges create gaps between application audit logs, such as those in SharePoint, and

SIEM solutions

Security Value None Output Format 2 formats: Windows Event Log,

Text Log Accessible to ArcSight via

Custom configuration of • Windows Unified

Connector • Flex Connector

Page 5 of 15

extension. Trace logs are also called ULS (Unified Logging Service) logs. Data in these logs range from actual errors to extremely verbose trace information for tracking the internal behavior and route that operations take through SharePoint code and plug-ins.

These logs do not contain precise duplication of information. The criteria for determining whether diagnostic data is logged to either log is not clearly known, but typically, higher-level and more critical diagnostics are sent to the Application event log while the trace log receives lower level, more verbose data. Administrators can fine-tune logging levels for both logs with the PowerShell command Set-SPLogLevel or in Central Administration.

SharePoint diagnostic logs are accessible to SIEM solutions like HP ArcSight via normal log collection of Windows event logs and text files but there are two important things to note about them:

• They offer no security value • They contain only the activity occurring on the local server. There are usually multiple servers

per SharePoint farm

Page 6 of 15

IIS Logs SharePoint is a web-based application hosted by Internet Information Server (IIS), the Windows version of a web server. IIS can record each HTTP transaction to a variety of log destinations for any Windows-based web application, including SharePoint. IIS logs include fields such as URL, query strings, IP address, and HTTP status data.

While IIS logs are the only SharePoint-related format for which a SmartConnector exists in HP ArcSight, the logs provide merely a protocol view of HTTP traffic between web clients and IIS. Someone with deep knowledge of both SharePoint and HTTP transactions can monitor aspects of SharePoint activity such as HTTP traffic regarding documents, but the events are simply the literal web page requests from the browser or other web client. IIS logs do not provide useful insight relative to IT security.

Usage Logs

Usage logs are part of the SharePoint Usage and Health Service that allows administrators to analyze usage of the SharePoint environment in the broadest of terms, and to troubleshoot issues.

The content of usage logs is the same as IIS logs just discussed – but with some SharePoint-specific context data appended, such as Application ID, Site ID and Web ID.

Each SharePoint server in the farm posts this usage data to the local file system, which in turn is periodically collected into a single SQL database specified in Central

Security Value Some Output Format SQL Database Accessible to ArcSight via

Possibly through custom configuration of Flex Connector

PartitionId 14 RowId 26FC53DB-D513-E211-9137-000C2958CF01 LogTime 57:13.9 MachineName SP FarmId 1AED515D-366F-4BA7-8E38-C58AD2144DEE SiteSubscriptionId 00000000-0000-0000-0000-000000000000 UserLogin MTG\rsmith CorrelationId B0470EF9-2F12-42BC-A23B-0169881C84C8 WebApplicationId 0F17234E-2953-4930-8F09-CC6EB8ED316A ServerUrl http://sp SiteId ACC1AC24-FAE6-4D6C-8D01-AF4BFD2E700F SiteUrl / WebId 557B334A-73DD-4229-82D3-ADB79BC45668 WebUrl contentcontrol DocumentPath /documents/forms/controlled document/docsethomepage.aspx ContentTypeId QueryString ?ID=263&FolderCTID=0x0120D52000216A22B1B083494FB55A61ACECFB92D9005 BytesConsumed 4479 HttpStatus 200 SessionId ReferrerUrl http://sp/ contentcontrol /Documents/Forms/AllItems.aspx ReferrerQueryString Browser IE7 User Agent Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; User Address 10.42.1.23 Request Count 5 Query Count 12 QueryDurationSum 25 ServiceCallCount 0 ServiceCallDurationSum 0 OperationCount 1 Duration 95 RequestType GET Title SharePoint Audit Logging with HP ArcSight and LOGbinder EX RowCreatedTime 00:02.5

Example Usage Log Entry: A document retrieved with IE7

Security Value Some Output Format W3C Extended Log

File Format Accessible to ArcSight via

SmartConnector™ for Microsoft IIS Multiple Server FileConnector

IIS logs are the only SharePoint related log for which a pre-built

SmartConnector exists... but they aren’t all that helpful in IT security

reviews.

Page 7 of 15

Administration or in PowerShell with Set-SPUsage commands.

Usage logs provide information about site traffic, response times and other statistics, as well as some security data. As you can see in the example event, usage logs can reveal security event information about document access (how and by whom).

The HP ArcSight FlexConnector parses usage logs from the SQL Server database. But bear in mind that usage logs track every HTTP request sent by the client during every session. The vast majority of these requests, like repetitive page views, are uninteresting.

Moreover, high-value security events such as changes to audit policy, permission or group membership changes and other privileged user operations hide from Usage log event tracking except in the sense of the very literal HTTP requests involved in those changes. Even if it were possible to identify the relevant HTTP requests for such security operations – out of thousands of single log entries -- the user name of the account receiving permissions and the group name (in the case of group membership changes) is absent; such data does not travel at the HTTP level. In this context, such operations resemble any other web page event where a user is clicking on column X of row Y. In addition to user and group name, other key elements missing from the Usage log, too.

Audit Logs Beginning with SharePoint 2007, Microsoft added a true audit capability to SharePoint that is flexible and designed to satisfy compliance requirements for audit logging. The problem is that the SharePoint audit log is inaccessible through any normal log

collection means, including HP ArcSight. LOGbinder SP™ exists to solve that problem.

SharePoint audit logs track end-user operations at the user account level. The event audit log includes the viewing of documents, document libraries, lists and list items. SharePoint can audit document check-in and check-out as well as updates to list items and documents.

Security Value High Output Format n/a Accessible to ArcSight via

Only with LOGbinder SP™

To enable auditing you must be a site collection administrator. From the Site Settings page, look under Site Collection Administration, as pictured here, for “Site collection audit settings”.

Page 8 of 15

Organizations can also obtain important administration-level site information such as permission changes, role updates, group membership changes and changes to the audit policy itself.

As shown in the screen shot of the Administrator’s Site collection audit settings page, auditing is controlled primarily at the root of each site collection, but you can also customize the audit function to include specific content types stored in libraries and lists. Once enabled, SharePoint will begin logging events to an internal audit log stored in the same SQL database as the content (documents and lists) of the SharePoint site.

SharePoint Audit Log Storage The fact that the audit log resides in the content database raises both resource and security issues. Audit logs can be extremely voluminous. If left unchecked they will artificially inflate the SharePoint content database, consume costly amounts of SQL Server storage and slow the system down dramatically.

As with all audit logs, it is critical for security and compliance requirements to get the audit log out of SharePoint and into the organization’s SIEM or log management solution such as HP ArcSight. Widely-accepted security best-practice dictates that we remove audit logs as quickly as possible (if not in real-time) to a separate and secure log repository. This process protects the integrity of the audit logs from intruders and nefarious administrators. SharePoint provides an automatic trimming feature to purge audit entries when they reach a certain age and can automatically export those events to the Excel-based reports found under the “View Auditing Reports” page in Site Collection Administration. However, the audit entries remain in the content database where they continue to occupy expensive SQL Server storage, inefficiently consume valuable resources and are vulnerable to tampering.

SharePoint Audit Reports The SharePoint audit log is accessible through the SharePoint API, PowerShell or the “View Auditing Reports” page in Site Collection Administration.

The auditing reports in Site Collection Administration are the only (somewhat) practical option for administrators who do not have access to a robust SIEM platform. These reports allow you to select a category of activity, date range and specific user or object (e.g. list, library) for further filtering. The report is generated as an Excel spreadsheet and stored in a specified Document Library. Users with appropriate access control retrieve the report by accessing that Library.

As you can see from the example Security Settings report the available audit trail is comprehensive. In this case it lists access control related events such as changes to groups, permissions and roles. But there is a plethora of unresolved ID codes, GUIDs and bit masks. Both the report and the other 2 log access methods listed above produce the raw data records that require additional processing before they are ready for human consumption (and log management). This is another gap that LOGbinder SP™ overcomes.

The SharePoint audit log is inaccessible through any normal

log-collection means.

Page 9 of 15

SharePoint Audit Log Alert Recommendations SharePoint has no built-in alerting capability for audit events. We believe this is more appropriately a function of SIEM/log management and recommend alerting on security changes, especially:

• Site collection administrator added • Audit policy changed • Audit entries deleted

Other candidates for alerting on high-security sites include:

• Audit policy changed on an individual object • Permission, role and group changes

Connecting SharePoint logs to HP ArcSight

Security Value Effort Required Connector

Diagnostic logs None Medium Custom configured Flex Connector

IIS Logs Low Low IIS SmartConnector Usage Logs Low High Custom configured Flex

Connector Audit Logs High Easy LOGbinder SP

Example SharePoint Audit Log Report

Page 10 of 15

Challenges of SharePoint Auditing As the previous section demonstrated, out of the box SharePoint provides a secure and purpose-built audit capability that is essentially bound to the SharePoint application. There is no built-in way to transfer this critical application security intelligence to HP ArcSight (and other SIEM platforms) in order to correlate SharePoint events with other security intelligence from across your network. The lack of automatic integration also keeps the information from benefitting from the protection of the secure archive of HP ArcSight. To recap, there are 6 key problems, or gaps:

SharePoint Audit Problems

1 SharePoint's audit log does not provide the names of users or objects.

Knowing it was “somebody” isn’t good enough. The SharePoint audit log fails to translate record IDs. You have no idea what object or user to which a many events refers. Click here for an example of an audit event from SharePoint before being processed by LOGbinder SP.

2

SharePoint's audit log is buried in SharePoint's SQL Server content database.

The value of SharePoint’s audit log is comprised. To ensure the integrity of audit logs, log files must be moved from the system where they are generated to a separate and secure archive. However, in SharePoint the audit log isn't really a log – it’s scattered throughout various tables within the SharePoint database. In addition, this storage location makes the audit log inaccessible to most log management solutions.

3

SharePoint's audit log has no reporting.

You can’t manage what you can’t see. In Windows SharePoint Services the log is totally inaccessible, and in Office SharePoint Services it's exposed through a few rudimentary, impractical reports in Excel.

4

Windows SharePoint Services provides no interface to enable auditing.

The audit function is there, but without custom programming there's no way to turn it on, much less access the logs.

5

SharePoint's built-in audit log trimming feature can delete audit events before they are exported.

Some editions of SharePoint provide automatic log trimming of old events, but there is no way to ensure events have been archived before it “trims” the log. Auditors and IT admins hate that for good reason. Think of all the events that could go unnoticed!

6

Missing: Audit Policy Management

Many organizations have thousands of site collections per farm. But administrators have no way to enforce consistent audit policy across all site collections. In a SharePoint farm, each site collection has its own audit policy. When creating a new site collection Administrators must remember to access the Site Collection's audit settings page and enable auditing or the site will be unmonitored. This is especially troublesome for farms with self-service site collection enabled because new sites can be created directly by users without Administrator involvement!

LOGbinder SP™ dramatically solves all 6 of these problems by serving as the conduit between SharePoint and your SIEM.

Page 11 of 15

Bridging the Gap with LOGbinder SP™ LOGbinder SP™ bridges the gap between the SharePoint audit log and HP ArcSight (as well as other SIEMs) by translating the cryptic data in raw SharePoint audit entries.

For each event, LOGbinder SP™ resolves the user and object IDs and other cryptic codes.

LOGbinder SP turns a raw SharePoint audit event like this one into a fully translated and easy to understand message like the one shown below it.

Then LOGbinder SP™ sends the (clarified) event to HP ArcSight via CEF (Common Event Format) over Syslog1. HP ArcSight can then supply you with needed archival, alerting and reports.

LOGbinder SP™ includes pre-defined reports, active channels, filters and rules2 for ArcSight ESM.

LOGbinder SP™ is a small, efficient Windows service installed on a non-production member of the farm or on any one of the existing servers in your SharePoint farm. It monitors the internal SharePoint audit log without making any changes to your SharePoint installation. Our application conforms to current security protocols and in no way jeopardizes your hardened network security.

1 Additional output formats such as the Windows event log and generic syslog are available to support other SIEMs 2 Similar integration content is available for other SIEM Synergy Partners. See http://www.logbinder.com/products/logbindersp/resources/thirdparty.aspx and http://www.logbinder.com/products/logbindersp/resources/reports.aspx

Page 12 of 15

LOGbinder SP™ Bridges Big Gaps

SharePoint Audit Log Gap Bridged by LOGbinder SP™ and HP ArcSight

1

SharePoint's audit log does not provide the names of users or objects.

LOGbinder SP™ resolves the user and object IDs and other cryptic codes, producing an easy-to-understand, plain-English translation

2

SharePoint's audit log is buried in SharePoint's SQL Server content database.

LOGbinder SP™ sends SharePoint audit events where they belong, to HP ArcSight, in Common Event Format so that no flex connector or parsing rules need be built.

3

SharePoint's audit log has no reporting.

LOGbinder SP™ pre-built content package for ArcSight includes:

• Active channels • Dashboards • Rules • Reports • Filters

4

Windows SharePoint Services provides no interface for enabling auditing at all.

LOGbinder SP™ provides a GUI-based list of all site collections and their audit policy.

5

SharePoint's built-in audit log trimming feature can delete audit events before they are exported.

LOGbinder SP™ purges events from the SharePoint audit file only after the events have been sent to HP ArcSight. LOGbinder SP™ detects unauthorized purging by administrators and generates appropriate alerts.

6

No way to manage audit policy LOGbinder SP™ allows you to specify

• Which site collections LOGbinder SP™ will process

• The audit policy applied to each site • Whether to purge old audit events • A farm-wide audit policy to be applied to all

sites by default – including new site collections

Page 13 of 15

Example screen shots of the LOGbinder SP™ content package for HP ArcSight

Page 14 of 15

Comprehensive Application Security Intelligence With LOGbinder SP™ bridging the gap between your SharePoint and HP ArcSight deployments, you can achieve comprehensive application security intelligence for SharePoint 2007, 2010 and 2013.

Furthermore, you can correlate that intelligence within ArcSight to the rest of your organization’s audit trails and system activity, for a superior, enhanced platform of detecting and responding to threats and suspicious events.

For more information on LOGbinder SP™ or other LOGbinder™ products (including Microsoft SQL Server and Microsoft Exchange Server), visit www.logbinder.com.

Low Impact

• Only one LOGbinder SP installation required per farm – regardless how many servers comprise the farm

• LOGbinder SP does not need to be installed on a production SharePoint Server

• LOGbinder SP does not install as a plug-in or make any customizations to your SharePoint environment

• LOGbinder SP uses only Microsoft supported APIs for accessing SharePoint

• LOGbinder SP does not connect to or directly access SQL Server

Save Money. Reduce Risk.

• Reduce time spent in investigating and monitoring audit trails

• Reduce overhead associated with maintaining secure content

• Manage reputational, compliance and financial risk with best-practice application audit procedures.

Core Functionality

• Safeguard SharePoint documents and records

• Compliance • Accountability over

privileged users • Application security

intelligence in your SIEM

SharePoint

HP ArcSight

Cryptic raw data

Easy-to-understand audit messages in CEF format

Page 15 of 15

About the Author

Randy Franklin Smith is an internationally recognized expert on the security and control of Windows and Active Directory. Randy publishes UltimateWindowsSecurity.com and wrote The Windows Server 2008 Security Log Revealed, the only book devoted to the Windows security log. Randy is the creator of LOGbinder™ software, which makes cryptic application logs understandable and available to log-management and SIEM solutions. Randy is a Microsoft Security Most Valuable Professional (MVP).

About LOGbinder™

LOGbinder software agents bridge the gap between application audit logs and SIEM solutions. LOGbinder is a division of Monterey Technology Group, Inc.