36
November 2013 Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide v1.4.1

Oracle_Enterprise_Manager_Plugin-in_for_Oracle_ZFS_Storage_Appliances_InstallGuide.pdf

Embed Size (px)

Citation preview

  • November 2013

    Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage ApplianceInstallation Guidev1.4.1

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance, Installation Guide v1.4.1

    Table of ContentsIntroduction.............................................................................................................................. 1About the Oracle ZFS Storage Appliance Plug-in................................................................. 2Software Requirements........................................................................................................... 2Prerequisites for Monitoring Oracle ZFS Storage Appliances.............................................. 3Configuring the Oracle ZFS Storage Appliance.................................................................... 3Configuring for Clustered Appliances.................................................................................... 4Configuring Database-to-Appliance Mappings ..................................................................... 4Plug-in Management in the Oracle Enterprise Manager Cloud Control 12c Environment..5Deploying the Plug-in............................................................................................................. 5Adding Instances for Monitoring............................................................................................ 7Adding Clustered Systems for Monitoring............................................................................. 8Viewing Metrics and Reports................................................................................................. 8Upgrading the Plug-in........................................................................................................... 10Un-deploying the Plug-in...................................................................................................... 10Metric and Collection Information....................................................................................... 11Metric Collections on Datasets............................................................................................. 17Metric Collections on Pools, Projects and Shares............................................................... 18Reports................................................................................................................................... 18Default Threshold Values..................................................................................................... 20Stateless Alerts...................................................................................................................... 24Oracle ZFS Storage Appliances Alerts................................................................................. 25Communicating Performance Issues to the Appliance Administrator.................................25Troubleshooting Monitoring Deployments........................................................................... 26Metric Collection Errors....................................................................................................... 29Oracle ZFS Storage Appliance Audit Logs......................................................................... 31Host Agent Logs .................................................................................................................. 32Additional Resources............................................................................................................ 33

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide v1.4.1

    Introduction

    The Oracle ZFS Storage Appliance Plug-in extends Oracle Enterprise Manager Cloud Control to add support for monitoring Oracle ZFS Storage Appliances. This guide provides an overview of the plug-in and how to install and deploy it for monitoring Oracle ZFS Storage Appliances.

    This guide is written for the Oracle Enterprise Manager Cloud Control administrator. As Cloud Control administrator, you will communicate closely with the Oracle ZFS Storage Appliance storage administrator to discuss performance and analysis of database deployments on the Oracle ZFS Storage Appliance.

    If you need additional information about the operation of plug-ins for the Oracle Enterprise Manager Cloud Control product, see the Oracle Enterprise Manager Concepts Guide. You can download the document, as well as other related documentation, from the Oracle Web site (see the section at the end of this document).

    Important: Oracle Enterprise Manager Cloud Control was previously called Oracle Enterprise Manager Grid Control (versions 10g and 11g). These older versions are no longer being supported, but users familiar with them should note that references to grid control are now replaced with cloud control. These legacy versions' documentation still reflects the old terminology.

    Also note that the Sun ZFS Storage Appliance is now called the Oracle ZFS Storage Appliance, and similarly to the Oracle Enterprise Manager software, legacy Appliance Kit software versions and their documentation may still reflect the former name. References to Sun ZFS Storage Appliance, Sun ZFS Storage 7000, and Sun ZFS Backup Appliance all refer to the same family of Oracle ZFS Storage Appliances.

    1

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    About the Oracle ZFS Storage Appliance Plug-in

    The Oracle ZFS Storage Appliance plug-in for Oracle Enterprise Manager Cloud Control provides first-class monitoring for all Oracle ZFS Storage Appliance models. The plug-in provides the following primary features:

    Gathers and presents storage system, configuration, and performance information for Oracle ZFS Storage Appliances

    Raises alerts for pre-selected configuration and monitoring data

    Provides out-of-the-box reports that complement Oracle ZFS Storage Analytics reports

    Ties together Oracle databases deployed on NFS shares from Oracle ZFS Storage Appliances

    Supports monitoring by remote agents in the Cloud Control environment

    When you configure the Oracle ZFS Storage Appliance to be monitored by Oracle Enterprise Manager Cloud Control, an analytics worksheet and datasets are enabled. The worksheet and datasets help bridge a Cloud Control administrator's view of the appliance to the storage administrator's more detailed view. This enables both administrators to be fully conversant in discussing performance or analysis of database deployments on the appliance.

    Software Requirements

    Deployment of the Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance requires that the following software versions already be installed:

    Oracle Enterprise Manager Cloud Control 12c Release 3 (12.1.0.3)

    Oracle ZFS Storage Appliance 2010 Q3.4.4 software or later

    Note: Additional software requirements, such as the correct Java version in Cloud Control, are met through the required applications listed, provided the correct versions are installed. The plug-in can be installed on any operating system in which Enterprise Manager Cloud Control is running.

    2

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    Prerequisites for Monitoring Oracle ZFS Storage Appliances

    Before you can deploy and monitor Oracle ZFS Storage Appliances, you must:

    Install the Oracle Enterprise Manager Cloud Control environment. Be sure to pay attention to security sections in the Oracle Enterprise Manager Administrator's Guide.

    Configure the Oracle ZFS Storage Appliances for monitoring using the workflow included with Oracle ZFS Storage Appliances, as identified in the next section.

    Configuring the Oracle ZFS Storage Appliance

    To set up each appliance for monitoring from Cloud Control, enter the Oracle ZFS Storage Appliance Browser User Interface (BUI) as root or another user with the proper role to create worksheets, enable datasets, and create users and user roles on the appliance.

    1. Log in to the BUI and go to the Maintenance/Workflows tab.

    2. Run the Configure for Oracle Enterprise Manager Monitoring workflow, entering and recording the password for the oracle_agent user. You can rerun the workflow later to reset the worksheet, re-enable data sets, or recreate the user. The following occurs when the workflow is run:

    A restricted role user named oracle_user is created for use by the monitoring agent. The restricted access role associated with the oracle_agent user is also named oracle_agent. An analytics worksheet is automatically created to ensure smooth communication between the Cloud Control administrator and the storage administrator tasked with administering the target Oracle ZFS Storage Appliance.

    3. Communicate the oracle_agent password to the Cloud Control Administrator along with the following information:

    IP address for the network interface that you want Cloud Control to use to access management information (this interface must allow administration).

    Full DNS name that is used to locate the system and corresponds to the IP address.

    3

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    Port that is used by the BUI (typically 215 unless your company has a requirement to change management ports).

    Important: If you are going to deploy/upgrade to v1.4.x of the plugin, you must carry out the following additional steps:

    1. Log in to the BUI and go to the Configuration/Users tab.

    2. Click to edit the Oracle Agent User name field.

    3. Uncheck Kiosk user if it is checked.

    Configuring for Clustered Appliances

    If the Oracle Enterprise Manager Cloud Control will monitor resources available on a clustered Oracle ZFS Storage Appliance, you must configure both cluster heads. The oracle_agent user and worksheet will automatically be created on the peer head, but datasets will not become active on the peer. If the head containing the monitored resources fails, Oracle Enterprise Manager Cloud Control monitors the resources on the head that takes over the exported resources from the failed head. If the configuration step was not run on this head, the datasets might not be active, resulting in Metric Collection errors with the message workflow parameters are invalid. (See Table 5: Common Metric Collection Errors .)

    Note: When running the Configure for Oracle Enterprise Manager Monitoring workflow on the second head, the oracle_agent user does not have to be "Recreated." If you choose the option to Recreate the user, be sure to use the same password as the peer head set (the oracle_agent password will always be the same on both heads of the appliance).

    Configuring Database-to-Appliance Mappings

    In order for reports that include database-to-appliance mappings to work, the database and the host on which the database resides must be monitored by Oracle Enterprise Manager Cloud Control. For instructions on monitoring host storage and database instances, refer to the documentation provided with Oracle Enterprise Manager Cloud Control software (see the References section at the end of this guide ).

    4

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    Plug-in Management in the Oracle Enterprise Manager Cloud Control 12c Environment

    This section describes how to deploy and use the Oracle Enterprise Manager System Monitoring Plug-in in the Oracle Enterprise Manager Cloud Control 12c environment. If you need to upgrade your Oracle Enterprise Manager Cloud Control 11g environment to Cloud Control 12c, please follow the instructions at:

    http://www.oracle.com/technetwork/oem/grid-control/downloads/oem-upgrade-console-502238.html

    The Oracle Enterprise Manager Cloud Control Upgrade Guide is located at:

    http://download.oracle.com/docs/cd/E24628_01/upgrade.121/e22625.pdf

    For further detailed instructions on the Oracle Enterprise Manager's Plug-in Manager and the plug-in deployment process, see section 18 "Plug-in Manager" of the Oracle Enterprise Manager Cloud Control 12c Administrator's Guide at:

    http://docs.oracle.com/cd/E24628_01/doc.121/e24473/plugin_mngr.htm

    Deploying the Plug-in

    Deploying the Oracle Enterprise Manager System Monitoring Plug-in in the Oracle Enterprise Manager Cloud Control 12c environment is a two-step process. First, the plug-in is imported into the Oracle Enterprise Manager Cloud Control 12c environment using the emcli command. Then the Oracle Enterprise Manager console is used to deploy the plug-in by installing it on the Oracle Enterprise Manager's Management Service and Management Agent, and adding the Oracle ZFS Storage Appliance as a target. The emcli command can also be used to monitor the installation of the plug-in. These steps are described in more detail in the following instructions.

    Note: The Plug-in must be deployed on Oracle Enterprise Manager's Management Service before being deployed on Management Agents. The Oracle ZFS Storage Appliances can only be added as targets for monitoring once the Plug-in is deployed on both the Management Service and the Management Agent.

    To import the plug-in into the Oracle Enterprise Manager Cloud Control 12c environment, complete these steps:

    1. Set up the software library path (if you have not already done this).5

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    (a) From a terminal window on the server hosting Oracle Enterprise Manager 12c , create a new directory for the software library; for example: /home/oracle/swlib1.

    (b) In the Oracle Enterprise Manager console, go to Enterprise>Provisioning and Patching> Software Library>Actions>Administration and click Add Button.

    (c) In the popup window, enter swlib1 and /home/oracle/swlib1.

    (d) Wait a minute or so. Then, on the Oracle Enterprise Manager host, verify that the following directory structure was created under /home/oracle/swlib1:

    [oracle@aie-4200i swlib1]$ ls

    0 11 14 17 2 22 25 28 30 33 36 39 41 44 47 5 52 55 58 60 63 66 69 71 9 1 12 15 18 20 23 26 29 31 34 37 4 42 45 48 50 53 56 59 61 64 67 7 72 10 13 16 19 21 24 27 3 32 35 38 40 43 46 49 51 54 57 6 62 65 68 70 8

    2. Install the emcli kit.(a) Download emclikit.jar by following instructions available on the Enterprise

    Manager Console: Setup>Command Line Interace.

    (b) Copy the downloaded kit to a new folder; for example: /home/oracle/emcli.(c) Install the kit and set it up to work with your EM by entering:

    java -jar emclikit.jar client -install_dir=/home/oracle/emcli

    ./emcli setup -dir='/home/oracle/emcli' -url='https://:port/em' -user=sysman

    (d) Download the plug-in file (*.opar file) to a directory; for example: /home/oracle/12g_plugins

    (e) Upload the plug-in using the emcli command as shown in the following example:

    ./emcli import_update -file= "/home/oracle/12g_plugins/12.1.0.1.0_oracle.sun.oss7_2000_0.opar" -omslocal

    Note: On a Windows system, the emcli command does not require any quotes.To deploy the plug-in on the Oracle Enterprise Manager using the Oracle Enterprise Manager console, complete these steps:

    6

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    1. Go to Setup > Extensibility > Self Update. Click on Plugin. Make sure Oracle ZFS Storage Appliances is listed as Downloaded. Select it and click on Actions > Readme for the details of the plug-in.

    2. Click on the Plug-in tab or Setup > Extensibility > Plugin. Expand the Servers, Storage and Network category.

    3. Select the plug-in Oracle ZFS Storage Appliances from the table displayed in step 2 and select: Actions > Deploy On > Management Servers.

    4. In the popup window, enter the Repository Password and click Continue.

    5. Wait until the prerequisite checks are done. Click Next and then Deploy. An emcli command is displayed that can be run to monitor the status of the deployment and let you know when it has completed successfully. For example:

    emcli get_plugin_deployment_status -plugin_id=oracle.sun.oss7

    To install the Oracle Enterprise Manager System Monitoring Plug-in on a local Management Agent, complete the following steps:

    1. Select Setup > Extensibility > Plugins.

    2. Click and highlight the plug-in that you installed on the Management Server.

    3. Select Deploy On > Management Agent.

    4. Click Add.

    5. In the Target Type drop-down list, select Agent and highlight the local agent displayed in the table, then click Select.

    6. Click Continue.

    7. Wait until the prerequisite checks are done. Click Next and then Deploy. An emcli command is displayed that can be run to monitor the status of the deployment and let you know when it has completed successfully. For example:

    emcli get_plugin_deployment_status -plugin_id=oracle.sun.oss7

    Note: It will take about 3 minutes to complete this process.

    Adding Instances for Monitoring

    To install the Oracle Enterprise Manager System Monitoring Plug-in on a local Management Agent, complete the following steps:

    1. Select Setup -> Add Target -> Add Targets Manually.

    7

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    2. Select Add Targets Declaratively by Specifying Target Monitoring Properties .

    3. In the Target Type drop-down list, select Oracle ZFS Storage Appliance.

    4. For Monitoring Agent, click Search. In the window that is displayed, for Target Type drop-down list, select Agent and from the table below, click and highlight the agent you want to use for monitoring your target. Click Select.

    5. Click Add Manually.

    6. Add the Target details and click OK. Wait until you see the confirmation and click OK.

    7. To find the target, click on Targets->All Targets.

    8. Select the Oracle ZFS Storage Appliance added above to go to the appliance home page.

    Adding Clustered Systems for Monitoring

    Clustered systems require consideration by the Oracle Enterprise Manager Cloud Control administrator. Unless Cloud Control is being used as a general purpose Configuration Management Database (CMDB) or general purpose asset and resources tracking, only the head or heads containing active exported resources in use by other monitored software resources should be monitored by Oracle Enterprise Manager Cloud Control (even though the configuration workflow must be run on both cluster heads).

    If a storage head fails and an alternate head takes over the resources, the new head owning the logical resources will become monitored (if the takeover head was properly configured). Within the metric reports, the status of the cluster is given. The cluster status is important, as shared resources in an active/active environment will become even more pressured and the actual event of a cluster failover can cause disruption in quality of service objectives.

    Viewing Metrics and Reports

    The two primary ways to gather information about monitored instances of an Oracle ZFS Storage Appliance are viewing metrics and accessing reports. These are each described in the following sections.

    IMPORTANT: By default, all metrics that collect information from datasets on Oracle ZFS Storage Appliances are disabled upon installation. Likewise, certain metrics for pools, projects, and shares are disabled too. Further details on types of metrics as well as their

    8

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    settings and how to manage them are included in the section Metric and Collection Information later in this document.

    Viewing Metrics

    As a rule, more point in time information is available in raw metric information than in reports. Reports are focused on information that database administrators can use to analyze database performance and storage capacity.

    To view the raw information collected from an Oracle ZFS Storage Appliance:

    1. Log in to Enterprise Manager Cloud Control.

    2. Go to Targets > All Targets and select the Oracle ZFS Storage Appliance as the target.

    3. From the target's home page, select Oracle ZFS Storage Appliance > Monitoring > All Metrics.

    4. View the categories and information collected from the last collection interval.

    The raw metric information that you have access to can be found in Table 1: Metric and Collection Information.

    Accessing Reports

    To view reports for an Oracle ZFS Storage Appliance:

    1. Log in to Enterprise Manager Cloud Control.

    2. Go to Targets > All Targets and select the Oracle ZFS Storage Appliance as the target.

    3. From the target's home page, select Target > Information Publisher Reports.

    The information to which you have access is the same reports as those in the Enterprise Manager 10g/11g environment. Please refer to Table 2: Reports

    Reports accessed using a target instance differ from reports accessed from the Reports tab within Oracle Enterprise Manager Cloud Control. All instance reports are available under the tabs Reports >Storage>Network Attached Storage. When accessed from the Reports tab, reports include configurable time spans for graph display where applicable.

    Note: Administrators can clone and customize all system reports from the Reports tab.

    9

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    Upgrading the Plug-in

    Ensure the following steps are performed on the Oracle ZFS Storage Appliance.

    1. Log in to the BUI and go to the Configuration/Users tab.

    2. Click to edit the Oracle Agent user field.

    3. Uncheck Kiosk user if it is checked.

    Note: If you are upgrading the Plug-in from v1.x to v1.4.x, you must ensure that you are on Oracle Enterprise Manager 12.1.0.3.0 and the following steps are performed.

    After importing v1.4.x of the Plug-in and deploying it on an Oracle Enterprise Manager Management Service and Management Agent, you must restart Oracle Enterprise Manager's Management Service. The steps to do so are:

    1. Log in to the host where Oracle Enterprise Manager's Management Service is residing.

    2. Navigate to where Oracle Enterprise Manager's Management Service is installed (typically in a folder called Middleware).

    3. Navigate to Middleware/oms/bin and run the following two commands:

    emctl stop oms

    emctl start oms

    Un-deploying the Plug-in

    Oracle Enterprise Manager System Monitoring Plug-in can be un-deployed from the agent and Oracle Enterprise Manager. By un-deploying the plug-in from the agent, all the associated targets on the Oracle ZFS Storage Appliance that are monitored by the agent are automatically removed.

    To un-deploy the plug-in from the agent and Oracle Enterprise Manager:

    1. Go to Setup > Extensibility > Plug-ins.

    2. Under the Servers, Storage and Network category, select Oracle ZFS Storage Appliances.

    3. Go to Undeploy From > Management Agent > Add.

    4. Select the Agent from which the plug-in is to be un-deployed and click Continue.

    10

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    An emcli command that you can run from the command prompt to monitor the status of the undeployment activity is displayed.

    5. To remove the plug-in from the management servers, repeat preceding steps 1-4 while selecting Management Servers in step 3.

    To permanently delete the plug-in:

    1. Go to Setup > Extensibility > Self Update > Plug-ins.

    2. Under the Servers, Storage and Network category, select Oracle ZFS Storage Appliances.

    3. Select Actions > Remove. This will remove the plug-in permanently from the system.

    Metric and Collection Information

    Oracle Enterprise Manager Cloud Control displays a direct mapping of information to datasets and information collected in the target Oracle ZFS Storage Appliances.

    Note: Information collected by the System Attributes from the Workflow dataset indicates items that cannot be enabled/disabled by an administrator. This information is collected through scripts on each Oracle ZFS Storage Appliance.

    TABLE 1: METRIC AND COLLECTION INFORMATION

    METRIC NAME

    (X=DISABLED BY DEFAULT)

    COLUMN (*=KEY FIELD,

    ! = CONDITIONS ENABLED)

    ORACLE ZFS STORAGE

    APPLIANCE DATASET

    POLLING

    INTERVAL

    (MINUTES)

    UPLOAD FREQUENCY

    TO EM REPOSITORY

    Response Status N/A (request for host name) 5 Every polling cycle.

    Monitoring Prerequisite Status! N/A 30 Upload when there is a

    severity change on the

    thresholds.Message

    X Appliance Logical Status Share Identifier* System Attribute from

    Workflow

    20 Every 72nd polling cycle

    (or once per day) or

    whenever there is a

    severity change on the

    thresholds.

    Available (GB)

    Total Allocated (GB)

    Data (GB)!

    11

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC NAME

    (X=DISABLED BY DEFAULT)

    COLUMN (*=KEY FIELD,

    ! = CONDITIONS ENABLED)

    ORACLE ZFS STORAGE

    APPLIANCE DATASET

    POLLING

    INTERVAL

    (MINUTES)

    UPLOAD FREQUENCY

    TO EM REPOSITORY

    Share Percentage Used (%)!

    Quota (GB)

    Reserved (GB)!

    Unused Reserved (GB)!

    SCSI LUN

    X Appliance Project Status Project Identifier* System Attribute from

    Workflow

    20 Every 72nd polling cycle

    (or once per day) or

    whenever there is a

    severity change on the

    thresholds.

    Available (GB)

    Total Allocated (GB)

    Project Data (GB)!

    Project Percentage Used!

    Quota (GB)

    Reserved (GB)!

    Unused Reserved (GB)!

    Total Share Quota (GB)!

    Project Over Allocated!

    Configuration Service Status Service Name* System Attribute from

    Workflow

    60 Every 24th polling cycle

    (or once per day) or

    whenever there is a

    severity change on the

    thresholds.

    Service Status!

    X Appliance Capacity Totals Storage Pool* System Attribute from

    Workflow

    20 Every 72nd polling cycle

    (or once per day) or

    whenever there is a

    severity change on the

    thresholds.

    Pool Owner

    Pool Status

    Data Profile

    Available (GB)

    12

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC NAME

    (X=DISABLED BY DEFAULT)

    COLUMN (*=KEY FIELD,

    ! = CONDITIONS ENABLED)

    ORACLE ZFS STORAGE

    APPLIANCE DATASET

    POLLING

    INTERVAL

    (MINUTES)

    UPLOAD FREQUENCY

    TO EM REPOSITORY

    Total Used (GB)

    Pool Used Percentage (%)!

    Total Capacity (GB)

    Data Disks

    Spare Disks!

    Log Disks

    Cache Disks

    Total Project Quota (GB)!

    Quota Allocation (GB)!

    Reservations (GB)!

    Pool Over Allocated!

    X Cache ARC Size (MB)! arc.size 30 Every polling cycle.

    X Cache L2ARC Size (MB)! arc.l2_size 30 Every polling cycle.

    X CPU Utilization %! cpu.utilization 30 Every polling cycle.

    X Disk I/O Ops Operations per Second! io.ops 30 Every polling cycle.

    X Disk I/O Ops Latency Operations per Second! io.ops[latency] 30 Every polling cycle.

    X Disk I/O Bytes KB per Second! io.bytes 30 Every polling cycle.

    X Network Device KB per Second! nic.kilobytes 30 Every polling cycle.

    X NFSv2 Ops Operations per Second! nfs2.ops 30 Every polling cycle.

    X NFSv3 Ops Operations per Second! nfs3.ops 30 Every polling cycle.

    X NFSv3 Ops Latency Operations per Second! nfs3.ops[latency] 30 Every polling cycle.

    X NFSv4 Ops Latency Operations per Second! nfs4.ops[latency] 30 Every polling cycle.

    X NFSv4 Ops Operations per Second! nfs4.ops 30 Every polling cycle.

    13

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC NAME

    (X=DISABLED BY DEFAULT)

    COLUMN (*=KEY FIELD,

    ! = CONDITIONS ENABLED)

    ORACLE ZFS STORAGE

    APPLIANCE DATASET

    POLLING

    INTERVAL

    (MINUTES)

    UPLOAD FREQUENCY

    TO EM REPOSITORY

    X SMB Ops Operations per Second! smb.ops 30 Every polling cycle.

    X SMB Ops Latency Operations per Second! smb.ops[latency] 30 Every polling cycle.

    X iSCSI Ops Operations per Second! iscsi.ops 30 Every polling cycle.

    X iSCSI Ops Latency Operations per Second! iscsi.ops[latency] 30 Every polling cycle.

    Initiator And Target Groups Protocol* System Attribute from

    Workflow

    60 Every 24th polling cycle

    (or once per day).

    Group Name*

    Address

    Initiator Or Target

    X Cache ARC Activity Activity* arc.accesses[hit/miss] 30 Every polling cycle.

    Accesses per Second! arc.accesses[hit/miss]

    X Cache L2ARC Activity Activity* arc.l2_accesses[hit/miss] 30 Every polling cycle.

    Accesses per Second! arc.l2_accesses[hit/miss]

    X Cache ARC Activity by Share Share* arc.accesses[share] 30 Every polling cycle.

    Accesses Per Second! arc.accesses[share]

    X Cache L2ARC Activity by

    Share

    Share* arc.l2_accesses[share] 30 Every polling cycle.

    Accesses Per Second! arc.l2_accesses[share]

    X NFS V3 Operations By Share NFS V3 Share* nfs3.ops[share] 30 Every polling cycle.

    Operations Per Second! nfs3.ops[share]

    X NFS V4 Operations By Share NFS V4 Share* nfs4.ops[share] 30 Every polling cycle.

    Operations Per Second! nfs4.ops[share]

    X Network I/O Direction Direction* nic.kilobytes[direction] 30 Every polling cycle.

    Kilobytes Per Second! nic.kilobytes[direction]

    X Network I/O by Device Device* nic.kilobytes[device] 30 Every polling cycle.

    14

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC NAME

    (X=DISABLED BY DEFAULT)

    COLUMN (*=KEY FIELD,

    ! = CONDITIONS ENABLED)

    ORACLE ZFS STORAGE

    APPLIANCE DATASET

    POLLING

    INTERVAL

    (MINUTES)

    UPLOAD FREQUENCY

    TO EM REPOSITORY

    Kilobytes Per Second! nic.kilobytes[device]

    X Network Interface

    Performance

    Interface* net.kilobytes[interface] 30 Every polling cycle.

    Kilobytes Per Second! net.kilobytes[interface]

    Physical Component Status Cluster State! System Attributes from

    Workflow

    60 Every 24th polling cycle

    (or once per day) or

    whenever there is a

    severity change on the

    thresholds.

    Cluster State Description

    Cluster Peer State

    Cluster Peer State Description

    Appliance Information Reported Node Name System Attributes from

    Workflow

    60 Every 24th polling cycle

    (or once per day). Product Model

    Appliance Serial Number

    Chassis Serial Number

    Appliance Software Version

    Install Time

    Last Update Time

    Power On Time

    Up Time

    Clustered

    Cluster Peer

    X Appliance Logical Information Share Identifier* System Attributes from

    Workflow

    60 Every 24th polling cycle

    (or once per day). Share Type

    Mount Point

    Data Profile

    Data Compression

    Cache Usage

    NFS

    CIFS

    15

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC NAME

    (X=DISABLED BY DEFAULT)

    COLUMN (*=KEY FIELD,

    ! = CONDITIONS ENABLED)

    ORACLE ZFS STORAGE

    APPLIANCE DATASET

    POLLING

    INTERVAL

    (MINUTES)

    UPLOAD FREQUENCY

    TO EM REPOSITORY

    IP Network Interfaces Interface* System Attributes from

    Workflow

    60 Every 24th polling cycle

    (or once per day).IP Address*

    CIDR

    IP Version

    Management Enabled

    IP Network Interface to Datalink

    Relationship

    Interface* System Attributes from

    Workflow

    60 Every 24th polling cycle

    (or once per day).Datalink*

    Jumbo Frames

    IP Datalink to Device

    Relationship

    Datalink* System Attributes from

    Workflow

    60 Every 24th polling cycle

    (or once per day).Link*

    Class

    IP Network Devices Network Device* System Attributes from

    Workflow

    60 Every 24th polling cycle

    (or once per day) or

    whenever there is a

    severity change on the

    thresholds.

    Device Speed!

    Device Up!

    Alert Log Contents Problem ID* System Attributes from

    Workflow

    20 Every polling cycle.

    Time Stamp

    Alert Description!

    Alert Log Problem Type!

    Fault Log Contents Problem ID* System Attributes from

    Workflow

    20 Every polling cycle.

    Time Stamp

    Description!

    Fault Log Problem Type!

    Problem Log Contents UUID* System Attributes from

    Workflow

    20 Every polling cycle.

    Description!

    Problem Severity!

    Problem Log Code!

    Replication Activity Problem ID* 20 Every polling cycle.

    16

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC NAME

    (X=DISABLED BY DEFAULT)

    COLUMN (*=KEY FIELD,

    ! = CONDITIONS ENABLED)

    ORACLE ZFS STORAGE

    APPLIANCE DATASET

    POLLING

    INTERVAL

    (MINUTES)

    UPLOAD FREQUENCY

    TO EM REPOSITORY

    System Attributes from

    Workflow

    Time Stamp

    Replication Description!

    Replication Log Problem Type

    Metric Collections on Datasets

    By default, all Metrics that collects information from datasets on Oracle ZFS Storage Appliances are disabled. If you are interested in any particular dataset, please enable the Metric Collection for that dataset.

    There are two methods to enable metric collection for datasets on Oracle Enterprise Manager.

    Method 1:

    1. Go to the Target Homepage.

    2. Click on the Oracle ZFS Storage Appliance Menu > Monitoring > All Metrics.

    3. Select the Metric that you wish to enable.

    4. Click on the Modify button.

    5. Click on the Enabled radio button for Data Collection and Click OK.

    Method 2:

    1. Go to the Target Homepage.

    2. Click on the Oracle ZFS Storage Appliance Menu > Monitoring > Metric and Collections Setting.

    3. Select All Metrics under the View drop-down menu.

    4. Find the Metric that you wish to enable.

    5. Select the Disabled link under the Collection Schedule column.

    6. Click on the Enabled button under Data Collection and click Continue.

    7. You will be brought back to the Metric and Collections Settings page with a notification box. Ensure that you click on OK.

    17

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    You can use the previous steps to disable a dataset collection at any time.

    Note: Once you enable the metric collection for any dataset, you must ensure that the storage administrators also enable the same dataset on the Oracle ZFS Storage Appliances; otherwise you will start receiving metric collection errors.

    Metric Collections on Pools, Projects and Shares

    The following metrics that collect information about the pool, projects, and shares information are disabled by default:

    Appliances Capacity Total

    Appliances Project Status

    Appliances Logical Status

    Appliances Logical Configuration

    Please follow the same steps shown previously for enabling dataset metrics if you would like to collect information on these metrics.

    Note: Metrics and information presented on charts can take up to 60 minutes to populate as a result of polling intervals for relatively static information. Also, due to differing polling cycles between statistics, some information can populate prior to other information.

    Reports

    The following table lists available reports and the information they can provide on the Oracle ZFS Storage Appliances status.

    18

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    TABLE 2: REPORTS

    REPORT DESCRIPTION PURPOSE

    Monitored System Home Page Contains a series of graphs that give a high-level view

    of system throughput and clients accessing the

    system. It is important to note whether a system is Up

    or Down. Up means the management interface is

    available and simple properties can be queried that

    do not require login. Down means the management

    interface is not available. If a system is set up with

    more than one interface that allows access to the

    data path, it could be Down when the data path

    interface is still accessible. The management

    interface being flagged as Down is sufficient to

    warrant attention to the system.

    Provides at-a-glance status of the system.

    Database to Appliance Mapping

    Report -- available from the

    Reports tab or from a

    preselected Target Report's tab

    Shows mapping of databases using NFS to access

    data residing on an Oracle ZFS Storage Appliance.

    Uses Detail Report links to view deeper information

    on mappings and component activity. Detail Reports

    provide information such as:

    Database file mapping to exports residing

    on appliances

    Export data on which files reside

    (including capacity consumption and

    cache activity)

    The appliance on which a share resides

    The network interface and its activity

    through which a database file is accessed

    Provides a detailed view of how an Oracle

    ZFS Storage Appliance is being accessed

    using Oracle databases. Provides easy

    navigation to drill into the configuration

    and activity of Oracle ZFS Storage

    Appliance components that relate to the

    database deployment.

    Storage Capacity Report --

    available from the Reports tab

    or from a preselected Target

    Report's tab

    Gives detailed information about how storage is used

    on an individual Oracle ZFS Storage Appliance.

    Shows how capacity is allocated on an

    Oracle ZFS Storage Appliance.

    Network Interface Information

    Report -- available from the

    Reports tab or from a

    preselected Target Report's tab

    Provides details of network configuration and

    generalities about network activity over the past 24

    hours.

    Provides a detailed view of how ports on

    an Oracle ZFS Storage Appliance are

    configured. Shows aggregate activity of

    the interfaces.

    19

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    REPORT DESCRIPTION PURPOSE

    Federated Database to

    Appliance Mapping Report --

    available from Reports tab>

    Storage>Network Attached

    Storage group

    Similar to the Database to Appliance Mapping Report,

    but applicable to all monitored Oracle ZFS Storage

    Appliances.

    Provides a detailed view of how all

    monitored Oracle ZFS Storage Appliances

    are being accessed for information

    through Oracle databases. Provides easy

    navigation to drill into the configuration

    and activity of individual Oracle ZFS

    Storage Appliances and their components.

    Federated Capacity and

    Configuration Report -- available

    from Reports tab>Storage>

    Network Attached Storage group

    Gives high-level information about all monitored

    Oracle ZFS Storage Appliances.

    Provides a side-by side-comparison of all

    Oracle ZFS Storage Appliances (and

    software installed on them) to help you

    make decisions on where to allocate

    capacity.

    Default Threshold Values

    You can set custom thresholds for some metrics within Oracle Enterprise Manager Cloud Control. The alerts received are contained within the product and are not set as Alerts and Thresholds on the Oracle ZFS Storage Appliance itself.

    The following table shows the metrics that have thresholds set with their default values.

    TABLE 3: METRICS AND DEFAULT THRESHOLD VALUES

    METRIC/COLUMNS COMPARISON

    OPERATOR

    WARNING CRITICAL PURPOSE

    Alert Log Contents/Alert Description Matches N/A N/A Issued when the Alert Description Matches (via

    Regular Expression) the thresholds.

    Alert Log Contents/Alert Log Problem

    Type

    Matches N/A N/A Issued when the Alert Log Problem Type Matches

    (via Regular Expression) the thresholds.

    Problem Type can be Minor/Major/Critical.

    Appliances Capacity Totals/Pool Over

    Allocated

    = N/A YES Issued when the Total Project Quota (GB) crosses

    the Pool's Total Capacity (GB).

    Appliances Capacity Totals/Pool Used

    Percentage

    >= 80 90 Issued when the total capacity available on this

    appliance crosses the threshold.

    Appliances Capacity Totals/Quota

    Allocations

    >= N/A N/A Issued when the Quota Allocations crosses the

    thresholds.

    20

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC/COLUMNS COMPARISON

    OPERATOR

    WARNING CRITICAL PURPOSE

    Appliances Capacity

    Totals/Reservations

    >= N/A N/A Issued when the Reservation crosses the

    thresholds.

    Appliances Capacity Totals/Spare

    Disks

    = N/A N/A Issued when the Total Project Quota crosses the

    thresholds.

    Appliances Logical Status/Reserved = N/A N/A Issued when the Reserved value is equal to the

    thresholds.

    Appliances Logical Status/Share Data >= N/A N/A Issued when the Share Data crosses the

    thresholds.

    Appliances Logical Status/Share

    Percentage Used

    >= 80 90 Any share issues these warnings. A Cloud Control

    administrator can make specific shares trigger on

    other values.

    Appliances Logical Status/Unused

    Reserved

    = N/A N/A Issued when the Project Data crosses the

    thresholds.

    Appliances Project Status/Project Over

    Allocated

    = N/A YES Issued when the Total Share Quota (GB) of any

    project crosses the corresponding Project Quota

    (GB).

    Appliances Project Status/Project

    Percentage Used

    >= 80 90 Issued when the Project Percentage used crosses

    the thresholds.

    Appliances Project Status/Reserved = 0 N/A Issued when the Reserved value is equal to the

    thresholds.

    Appliances Project Status/Total Share

    Quota

    >= N/A N/A Issued when the Total Share Quota used crosses

    the thresholds.

    Appliances Project Status/Unused

    Reserved

    = N/A N/A Issued when Cache ARC (MB) crosses the

    thresholds.

    21

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC/COLUMNS COMPARISON

    OPERATOR

    WARNING CRITICAL PURPOSE

    Cache ARC Activity >= N/A N/A Issued when Cache ARC Activity (Accesses per

    Second) crosses the thresholds.

    Cache ARC Activity by Share >= N/A N/A Issued when Cache ARC Activity by Share

    (Accesses per Second) crosses the thresholds.

    Cache L2ARC >= N/A N/A Issued when Cache L2ARC (MB) crosses the

    thresholds.

    Cache L2ARC Activity >= N/A N/A Issued when Cache L2ARC Activity (Accesses per

    Second) crosses the thresholds.

    Cache L2ARC Activity by Share >= N/A N/A Issued when Cache L2ARC Activity by Share

    (Accesses per Second) crosses the thresholds.

    Configuration Service Status = N/A N/A This threshold is enabled but left off because a

    service that is down does not necessarily indicate a

    problem (the service might not be configured or

    operating by plan).

    CPU Utilization % >= 80.00% 90.00% Issued after three separate polling intervals. See

    the CPU Utilization above this percentage. Overuse

    of CPU can result in lower performance.

    Disk I/O Bytes >= N/A N/A Issued when Disk I/O Bytes (KB/S) crosses the

    thresholds.

    Disk I/O Ops >= N/A N/A Issued when Disk I/O Ops (Operation per Second)

    crosses the thresholds.

    Disk I/O Ops Latency >= N/A N/A Issued when Disk I/O Ops Latency (Operation per

    Second) crosses the thresholds.

    Fault Log Contents/Description Matches N/A N/A Issued when the Fault Description Matches (via

    Regular Expression) the thresholds.

    Fault Log Contents/Fault Log Problem

    Type

    Matches N/A N/A Issued when the Fault Log Problem Type Matches

    (via Regular Expression) the thresholds.

    Problem Type can be Minor/Major/Critical.

    IP Network Device/Device Speed

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC/COLUMNS COMPARISON

    OPERATOR

    WARNING CRITICAL PURPOSE

    IP Network Device/Device Up = N/A N/A This threshold is enabled but left off because a

    device that is down does not necessarily indicate a

    problem (the device might not be configured or

    operating by plan).

    iSCSI Ops >= N/A N/A Issued when iSCSI Ops (Operation per Second)

    crosses the thresholds.

    iSCSI Ops Latency >= N/A N/A Issued when iSCSI Ops Latency (Operation per

    Second) crosses the thresholds.

    Monitoring Prerequisite/Status = N/A 0 Issued when the credentials provided are incorrect.

    Once this alert is triggered, all metric collection will

    stop until this alert is cleared.

    Network Device >= N/A N/A Issued when Network Device (KB/S) crosses the

    thresholds.

    Network I/O by Device >= N/A N/A Issued when Network I/O by Device (KB/S) crosses

    the thresholds.

    Network Interface Performance >= N/A N/A Issued when Network Interface Performance

    (KB/S) crosses the thresholds.

    NFSv2 Ops >= N/A N/A Issued when NFSv2 Ops (KB/S) crosses the

    thresholds.

    NFSv3 Ops >= N/A N/A Issued when NFSv3 Ops (KB/S) crosses the

    thresholds.

    NFSv3 Ops by Share >= N/A N/A Issued when NFSv3 Ops by Share Ops (KB/S)

    crosses the thresholds.

    NFSv3 Ops Latency >= N/A N/A Issued when NFSv3 Ops Latency (KB/S) crosses

    the thresholds.

    NFSv4 Ops >= N/A N/A Issued when NFSv4 Ops (KB/S) crosses the

    thresholds.

    NFSv4 Ops by Share >= N/A N/A Issued when NFSv4 Ops by Share Ops (KB/S)

    crosses the thresholds.

    NFSv4 Ops Latency >= N/A N/A Issued when NFSv4 Ops Latency (KB/S) crosses

    the thresholds.

    23

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC/COLUMNS COMPARISON

    OPERATOR

    WARNING CRITICAL PURPOSE

    Physical Components Status/Cluster

    States

    = AKCS_

    OWNER

    AKCS_

    STRIPPED

    State of the cluster. OWNER means this head is

    the owner of the cluster and that resources are

    likely in contention. STRIPPED means that the

    head's resources are owned by its peer and must

    have the takeover redone.

    Problem Log Contents/Description Matches N/A N/A Issued when the Problem Description Matches (via

    Regular Expression) the thresholds.

    Problem Log Contents/Problem Log

    Code

    Matches N/A N/A Issued when the Problem Log Code Matches (via

    Regular Expression) the thresholds.

    Problem Log Contents/Problem

    Severity

    Matches N/A Major|

    Critical

    Issued when the Problem Severity Matches (via

    Regular Expression) the thresholds.

    Problem Type can be Minor/Major/Critical.

    Response/Status = N/A Down Provides information to the Home Page report on

    availability of the system's management path.

    Replication Activity/Replication

    Description

    Matches N/A Replication

    of.*

    Issued when the Replication Description Matches

    (via Regular Expression) the thresholds.

    SMB Ops >= N/A N/A Issued when SMB Ops (Operation per Second)

    crosses the thresholds.

    Stateless Alerts

    Most of the alerts that are set in the preceding table are stateful alerts. They will be automatically cleared once the underlying conditions are cleared. F or example, if a threshold of 20 is set for SMB Ops, an alert will be triggered once the value crosses 20, and cleared once the value drops below 20.

    However, there are a few special cases where the alerts cannot be cleared automatically. In these scenarios, these alerts are defined as stateless and you must manually clear them. All alerts from the following metrics are stateless alerts:

    Alert Log

    Fault Log

    24

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    Problem Log

    Replication

    To clear stateless alerts, follow these steps:

    1. Click on the Alert either from the Homepage or Incident Manager.

    2. Click on the Clear button.

    3. Click on Yes to permanently clear the alert.

    Oracle ZFS Storage Appliances Alerts

    If real-time monitoring for performance is required, the storage administrator should create appliance alerts and thresholds on the Oracle ZFS Storage Appliances user interfaces. This allows the storage administrator to view performance issues using real-time indicators, as opposed to the slower polling interval required by Oracle Enterprise Manager Cloud Control.

    Communicating Performance Issues to the Appliance Administrator

    The most common use of the Oracle Enterprise Manager Cloud Control Plug-in (besides simple capacity monitoring and high-level information collection) is analysis of application performance degradation relating to a database. Using the information within Oracle Enterprise Manager Cloud Control, you can provide to the Oracle ZFS Storage Appliance administrator information related to performance degradation for an application on the appliance.

    An Oracle ZFS Storage Appliance administrator has the advantage of real-time appliance analytics that can be viewed in various time spans (including initial configuration), monitored by Oracle Enterprise Manager Cloud Control. Some datasets are available from the first power-on of the appliance.

    When you present performance information to the storage administrator, you can start the analysis process by using the Oracle Enterprise Manager worksheet. This worksheet displays a down to the second view of the data compared to the less useful view of the data available to Oracle Enterprise Manager Cloud Control (often a 10- or 20-minute

    25

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    collection cycle averaged over the period, resulting in a much less dramatic performance curve).

    Using the worksheet and/or other datasets available to the storage administrator, you can analyze the cause down to the hardware or software component level. In the event of a resource contention issue, you can study levels of client access to determine how and when individual clients are accessing a system, along with the resources they are accessing. These can include network interfaces, logical resources (such as shares), and more.

    When initial communication is complete, the storage administrator should complete the analysis of the problem and propose a way to resolve a disruption in quality of service. If the problem is not likely associated with the Oracle ZFS Storage Appliance, the storage administrator can provide this information.

    Troubleshooting Monitoring Deployments

    A variety of reasons could cause a monitored instance in the Oracle Enterprise Manager Cloud Control to return information improperly. These issues are typically associated with misconfiguration, incorrect permissions, or changing the configuration within one of the products.

    26

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    TABLE 4: APPLIANCE PROBLEMS AND SOLUTIONS

    PROBLEM SOLUTION

    Appliance does not appear to be Up

    after being added as an instance to

    be monitored.

    If the appliance previously reported as Up but is now indicating it is Down, this is an indication

    that the management path is no longer available on the target Oracle ZFS Storage Appliance.

    The data path might still be available on the appliance, but this critical problem must be

    addressed.

    If this is the first time an appliance was added and it has never reported as being Up, an error

    could have been made when setting up the instance for monitoring. Check the following:

    Is the designated DNS name or IP address correct in the asset's Monitoring

    Configuration?

    Is the port number correct? The Oracle ZFS Storage Appliance default port is 215.

    Is the named asset accessible from the Agent system?

    Note: The Up/Down indicator does not require authentication to the target Oracle ZFS Storage Appliance, so the user ID and password are not relevant here.

    Checking the value of the Response Metric should show 0. This indicates that the Agent

    properly invoked the metric collection program, but the metric collection program could not

    connect to the target Oracle ZFS Storage Appliance given the parameters in the Monitoring

    Configuration.

    After determining 0 is shown in the field, ensure the appliance is available at the configured IP

    address in the monitoring configuration. However, if the value of Response Metric is a 1, the

    appliance is likely available. In this case, refresh the Cloud Control page reporting that the

    appliance is down. It will likely return to "Up".

    Appliance is Up, but all metrics are

    not being collected.

    Check the metric collection errors. (See Table 4: Common Metric Collection Errors.) The most

    common problem is misconfiguration of the password for the oracle_agent user within the Oracle Enterprise Manager Cloud Control environment.

    For most metric collection errors, a summary message and Java Stack Trace is generated

    and viewable by clicking the Metric Collection Errors link under the Home Page reports, then

    clicking through the text Exception. The initial summary message will contain information

    about the problem.

    27

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    PROBLEM SOLUTION

    Appliance is Up, but some metrics

    are not being collected.

    Determine what metrics are not being collected. Most likely, it is a group of metrics such as:

    ARC Activity by Share

    L2ARC Activity

    L2ARC Activity by Share

    NFS V3 Operations by Share

    NFS V4 Operations by Share

    There are usually three reasons for selective metrics not being collected:

    A component does not exist on a system This is common with the L2ARC metrics. In

    this case, you will not receive a metrics collection error; rather, the Last Upload time on the

    All Metrics page will remain empty (or include a -).

    The Configure for Oracle Enterprise Manager Monitoring workflow was not run on

    the target Oracle ZFS Storage Appliance. This problem embodies in metrics that

    are not collected by default on a target appliance, such as those listed above. In

    this case, Metric Collection Errors will occur within the Oracle Enterprise Manager

    Cloud Control until the target Oracle ZFS Storage Appliance is correctly

    configured (See Table 4: Common Metric Collection Errors.)

    A storage administrator either disabled or removed datasets on the target system.

    Ask the storage administrator to re-enable the dataset name that is in error. See

    Metric Information in Table 1: Metric and Collection Information.

    The list of shares in the Appliance

    Logical Configuration and the

    Appliance Logical Status appear to

    be out of sync.

    The layout of filesystems and their values are collected at different intervals than the capacity

    information for them. In the worst case, a metric could be as much as 60 minutes out of sync

    with a storage appliance. If you have a need for real-time information, refer to the Oracle ZFS

    Storage Appliance BUI.

    A Report or Metric Detail is not

    showing a change made on the

    system.

    Metrics and information about system values are collected in varying intervals. In the worst

    case, a metric could be as much as 60 minutes out of sync with a storage appliance. If you

    have a need for real-time information, refer to the Oracle ZFS Storage Appliance BUI.

    Unexpected Workflow Errors in

    running the Configuration Workflow

    on the Oracle ZFS Storage

    Appliance, such as aksh: uncaught

    error exception (code EAKSH_

    GENERIC_DATASET_

    INVALID)

    Be sure you are running the configuration script as a user with root access (or equivalent)

    with respect to creating and manipulating datasets and users.

    28

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    PROBLEM SOLUTION

    Database mapping reports do not

    show database artifacts that are

    mapped to NFS v3 or v4 shares on

    an Oracle ZFS Storage Appliance.

    The host, database, or both are not being monitored by Cloud Control. Install the proper

    agents on the host to ensure that storage attached to the system is being monitored.

    Refer to the Oracle Enterprise Manager Cloud Control documentation for information on

    monitoring host and database assets. See the Oracle Enterprise Manager Cloud Control

    documentation listed in the Additional References section at the end of this document .

    Database mapping reports do not

    show database artifacts that are

    mapped to iSCSI or Fibre Channel

    shares on an Oracle ZFS Storage

    Appliance.

    iSCSI and Fibre Channel mappings are not supported in this release.

    Metric Collection Errors

    Some common configuration problems manifest as Metric Collection Errors. In most cases, these error messages contain two parts a short string followed by a longer Java Stack Trace. The first string usually contains enough information to isolate the problem. The Java Stack Trace is used by the Support Center (if the problem is complex). Common Metric Collection problems are as follows:

    TABLE 5: COMMON METRIC COLLECTION ERRORS

    METRIC COLLECTION ERROR PROBLEM / SOLUTION

    login denied Problem: The user ID/password combination on the target Oracle ZFS Storage Appliance is incorrect.

    Possible causes:

    The Monitoring Configuration in Oracle Enterprise Manager Cloud Control is

    using the incorrect user ID/password. The user ID should be oracle_agent and the password should be the one provided by the storage administrator.

    The target Oracle ZFS Storage Appliance does not have the proper user ID and

    role.

    Solution:

    Change the monitoring configuration to use the proper user ID/password provided by the

    storage administrator, or reconfigure the target Oracle ZFS Storage Appliance to use the

    correct oracle_agent password.

    29

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC COLLECTION ERROR PROBLEM / SOLUTION

    workflow parameters are invalid Problem: The parameters passed to the Oracle ZFS Storage Appliance are invalid.

    Possible causes:

    A dataset on the Oracle ZFS Storage Appliance is not enabled.

    Resources from a monitored cluster head failed to the secondary head, for which

    the storage administrator did not run the Configuration workflow.

    An internal error occurred and the Agent is calling an incompatible workflow.

    Solution:

    Run the Configuration workflow on the target Oracle ZFS Storage Appliance and

    recreate the worksheet.

    Ask the storage administrator to ensure that all of the datasets in the created

    worksheet are being collected properly.

    (dataset name) is not enabled on the

    filer. Please enable the dataset on the

    filer or disable the collection of this

    metric.

    Problem: A dataset cannot be found on the Oracle ZFS Storage Appliance.

    Possible causes:

    A dataset on the Oracle ZFS Storage Appliance is not enabled.

    Solution:

    Ask the storage administrator to enable the dataset on the Oracle ZFS Storage

    Appliance.

    Disable the collection of this dataset metric with the following steps:

    1. Navigate to Metric and Collection Settings of the target.

    2. Change the Drop-down View option from All Metrics.

    3. Look for the metric name you wish to disable.

    4. Click on the Collection Interval (such as Every 30 minutes).

    5. Click the Disable button to disable collection of this metric.

    30

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    METRIC COLLECTION ERROR PROBLEM / SOLUTION

    cannot find specified workflow Problem: A workflow that collects and returns metrics to the Agent cannot be found on the Oracle ZFS Storage Appliance.

    Possible causes:

    The oracle_agent user (or user that you configured in the Monitoring Configuration) does not have workflow.*.* read authorization.

    An Oracle ZFS Storage Appliance is being used without the workflow on it, either

    because the version of the Oracle ZFS Storage Appliance software is incorrect,

    or the workflow was removed by a storage administrator.

    Solution:

    1. Run the Unconfiguration workflow on the target Oracle ZFS Storage Appliance

    and remove the oracle_agent user.2. Run the Configuration workflow on the target Oracle ZFS Storage Appliance and

    recreate the oracle_agent user.3. Ask the storage administrator to ensure the Configure/Unconfigure workflows are

    on the system.

    4. Contact the Support Center if the Agent fails to start collecting the dataset again.

    workflow failed Problem: The Agent is accessing an Oracle ZFS Storage Appliance, but the workflow is failing in some way.

    Possible causes:

    A problem in the metric collection logic is causing it to fail on the Oracle ZFS

    Storage Appliance.

    Solution:

    Run the Configuration workflow on the Oracle ZFS Storage Appliance and recreate the

    worksheet.

    If you are still receiving this error after recreating the worksheet, contact the Support Center.

    Oracle ZFS Storage Appliance Audit Logs

    For detailed debug information, use the audit log on the target Oracle ZFS Storage Appliance to determine which actions occurred with the workflows. Creation and deletion of artifacts (or the failure to create or delete an artifact) are traced in the audit log.

    Note: Debugging the reason that an Agent is not properly collecting statistics from an Oracle ZFS Storage Appliance can be more difficult and should only be done by service personnel. Contact the Oracle Support Center (see Additional Resources at the end of this document).

    31

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    Host Agent Logs

    Most problems (other than version mismatches) occur during the collection of metrics or response information. Look for the following logs on the Agent host. The Agent host may differ from the management station location. In the following file locations, %AGENT_LOCATION% indicates the home directory of the Agent, typically similar to /export/home/oracle/OracleHomes/agent10g. An asterisk (*) indicates there are several files with additional extensions.

    TABLE 6: HOST AGENT LOGS

    Agent Log Location Description

    %AGENT_HOME%/sysman/emd/oracle_zfs_plugin/logs/{target_ip}/sun_storage_7000_pipe.log*

    Contains errors that occurred below the Oracle Enterprise

    Manager Cloud Control framework during attempts to

    communicate with the Oracle ZFS Storage Appliance. These files

    will typically contain fine-grained details on connection problems

    with an Oracle ZFS Storage Appliance or failures that occur while

    retrieving information.

    %AGENT_HOME%/sysman/log/emagent.trc* These logs also contain connection exceptions as well as any information on using the data returned from the system to

    populate database tables.

    %AGENT_HOME%/sysman/log/* Remaining logs will contain finer-grained details on various elements in the Cloud Control Framework and 90% of the issues

    can be diagnosed with the above.

    Note: When interacting with the Support Center or developers within your own help desk, include the sun_storage_7000_pipe.log* files and emagent.trc* files for reference.

    32

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance Installation Guide, v1.4.1

    Additional Resources

    Oracle Support Centerhttp://www.oracle.com/support

    Patches and updates downloads from My Oracle Support (MOS)https://support.oracle.com/CSP/ui/flash.html#tab=PatchHomePage%28page=PatchHomePage&id=g1ws8maj%28%29%29

    Oracle Unified Storage Systems Documentationhttp://www.oracle.com/technetwork/documentation/oracle-unified-ss-193371.html

    Oracle Enterprise Manager Concepts Guidehttp://docs.oracle.com/cd/B10501_01/em.920/a96674/toc.htm

    Management of Heterogeneous Data Centers Using Oracle Enterprise Manager Plug-ins, An Oracle White Paperhttp://www.oracle.com/technology/products/oem/pdf/dc_mgmt.pdf

    Oracle Enterprise Manager Cloud Control 12c documentation, including Oracle Enterprise Manager Concepts and Oracle Enterprise Manager Administration Guidehttp://docs.oracle.com/cd/E24628_01/index.htm

    Oracle Enterprise Manager Cloud Control Upgrade Guidehttp://download.oracle.com/docs/cd/E24628_01/upgrade.121/e22625.pdf

    Oracle Enterprise Manager, Cloud Control Extensibility Programmers Reference,12c Release3 (12.1.0.3)http://docs.oracle.com/cd/E24628_01/doc.121/e25161/title.htm

    33

  • Oracle Enterprise Manager System Monitoring Plug-in for Oracle ZFS Storage Appliance, Installation GuideNovember 2013, Version 1.4.1Oracle Application Integration Engineering

    Oracle CorporationWorld Headquarters500 Oracle ParkwayRedwood Shores, CA 94065U.S.A.

    Worldwide Inquiries:Phone: +1.650.506.7000Fax: +1.650.506.7200

    oracle.com

    Copyright 2013, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.

    Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

    Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark licensed through X/Open Company, Ltd. 0611

    Configuring the Oracle ZFS Storage ApplianceConfiguring for Clustered AppliancesConfiguring Database-to-Appliance Mappings Deploying the Plug-inAdding Instances for MonitoringAdding Clustered Systems for MonitoringViewing Metrics and ReportsViewing MetricsAccessing Reports

    Upgrading the Plug-inUn-deploying the Plug-inMetric Collections on DatasetsMetric Collections on Pools, Projects and SharesStateless AlertsOracle ZFS Storage Appliances AlertsMetric Collection ErrorsOracle ZFS Storage Appliance Audit LogsHost Agent Logs