EGL2 Dell Compellent Storage Center-11

Embed Size (px)

Citation preview

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    1/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    1 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Dell Compellent Storage Center

    The Dell Compellent Storage Center is a block storage system aimed at the mid-tier enterprise market. Thelargest models are the SC9000 and the earlier SC8000. A scaled down version is offered as the SC4000, targeted

    at more cost competitive markets and the SCv2000 has three models in the family as economical entry level

    systems. Dell calls their design Fluid Data Architecture. The concept of Fluid Data that is embodied with the

    architecture of the Compellent systems is that data can be automatically stored, moved, exported, or imported

    based on the intelligence in the Compellent system. The advanced features of the Compellent system are

    centered on managing and moving data to optimize the storage. In addition to block storage, the Compellent

    system can have a NAS controller element added for file-based access.

    As a midrange storage system, Compellent competes with EMC VNX and VNXe, HDS VSP G-Series, IBM Storwize

    V7000, V5000, and V3700, NetApp FAS series, and the NetApp Engenio-based (E-Series) storage systems.

    Highlights

    Virtualization of the attached disk storage

    Wide-striping of data blocks across all physical disks

    RAID protection levels 0, 10, 5, 6

    Data prioritization control of service level assignment to prioritize data access

    Automated tiered storage

    Add-on NAS controller the FS8600

    Portable volumes to populate other Compellent storage systems

    Snapshots with consistency groups

    Additional element to provide file access using NAS protocols Live Volume storage hypervisor to present storage volume to two storage arrays for workload

    migration and failover Active-active stretched cluster support

    Perpetual license purchase of Storage Controller software once; continued on through multiple

    generations of hardware

    Two tiers of SSDs

    SSD 3D NAND flash at 3.8TB supported

    84 3.5 drives in 5U package

    Encryption with Self-Encrypting Drives

    Federation of multiple systems for management and volume migration

    Evaluator Group Comment: The Dell Compellent Storage Center systems are targeted for a range of

    customers from the higher end of the mid-tier market which is both price sensitive and has the

    expectation for features originally resident in high-end systems to the most economical entry-level

    market. There are many of the desired features in the Compellent system. The system is a very

    popular choice by customers for its economics and the basic features. Additional features will help

    expand the opportunity for Dell Compellent.

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    2/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    2 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Overview

    The Dell Compellent Storage Center offers multiple models with usage as midrange storage and entry-level systems providing block storage. In addition to the block storage offered with the Compellent

    system, another component may be added to the system to provide file-based storage as well. The

    block storage system is a dual controller system where both controllers are active and support high

    availability with failover from one controller to the other.

    The dual controller system has implemented virtualization of the physical disks attached which has

    enabled many advanced features including thin provisioning and wide striping. Highly valued advanced

    features such as snapshots and remote replication are included with the system.

    The add-on file-based capability is accomplished with the FS8600 Fluid File System which is a separate

    appliance. The add-on NAS capability can scale from one appliance with two controllers to fourapplicance with eight controllers. Prior to the availability of the FS8600, NAS attachment for Compellent

    was done with a server running the ZFS file system or the NX3000 NAS system (Windows Storage Server-

    based) attached to the dual controller system. With this added element, both block and file-based

    access can be supported.

    Figure 1: Dell Compellent Storage Center Family (source: Dell)

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    3/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    3 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Figure 2: Dell Compellent Controller and SAS Drive Enclosures

    http://www.compellent.com/Products/Hardware/Enclosures.aspx%23%23
  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    4/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    4 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Product Architecture

    The Dell Compellent Storage Center is a dual controller design with redundant components that allow

    for failover in the case of failure of a controller and continued operation in the case of failure of other

    components. The dual controller system uses communication through switches to detect failure in the

    paired controller and handle the failover.

    Physical disks are virtualized with blocks to be stored or retrieved managed across the disks in virtual

    volumes. Using the indirection with the block locations not tied to specific disks prior to storing allows

    for the features such as automated tiering, use of all physical resources, and redistribution of data to

    avoid hotspots to be implemented. The virtualized architecture with blocks being handled

    independently also gives flexibility in handling of the RAID protection and the ability to do wide striping

    allowing for more simultaneous disk access.

    Dell supports multiple types of SSDs as separate tiers of storage with migration between the tiers

    handled by the standard Data Progression software.

    Multiple RAID levels are supported with dynamic hot sparing. Any number of hot spares may be utilized

    and the failover and rebuild are automatic.

    The cache management allows for multiple threads to access cache with read-ahead algorithm for read

    caching of data in system memory and a separate write cache mirrored across the controllers. The write

    cache is battery backed to maintain data in the case of a failure or power loss.

    I/O operations are monitored and optimized for performance by merging contiguous block requests into

    a single request to minimize physical disk revolutions, sorting I/O request to minimize seek operations,

    and deep tagged command queuing for multiple outstanding I/O requests to disks.

    Another built-in unique feature is the ability to export and import volumes to a portable USB-attached

    disk drive.

    The controller has internal SSD disks used for operating system boot and for offloading battery-backed

    cache in the case of a power failure. The operating system is a 64 bit OS beginning with SC 6.1.

    The SC9000, SC8000, SC4000, and SCv2000 also support the Dell Fresh Air initiative and can operate at

    elevated temperatures to reduce the need for air conditioning and associated power consumption. The

    operating temperature can be up to 115 degrees F.

    The file-based access is a NAS controller is the FS8600 Fluid File System (described in a separate

    Evaluator Group document). The FS8600 is an appliance with two controllers. From one to four

    appliances may be attached and provide a global namespace. Prior to the availability of the FS8600, NAS

    was provided by either a server running a Linux operating system with the ZFS file system or the Dell

    NX3000 NAS system. The NAS controller attaches to the Dell Compellent Storage Center for its block

    storage while managing the file access using CIFS, NFS, and HTTP protocols. The block storage system

    features such as thin provisioning and snapshots can be exploited with the NAS system.

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    5/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    5 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Evaluator Group Comments: Using dual controller architecture brings the high availability

    characteristics expected in mid-tier systems to a broader market because of the economics of the

    Compellent system. Probably the most important element of the architecture is that it abstracts the

    physical disks and allows blocks to be moved and managed so that resources are optimized. This was

    accomplished by a design that from the start targeted this abstraction and did not graft it on to an

    existing design like some competing midrange storage systems.

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    6/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    6 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    System Software

    Disk Virtualization

    All disks are arranged as a pool and logical volumes are created from the pool and presented to attachedservers for usage. The volumes created from the pool can be of any size and are thinly provisioned.

    Copy-Mirror-Migrate

    A local clone (copy) or mirror of a volume may be made using the same mechanism as the data

    migration feature.

    Disk Optimizer

    When additional disks are added to the pool, data is restriped across the disks as a background activity

    to spread the performance (disk access) across the disks in the pool

    System Management

    Enterprise Manager

    The system management includes a web-based GUI for administration and resource management with

    monitoring and reporting function built-in. The Enterprise Manager software contains monitoring and

    reporting capabilities in addition to configuration and management. Enterprise Manager functions

    include:

    Monitoring and management of multiple local and remote Compellent systems

    Configuration and control of remote replication

    Monitoring and reporting on capacity and disk utilization Generate reports on storage utilization and performance

    Manage alert notifications

    Configure phone home support

    Chargeback configuration (associating volumes to groups) and reporting

    Dashboard display of status

    CLI for manual interaction or scripting

    Consolidated management with the FS8600 NAS

    Enterprise Manager is a suite of individual elements with groupings into three levels. The Foundation

    level is basic and is included with the product. The Reporter and Chargeback levels are optional and

    additional charge.

    Enterprise Manager Foundation Discovery, replication management, VMware SRM adapter, event

    management, free space recovery

    Enterprise Manager Reporter Performance management, capacity management, threshold alerting

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    7/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    7 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Enterprise Manager Chargeback Storage-based chargeback, power savings, and economic savings

    (called Hero by Compellent)

    Evaluator Group Comment: For a system targeted as a mid-range storage system, there aresystem management features that are typically on ly seen in high-end enterprise systems.

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    8/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    8 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Feature / FunctionSC9000 SC8000 SC4020

    Form Factor Rack Mount

    Number of Controllers2 with two 3.2 GHz 8-

    core Xeon processors

    2 with two 2.5 GHz 6-

    core Xeon processors

    1 or 2 with single 2.5 GHz

    quad-core Xeon

    processors

    RAID Levels 0, 5, 6, 10

    Maximum Disk DrivesSAS 960 - 12, 24 and

    84 drives per enclosure

    SAS 960 - 12, 24 and

    84 drives per enclosure

    120 (24 internal plus 96

    external)

    Drive Sizes Supported

    SSD: 400 GB, 1.6 TB / 3.8 TB SAS

    SAS 3.5: 450, 600 GB 15,000 RPM, 1, 2, 3, 4 TB 7,200 RPM

    SAS 2.5: 146 GB 15,000 RPM, 450, 600, 900 GB 10,000 RPM

    NL-SAS: 1 TB 7,200 RPM

    Maximum Raw

    Capacity3PB 3PB 500 TB

    Device Interface Speed SAS: 12 Gb/s SAS: 6 Gb/s SAS: 6 Gb/s

    Maximum backend

    portsSAS 40 SAS 20 4

    System Memory 256 GB per controller 32 128 GB 32 GB

    NVRAM Capacity 512 MB 512 MB N/A

    Drive intermix Yes Yes Yes

    Drive spin-down No No No

    Max iSCSI ports 20 @ 10 Gb/s 10 @ 10 Gb/s 4 @ 10 Gb/s

    Max Fibre channel

    ports

    20 @ 16 Gb/s, 40 @

    8Gb/s16 @ 16 Gb/s 8 @ 16 Gb/s

    Max FCoE ports 12 @ 10 Gb/s 10 @ 10 Gb/s N/A

    Table 1: Dell Compellent Model SC9000, SC8000 and SC4000 Comparison

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    9/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    9 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Feature / FunctionModel SCv2080 SCv2020 SCv2000

    Form Factor Rack Mount

    Number of Controllers2 with quad-core Xeon

    processors

    1 or 2 with quad-core Xeon

    processors

    1 or 2 with quad-core

    Xeon processors

    RAID Levels 0, 5, 6, 10

    Maximum Disk Drives 168 168 168

    Drive Sizes Supported

    SSD: 400 GB, 1.6 TB SAS

    SAS 3.5: 450, 600 GB 15,000 RPM, 1, 2, 3, 4 TB 7,200 RPM

    SAS 2.5: 450, 600, 900 GB 10,000 RPM

    NL-SAS: 1 TB 7,200 RPM

    Maximum Raw Capacity 504 TB 504 TB 504 TB

    Maximum backend ports 2 @ 6 Gb/s

    System Memory 8 GB per controller 8 GB per controller 8 GB per controller

    NVRAM Capacity 512 MB N/A

    Drive intermix Yes Yes

    Drive spin-down No No

    iSCSI ports 2 @ 10 Gb/s, 4 @ 1Gb/s 2 @ 10 Gb/s, 4 @ 1Gb/s 2 @ 10 Gb/s, 4 @ 1Gb/s

    Fibre channel ports 2 @ 16 Gb/s, 4 @ 8 Gb/s 2 @ 16 Gb/s, 4 @ 8 Gb/s 2 @ 16 Gb/s, 4 @ 8 Gb/s

    SAS ports 4 @ 12 Gb/s 4 @ 12 Gb/s 4 @ 12 Gb/s

    Table 2: Dell SCv2000 Models Comparison

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    10/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    10 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Host Operating System Support

    Microsoft Windows Server, Hyper-V

    Sun Solaris

    HP-UX

    Linux

    IBM AIX

    Novell Netware

    Apple MacOS

    Tru64

    VMware ESX

    Citrix XenServer

    Microsoft Cluster Server Support

    Table 3: Host OS Support

    Multipath I/O failover is supported for the Windows MPIO driver. Microsofts Volume Shadow Copy

    Service (VSS) is also supported for backup processes.

    SMI-S is not specified as being supported.

    Evaluator Group Comment: The specifications bring out the fact that the amount of cache in the

    controllers is less than some competing products. Additional cache would allow for more cache hits

    and write data for performance optimization.

    The NAS controller for the Compellent Storage Center is the Dell FS8600 Fluid File System. This product

    is described in the NAS section of the Evaluator Group Research. Prior to the availability of the FS8600,

    there were two NAS controllers that were used to provide file access to Compellent.

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    11/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    11 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Reliability, Availability and Serviceability

    The Compellent storage systems maintain data integrity and provide data availability via the

    following mechanisms:

    Dual controllers with failover for high availability

    Redundant, how-swappable components:

    - Controllers

    - Fans

    - Power supplies

    Dual loop, switched connections to storage devices

    Automatic drive failover

    Automatic drive rebuild to spare drive

    Configurable number of hot spare disks

    RAID protection across storage devices

    Remote reporting of status and automatic trigger of service actions

    Mirrored, battery backed cache

    Advanced Features

    Dynamic Capacity (Thin Provisioning) Dynamic Capacity is Compellents name for thinprovisioning. With the architecture to abstract the physical disks and deal with blocks individually, as

    capacity is required to store blocks of data, it is allocated dynamically and the space assigned to the

    required volume. With the abstraction or virtualization of storage, all physical storage can be assigned

    to a common pool to allow for better utilization. The allocation is done on a page basis where the page

    size is 2 MB. For Windows environments only, there is a reclaim deleted or unused space back into the

    common storage pool.

    Automated Tiered Storage (Data Progression) Based on policy settings, the Compellent Storage

    Center will automatically migrate data between tiers. The tiers that can be defined are for Solid State

    Disk, high performance disk, or lower performance disk. In addition to the tiering function, additional

    drives may be added to any tier and immediately utilized.

    Data Instant Replay (Snapshot) Snapshots or Replays can be created at any time interval. The

    Snapshots are copy on write with incremental changes captured to the read/write snapshot volume.With the management interface, a snapshot (or replay) can be restored to any point in time. There is no

    logical limit on the number of snapshots that can be made.

    Remote Instant Replay (Replication) Remote Instant Replay does an initial asynchronous

    replication of a volume and then incremental changes are replicated asynchronously. Either Fibre

    Channel or native IP interfaces may be used for the replication. One usage may be to use Portable

    Volumes for the initial synchronization with changes only after the volume has been introduced at the

    remote site. Synchronous replication of Live Volumes is supported over Fibre Channel networks.

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    12/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    12 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Dynamic Controllers (Clustering) Storage controllers may be run in clustering mode to provide for

    high availability and additional performance. Multi-path software such as Microsoft MPIO is used to

    failover access from servers. Cache is mirrored between the controllers.

    Portable Volumes Portable Volumes are USB attached removable disks where a volume may be

    copied and then introduced on another Compellent system. The Portable Volume can be used for

    populated a remote system, seeding the replication volume in a remote replication solution, or for

    creating a removable copy of a volume. Currently the Portable Volumes are 2 TB and have a 128-bit AES

    encryption capability for the drive. The connection is USB 2.0.

    Live Volume

    Live Volume is a software-based function that introduces the concept of a storage hypervisor into the

    Compellent system and provide Active-Active Stretched Cluster functionality. With Live Volume, storage

    volumes can be presented to two Compellent storage systems at the same time. This allows

    administrators to shift volumes between systems for application movement or high availabilityrequirements in case a system is being made unavailable for some reason. There is no limit to the

    number of volumes or the number of transfers that may be done.

    Live Volume is built upon Dell Compellent asynchronous replication where a volume it initially

    synchronized to another Compellent system and then periodic snapshots are done with the changed

    data replicated. Before a controlled swap is done, another snapshot is taken and the changed data

    replicated. The switch of server access to a volume on the companion Compellent system involves

    spoofing of the World Wide Name for the original access. The Live Volume can be switched back to the

    original system with the same replication mechanisms. The configuration for the Live Volume is set up

    through Enterprise Manager GUI and the swap can be initiated through the interface as well.

    Live Volume works with vMotion and Microsoft Live Migration to allow the storage volume to move

    when the virtual machines are moved. With the integration with the virtual machine software, when a

    virtual machine is moved, the Live Volume can automatically be moved as well. Live Volumes have the

    same capabilities of thin provisioning and tiering as other storage volumes.

    VMware VAAI Support

    The Dell Compellent system supports the VMware vStorage APIs for Array Integration (VAAI) which is in

    vSphere 4.1. VAAI is a set of mechanisms that allow processing for certain data-related services

    copying data when creating a new VM, for exampleto be offloaded from the ESX host to a storage

    array. The intent of these APIs is to streamline the functioning of the ESX server and speed-up delivery ofstorage-supported services.

    Full copy Enables the storage system to make full copies of data within the storage system

    without having the ESX host read and write the data.

    Block zeroing Enables storage systems to zero out a large number of blocks to speed

    provisioning of virtual machines.

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    13/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    13 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Hardware-assisted locking The system uses an atomic test and set implementation rather

    than a SCSI lock/reservation to protect the metadata for VMFS cluster file systems, thereby

    improving the scalability of large ESX host farms sharing a Datastore.

    Encryption

    Encryption of data at rest is supported through the use of Self-Encrypting Drives. FIPS 140-2 certification

    is included in the support. Storage Pools may be selected for encryption such that the entire system

    does not require having all data encrypted.

    Evaluator Group Comment: There are some very valuable features that can be exploited by

    customers. The snapshot and asynchronous remote replication are very useful to operations.

    An interesting feature is the Portable Volume. This feature has some unique usages and some

    potential protection opportunities for customers. There are probably some unique cases that

    customers have that will be very interesting to understand.

    The Live Volume feature can be a very useful capability both for the movement of virtual machines

    and to handle site failovers / switchovers. The granularity of one Compellent system to a second one

    is a per-volume basis which means there could be multiple sites involving Compellent systems linked

    together. Operationally, there can be advantages to being able to seamlessly direct the switchovers

    as needed.

    PerformanceNo SPC-1 performance information is available on the Storage Performance Council website at this time

  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    14/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    14 of

    2015 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

    Evaluator Group Comments

    The Dell Compellent Storage Center continues a very big success. It is offered through the reseller

    channel and is a popular offering because of the price and the basic capabilities it provides for thetarget market of mid-size enterprises. The new controller will allow for greater performance and

    scaling up market.

    Positives:

    Probably the most important thing to realize is that the Compellent system was designed from the

    ground up as a system that virtualized the attached storage devices. This allowed for a design that

    could build upon that without the constraints of other systems that evolved from fixed type device

    mapping. The capabilities of thin provisioning and wide striping as well as the flexibility in handling

    RAID protection were part of the initial design of virtualization.

    The advanced features currently available are valued by customers most specifically the snapshot

    and the asynchronous remote replication. Unique to the Compellent system is the Portable Volume

    feature. This feature can be exploited in a number of ways and probably will lead competition to add

    a similar capability at some point but after Compellent has already proved it to be a success.

    The Live Volume feature has great operational value for customers in that it can be used to provide

    access to volumes that are dynamically replicated between two Storage Center systems by servers in

    case of failure or movement of the application to another server. The flexibility of providing access to

    a volume through on Storage Center system or another that is transparent and can be seamlessly

    switched is a feature that is usually only found on much more expensive systems. The simplicity ofswitching makes this capability more likely to be used by IT personnel.

    Potential Concerns:

    As the Compellent system is considered by customers who have had midrange systems with additional

    advanced features, some of those features will be determining factors in purchases. Compellent will

    need to continue to add more features such as a larger caching area and deduplication. These will be

    competitive issues at some point in time to expand the market opportunity. The current product is still

    very applicable for customers in the target market with excellent features and capabilities.

    There is confusion sometimes with the Dell EqualLogic systems, which creates problems with

    customers in making a selection. The potential acquisition by EMC also adds to some confusion.

    More detailed information is available athttp://evaluatorgroup.com

    Copyright 2015 Evaluator Group, Inc. All rights reserved.

    No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including

    photocopying and recording, or stored in a database or retrieval system for any purpose without the express written consent of Evaluator

    Group Inc. The information contained in this document is subject to change without notice. Evaluator Group assumes no responsibility for

    http://evaluatorgroup.com/http://evaluatorgroup.com/http://evaluatorgroup.com/http://evaluatorgroup.com/
  • 7/25/2019 EGL2 Dell Compellent Storage Center-11

    15/15

    Dell Compellent Storage Center - Product Analysis

    Evaluator Series Research - October 26, 2015

    Pag

    15 of

    2015 Evaluator Group Inc All rights reserved Reproduction of this publication in any form without prior written permission is prohibited

    errors or omissions. Evaluator Group makes no expressed or implied warranties in this document relating to the use or operation of the

    products described herein. In no event shall Evaluator Group be liable for any indirect, special, inconsequential or incidental damages

    arising out of or associated with any aspect of this publication, even if advised of the possibility of such damages. The Evaluator Series is a

    trademark of Evaluator Group, Inc. All other trademarks are the property of their respective companies.