Certification Report for Apache Cassandra 4.0

© Instaclustr Pty Ltd, 2021 | 1 of 19

Certified Apache Cassandra Certification Report for Apache Cassandra 4.0.0 Document Identifier: ICF_CCR_4.0.0 Document Version: 1.0 Issue Date: 25 OCT 2021


Table of Contents Table of Contents............................................................................................................................ 2

Executive Summary ........................................................................................................................ 3

Significant Changes .................................................................................................................... 3

Stability and Testing................................................................................................................. 3

Full Query Logging, Audit, and Diagnostic Events ................................................................... 3

Virtual Tables ........................................................................................................................... 3

Netty Transport ........................................................................................................................ 4

Support Caveats ............................................................................................................................. 4

Unit Test and D-Test Results .......................................................................................................... 5

Unit Tests .................................................................................................................................... 5

D-Tests........................................................................................................................................ 5

Performance Test Results ............................................................................................................... 6

Performance Result Summary ..................................................................................................... 6

Test Approach ............................................................................................................................. 6

Throughput .............................................................................................................................. 6

Throughput—only assessing latency ....................................................................................... 7

Latency .................................................................................................................................... 7

Three Medium Node Cluster Test ................................................................................................ 7

Throughput .............................................................................................................................. 7

Throughput—only assessing latency ....................................................................................... 8

Latency .................................................................................................................................... 9

Three Large Node Cluster Test ................................................................................................. 10

Throughput ............................................................................................................................ 10

Throughput—only assessing latency ..................................................................................... 11

Latency .................................................................................................................................. 12

Nine Medium Node Cluster Test ................................................................................................ 13

Throughput ............................................................................................................................ 13

Latency .................................................................................................................................. 14

Soak Test Results ......................................................................................................................... 15

Integrations Shakedown Results ................................................................................................... 16

Security Vulnerability Assessment Results.................................................................................... 16


Executive Summary In order to give a Cassandra Version an Instaclustr certification, we run it through a rigorous testing process, in order to identify issues which may occur when running that version in your production environment. Our aim is to wrap the Apache project releases in some further QA activities and to provide an open testing process and set of results to provide a higher level of assurance on the quality of the release, identify any performance regression, or potential caveats, as well as some further guidance on how to use the release. In order to do this, we run a full suite of Unit, and D-Tests, to help identify any regressions. Performance tests are run on multiple sized clusters (in terms of size and number of nodes), in order to highlight any potentially introduced performance drops. We also run Soak Testing, where we run a production-like load at a 9-node cluster, simulating node outages, and triggering repairs. Finally, we confirm that all major driver versions are compatible with the new Cassandra release. Towards the end of our testing, a bug was found in Cassandra 4.0 which resulted in the release of Cassandra 4.01. We recommended deploying Cassandra 4.0.1 and expect results to be very similar to those presented here for Cassandra 4.0. A detailed description of our testing is available in our Certified Cassandra Test Plan, v1.1.

Significant Changes

Stability and Testing One of the explicit goals for Apache Cassandra 4.0.0 was to be the “most stable major release of Cassandra ever” (https://instac.io/37KfiAb) For more details see our blog on the topic: https://www.instaclustr.com/apache-cassandra-4-0-stability-and-testing/

Full Query Logging, Audit, and Diagnostic Events Apache Cassandra 4.0.0 brings about a long-awaited feature for tracking and logging database user activity. Primarily aimed at providing a robust set of audit capabilities allowing operators of Cassandra to meet external compliance obligations, it brings yet another enterprise feature into the database. Combining work for the full query log capability, the audit log capability provides operators with the ability to audit all DML, DDL, and DCL changes to either a binary file or a user configurable source (including the new Diagnostics notification changes). For more details see our blog on the topic: https://www.instaclustr.com/apache-cassandra-4-0-audit/

Virtual Tables Among the many exciting new features, Cassandra 4.0.0 boasts is the implementation of Virtual Tables. Up until now, JMX access has been required for revealing Cassandra details such as running compactions, metrics, clients, and various configuration settings. With Virtual Tables, users will be able to easily query this data as CQL rows from a read-only system table. For more details see our blog on the topic: https://www.instaclustr.com/apache-cassandra-4-0-virtual-tables/

https://instac.io/37KfiAb

https://www.instaclustr.com/apache-cassandra-4-0-stability-and-testing/

https://www.instaclustr.com/apache-cassandra-4-0-audit/

https://www.instaclustr.com/apache-cassandra-4-0-virtual-tables/

Ben Slater

will need a link


Netty Transport One of the headline features for Apache Cassandra 4.0.0 is the refactor of internode messaging to use Javas (non-blocking) NIO capability (https://issues.apache.org/jira/browse/CASSANDRA-8457 and https://issues.apache.org/jira/browse/CASSANDRA-15066) via the Netty library (link to Netty).

This provides performance improvements around operations that require streaming data between the nodes. For more details see our blog on the topic: https://www.instaclustr.com/apache-cassandra-4-0-netty-transport/ For the full list of changes see the official CHANGES.txt file.

Support Caveats The following features either have known issues or limited known production exposure and are supported by Instaclustr with the specified caveats. If a feature is not listed, Instaclustr provides full support for the feature.

Feature Caveat

Materialized Views Not Recommended. The project has moved this feature officially to experimental. Supported with known limitations

See https://www.instaclustr.com/apache-cassandra-materialized-view-instaclustr-support/

Change Data Capture

Not supported. Functionality is relatively new and complex and we have not seen any adoption in our customer base and are not aware of significant production usage. Community consensus appears to be that this feature should be considered experimental.

SASI Bug fix support only for demonstrated regressions against previous releases. We have not seen wide production use of this technology and there have been numerous major bugs in recent versions. Community consensus appears to be that this feature should be considered experimental although we are aware of significant production users.

Triggers Not supported. While triggers have been available for some time and the base functionality is generally stable, they are not widely used and tested in production, and the use of triggers is complex and may have performance side effects. Community consensus appears to be that this feature should be considered experimental.

https://issues.apache.org/jira/browse/CASSANDRA-8457



https://www.instaclustr.com/apache-cassandra-4-0-netty-transport/

https://www.instaclustr.com/apache-cassandra-4-0-netty-transport/

https://github.com/apache/cassandra/blob/cassandra-4.0.0/CHANGES.txt

https://www.instaclustr.com/apache-cassandra-materialized-view-instaclustr-support/

https://www.instaclustr.com/apache-cassandra-materialized-view-instaclustr-support/


User Defined Functions, Aggregates, and Types

Bug fix support only for demonstrated regressions against previous releases. The functionality has reasonable adoption although some edge case behaviour may not be well defined and individual use cases should be thoroughly tested.

Cassandra Hadoop Interface

Not supported. Hadoop API was not widely used and is now deprecated.

Thrift Thrift API has been removed.

Unit Test and D-Test Results

Unit Tests

Test Run Total Tests Tests Passing Tests Failing ant-test-all 5682 5682 0 ant-test-long 96 96 0 ant-test-fqltool 25 25 0 ant-stress-test 20 20 0

Failed tests and commentary:

Test Name Impact/Commentary

None No failing tests

D-Tests

Test Run Total Tests Test Passing Tests Failing

dtests-with-vnodes 859 857 2

dtests-no-vnodes 920 918 2


Failed tests and commentary:

Test Name Impact/Commentary

test_revoked_login Test created after 4.0.0 release to test invalidating auth cache from nodetool. Functionality will be in 4.1 CASSANDRA-16404.

test_revoked_dc_access Test created after 4.0.0 release to test invalidating auth cache from nodetool. Functionality will be in 4.1 CASSANDRA-16404.

Testing Notes:

● We have run the unit and D-tests through Circle Ci with the standard project settings with increased parallelism (4 -> 70) and resource_class (medium -> xlarge) for speed of results

● The Dtest repository is always targeting trunk so any tests added after a release are unlikely to pass

● The failing test has now been modified not to run on versions before 4.1—after re-testing these passed

Performance Test Results

Performance Result Summary In most cases (with an important caveat in the next paragraph) Cassandra 4.0.0 is able to achieve higher throughput than Cassandra 3.11.8. In particular, when just assessing cluster load in terms of the resultant operation latency, Cassandra 4.0.0 performs better then (or at least as well as) Cassandra 3.11.8 in most tests. Latency is also generally improved, particularly at the p95 and p99 level. In the longer running Soak Tests, while there was a slight increase in median and p95 latency, p99 latency was significantly improved and garbage collection activity was also reduced. That said, the results do suggest that operations involving medium-sized writes (~12KiB in our tests) may have some degradation of performance on Cassandra 4.0.0. We recommend that users test their use case(s) before upgrading.

Test Approach

Throughput Comparison of max throughput across versions with <5ms median write latency, <20ms median read latency, OS load normalized to the number of CPU cores < 4 and (for medium payloads) the slope of pending compactions < 0.004.




Throughput—only assessing latency Comparison of max throughput across versions with <5ms median write latency, <20ms median read latency.

Latency Comparison of latency across versions with constant ops/sec

Three Medium Node Cluster Test

Throughput

Test Ops/Sec

3.11.8 4.0.0 insert-small 16642 17055

read-small 13485 18225

mixed-small 11257 11375

insert-medium 265 186

read-medium 11963 9653

mixed-medium 553 407


Throughput—only assessing latency

Test Ops/Sec

3.11.8 4.0.0 insert-small 25359 27669







Latency

Latencies

Median p95 p99

Test Ops/Sec 3.11.8 4.0.0 3.11.8 4.0.0 3.11.8 4.0.0

insert-small 14974 2.00 2.00 28.73 22.47 99.67 105.27

read-small 9437 1.77 1.70 4.20 2.60 43.07 10.73

mixed-small 7878 1.77 1.80 14.47 5.87 70.73 46.17

insert-medium 238 2.07 2.07 2.77 2.60 2.87 2.80

read-medium 8373 2.00 2.70 30.80 17.93 83.47 46.03

mixed-medium 387 2.13 2.37 2.87 3.53 3.33 6.17


Three Large Node Cluster Test

Throughput

Test Ops/Sec

3.11.8 4.0.0

insert-small 24543 26979







Throughput—only assessing latency

Test Ops/Sec

3.11.8 4.0.0

insert-small 24128 28011







Latency

Latencies

Median p95 p99

Test Ops/Sec 3.11.8 4.0.0 3.11.8 4.0.0 3.11.8 4.0.0

insert-small 22085 3.27 2.60 45.60 30.15 106.57 94.6

read-small 13524 2.40 2.23 5.23 3.43 33.67 14.03

mixed-small 10842 2.37 2.30 28.50 12.03 71.10 51.30

insert-medium 230 2.40 2.37 3.07 3.10 3.23 3.27

read-medium 12050 2.77 2.60 29.77 6.70 71.40 25.07

mixed-medium 371 2.47 2.57 3.30 3.40 3.60 3.97


Nine Medium Node Cluster Test

Throughput

Test Ops/Sec

3.11.8 4.0.0 insert-small 37817 37509







Latency

Latencies

Median p95 p99

Test Ops/Sec 3.11.8 4.0.0 3.11.8 4.0.0 3.11.8 4.0.0

insert-small 34031 1.83 1.87 13.97 15.47 57.37 62.87

read-small 27226 1.83 1.70 2.87 2.70 11.77 10.87

mixed-small 23988 1.80 1.77 7.70 6.57 46.13 39.17

insert-medium 779 2.10 1.97 2.70 2.67 2.87 2.87

read-medium 22248 2.10 2.17 24.00 7.20 88.63 28.15

mixed-medium 1849 2.03 2.13 2.77 3.17 4.43 5.13


Soak Test Results

Measure 3.11.8 4.0.0

Operation rate 23,991 23,991

Latency median 1.5 1.7

Latency 95th percentile 2.2 2.6

Latency 99th percentile 29.3 19.6

Total GC time 220.4 166.2

Avg GC time 84.8 80.9

StdDev GC time 27.8 23

Total GC count 2600 2053

Errors 0 0

Results


Integrations Shakedown Results

Driver Status Version

cqlsh PASSED cqlsh 6.0.0

java PASSED 3.10.2

python PASSED Version: 3.25.0

ruby PASSED cassandra-driver (3.2.5)

go PASSED bc256bbb90de7113a74ad4d777beeec75eb9c4e7

clojure PASSED 4.2.2

Security Vulnerability Assessment Results

Dependency CVE Severity Classification Explanatory Text guava-27.0-jre.jar CVE-2020-8908 LOW Not vulnerable The vulnerable

component is only used in tests.

jbcrypt-0.3m.jar CVE-2015-0886 MEDIUM Partially vulnerable

To be vulnerable to this CVE Cassandra would need to have the system property ‘-Dcassandra.auth_bcrypt_gensalt_log2_rounds’ set to 31. By default it is set to 10. Instaclustr Managed Platform uses the default so it is not vulnerable. There is an Apache Cassandra project ticket to address this issue: CASSANDRA-9384

netty-all-4.1.58.Final.jar

CVE-2021-21290

MEDIUM Not vulnerable The CVE relates to http requests that are not used by Cassandra 4.

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2020-8908






netty-all-4.1.58.Final.jar CVE-2021-21295

MEDIUM Not vulnerable The CVE relates to

http requests that are not used by Cassandra 4

netty-all-4.1.58.Final.jar CVE-2021-21409 MEDIUM Not vulnerable

The CVE relates to http requests that are

not used by Cassandra 4

cassandra-driver-core-3.11.0-shaded.jar

CVE-2020-17516 HIGH False Positive Only affects earlier versions of Cassandra


CVE-2020-13946 MEDIUM False Positive Only affects earlier

versions of Cassandra


CVE-2018-8016 CRITICAL False Positive Only affects earlier

versions of Cassandra

chronicle-wire-2.20.117.jar

CVE-2018-8909, CVE-2020-

15258, CVE-2020-

27853, CVE-2021-

21301, CVE-2021-

32665, CVE-2021-

32666, CVE-2021-

32755,

Highest is CRITICAL False Positive Identified CVEs are for

a different application



















Documents

Certification Report for Apache Cassandra 4.0