1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Troubleshooting KerberosIn Hadoop :Taming the BeastDataWorks Summit

Sept 2017

2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Author Profile

Vipin Rathor

Sr. Product Specialist (HDP Security)

Contributed to Kerberos, Apache Zeppelin, Apache Atlas

vrathor@hortonworks.com / @VipinRathor46

3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Agenda

• Why Kerberos?

• Where is Kerberos used across the Hadoop Stack?

• What is Kerberos & how does it work

• Realms, Principals and Keytabs

• Systematic Approach to Kerberos Nirvana

• Tools available in Hadoop

• Native Kerberos Tools / Debug Options

• Kerberos Checklist

• Most Common Kerberos Error Messages (& their meanings)

4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Why Kerberos?

• Universal Authentication mechanism for Hadoop stack

• Integrates with enterprise user management (e.g. Active Directory)

Solves:

• How can parts of a cluster trust each other(NameNodes, DataNodes, YARN, HBase, ZooKeeper...)

• How can users trust the system?

• How can the system trust users?

• Foundation for: how can users delegate rights to applications?

• Without Kerberos: your cluster has NO security

Hadoop clusters are some of the largest Kerberos systems ever!!

5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Where is Kerberos used across the Hadoop Stack?

• Ubiquitous End-User / Hadoop Service Authentication mechanism

• Hadoop DelegationToken (Delegated authentication to NameNode)

• != Kerberos Tickets

• Bootstrapped with Kerberos authentication token

• HTTP Authentication

• Using SPNEGO (RFC 4559)

• Via Browsers / cURL (curl --negotiate)

• RPC Authentication

• Using Simple Authentication & Security Layer aka SASL (!= SSL)

• Java API Based Kerberos login

• Using JGSS / JAAS

• GSS-API (RFC 2743)

6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

What is Kerberos

• Open source, Developed by MIT

• Password is NEVER transmitted over wire

• Central trusted authority – Key Distribution Center (KDC)

• Symmetric key (common shared key)

• Flavors:• MIT Kerberos

• Active Directory

• Heimdal Kerberos (OS X)

7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

How does Kerberos work

End User

- Does kinit (1 & 2)

- Runs HDFS

command (3 - 6)

Hadoop NameNode

- Starts up with nn.service.keytab

- Verifies user and gives access to

HDFS

KDC

- Provisions user keys and

service keytabs (e.g.

nn.service.keytab)

- Provides TGT and TGS

8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Realms, Principals and Keytabs

• Realm

• User Principal• E.g. user1@HWX.COM

• ken/admin@HWX.COM

• ken/sandbox.hortonworks.com@HWX.COM

• Service Principal• E.g. HTTP/sandbox.hortonworks.com@HWX.COM

• nn/node1.hortonworks.com@HWX.COM

• dn/node2.hortonworks.com@HWX.COM

• dn/_HOST@HWX.COM

• Keytabs• Service keytabs (for service)

• Headless keytabs (for user)

9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Systematic Approach to Kerberos Nirvana

• Identify the involved parties (user, service, keytabs, nodes)

• Identify the stage where Kerberos is failing

• Based on stage & error message, narrow down between client-side or service-side issue

• Check & verify configurations for correctness using the appropriate tools

• Repeat as necessary

10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Kerberos Tools Available in Hadoop

• Kdiag

• Runs a series of diagnostic checks & gives suggestions

• hadoop org.apache.hadoop.security.KDiag

11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Kerberos Tools Available in Hadoop (cntd..)

• HadoopKerberosName

• Checks Auth_to_local rules (Kerberos Principal to Unix user name conversion)

• hadoop org.apache.hadoop.security.HadoopKerberosName

nn/bali1.openstacklocal@LAB.HORTONWORKS.NET

12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Native Kerberos Tools / Debug Options

• via command line• kinit

• klist -eaf / klist –kte

• kvno

• kdestroy

• export KRB5_TRACE=/tmp/krb5-curl.outcurl -ivL --negotiate -u: "http://namenode-host:50070/webhdfs/v1/?op=LISTSTATUS"

• via debug messages• export HADOOP_JAAS_DEBUG=true

• export HADOOP_ROOT_LOGGER=DEBUG,console

• via Java library• -Dsun.security.krb5.debug=true

• -Dsun.security.spnego.debug=true

• export OPTS=“$OPTS -Dsun.security.krb5.debug=true”

13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Kerberos Checklist

• FQDN

• Name Resolution

• If DNS is configured, then check reverse lookup

• Date/Time sync (< 5 minutes)

• Configuration file - /etc/krb5.conf

• Principal Names

• Stale Keytabs (via kvno)

• Credential Cache location (JAAS config)

• Which Java suite, JCE policy

• KDC log file - /var/log/kerberos/krb5kdc.log

14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Most Common Kerberos Error Messages (& their meaning)

• <unknown client> for <unknown service>

• Decrypt Integrity Check Failed

• AES256 EncType not supported

• Clock skew too great

• Kerberos service principal not found in the database

• Client not found in the database

• No valid initial credential found

15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

References

• http://web.mit.edu/kerberos/

• http://www.kerberos.org/software/tutorial.html

• https://github.com/steveloughran/kerberos_and_hadoop

16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Thank you !