September Cloudera Customer Advisory

Key Distribution Center (KDC) may be overwhelmed on large Kerberized Impala clusters

After cluster startup, when Impala runs a query that shuffles data amongst all nodes or a large subset of nodes in a Kerberos-secured cluster, every node will need to acquire a Ticket Granting Server (TGS) for every other node. In a cluster of hundreds of nodes or more, this can overwhelm the KDC, and queries can exit with an error ("Could not contact KDC for realm").

For example, queries that could fail with kerberos errors just after cluster startup are those that involve one or more joins on large tables that have data spread out over the entire cluster.

Users affected:

Affected users are those running Kerberized Impala clusters with 100 nodes or up.

Impact:

Failed queries for a few minutes immediately after cluster startup.

Action required:

A simple workaround is to run a warm-up query a few times until all the tickets are established (which can take a few minutes after cluster startup). Any KDC related errors from the warm-up query can be ignored, since they are expected.

An example query is given below:

   select count(*) from t1 join [SHUFFLE] t2;

t1 should be a large table whose data is spread on all hosts.
t2 can be a table that has a moderate amount of data, i.e. around 200M-300M of data.

Once the service tickets are established between nodes, this will not be a problem until the said service tickets have to be renewed again, thereby incentivizing longer ticket lifetimes. A good balance between security and performance is to have tickets with the lifetime of at least a day.

A secondary workaround is to scale the KDC (e.g. with secondary KDC nodes). If KDC’s are managed by Cloudera Manager, please follow the documentation found here.

All rights belong to: Cloudera, Inc. 395 Page Mill Rd, 3rd Floor · Palo Alto, CA 94306