Cloudera Technical Service Bulletin 2017-237 (TSB): Keystore password for the Spark History Server not properly secured

Title: Keystore password for the Spark History Server not properly secured 

Products affected: Cloudera Manager

Releases affected: 5.11.0

Users affected: All users with TLS enabled for the Spark History Server.

Date/time of detection: April 18, 2017

Severity (Low/Medium/High): Medium

Impact: The keystore password for the Spark History Server is exposed in a world-readable file on the machine running the Spark History Server. The keystore file itself is not exposed.

The password is also visible in the Cloudera Manager UI, by looking at the Spark History Server process’s configuration files.

CVE: CVE-2017-9326

Immediate action required: Upgrade to Cloudera Manager 5.11.1.

Addressed in release/refresh/patch: 5.11.1 or higher.

For the latest update on this issue see the corresponding Knowledge article:

TSB 2017-237: Keystore password for the Spark History Server not properly secured

All rights belong to: Cloudera, Inc. 395 Page Mill Rd, 3rd Floor · Palo Alto, CA 94306 USA

Cloudera Technical Service Bulletin 2017-242 (TSB): CDH Services crash with Java Virtual Machine error after application of Linux Security Kernel Patch for CVE-2017-1000364

CDH Services crash with Java Virtual Machine error after application of Linux Security Kernel Patch for CVE-2017-1000364

Cloudera has received a number of customer cases where CDH services using the JSVC set of libraries crash with a Java Virtual Machine error in CDH clusters after applying a recent Kernel Security Patch. The list below contains the announcements about the patch for a number of distributions.

• RHSA-2017-1484 (RHEL 7). RedHat has fixed this problem for RHEL 7, marked this patch as outdated and superseded by RHSA-2017-1674.
• RHSA-2017-1486 (RHEL 6). RedHat has fixed this problem for RHEL 6, marked this as outdated and superseded by RHSA-2017-1723.
• ELSA-2017-1484 (Oracle Linux 7). Oracle has also added the fix for Oracle Linux 7 in ELBA-2017-1674.
• ELSA-2017-1486 (Oracle Linux 6). Oracle has fixed this problem in ELSA-2017-1723.
• USN-3335-1 (Ubuntu 14.04). Fixed in USN-3338-2.
• CVE-2017-1000364 (SUSE). SUSE has also fixed this problem and the patch names are included in this same advisory.

We have been able to reproduce Cloudera services not starting after the application of the original patch.

Products affected: Impala, HDFS DataNodes, HDFS NFS Gateway

Releases affected: Any CDH release that runs on RHEL, CentOS, Oracle Linux, SUSE Linux and Ubuntu where this Security Kernel Patch is applied.

Users affected: All Impala & HDFS users

Severity (Low/Medium/High): High

Impact:

After applying the patch the service's daemon crashes with an error similar to the following:

----
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0x7) at pc=0x00007fe91ef6cebc, pid=30321, tid=0x00007fe930c67700
> #
> ----

Immediate action required:

Apply a kernel update containing the fix for your Operating System.

If you are unable to apply a kernel update, the Java thread stack size may be set for the affected service(s). The Java Configuration Options will need to include the following value: "-Xss1280k". For roles that do not have a parameter for Java Configuration Options in Cloudera Manager, the value will need to be set in the Environment Advanced Configuration Snippet (Safety Valve).

Example for adding the value in the Java Configuration Options:

1. In Cloudera Manager, Navigate to Impala > Configuration > Category (Advanced)
2. Type “Java Configuration Options” in the configuration search box and locate the required field
3. Add “-Xss1280k” to the field
4. Save Changes and restart the affected roles

Example for adding the value in the Advanced Configuration Snippet (Safety Valve):

1. In Cloudera Manager, Navigate to Impala > Configuration > Category (Advanced)
2. Type “Advanced Configuration Snippet” in the configuration search box and locate the required field
3. Add “JAVA_TOOL_OPTIONS=-Xss1280k” to the field.
4. Save Changes and restart the affected roles

The parameters to change for all affected services are shown in the table below.

Service	Parameter
HDFS DataNodes	Java Configuration Options for DataNode
HDFS NFS Gateway	Java Configuration Options for NFS Gateway
Impala	Impala Service Environment Advanced Configuration Snippet (Safety Valve)
Impala	Java Configuration Options for Catalog Server
Impala	Impala Daemon Environment Advanced Configuration Snippet

For the latest update on this issue see the corresponding Knowledge article:

TSB 2017-242: CDH Services crash with Java Virtual Machine error after application of Linux Security Kernel Patch for CVE-2017-1000364

All rights belong to: Cloudera, Inc. 395 Page Mill Rd, 3rd Floor · Palo Alto, CA 94306 USA

Cloudera Technical Service Bulletin 2017-236 (TSB): Sample solrconfig.xml file for enabling Solr/Sentry Authorization is missing critical attribute

Title: Sample solrconfig.xml file for enabling Solr/Sentry Authorization is missing critical attribute 

The solrconfig.xml.secure sample configuration which was provided with CDH, if used to create solrconfig.xml, does not enforce Sentry authorization on the request URI /update/json/docs because it is missing a necessary attribute.

Products affected: Solr (if Sentry enabled)

Releases affected:
CDH 5.8 and lower
CDH 5.9.2 and lower
CDH 5.10.1 and lower
CDH 5.11.1 and lower

Users affected: Those who are using Sentry authorization with Cloudera Search and who have used the provided sample configuration and have not specified the below attributes in their solrconfig.xml file.

Date/time of detection: May 18, 2017

Detected by: István Farkas, Hrishikesh Gadre

Severity (Low/Medium/High): High

Impact: Unauthorized users using the request URI /update/json/docs may insert, update, or delete documents.

CVE: CVE-2017-9325

Immediate action required: 

Every solrconfig.xml of a collection protected by Sentry should be updated in Zookeeper.
Line of:

<updateRequestProcessorChain name="updateIndexAuthorization">
Must be replaced with 

<updateRequestProcessorChain name="updateIndexAuthorization" default="true">

After updating the configuration in Zookeeper, the collections must be reloaded.

Addressed in release/refresh/patch:
The following releases will contain the fixed sample configuration file.
CDH 5.9.3 and higher
CDH 5.10.2 and higher
CDH 5.11.2 and higher
CDH 5.12.0 and higher

Upgrading will only correct the sample configuration file. The fix mentioned above will still need to be applied on the affected cluster.

For the latest update on this issue see the corresponding Knowledge article: 

TSB 2017-236: Sample solrconfig.xml file for enabling Solr/Sentry Authorization is missing critical attribute

All rights belong to: Cloudera, Inc. 395 Page Mill Rd, 3rd Floor · Palo Alto, CA 94306 USA

Cloudera Technical Service Bulletin 2017-226 (TSB): Impala Statestore exposes plaintext data with SSL TLS enabled

Title: Impala Statestore exposes plaintext data with SSL TLS enabled 

During a security analysis, Cloudera found that despite TLS being enabled for “internal” Impala ports, the Statestore thrift port did not actually use TLS. This gap would allow an adversary with network access to eavesdrop and potentially modify the packets going to and coming from that port.

Products affected: Impala

Releases affected:
CDH 5.7 and lower
CDH 5.8.0, 5.8.1, 5.8.2, 5.8.3, 5.8.4
CDH 5.9.0, 5.9.1, 5.9.2
CDH 5.10.0, 5.10.1
CDH 5.11.0

Users affected: Deployments that use “internal” TLS (TLS between Impala daemons).

Date/time of detection: April 27, 2017

Detected by: Cloudera

Severity (Low/Medium/High): High

Impact: Data on the wire may be intercepted by a malicious server.

CVE: CVE-2017-5652

Immediate action required: Affected customers should upgrade to latest maintenance version with the fix.

Addressed in release/refresh/patch:
CDH 5.8.5, CDH 5.9.3, CDH 5.10.2, CDH 5.11.1, CDH 5.12.0

For the latest update on this issue see the corresponding Knowledge article: 

TSB 2017-226: Impala Statestore exposes plaintext data with SSL TLS enabled

All rights belong to: Cloudera, Inc. 395 Page Mill Rd, 3rd Floor · Palo Alto, CA 94306 USA

September Cloudera Customer Advisory

Key Distribution Center (KDC) may be overwhelmed on large Kerberized Impala clusters

After cluster startup, when Impala runs a query that shuffles data amongst all nodes or a large subset of nodes in a Kerberos-secured cluster, every node will need to acquire a Ticket Granting Server (TGS) for every other node. In a cluster of hundreds of nodes or more, this can overwhelm the KDC, and queries can exit with an error ("Could not contact KDC for realm").

For example, queries that could fail with kerberos errors just after cluster startup are those that involve one or more joins on large tables that have data spread out over the entire cluster.

Users affected:

Affected users are those running Kerberized Impala clusters with 100 nodes or up.

Impact:

Failed queries for a few minutes immediately after cluster startup.

Action required:

A simple workaround is to run a warm-up query a few times until all the tickets are established (which can take a few minutes after cluster startup). Any KDC related errors from the warm-up query can be ignored, since they are expected.

An example query is given below:

   select count(*) from t1 join [SHUFFLE] t2;

t1 should be a large table whose data is spread on all hosts.
t2 can be a table that has a moderate amount of data, i.e. around 200M-300M of data.

Once the service tickets are established between nodes, this will not be a problem until the said service tickets have to be renewed again, thereby incentivizing longer ticket lifetimes. A good balance between security and performance is to have tickets with the lifetime of at least a day.

A secondary workaround is to scale the KDC (e.g. with secondary KDC nodes). If KDC’s are managed by Cloudera Manager, please follow the documentation found here.

All rights belong to: Cloudera, Inc. 395 Page Mill Rd, 3rd Floor · Palo Alto, CA 94306

Cloudera Technical Service Bulletin 2017-249: Unsafe deserialization in Apache Spark launcher API

Title: Unsafe deserialization in Apache Spark launcher API 

In Apache Spark 1.6.0 until 2.1.1, the launcher API performs unsafe deserialization of data received by its socket. This makes applications launched programmatically using the SparkLauncher#startApplication() API potentially vulnerable to arbitrary code execution by an attacker with access to any user account on the local machine. It does not affect apps run by spark-submit or spark-shell. The attacker would be able to execute code as the user that ran the Spark application. Users are encouraged to update to Spark version 2.2.0 or later.

Products affected: Cloudera's distribution of Apache Spark and Spark in CDH.

Releases affected: 

• CDH: 5.9.0-5.9.2, 5.10.0-5.10.1, 5.11.0-5.11.1
• Cloudera's Distribution of Apache Spark: 2.0 Release 1, 2.0 Release 2, 2.1 Release 1

Users affected: All

Date/time of detection: June 1, 2017

Detected by: Aditya Sharad (Semmle)

Severity (Low/Medium/High): Medium

Impact: Privilege escalation to the user who ran the Spark application

CVE: CVE-2017-12612

Immediate action required: 

• Spark 1.x users: update to CDH 5.9.3, 5.10.2, 5.11.2, or 5.12.0 or later
• Spark 2.x users: update to Cloudera's Distribution of Apache Spark 2.2 Release 1 or later
• Or, discontinue use of the programmatic launcher API

Addressed in release/refresh/patch: 

• CDH: 5.9.3, 5.10.2, 5.11.2, 5.12.0
• Cloudera's Distribution of Apache Spark: 2.2 Release 1

For the latest update on this issue see the corresponding Knowledge article: 

TSB 2017-249: Unsafe deserialization in Apache Spark launcher API *

All rights belong to: Cloudera, Inc. 395 Page Mill Rd, 3rd Floor · Palo Alto, CA 94306