- Published on
- // 15 min read
Getting started with RHACS Scanner V4
- Authors
- Name
- Shane Boulden
- @shaneboulden
Red Hat Advanced Cluster Security for Kubernetes (RHACS) v4.4 is now out, and includes a really interesting new feature in technology preview - Scanner V4.
In this article I'll take a closer look at Scanner V4, and look at how it differs from the existing StackRox Scanner. Let's dive in!
Exploring Scanner V4
The RHACS Scanner V4 is based on ClaireCore. ClairCore is the engine that sits behind the Clair v4 scanner, the latest iteration of the Clair container scanner, originally created by CoreOS.
Clair scans each container layer and provides a notification of vulnerabilities that may be a threat, based on the Common Vulnerabilities and Exposures database (CVE) and similar databases from Red Hat, Ubuntu, and Debian. Since layers can be shared between many containers, introspection is vital to build an inventory of packages and match that against known CVEs.
Clair v4 utilizes the ClairCore library as its engine for examining contents and reporting vulnerabilities. At a high level you can consider Clair a service wrapper to the functionality provided in the ClairCore library. If you want to read more about ClairCore there's an entire book dedicated to the topic.
You can see an example here showing how Clair v4 essentially provides a service wrapper around the ClairCore engine.
In the same way, StackRox (RHACS) also provides a service wrapper around the ClairCore engine. This means that RHACS now provides two scanners - the colloquially named "StackRox Scanner" (based on Clair v2) and Scanner V4, based on ClairCore.
The "StackRox Scanner" supported a number of programming languages and base image OS flavours, shown in the following tables.
Supported base images (StackRox Scanner)
| Distribution | Version |
|---|---|
| Alpine Linux | alpine:3.2,alpine:3.3, alpine:3.4, alpine:3.5, alpine:3.6, alpine:3.7, alpine:3.8, alpine:3.9, alpine:3.10, alpine:3.11, alpine:3.12, alpine:3.13, alpine:3.14, alpine:3.15, alpine:3.16, alpine:3.17, alpine:3.18, alpine:edge |
| Amazon Linux | amzn:2018.03, amzn:2 |
| CentOS | centos:6, centos:7, centos:8 |
| Debian | ebian:10, debian:11, debian:12, debian:unstable, distroless |
| Red Hat Enterprise Linux (RHEL) | rhel:6, rhel:7, rhel:8, rhel:9 |
| Ubuntu | ubuntu:14.04, ubuntu:16.04, ubuntu:18.04, ubuntu:20.04, ubuntu:21.04, ubuntu:21.10, ubuntu:22.04, ubuntu:22.10, ubuntu:23.04, ubuntu:23.10. The following vulnerability sources are not updated by the vendor: ubuntu:12.04, ubuntu:12.10, ubuntu:13.04, ubuntu:14.10, ubuntu:15.04, ubuntu::15.10, ubuntu::16.10, ubuntu:17.04, ubuntu:17.10, ubuntu:18.10, ubuntu:19.04, ubuntu:19.10, ubuntu:20.1 |
Supported programming languages (StackRox Scanner)
| Language | Artifact |
|---|---|
| Java | JAR / WAR / EAR |
| JavaScript | Node.js / npm package.json |
| Python | egg and wheel formats |
| Ruby | gem |
Because of the move to ClairCore, Scanner V4 introduces new capabilities for image scanning in Red Hat Advanced Cluster Security for Kubernetes:
New supported programming languages: Scanner V4 now supports Go binaries. If the binaries are built with module support (
go.mod) then the dependencies of the binary are also analyzed.New supported operating systems: Scanner V4 introduces support for Amazon Linux 2023, Alpine Linux 3.19, SLES 15, openSUSE Leap 15.1, Photon OS and Oracle Linux.
New CVE sources: Scanner V4 also introduces a new source for vulnerability data, OSV.dev. OSV.dev is an open source vulnerability database created by Google Security in 2021, with the goal of making it easier to track CVEs and vulnerabilities across open source packages.
The use of OSV (Open Source Vulnerabilties) is a pretty exciting change. OSV automates a lot of analysis that typically happens behind-the-scenes for open source package vulnerability analysis.
OSV does this by providing both the commits that introduce and fix bugs in the package. If that information is not available, OSV requires providing a reproduction test case and steps to generate an application build, and then it performs bisection to find these commits in an automated fashion. OSV takes care of the rest of the analysis to figure out impacted commit ranges (accounting for cherry picks) and versions/tags.
There's a great diagram here from the Google Security blog on how OSV works.
This means that the supported base images and programming languages table now looks like this with Scanner V4:
Supported base images (Scanner V4)
| Distribution | Version |
|---|---|
| Alpine Linux | alpine:3.2,alpine:3.3, alpine:3.4, alpine:3.5, alpine:3.6, alpine:3.7, alpine:3.8, alpine:3.9, alpine:3.10, alpine:3.11, alpine:3.12, alpine:3.13, alpine:3.14, alpine:3.15, alpine:3.16, alpine:3.17, alpine:3.18, alpine:3.19, alpine:edge |
| Amazon Linux | amzn:2018.03, amzn:2, amzn:2023 |
| CentOS | centos:6, centos:7, centos:8 |
| Debian | ebian:10, debian:11, debian:12, debian:unstable, distroless |
| Oracle Linux | versions 5-9 |
| Photon OS | 1.0, 2.0, 3.0 |
| Red Hat Enterprise Linux (RHEL) | rhel:6, rhel:7, rhel:8, rhel:9 |
| SUSE | SLES 11, 12, 15; openSUSE Leap 42.3, 15.0, 15.1; SUSE Linux |
| Ubuntu | ubuntu:14.04, ubuntu:16.04, ubuntu:18.04, ubuntu:20.04, ubuntu:21.04, ubuntu:21.10, ubuntu:22.04, ubuntu:22.10, ubuntu:23.04, ubuntu:23.10. The following vulnerability sources are not updated by the vendor: ubuntu:12.04, ubuntu:12.10, ubuntu:13.04, ubuntu:14.10, ubuntu:15.04, ubuntu::15.10, ubuntu::16.10, ubuntu:17.04, ubuntu:17.10, ubuntu:18.10, ubuntu:19.04, ubuntu:19.10, ubuntu:20.1 |
Supported programming languages (Scanner V4)
| Language | Artifact |
|---|---|
| Go | Binaries: The standard library version used to build the binary is analyzed. If the binaries are built with module support (go.mod), then the dependencies are also analyzed. |
| Java | JAR / WAR / EAR |
| JavaScript | Node.js / npm package.json |
| Python | egg and wheel formats |
| Ruby | gem |
Getting started with Scanner V4
The Red Hat Advanced Cluster Security for Kubernetes (RHACS) operator exposes settings for Scanner V4, which you can use to enable this alongside the StackRox Scanner (based on Clair v2).
What is a Red Hat technology preview?
Scanner V4 is a Red Hat Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
In other words, I wouldn't recommend using this in production. Try out Scanner V4 in a non-production environment while it's a Technology Preview.
You can find the complete list of Scanner V4 configuration items documented here.
When you deploy a RHACS Central v4.4 instance it will configure settings for Scanner v4:
scannerV4:
db:
persistence:
persistentVolumeClaim:
claimName: scanner-v4-db
indexer:
scaling:
autoScaling: Enabled
maxReplicas: 5
minReplicas: 2
replicas: 3
matcher:
scaling:
autoScaling: Enabled
maxReplicas: 5
minReplicas: 2
replicas: 3
scannerComponent: Default
However, Scanner V4 is not yet enabled, as the scannerComponent is set to Default. Changing this to Enabled will deploy the scanner v4 components, and configure RHACS Central to use this as the default.
scannerV4:
db:
persistence:
persistentVolumeClaim:
claimName: scanner-v4-db
indexer:
scaling:
autoScaling: Enabled
maxReplicas: 5
minReplicas: 2
replicas: 3
matcher:
scaling:
autoScaling: Enabled
maxReplicas: 5
minReplicas: 2
replicas: 3
scannerComponent: Enabled
Once you've made this change, you should see the Scanner v4 components deploying to the cluster.
After enabling the Scanner V4 component you may find that some of the pods are stuck in a Pending state. This is likely due to insufficient memory / CPU for the Scanner v4 indexers, which you can see below:
There's a couple of ways to resolve this:
Scaling nodes vertically and horizontally
You can scale OpenShift nodes vertically by updating the machineset. Here I've changed the worker nodes from the AWS m6i.xlarge size to m6i.2xlarge.
$ oc get machineset cluster1-fwbxz-worker-ap-southeast-2c -o json | jq -r '.spec.template.spec.providerSpec.value.instanceType'
m6i.2xlarge
Once updated, if you scale the machineset to zero-replicas and back again the changes will take effect.
$ oc scale machineset cluster1-fwbxz-worker-ap-southeast-2c --replicas 0
You can also scale the nodes horizontally in the same step:
oc scale machineset cluster1-fwbxz-worker-ap-southeast-2c --replicas 2
Configuring Scanner V4 resource requests
You can also resolve this issue by updating the Scanner V4 resource requests.
The default resource requests are created by the RHACS Central operator, and are pretty heavy. You can see an example here for the scanner-v4-indexer containers:
containers:
- resources:
limits:
cpu: '2'
memory: 3Gi
requests:
cpu: '1'
memory: 1500Mi
The RHACS Central operator exposes a number of settings you can use to tune these resource requests, and deploy Scanner V4 on smaller clusters.
You can see the resources / limits summarised in the following table:
| Parameter | Description |
|---|---|
| scannerV4.db.resources.limits | Use this parameter to override the default resource limits for Scanner V4 DB. |
| scannerV4.db.resources.requests | Use this parameter to override the default resource requests for Scanner V4 DB. |
| scannerV4.indexer.resources.limits | Use this parameter to override the default resource limits for the Scanner V4 Indexer. |
| scannerV4.indexer.resources.requests | Use this parameter to override the default resource requests for the Scanner V4 Indexer. |
| scannerV4.matcher.resources.limits | Use this parameter to override the default resource limits for the Scanner V4 Matcher. |
| scannerV4.matcher.resources.requests | Use this parameter to override the default resource requests for the Scanner V4 Matcher. |
Scanner V4 initialization
Once you've updated either the ScannerV4 configuration or scaled the nodes, you should see all Scanner v4 pods deploy.
It's not quite ready to use though. If you try running an image scan you'll see an error like below:
ERROR: checking image failed after 3 retries: could not check build-time alerts: rpc error: code = Internal desc = image enrichment error: error scanning image: quay.io/argoproj/argocd:v2.9.17 error: scanning "quay.io/argoproj/argocd:v2.9.17" with scanner "Scanner V4": index and scan image report (reference: "quay.io/argoproj/argocd@sha256:42d7cef523701c74c5540d721b58615d8c8526bc1d7ef264df6670dd28c4be81"): get vulns: rpc error: code = FailedPrecondition desc = the matcher is not initialized: initial load for the vulnerability store is in progress
You can also see these errors reporting in the RHACS Central System Health dashboard.
As the error highlights, the Scanner V4 matcher component is still initializing. You can see this in the matcher pod logs:
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"matcher/updater/vuln/Updater.Start","time":"2024-06-10T23:52:17Z","message":"completed update"}
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"matcher/updater/distribution/Updater.Start","time":"2024-06-10T23:52:26Z","message":"vulnerability updater is running, wait 15m0s and try again..."}
You should also be able to see the matcher applying CVE updates using the Red Hat OVAL sources:
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"matcher/updater/distribution/Updater.Start","time":"2024-06-11T00:22:01Z","message":"vulnerability updater is running, wait 15m0s and try again..."}
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"libvuln/OfflineImporter","updater":"RHEL7-rhel-7.2-eus","time":"2024-06-11T00:22:02Z","message":"fingerprint match, skipping"}
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"libvuln/OfflineImporter","updater":"RHEL8-storage-ceph-4-including-unpatched","time":"2024-06-11T00:22:02Z","message":"fingerprint match, skipping"}
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"libvuln/OfflineImporter","updater":"RHEL7-rhel-7.6-tus","time":"2024-06-11T00:22:03Z","message":"fingerprint match, skipping"}
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"libvuln/OfflineImporter","updater":"RHEL6-jboss-ws-5-including-unpatched","time":"2024-06-11T00:22:03Z","message":"fingerprint match, skipping"}
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"libvuln/OfflineImporter","updater":"RHEL9-rhel-9.2-eus","time":"2024-06-11T00:22:05Z","message":"fingerprint match, skipping"}
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"libvuln/OfflineImporter","updater":"RHEL7-satellite-6.8","time":"2024-06-11T00:22:05Z","message":"fingerprint match, skipping"}
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"libvuln/OfflineImporter","updater":"RHEL7-rhel-7.1-eus","time":"2024-06-11T00:22:06Z","message":"fingerprint match, skipping"}
{"level":"info","host":"scanner-v4-matcher-6948d79d87-zmrrj","component":"libvuln/OfflineImporter","updater":"RHEL9-jboss-ws-5","time":"2024-06-11T00:22:06Z","message":"fingerprint match, skipping"}
If we give this about 45 minutes the matcher will have updated its CVE definitions and we can start using it. If the matcher hasn't initialized after 45 mins you may need to bump up the limits for the pods via the RHACS Central CR. Here I've updated the matcher to allow it to use up to 4 cores and 10GB of RAM:
scannerV4:
db:
persistence:
persistentVolumeClaim:
claimName: scanner-v4-db
indexer:
scaling:
autoScaling: Enabled
maxReplicas: 5
minReplicas: 2
replicas: 3
matcher:
resources:
limits:
cpu: 4
memory: 10Gi
scaling:
autoScaling: Enabled
maxReplicas: 5
minReplicas: 2
replicas: 3
scannerComponent: Enabled
Trying out Scanner V4
Scanner V4 is now up-and-running now on this cluster. so let's give it a shot!
One of the limitations with StackRox Scanner (based on Clair v2) was that it couldn't support Go binaries. Let's try out Scanner V4 and make sure we get a result for Go. You can initiate an image scan and policy check with roxctlfor the quay.io/argoproj/argocd:v2.9.17 image:
$ export ROX_API_TOKEN=...
$ roxctl -e "central-acs-central.apps.your-cluster.example.com:443" --insecure-skip-tls-verify image check --image quay.io/argoproj/argocd:v2.9.17
You should get a result like this:
Policy check results for image: quay.io/argoproj/argocd:v2.9.17
(TOTAL: 3, LOW: 1, MEDIUM: 0, HIGH: 1, CRITICAL: 1)
+--------------------------------+----------+--------------+--------------------------------+-------------------------------------------------------------------------------+--------------------------------+
| POLICY | SEVERITY | BREAKS BUILD | DESCRIPTION | VIOLATION | REMEDIATION |
+--------------------------------+----------+--------------+--------------------------------+-------------------------------------------------------------------------------+--------------------------------+
| Rapid Reset: Denial of Service | CRITICAL | - | Alert on deployments with | - CVE-2023-44487 (CVSS | Upgrade vulnerable components |
| Vulnerability in HTTP/2 | | | images containing components | 5.3) (severity Moderate) | or images to the latest |
| Protocol | | | that are susceptible to | found in component | version. |
| | | | a Denial of Service (DoS) | 'google.golang.org/grpc' | |
| | | | vulnerability for HTTP/2 | (version v1.56.2) | |
| | | | servers. | | |
+--------------------------------+----------+--------------+--------------------------------+-------------------------------------------------------------------------------+--------------------------------+
| Fixable Severity at least | HIGH | X | Alert on deployments with | - Fixable CVE-2023-47108 (CVSS 7.5) (severity Important) found in component | Use your package manager to |
| Important | | | fixable vulnerabilities with | 'go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc' | update to a fixed version in |
| | | | a Severity Rating at least | (version v0.42.0), resolved by version 0.46.0 | future builds or speak with |
| | | | Important | | your security team to mitigate |
| | | | | - Fixable CVE-2023-5528 | the vulnerabilities. |
| | | | | (CVSS 8.8) (severity | |
| | | | | Important) found in component | |
| | | | | 'k8s.io/kubernetes' (version | |
| | | | | v1.24.17), resolved by version | |
| | | | | 1.25.16 | |
| | | | | | |
| | | | | - Fixable GHSA-m425-mq94-257g | |
| | | | | (CVSS 7.5) (severity | |
| | | | | Important) found in component | |
| | | | | 'google.golang.org/grpc' | |
| | | | | (version v1.56.2), resolved by | |
| | | | | version 1.56.3 | |
+--------------------------------+----------+--------------+--------------------------------+-------------------------------------------------------------------------------+--------------------------------+
| Ubuntu Package Manager in | LOW | - | Alert on deployments | - Image includes component | Run `dpkg -r --force-all |
| Image | | | with components of the | 'apt' (version 2.4.12) | apt apt-get && dpkg -r |
| | | | Debian/Ubuntu package | | --force-all debconf dpkg` in |
| | | | management system in the | - Image includes | the image build for production |
| | | | image. | component 'dpkg' (version | containers. |
| | | | | 1.21.1ubuntu2.3) | |
+--------------------------------+----------+--------------+--------------------------------+-------------------------------------------------------------------------------+--------------------------------+
We can see that Scanner V4 has detected CVE-2023-44487 inside one of the Go binaries packaged inside this container image. Let's take a closer look at this image in the RHACS UI.
Great! We can see from these images that Scanner V4 was used to analyze the container image, and was able to detect Go components. Importantly, Scanner V4 identified CVEs inside a Go binary packaged inside the container image. It identified that the file /usr/local/bin/argocd packaged inside the container image is vulnerable to CVE-2023-44487, and that this is fixed in version 1.56.3 of the google.golang.org/grpc dependency.
Wrapping up
In this article I took a closer look at the new Scanner V4 introduced with Red Hat Advanced Cluster Security for Kubernetes (RHACS) v4.4. Scanner V4 is based on ClairCore and introduces support for Go and other programming languages.
Let me know in the comments how you go trying it out for yourself!