Complete and Automated Security Experience Using Resmo and Trivy

Security is a critical aspect of both software development and product usage in today's world and should be given the attention it deserves to protect any critical data and maintain trust between users and products. As Resmo, we believe that ensuring your environment's security requires a combination of solid identification of any security issue and tracking changes on your system.

One of the most effective ways to identify and mitigate vulnerabilities is through the use of software bill of materials (SBOM) scanning. An SBOM is a detailed inventory of all the software components and dependencies used in a particular application or system. By scanning an SBOM, organizations can identify any known vulnerabilities within the software components and take steps to remediate them.

There have been several high-profile software vulnerabilities that have been discovered in recent years, such as OpenSSL Heartbleed and Log4Shell. These vulnerabilities have had significant impacts on organizations, as they have been exploited by attackers to gain unauthorized access to sensitive information. Quickly identifying vulnerable software deployed among your infrastructure is the first and most crucial step in order to mitigate possible exploits.

Trivy and Resmo Integration

To help with quickly identifying the deployed software and dependencies, we have developed an integration with Trivy, an open-source vulnerability and misconfiguration scanner developed by Aqua Security. Trivy is designed to be fast and easy to use and can be integrated into the development workflow to identify and fix vulnerabilities early in the development process. Trivy can scan a variety of targets, including container images and repositories.

At Resmo, we use the CycloneDX Format for ingesting Software Bill of Materials (SBOM) files. By storing these files in a queryable state using our SQL engine, users are able to easily associate the actual software that is running with any identified vulnerabilities. This allows users to prioritize remediation actions based on various criteria such as severity and risk by making use of the information Resmo already collects from more than 60 integrations. The flexibility of SQL will help you to prioritize your actions.

Example CyloneDX File in JSON Format
Example CyloneDX File in JSON Format

To ensure continuous coverage of deployment, we use Trivy to scan for vulnerabilities both at the time of SBOM ingestion and periodically. This allows us to detect and address vulnerabilities as soon as they are discovered. With this approach, users can be confident that their systems are secure and that any vulnerabilities are identified and addressed in a timely manner.

The details of Trivy integration are in our documentation.

Balancing Security and Agility with Vulnerability Management

Blocking deployments at the Continuous Integration (CI) stage due to vulnerabilities can be an effective way to ensure that only secure software is deployed to production. However, it also has an impact on the agility of the development process.

On the one hand, blocking deployments at the CI stage can prevent the deployment of vulnerable software and help to mitigate the risk of security breaches. This can be especially important for organizations that handle sensitive data or are subject to strict regulatory requirements.

On the other hand, blocking deployments at the CI stage can slow down the development process and impede the ability to quickly deliver new features and updates. This can be particularly challenging for organizations that operate in fast-paced or highly competitive industries.

To balance the need for security and agility, you should use a risk-based approach to vulnerability management. This can involve prioritizing vulnerabilities based on their severity and the likelihood of exploitation and allowing deployments to proceed if the risk is deemed acceptable.

For instance, you can allow new deployments without any new critical vulnerability to be deployed to the production environments, and the rest of the vulnerabilities can be given a due date based on severities, implementing an SLA. An example SLA in an organization might be as follows:

  • Critical
  • Block deployment except for hotfixes
  • 1-day after discovery if discovered after deployment
  • High - resolved in 2 weeks
  • Medium-  in 4 weeks
  • Low - in 6 weeks

Also, it's best to remember that just because vulnerable software or a library is present, it does not mean it's an immediate threat. As with everything in computing and life, the actual answer is always "it depends." As an example, a Critical sudo vulnerability would probably mean nothing because the container is already mapped to a non-privileged user, and sudoers is not set up.

Another example, a Python vulnerability might be meaningless to a Java-based container. However, it's also best practice to reduce your attack surface as much as possible by removing unused dependencies and components in software and container images, to reduce both noises in the alerts and any future actual vulnerabilities that might actually hurt you in other ways.

Security and vulnerability scanning should be more than just a compliance checkbox item. If not designed properly, you'd get bombarded with alerts for vulnerabilities that are not actually important, cause alert fatigue among developers and the security team, and actually miss the important ones when they occur.

If you have alert fatigue from your security alerts, we offer a free 30-minute consultancy!

Docker Image Names and Tags are mutable

One thing I resent from the beginning of Docker is that container images are hard to identify because image names and tags are not immutable, meaning they can change over time. For instance, there is java:17 image which is actually java:17.0.5. This can make it difficult to track the specific version of a container that is running in an environment.

To address this issue, we make use of image IDs, sha256, in vulnerability scanning instead of image names and tags at Resmo. Image IDs are unique and immutable, meaning they will not change over time. This helps to mitigate the risk of confusion and missing vulnerabilities and ensure that any vulnerabilities that are discovered can be accurately tracked and addressed. Most tools overlook this fact, and you need to be aware of it.

Example: A vulnerable Go-based repo deployed to Kubernetes Cluster

To illustrate how Trivy integration works, we prepared a sample, intentionally vulnerable repository. The code is written in Go, using the client-go library, with an old version. The base image is also based on Debian 9, which is quite old and has critical vulnerabilities. The repository is available on GitHub.

With this example, we will show how we capture SBOM in build time and collect running container images from our Kubernetes cluster.

Our example is based on running a local Minikube cluster but would work with any Kubernetes cluster with an outbound internet connection. Assuming you have created a free Resmo account and configured Kubernetes integration and Trivy Integration, we can move to the next steps to build our Docker image and generate an SBOM file in CycloneDX format, and upload it to Resmo API.


$ eval $(minkube docker-env)
// Build the image and write the Image ID to file $(imgId1) and deploy to Kubernetes cluster
$ docker build --iidfile=imgId1 -t "pod-watcher:1.0" .
$ kubectl apply -f manifest.yml

// Generate a SBOM in the Cyclonedx format of the resulting image with Trivy, and upload it to Resmo
$ trivy image --format cyclonedx $(cat imgId1) > step1.json

$ CUSTOMER_DOMAIN="acme"
$ INGEST_KEY="7c3d9dbe-48ec-411a-92ff-f159cf2f1473" # Available on the Resmo UI in Trivy Integration
$ COMPONENT="trivy-example-project"

$ curl --request POST \
  --compressed \
  --url "https://${CUSTOMER_DOMAIN}.resmo.app/integration/trivy/event?componentName=${COMPONENT}" \
  --header 'Content-Type: application/json' \
  --header "X-Ingest-Key: ${INGEST_KEY}" \
  --data "@step1.json"
  

After completing the above steps, hop on to Resmo UI to Explore the menu and Resources submenu and find out that there have been 77 vulnerabilities found by Trivy! It's huge. Note that, Trivy can generate an SBOM from a container image for both OS package dependencies and even compiled Go binaries without even needing the source (if you are not using upx or anything similar).

The support for languages can be found on Trivy documentation. Please note that not all languages support extracting SBOM from binaries, so it might be a good idea to scan both source code and the resulting binaries for best results.

Trivy vulnerability monitoring on Resmo
77 Vulnerabilities have been identified from pod-watcher:1.0 Docker Image.

As we also configured a Kubernetes integration, we automatically map the SBOM components and vulnerabilities to the actual running pods! Before we will show how you can make use of this information in SQL, see how beautiful it looks to map relationships:

pod watcher
You can see the relationship of a Pod with a ReplicaSet, and the associated vulnerabilities.

The good and best part of integrating resource information from various integrations to Resmo is the ability to make use of all information together. It looks nice in a graph, but with Resmo, you can also write comprehensive SQL queries to identify issues and create alerts on them.

Trivy security query on Resmo
You can see the relationship of a Pod with a ReplicaSet, and the associated vulnerabilities.

After looking at the screen in disappointment with the huge list of vulnerabilities from a very tiny Go program, you do your best by just updating both the Docker image and the dependencies to their latest versions. Normally, blindly updating the dependencies might be problematic in huge projects because some libraries are not backward incompatible or there can be dependency conflicts.

However, for simplicity, we consider our tests comprehensive enough to catch errors, so you can use dependabot or renovate to automatically create PRs to update dependencies for your projects.


-FROM gcr.io/distroless/base-debian9
+FROM gcr.io/distroless/base-debian11

require (
-       k8s.io/apimachinery v0.18.0
-       k8s.io/client-go v0.18.0
+       k8s.io/apimachinery v0.26.1
+       k8s.io/client-go v0.26.1
)


After updating the dependencies, building the docker image, and deploying to our Kubernetes cluster, we send the new SBOM to Resmo API. Notice there are no more Critical or High severity vulnerabilities!

Trivy vulnerability tracking on Resmo

Kubernetes is just an example, and there are other environments you can deploy containers, like AWS Fargate, AWS Lambda, Google Cloud Run, Azure, Fly.io, Heroku, and others! Luckily Resmo supports them all, and with Container abstraction, you don't need to write queries for each container provider.

Conclusion

The integration of Resmo and Trivy allows organizations to gain a comprehensive view of their systems and the vulnerabilities associated with them. The ability to prioritize remediation actions based on severity and risk, along with the flexibility of SQL queries, enables organizations to effectively manage vulnerabilities and minimize their risk. Furthermore, the continuous monitoring and alerting of new vulnerabilities enable organizations to take proactive measures to address vulnerabilities before they can be exploited.

Continue Reading