Docker Image Scanner for Vulnerabilities With Clair

I’m gonna tell you how you can add a step in your CI pipeline to check if the Docker image you’re build contains vulnerabilities or not.

Pre-requisites

I assume you’ve Docker installed on your system. We’re gonna use Jenkins but is optional.

Clair

Clair is an open source project for the static analysis of vulnerabilities in appc and docker containers. It’s been developed by CoreOS.

Vulnerabilities Database

Clair relay on Postgres to keep the database. We’ve two different options here. We can create a new Postgres database which will be initializes when Clair starts or we can use a database already provisioned. 

If you want to try the first option keep in mind it takes a while to feed the database (10~15 minutes depends on network).

We’re using the second option here. Go to a terminal and run this command:

$ docker network create ci
$ docker volume create --name clair-postgres
$ docker run --detach \
   --name clair-postgres \
   --publish 5432:5432 \
   --net ci \
   --volume clair-postgres:/var/lib/postgresql/data \
   arminc/clair-db:latest

We just have created a new network to Clair can resolve the database by its name, a volume and a Docker container. After a couple seconds we can check the database is up and ready.

$ docker logs --tail 1 clair-postgres 
2019-05-15 13:36:00.068 UTC [1] LOG:  database system is ready to accept connections

Clair Service

Now it’s time to run the service. But first, a little configuration is needed. We must set the database for Clair in the config file. Run the following command:

$ curl --silent https://raw.githubusercontent.com/nordri/config-files/master/clair/config-clair.yaml | sed "s/POSTGRES_NAME/clair-postgres/" > config.yaml

We’re changing the string POSTGRES_NAME by clair-postgres which is the name we’ve gave to the Postgres container. Now, we can launch the Clair container by running:

$ docker run --detach \
  --name clair \
  --net ci \
  --publish 6060:6060 \
  --publish 6061:6061 \
  --volume ${PWD}/config.yaml:/config/config.yaml \
  quay.io/coreos/clair:latest -config /config/config.yaml

And that is.

Launch an scanner

It’s time to check everything is working properly. To run the Clair scanner I’ve just built a container with the latest release. You can found it here and then build the image using this Dockerfile.

FROM debian:jessie

COPY clair-scanner_linux_amd64 /clair-scanner
RUN chmod +x /clair-scanner

I already did so you can use this image: nordri/clair-scanner.

Let’s check some images now. One of Clair limitations is it cannot scan remote images. All images must be locals. To scan an image just launch

$ docker run -ti \
  --rm \
  --name clair-scanner \
  --net ci \
  -v /var/run/docker.sock:/var/run/docker.sock \
  nordri/clair-scanner:latest /bin/bash

Now we’re inside the container and we’re able to launch the scanner:

# export IP=$(ip r | tail -n1 | awk '{ print $9 }')
# /clair-scanner --ip ${IP} --clair=http://clair:6060 debian:jessie

If we launch that as it, we’ll see an endless list of vulnerabilities, which is more noise than something useful. So, we can filter by severity using this flag:

-t, --threshold="Unknown"             CVE severity threshold. Valid values; 'Defcon1', 'Critical', 'High', 'Medium', 'Low', 'Negligible', 'Unknown'

We can choose only check for critical or higher and then the list will be much more gentle because all the vulnerabilities will be treat as approved, and, which is much more interested, the command will exit with 0. Then we can add this to a CI pipeline.

Jenkins Integration

If you’ve Jenkins running in your infrastructure, you’ll be probably interested in check the images you’re delivering to your customers.

This simple Jenkinsfile can solve the problem

node {

   docker.image('docker').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
      
       stage ('Checkout') {
         checkout scm
       }

       stage ('Build Docker image') {
           // Build docker image
           // docker build... DOCKER_IMAGE
       }
   }

   docker.image('nordri/clair-scanner').inside('--net ci') {

       stage ('Security scanner') {
           sh '''
             IP=$(ip r | tail -n1 | awk '{ print $9 }')
             /clair-scanner --ip ${IP} --clair=http://clair:6060 --threshold="Critical" DOCKER_IMAGE
           '''
       }
   }
}

As we can see, we’re using a container to build the image and another one to scan that image for vulnerabilities, we set the threshold to Critical so only really big problem will made the pipeline to fail.

References

Posted in Docker, Jenkins, Seguridad | Tagged , , , | Leave a comment

Recovering Untagged Images From ECR

Happened today that our CI system created a new Docker Image with a tag that was supposed to be used in production. So, we faced with a Docker Image tagged like the production one but it wasn’t the production one at all and, on the other hand, the Docker Image from production was there untagged. What should we do?

Actually, the process is quite simple if you know how to cope with that. First we have to log into the ECR Service.

$ $(aws ecr get-login --no-include-email --region eu-west-1)

Login Succeeded

Then locate the sha256 and download the manifest.

$ MANIFEST=$(aws ecr batch-get-image --repository-name sandbox --image-ids imageDigest=sha256:e226e9aaa12beb32bfe65c571cb60605b2de13338866bc832bba0e39f6819365 --query 'images[].imageManifest' --output text)

You can find the sha256 in the Image list, copy and paste the one belongs to the untagged image.

The manifest is basically all the layers which conform the Docker Image.

Then tag the image

$ aws ecr put-image --repository-name sandbox --image-tag backup --image-manifest "$MANIFEST"

Now, you are able to pull the new tagged image

docker pull myarn.amazonaws.com/sandbox:backup

Reference:

Posted in AWS, Docker, Tips | Tagged , , | Leave a comment

Azure Batch: Task in Containers

In today’s post I’ll be talking about how to send tasks to Microsoft Azure Batch able to run in containers. The task I want to solve in this example is calculating whether a number is prime or not. This Python code does the work for us. Then, I’ve write a Dockerfile adding the piece of code to the image. Now, we’re able to run the script from a Docker container.

Let’s move forward to Azure Batch, you need to create a Docker registry where you’ll push the Docker image and an Azure Batch account.

Docker Registry

Login into your Azure account and move to All resources, click Add and look for Container Registry. Then click in Create. Fill up the information with the name of the registry, the resource group, choose a location closer to you, enable Admin user to be able to push to the repo and the SKU (choose standard here).

Then Create

In a few seconds the registry will be ready. So go to the Dashboard and click on the registry name (the one you chose before). Click in Settings -> Access keys. Here are the credentials you’ll need to manage the registry.

Batch Account

From the All resources look for Batch service. Fill up the information with the Account name and Location, Subscription and Resource group should be ready.

Click Review + create and then Create. In a few seconds the service should be ready.

Building the Container

Clone the repo

git clone https://github.com/nordri/microsoft-azure

and build and push the container

cd batch-containers
docker build -t YOUR_REGISTRY_SERVER/YOUR_REGISTRY_NAME/YOUR_IMAGE_NAME .
# for example:
docker build -t pythonrepo.azurecr.io/pythonrepo/is_prime:latest .
# Check the image works:
docker run -ti --rm pythonrepo.azurecr.io/pythonrepo/is_prime python is_prime.py 7856
The number: 7856 is not prime
docker run -ti --rm pythonrepo.azurecr.io/pythonrepo/is_prime python is_prime.py 2237
The number: 2237 is prime
# login first
docker login pythonrepo.azurecr.io
Username: pythonrepo
Password: 
# Push
docker push pythonrepo.azurecr.io/pythonrepo/is_prime

Azure Batch

Now it’s time to send the task to Azure Batch. To do this, I’ve worked over this Python script. This script creates a pool, a job and three tasks to upload files to Azure Storage. So, I’ve made some modifications to fit it to my needs.

Creating the Pool

I need my pool to be created using instances able to run containers

...
def create_pool(batch_service_client, pool_id):
    print('Creating pool [{}]...'.format(pool_id))

    image_ref_to_use = batch.models.ImageReference(
        publisher='microsoft-azure-batch',
        offer='ubuntu-server-container',
        sku='16-04-lts',
        version='latest'
    )

    # Specify a container registry
    # We got the credentials from config.py
    containerRegistry = batchmodels.ContainerRegistry(
        user_name=config._REGISTRY_USER_NAME, 
        password=config._REGISTRY_PASSWORD, 
        registry_server=config._REGISTRY_SERVER
    )

    # The instance will pull the images defined here
    container_conf = batchmodels.ContainerConfiguration(
        container_image_names=[config._DOCKER_IMAGE],
        container_registries=[containerRegistry]
    )

    new_pool = batch.models.PoolAddParameter(
        id=pool_id,
        virtual_machine_configuration=batchmodels.VirtualMachineConfiguration(
            image_reference=image_ref_to_use,
            container_configuration=container_conf,
            node_agent_sku_id='batch.node.ubuntu 16.04'),
        vm_size='STANDARD_A2',
        target_dedicated_nodes=1
    )

    batch_service_client.pool.add(new_pool)
...

The key is the ImageReference where we set the instances to run with an OS able to run Docker. You must set the registry credentials and the default Docker image that will be pulled when the instance boots.

Creating the Task

I’ve also changed the Task for the same reason. This task is ready to launch a container in the instance.

...
def add_tasks(batch_service_client, job_id, task_id, number_to_test):
    print('Adding tasks to job [{}]...'.format(job_id))

    # This is the user who run the command inside the container.
    # An unprivileged one
    user = batchmodels.AutoUserSpecification(
        scope=batchmodels.AutoUserScope.task,
        elevation_level=batchmodels.ElevationLevel.non_admin
    )

    # This is the docker image we want to run
    task_container_settings = batchmodels.TaskContainerSettings(
        image_name=config._DOCKER_IMAGE,
        container_run_options='--rm'
    )
    
    # The container needs this argument to be executed
    task = batchmodels.TaskAddParameter(
        id=task_id,
        command_line='python /is_prime.py ' + str(number_to_test),
        container_settings=task_container_settings,
        user_identity=batchmodels.UserIdentity(auto_user=user)
    )

    batch_service_client.task.add(job_id, task)
...

You can see how I’ve defined the user inside the container as a non admin user. The Docker image we want to use and the arguments we need to pass in the command line, remember we launch the container like:

docker ... imagename python /is_prime.py number

Launching the Script

Configure

In order to launch the script we need to fill up some configuration. Open the config.py file and write all the credentials needed. Remember, all the credentials are in the Access keys section.

Installing Dependencies

You need Azure Python SDK installed to run the script.

pip install -r requirements.txt

Let’s go

Now we’re ready to launch the script:

python batch_containers.py 89
Sample start: 2018-11-10 10:11:11

Creating pool [ContainersPool]...
No handlers could be found for logger "msrest.pipeline.requests"
Creating job [ContainersJob]...
Adding tasks to job [ContainersJob]...
Monitoring all tasks for 'Completed' state, timeout in 0:30:00.....................................................................................................................................................................
  Success! All tasks reached the 'Completed' state within the specified timeout period.
Printing task output...
Task: ContainersTask
Standard output:
The number: 89 is prime

Standard error:


Sample end: 2018-11-10 10:14:31
Elapsed time: 0:03:20

Delete job? [Y/n] y
Delete pool? [Y/n] y

Press ENTER to exit...

If there’s a problem with the script we’ll see the error on stderr.txt.

Sample start: 2018-11-10 11:29:56

Creating pool [ContainersPool]...
No handlers could be found for logger "msrest.pipeline.requests"
Creating job [ContainersJob]...
Adding tasks to job [ContainersJob]...
Monitoring all tasks for 'Completed' state, timeout in 0:30:00..................................................................................................................................................................
  Success! All tasks reached the 'Completed' state within the specified timeout period.
Printing task output...
Task: ContainersTask
Standard output:

Standard error:
usage: is_prime.py [-h] number
is_prime.py: error: argument number: invalid int value: 'o'


Sample end: 2018-11-10 11:33:10
Elapsed time: 0:03:14

Delete job? [Y/n] y
yDelete pool? [Y/n] y

Press ENTER to exit...

Remember at the end to eliminate resources so that they do not infringe costs.

References

batch-python-quickstart
Run container applications on Azure Batch

Posted in Azure, Microsoft | Tagged , , , , , | Comments Off on Azure Batch: Task in Containers

Kubernetes Pipeline

Let’s explain an easy way to build an integration pipeline (CI) on Minikube.

Launch Minikube

If you don’t have Minikube running on your system,

$ minikube start --memory 4000 --cpus 2

Wait for a few minutes, you’ll see something like.

Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

Installing Helm

Helm is The Kubernetes Package Manager, it helps you to deploy services into Kubernetes.

$ wget https://storage.googleapis.com/kubernetes-helm/helm-v2.10.0-linux-amd64.tar.gz -O helm.tar.gz
$ tar zxf helm.tar.gz
$ sudo cp linux-amd64/helm /usr/local/bin/helm
$ sudo chmod +x /usr/local/bin/helm

Applying the RBAC policy

$ kubectl create -f https://raw.githubusercontent.com/nordri/kubernetes-experiments/master/Pipeline/ServiceAccount.yaml

and then launch helm.

helm init --service-account tiller

Checking

$ helm version
Client: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}

Deploying Jenkins

I’m using a custom values file for this chart. What I’m adjusting is:

AdminPassword: set to admin1234
ServiceType: set to NodePort (because is Minikube)
In plugins:
– kubernetes:1.2
– workflow-aggregator:2.5
– workflow-job:2.17
– credentials-binding:1.15
– git:3.7.0

And then the deployment:

$ helm install --name jenkins -f jenkins-helm-values.yaml stable/jenkins

After a few minutes we could be able to access Jenkins with:

$ minikube service jenkins

Configuring Jenkins

First, set the credentials to access Docker Hub where we’ll push the Docker images. The only field you must keep is ID because is needed by the pipeline in a next step. Fill it with your information:

Back to Jenkins main screen, add a new item type Pipeline

And finally, configure the pipeline in the Pipeline section:

Save the changes and click on Build now

And that’s it!

The pipeline

Let’s deep into the pipeline

The head

The pipeline starts setting the worker id so the pod has different label on each execution.

Follow the pod definition where we can define the containers who will run inside the pod. For this example we’ll need:

  1. maven
  2. docker
  3. mysql, this one with environment variables
  4. java, also with environment variables

Then the volumes, we need the docker sock in order to run docker in docker and a folder to save the artefacts downloaded from the Internet (it’s a Maven project!) between the executions. Saving time and bandwidth.

Cloning the repo…

What we do here is clean the workspace and clone the repository. It a SpringBoot application with MySQL.

Building…

We build the package using maven container.

Testing…

In this stage we launch our app inside Java container and after 30 seconds we check if it online, a simple smoky test. We save the return value in RES to decide if it’s ok or not. If not, finish with fail. As we defined all the containers at the beginning there’s a MySQL running inside the pod.

Building & Uploading Docker images…

If the testing stage went OK, we can push it to Docker Hub. To set the tag we use the commit ID cut to eight characters. To login into Docker Hub we use the withCredentials who takes a credential by id and fill the environment variables.

References

Set Up a Jenkins CI/CD Pipeline with Kubernetes

Repository

GitHub

Posted in Jenkins, Kubernetes | Tagged , , , , , | Comments Off on Kubernetes Pipeline

Elastest & Amazon Dash Button

Here, at Universidad Rey Juan Carlos, where we’re working hard to deploy Elastest, we’ve the need to run instances on AWS all the time. We, of course, have defined CloudFormation Templates to do the work but we can go further with just push a button (a physical one, indeed) so every time a team member needs to test something on Elastest, push the button!

Let’s see how it works

First of all, you need an Amazon Dash Button, I chose Optimum Nutrition a brand for gym diet, don’t ask why.

To configure the device, we follow the steps from Amazon’ guide till we have to choose a product. We avoid that because we don’t want to order products each time we push the button. So we’ve the device attached to our network and we need to know the MAC address.

We’re gonna use a Python project Amazon Dash to do that. First, we install it:

$ sudo pip install amazon-dash  # and after:
$ sudo python -m amazon_dash.install

Scan the network to find the device:

$ sudo amazon-dash discovery

Launch the command and wait like 10 seconds, then push the button and you’ll see the MAC address. Copy it for the next step.

Dash Button is based on systemd (which means that you must stop and start it on each change) and it has a yaml file for configuration (in /etc/amazon-dash.yml). It’s quite simple for out needs, should look like:

# amazon-dash.yml
# ---------------
settings:
  # On seconds. Minimum time that must pass between pulsations.
  delay: 10
devices:
  fc:65:de:1b:2e:8a:
    name: Elastest-dashbutton
    user: nordri
    cmd: PATH/TO/deploy-elastest.sh

For each device we have, we can define a stanza with a command to run. We set the user that will be the user who run the command. As you can imagine, the script is an AWS Cli command for CloudFormation like this one:

#!/bin/bash

DATE_SUFFIX=$(date +%s)

aws cloudformation create-stack \
  --stack-name Elastest-dashbutton-$DATE_SUFFIX \
  --template-url https://s3-eu-west-1.amazonaws.com/aws.elastest.io/cloud-formation-latest.json \
  --parameters '[{ "ParameterKey": "KeyName", "ParameterValue": "kms-aws-share-key" }]' \
  --profile naeva

Feel free to use the script on its own. Just remember to change the RSA key you use to access your instances on AWS and the auth profile.

That’s it, from now on, every push will result in an Elastest instance running on AWS.

Here, one could say

OK, mate, that’s pretty cool but deploying Elastest takes like 400 seconds and pushing a button won’t tell me when the instance is ready and, where should I connect to use the platform

Fair enough! Let’s configure the notifications.

Amazon Dash has confirmations to actually confirm that the button was pushed, but as our friend said before, we should notify when the instance is ready, to do that, we’re using a Telegram Bot. It’s quite simple to ask the Telegram’ @botfather for an Auth Token and start using it. So, out script to deploy Elastest now is like:

#!/bin/bash

DATE_SUFFIX=$(date +%s)

# Telegram data
USERID="...."
KEY="...."
TIMEOUT="10"
URL="https://api.telegram.org/bot$KEY/sendMessage"

aws cloudformation create-stack \
  --stack-name Elastest-dashbutton-$DATE_SUFFIX \
  --template-url https://s3-eu-west-1.amazonaws.com/aws.elastest.io/cloud-formation-latest.json \
  --parameters '[{ "ParameterKey": "KeyName", "ParameterValue": "kms-aws-share-key" }]' \
  --profile naeva

aws cloudformation wait stack-create-complete --stack-name Elastest-dashbutton-$DATE_SUFFIX --profile naeva
ELASTEST_URL=$(aws cloudformation describe-stacks --stack-name Elastest-dashbutton-$DATE_SUFFIX --profile naeva | jq --raw-output '.Stacks[0] | .Outputs[0] | .OutputValue')

TEXT="New Elastest deployed on $ELASTEST_URL"
curl -s --max-time $TIMEOUT -d "chat_id=$USERID&disable_web_page_preview=1&text=$TEXT" $URL > /dev/null

And, anyway, let’s configure the confirmation, so we’ll know when the button is pushed. To do so, see this Amazon Dash’ configuration file:

devices:
  fc:65:de:1b:2e:8a:
    name: Elastest-dashbutton
    user: nordri
    cmd: PATH/TO/deploy-elastest.sh
    confirmation: Telegram
confirmations:
  Telegram:
    service: telegram
    token: '...'
    to: ...
    is_default: true

Now, every time one of the team member needs an AWS instance to test Elastest, we’ll receive a notification at the beginning and another as soon as it’s ready.

Dash button cost 5€ and keep me busy for a day, not bad investment at all 😉

Posted in Automation, AWS, Tips | Tagged , , , , | Comments Off on Elastest & Amazon Dash Button