Azure Batch: Task in Containers

Posted by on 10/11/2018

In today’s post I’ll be talking about how to send tasks to Microsoft Azure Batch able to run in containers. The task I want to solve in this example is calculating whether a number is prime or not. This Python code does the work for us. Then, I’ve write a Dockerfile adding the piece of code to the image. Now, we’re able to run the script from a Docker container.

Let’s move forward to Azure Batch, you need to create a Docker registry where you’ll push the Docker image and an Azure Batch account.

Docker Registry

Login into your Azure account and move to All resources, click Add and look for Container Registry. Then click in Create. Fill up the information with the name of the registry, the resource group, choose a location closer to you, enable Admin user to be able to push to the repo and the SKU (choose standard here).

Then Create

In a few seconds the registry will be ready. So go to the Dashboard and click on the registry name (the one you chose before). Click in Settings -> Access keys. Here are the credentials you’ll need to manage the registry.

Batch Account

From the All resources look for Batch service. Fill up the information with the Account name and Location, Subscription and Resource group should be ready.

Click Review + create and then Create. In a few seconds the service should be ready.

Building the Container

Clone the repo

git clone https://github.com/nordri/microsoft-azure

and build and push the container

cd batch-containers
docker build -t YOUR_REGISTRY_SERVER/YOUR_REGISTRY_NAME/YOUR_IMAGE_NAME .
# for example:
docker build -t pythonrepo.azurecr.io/pythonrepo/is_prime:latest .
# Check the image works:
docker run -ti --rm pythonrepo.azurecr.io/pythonrepo/is_prime python is_prime.py 7856
The number: 7856 is not prime
docker run -ti --rm pythonrepo.azurecr.io/pythonrepo/is_prime python is_prime.py 2237
The number: 2237 is prime
# login first
docker login pythonrepo.azurecr.io
Username: pythonrepo
Password: 
# Push
docker push pythonrepo.azurecr.io/pythonrepo/is_prime

Azure Batch

Now it’s time to send the task to Azure Batch. To do this, I’ve worked over this Python script. This script creates a pool, a job and three tasks to upload files to Azure Storage. So, I’ve made some modifications to fit it to my needs.

Creating the Pool

I need my pool to be created using instances able to run containers

...
def create_pool(batch_service_client, pool_id):
    print('Creating pool [{}]...'.format(pool_id))

    image_ref_to_use = batch.models.ImageReference(
        publisher='microsoft-azure-batch',
        offer='ubuntu-server-container',
        sku='16-04-lts',
        version='latest'
    )

    # Specify a container registry
    # We got the credentials from config.py
    containerRegistry = batchmodels.ContainerRegistry(
        user_name=config._REGISTRY_USER_NAME, 
        password=config._REGISTRY_PASSWORD, 
        registry_server=config._REGISTRY_SERVER
    )

    # The instance will pull the images defined here
    container_conf = batchmodels.ContainerConfiguration(
        container_image_names=[config._DOCKER_IMAGE],
        container_registries=[containerRegistry]
    )

    new_pool = batch.models.PoolAddParameter(
        id=pool_id,
        virtual_machine_configuration=batchmodels.VirtualMachineConfiguration(
            image_reference=image_ref_to_use,
            container_configuration=container_conf,
            node_agent_sku_id='batch.node.ubuntu 16.04'),
        vm_size='STANDARD_A2',
        target_dedicated_nodes=1
    )

    batch_service_client.pool.add(new_pool)
...

The key is the ImageReference where we set the instances to run with an OS able to run Docker. You must set the registry credentials and the default Docker image that will be pulled when the instance boots.

Creating the Task

I’ve also changed the Task for the same reason. This task is ready to launch a container in the instance.

...
def add_tasks(batch_service_client, job_id, task_id, number_to_test):
    print('Adding tasks to job [{}]...'.format(job_id))

    # This is the user who run the command inside the container.
    # An unprivileged one
    user = batchmodels.AutoUserSpecification(
        scope=batchmodels.AutoUserScope.task,
        elevation_level=batchmodels.ElevationLevel.non_admin
    )

    # This is the docker image we want to run
    task_container_settings = batchmodels.TaskContainerSettings(
        image_name=config._DOCKER_IMAGE,
        container_run_options='--rm'
    )
    
    # The container needs this argument to be executed
    task = batchmodels.TaskAddParameter(
        id=task_id,
        command_line='python /is_prime.py ' + str(number_to_test),
        container_settings=task_container_settings,
        user_identity=batchmodels.UserIdentity(auto_user=user)
    )

    batch_service_client.task.add(job_id, task)
...

You can see how I’ve defined the user inside the container as a non admin user. The Docker image we want to use and the arguments we need to pass in the command line, remember we launch the container like:

docker ... imagename python /is_prime.py number

Launching the Script

Configure

In order to launch the script we need to fill up some configuration. Open the config.py file and write all the credentials needed. Remember, all the credentials are in the Access keys section.

Installing Dependencies

You need Azure Python SDK installed to run the script.

pip install -r requirements.txt

Let’s go

Now we’re ready to launch the script:

python batch_containers.py 89
Sample start: 2018-11-10 10:11:11

Creating pool [ContainersPool]...
No handlers could be found for logger "msrest.pipeline.requests"
Creating job [ContainersJob]...
Adding tasks to job [ContainersJob]...
Monitoring all tasks for 'Completed' state, timeout in 0:30:00.....................................................................................................................................................................
  Success! All tasks reached the 'Completed' state within the specified timeout period.
Printing task output...
Task: ContainersTask
Standard output:
The number: 89 is prime

Standard error:


Sample end: 2018-11-10 10:14:31
Elapsed time: 0:03:20

Delete job? [Y/n] y
Delete pool? [Y/n] y

Press ENTER to exit...

If there’s a problem with the script we’ll see the error on stderr.txt.

Sample start: 2018-11-10 11:29:56

Creating pool [ContainersPool]...
No handlers could be found for logger "msrest.pipeline.requests"
Creating job [ContainersJob]...
Adding tasks to job [ContainersJob]...
Monitoring all tasks for 'Completed' state, timeout in 0:30:00..................................................................................................................................................................
  Success! All tasks reached the 'Completed' state within the specified timeout period.
Printing task output...
Task: ContainersTask
Standard output:

Standard error:
usage: is_prime.py [-h] number
is_prime.py: error: argument number: invalid int value: 'o'


Sample end: 2018-11-10 11:33:10
Elapsed time: 0:03:14

Delete job? [Y/n] y
yDelete pool? [Y/n] y

Press ENTER to exit...

Remember at the end to eliminate resources so that they do not infringe costs.

References

batch-python-quickstart
Run container applications on Azure Batch

Azure, Microsoft, , , , ,

Comments are closed.