{"id":842,"date":"2022-01-14T15:25:07","date_gmt":"2022-01-14T15:25:07","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2022\/01\/14\/how-to-build-docker-images-in-a-gitlab-ci-pipeline-cloudsavvy-it\/"},"modified":"2022-01-14T15:25:07","modified_gmt":"2022-01-14T15:25:07","slug":"how-to-build-docker-images-in-a-gitlab-ci-pipeline-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2022\/01\/14\/how-to-build-docker-images-in-a-gitlab-ci-pipeline-cloudsavvy-it\/","title":{"rendered":"How to Build Docker Images In a GitLab CI Pipeline \u2013 CloudSavvy IT"},"content":{"rendered":"

<\/a>\n<\/p>\n

\n

\"Graphic<\/p>\n

One common use case for CI pipelines is building the Docker images you\u2019ll use to deploy your application. GitLab CI is a great choice for this as it supports an integrated pull proxy service, meaning faster pipelines, and a built-in registry to store your built images.<\/p>\n

In this guide, we\u2019ll show you how to set up Docker builds that use both the above features. The steps you need to take vary slightly depending on the GitLab Runner executor<\/a> type you\u2019ll use for your pipeline. We\u2019ll cover the Shell and Docker executors below.<\/p>\n

Building With the Shell Executor<\/h2>\n

If you\u2019re using the Shell executor, make sure you\u2019ve got Docker installed<\/a> on the machine that hosts your runner. The executor works by running regular shell commands using the docker<\/code> binary on the Runner\u2019s host.<\/p>\n

Head to the Git repository for the project you want to build images for. Create a .gitlab-ci.yml<\/code> file at the root of the repository. This file defines the GitLab CI pipeline that will run when you push changes to your project.<\/p>\n

Add the following content to the file:<\/p>\n

\n
\n
\n
\n
\n
\n
\n
stages<\/span>:\n<\/span>  - build\n\ndocker_build<\/span>:\n  stage<\/span>: <\/span>build\n  script<\/span>:\n<\/span>    - docker build -t example.com\/example-image:latest .\n    - docker push example.com\/example-image:latest<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
This simplistic configuration is enough to demonstrate the basics of pipeline-powered image builds. GitLab automatically clones your Git repository into the build environment so running docker build<\/code> will use your project\u2019s Dockerfile<\/code> and make the repository\u2019s content available as the build context<\/a>.<\/p>\n
After the build completes, you can docker push<\/code> the image to your registry. Otherwise it would only be available to the local Docker installation that ran the build. If you\u2019re using a private registry, run docker login<\/code> first to supply proper authentication details:<\/p>\n
script:
\n  - docker login -u $DOCKER_REGISTRY_USER -p $DOCKER_REGISTRY_PASSWORD<\/pre>\n
Define the values of the two credential variables by heading to Settings > CI\/CD > Variables in the GitLab web UI. Click the blue \u201cAdd variable\u201d button to create a new variable and assign a value. GitLab will make these variables available in the shell environment used to run your job.<\/p>\n
\"\"<\/p>\n
Building With the Docker Executor<\/h2>\n
GitLab Runner\u2019s Docker executor is commonly used to provide a completely clean environment for each job. The job will execute in an isolated container so the docker<\/code> binary on the Runner host will be inaccessible.<\/p>\n
The Docker executor gives you two possible strategies for building your image: either use Docker-in-Docker, or bind the host\u2019s Docker socket into the Runner\u2019s build environment. You then use the official Docker container image as your job\u2019s image, making the docker<\/code> command available in your CI script.<\/p>\n
Docker-in-Docker<\/h3>\n
Using Docker-in-Docker<\/a> (DinD) to build your images gives you a fully isolated environment for each job. The Docker process that performs the build will be a child of the container that GitLab Runner creates on the host to run the CI job.<\/p>\n
You need to register your GitLab Runner Docker executor with privileged mode<\/a> enabled to use DinD. Add the --docker-privileged<\/code> flag when you register your runner<\/a>:<\/p>\n
sudo gitlab-runner register -n 
\n  --url https:\/\/example.com 
\n  --registration-token $GITLAB_REGISTRATION_TOKEN 
\n  --executor docker 
\n  --description \"Docker Runner\" 
\n  --docker-image \"docker:20.10\" 
\n  --docker-volumes \"\/certs\/client\" 
\n  --docker-privileged<\/pre>\n
Within your CI pipeline, add the docker:dind<\/code> image as a service<\/a>. This makes Docker available as a separate image that\u2019s linked to the job\u2019s image. You\u2019ll be able to use the docker<\/code> command to build images using the Docker instance in the docker:dind<\/code> container.<\/p>\n
\n
\n
\n
\n
\n
\n
\n
services<\/span>:\n<\/span>  - docker:dind\n\ndocker_build<\/span>:\n  stage<\/span>: <\/span>build\n  image<\/span>: <\/span>docker:latest\n  script<\/span>:\n<\/span>    - docker build -t example-image:latest .<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
Using DinD gives you fully isolated builds that can\u2019t impact each other or your host. The major drawback is more complicated caching behavior: each job gets a new environment where previously built layers won\u2019t be accessible. You can partially address this by trying to pull the previous version of your image before you build, then using the --cache-from<\/code> build flag<\/a> to make the pulled image\u2019s layers available as a cache source:<\/p>\n
\n
\n
\n
\n
\n
\n
\n
docker_build<\/span>:\n  stage<\/span>: <\/span>build\n  image<\/span>: <\/span>docker:latest\n  script<\/span>:\n<\/span>    - docker pull $CI_REGISTRY_IMAGE:latest || true\n    - docker build --cache-from $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:latest .<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
Socket Bind Mounts<\/h3>\n
Mounting your host\u2019s Docker socket into your job\u2019s environment is an alternative option when you\u2019re using the Docker executor. This gives you seamless caching and removes the need to add the docker:dind<\/code> service to your CI config.<\/p>\n
To set this up, register your Runner with a docker-volumes<\/code> flag that binds the host\u2019s Docker socket to \/var\/run\/docker.sock<\/code> inside job containers:<\/p>\n
sudo gitlab-runner register -n 
\n  --url https:\/\/example.com 
\n  --registration-token $GITLAB_REGISTRATION_TOKEN 
\n  --executor docker 
\n  --description \"Docker Runner\" 
\n  --docker-image \"docker:20.10\" 
\n  --docker-volumes \/var\/run\/docker.sock:\/var\/run\/docker.sock<\/pre>\n
Now jobs that run with the docker<\/code> image will be able to use the docker<\/code> binary as normal. Operations will actually occur on your host machine, becoming siblings of the job\u2019s container instead of children.<\/p>\n
This is effectively similar to using the shell executor with your host\u2019s Docker installation. Images will reside on the host, facilitating seamless use of regular docker build<\/code> layer caching.<\/p>\n
While this approach can lead to higher performance, less configuration, and none of the limitations of DinD, it comes with its own unique issues. Most prominent among these are the security implications: jobs could execute arbitrary Docker commands on your Runner host, so a malicious project in your GitLab instance might run docker run -it malicious-image:latest<\/code> or docker rm -f $(docker ps -a)<\/code> with devastating consequences.<\/p>\n
GitLab also cautions<\/a> that socket binding can cause problems when jobs run concurrently. This occurs when you rely on containers being created with specific names. If two instances of a job run in parallel, the second one will fail as the container name will already exist on your host.<\/p>\n
You should consider using DinD instead if you expect either of these issues will be troublesome. While DinD is no longer generally recommended<\/a>, it can make more sense for public-facing GitLab instances that run concurrent CI jobs.<\/p>\n
Pushing Images to GitLab\u2019s Registry<\/h2>\n
GitLab projects have the option of an integrated registry<\/a> which you can use to store your images. You can view the registry\u2019s content by navigating to Packages & Registries > Container Registry in your project\u2019s sidebar. If you don\u2019t see this link, enable the registry by going to Settings > General > Visibility, Project, Features & Permissions and activating the \u201cContainer registry\u201d toggle.<\/p>\n
\"\"<\/p>\n
\u00a0<\/p>\n
GitLab automatically sets environment variables in your CI jobs which let you reference your project\u2019s container registry. Adjust the script<\/code> section to login to the registry and push your image:<\/p>\n
\n
\n
\n
\n
\n
\n
\n
script<\/span>:\n<\/span>  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY\n  - docker build -t $CI_REGISTRY_IMAGE:latest .\n  - docker push $CI_REGISTRY_IMAGE:latest<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
GitLab generates a secure set of credentials for each of your CI jobs. The $CI_JOB_TOKEN<\/code> environment variable will contain an access token the job can use to connect to the registry as the gitlab-ci-token<\/code> user. The registry server URL is available as $CI_REGISTRY<\/code>.<\/p>\n
The final variable, $CI_REGISTRY_IMAGE<\/code>, provides the complete path to your project\u2019s container registry. This is a suitable base for your image tags. You can extend this variable to create sub-repositories, such as $CI_REGISTRY_IMAGE\/production\/api:latest<\/code>.<\/p>\n
\u00a0<\/p>\n
Other Docker clients can pull images from the registry by authenticating using an access token. You can generate these on your project\u2019s Settings > Access Tokens screen. Add the read_registry<\/code> scope, then use the displayed credentials to docker login<\/code> to your project\u2019s registry.<\/p>\n
Using GitLab\u2019s Dependency Proxy<\/h2>\n
GitLab\u2019s Dependency Proxy<\/a> provides a caching layer for the upstream images you pull from Docker Hub. It helps you stay within Docker Hub\u2019s rate limits<\/a> by only pulling the content of images when they\u2019ve actually changed. This will also improve the performance of your builds.<\/p>\n
\"\"<\/p>\n
The Dependency Proxy is activated at the GitLab group level by heading to Settings > Packages & Registries > Dependency Proxy. Once it\u2019s enabled, prefix image references in your .gitlab-ci.yml<\/code> file with $CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX<\/code> to pull them through the proxy:<\/p>\n
\n
\n
\n
\n
\n
\n
\n
docker_build<\/span>:\n  stage<\/span>: <\/span>build\n  image<\/span>: <\/span>$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX\/docker:latest\n  services<\/span>:\n    - name<\/span>: <\/span>$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX\/docker:dind\n      alias<\/span>: <\/span>docker<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
That\u2019s all there is to it! GitLab Runner automatically logs into the dependency proxy registry so there\u2019s no need to manually supply your credentials.<\/p>\n
GitLab will now cache your images, giving you improved performance as well as resiliency to network outages. Note that the services<\/code> definition has had to be adjusted too \u2013 environment variables don\u2019t work with the inline form used earlier, so the full image name<\/code> must be specified, then a command alias<\/code> to reference in your script<\/code> section.<\/p>\n
While we\u2019ve now set up the proxy for images directly used by our job stages, more work\u2019s needed to add support for the base image in the Dockerfile<\/code> to build. A regular instruction like this won\u2019t go through the proxy:<\/p>\n
FROM ubuntu:latest<\/pre>\n
To add this final piece, use Docker\u2019s build arguments to make the dependency proxy URL available when stepping through the Dockerfile:<\/p>\n
ARG GITLAB_DEPENDENCY_PROXY
\nFROM ${GITLAB_DEPENDENCY_PROXY}\/ubuntu:latest<\/pre>\n
Then modify your docker build<\/code> command to define the variable\u2019s value:<\/p>\n
\n
\n
\n
\n
\n
\n
\n
script<\/span>:\n<\/span>  >\n<\/span>    - docker build \n        --build-arg GITLAB_DEPENDENCY_PROXY=${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX} \n        -t example-image:latest .<\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
Now your base image will be pulled through the dependency proxy too.<\/p>\n
Summary<\/h2>\n
Docker image builds are easily integrated into your GitLab CI pipelines. After initial Runner configuration, docker build<\/code> and docker push<\/code> commands in your job\u2019s script<\/code> section are all you need to create an image with the Dockerfile<\/code> in your repository. GitLab\u2019s built-in container registry gives you private storage for your project\u2019s images.<\/p>\n
\"\"<\/p>\n
Beyond basic builds, it\u2019s worth integrating GitLab\u2019s dependency proxy to accelerate performance and avoid hitting Docker Hub rate limits. You should also check the security of your installation by assessing whether your selected method allows untrusted projects to run commands on your Runner host. Although it carries its own issues, Docker-in-Docker is the safest approach when your GitLab instance is publicly accessible or accessed by a large user base.<\/p>\n<\/div>\n