In this post, we’ll walk through how to manage microservice sprawl with Docker Compose by importing subprojects from other Git repos.

Maintaining simplicity

Now, perhaps more than ever, that simplicity is key. The complexity of modern software development is undeniable regardless of whether you’re using microservices or a monolith, deploying to the cloud or on-prem, or writing in JavaScript or C. 

Compose has not kept up with this “development sprawl” and is even sometimes an obstacle when working on larger, more complex projects. Maintaining Compose to accurately represent your increasingly complex application can require its own expertise, often resulting in out-of-date configuration in YAML or complex makefile tasks.

As an open source project, Compose serves everyone from home lab enthusiasts to transcontinental corporations, which is no small feat, and our commitment to maintaining Compose’s signature simplicity for all users hasn’t changed.

The increased flexibility afforded by Compose watch and include means your project no longer needs to be one-size-fits-all. Now, it’s possible to split your project across Git repos and import services as needed, customizing their configuration in the process.

Application architecture

Let’s take a look at a hypothetical application architecture. To begin, the application is split across two Git repos:

• backend — Backend in Python/Flask
• frontend — Single-page app (SPA) frontend in JavaScript/Node.js

While working on the frontend, the developers run without using Docker or Compose, launching npm start on their laptops directly and proxy API requests to a shared staging server (as opposed to running the backend locally). Meanwhile, while working on the backend, developers and CI (for integration tests) share a Compose file and rely on command-line tools like cURL to manually test functionality locally.

We’d like a flexible configuration that enables each group of developers to use their optimal workflow (e.g., leveraging hot reload for the frontend) while also allowing reuse to share project configuration between repos. At first, this seems like an impossible situation to resolve.

Frontend

We can start by adding a compose.yaml file to frontend:

services:
  frontend:
  pull_policy: build
    build:
      context: .
    environment:
      BACKEND_HOST: ${BACKEND_HOST:-https://staging.example.com}
    ports:
      - 8000:8000

Note: If you’re wondering what the Dockerfile looks like, take a look at this samples page for an up-to-date example of best practices generated by docker init.

This is a great start! Running docker compose up will now build the Node.js frontend and make it accessible at http://localhost:8000/.

The BACKEND_HOST environment variable can be used to control where upstream API requests are proxied to and defaults to our shared staging instance.

Unfortunately, we’ve lost the great developer experience afforded by hot module reload (HMR) because everything is inside the container. By adding a develop.watch section, we can preserve that:

services:
  frontend:
    pull_policy: build
    build:
      context: .
    environment:
      BACKEND_HOST: ${BACKEND_HOST:-https://staging.example.com}
    ports:
      - 8000:8000
    develop:
      watch:
        - path: package.json
          action: rebuild
        - path: src/
          target: /app/src
          action: sync

Now, while working on the frontend, developers continue to benefit from the rapid iteration cycles due to HMR. Whenever a file is modified locally in the src/ directory, it’s synchronized into the container at /app/src. 

If the package.json file is modified, the entire container is rebuilt, so that the RUN npm install step in the Dockerfile will be re-executed and install the latest dependencies. The best part is the only change to the workflow is running docker compose watch instead of npm start.

Backend

Now, let’s set up a Compose file in backend:

services:
  backend:
    pull_policy: build
    build:
      context: .
    ports:
      - 1234:8080
    develop:
      watch:
        - path: requirements.txt
          action: rebuild
        - path: ./
          target: /app/
          action: sync

include:
  - path: git@github.com:myorg/frontend.git
    env_file: frontend.env

frontend.env

BACKEND_HOST=http://backend:8080

Much of this looks very similar to the frontend compose.yaml.

When files in the project directory change locally, they’re synchronized to /app inside the container, so the Flask dev server can handle hot reload. If the requirements.txt is changed, the entire container is rebuilt, so that the RUN pip install step in the Dockerfile will be re-executed and install the latest dependencies.

However, we’ve also added an include section that references the frontend project by its Git repository. The custom env_file points to a local path (in the backend repo), which sets BACKEND_HOST so that the frontend service container will proxy API requests to the backend service container instead of the default.

Note: Remote includes are an experimental feature. You’ll need to set COMPOSE_EXPERIMENTAL_GIT_REMOTE=1 in your environment to use Git references.

With this configuration, developers can now run the full stack while keeping the frontend and backend Compose projects independent and even in different Git repositories.

As developers, we’re used to sharing code library dependencies, and the include keyword brings this same reusability and convenience to your Compose development configurations.

What’s next?

There are still some rough edges. For example, the remote project is cloned to a temporary directory, which makes it impractical to use with watch mode when imported, as the files are not available for editing. Enabling bigger and more complex software projects to use Compose for flexible, personal environments is something we’re continuing to improve upon.

If you’re a Docker customer using Compose across microservices or repositories, we’d love to hear how we can better support you. Get in touch!

Learn more

• Get the latest release of Docker Desktop.
• Vote on what’s next! Check out our public roadmap.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

We’re developers, too, so we explored ways to speed up build time without sacrificing the local development experience we love. That’s how Docker Build Cloud was born. 

“The new products we announced meet development teams where they are with ‘just enough cloud,’ seamlessly blurring the boundaries between local and remote development. In doing so, we’re enabling these teams to accelerate their delivery of secure applications critical to their businesses.” — Giri Sreenivas, Chief Product Officer, Docker

Temporary fixes don’t last 

Developers and their employers have attempted to address resource constraints that slow build times and drain productivity in a few common ways, the Incredibuild study says, including upgrading hardware (cited by 44% of respondents) and reducing codebase size (acknowledged by 33%). Although these tactics offer a temporary boost, they’re far from sustainable. A better solution is required — one that speeds up build times and improves team collaboration by eliminating redundant builds. 

Build up to 39x faster with Docker Build Cloud 

In the constantly evolving landscape of software development, two investment areas continue to trend upwards: moving development to the cloud and accelerating builds. The concept behind Docker Build Cloud is simple. By leveraging cloud compute and cache, developers can build faster and improve collaboration with their team. 

Building in the cloud speeds up build times because the cloud provides access to faster compute resources than are available on a developer’s local machine, and this approach provides more consistency between developers who may have newer or older machines. 

Shared cache speeds up build times because when one team member initiates a build, the cached results become instantly accessible to others, thereby eliminating unnecessary builds and speeding up the development cycle. No more waiting around for each build to complete independently. 

The combination of building in the cloud and shared cache helps developers save time and improve productivity. Developers can get back to coding on parallel tasks while builds complete, and they get the results of builds back faster to incorporate into their work.  

For example, one of our technology customers who develops enterprise collaboration software was able to reduce their build time from an average duration of 15-20 minutes to less than 2 minutes using Docker Build Cloud. 

Multi-architecture builds made effortless

Today, developers who need to create applications for both Intel (AMD64) and Apple Silicon/AWS Graviton (Arm64) chipsets must have multiple native builders or configure slow emulators to successfully build for their deployment targets. Docker Build Cloud offers native support for multi-architecture builds, eliminating the need for setting up and maintaining multiple native builders. This support removes the challenges associated with emulation, further improving build efficiency.

An e-commerce customer simplified their CI toolchain. Prior to Docker Build Cloud, they were leveraging GitHub Actions, GitLab runners, plus a custom GitLab runner to handle ARM architecture. Due to Build Cloud’s dual AMD and ARM builders, our customer reduced their complexity and sped up their pipelines. 

Seamless integration with the tools you love

Developer tools should enhance developer experience, not add new points of friction. We’ve designed Docker Build Cloud to be easy to set up wherever you run your builds without requiring a massive lift-and-shift effort. Docker Build Cloud also works well with Docker Compose, GitHub Actions, and other CI solutions. This means you can seamlessly incorporate Docker Build Cloud into your existing development tools and services and immediately start reaping the benefits of enhanced speed and efficiency.

Try Docker Build Cloud today

Docker Build Cloud will enable a faster, more efficient Docker image-building process. Whether you’re a seasoned developer or just getting started, embrace the power of the cloud in your local development environment. Welcome to the future of Docker image builds with Docker Build Cloud. 

Try Docker Build Cloud now. 

Learn more

• See the video demo Introducing Docker Build Cloud.
• Watch the webinar Docker Build Cloud: Reclaim Your Dev Time with Fast, Multi-Architecture Builds.
• Get the latest release of Docker Desktop.
• Vote on what’s next! Check out our public roadmap.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

With containerized application development, there are more steps than Alt+Tab and hitting reload in your browser. Even with caching, rebuilding the image and re-creating the container — especially after waiting on stop and start time — can disrupt focus.

We built Docker Compose Watch to smooth away these workflow papercuts. We have learned from many people using our open source Docker Compose project for local development. Now we are natively addressing common workflow friction we observe, like the use case of hot reload for frontend development. 

Bind mount vs. Watch

A common workaround to get hot reload to work is to set up a bind mount to mirror file changes between the local system and a container. This method uses operating system and hypervisor APIs to expose a local directory to the otherwise isolated file system in the container.

The workaround is not trivial engineering since how bind mounts function in Docker Desktop differs from Docker Engine on Linux. For parity, Docker Desktop must provide seamless and efficient file sharing between your machine and its virtual machine (VM), ensuring permissions, replicating file notifications, and maintaining low-level filesystem consistency to prevent corruption. 

In contrast, Docker Compose Watch is specifically targeting development use cases. You may want to ensure your code changes sync into the container, allowing React or NextJS to kick off its own live reload. However, you don’t want the changes you’ve made in the container for ad-hoc testing to reflect in your local directory. For this reason, the tradeoffs we make for Docker Compose Watch favor fine-grained control for common development workflows with Docker Compose (Figures 1 and 2).

Improving Watch for development

Since the alpha launch (in Compose v2.17, bundled with Docker Desktop 4.18), we’ve responded to early feedback by making Docker Compose Watch faster and more robust. This improvement avoids hiccups on common development tasks that kick off many changes, such as merging in the latest main or switching branches. 

Your code sync operation now batches, debounces, and ignores unimportant changes:

• What previously would be many API calls are now batched as a single API call to the Docker Engine.
• We’ve fine-tuned the streaming of changes to your containers for improved transfer performance. A new built-in debounce mechanism prevents unnecessary transfers in case of back-to-back writes to the same file. This optimizes CPU usage by preventing unnecessary incremental compiles.
• The built-in filters have been refined to ignore standard temporary files generated by common code editors and integrated development environments (IDEs).

Previously, Docker Compose Watch required attaching to an already running Compose project. Docker Compose Watch now automatically builds and starts all required services at launch. One command is all you need: docker compose watch.

Try Docker Compose Watch in Docker Desktop 4.24

As of Compose 2.22.0, bundled with Docker Desktop 4.24, Docker Compose Watch is now Generally Available. Make sure to upgrade to the latest version of Docker Desktop and develop more efficiently with docker compose watch.

Let us know how Docker Compose Watch supports your use case and where it can improve. Or you can contribute directly to the open source Docker Compose project.

Learn more

• Read the Docker Desktop Release Notes.
• Learn about Docker Desktop 4.24.
• Get the latest release of Docker Desktop.
• Contribute to the open source Docker Compose project.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

Resource Saver 

In 4.22 we have added a new Resource Saver feature for Mac and Windows which detects when Docker Desktop is not running containers and massively reduces its memory and CPU footprint (WSL has CPU optimizations only at this stage). When Resource Saver detects that Docker Desktop is idle without any active containers for a duration of 30 seconds, it automatically reduces the memory and CPU footprint. This optimizes Docker Desktop for your system and helps to free up resources on your machine for other tasks. When a container needs resources, they’re quickly allocated on demand.

To see this feature in action, start Docker Desktop and leave it idle for 30 seconds with no containers running. A leaf will appear over the whale icon in your Docker Desktop menu and the sidebar of the Docker Desktop dashboard, indicating that Resource Saver mode is activated.

Previously, Docker Desktop introduced some CPU optimizations of Resource Saver, which, at the time of writing, are already saving up to a staggering 38,500 CPU hours every single day across all Docker Desktop users.

Split complex Compose projects into multiple subprojects with ‘include’

If you’re working with complex applications, use the new include section in your Compose file to split your project into manageable subprojects. Compared to merging files with CLI flags or using extends to share common attributes of a single service from another file, include loads external Compose files as self-contained building blocks, making it easier to collaborate on services across teams and share common dependency configurations within your organization.

For more on how you can try out this feature, read “Improve Docker Compose Modularity with `include`” or refer to the documentation.

Editor role available for organizations

With the addition of the Editor role, admins can provision users to manage repositories without full administrator privileges. Users assigned to the Editor role can:

1. Create public and private repositories
2. Pull, push, view, edit, and delete a repository
3. Update repository description
4. Assign team permissions to repos
5. Update scanning settings
6. Delete tags
7. Add webhooks
8. Change repo visibility settings

For further details on roles and permissions, refer to the documentation. 

Organization owners can assign the Editor role to a member of their organization in either Docker Hub or Docker Admin.

Conclusion

Upgrade now to explore what’s new in the 4.22 release of Docker Desktop. Do you have feedback? Leave feedback on our public GitHub roadmap and let us know what else you’d like to see in upcoming releases.

Learn more

• Read Improve Docker Compose Modularity with include.
• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

The docker command line supports many flags to fine-tune your container, and it’s difficult to remember them all when replicating an environment. Doing so is even harder when your application is not a single container but a combination of many containers with various dependency relationships. Based on this, Docker Compose quickly became a popular tool because it lets users declare all the infrastructure details for a container-based application into a single YAML file using a simple syntax directly inspired by the docker run… command line.

Still, an issue persists for large applications using dozens, maybe hundreds, of containers, with ownership distributed across multiple teams. When using a monorepo, teams often have their own “local” Docker Compose file to run a subset of the application, but then they need to rely on other teams to provide a reference Compose file that defines the expected way to run their own subset.

This issue is not new and was debated in 2014 when Docker Compose was a fresh new project and issue numbers had only three digits. Now we’ve introduced the ability to “compose compose files” to address this need — even before this issue reaches its 10th anniversary!

In this article, we’ll show how the new include attribute, introduced in Docker Compose 2.20, makes Compose files more modular and reusable.

Extend a Compose file

Docker Compose lets you reuse an existing Compose file using the extends mechanism. This special configuration attribute lets you refer to another Compose file and select a service you want to also use in your own application, with the ability to override attributes for your own needs.

services:
  database:
    extends:
        file: ../commons/compose.yaml
        service: db

That’s a good solution as long as you only need a single service to be shared, and you know about its internal details so you know how to tweak configuration. But, it is not an acceptable solution when you want to reuse someone else’s configuration as a “black box” and don’t know about its own dependencies.

Merge Compose files

Another option is to merge a set of Compose files together. Docker Compose accepts a set of files and will merge and override the service definition to eventually create a composite Compose application model.

This approach has been utilized for years, but it comes with a specific challenge. Namely, Docker Compose supports relative paths for the many resources to be included in the application model, such as: build context for service images, location of file defining environment variables, path to a local directory used in a bind-mounted volume.

With such a constraint, code organization in a monorepo can become difficult, as a natural choice would be to have dedicated folders per team or component, but then the Compose files relative paths won’t be relevant.

Let’s play the role of the “database” team and define a Compose file for the service we are responsible for. Next, we build our own image from a Dockerfile and have a reference environment set as an env file:

services:
  database:
    builld: . 
    env-file:
      - ./db.env

Now, let’s switch to another team and build a web application, which requires access to the database:

services:
  webapp:
    depends_on:
    - database 

Sounds good, until we try to combine those, running the following from the webapp directory: docker compose -f compose.yaml -f ../database/compose.yaml.

In doing so, the relative paths set by the second Compose file won’t get resolved as designed by the authors but from the local working directory. Thus, the resulting application won’t work as expected.

Reuse content from other teams

The include flag was introduced for this exact need. As the name suggests, this new top-level attribute will get a whole Compose file included in your own application model, just like you did a full copy/paste. The only difference is that it will manage relative path references so that the included Compose file will be parsed the way the author expects, running from its original location. This capability makes it way easier for you to reuse content from another team, without needing to know the exact details.

include:
../database/compose.yaml

services:
  webapp:
    depends_on:
     - database 

In this example, an infrastructure team has prepared a Compose file to manage a database service, maybe including some replicas, web UI to inspect data, isolated networks, volumes for data persistence, etc.

An application relying on this service doesn’t need to know about those infrastructure details and will consume the Compose file as a building block it can rely on. Thus, the infrastructure team can refactor its own database component to introduce additional services without the dependent teams being impacted.

This approach also comes with the bonus that the dependency on another Compose file is now explicit, and users don’t need to include additional flags on each Compose command they run. They can instead rely on the beloved docker compose up command without any additional knowledge of the application architecture.

Conclusion

With microservices and monorepo, it becomes common for an application to be split into dozens of services, and complexity is moved from code into infrastructure and configuration file. Docker Compose fits well with simple applications but is harder to use in such a context. At least it was, until now.

With include support, Docker Compose makes it easier to modularize such complex applications into sub-compose files. This capability allows application configuration to be simpler and more explicit. It also helps to reflect the engineering team responsible for the code in the config file organization. With each team able to reflect in configuration the way it depends on other’s work, there’s a natural approach to compose.yaml files organization.

Read more about this new include feature on the dedicated Compose specification page and experiment with it by upgrading Docker Compose to v2.20 or later.

Learn more

• Read the include documentation
• Get the latest release of Docker Desktop.
• Vote on what’s next! Check out our public roadmap.
• Have questions? The Docker community is here to help
• New to Docker? Get started.

Note: Docker Init should not be confused with the internally -used docker-init executable, which is invoked by Docker when utilizing the –init flag with the docker run command.

Create assets automatically

The new  docker init command automates the creation of necessary Docker assets, such as Dockerfiles, Compose files, and .dockerignore files, based on the characteristics of the project. By executing the docker init command, developers can quickly containerize their projects. Docker init is a valuable tool for developers who want to experiment with Docker, learn about containerization, or integrate Docker into their existing projects.

To use docker init, developers need to upgrade to the version 4.19.0 or later of Docker Desktop and execute the command in the target project folder. Docker init will detect the project definitions, and it will automatically generate the necessary files to run the project in Docker. 

The current Beta release of docker init supports Go, Node, and Python, and our development team is actively working to extend support for additional languages and frameworks, including Java, Rust, and .NET. If there is a language or stack that you would like to see added or if you have other feedback about docker init, let us know through our Google form.

In conclusion, docker init is a valuable tool for developers who want to simplify the process of adding Docker support to their projects. It automates the creation of necessary Docker assets  and can help standardize the creation of Docker assets across different projects. By enabling developers to focus on developing their applications and reducing the risk of errors and inconsistencies, Docker init can help accelerate the adoption of Docker and containerization.

See Docker Init in action

To see docker init in action, check out the following overview video by Francesco Ciulla, which demonstrates building the required Docker assets to your project.

Check out the documentation to learn more.

5x faster container-to-host networking on macOS

In Docker Desktop 4.19, we’ve made container-to-host networking performance 5x faster on macOS by replacing vpnkit with the TCP/IP stack from the gVisor project.

Many users work on projects that have containers communicating with a server outside their local Docker network. One example of this would be workloads that download packages from the internet via npm install or apt-get. This performance improvement should help a lot in these cases.

Over the next month, we’ll keep track of the stability of this new networking stack. If you notice any issues, you can revert to using the legacy vpnkit networking stack by setting "networkType":"vpnkit" in Docker Desktop’s settings.json config file.

Docker Init (Beta): Support for Node and Python

In our 4.18 release, we introduced docker init, a CLI command in Beta that helps you easily add Docker to any of your projects by creating the required assets for you. In the 4.19 release, we’re happy to add to this and share that the feature now includes support for Python and Node.js. 

You can try docker init with Python and Node.js by updating to the latest version of Docker Desktop (4.19) and typing docker init in the command line while inside a target project folder. 

The Docker team is working on adding more languages and frameworks for this command, including Java, Rust, and .Net. Let us know if you would like us to support a specific language or framework. We welcome any feedback you may have as we continue to develop and improve Docker Init (Beta).

Docker Scout (Early Access)

With Docker Desktop release 4.19, we’ve made it easier to view Docker Scout data for all of your images directly in Docker Desktop. Whether you’re using an image stored locally in Docker Desktop or a remote image from Docker Hub, you can see all that data without leaving Docker Desktop.

A nudge toward Compose v2

Compose v1 has reached end-of-life and will no longer be bundled with Docker Desktop after June 2023.

In preparation, a new warning will be shown in the terminal when running Compose v1 commands. Set the COMPOSE_V1_EOL_SILENT=1 environment variable to suppress this message.

You can upgrade by enabling Use Compose v2 in the Docker Desktop settings. When active, Docker Desktop aliases docker-compose to Compose v2 and supports the recommended docker compose syntax.

Moby 23

We updated the Docker Engine and the CLI to Moby 23.0,  where we are upstreaming open source internal developments such as the containerd integration and Wasm support, which will ship with Moby 24.0. Moby 23.0 includes additional enhancements such as the --format=json shorthand variant of --format=“{{ json . }}” and support of relative source paths to the run command in the -v/--volume and -m/--mount flags. You can read more about Moby 23.0 in the release notes.

Conclusion

We love hearing your feedback. Please leave any feedback on our public GitHub roadmap and let us know what else you’d like to see. Check out the Docker Desktop 4.19 release notes for a full breakdown of what’s new in the latest release.

And, although “building a Docker image” was previously considered a last step in release pipelines, it’s safe to say that containers have since become an essential part of our daily workflow. Still, concerns around slow builds and developer experience have often been a barrier towards the adoption of containers for local development.

We’ve come a long way, though. For starters, Docker Compose v2 now has deep integration with BuildKit, so you can use features like RUN cache mounts, SSH Agent forwarding, and efficient COPY with –link to speed up and simplify your builds. We’re also constantly making quality-of-life tweaks like enhanced progress reporting and improving consistency across the Docker CLI ecosystem.

As a result, more developers are running docker compose build && docker compose up to keep their running development environment up-to-date as they make code changes. In some cases, you can even use bind mounts combined with a framework that supports hot reload to avoid the need for an image rebuild, but this approach often comes with its own set of caveats and limitations.

An early look at the watch command

Starting with Compose v2.17, we’re excited to share an early look at the new development-specific configuration in Compose YAML as well as an experimental file watch command (Figure 1) that will automatically update your running Compose services as you edit and save your code.

This preview is brought to you in no small part by Compose developer Nicolas De Loof (in addition to more than 10 bugfixes in this release alone).

An optional new section, x-develop, can be added to a Compose service to configure options specific to your project’s daily flow. In this release, the only available option is watch, which allows you to define file or directory paths to monitor on your computer and a corresponding action to take inside the service container.

Currently, there are two possible actions: 

• sync — Copy changed files matching the pattern into the running service container(s).
• rebuild — Trigger an image build and recreate the running service container(s).

services:
  web:
    build: .
    x-develop:
      watch:
        - action: sync
          path: ./web
          target: /src/web
        - action: rebuild
          path: package.json

In the preceding example, whenever a source file in the web/ directory is changed, Compose will copy the file to the corresponding location under /src/web inside the container. Because Webpack supports Hot Module Reload, the changes are automatically detected and applied.

Unlike source code files, adding a new dependency cannot be done on the fly, so whenever package.json is changed, Compose will rebuild the image and recreate the web service container.

Behind the scenes, the file watching code shares its core with Tilt. The intricacies and surprises of file watching have always been near and dear to the Tilt team’s hearts, and, as Dockhands, the geeking out has continued. 

We are going to continue to build out the experience while gated behind the new docker compose alpha command and x-develop Compose YAML section. This approach will allow us to respond to community feedback early in the development process while still providing a clear path to stabilization as part of the Compose Spec.

Docker Compose powers countless workflows today, and its lightweight approach to containerized development is not going anywhere — it’s just learning a few new tricks.

Try it out

Follow the instructions at dockersamples/avatars to quickly run a small demo app, as follows:

git clone https://github.com/dockersamples/avatars.git
cd avatars
docker compose up -d
docker compose alpha watch
# open http://localhost:5735 in your browser

If you try it out on your own project, you can comment on the proposed specification on GitHub issue #253 in the compose-spec repository.

We’ll walk through new Docker Compose features the team has built, share what we plan to work on next, and remind you to switch to Compose V2 as soon as possible.

Compose V1 support will no longer be provided after June 2023 and will be removed from all future Docker Desktop versions. If you’re still on Compose V1, we recommend you switch as soon as possible to leave time to address any issues with running your Compose applications. (Until the end of June 2023, we’ll monitor Compose issues to address challenges related to V1 migration to V2.)

Compose V1: So long and farewell, old friend!

In the Compose V2 GA announcement we proposed the following timeline:

We’ve extended the timeline, so support now ends after June 2023. 

Switching is easy. Type docker compose instead of docker-compose in your favorite terminal.

An even easier way is to choose Compose V2 by default inside Docker Desktop settings. Activating this option creates a symlink for you so you can continue using docker-compose to preserve your potential existing scripts, but start using the newest version of Compose.

For more on the differences between V1 and V2, see the Evolution of Compose in docs.

What’s new?

Build improvements

During the past few months, the main effort of the team was to focus on improving the build experience within Compose. After collecting all the proposals opened in the Compose specification, we started to ship the following new features incrementally:

• cache_to support to allow sharing layers from intermediary images in a multi-stage build. One of the best ways to use this option is sharing cache in your CI between your workflow steps.
• no-cache to force a full rebuild of your service.
• pull to trigger a registry sync for force-pulling your base images.
• secrets to use at build time.
• tags to define a list associated with your final build image.
• ssh to use your local ssh configuration or pass keys to your build process. This allows you to clone a private repo or interact with protected resources; the ssh info won’t be stored in the final image.
• platforms to define multiple platforms and let Compose produce multi-arch images of your services.

Let’s dive deeper into those last two improvements.

Using ssh resources

ssh was introduced in Compose V2.4.0 GA and lets you use ssh resources at build time. Now you’re able to use your local ssh configuration or public/private keys when you build your service image. For example, you can clone a private Git repository inside your container or connect to a remote server to use critical resources during the build process of your services.

The ssh resources are only used during the build process and won’t be available in your final image.

There are different possibilities for using ssh with Compose. The first one is the new ssh attribute of the build section in your Compose file:

services:
 myservice:
   image: build-test-ssh
   build:
     context: .
     ssh:
       - fake-ssh=./fixtures/build-test/ssh/fake_rsa

And you need to reference the ID of your ssh resource inside your Dockerfile:

FROM alpine
RUN apk add --no-cache openssh-client

WORKDIR /compose
COPY fake_rsa.pub /compose/

RUN --mount=type=ssh,id=fake-ssh,required=true diff <(ssh-add -L) <(cat /compose/fake_rsa.pub)

This example is a simple demonstration of using keys at build time. It copies a public ssh key, mounts the private key inside the container, and checks if it matches the public key previously added.

It’s also possible to directly use the CLI with the new --ssh flag. Let’s try to use it to copy a private Git repository. 

The following Dockerfile adds GitHub as a known host in the ssh configuration of the image and then mounts the ssh local agent to clone the private repository:

# syntax=docker/dockerfile:1
FROM alpine:3.15

RUN apk add --no-cache openssh-client git
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
RUN --mount=type=ssh git clone git@github.com:glours/secret-repo.git

CMD ls -lah secret-repo

And using the docker compose build --no-cache --progress=plain --ssh default command will pass your local ssh agent to Compose.

Build multi-arch images with Compose

In Compose version V2.11.0, we introduced the ability to add platforms in the build section and let Compose do a cross-platform build for you.

The following Dockerfile logs the name of the service, the targeted platform to build, and the platform used for doing this build:

FROM --platform=$BUILDPLATFORM golang:alpine AS build

ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG SERVICENAME
RUN echo "I am $SERVICENAME building for $TARGETPLATFORM, running on $BUILDPLATFORM" > /log

FROM alpine
COPY --from=build /log /log

This Compose file defines an application stack with two services (A and B) which are targeting different build platforms:

services:
 serviceA:
   image: build-test-platform-a:test
   build:
     context: .
     args:
       - SERVICENAME=serviceA
     platforms:
       - linux/amd64
       - linux/arm64
 serviceB:
   image: build-test-platform-b:test
   build:
     context: .
     args:
       - SERVICENAME=serviceB
     platforms:
       - linux/386
       - linux/arm64

Be sure to create and use a docker-container build driver that allows you to build multi-arch images: 

docker buildx create --driver docker-container --use

To use the multi-arch build feature:

> docker compose build --no-cache

Additional updates

We also fixed issues, managed corner cases, and added features. For example, you can define a secret from the environment variable value:

services:
 myservice:
   image: build-test-secret
   build:
     context: .
     secrets:
       - envsecret

secrets:
 envsecret:
   environment: SOME_SECRET

We’re now providing Compose binaries for windows/arm64 and linux/riscv64.

We overhauled the way Compose manages .env files, environment variables, and precedence interpolation. Read the environment variables precedence documentation to learn more. 

To see all the changes we’ve made since April 2022, check out the Compose release page or the comparing changes page.

What’s next?

The Compose team is focused on improving the developer inner loop using Compose. Ideas we’re working on include:

• A development section in the Compose specification, including a watch mode so you will be able to use the one defined by your programming tooling or let Compose manage it for you 
• Capabilities to add specific debugging ports, or use profiling tooling inside your service containers
• Lifecycle hooks to interact with services at different moments of the container lifecycle (for example, letting you execute a command when a container is created but not started, or when it’s up and healthy)
• A --dry-run flag to test a Compose command before executing it

If you’d like to see something in Compose to improve your development workflow, we invite your feedback in our Public Roadmap.

To take advantage of ongoing improvements to Compose and surface any issues before support ends June 2023, make sure you’re on Compose V2. Use the docker compose CLI or activate the option in Docker Desktop settings.

To learn more about the differences between V1 and V2, check out the Evolution of Compose in our documentation.

Rapid7 is a Boston-based provider of security analytics and automation solutions enabling organizations to implement an active approach to cybersecurity. Over 10,000 customers rely on Rapid7 technology, services, and research to improve security outcomes and securely advance their organizations.

The security space is constantly changing, with new threats arising every day. To meet their customers’ needs, Rapid7 focuses on increasing the reliability and velocity of software builds while also maintaining their quality.

That’s why Rapid7 turned to Docker. Their teams use Docker to help development, support the sales pipeline, provide testing environments, and deploy to production in an automated, reliable way. 

By using Docker, Rapid7 transformed their onboarding process by automating manual processes. Setting up a new development environment now takes minutes instead of days. Their developers can produce faster builds that enable regular code releases to support changing requirements.

Automating the onboarding process

When developers first joined Rapid7, they were met with a static, manual process that was time consuming and error-prone. Configuring a development environment isn’t exciting for most developers. They want to spend most of their time creating! And setting up the environment is the least glamorous part of the process.

Docker helped automate this cumbersome process. Using Docker, Rapid7 could create containerized systems that were preconfigured with the right OS and developer tools. Docker Compose enabled multiple containers to communicate with each other, and it had the hooks needed to incorporate custom scripting and debugging tools.

Once the onboarding setup was configured through Docker, the process was simple for other developers to replicate. What once took multiple days now takes minutes.

Expanding containers into production

The Rapid7 team streamlined the setup of the development environment by using a Dockerfile. This helped them create an image with every required dependency and software package.

But they didn’t stop there. As this single Docker image evolved into a more complex system, they realized that they’d need more Docker images and container orchestration. That’s when they integrated Docker Compose into the setup.

Docker Compose simplified Docker image builds for each of Rapid7’s environments. It also encouraged a high level of service separation that split out different initialization steps into separate bounded contexts. Plus, they could leverage Docker Compose for inter-container communication, private networks, Docker volumes, defining environment variables with anchors, and linking containers for communication and aliasing.

This was a real game changer for Rapid7, because Docker Compose truly gave them unprecedented flexibility. Teams then added scripting to orchestrate communication between containers when a trigger event occurs (like when a service has completed).

Using Docker, Docker Compose, and scripting, Rapid7 was able to create a solution for the development team that could reliably replicate a complete development environment. To optimize the initialization, Rapid7 wanted to decrease the startup times beyond what Docker enables out of the box.

Optimizing build times even further

After creating Docker base images, the bottom layers rarely have to change. Essentially, that initial build is a one-time cost. Even if the images change, the cached layers make it a breeze to get through that process quickly. However, you do have to reinstall all software dependencies from scratch again, which is a one-time cost per Docker image update.

Committing the installed software dependencies back to the base image allows for a simple, incremental, and often skippable stage. The Docker image is always usable in development and production, all on the development computer.

All of these efficiencies together streamlined an already fast 15 minute process down to 5 minutes — making it easy for developers to get productive faster.

How to build it for yourself

Check out code examples and explanations about how to replicate this setup for yourself. We’ll now tackle the key steps you’ll need to follow to get started.

Downloading Docker

Download and install the latest version of Docker to be able to perform Docker-in-Docker. Docker-in-Docker lets your Docker environment have Docker installed within a container. This lets your container run other containers or pull images.

To enable Docker-in-Docker, you can apt install the docker.io distribution as one of your first commands in your Dockerfile. Once the container is configured, mount the Docker socket from the host installation:

# Dockerfile

FROM ubuntu:20.04

# Install dependencies

RUN apt update &&  \
           apt install -y docker.io

Next, build your Docker image by running the following command in your CLI or shell script file:

docker build -t <docker-image-name>

Then, start your Docker container with the following command:

docker run -v /var/run/docker.sock:/var/run/docker.sock -ti <docker-image-name>

Using a Docker commit script

Committing layered changes to your base image is what drives the core of the Dev Environments in Docker. Docker fetches the container ID based on the service name, and the changes you make to the running container are committed to the desired image. 

Because the host Docker socket is mounted into the container when executing the docker commit command, the container will apply the change to the base image located in the host Docker installation.

# ! /bin/bash

SERVICE=${1}
IMAGE=${2}

# Commit changes to image
CONTAINER_ID=$(docker ps -aqf “name=${SERVICE}”)

if [ ! -z “$CONTAINER_ID”]; then
	Echo “--- Committing changes from $SERVICE to $IMAGE --- ”
	docker commit $CONTAINER_ID $IMAGE
fi

Updating your environment

Mount the Docker socket from the host installation. Mounting the source code is insufficient without the :z property, which tells Docker that the content will be shared between containers.

You’ll have to mount the host machine’s Docker socket into the container. This lets any Docker operations performed inside the container actually modify the host Docker images and installation. Without this, changes made in the container are only going to persist in the container until it’s stopped and removed.

Add the following code into your Docker Compose file:

# docker-compose.yaml

services:
  service-name:
    image: image-with-docker:latest
    volumes:
        - /host/code/path:/container/code/path:z
        - /var/run/docker.sock:/var/run/docker.sock

Orchestrating components

Once Docker Compose has the appropriate services configured, you can start your environment in two different ways. Use either the docker-compose up command or start the environment by running the individual service with the linked services with the following command:

docker compose start webserver

The main container references the linked service via the linked names. This makes it very easy to override any environment variables with the provided names. Check out the YAML file below:

services:
  webserver:
  mysql:
    ports:
      - '3306:3306'
    volume
	 - dbdata:var/lib/mysql
  redis:
    ports:
      - 6379:6379
    volumes:
	 - redisdata:/data

volumes:
  dbdata:
  redisdata:

Notes: For each service, you’ll want to choose and specify your preferred Docker Official Image version. Additionally, the MySQL Docker Official Image comes with important environment variables defaulted in — though you can specify them here as needed.

Managing separate services

Starting a small part of the stack can also be useful if a developer only needs that specific piece. For example, if we just wanted to start the MySQL service, we’d run the following command:

docker compose start mysql

We can stop this service just as easily with the following command:

docker compose stop mysql

Configuring your environment

Mounting volumes into the database services lets your containers apply the change to their respective databases while letting those databases remain as ephemeral containers.

In the main entry point and script orchestrator, provide a -p attribute to ./start.sh to set the PROD_BUILD environment variable. The build reads the variable inside the entry point and optionally builds a production or development version of the development environment.  

First, here’s how that script looks:

# start.sh

while [ "$1" != ""];
do
	case $1 in
		-p |  --prod) PROD_BUILD="true";;

	esac
		shift
done

Second, here’s a sample shell script:

export PROD_BUILD=$PROD_BUILD

Third, here’s your sample Docker Compose file:

# docker-compose.yaml

services:
  build-frontend:
    entrypoint:
      - bash
      - -c
      - "[[ \"$PROD_BUILD\" == \"true\" ]] && make fe-prod || make fe-dev"

Note: Don’t forget to add your preferred image under build-frontend if you’re aiming to make a fully functional Docker Compose file.

What if we need to troubleshoot any issues that arise? Debugging inside a running container only requires the appropriate debugging library in the mounted source code and an open port to mount the debugger. Here’s our YAML file:

# docker-compose.yaml

services:
  webserver:
    ports:
	- '5678:5678'
    links:
	- mysql
	 - redis
	entrypoint:
      - bash
      - -c
      - ./start-webserver.sh

Note: Like in our previous examples, don’t forget to specify an image underneath webserver when creating a functional Docker Compose file.

In your editor of choice, provide a launch configuration to attach the debugger using the specified port. Once the container is running, run the configuration and the debugger will be attached:

#launch-setting.json
{
	"configurations" : [
		{
			"name":  "Python: Remote Attach",
			"type": "python",
			"request": "attach",
			"port": 5678,
			"host": "localhost",
			"pathMappings": [
				{
					"localRoot": "${workspaceFolder}",
					"remoteRoot": "."
				}
			]
		}
	]
}

Confirming that everything works

Once the full stack is running, it’s easy to access the main entry point web server via a browser on the defined webserver port.

The docker ps command will show your running containers. Docker is managing communication between containers. 

The entire software service is now running completely in Docker. All the code lives on the host computer inside the Docker container. The development environment is now completely portable using only Docker.

Remembering important tradeoffs

This approach has some limitations. First, running your developer environment in Docker will incur additional resource overhead. Docker has to run and requires extra computing resources as a result. Also, including multiple stages will require scripting as a final orchestration layer to enable communication between containers.

Wrapping up

Rapid7’s development team uses Docker to quickly create their development environments. They use the Docker CLI, Docker Desktop, Docker Compose, and shell scripts to create an extremely unique and robust Docker-friendly environment. They can use this to spin up any part of their development environment.

The setup also helps Rapid7 compile frontend assets, start cache and database servers, run the backend service with different parameters, or start the entire application stack. Using a “Docker-in-Docker” approach of mounting the Docker socket within running containers makes this possible. Docker’s ability to commit layers to the base image after dependencies are either updated or installed is also key. 

The shell scripts will export the required environment variables and then run specific processes in a specific order. Finally, Docker Compose makes sure that the appropriate service containers and dependencies are running.

Achieving future development goals

Relying on the Docker tool chain has been truly beneficial for Rapid7, since this has helped them create a consistent environment compatible with any part of their application stack. This integration has helped Rapid7 do the following: 

• Deploy extremely reliable software to advanced customer environments
• Analyze code before merging it in development
• Deliver much more stable code
• Simplify onboarding 
• Form an extremely flexible and configurable development environment

By using Docker, Rapid7 is continuously refining its processes to push past the boundaries of what’s possible. Their next goal is to deliver production-grade stable builds on a daily basis, and they’re confident that Docker can help them get there.