Pulumi is an Infrastructure as Code (IaC) platform that simplifies resource management across any cloud or SaaS provider, including Docker. Pulumi providers are integrations with useful tools and vendors. Pulumi’s new Docker Build provider is about making your builds even easier, faster, and more reliable. 

In this post, we will dive into how Pulumi’s new Docker Build provider works with Docker Build Cloud to streamline building, deploying, and managing containerized applications. First, we’ll set up a project using Docker Build Cloud and Pulumi. Then, we’ll explore cool use cases that showcase how you can leverage this provider to simplify your build and deployment pipelines.  

Pulumi Docker Build provider features

Top features of the Pulumi Docker Build provider include the following:

• Docker Build Cloud support: Offload your builds to the cloud and free up your local resources. Faster builds mean fewer headaches.
• Multi-platform support: Build Docker images that work on different hardware architectures without breaking a sweat.
• Advanced caching: Say goodbye to redundant builds. In addition to the shared caching available when you use Docker Build Cloud, this provider supports multiple cache backends, like Amazon S3, GitHub Actions, and even local disk, to keep your builds efficient.
• Flexible export options: Customize where your Docker images go after they’re built — export to registries, filesystems, or wherever your workflow needs.

Getting started with Docker Build Cloud and Pulumi  

Docker Build Cloud is Docker’s newest offering that provides a pair of AMD and Arm builders in the cloud and shared cache for your team, resulting in up to 39x faster image builds. Docker Personal, Pro, Team, and Business plans include a set number of Build Cloud minutes, or you can purchase a Build Cloud Team plan to add minutes. Learn more about Docker Build Cloud plans. 

The example builds an NGINX Dockerfile using a Docker Build Cloud builder. We will create a Docker Build Cloud builder, create a Pulumi program in Typescript, and build our image.

Prerequisites

• Docker
• Pulumi CLI
• Pulumi Cloud account (Individual free tier works)
• A Pulumi-supported language
• Docker Build Cloud Team plan or a Docker Core Subscription with available Docker Build Cloud minutes

Step 1: Set up your Docker Build Cloud builder

Building images locally means being subject to local compute and storage availability. Pulumi allows users to build images with Docker Build Cloud.

The Pulumi Docker Build provider fully supports Docker Build Cloud, which unlocks new capabilities, as individual team members or a CI/CD pipeline can fully take advantage of improved build speeds, shared build cache, and native multi-platform builds.

If you still need to create a builder, follow the steps below; otherwise, skip to step 1C.

A. Log in to your Docker Build Cloud account.

B. Create a new cloud builder named my-cool-builder. 

C. In your local machine, sign in to your Docker account.

$ docker login

D. Add your existing cloud builder endpoint.

$ docker buildx create --driver cloud ORG/BUILDER_NAME
# Replace ORG with the Docker Hub namespace of your Docker organization. 
# This creates a builder named cloud-ORG-BUILDER_NAME.

# Example:
$ docker buildx create --driver cloud pulumi/my-cool-builder
# cloud-pulumi-my-cool-builder

# check your new builder is configured
$ docker buildx ls

E. Optionally, see that your new builder is available in Docker Desktop.

For additional guidance on setting up Docker Build Cloud, refer to the Docker docs.

Step 2: Set up your Pulumi project

To create your first Pulumi project, start with a Pulumi template. Pulumi has curated hundreds of templates that are directly integrated with the Pulumi CLI via pulumi new. In particular, the Pulumi team has created a Pulumi template for Docker Build Cloud to get you started.

The Pulumi programming model centers around defining infrastructure using popular programming languages. This approach allows you to leverage existing programming tools and define cloud resources using familiar syntaxes such as loops and conditionals.

To copy the Pulumi template locally:

$ pulumi new https://github.com/pulumi/examples/tree/master/dockerbuildcloud-ts --dir hello-dbc
# project name: hello-dbc 
# project description: (default)
# stack name: dev
# Note: Update the builder value to match yours
# builder: cloud-pulumi-my-cool-builder 
$ cd hello-dbc

# update all npm packages (recommended)
$ npm update --save

Optionally, explore your Pulumi program. The hello-dbc folder has everything you need to build a Dockerfile into an image with Pulumi. Your Pulumi program starts with an entry point, typically a function written in your chosen programming language. This function defines the infrastructure resources and configurations for your project. For TypeScript, that file is index.ts, and the contents are shown below:

import * as dockerBuild from "@pulumi/docker-build";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();
const builder = config.require("builder");

const image = new dockerBuild.Image("image", {

   // Configures the name of your existing buildx builder to use.
   // See the Pulumi.<stack>.yaml project file for the builder configuration.
   builder: {
       name: builder, // Example, "cloud-pulumi-my-cool-builder",
   },
   context: {
       location: "app",
   },
   // Enable exec to run a custom docker-buildx binary with support
   // for Docker Build Cloud (DBC).
   exec: true,
   push: false,
});

Step 3: Build your Docker image

Run the pulumi up command to see the image being built with the newly configured builder:

$ pulumi up --yes

You can follow the browser link to the Pulumi Cloud dashboard and navigate to the Image resource to confirm it’s properly configured by noting the builder parameter.

Optionally, also check your Docker Build Cloud dashboard for build minutes usage:

Congratulations! You have built an NGINX Dockerfile with Docker Build Cloud and Pulumi. This was achieved by creating a new Docker Build Cloud builder and passing that to a Pulumi template. The Pulumi CLI is then used to deploy the changes.

Advanced use cases with buildx and BuildKit

To showcase popular buildx and BuildKit features, test one or more of the following Pulumi code samples. These include multi-platform, advanced caching,  and exports. Note that each feature is available as an input (or parameter) in the Pulumi Docker Build Image resource. 

Multi-platform image builds for Docker Build Cloud

Docker images can support multiple platforms, meaning a single image may contain variants for architectures and operating systems. 

The following code snippet is analogous to invoking a build from the Docker CLI with the --platform flag to specify the target platform for the build output.

import * as dockerBuild from "@pulumi/docker-build";

const image = new dockerBuild.Image("image", {
   // Build a multi-platform image manifest for ARM and AMD.
   platforms: [
       dockerBuild.Platform.Linux_amd64,
       dockerBuild.Platform.Linux_arm64,
   ],
   push: false,

});

Deploy the changes made to the Pulumi program:

$ pulumi up --yes

Caching from and to AWS ECR

Maintaining cached layers while building Docker images saves precious time by enabling faster builds. However, utilizing cached layers has been historically challenging in CI/CD pipelines due to recycled environments between builds. The cacheFrom and cacheTo parameters allow programmatic builds to optimize caching behavior. 

Update your Docker image resource to take advantage of caching:

import * as dockerBuild from "@pulumi/docker-build";
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws"; // Required for ECR

// Create an ECR repository for pushing.
const ecrRepository = new aws.ecr.Repository("ecr-repository", {});

// Grab auth credentials for ECR.
const authToken = aws.ecr.getAuthorizationTokenOutput({
   registryId: ecrRepository.registryId,
});

const image = new dockerBuild.Image("image", {
   push: true,
   // Use the pushed image as a cache source.
   cacheFrom: [{
       registry: {
           ref: pulumi.interpolate`${ecrRepository.repositoryUrl}:cache`,
       },
   }],
   cacheTo: [{
       registry: {
           imageManifest: true,
           ociMediaTypes: true,
           ref: pulumi.interpolate`${ecrRepository.repositoryUrl}:cache`,
       },
   }],
   // Provide our ECR credentials.
   registries: [{
       address: ecrRepository.repositoryUrl,
       password: authToken.password,
       username: authToken.userName,
   }],
})

Notice the declaration of additional resources for AWS ECR. 

Export builds as a tar file

Exporting allows us to share or back up the resulting image from a build invocation. 

To export the build as a local .tar file, modify your resource to include the exports Input:

const image = new dockerBuild.Image("image", {
   push: false,
   exports: [{
       docker: {
           tar: true,
       },
   }],
})

Deploy the changes made to the Pulumi program:

$ pulumi up --yes

Review the Pulumi Docker Build provider guide to explore other Docker Build features, such as build arguments, build contexts, remote contexts, and more.

Next steps

Infrastructure as Code (IaC) is key to managing modern cloud-native development, and Docker lets developers create and control images with Dockerfiles and Docker Compose files. But when the situation gets more complex, like deploying across different cloud platforms, Pulumi can offer additional flexibility and advanced infrastructure features. The Docker Build provider supports Docker Build Cloud, streamlining building, deploying, and managing containerized applications, which helps development teams work together more effectively and maintain agility.

Pulumi’s latest Docker Build provider, powered by BuildKit, improves flexibility and efficiency in Docker builds. By applying IaC principles, developers manage infrastructure with code, even in intricate scenarios. This means you can focus on building and deploying your containerized workloads without the hassle of complex infrastructure challenges.

Visit Pulumi’s launch announcement and the provider documentation to get started with the Docker Build provider. 

Register for the June 25 Pulumi and Docker virtual workshop: Automating Docker Image Builds using Pulumi and Docker.

Learn more

• Get started with the Pulumi Docker Build provider.
• Read the Pulumi Docker Build provider documentation.
• Read the Docker Build documentation.
• Learn about Docker Build Cloud.
• See Docker Build Cloud pricing.
• Find out about Docker Core Subscriptions.

Along with our investments in bringing access to cloud resources within the local Docker Desktop experience with Docker Build Cloud Builds view, this release provides a more efficient and flexible platform for development teams.

Introducing enhanced file-sharing controls in Docker Desktop Business 

As we continue to innovate and elevate the Docker experience for our business customers, we’re thrilled to unveil significant upgrades to the Docker Desktop’s Hardened Desktop feature. Recognizing the importance of administrative control over Docker Desktop settings, we’ve listened to your feedback and are introducing enhancements prioritizing security and ease of use.

For IT administrators and non-admin users, Docker now offers the much-requested capability to specify and manage file-sharing options directly via Settings Management (Figure 1). This includes:

• Selective file sharing: Choose your preferred file-sharing implementation directly from Settings > General, where you can choose between VirtioFS, gRPC FUSE, or osxfs. VirtioFS is only available for macOS versions 12.5 and above and is turned on by default.
• Path allow-listing: Precisely control which paths users can share files from, enhancing security and compliance across your organization.

We’ve also reimagined the Settings > Resources > File Sharing interface to enhance your interaction with Docker Desktop (Figure 2). You’ll notice:

• Clearer error messaging: Quickly understand and rectify issues with enhanced error messages.
• Intuitive action buttons: Experience a smoother workflow with redesigned action buttons, making your Docker Desktop interactions as straightforward as possible.

These enhancements are not just about improving current functionalities; they’re about unlocking new possibilities for your Docker experience. From increased security controls to a more navigable interface, every update is designed with your efficiency in mind.

Refining development with Docker Desktop’s Builds view update 

Docker Desktop’s previous update introduced Docker Build Cloud integration, aimed at reducing build times and improving build management. In this release, we’re landing incremental updates that refine the Builds view, making it easier and faster to manage your builds.

New in Docker Desktop 4.28:

• Dedicated tabs: Separates active from completed builds for better organization (Figure 3).
• Build insights: Displays build duration and cache steps, offering more clarity on the build process.
• Reliability fixes: Resolves issues with updates for a more consistent experience.
• UI improvements: Updates the empty state view for a clearer dashboard experience (Figure 4).

These updates are designed to streamline the build management process within Docker Desktop, leveraging Docker Build Cloud for more efficient builds.

To explore how Docker Desktop and Docker Build Cloud can optimize your development workflow, read our Docker Build Cloud blog post. Experience the latest Builds view update to further enrich your local, hybrid, and cloud-native development journey.

These Docker Desktop updates support improved platform security and a better user experience. By introducing more detailed file-sharing controls, we aim to provide developers with a more straightforward administration experience and secure environment. As we move forward, we remain dedicated to refining Docker Desktop to meet the evolving needs of our users and organizations, enhancing their development workflows and agility to innovate.

Join the conversation and make your mark

Dive into the dialogue and contribute to the evolution of Docker Desktop. Use our feedback form to share your thoughts and let us know how to improve the Hardened Desktop features. Your input directly influences the development roadmap, ensuring Docker Desktop meets and exceeds our community and customers’ needs.

Learn more

• Authenticate and update to receive the newest Docker Desktop features per your subscription level.
• New to Docker? Create an account. 
• Read our latest blog on synchronized file shares.
• Read about what rolled out in Docker Desktop 4.27, including synchronized file shares, Docker Init GA, a private marketplace for extensions, Moby 25, support for Testcontainers with ECI, Docker Build Cloud, and Docker Debug Beta.
• Learn about Docker Build Cloud.
• Subscribe to the Docker Newsletter.

Additionally, we explore the support for Testcontainers with Enhanced Container Isolation, the general availability of docker init with expanded language support, and the beta release of Docker Debug. These updates represent significant strides in improving development workflows, enhancing security, and offering advanced customization for Docker users.

Docker Desktop synchronized file shares GA

We’re diving into some fantastic updates for Docker Desktop, and we’re especially thrilled to introduce our latest feature, synchronized file shares, which is available now in version 4.27 (Figure 1). Following our acquisition announcement in June 2023, we have integrated the technology behind Mutagen into the core of Docker Desktop.

You can now say goodbye to the challenges of using large codebases in containers with virtual filesystems. Synchronized file shares unlock native filesystem performance for bind mounts and provides a remarkable 2-10x boost in file operation speeds. For developers managing extensive codebases, this is a game-changer.

To get started, log in to Docker Desktop with your subscription account (Pro, Teams, or Business) to harness the power of Docker Desktop synchronized file shares. You can read more about this feature in the Docker documentation.

Collaborate on shared Docker Build Cloud builds in Docker Desktop

With the recent GA of Docker Build Cloud, your team can now leverage Docker Desktop to use powerful cloud-based build machines and shared caching to reduce unnecessary rebuilds and get your build done in a fraction of the time, regardless of your local physical hardware.

New builds can make instant use of the shared cache. Even if this is your first time building the project, you can immediately speed up build times with shared caches.

We know that team members have varying levels of Docker expertise. When a new developer has issues with their build failing, the Builds view makes it effortless for anyone on the team to locate the troublesome build using search and filtering. They can then collaborate on a fix and get unblocked in no time.

When all your team is building on the same cloud builder, it can get noisy, so we added filtering by specific build types, helping you focus on the builds that are important to you.

Link to builder settings for a build

Previously, to access builder settings, you had to jump back to the build list or the settings page, but now you can access them directly from a build (Figure 2).

Delete build history for a builder

And, until now you could only delete build in batches, which meant if you wanted to clear the build history it required a lot of clicks. This update enables you to clear all builds easily (Figure 3).

Refresh storage data for your builder at any point in time

Refreshing the storage data is an intensive operation, so it only happens periodically. Previously, when you were clearing data, you would have to wait a while to see the update. Now it’s just a one-click process (Figure 4).

New feature: Private marketplace for extensions available for Docker Business subscribers

Docker Business customers now have exclusive access to a new feature: the private marketplace for extensions. This enhancement focuses on security, compliance, and customization, and empowering developers, providing:

• Controlled access: Manage which extensions developers can use through allow-listing.
• Private distribution: Easily distribute company-specific extensions from a private registry.
• Customized development: Deploy customized team processes and tools as unpublished/private Docker extensions tailored to a specific organization.

The private marketplace for extensions enables a secure, efficient, and tailored development environment, aligning with your enterprise’s specific needs. Get started today by learning how to configure a private marketplace for extensions.

Moby 25 release — containerd image store 

We are happy to announce the release of Moby 25.0 with Docker Desktop 4.27. In case you’re unfamiliar, Moby is the open source project for Docker Engine, which ships in Docker Desktop. We have dedicated significant effort to this release, which marks a major release milestone for the open source Moby project. You can read a comprehensive list of enhancements in the v25.0.0 release notes.

With the release of Docker Desktop 4.27,  support for the containerd image store has graduated from beta to general availability. This work began in September 2022 when we started extending the Docker Engine integration with containerd, so we are excited to have this functionality reach general availability.

This support provides a more robust user experience by natively storing and building multi-platform images and using snapshotters for lazy pulling images (e.g., stargz) and peer-to-peer image distribution (e.g., dragonfly, nydus). It also provides a foundation for you to run Wasm containers (currently in beta). 

Using the containerd image store is not currently enabled by default for all users but can be enabled in the general settings in Docker Desktop under Use containers for pulling and storing images (Figure 5).

Going forward, we will continue improving the user experience of pushing, pulling, and storing images with the containerd image store, help migrate user images to use containerd, and work toward enabling it by default for all users. 

As always, you can try any of the features landing in Moby 25 in Docker Desktop.

Support for Testcontainers with Enhanced Container Isolation

Docker Desktop 4.27 introduces the ability to use the popular Testcontainers framework with Enhanced Container Isolation (ECI). 

ECI, which is available to Docker Business customers, provides an additional layer of security to prevent malicious workloads running in containers from compromising the Docker Desktop or the host by running containers without root access to the Docker Desktop VM, by vetting sensitive system calls inside containers and other advanced techniques. It’s meant to better secure local development environments. 

Before Docker Desktop 4.27, ECI blocked mounting the Docker Engine socket into containers to increase security and prevent malicious containers from gaining access to Docker Engine. However, this also prevented legitimate scenarios (such as Testcontainers) from working with ECI.   

Starting with Docker Desktop 4.27, admins can now configure ECI to allow Docker socket mounts, but in a controlled way (e.g., on trusted images of their choice) and even restrict the commands that may be sent on that socket. This functionality, in turn, enables users to enjoy the combined benefits of frameworks such as Testcontainers (or any others that require containers to access the Docker engine socket) with the extra security and peace of mind provided by ECI.

Docker init GA with Java support 

Initially released in its beta form in Docker 4.18, docker init has undergone several enhancements. The docker init command-line utility aids in the initialization of Docker resources within a project. It automatically generates Dockerfiles, Compose files, and .dockerignore files based on the nature of the project, significantly reducing the setup time and complexity associated with Docker configurations. 

The initial beta release of docker init only supported Go and generic projects. The latest version, available in Docker 4.27, supports Go, Python, Node.js, Rust, ASP.NET, PHP, and Java (Figure 6).

The general availability of docker init offers an efficient and user-friendly way to integrate Docker into your projects. Whether you’re a seasoned Docker user or new to containerization, docker init is ready to enhance your development workflow. 

Beta release of Docker Debug 

As previously announced at DockerCon 2023, Docker Debug is now available as a beta offering in Docker Desktop 4.27.

Developers can spend as much as 60% of their time debugging their applications, with much of that time taken up by sorting and configuring tools and setup instead of debugging. Docker Debug (available in Pro, Teams, or Business subscriptions) provides a language-independent, integrated toolbox for debugging local and remote containerized apps — even when the container fails to launch — enabling developers to find and solve problems faster.

To get started, run docker debug <Container or Image name> in the Docker Desktop CLI while logged in with your subscription account.

Conclusion

Docker Desktop’s latest updates and features, from synchronized file shares to the first beta release of Docker Debug, reflect our ongoing commitment to enhancing developer productivity and operational efficiency. Integrating these capabilities into Docker Desktop streamlines development processes and empowers teams to collaborate more effectively and securely. As Docker continues to evolve, we remain dedicated to providing our community and customers with innovative solutions that address the dynamic needs of modern software development.

Stay tuned for further updates and enhancements, and as always, we encourage you to explore these new features to see how they can benefit your development workflow.

Upgrade to Docker Desktop 4.27 to explore these updates and experiment with Docker’s latest features.

Learn more

• Read the Docker Desktop Release Notes.
• Install and authenticate against the latest release of Docker Desktop.
• Learn more about synchronized file shares.
• Check out Docker Build Cloud.
• Read Streamline Dockerization with Docker Init GA
• Read Docker Init: Initialize Dockerfiles and Compose files with a single CLI command.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

We’re developers, too, so we explored ways to speed up build time without sacrificing the local development experience we love. That’s how Docker Build Cloud was born. 

“The new products we announced meet development teams where they are with ‘just enough cloud,’ seamlessly blurring the boundaries between local and remote development. In doing so, we’re enabling these teams to accelerate their delivery of secure applications critical to their businesses.” — Giri Sreenivas, Chief Product Officer, Docker

Temporary fixes don’t last 

Developers and their employers have attempted to address resource constraints that slow build times and drain productivity in a few common ways, the Incredibuild study says, including upgrading hardware (cited by 44% of respondents) and reducing codebase size (acknowledged by 33%). Although these tactics offer a temporary boost, they’re far from sustainable. A better solution is required — one that speeds up build times and improves team collaboration by eliminating redundant builds. 

Build up to 39x faster with Docker Build Cloud 

In the constantly evolving landscape of software development, two investment areas continue to trend upwards: moving development to the cloud and accelerating builds. The concept behind Docker Build Cloud is simple. By leveraging cloud compute and cache, developers can build faster and improve collaboration with their team. 

Building in the cloud speeds up build times because the cloud provides access to faster compute resources than are available on a developer’s local machine, and this approach provides more consistency between developers who may have newer or older machines. 

Shared cache speeds up build times because when one team member initiates a build, the cached results become instantly accessible to others, thereby eliminating unnecessary builds and speeding up the development cycle. No more waiting around for each build to complete independently. 

The combination of building in the cloud and shared cache helps developers save time and improve productivity. Developers can get back to coding on parallel tasks while builds complete, and they get the results of builds back faster to incorporate into their work.  

For example, one of our technology customers who develops enterprise collaboration software was able to reduce their build time from an average duration of 15-20 minutes to less than 2 minutes using Docker Build Cloud. 

Multi-architecture builds made effortless

Today, developers who need to create applications for both Intel (AMD64) and Apple Silicon/AWS Graviton (Arm64) chipsets must have multiple native builders or configure slow emulators to successfully build for their deployment targets. Docker Build Cloud offers native support for multi-architecture builds, eliminating the need for setting up and maintaining multiple native builders. This support removes the challenges associated with emulation, further improving build efficiency.

An e-commerce customer simplified their CI toolchain. Prior to Docker Build Cloud, they were leveraging GitHub Actions, GitLab runners, plus a custom GitLab runner to handle ARM architecture. Due to Build Cloud’s dual AMD and ARM builders, our customer reduced their complexity and sped up their pipelines. 

Seamless integration with the tools you love

Developer tools should enhance developer experience, not add new points of friction. We’ve designed Docker Build Cloud to be easy to set up wherever you run your builds without requiring a massive lift-and-shift effort. Docker Build Cloud also works well with Docker Compose, GitHub Actions, and other CI solutions. This means you can seamlessly incorporate Docker Build Cloud into your existing development tools and services and immediately start reaping the benefits of enhanced speed and efficiency.

Try Docker Build Cloud today

Docker Build Cloud will enable a faster, more efficient Docker image-building process. Whether you’re a seasoned developer or just getting started, embrace the power of the cloud in your local development environment. Welcome to the future of Docker image builds with Docker Build Cloud. 

Try Docker Build Cloud now. 

Learn more

• See the video demo Introducing Docker Build Cloud.
• Watch the webinar Docker Build Cloud: Reclaim Your Dev Time with Fast, Multi-Architecture Builds.
• Get the latest release of Docker Desktop.
• Vote on what’s next! Check out our public roadmap.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.