As the name suggests, docker COPY copies stuff from the host to the docker image, right? Right. Well, mostly right...
TLDR
You need to watch out for small details of how COPY in Dockerfile works. It is copying source directory contents and never the directory itself.
Debug story
Let me tell you a story of how to wasted a few hours debugging build problems in CI because of a small change in COPY.
Project structure
Let's assume we have a small PHP project.
.
├── bin
│ └── console
├── composer.json
├── composer.lock
├── Dockerfile
├── .dockerignore
├── .gitignore
└── README.md
composer.json contents:
{
"name": "blog/composer-copy",
"type": "project",
"require": {
"symfony/console": "^6.3"
},
"autoload": {
"psr-4": {
"Blog\\ComposerCopy\\": "src/"
}
},
"minimum-stability": "stable",
"require-dev": {
"vimeo/psalm": "^5.15"
},
"config": {
"platform": {
"php": "8.2.8"
}
}
}
Dockerfile contents:
FROM php:8.2.10-cli-bookworm
ARG DEBIAN_FRONTEND=noninteractive
ARG COMPOSER_ALLOW_SUPERUSER=1
RUN apt-get update && apt-get install unzip
# install composer
COPY --from=composer:2.6 /usr/bin/composer /usr/bin/composer
WORKDIR /app
COPY composer.json composer.lock ./
RUN composer install
COPY . .
CMD [ "php", "bin/console" ]
The project is building and working fine on local dev and in CI. All good so far.
Devs want a change
Devs don't want to have tools like vimeo/psalm in project dev dependencies, so their requirements will not interfere with app requirements. They will use tools/ directory to store all of them now. They even already updated and pushed modified composer.json files in the main project directory and tools/. Now a project looks like this:
.
├── bin
│ └── console
├── tools
│ └── composer.json
├── composer.json
├── composer.lock
├── Dockerfile
├── .dockerignore
├── .gitignore
└── README.md
But they have a problem because somehow in CI calling psalm using tools/vendor/bin/psalm does not work.
Let's fix it
There is no psalm because there is no instruction to install it in Dockerfile (there is only composer install for the main composer.json file). The COPY instruction for tools/ is also missing. New version of Dockerfile fixing this looks like this:
FROM php:8.2.8-cli-bookworm
ARG DEBIAN_FRONTEND=noninteractive
ARG COMPOSER_ALLOW_SUPERUSER=1
RUN apt-get update && apt-get install unzip
# install composer
COPY --from=composer:2.6 /usr/bin/composer /usr/bin/composer
WORKDIR /app
# copy composer config for app and tools
COPY composer.json composer.lock tools/ ./
# install app dependencies & tools
RUN composer install && composer --working-dir=tools/ install
COPY . .
CMD [ "php", "bin/console" ]
git add, commit, push and fail
All looks good in Dockerfile but now I get this weird error in CI.
=> ERROR [stage-0 6/7] RUN composer install && composer --working-dir=tools/ install 0.6s
------
> [stage-0 6/7] RUN composer install && composer --working-dir=tools/ install:
0.568 Installing dependencies from lock file (including require-dev)
0.571 Verifying lock file contents can be installed on current platform.
0.578 Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. It is recommended that you run `composer update` or `composer update <package name>`.
0.580 - Required package "vimeo/psalm" is not present in the lock file.
0.580 This usually happens when composer files are incorrectly merged or the composer.json file is manually edited.
0.580 Read more about correctly resolving merge conflicts https://getcomposer.org/doc/articles/resolving-merge-conflicts.md
0.580 and prefer using the "require" command over editing the composer.json file directly https://getcomposer.org/doc/03-cli.md#require-r
Quickly checking if composer install command works on local dev env shows no problems:
$ docker run --rm -it -v $PWD:/app --user $(id -u):$(id -g) blog-copy:dev composer install
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Nothing to install, update or remove
Generating autoload files
8 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
$ docker run --rm -it -v $PWD:/app --user $(id -u):$(id -g) blog-copy:dev composer --working-dir=tools/ install
No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
Loading composer repositories with package information
Cannot create cache directory /.composer/cache/repo/https---repo.packagist.org/, or directory is not writable. Proceeding without cache. See also cache-read-only config if your filesystem is read-only.
Info from https://repo.packagist.org: #StandWithUkraine
Updating dependencies
Lock file operations: 31 installs, 0 updates, 0 removals
- Locking amphp/amp (v2.6.2)
...
- Installing amphp/byte-stream (v1.8.1): Extracting archive
- Installing vimeo/psalm (5.15.0): Extracting archive
3 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating autoload files
17 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Tried some tricks to fix it like updating composer.lock on local dev env using composer update, removing and recreating the lock file from scratch etc. I even added --no-cache to docker build command because I thought it could be some broken docker cache (highly unlikely but still just in case).
Because RUN command in Dockerfile contains two composer install commands I decided to split them into separate RUN commands like this:
# install app dependencies
RUN composer install
# install tools
RUN composer --working-dir=tools/ install
I wanted to see which one is failing exactly (at first glance it should be the second composer install).
I've pushed a new Dockerfile and waited on CI to fail on tools installation. I was quite surprised when the first RUN failed with The lock file is not up to date with the latest changes in composer.json. It should be the second RUN that fails. I don't have psalm anywhere in composer.json or in a lock file.
What is going on?
Local debug
To not wait for CI each time, I decided to replicate the problem in local dev. To my even bigger surprise, I could not replicate the same error. The build failed on the second RUN instruction, which was expected, but with an unexpected error message:
$ docker build -t blog-copy:dev .
[+] Building 3.7s (13/14) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 507B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 49B 0.0s
=> [internal] load metadata for docker.io/library/php:8.2.8-cli-bookworm 1.0s
=> [internal] load metadata for docker.io/library/composer:2.6 1.0s
=> [internal] load build context 0.0s
=> => transferring context: 82.49kB 0.0s
=> [stage-0 1/8] FROM docker.io/library/php:8.2.8-cli-bookworm@sha256:6832edf1ccadcab4e627d98ce3d6262235fbd03d5b2409ca96988a1ea79fb6c9 0.0s
=> FROM docker.io/library/composer:2.6@sha256:fd0b4f28a5070d4361d5cbfe7f7807a10bf2efe9396e42009f9e63f7fa307e38 0.0s
=> CACHED [stage-0 2/8] RUN apt-get update && apt-get install unzip 0.0s
=> CACHED [stage-0 3/8] COPY --from=composer:2.6 /usr/bin/composer /usr/bin/composer 0.0s
=> CACHED [stage-0 4/8] WORKDIR /app 0.0s
=> [stage-0 5/8] COPY composer.json composer.lock tools/ ./ 0.0s
=> [stage-0 6/8] RUN composer install 2.2s
=> ERROR [stage-0 7/8] RUN composer --working-dir=tools/ install 0.4s
------
> [stage-0 7/8] RUN composer --working-dir=tools/ install:
0.351
0.355 In Application.php line 430:
0.355
0.355 Invalid working directory specified, tools/ does not exist.
0.355
0.355
------
Dockerfile:20
--------------------
18 |
19 | # install tools
20 | >>> RUN composer --working-dir=tools/ install
21 |
22 | COPY . .
--------------------
ERROR: failed to solve: process "/bin/sh -c composer --working-dir=tools/ install" did not complete successfully: exit code: 1
Maybe I have something different on local dev env somehow? Git status showed only that tools/composer.lock was untracked, so I removed it. Now build is failing on the first RUN. Finally, some "success".
That gave me a hint that something was not right with the files inside the image when I tried to install dependencies. To check this, I commented everything after COPY instruction.
FROM php:8.2.8-cli-bookworm
ARG DEBIAN_FRONTEND=noninteractive
ARG COMPOSER_ALLOW_SUPERUSER=1
RUN apt-get update && apt-get install unzip
# install composer
COPY --from=composer:2.6 /usr/bin/composer /usr/bin/composer
WORKDIR /app
# copy composer config for app and tools
COPY composer.json composer.lock tools/ ./
# install app dependencies
#RUN composer install
# install tools
#RUN composer --working-dir=tools/ install
#COPY . .
#CMD [ "php", "bin/console" ]
The build is now successful:
$ docker build -t blog-copy:dev .
[+] Building 0.9s (12/12) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 510B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 49B 0.0s
=> [internal] load metadata for docker.io/library/composer:2.6 0.8s
=> [internal] load metadata for docker.io/library/php:8.2.8-cli-bookworm 0.8s
=> [stage-0 1/5] FROM docker.io/library/php:8.2.8-cli-bookworm@sha256:6832edf1ccadcab4e627d98ce3d6262235fbd03d5b2409ca96988a1ea79fb6c9 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 172B 0.0s
=> FROM docker.io/library/composer:2.6@sha256:fd0b4f28a5070d4361d5cbfe7f7807a10bf2efe9396e42009f9e63f7fa307e38 0.0s
=> CACHED [stage-0 2/5] RUN apt-get update && apt-get install unzip 0.0s
=> CACHED [stage-0 3/5] COPY --from=composer:2.6 /usr/bin/composer /usr/bin/composer 0.0s
=> CACHED [stage-0 4/5] WORKDIR /app 0.0s
=> CACHED [stage-0 5/5] COPY composer.json composer.lock tools/ ./ 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:d6072717482abdd61bc3ebd5ae67b356270e1fc161a6803f5455ff0a7da68696 0.0s
=> => naming to docker.io/library/blog-copy:dev
I checked what is inside the image:
$ docker run --rm blog-copy:dev ls -la
total 88
drwxr-xr-x 1 root root 4096 Oct 5 06:50 .
drwxr-xr-x 1 root root 4096 Oct 6 06:28 ..
-rw-r--r-- 1 root root 49 Oct 3 19:38 composer.json
-rw-r--r-- 1 root root 74830 Oct 5 06:30 composer.lock
Why is tools/ directory missing!? I've checked composer.json:
$ docker run --rm blog-copy:dev cat composer.json
{
"require": {
"vimeo/psalm": "^5.15"
}
}
And at that moment I remembered something. A long time ago when reading Dockerfile spec I noticed that they mentioned that COPY works differently than standard Linux cp. I've checked again and yeah...
COPY copies contents of a directory and never the directory itself
They mention it in a note:
Note
The directory itself is not copied, just its contents.
The working version
Working Dockerfile looks like this:
FROM php:8.2.8-cli-bookworm
ARG DEBIAN_FRONTEND=noninteractive
ARG COMPOSER_ALLOW_SUPERUSER=1
RUN apt-get update && apt-get install unzip
# install composer
COPY --from=composer:2.6 /usr/bin/composer /usr/bin/composer
WORKDIR /app
# copy composer config for app
COPY composer.json composer.lock ./
# copy tools
COPY tools/ tools/
# install app dependencies
RUN composer install && composer --working-dir=tools/ install
COPY . .
CMD [ "php", "bin/console" ]
The build is now working:
$ docker build -t blog-copy:dev .
[+] Building 5.1s (15/15) FINISHED docker:default
=> [internal] load .dockerignore 0.0s
=> => transferring context: 49B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 505B 0.0s
=> [internal] load metadata for docker.io/library/composer:2.6 0.4s
=> [internal] load metadata for docker.io/library/php:8.2.8-cli-bookworm 0.4s
=> [stage-0 1/8] FROM docker.io/library/php:8.2.8-cli-bookworm@sha256:6832edf1ccadcab4e627d98ce3d6262235fbd03d5b2409ca96988a1ea79fb6c9 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 784B 0.0s
=> FROM docker.io/library/composer:2.6@sha256:fd0b4f28a5070d4361d5cbfe7f7807a10bf2efe9396e42009f9e63f7fa307e38 0.0s
=> CACHED [stage-0 2/8] RUN apt-get update && apt-get install unzip 0.0s
=> CACHED [stage-0 3/8] COPY --from=composer:2.6 /usr/bin/composer /usr/bin/composer 0.0s
=> CACHED [stage-0 4/8] WORKDIR /app 0.0s
=> [stage-0 5/8] COPY composer.json composer.lock ./ 0.0s
=> [stage-0 6/8] COPY tools/ tools/ 0.0s
=> [stage-0 7/8] RUN composer install && composer --working-dir=tools/ install 3.5s
=> [stage-0 8/8] COPY . . 0.0s
=> exporting to image 1.0s
=> => exporting layers 1.0s
=> => writing image sha256:750cc2aec8f79312d0ade99841dd6d96588b2430d79149fca821f40abe77d535 0.0s
=> => naming to docker.io/library/blog-copy:dev