Fix forthcoming via https://github.com/compose-spec/compose-go/pull/436
which addresses some symlink limitations. These can
actually effect other platforms but are most common
on macOS because the test creates temporary directories,
which are symlinked on macOS.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Lots of our phony Compose files launch pointless long-lived processes
so we can assert on state. However, this means they often don't respond
well to signals on their own, requiring Compose to timeout and kill
them when doing a `down`.
Add in lots of `init: true` where appropriate so that we don't block
for no reason while running E2E tests all over the place.
Additionally, a couple tests have gotten a cleanup so they don't leave
behind containers. I still want to build this into the framework in
the future, but this is easier for the moment and won't cause any
trouble in the future.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Add an end-to-end test that covers the core watch functionality,
i.e. CRUD on files & directories.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
* Run `down` before and after test to not leave around containers
* Kill the `wait` process that's waiting on `infinity`
* NOTE: If the test is actually working, this should exit once
the `down` happens, but this ensures that we kill everything
we start
I'd like to generalize more of this into the framework, but this
is a quick fix to prevent filling up CI machines with tons of
processes over time.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
closes#10783
Compose Spec mentions that default values for secrets is `0444` aka. world-readable permissions. However, the value was previously set to `0400`.
Signed-off-by: Shan Desai <shantanoo.desai@gmail.com>
As part of the fix for #10668, the logic was adjusted so that the
default (highest-priority) network is used in the `ContainerCreate`,
and then the remaining networks are connected via calls to
`NetworkConnect` before starting the container.
Unfortunately, `ServiceConfig::NetworksByPriority` is neither
deterministic nor stable when networks have the same priority.
It's non-deterministic because the order of networks from parsing
YAML is random, since they are loaded into a Go map (which have
random iteration order). Additionally, it's not using a `SortStable`
in `compose-go`, so even if the load order was predictable, it
still might produce different results.
While I look at improving `compose-go` here to prevent this from
tripping us up in the future, this fix looks at _all_ networks for
a service and ignores the "default" one now. Before, it would
always skip the first one in the slice since that _should_ have
been the "default".
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Engine API only allows at most one network to be connected as
part of the ContainerCreate API request. Compose will pick the
highest priority network.
Afterwards, the remaining networks (if any) are connected before
the container is actually started.
The big change here is that, previously, the highest-priority
network was connected in the create, and then disconnected and
immediately reconnected along with all the others. This was
racy because evidently connecting the container to the network
as part of the create isn't synchronous, so sometimes when Compose
tried to disconnect it, the API would return an error like:
```
container <id> is not connected to the network <network>
```
To avoid needing to disconnect and immediately reconnect, the
network config logic has been refactored to ensure that it sets
up the network config correctly the first time.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Attempting to fix the state of codecov action checks right now,
which are behaving very erratically.
Using the new functionality in Go 1.20 to merge multiple reports,
so now the unit & E2E coverage data reports are stored as artifacts
and then downloaded, merged, and finally uploaded to codecov as a
new job.
Additionally, add a `codecov.yml` config and try to turn down the
aggressiveness of it for CI checks.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Previously, this was telling us "but was not created for project
[project-it-was-created-for]", which is wrong. I opted to make the
message super explicit and print both the actual project and the
expected project.
Signed-off-by: Jeremy Fleischman <jeremyfleischman@gmail.com>
Write the warning using `logrus.Warn`. The function being used was
coming from `cfssl`'s log package, which was presumably the result
of auto-import being _slightly_ too aggressive.
(Note: `cfssl` is still an indirect dependency after this.)
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Some error messages have been tweaked slightly, this adapts the
assertions to work on both Engine v20.10.x and v23.x.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
If we go to inspect a container that we got an event for and it
no longer exists on the server, handle clean up without erroring
out.
Fixes#10373.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
The IndexServerAddress field was as part of the initial Windows implementation
of the engine. For legal reasons, Microsoft Windows (and thus Docker images
based on Windows) were not allowed to be distributed through non-Microsoft
infrastructure. As a temporary solution, a dedicated "registry-win-tp3.docker.io"
registry was created to serve Windows images.
Using separate registries was not an ideal solution, and a more permanent
solution was created by introducing "foreign image layers" in the distribution
spec, after which the "registry-win-tp3.docker.io" ceased to exist, and
removed from the engine.
This replaces the code that calls out to the "/info" endpoint to use the
GetAuthConfigKey() function instead.
Related PR in docker/cli:
b4ca1c7368
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use latest Go minor release. Note: this release included fixes for
several CVEs, but they do not impact Compose.
Small errors have been fixed to keep the linter happy.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
This was running two tests in parallel that would build/delete the
same images. Run in serial instead since that's not safe.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Was getting segfaults with multiple services using
`x-develop` and `watch` at the same time. Turns out
the Moby path matcher lazily initializes the regex
pattern internally the first time it's used, so it's
not goroutine-safe.
Change here is to not use a global instance for the
ephemeral path matcher, but a per-watcher instance.
Additionally, the data race detector caught a couple
other issues that were easy enough to fix:
* Use the lock that's used elsewhere for convergence
before manipulating
* Eliminate concurrent map access when triggering
rebuilds
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
This approach mimics Tilt's behavior[^1]:
1. At sync time, `stat` the path on host
2. If the path does not exist -> `rm` from container
3. If the path exists -> sync to container
By handling things this way, we're always syncing based on the true
state, regardless of what's happened in the interim. For example, a
common pattern in POSIX tools is to create a file and then rename it
over an existing file. Based on timing, this could be a sync, delete,
sync (every file gets seen & processed) OR a delete, sync (by the
the time we process the event, the "temp" file is already gone, so
we just delete it from the container, where it never existed, but
that's fine since we deletes are idempotent thanks to the `-f` flag
on `rm`).
Additionally, when syncing, if the `stat` call shows it's for a
directory, we ignore it. Otherwise, duplicate, nested copies of the
entire path could get synced in. (On some OSes, an event for the
directory gets dispatched when a file inside of it is modified. In
practice, I think we might want this pushed further down in the
watching code, but since we're already `stat`ing the paths here now,
it's a good place to handle it.)
Lastly, there's some very light changes to the text when it does a
full rebuild that will list out the (merged) set of paths that
triggered it. We can continue to improve the output, but this is
really helpful for understanding why it's rebuilding.
[^1]: db7f887b06/internal/controllers/core/liveupdate/reconciler.go (L911)
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
This test keeps failing with a timeout in Windows. I don't actually
think it should take that long to bring up an nginx container, so
I'm guessing that there's something else going on that's causing
trouble.
Increase the verbosity when running Compose commands: I think this
will generally make E2E test failures easier to diagnose by always
logging the full command that's going to be run and also capturing
stdout.
Add a health check and use `--wait` when launching the fixture for
the pause test. Combined with the verbosity increase, this should
make it easier to understand what's going on here.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Big change here is to import the ephemeral ignore set from Tilt.
The `.git` directory is also ignored for now: this restriction
should probably be lifted and made configurable in the future,
but it's not generally important to watch and triggers a LOT of
events (e.g. Git creates `index.lock` files that will appear and
disappear rapidly as terminals/IDEs/etc interact with Git, even
for read-only operations).
The Tilt-provided ephemeral file set has been slowly devised over
time based on temporary files that can cause trouble. We can also
look at a more robust/configurable solution here in the future,
but thse provide a reasonable out-of-the-box configuration for
the moment.
There's also some small tweaks to the output to add missing
newlines in a few edge cases and such.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Currently the emacs ignore patterns include `**/.#*` (lock files), but
doesn't include `**/#*#` (autosave files;
https://www.emacswiki.org/emacs/AutoSave, not to be confused with
`**/*~` backup files which are ignored.)
Add autosave files.
When creating files in Go, the stdlib will create (and then rapidly
delete) files ending with `-go-tmp-umask` to determine the umask
to use for permission purposes.
This can cause trouble with Live Update because the files tend to
vanish underneath it, for example.
Fixes#5117.
Milas Bowman
Guillaume Lours
Nicolas De Loof
Shan Desai
cui fliter
Ulysses Souza
Jes Cok
Jeremy Fleischman
Mateus Esdras
TP-O
Benjamín Guzmán
Tran Phong
Sebastiaan van Stijn
Luis Rascao
Laura Brehm
Josh Wilson
Milas Bowman
Iggy Jackson
Nick Santos
Matt Landis
Maia McCormick
Dan Miller