Commit Graph

973 Commits

Author SHA1 Message Date
Nicolas De Loof 6727908803 introduce --resolve-image-digests for publish to seal service images by digest
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-11-02 10:43:45 +01:00
kimdcottrell 616bba0afd linter errors fixed
Signed-off-by: kimdcottrell <me@kimdcottrell.com>
2023-10-29 22:08:00 +01:00
kimdcottrell ee6e3c2a44 NetworkList to NetworkInspect for ID search
Signed-off-by: kimdcottrell <me@kimdcottrell.com>
2023-10-29 22:08:00 +01:00
kimdcottrell c7e31a3c15 Squashing feature branch commits in order to add signoff message.
- added clarity with error handling. added test to show issue.

- in manual testing, this fixes the issue and allows watch to run after rebuild

- added cleanup back in

- fixed issue where watch extnet rebuild test would start all containers listed in the fixture

Signed-off-by: kimdcottrell <me@kimdcottrell.com>
2023-10-29 22:08:00 +01:00
Nicolas De Loof dd0803dba1 fix SIGTERM support to stop/kill stack
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-10-26 12:19:50 +02:00
Nicolas De Loof 39008c539c align with OCI artifact best practices
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-10-23 20:26:13 +02:00
Eng Zer Jun 6c345b3755
refactor(cmd/compose/run): remove redundant `len` check
From the Go specification [1]:

  "1. For a nil slice, the number of iterations is 0."

`len` returns 0 if the slice is nil [2]. Therefore, checking
`len(v) > 0` before a loop is unnecessary.

[1]: https://go.dev/ref/spec#For_range
[2]: https://pkg.go.dev/builtin#len

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-10-23 16:06:52 +08:00
Guillaume Lours b8773ad1c5
Merge pull request from ndeloof/wait_missing
fail start if depependency is missing
2023-10-19 10:04:32 +02:00
Nicolas De Loof 1ffa194e12
fail start if depependency is missing
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-10-19 09:50:37 +02:00
Guillaume Lours 5e1d3f2b41
Merge pull request from ndeloof/warn_remote_disabled
warn user remote resource is disabled
2023-10-18 12:54:40 +02:00
Guillaume Lours 3d0207ebc8
remove uncessary return value of pullComposeFiles function
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-10-18 12:42:11 +02:00
Guillaume Lours 818bc3c34a add sync+restart action to watch attribute
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-10-18 12:11:02 +02:00
Guillaume Lours 9dd081b92e add support of COMPOSE_ENV_FILES env variable to pass a list of env files
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-10-05 21:55:43 +02:00
Guillaume Lours 2c0b023273 add dry-run support for publish command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-10-05 21:55:00 +02:00
Nicolas De Loof 599e4b242a
extract method to reduce cyclomatic complexity
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-29 10:18:11 +02:00
Nicolas De Loof fe8c2780c8 warn user remote resource is disabled
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-29 09:30:38 +02:00
Matthieu MOREL 4f694919ff deps: remove deprecated github.com/pkg/errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-09-29 06:28:58 +02:00
Matthew Walowski 6ecab95775 Include image name in error message
Signed-off-by: Matthew Walowski <mattwalowski@gmail.com>
2023-09-29 06:25:10 +02:00
Bjorn Neergaard 12e0ac898a pkg/compose/publish: use empty config descriptor mediaType
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-09-29 05:27:14 +02:00
Bjorn Neergaard a6b7d78575 pkg/remote/oci: check artifactType instead of config.mediaType
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-09-29 05:27:14 +02:00
Bjorn Neergaard 991901f2ef pkg/remote/oci: refer to the manifest as manifest
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
2023-09-29 05:27:14 +02:00
Guillaume Lours 2d971fc97d
update the watch warning message when no services with a develop section
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-09-27 23:28:45 +02:00
Guillaume Lours 52a641bf6d
Merge pull request from glours/move-watch-to-main-cmd
move watch from alpha to main command
2023-09-21 10:08:23 +02:00
Guillaume Lours 61c8be11c0 remove --timeout=0 flag to cleanup function of watch e2e test
compose down command need the watch process to be killed to succeed

Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-09-20 21:32:03 +02:00
Guillaume Lours 6be5f3003a move watch from alpha to main command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-09-20 18:55:08 +02:00
Nicolas De Loof c34c306cb9
TestWatch to use new `develop` section
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-20 18:39:32 +02:00
Nicolas De Loof 5ca35c88be implement publish
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-20 18:14:58 +02:00
Nicolas De loof 805541be9d
watch: use official `develop` section ()
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-20 11:51:53 -04:00
Nicolas De Loof 8af49ff369 resolve service reference into container based on observed state
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-14 15:29:50 +02:00
Nicolas De Loof f6e31dbc6a don't rely on depends_on to resolve volume_from, better use observed state
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-14 15:29:50 +02:00
Guillaume Lours 1a98a70b8a add scale command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-09-13 11:15:14 +02:00
Milas Bowman 19bbb12fac
ci: tweak restricted imports in linter ()
* Eliminate direct dependency on gopkg.in/yaml.v2
* Add gopkg.in/yaml.v2 as a restricted import
* Add github.com/distribution/distribution as a restricted dependency in favor of distribution/reference which is the subset of functionality that Compose needs
* Remove an unused exclusion

NOTE: This does change the `compose config` output slightly but does NOT  change the semantics:
* YAML indentation is slightly different for lists (this is a `v2` / `v3` thing)
* JSON is now "minified" instead of pretty-printed (I think this generally desirable and more consistent with other JSON command outputs)

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-09-11 15:53:19 +00:00
Milas Bowman d7b0b2bd7d
watch: build & launch the project at start ()
The `alpha watch` command current "attaches" to an already-running
Compose project, so it's necessary to run something like
`docker compose up --wait` first.

Now, we'll do the equivalent of an `up --build` before starting the
watch, so that we know the project is up-to-date and running.

Additionally, unlike an interactive `up`, the services are not stopped
when `watch` exits (e.g. via `Ctrl-C`). This prevents the need to start
from scratch each time the command is run - if some services are already
running and up-to-date, they can be used as-is. A `down` can always be
used to destroy everything, and we can consider introducing a flag like
`--down-on-exit` to `watch` or changing the default.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-09-07 13:27:23 -04:00
Nicolas De Loof e0f39ebbef pull OCI remote resource
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-07 07:27:32 +02:00
Nicolas De Loof c9d54f09cf introduce publish (alpha) command
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-07 07:27:32 +02:00
Sebastiaan van Stijn f4f2e934ac
migrate to github.com/distribution/reference
The "reference" package was moved to a separate module, which was extracted
from b9b19409cf

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-04 14:36:36 +02:00
Nicolas De Loof 32c3d0a3ff Enable service explicitly requested to be restarted
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-09-01 12:01:02 +02:00
Milas Bowman 1fdbcb6255 build: pass BuildOptions around explicitly & fix multi-platform issues
The big change here is to pass around an explicit `*BuildOptions` object
as part of Compose operations like `up` & `run` that may or may not do
builds. If the options object is `nil`, no builds whatsoever will be
attempted.

Motivation is to allow for partial rebuilds in the context of an `up`
for watch. This was broken and tricky to accomplish because various parts
of the Compose APIs mutate the `*Project` for convenience in ways that
make it unusable afterwards. (For example, it might set `service.Build = nil`
because it's not going to build that service right _then_. But we might
still want to build it later!)

NOTE: This commit does not actually touch the watch logic. This is all
      in preparation to make it possible.

As part of this, a bunch of code moved around and I eliminated a bunch
of partially redundant logic, mostly around multi-platform. Several
edge cases have been addressed as part of this:
 * `DOCKER_DEFAULT_PLATFORM` was _overriding_ explicitly set platforms
   in some cases, this is no longer true, and it behaves like the Docker
   CLI now
 * It was possible for Compose to build an image for one platform and
   then try to run it for a different platform (and fail)
 * Errors are no longer returned if a local image exists but for the
   wrong platform - the correct platform will be fetched/built (if
   possible).

Because there's a LOT of subtlety and tricky logic here, I've also tried
to add an excessive amount of explanatory comments.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-09-01 08:32:56 +02:00
Milas Bowman 407a0d5b53
up: fix various race/deadlock conditions on exit ()
If running `up` in foreground mode (i.e. not `-d`),
when exiting via `Ctrl-C`, Compose stops all the
services it launched directly as part of that `up`
command.

In one of the E2E tests (`TestUpDependenciesNotStopped`),
this was occasionally flaking because the stop
behavior was racy: the return might not block on
the stop operation because it gets added to the
error group in a goroutine. As a result, it was
possible for no services to get terminated on exit.

There were a few other related pieces here that
I uncovered and tried to fix while stressing this.
For example, the printer could cause a deadlock if
an event was sent to it after it stopped.

Also, an error group wasn't really appropriate here;
each goroutine is a different operation for printing,
signal-handling, etc. If one part fails, we don't
actually want printing to stop, for example. This has
been switched to a `multierror.Group`, which has the
same API but coalesces errors instead of canceling a
context the moment the first one fails and returning
that single error.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-31 10:47:14 -04:00
Sebastiaan van Stijn 8caa6f1f3e
pkg/api: replace uuid for basic random id
The uuid package in distribution was created as a utility for the distribution
project itself, to cut down external dependencies (see [1][1]).

For compose, this has the reverse effect, as it now brings all the dependencies
of the distribution module with it.

This patch switches to the uuid generation to crypto/rand to produce a random
id. I was considering using a different uuid implementation, or docker's
"stringid.GenerateRandomID", but all of those are doing more than needed,
so keep it simple.

Currently, this change has little effect, because compose also uses the
distribution module for other purposes, but the distribution project is
in the process of moving the "reference" package to a separate module,
in which case we don't want to depend on the distribution module only for
the uuid package.

[1]: 36e34a55ad

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-31 10:13:32 +02:00
Milas Bowman d6f842b042
test: e2e test reliability improvements ()
* Use unique project name prefixes (some of these tests assert
  on output using the project name as a magic string, so could
  be impacted by other tests with the same project name prefix)
* Tear down port range project before starting to try and avoid
  race conditions with the engine and port assignment

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-30 16:01:15 -04:00
Nicolas De Loof 41682acc77 add support for attributes exposed by `docker ps`
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-08-25 16:36:45 +02:00
Nicolas De Loof 1054792b47 align docker compose ps with docker CLI to support --format
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-08-25 16:36:45 +02:00
Milas Bowman 19f66918cc watch: only allow a single instance per-project
This is a good place to start introducing (local) exclusivity
to Compose. Now, when `alpha watch` launches, it will check for
the existence of a PID file in the user XDG runtime directory,
and create one if the existing one is stale or does not exist.
If the PID file exists and is valid, an error is returned and
Compose exits.

A slight tweak to the experimental remote Git loader has been
made to use the XDG package for consistency.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-25 15:49:28 +02:00
Milas Bowman bc9d696fa0
Merge pull request from thaJeztah/replace_dockerignore
replace dockerfile/dockerignore with patternmatcher/ignorefile
2023-08-23 16:04:19 -04:00
Nicolas De loof 6204fb1c94
logs: fix for missing output on container exit ()
We can't assume we receive container logs line by line. Some framework won't buffer output and will send char by char, and we also can receive looong lines which get buffered to 32kb and then cut into multiple logs.

This assumes we will catch container streams being closed before we receive a die event for container, which could be subject to race condition, but at least the impact here is minimal and the fix works for reproduction examples provided in linked issues.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-08-23 08:57:18 -04:00
Sebastiaan van Stijn 5d732010a7
replace dockerfile/dockerignore with patternmatcher/ignorefile
The BuildKit dockerignore package was integrated in the patternmatcher
repository / module. This patch updates our uses of the BuildKit package
with its new location.

A small local change was made to keep the format of the existing error message,
because the "ignorefile" package is slightly more agnostic in that respect
and doesn't include ".dockerignore" in the error message.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-23 00:43:17 +02:00
Nicolas De loof dd34f7a22b
include: add experimental support for Git resources ()
Requires setting `COMPOSE_EXPERIMENTAL_GIT_REMOTE=1`.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-08-18 09:16:45 -04:00
Milas Bowman caad72713b up: handle various attach use cases better
By default, `compose up` attaches to all services (i.e.
shows log output from every associated container). If
a service is specified, e.g. `compose up foo`, then
only `foo`'s logs are tailed. The `--attach-dependencies`
flag can also be used, so that if `foo` depended upon
`bar`, then `bar`'s logs would also be followed. It's
also possible to use `--no-attach` to filter out one
or more services explicitly, e.g. `compose up --no-attach=noisy`
would launch all services, including `noisy`, and would
show log output from every service _except_ `noisy`.
Lastly, it's possible to use `up --attach` to explicitly
restrict to a subset of services (or their dependencies).

How these flags interact with each other is also worth
thinking through.

There were a few different connected issues here, but
the primary issue was that running `compose up foo` was
always attaching dependencies regardless of `--attach-dependencies`.

The filtering logic here has been updated so that it
behaves predictably both when launching all services
(`compose up`) or a subset (`compose up foo`) as well
as various flag combinations on top of those.

Notably, this required making some changes to how it
watches containers. The logic here between attaching
for logs and monitoring for lifecycle changes is
tightly coupled, so some changes were needed to ensure
that the full set of services being `up`'d are _watched_
and the subset that should have logs shown are _attached_.
(This does mean faking the attach with an event but not
actually doing it.)

While handling that, I adjusted the context lifetimes
here, which improves error handling that gets shown to
the user and should help avoid potential leaks by getting
rid of a `context.Background()`.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-18 12:38:38 +02:00
Nicolas De loof 792afb8d13
build: use correct values for proxy variables ()
clone variable before we capture a pointer

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-08-17 14:25:28 -04:00
Nicolas De Loof 150449bbd2 warn user secret uid/gid/mode is not supported
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-08-16 19:17:28 +02:00
Milas Bowman bfa54081d4
build: fix missing proxy build args for classic builder ()
Refactor to use a consistent code path for determining the build
args for a service image regardless of whether BuildKit or the
classic builder is being used.

After recent changes, these code paths had diverged, so the classic
builder was missing the proxy variables from the Docker client
config.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-10 08:57:28 -04:00
Milas Bowman fd8ab2f7ac
watch: enable tar-based syncer by default ()
Swap the default implementation now that batching is merged.
Keeping the `docker cp` based implementation around for the
moment, but it needs to be _explicitly_ disabled now by setting
`COMPOSE_EXPERIMENTAL_WATCH_TAR=0`.

After the next release, we should remove the `docker cp`
implementation entirely.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-04 16:58:01 -04:00
Silvin Lubecki 0a9d1277c5 Display builder's name on the first build line.
Code borrowed from buildx commands/build.go.

Signed-off-by: Silvin Lubecki <silvin.lubecki@docker.com>
2023-08-04 17:11:44 +02:00
Milas Bowman c350f80d4b
up: do not warn on successful optional dependency complete ()
If an optional dependency exits successfully (exit code of 0),
with a service condition of `service_completed_successfully`,
don't log a warning.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-03 21:00:49 +00:00
Milas Bowman 80856eacaf
progress: minor correctness fixes ()
* When waiting for dependencies, `select` on the context as well
  as the ticker
* Write multiple progress events "transactionally" (i.e. hold the
  lock for the duration to avoid other events being interleaved)
* Do not change "finished" steps back to "in progress" to prevent
  flickering

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-03 15:14:17 -04:00
Praful Gupta d7b1972d5e
doc: update Config() comment in API Service interface ()
Update Config comment in Service interface

Signed-off-by: Praful Gupta <prafulgupta6@gmail.com>
2023-08-03 15:13:26 -04:00
Silvin Lubecki 7c42776770
Improve buildkit node creation ()
Move builder and nodes initialization code up, avoiding to recreate/load them for every service build.

Signed-off-by: Silvin Lubecki <silvin.lubecki@docker.com>
2023-08-03 15:11:16 -04:00
Milas Bowman 3b0742fd57
watch: batch & de-duplicate file events ()
Adjust the debouncing logic so that it applies to all inbound file
events, regardless of whether they match a sync or rebuild rule.

When the batch is flushed out, if any event for the service is a
rebuild event, then the service is rebuilt and all sync events for
the batch are ignored. If _all_ events in the batch are sync events,
then a sync is triggered, passing the entire batch at once. This
provides a substantial performance win for the new `tar`-based
implementation, as it can efficiently transfer the changes in bulk.

Additionally, this helps with jitter, e.g. it's not uncommon for
there to be double-writes in quick succession to a file, so even if
there's not many files being modified at once, it can still prevent
some unnecessary transfers.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-03 14:53:02 -04:00
Milas Bowman f94cb49062
test: fix e2e test for privileged builds ()
We cannot guarantee the exact value of `CapEff` across
environments, and this test has started failing some places,
e.g. Docker Desktop, and now GitHub Actions (likely due to
a kernel upgrade on the runners or similar).

By setting `privileged: true` on the build, we're asking for
the `security.insecure` entitlement on the build. A safe
assumption is that will include `CAP_SYS_ADMIN`, which won't
be present otherwise, so mask the `CapEff` value and check
for that.

It's worth noting that realistically, the build won't even
be able to complete without the correct entitlement, since the
`Dockerfile` uses `RUN --security=insecure`, so this is really
an additional sanity check.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-03 12:23:24 -04:00
Milas Bowman f65fd02383
watch: add tar sync implementation ()
Brought to you by Tilt ❤️ 

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-08-01 14:39:08 -04:00
Milas Bowman cb17c3c8a6 watch: move sync logic into separate package
Just moving some code around in preparation for an alternative
sync implementation that can do bulk transfers by using `tar`.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-07-19 12:25:13 +02:00
Sebastiaan van Stijn 4eb43c53fa pkg/compose: RunOneOffContainer: don't use NewStartOptions()
It's no longer used in docker/cli, and doesn't do anything other than
creating an empty struct, so replacing it (as we're planning to
deprecate that function)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-07-19 12:13:40 +02:00
Guillaume Lours 150b88ab5d
Merge pull request from milas/e2e-watch-test-fix
test: watch e2e reliability tweaks
2023-07-19 12:07:10 +02:00
Milas Bowman 1ae191a936 trace: instrument `compose up` at a high-level
* Image pull
* Image build
* Service apply
  * Scale down/up (event)
  * Recreate container (event)
  * Scale up (event)
  * Container start (event)

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-07-19 11:26:12 +02:00
Nicolas De Loof 47778f8b77 check secret target is an absolute windows path
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-07-19 10:57:22 +02:00
Milas Bowman 636c13f818 build: do not attempt to push unnamed service images
When building, if images are being pushed, ensure that only
named images (i.e. services with a populated `image` field)
are attempted to be pushed.

Services without `image` get an auto-generated name, which
will be a "Docker library" reference since they're in the
format `$project-$service`, which is implicitly the same as
`docker.io/library/$project-$service`. A push for that is
never desirable / will always fail.

The key here is that we cannot overwrite the `<svc>.image`
field when doing builds, as we need to be able to check for
its presence to determine whether a push makes sense.

Fixes .

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-07-19 09:58:37 +02:00
Milas Bowman ddceb1ac9d test: do not run watch e2e tests in parallel
This isn't playing nicely with the GHA CI runner.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-07-18 18:54:23 -04:00
Milas Bowman d48f28c72c test: skip watch e2e test on macOS for the moment
Fix forthcoming via https://github.com/compose-spec/compose-go/pull/436
which addresses some symlink limitations. These can
actually effect other platforms but are most common
on macOS because the test creates temporary directories,
which are symlinked on macOS.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-07-18 18:53:26 -04:00
Guillaume Lours 2d16a05afa
only check if a dependency is required when something unexpected happens
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-07-18 23:45:31 +02:00
Guillaume Lours bb94ea034e add support of depends_on.required attribute
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-07-18 23:13:47 +02:00
Milas Bowman 3bc871e64b test: speed up the e2e test suite
Lots of our phony Compose files launch pointless long-lived processes
so we can assert on state. However, this means they often don't respond
well to signals on their own, requiring Compose to timeout and kill
them when doing a `down`.

Add in lots of `init: true` where appropriate so that we don't block
for no reason while running E2E tests all over the place.

Additionally, a couple tests have gotten a cleanup so they don't leave
behind containers. I still want to build this into the framework in
the future, but this is easier for the moment and won't cause any
trouble in the future.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-07-18 11:08:06 +02:00
Milas Bowman 3dc8734897
watch: add end-to-end test ()
Add an end-to-end test that covers the core watch functionality,
i.e. CRUD on files & directories.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-07-17 10:47:36 -04:00
Guillaume Lours 8339269e13
Merge pull request from ndeloof/run_no_deps
Apply no-deps before we select and mutate target service
2023-07-10 15:46:01 +02:00
Guillaume Lours ee6aeed84e
Merge pull request from ndeloof/attach
support `attach`
2023-07-10 15:17:44 +02:00
Guillaume Lours 7a9dfa4284
Merge pull request from milas/e2e-process-leak
test: fix process leak in wait e2e test
2023-07-10 15:09:41 +02:00
Milas Bowman 8dea7b5cae test: fix process leak in wait e2e test
* Run `down` before and after test to not leave around containers
* Kill the `wait` process that's waiting on `infinity`
  * NOTE: If the test is actually working, this should exit once
    the `down` happens, but this ensures that we kill everything
    we start

I'd like to generalize more of this into the framework, but this
is a quick fix to prevent filling up CI machines with tons of
processes over time.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-07-10 08:42:09 -04:00
Nicolas De Loof e6a7694b8d
Apply no-deps before we select and mutate target service
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-07-10 14:37:42 +02:00
Nicolas De Loof 46d936c750 support `attach`
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-07-10 14:34:28 +02:00
Shan Desai fd0e0a2cbd
fix(secrets): file permission value does not comply with spec
closes 

Compose Spec mentions that default values for secrets is `0444` aka. world-readable permissions. However, the value was previously set to `0400`. 


Signed-off-by: Shan Desai <shantanoo.desai@gmail.com>
2023-07-07 18:58:21 +02:00
Nicolas De Loof b0af2deb2b when --index is not set select first service container
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-07-07 14:08:24 +02:00
Milas Bowman be22bc735a network: fix random missing network when service has more than one
As part of the fix for , the logic was adjusted so that the
default (highest-priority) network is used in the `ContainerCreate`,
and then the remaining networks are connected via calls to
`NetworkConnect` before starting the container.

Unfortunately, `ServiceConfig::NetworksByPriority` is neither
deterministic nor stable when networks have the same priority.

It's non-deterministic because the order of networks from parsing
YAML is random, since they are loaded into a Go map (which have
random iteration order). Additionally, it's not using a `SortStable`
in `compose-go`, so even if the load order was predictable, it
still might produce different results.

While I look at improving `compose-go` here to prevent this from
tripping us up in the future, this fix looks at _all_ networks for
a service and ignores the "default" one now. Before, it would
always skip the first one in the slice since that _should_ have
been the "default".

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-07-07 09:18:01 +02:00
Nicolas De Loof b5f5e27597 don't use unitialized cli to setup DryRunClient
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-07-06 14:27:57 +02:00
cui fliter 25ca75db4d fix some comments
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-07-04 11:34:49 +08:00
Guillaume Lours 28301fb1a4 add support of --builder and BUILDX_BUILDER
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-07-03 10:11:18 +02:00
Ulysses Souza edd76bfd70 Add `docker compose wait`
Signed-off-by: Ulysses Souza <ulyssessouza@gmail.com>
2023-06-30 16:07:03 +02:00
Milas Bowman 10b290e682 up: fix race condition on network connect
Engine API only allows at most one network to be connected as
part of the ContainerCreate API request. Compose will pick the
highest priority network.

Afterwards, the remaining networks (if any) are connected before
the container is actually started.

The big change here is that, previously, the highest-priority
network was connected in the create, and then disconnected and
immediately reconnected along with all the others. This was
racy because evidently connecting the container to the network
as part of the create isn't synchronous, so sometimes when Compose
tried to disconnect it, the API would return an error like:
```
container <id> is not connected to the network <network>
```

To avoid needing to disconnect and immediately reconnect, the
network config logic has been refactored to ensure that it sets
up the network config correctly the first time.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-06-29 16:00:55 -04:00
Jes Cok 1a41678c58 fix typos
Signed-off-by: Jes Cok <xigua67damn@gmail.com>
2023-06-27 16:12:25 +02:00
Guillaume Lours 035276e027
watch: add warning when a path is already used by a bind mount volume ()
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-06-26 18:56:04 -04:00
Milas Bowman 061b52da9a ci: build fix for new buildx
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-06-21 11:17:39 +02:00
Milas Bowman 04aa155878 ci: upgrade to buildx v0.11
https://github.com/docker/buildx/releases/tag/v0.11.0

Several `replace` directives have been removed and dependencies
aligned with buildx as needed.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-06-21 11:17:39 +02:00
Guillaume Lours 2d4f8d31fc
Merge pull request from ndeloof/secret_uid
warn user build.secrets uid,gid,mode are not implemented
2023-06-21 10:19:43 +02:00
Nicolas De Loof a2ce602f6c fix race condition, waiting for containers when one exit
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-06-20 16:17:15 +02:00
Nicolas De Loof c61b8aa5ac introduce run --cap-add to run maintenance commands using service image
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-06-19 14:20:20 +02:00
Nicolas De Loof 7ffe83dc95 don't apply "rebuild" watch strategy by default
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-06-15 15:30:21 +02:00
Nicolas De Loof d20c2551f2 warn user build.secrets uid,gid,mode are not implemented
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-06-14 10:21:20 +02:00
Nicolas De Loof cfe91becc7 use `--progress` to configure progress UI stylet push
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-06-12 16:53:42 +02:00
Milas Bowman e63ab14b1e
ci: merge Go coverage reports before upload ()
Attempting to fix the state of codecov action checks right now,
which are behaving very erratically.

Using the new functionality in Go 1.20 to merge multiple reports,
so now the unit & E2E coverage data reports are stored as artifacts
and then downloaded, merged, and finally uploaded to codecov as a
new job.

Additionally, add a `codecov.yml` config and try to turn down the
aggressiveness of it for CI checks.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-06-08 14:58:21 -04:00
Guillaume Lours 32cf776ecd
Merge pull request from ndeloof/Building
do not render `Building` when no build is needed
2023-06-08 12:16:46 +02:00
Guillaume Lours 955784c406
Merge pull request from milas/bump-deps
ci: upgrade to Go 1.20.5 and Moby v24.x
2023-06-08 12:03:23 +02:00
Nicolas De Loof 852c9e80b4 create directory in container using `mkdir -p`
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-06-08 11:28:22 +02:00
Milas Bowman 37850f7955 ci: upgrade to Go 1.20.5 and Moby v24.x
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-06-07 10:41:49 -04:00
Nicolas De Loof 4bf2fe9fed assume we receive logs by lines and don't ignore those without EOL
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-06-07 10:40:37 +02:00
Nicolas De Loof 629c9f62e9 better diagnostic message on network label mismatch
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-06-05 16:54:58 +02:00
Guillaume Lours 7c3fe359b7
Merge pull request from ndeloof/logs_follow
fix `compose -p x logs -f` detect new services started after command
2023-06-02 09:39:20 +02:00
Nicolas De Loof d2aa15c06e bump buildx and use confutil.ConfigDir
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-06-01 12:09:13 +02:00
Guillaume Lours 6530880361
Merge pull request from jfly/jfly/tweak-warning-message
Fix typo in warning about existing volume
2023-06-01 08:50:33 +02:00
Nicolas De loof 1bd8a773a7
detect network conflict as name is not guaranteed to be unique ()
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-31 14:46:23 -04:00
Nicolas De Loof fed8ef6b79 forward signal to container
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-31 15:10:11 +02:00
Nicolas De Loof 06ec06472f up should not silently ignore missing depends_on service
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-26 21:59:29 +02:00
Nicolas De Loof 466e1d3197
prevent buildkt's progress to render `Building` when no built is needed
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-26 15:04:37 +02:00
Nicolas De Loof 0d6b99e6f9
e2e test to cover logs -f managing service being added/scaled
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-26 14:51:23 +02:00
Nicolas De Loof 01d91c490c detect new container from project have started when running `compose logs` with no explicit services
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-26 14:15:48 +02:00
Nicolas De Loof 6f6e1635fd compute service hash with a default DeployConfig
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-26 14:15:33 +02:00
Jeremy Fleischman 3d05a1becf
Fix typo in warning about existing volume
Previously, this was telling us "but was not created for project
[project-it-was-created-for]", which is wrong. I opted to make the
message super explicit and print both the actual project and the
expected project.

Signed-off-by: Jeremy Fleischman <jeremyfleischman@gmail.com>
2023-05-25 17:16:47 -07:00
Milas Bowman b05a94fd66
progress: remove errant import ()
Write the warning using `logrus.Warn`. The function being used was
coming from `cfssl`'s log package, which was presumably the result
of auto-import being _slightly_ too aggressive.

(Note: `cfssl` is still an indirect dependency after this.)

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-05-24 17:08:46 -04:00
Nicolas De Loof c7afc6188b detect conflict removing volume/image and warn user
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-24 17:32:32 +02:00
Nicolas De Loof ca19b7fcc9 introduce WithRootNodesAndDown to walk the graph from specified nodes and down
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-24 17:32:32 +02:00
Nicolas De Loof 93bd27a0cc introduce ability to select service to be stopped by `compose down`
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-24 17:32:32 +02:00
Nicolas De loof 68c462e607
scale: sort containers by creation date to remove older ones first ()
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-23 13:58:32 -04:00
Nicolas De loof 916aac6c27
watch: only monitor configured paths ()
For performance, don't watch the entire build context.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-23 13:35:50 -04:00
Nicolas De loof eafcd1b35e
secrets: only set CopyUIDGID when required ()
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-23 13:34:24 -04:00
Nicolas De Loof cfe1a860ff
fix detection of swarm mode
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-22 08:26:22 +02:00
Guillaume Lours cd0fc214a5 only check the platform of cached image if image found
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-05-17 08:04:54 +02:00
Nicolas De Loof 9b5a4588f9 introduce --no-path-resolution to skip relative path to be resolved
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-16 13:02:56 +02:00
Nicolas De Loof 00f72cb553 report external network not found when swarm is disabled
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-15 20:57:19 +02:00
Nicolas De Loof 18a112e88c detect terminal is not a `console.File` to avoid a panic
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-15 20:56:33 +02:00
Mateus Esdras 77dc9b54f3
rm: remove debugging output ()
For example, when no container was being removed,
this would print `[]`.

Signed-off-by: Mateus Esdras <linux.esdras@gmail.com>
2023-05-11 16:32:29 -04:00
Nicolas De loof bceb3c1876
detect active endpoint trying to remove network and skip with a warning ()
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-11 16:31:29 -04:00
Nicolas De loof a14abb9044
cli: option to write status messages on stdout ()
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-11 12:45:00 -04:00
Nicolas De loof 0363d9260a
fix local image removal when `compose down` is ran with `--project-name` ()
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-11 12:41:14 -04:00
Nicolas De Loof b776826d92 check local image matches the required platform
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-10 16:57:21 +02:00
Nicolas De Loof e92c5d1392 fix race condition running `compose up` with --parallel < number of services
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-10 11:39:07 +02:00
Nicolas De loof 67455e9f33
fix builkit progressui integration ()
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-08 09:21:41 -04:00
Guillaume Lours 5fdcaa0fe1
Merge pull request from glours/dry-run-up-support
add dry-run support to up command
2023-05-05 09:41:18 +02:00
Guillaume Lours 2e4faf80f5 add dry-run support to up command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-05-04 15:11:11 +02:00
Nicolas De Loof b45ca82791
let user declare build secret target (id)
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-05-04 10:01:10 +02:00
TP-O b304c4e154 stop containers after termination
Signed-off-by: TP-O <letranphong2k1@gmail.com>
2023-05-03 11:47:56 +02:00
Guillaume Lours eca1365d42
cli: dry run support for `build` ()
* add dry-run support for classic builder
* add dry-run support for buildkit

Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-05-02 14:23:26 -04:00
Guillaume Lours 03f4c0e631
progress: make title configurable ()
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-05-02 14:15:35 -04:00
Nicolas De Loof 07c4849cb9 TailMsgf must format with args
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-04-27 22:12:12 +02:00
Nicolas De Loof d01ef5887a restore support for `--memory`
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-04-25 15:41:08 +02:00
Nicolas De Loof dec608f3cd don't block events loop collecting logs
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-04-25 15:20:05 +02:00
Milas Bowman 1383ab09ec test: fix E2E tests under Engine v23 / DD 4.19
Some error messages have been tweaked slightly, this adapts the
assertions to work on both Engine v20.10.x and v23.x.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-04-25 08:27:42 +02:00
Benjamín Guzmán 5eaafe4237 Fixed issue when project name contains dashes (`-`)
Signed-off-by: Benjamín Guzmán <bg@benjaminguzman.dev>
2023-04-24 12:18:37 +02:00
Benjamín Guzmán 7840a92c40 Added tests to `viz` subcommand
Signed-off-by: Benjamín Guzmán <bg@benjaminguzman.dev>
2023-04-24 12:18:37 +02:00
Nicolas De Loof 65fda18821 bump compose-go
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-04-17 17:19:56 +02:00
Tran Phong 0e7e1b940b
Remove redundant goroutine while removing containers ()
don't use goroutine to stop container while removing

Signed-off-by: TP-O <letranphong2k1@gmail.com>
2023-04-17 10:57:29 -04:00
Milas Bowman af6f0ffb9e
Merge pull request from thaJeztah/simplify_auth
Don't use "info.IndexServerAddress" for authentication
2023-04-12 12:19:04 -04:00
Milas Bowman 9ef173a3ac
log: fix race on container kill ()
If we go to inspect a container that we got an event for and it
no longer exists on the server, handle clean up without erroring
out.

Fixes .

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-04-12 12:15:58 -04:00
Sebastiaan van Stijn 1892be8c70
Don't use "info.IndexServerAddress" for authentication
The IndexServerAddress field was  as part of the initial Windows implementation
of the engine. For legal reasons, Microsoft Windows (and thus Docker images
based on Windows) were not allowed to be distributed through non-Microsoft
infrastructure. As a temporary solution, a dedicated "registry-win-tp3.docker.io"
registry was created to serve Windows images.

Using separate registries was not an ideal solution, and a more permanent
solution was created by introducing "foreign image layers" in the distribution
spec, after which the "registry-win-tp3.docker.io" ceased to exist, and
removed from the engine.

This replaces the code that calls out to the "/info" endpoint to use the
GetAuthConfigKey() function instead.

Related PR in docker/cli:
b4ca1c7368

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-04-11 23:10:22 +02:00
Guillaume Lours 7fb87856aa add dry-run support to down command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-04-07 10:41:51 +02:00
Guillaume Lours cb688b5fd4
fix gocyclo lint error which currently block Compose CI
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-04-07 10:19:06 +02:00
Nicolas De Loof 8b5b78fbb6 can't watch a service without a build section
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-04-06 08:55:49 +02:00
Nicolas De loof d762f5f473
better support NO_COLOR by disabling colors, not ANSI TUI ()
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-04-04 17:00:10 -04:00
Milas Bowman 7ce0096f40 ci: bump Go to 1.20.3 and various dependencies
Use latest Go minor release. Note: this release included fixes for
several CVEs, but they do not impact Compose.

Small errors have been fixed to keep the linter happy.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-04-04 15:15:02 -04:00
Nicolas De Loof a10c4c6df5
restore `--timeout` flag renamed by mistake
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-04-04 08:48:00 +02:00
Guillaume Lours 6a37428491
Merge pull request from glours/dry-run-create-support
add dry-run support to create command
2023-04-03 18:25:55 +02:00
Nicolas De Loof 981cb2024e
prevent panic using classic builder
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-31 11:59:22 +02:00
Guillaume Lours b83edbd039 add dry-run support to create command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-03-30 10:25:56 +02:00
Nicolas De Loof c5317496ac
workaround race condition in ContainerList
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-28 12:28:41 +02:00
Guillaume Lours 72a61c0602 add dry-run support to run command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-03-27 18:57:11 +02:00
Milas Bowman d818bf6f34
Merge pull request from milas/deps-update
ci: upgrade to Go 1.20.2 & bump deps
2023-03-24 11:05:53 -04:00
Milas Bowman cd17c8a950 test: update error message
Validation got improved in `compose-go` so the error message is
slightly different.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-03-24 10:42:43 -04:00
Milas Bowman 36625ed229 test: fix race in e2e build test
This was running two tests in parallel that would build/delete the
same images. Run in serial instead since that's not safe.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-03-24 10:32:55 -04:00
Nicolas De Loof d637cc3ade watch involves up --build after change has been detected
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-24 10:49:27 +01:00
Milas Bowman 16d5354d70 watch: add note about goroutine-safety & test
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-03-23 11:33:03 -04:00
Milas Bowman 7aaea283ca watch: data race / segfault fixes
Was getting segfaults with multiple services using
`x-develop` and `watch` at the same time. Turns out
the Moby path matcher lazily initializes the regex
pattern internally the first time it's used, so it's
not goroutine-safe.

Change here is to not use a global instance for the
ephemeral path matcher, but a per-watcher instance.

Additionally, the data race detector caught a couple
other issues that were easy enough to fix:
 * Use the lock that's used elsewhere for convergence
   before manipulating
 * Eliminate concurrent map access when triggering
   rebuilds

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-03-22 18:05:56 -04:00
Nicolas De Loof a11515e038 introduce `ignore` attribute for watch triggers
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-21 18:29:49 +01:00
Nicolas De Loof 6c1f06e420 Run classic builder with BuildConfig, not buildx.Options
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-21 15:37:55 +01:00
Nicolas De Loof 88b0d17ff8 use `build` as common API for build scenarios
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-21 15:37:55 +01:00
Nicolas De Loof 9e19bc8441 use progress to show copy status
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-21 14:40:52 +01:00
Milas Bowman 105a7c5b70 watch: add file delete/rename handling
This approach mimics Tilt's behavior[^1]:
 1. At sync time, `stat` the path on host
 2. If the path does not exist -> `rm` from container
 3. If the path exists -> sync to container

By handling things this way, we're always syncing based on the true
state, regardless of what's happened in the interim. For example, a
common pattern in POSIX tools is to create a file and then rename it
over an existing file. Based on timing, this could be a sync, delete,
sync (every file gets seen & processed) OR a delete, sync (by the
the time we process the event, the "temp" file is already gone, so
we just delete it from the container, where it never existed, but
that's fine since we deletes are idempotent thanks to the `-f` flag
on `rm`).

Additionally, when syncing, if the `stat` call shows it's for a
directory, we ignore it. Otherwise, duplicate, nested copies of the
entire path could get synced in. (On some OSes, an event for the
directory gets dispatched when a file inside of it is modified. In
practice, I think we might want this pushed further down in the
watching code, but since we're already `stat`ing the paths here now,
it's a good place to handle it.)

Lastly, there's some very light changes to the text when it does a
full rebuild that will list out the (merged) set of paths that
triggered it. We can continue to improve the output, but this is
really helpful for understanding why it's rebuilding.

[^1]: db7f887b06/internal/controllers/core/liveupdate/reconciler.go (L911)

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-03-21 08:37:18 -04:00
Luis Rascao bfeb1dc277 Fix concurrent map read/write when recreating containers
Signed-off-by: Luis Rascao <luis.rascao@gmail.com>
2023-03-19 21:56:06 +01:00
Laura Brehm 200f47e5be Add support for `additional_contexts` in `build` service config
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
2023-03-15 14:00:51 +01:00
Nicolas De Loof e0aaccf430 introduce dockerfile_inline
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-15 10:19:04 +01:00
Nicolas De Loof 754c06886f one off container name use configured Separator for naming
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-15 09:47:43 +01:00
Nicolas De Loof e492330dd5 collect built image IDs
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-15 09:47:32 +01:00
Nicolas De Loof bbe1b77a67 progress writer uses dockercli.Err stream
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-15 09:47:13 +01:00
Nicolas De Loof 85ddfde5d6 use go 1.20 -cover support
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-10 16:54:39 +00:00
Nicolas De Loof 6a0398d786 pad can be negative on small terminal
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-09 10:27:43 +00:00
Guillaume Lours 4434cea535 add dry-run support for push command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-03-08 14:23:57 +00:00
Guillaume Lours 3f7d3c2661 add dry-run support for pull command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-03-05 22:04:32 +01:00
Guillaume Lours 167c6a89b1 add dry-run support to restart command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-03-02 10:14:59 +01:00
Guillaume Lours 3cfbac6624
restart only needed services by checking depends_on relations
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-03-01 18:22:40 +01:00
Nicolas De Loof 4ea44797f5 only consider containers with config_hash labels (i.e, created by compose)
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-03-01 15:46:50 +01:00
Milas Bowman e31b95c16d test: tweak pause test to try and prevent failures in Windows CI
This test keeps failing with a timeout in Windows. I don't actually
think it should take that long to bring up an nginx container, so
I'm guessing that there's something else going on that's causing
trouble.

Increase the verbosity when running Compose commands: I think this
will generally make E2E test failures easier to diagnose by always
logging the full command that's going to be run and also capturing
stdout.

Add a health check and use `--wait` when launching the fixture for
the pause test. Combined with the verbosity increase, this should
make it easier to understand what's going on here.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-02-27 14:20:20 -05:00
Milas Bowman da1ca578b5 watch: ignore ephemeral files & minor output tweaks
Big change here is to import the ephemeral ignore set from Tilt.

The `.git` directory is also ignored for now: this restriction
should probably be lifted and made configurable in the future,
but it's not generally important to watch and triggers a LOT of
events (e.g. Git creates `index.lock` files that will appear and
disappear rapidly as terminals/IDEs/etc interact with Git, even
for read-only operations).

The Tilt-provided ephemeral file set has been slowly devised over
time based on temporary files that can cause trouble. We can also
look at a more robust/configurable solution here in the future,
but thse provide a reasonable out-of-the-box configuration for
the moment.

There's also some small tweaks to the output to add missing
newlines in a few edge cases and such.

Signed-off-by: Milas Bowman <milas.bowman@docker.com>
2023-02-24 14:34:51 -05:00
Josh Wilson 6fae6a41f9 Update emacs ignore patterns ()
Currently the emacs ignore patterns include `**/.#*` (lock files), but
doesn't include `**/#*#` (autosave files;
https://www.emacswiki.org/emacs/AutoSave, not to be confused with
`**/*~` backup files which are ignored.)

Add autosave files.
2022-07-26 08:57:40 -05:00
Milas Bowman dd5ea044bb ignore: add Go umask files to ephemeral set ()
When creating files in Go, the stdlib will create (and then rapidly
delete) files ending with `-go-tmp-umask` to determine the umask
to use for permission purposes.

This can cause trouble with Live Update because the files tend to
vanish underneath it, for example.

Fixes .
2022-04-27 16:42:51 -04:00
Milas Bowman 12de97b8d1 filewatch: use apiserver FileWatch model in EngineState ()
This follows the "action-first" approach to use the apiserver model
for `FileWatch` and dispatch simple create/update/delete actions.
2021-03-09 13:05:32 -05:00
Iggy Jackson 62b5f78fd9 Add .kate-swp files to ignore pattern ()
KDE's text editor, kate, uses a file similar to Vim's .swp files. Ignore
these files so we don't rebuild on every keypress.

Fixes 
2020-05-27 07:36:06 -07:00
Nick Santos 3f526c5c7b change org name from windmilleng to tilt-dev () 2020-05-15 10:34:39 -04:00
Matt Landis 12916b75a2 tilt: ignore a few more vim swap files () 2019-09-12 11:57:39 -04:00
Maia McCormick e3948f6bae ignore: auto-ignore jetbrains .idea file () 2019-08-15 14:38:44 -04:00
Dan Miller 91a2bdd6de model: move to pkg () 2019-08-09 12:52:31 -04:00
Nick Santos ce61e7bf18 ignore: improve the ephemeral temp file patterns [ch2663] () 2019-07-29 11:18:22 -04:00
Nicolas De Loof e831ea826b add support for `restart` for `depends_on`
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-18 12:55:29 +01:00
Milas Bowman 762cf9d998
Merge pull request from glours/dry-run-exec-support
support dry-run for exec command
2023-02-17 10:05:31 -05:00
Nicolas De Loof 24ff098252 compact TUI to monitor layers download progress
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-16 17:55:59 +01:00
Nicolas De Loof 313b82e94c ignore services without a build section
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-14 15:18:44 +01:00
Nicolas De Loof 9ac0392baf introduce --timeout on `up`
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-14 12:16:41 +01:00
Nicolas De Loof 92e0cd4047 also restart dependent services after a service has been restarted
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-14 11:30:15 +01:00
Nicolas De Loof 256ec49974 exclude unstable labels from config hash
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-13 17:02:28 +01:00
Nicolas De Loof 9765f171cd
store exec details to offer better dry-run status on ExecStart
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-13 11:59:01 +01:00
Nicolas De Loof b19df5c96c add support for `excludes` and `rebuild`
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-10 17:24:48 +01:00
Nicolas De Loof 7a42ba7eec use CGO to enable fsevent on OSX
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-10 17:24:48 +01:00
Guillaume Lours eb1c798912 support dry-run for rm command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-02-09 17:34:10 +01:00
Guillaume Lours 78b9404767 support dry-run for stop command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-02-09 17:34:10 +01:00
Guillaume Lours 25be264ed8 support dry-run for exec command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-02-08 11:21:47 +01:00
Guillaume Lours 70ab9f8f33 bump docker engine and cli version to 23.0.0 with buildkit(v0.11.2) and buildx (v0.10.2)
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-02-08 10:11:50 +01:00
Nicolas De Loof 93bffd9a7f
prevent assignment to entry in nil map
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-07 20:50:15 +01:00
Nicolas De Loof 52478f0c6e wait on service containers as dependencies to be deterministic
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-07 16:36:45 +01:00
Nicolas De Loof b5f0a4eefa use containers we expect to start for wait condition
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-07 16:36:45 +01:00
Nicolas De Loof 0f5b5ccbd0 detect replacement container is created and inform printer so it attach and don't stop
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-06 11:23:13 +01:00
Guillaume Lours fdc1738143 add log when copying files/directories between host and containers (both way)
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-02-03 22:44:26 +01:00
Guillaume Lours 2336d9fe35 support dry-run for cp command
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
2023-02-03 22:44:26 +01:00
Milas Bowman bf0ed9a4d4
Merge pull request from benmoss/add-remote-builder
Add remote buildx driver
2023-02-03 15:03:49 -05:00
Nicolas De Loof 1640f155e9 initial support for `sync`
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-02 14:59:30 +01:00
Nicolas De Loof e63cbfba0e use tilt watcher to track filesystem changes
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-02 14:59:30 +01:00
Nicolas De Loof 25576289c8 adjust code and dependencies
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2023-02-02 14:59:30 +01:00
Nick Santos 7d6ee74e62 cleanup deprecated ioutil functions ()
Signed-off-by: Nick Santos <nick.santos@docker.com>

Signed-off-by: Nick Santos <nick.santos@docker.com>
2023-02-02 14:59:30 +01:00
Nick Santos c08e07714a watch: optimization to help avoid inotify nodes for large file trees ()
fixes https://github.com/tilt-dev/tilt/issues/5764
2023-02-02 14:59:30 +01:00
Matt Landis cf31462398 tests: make test fixtures responsible for tearing themselves down () 2023-02-02 14:59:30 +01:00
Ken Sipe 1b71e3efb3 lint: fix local-prefixes for goimports ()
Fix local-prefixes on linters imports then resolved imports to be as expected

Signed-off-by: Ken Sipe <kensipe@gmail.com>
2023-02-02 14:59:30 +01:00
Milas Bowman 4d822676ce ci: add exportloopref linter and upgrade to golangci-lint v1.43.0 ()
`exportloopref` - detects captures of loop variable without
re-assignment

NOTE: There can be false negatives with this linter to avoid being
            overly strict and annoying!

Also upgraded `golangci-lint` to latest (v1.43.0 published 2021-11-03).
2023-02-02 14:59:30 +01:00
Milas Bowman 6fc0b6ff27 build: ensure file handles properly closed ()
I've been running the test suite (`./internal/engine` in particular)
with `-count X` a lot recently to catch timing-related test failures.

After running enough times, the tests start failing due to too many
open files, so I did an audit and found a few places where files or
readers weren't always being closed.
2023-02-02 14:59:30 +01:00
Nick Sieger ab84b6ac5b Fix local go_lintfix errors ()
* lint: add make lintfix and run it

Fixes all errors like:
```
File is not `goimports`-ed with -local github.com/tilt-dev/tilt (goimports)
```

* git: change to use TrimSuffix

* build: remove unnecessary cast
2023-02-02 14:59:30 +01:00
Milas Bowman 7161778ccf watch: use WalkDir to speed up file listing ()
`WalkDir` is new in Go 1.16 and avoids calling `os.Lstat` on
every visited file and directory. In most cases, we don't need
that info, so this will help reduce I/O when listing files,
which can be helpful for particularly big monorepos.
2023-02-02 14:59:30 +01:00
Nick Santos 1f5bfe8882 test: clean up test loggers a bit () 2023-02-02 14:59:30 +01:00
Milas Bowman f79c75ab10 test: fix lint errors on Darwin (macOS) ()
Unused code linter isn't particularly smart about platform build
tags, so since this func is only used by the "naive" (non-macOS)
file watcher, it needs to live with that or it gets flagged as
dead code when linting on macOS.
2023-02-02 14:59:30 +01:00
Nick Santos e62993a854 test: enable unused check, remove dead code () 2023-02-02 14:59:30 +01:00
Nick Santos 28251e8be5 watch: improve error messages when you run out of inotify instances () 2023-02-02 14:59:30 +01:00
Nick Santos c7ba7d9de5 dockerignore: convert ignore patterns to absolute paths [ch9237] ()
In most places in Tilt, we try to use absolute paths everywhere.
So this makes things more consistent with the rest of Tilt, and lets us be
a bit more flexible in how we handle subdirs and parent dirs in ignores.

Fixes https://github.com/tilt-dev/tilt/issues/3740
2023-02-02 14:59:30 +01:00
Nick Santos 8b39322365 watch: fix a dumb errcheck () 2023-02-02 14:59:30 +01:00
Nick Santos b3615d64e2 watch: increase the windows watch i/o buffer ()
fixes https://github.com/tilt-dev/tilt/issues/3556
2023-02-02 14:59:30 +01:00
Nick Santos 1a1d1707ed change org name from windmilleng to tilt-dev () 2023-02-02 14:59:30 +01:00
Nick Santos fd3e0bbe2b watch: use the recursive watcher on windows () 2023-02-02 14:59:30 +01:00
Nick Santos d2d4d05264 Revert "watch: fix inotify tests on windows" ()
This reverts commit 74ac7997b1c8f497babbbd499ff1f047563d699a.
2023-02-02 14:59:30 +01:00
Nick Santos dda0362b6e watch: fix inotify tests on windows () 2023-02-02 14:59:30 +01:00
Nick Santos ddc88ec41b circleci: run make shorttest on windows. flag off a large swath of broken tests () 2023-02-02 14:59:30 +01:00
Matt Landis 92c6a65a03 tilt: enable errcheck on tests () 2023-02-02 14:59:30 +01:00
Dan Miller 00b3caecbb ci: enable staticcheck, fix problems that were surfaced () 2023-02-02 14:59:30 +01:00
Dan Miller 17087447e9 logger: move to pkg () 2023-02-02 14:59:30 +01:00
Nick Santos ec781687fd watch: add retry counts to fsync () 2023-02-02 14:59:30 +01:00
Nick Santos b22dde9f18 dockerignore: improve MatchesEntireDir () 2023-02-02 14:59:30 +01:00
Nick Santos a31350ede1 watch: move more of the directory-skipping logic into the interface () 2023-02-02 14:59:30 +01:00
Nick Santos d744c97f13 watch: add a simple check when there are no exclusions () 2023-02-02 14:59:30 +01:00
Nick Santos 37647bc600 watch: skip setup if nothing is being watched () 2023-02-02 14:59:30 +01:00
Nick Santos 21e5d564af watch: FileEvents must always be absolute ()
there were a lot of confused tests that were using relative paths, then trying to workaround this
2023-02-02 14:59:30 +01:00