Ensure config_hash is updated when volumes_from mode is changed, and
service is recreated on next up as a result.
Signed-off-by: Joffrey F <joffrey@docker.com>
We were outputting an extra line, which in *some* cases, on *some*
terminals, was causing the output of parallel actions to get messed up.
In particular, it would happen when the terminal had just been cleared
or hadn't yet filled up with a screen's worth of text.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
- make it a positional arg, since it's required
- make it the first argument for all functions that require it
- remove an unnecessary one-line function that was only called in one place
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Includes some refactoring of log_printer_test to support checking for flush(), and so that each test calls the unit-under-test directly, instead of through a helper function.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
It has been an unnecessary wrapper around container.start() for a little while now, so we can call it directly.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Code cleanup. We no longer need this as the api returns a 304 for any
stopped containers, which doesn't raise an error.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
So we're not displaying output of all previous logs for a container, we attach,
if possible, to a container before the container is started.
LogPrinter checks if a container has a log_stream already attached and
print from that rather than always attempting to attach one itself.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Mutable objects are harder to debug and harder to reason about. ServiceLoader was almost immutable. There was just a single function which set fields for a second function. Instead of mutating the object, we can pass those values as parameters to the next function.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
- fixed a docstring to make it PEP257 compliant
- wrapped some long lines
- used a more specific error
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Remove some unnecessary newlines.
Remove a unittest that was attempting to test behaviour that was removed a while ago, so isn't testing anything.
Updated some unit tests to use mocks instead of a custom fake.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Environment keys that contain no value, get populated with values taken
from the environment not from the build phase but from running the command `up`.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This change adds cgroup-parent support to compose project. It allows
each service to specify a 'cgroup_parent' option.
Signed-off-by: Mohit Soni <mosoni@paypal.com>
splitdrive doesn't handle relative paths, so if volume_path contains
a relative path, we handle that differently and manually set drive to ''.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Including examples of more boolean types, eg yes/N as it's not
always immediately clear that they are treated as booleans.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Rather than inefficiently looping through all the containers that a
service has and overriding each volumes_from value, pick the first
one and return that.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
An expanded windows path of c:\shiny\thing:\shiny:rw
needs to be changed to be linux style path, including the drive,
like /c/shiny/thing /shiny
to be mounted successfully by the engine.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When a relative path is expanded and we're on a windows platform,
it expands to include the drive, eg C:\ , which was causing a ConfigError
as we split on ":" in parse_volume_spec and that was giving too many parts.
Use os.path.splitdrive instead of manually calculating the drive.
This should help us deal with windows drives as part of the volume
path better than us doing it manually.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Only expand volume host paths if they begin with a dot.
This is a breaking change. The deprecation warning preparing users for
this change has been removed.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
create_host_config from docker.utils will be deprecated so that
the new create_host_config has access to the _version so
we can ensure that network_mode only gets set to 'default' by
default if the version is high enough and won't explode.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
By doing this we were over-riding any of the daemon's defaults.
Instead we can send an empty string which docker-py sends on
and the daemon interprets as, 'json-file' as a default if it
hasn't got any other daemon level config options.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
One of the use cases is swarm requires at least : character, so going
from conservative to relaxed.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We're going to warn people that allowing a boolean in the environment is
being deprecated, so in a future release we can disallow it. This is to
ensure boolean variables are quoted in strings to ensure they don't get
mis-parsed by YML.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When users were putting true/false/yes/no in the environment key,
the YML parser was converting them into True/False, rather than leaving
them as a string.
This change will force people to put them in quotes, thus ensuring
that the value gets passed through as intended.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
oneOf schema ValidationError takes a little more work to parse and
pull out more detail so we can give a better error message back to
the user.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
English language is a tricky old thing and I've pulled out the validator type
parsing so that we can prefix our validator types with the correct article,
'an' or 'a'.
Doing a bit of extra hard work to ensure the error message is clear and
well constructed english.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
For simple definitions where a field can be multiple types, we can
specify the allowed types in an array. It's simpler and clearer.
This is only applicable to *simple* definitions, like number, string, list,
object without any other constraints.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
'~/' in a path currently doesnt work, you get the following error:
[Errno 2] No such file or directory: u'/home/USER/folder/~/some/path/.yml'
Signed-off-by: Nick H <nick.humrich@gmail.com>
The command value can be a list, which would be a Unix command-line
invocation broken up into individual values, thus needing the ability to
have non unique values.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Previously on Buffy... The process_errors was parsing a load of
ValidationErrors that we get back from jsonschema which included
assumptions about the state of the instance we're validating.
Now it's split in two and we're doing field separate to service,
those assumptions don't hold and we can't logically retrieve the
service_name from the error parsing when we're doing service schema
validation, have to explicitly pass this in.
process_errors is high on my list for some future re-factoring to help
make it a bit clearer, smaller state of doing things.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Now validation is split in two, the integration tests helped
highlight some places where the schema definition was incorrect.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If make_service_dict is our factory function then we'll give it the
responsibility of validation/construction and resolving.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Now the schema has been split into two, we need to modify the
process_errors function to accomodate.
Previously if an error.path was empty then it meant they were root
errors. Now that service_schema checks after the service has been
resolved, our service name is a key within the dictionary and
so our root error logic check is no longer true.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We want to give feedback to the user as soon as possible about the
validity of the config supplied for the services.
When extending a service, we can validate that the fields are
correct against our schema but we must wait until the *end* of
the extends cycle once all of the extended dicts have been merged
into the service dict, to perform the final validation check on the
config to ensure it is a complete valid service.
Doing this before that had happened resulted in false reports of
invalid config, as common config when split out, by itself, is not
a valid service but *is* valid config to be included.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This refactoring is now really coming together. Construction is
happening in the __init__, which is a constructor and helps
clean up the design and clarity of intent of the code. We can now
see (nearly) everything that is being constructed when a ServiceLoader
is created. It needs all of these data constructs to perform the
domain logic and actions. Which are now clearer to see and moving
more towards the principle of functions doing (mostly)one thing and
function names being more descriptive.
resolve_extends is now concerned with the resolving of extends, rather
than the construction, validation, pre processing and *then* resolving
of extends.
Happy days :)
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Separating out the steps we need to resolve extends, so that it
will be clear to insert pre-processing of interpolation and
validation.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
resolve_environment is specific to ServiceLoader, the function
does not need to be on the global scope, it is a part of the
ServiceLoader object. The environment needs to be resolved
before we can make any service dicts, it belongs in the
constructor.
This is cleaning up the design a little and being clearer
about intent and scope of functions.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Moving service name and dict out of the function make_service_dict
and into __init__. We always call make_service_dict with those so
let's put them in the initialiser. Slightly cleaner design intent.
The whole purpose of the ServiceLoader is to take a
service name&service dictionary then validate, process and return
service dictionaries ready to be created.
This is also another step towards cleaning the code up so we can
interpolate and validate an extended dictionary.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
While it can be set to ultimately a value of None, when a
config file is read in from stdin, it is not optional.
We kinda make use of it's ability to be set to None in our
tests but functionally and design wise, it is required.
If filename is not set, extends does not work.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Check for this in the init so we can remove the duplication of
raising in further functions.
A ServiceLoader isn't valid without one.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
In particular it includes:
- some extension of CONTRIBUTING.md
- one fix for Python 2.6 in tests/integration/cli_test.py
- one fix for Python 3.3 in tests/integration/service_test.py
- removal of unused imports
Make stream_output Python 3-compatible
Signed-off-by: Frank Sachsenheim <funkyfuture@riseup.net>
If we get back an error that wasn't an APIError, it was causing the
thread to hang. This catch all, while I appreciate feels risky to
have a catch all, is better than not catching and silently failing,
with a never ending thread.
If something worse than an APIError has gone wrong, we want to stop
the incredible journey of what we're doing.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
In order to validate a service name that has been specified as an
integer we need to run that as a pre-process validation step
*before* we pass the config to be validated against the schema.
It is not possible to validate it *in* the schema, it causes a
type error. Even though a number is a valid service name, it
must be a cast as a string within the yaml to avoid type error.
Taken this opportunity to move the code design in a direction
towards:
1. pre-process
2. validate
3. construct
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We use $ref in the schema to allow us to specify multiple type, eg
command, it can be a string or a list of strings.
It required some extra parsing to retrieve a helpful type to display
in our error message rather than 'string or string'. Which while
correct, is not helpful. We value helpful.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When a schema type is set as unique, we should display the validation
error to indicate that non-unique values have been provided for a key.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
The validation message was confusing by displaying only 1 level of
property of the service, even if the error was another level down.
Eg. if the 'files' property of 'extends' was the incorrect format,
it was displaying 'an invalid value for 'extends'', rather than
correctly retrieving 'files'.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Unfortunately the way that jsonschema is calling %r on its property
and then encoding the complete message means I've had to do this
manual way of removing the literal string prefix, u'.
eg:
key = 'extends'
message = "Invalid value for %r" % key
error.message = message.encode("utf-8")"
results in:
"Invalid value for u'extends'"
Performing a replace to strip out the extra "u'", does not change the
encoding of the string, it is at this point the character u followed
by a '.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
While it was intended as a positive to be stricter in validation
it would in fact break backwards compatibility, which we do not
want to be doing.
Consider re-visiting this later and include a deprecation warning if
we want to be stricter.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If a container is in the process of being removed, or removal has
failed, it can sometimes appear in the output of GET /containers/json
but not have a 'Name' key. In that case, rather than crashing, we can
ignore it.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Rather than implement the logic a second time, use docker-py
split_port function to test if the ports is valid.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Move validation out into its own file without causing circular
import errors.
Fix some of the tests to import from the right place.
Also fix tests that were not using valid test data, as the validation
schema is now firing telling you that you couldn't "just" have this
dict without a build/image config key.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
jsonschema provides a rich error tree of info, by parsing each error
we can pull out relevant info and re-write the error messages.
This covers current error handling behaviour.
This includes new error handling behaviour for types and formatting of
the ports field.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Define a schema that we can pass to jsonschema to validate against the
config a user has supplied. This will help catch a wide variety of common
errors that occur.
If the config does not pass schema validation then it raises an exception
and prints out human readable reasons.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
* Add support for volume_driver parameter in compose yml
* Don't expand volume host paths if a volume_driver is specified
(i.e., disable compose feature "relative to absolute path
transformation" when volume drivers are in use, since volume drivers
can use name where host path is normally specified; this is a
heuristic)
Signed-off-by: Luke Marsden <luke@clusterhq.com>
When an image declares a volume such as `/var/lib/mysql`, and a Compose
file has a line like `./data:/var/lib/mysql/` (note the trailing slash),
Compose creates duplicate volume binds when *recreating* the container.
(The first container is created without a hitch, but contains multiple
entries in its "Volumes" config.)
Fixed by normalizing all paths in volumes config.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Hard-coding the API version to '1.18' with the docker-py constructor will
cause the docker-py logic at
https://github.com/docker/docker-py/blob/master/docker/client.py#L143-L146
to always fail, which will cause authentication issues if you're using a
remote daemon using API version 1.19 - regardless of the API version of
the registry.
Allow the user to set the API version via an environment variable. If
the variable is not present, it will still default to '1.18' like it
does today.
Signed-off-by: Reilly Herrewig-Pope <reilly.herrewigpope@mandiant.com>
It was harder to see when there are errors if they came straight after
the other output. Putting a newline in there gives it a bit of visual
room.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Refactored parallel execute and execute create into a single function
parallel_execute that can now handle both cases. This helps untangle it
from being so tightly coupled to the container.
Updated all the relevant operations to use the refactored function.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
The concurrent.futures backport doesn't play well with
KeyboardInterrupt, so I'm using Thread and Queue instead.
Since thread pooling would likely be a pain to implement, I've just
removed `COMPOSE_MAX_WORKERS` for now. We'll implement it later if we
decide we need it.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Aanand Prasad
Daniel Nephin
Joffrey F
Joffrey F
Daniel Nephin
Mazz Mosley
mnowster
Viranch Mehta
Cameron Eagans
Mohit Soni
Karol Duleba
Vincent Demeester
Vojta Orgon
Christophe Labouisse
Nick H
Lucas N. Munhoz
Lachlan Pease
Frank Sachsenheim
Joel Hansson
Yuval Kohavi
Luke Marsden
Reilly Herrewig-Pope