From 3a8a25b9b35002368615084a9a1cbc6a6fd0e7a6 Mon Sep 17 00:00:00 2001 From: Harald Albers Date: Fri, 13 Feb 2015 10:25:18 +0100 Subject: [PATCH 01/46] Favour yml and yaml extensions in bash completion for -f Signed-off-by: Harald Albers --- contrib/completion/bash/docker-compose | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/completion/bash/docker-compose b/contrib/completion/bash/docker-compose index 03721f505..a0007fcbd 100644 --- a/contrib/completion/bash/docker-compose +++ b/contrib/completion/bash/docker-compose @@ -94,7 +94,7 @@ _docker-compose_build() { _docker-compose_docker-compose() { case "$prev" in --file|-f) - _filedir + _filedir y?(a)ml return ;; --project-name|-p) From 34c6920b37a22ea918abe51455ab0e51cf2a9220 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Tue, 24 Feb 2015 11:43:14 +0000 Subject: [PATCH 02/46] Point at official Docker install instructions rather than repeating them Signed-off-by: Aanand Prasad --- docs/install.md | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/docs/install.md b/docs/install.md index ef0da087e..b0c1b0e27 100644 --- a/docs/install.md +++ b/docs/install.md @@ -10,20 +10,11 @@ Compose with a `curl` command. ### Install Docker -First, you'll need to install Docker version 1.3 or greater. +First, install Docker version 1.3 or greater: -If you're on OS X, you can use the -[OS X installer](https://docs.docker.com/installation/mac/) to install both -Docker and the OSX helper app, boot2docker. Once boot2docker is running, set the -environment variables that'll configure Docker and Compose to talk to it: - - $(boot2docker shellinit) - -To persist the environment variables across shell sessions, add the above line -to your `~/.bashrc` file. - -For complete instructions, or if you are on another platform, consult Docker's -[installation instructions](https://docs.docker.com/installation/). +- [Instructions for Mac OS X](http://docs.docker.com/installation/mac/) +- [Instructions for Ubuntu](http://docs.docker.com/installation/ubuntulinux/) +- [Instructions for other systems](http://docs.docker.com/installation/) ### Install Compose From 35d5d1a5b1889956d72778dfc08427c628990fa9 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Wed, 25 Feb 2015 14:04:30 +0000 Subject: [PATCH 03/46] Build and link to getting started guides Signed-off-by: Aanand Prasad --- docs/index.md | 7 +++++-- docs/mkdocs.yml | 3 +++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/index.md b/docs/index.md index b44adc8e2..a75e7285a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -185,6 +185,9 @@ your services once you've finished with them: $ docker-compose stop -At this point, you have seen the basics of how Compose works. - +At this point, you have seen the basics of how Compose works. +- Next, try the quick start guide for [Django](django.md), + [Rails](rails.md), or [Wordpress](wordpress.md). +- See the reference guides for complete details on the [commands](cli.md), the + [configuration file](yml.md) and [environment variables](env.md). diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index fc725b28c..14335873d 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -5,3 +5,6 @@ - ['compose/yml.md', 'Reference', 'Compose yml'] - ['compose/env.md', 'Reference', 'Compose ENV variables'] - ['compose/completion.md', 'Reference', 'Compose commandline completion'] +- ['compose/django.md', 'Examples', 'Getting started with Compose and Django'] +- ['compose/rails.md', 'Examples', 'Getting started with Compose and Rails'] +- ['compose/wordpress.md', 'Examples', 'Getting started with Compose and Wordpress'] From c41342501bef738c7f39e7dfbee0b571b9b77e33 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Wed, 25 Feb 2015 18:25:15 +0000 Subject: [PATCH 04/46] Update README with new docs URL and IRC channel Signed-off-by: Aanand Prasad --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 52e6f3607..c943c70d8 100644 --- a/README.md +++ b/README.md @@ -53,4 +53,5 @@ Compose has commands for managing the whole lifecycle of your application: Installation and documentation ------------------------------ -Full documentation is available on [Fig's website](http://www.fig.sh/). +- Full documentation is available on [Docker's website](http://docs.docker.com/compose/). +- Hop into #docker-compose on Freenode if you have any questions. From 882dc673ce84b0b29cd59b6815cb93f74a6c4134 Mon Sep 17 00:00:00 2001 From: Fred Lifton Date: Thu, 26 Feb 2015 18:58:06 -0800 Subject: [PATCH 05/46] Edits and revisions to Compose Quickstart guides --- docs/django.md | 66 +++++++++++++++++++++++++++++++++-------------- docs/rails.md | 66 +++++++++++++++++++++++++++++++---------------- docs/wordpress.md | 50 +++++++++++++++++++++++++---------- 3 files changed, 127 insertions(+), 55 deletions(-) diff --git a/docs/django.md b/docs/django.md index 48d3e6d01..0605c86b6 100644 --- a/docs/django.md +++ b/docs/django.md @@ -1,14 +1,23 @@ ---- -layout: default -title: Getting started with Compose and Django ---- +page_title: Quickstart Guide: Compose and Django +page_description: Getting started with Docker Compose and Django +page_keywords: documentation, docs, docker, compose, orchestration, containers, +django -Getting started with Compose and Django -=================================== -Let's use Compose to set up and run a Django/PostgreSQL app. Before starting, you'll need to have [Compose installed](install.md). +## Getting started with Compose and Django -Let's set up the three files that'll get us started. First, our app is going to be running inside a Docker container which contains all of its dependencies. We can define what goes inside that Docker container using a file called `Dockerfile`. It'll contain this to start with: + +This Quick-start Guide will demonstrate how to use Compose to set up and run a +simple Django/PostgreSQL app. Before starting, you'll need to have +[Compose installed](install.md). + +### Define the project + +Start by setting up the three files you'll need to build the app. First, since +your app is going to run inside a Docker container containing all of its +dependencies, you'll need to define exactly what needs to be included in the +container. This is done using a file called `Dockerfile`. To begin with, the +Dockerfile consists of: FROM python:2.7 ENV PYTHONUNBUFFERED 1 @@ -18,14 +27,21 @@ Let's set up the three files that'll get us started. First, our app is going to RUN pip install -r requirements.txt ADD . /code/ -That'll install our application inside an image with Python installed alongside all of our Python dependencies. For more information on how to write Dockerfiles, see the [Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/). +This Dockerfile will define an image that is used to build a container that +includes your application and has Python installed alongside all of your Python +dependencies. For more information on how to write Dockerfiles, see the +[Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/). -Second, we define our Python dependencies in a file called `requirements.txt`: +Second, you'll define your Python dependencies in a file called +`requirements.txt`: Django psycopg2 -Simple enough. Finally, this is all tied together with a file called `docker-compose.yml`. It describes the services that our app comprises of (a web server and database), what Docker images they use, how they link together, what volumes will be mounted inside the containers and what ports they expose. +Finally, this is all tied together with a file called `docker-compose.yml`. It +describes the services that comprise your app (here, a web server and database), +which Docker images they use, how they link together, what volumes will be +mounted inside the containers, and what ports they expose. db: image: postgres @@ -39,20 +55,28 @@ Simple enough. Finally, this is all tied together with a file called `docker-com links: - db -See the [`docker-compose.yml` reference](yml.html) for more information on how it works. +See the [`docker-compose.yml` reference](yml.html) for more information on how +this file works. -We can now start a Django project using `docker-compose run`: +### Build the project + +You can now start a Django project with `docker-compose run`: $ docker-compose run web django-admin.py startproject composeexample . -First, Compose will build an image for the `web` service using the `Dockerfile`. It will then run `django-admin.py startproject composeexample .` inside a container using that image. +First, Compose will build an image for the `web` service using the `Dockerfile`. +It will then run `django-admin.py startproject composeexample .` inside a +container built using that image. This will generate a Django app inside the current directory: $ ls Dockerfile docker-compose.yml composeexample manage.py requirements.txt -First thing we need to do is set up the database connection. Replace the `DATABASES = ...` definition in `composeexample/settings.py` to read: +### Connect the database + +Now you need to set up the database connection. Replace the `DATABASES = ...` +definition in `composeexample/settings.py` to read: DATABASES = { 'default': { @@ -64,7 +88,9 @@ First thing we need to do is set up the database connection. Replace the `DATABA } } -These settings are determined by the [postgres](https://registry.hub.docker.com/_/postgres/) Docker image we are using. +These settings are determined by the +[postgres](https://registry.hub.docker.com/_/postgres/) Docker image specified +in the Dockerfile. Then, run `docker-compose up`: @@ -83,13 +109,15 @@ Then, run `docker-compose up`: myapp_web_1 | Starting development server at http://0.0.0.0:8000/ myapp_web_1 | Quit the server with CONTROL-C. -And your Django app should be running at port 8000 on your docker daemon (if you're using boot2docker, `boot2docker ip` will tell you its address). +Your Django app should nw be running at port 8000 on your Docker daemon (if +you're using Boot2docker, `boot2docker ip` will tell you its address). -You can also run management commands with Docker. To set up your database, for example, run `docker-compose up` and in another terminal run: +You can also run management commands with Docker. To set up your database, for +example, run `docker-compose up` and in another terminal run: $ docker-compose run web python manage.py syncdb -## Compose documentation +## More Compose documentation - [Installing Compose](install.md) - [User guide](index.md) diff --git a/docs/rails.md b/docs/rails.md index 67682cf59..1fe484990 100644 --- a/docs/rails.md +++ b/docs/rails.md @@ -1,14 +1,20 @@ ---- -layout: default -title: Getting started with Compose and Rails ---- +page_title: Quickstart Guide: Compose and Rails +page_description: Getting started with Docker Compose and Rails +page_keywords: documentation, docs, docker, compose, orchestration, containers, +rails -Getting started with Compose and Rails -================================== -We're going to use Compose to set up and run a Rails/PostgreSQL app. Before starting, you'll need to have [Compose installed](install.md). +## Getting started with Compose and Rails -Let's set up the three files that'll get us started. First, our app is going to be running inside a Docker container which contains all of its dependencies. We can define what goes inside that Docker container using a file called `Dockerfile`. It'll contain this to start with: +This Quickstart guide will show you how to use Compose to set up and run a Rails/PostgreSQL app. Before starting, you'll need to have [Compose installed](install.md). + +### Define the project + +Start by setting up the three files you'll need to build the app. First, since +your app is going to run inside a Docker container containing all of its +dependencies, you'll need to define exactly what needs to be included in the +container. This is done using a file called `Dockerfile`. To begin with, the +Dockerfile consists of: FROM ruby:2.2.0 RUN apt-get update -qq && apt-get install -y build-essential libpq-dev @@ -18,14 +24,14 @@ Let's set up the three files that'll get us started. First, our app is going to RUN bundle install ADD . /myapp -That'll put our application code inside an image with Ruby, Bundler and all our dependencies. For more information on how to write Dockerfiles, see the [Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/). +That'll put your application code inside an image that will build a container with Ruby, Bundler and all your dependencies inside it. For more information on how to write Dockerfiles, see the [Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/). -Next, we have a bootstrap `Gemfile` which just loads Rails. It'll be overwritten in a moment by `rails new`. +Next, create a bootstrap `Gemfile` which just loads Rails. It'll be overwritten in a moment by `rails new`. source 'https://rubygems.org' gem 'rails', '4.2.0' -Finally, `docker-compose.yml` is where the magic happens. It describes what services our app comprises (a database and a web app), how to get each one's Docker image (the database just runs on a pre-made PostgreSQL image, and the web app is built from the current directory), and the configuration we need to link them together and expose the web app's port. +Finally, `docker-compose.yml` is where the magic happens. This file describes the services that comprise your app (a database and a web app), how to get each one's Docker image (the database just runs on a pre-made PostgreSQL image, and the web app is built from the current directory), and the configuration needed to link them together and expose the web app's port. db: image: postgres @@ -41,11 +47,16 @@ Finally, `docker-compose.yml` is where the magic happens. It describes what serv links: - db -With those files in place, we can now generate the Rails skeleton app using `docker-compose run`: +### Build the project + +With those three files in place, you can now generate the Rails skeleton app +using `docker-compose run`: $ docker-compose run web rails new . --force --database=postgresql --skip-bundle -First, Compose will build the image for the `web` service using the `Dockerfile`. Then it'll run `rails new` inside a new container, using that image. Once it's done, you should have a fresh app generated: +First, Compose will build the image for the `web` service using the +`Dockerfile`. Then it'll run `rails new` inside a new container, using that +image. Once it's done, you should have generated a fresh app: $ ls Dockerfile app docker-compose.yml tmp @@ -54,17 +65,26 @@ First, Compose will build the image for the `web` service using the `Dockerfile` README.rdoc config.ru public Rakefile db test -Uncomment the line in your new `Gemfile` which loads `therubyracer`, so we've got a Javascript runtime: +Uncomment the line in your new `Gemfile` which loads `therubyracer`, so you've +got a Javascript runtime: gem 'therubyracer', platforms: :ruby -Now that we've got a new `Gemfile`, we need to build the image again. (This, and changes to the Dockerfile itself, should be the only times you'll need to rebuild). +Now that you've got a new `Gemfile`, you need to build the image again. (This, +and changes to the Dockerfile itself, should be the only times you'll need to +rebuild.) $ docker-compose build -The app is now bootable, but we're not quite there yet. By default, Rails expects a database to be running on `localhost` - we need to point it at the `db` container instead. We also need to change the database and username to align with the defaults set by the `postgres` image. +### Connect the database -Open up your newly-generated `database.yml`. Replace its contents with the following: +The app is now bootable, but you're not quite there yet. By default, Rails +expects a database to be running on `localhost` - so you need to point it at the +`db` container instead. You also need to change the database and username to +align with the defaults set by the `postgres` image. + +Open up your newly-generated `database.yml`file. Replace its contents with the +following: development: &default adapter: postgresql @@ -79,23 +99,25 @@ Open up your newly-generated `database.yml`. Replace its contents with the follo <<: *default database: myapp_test -We can now boot the app. +You can now boot the app with: $ docker-compose up -If all's well, you should see some PostgreSQL output, and then—after a few seconds—the familiar refrain: +If all's well, you should see some PostgreSQL output, and then—after a few +seconds—the familiar refrain: myapp_web_1 | [2014-01-17 17:16:29] INFO WEBrick 1.3.1 myapp_web_1 | [2014-01-17 17:16:29] INFO ruby 2.2.0 (2014-12-25) [x86_64-linux-gnu] myapp_web_1 | [2014-01-17 17:16:29] INFO WEBrick::HTTPServer#start: pid=1 port=3000 -Finally, we just need to create the database. In another terminal, run: +Finally, you need to create the database. In another terminal, run: $ docker-compose run web rake db:create -And we're rolling—your app should now be running on port 3000 on your docker daemon (if you're using boot2docker, `boot2docker ip` will tell you its address). +That's it. Your app should now be running on port 3000 on your Docker daemon (if +you're using Boot2docker, `boot2docker ip` will tell you its address). -## Compose documentation +## More Compose documentation - [Installing Compose](install.md) - [User guide](index.md) diff --git a/docs/wordpress.md b/docs/wordpress.md index 1fa1d9e33..5a9c37a8d 100644 --- a/docs/wordpress.md +++ b/docs/wordpress.md @@ -1,25 +1,40 @@ ---- -layout: default -title: Getting started with Compose and Wordpress ---- +page_title: Quickstart Guide: Compose and Wordpress +page_description: Getting started with Docker Compose and Rails +page_keywords: documentation, docs, docker, compose, orchestration, containers, +wordpress -Getting started with Compose and Wordpress -====================================== +## Getting started with Compose and Wordpress -Compose makes it nice and easy to run Wordpress in an isolated environment. [Install Compose](install.md), then download Wordpress into the current directory: +You can use Compose to easily run Wordpress in an isolated environment built +with Docker containers. + +### Define the project + +First, [Install Compose](install.md) and then download Wordpress into the +current directory: $ curl https://wordpress.org/latest.tar.gz | tar -xvzf - -This will create a directory called `wordpress`, which you can rename to the name of your project if you wish. Inside that directory, we create `Dockerfile`, a file that defines what environment your app is going to run in: +This will create a directory called `wordpress`. If you wish, you can rename it +to the name of your project. + +Next, inside that directory, create a `Dockerfile`, a file that defines what +environment your app is going to run in. For more information on how to write +Dockerfiles, see the +[Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the +[Dockerfile reference](http://docs.docker.com/reference/builder/). In this case, +your Dockerfile should be: ``` FROM orchardup/php5 ADD . /code ``` -This instructs Docker on how to build an image that contains PHP and Wordpress. For more information on how to write Dockerfiles, see the [Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/). +This tells Docker how to build an image defining a container that contains PHP +and Wordpress. -Next up, `docker-compose.yml` starts our web service and a separate MySQL instance: +Next you'll create a `docker-compose.yml` file that will start your web service +and a separate MySQL instance: ``` web: @@ -37,7 +52,9 @@ db: MYSQL_DATABASE: wordpress ``` -Two supporting files are needed to get this working - first up, `wp-config.php` is the standard Wordpress config file with a single change to point the database configuration at the `db` container: +Two supporting files are needed to get this working - first, `wp-config.php` is +the standard Wordpress config file with a single change to point the database +configuration at the `db` container: ``` Date: Sun, 22 Feb 2015 23:13:15 +0000 Subject: [PATCH 06/46] Add -f flag as option on rm. Signed-off-by: Kingsley Kelly --- compose/cli/main.py | 4 ++-- tests/integration/cli_test.py | 6 ++++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/compose/cli/main.py b/compose/cli/main.py index cfe29cd07..858f19479 100644 --- a/compose/cli/main.py +++ b/compose/cli/main.py @@ -238,8 +238,8 @@ class TopLevelCommand(Command): Usage: rm [options] [SERVICE...] Options: - --force Don't ask to confirm removal - -v Remove volumes associated with containers + -f, --force Don't ask to confirm removal + -v Remove volumes associated with containers """ all_containers = project.containers(service_names=options['SERVICE'], stopped=True) stopped_containers = [c for c in all_containers if not c.is_running] diff --git a/tests/integration/cli_test.py b/tests/integration/cli_test.py index cf9398379..32c4294cc 100644 --- a/tests/integration/cli_test.py +++ b/tests/integration/cli_test.py @@ -295,6 +295,12 @@ class CLITestCase(DockerClientTestCase): self.assertEqual(len(service.containers(stopped=True)), 1) self.command.dispatch(['rm', '--force'], None) self.assertEqual(len(service.containers(stopped=True)), 0) + service = self.project.get_service('simple') + service.create_container() + service.kill() + self.assertEqual(len(service.containers(stopped=True)), 1) + self.command.dispatch(['rm', '-f'], None) + self.assertEqual(len(service.containers(stopped=True)), 0) def test_kill(self): self.command.dispatch(['up', '-d'], None) From 98f32a21e740cc30abdfd9427b047999341b98a6 Mon Sep 17 00:00:00 2001 From: Ben Firshman Date: Fri, 27 Feb 2015 11:18:41 +0000 Subject: [PATCH 07/46] Remove @d11wtq as a maintainer Thanks for all the help! Signed-off-by: Ben Firshman --- MAINTAINERS | 1 - 1 file changed, 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index 0fd7f81c8..8ac3985fa 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1,4 +1,3 @@ Aanand Prasad (@aanand) Ben Firshman (@bfirsh) -Chris Corbyn (@d11wtq) Daniel Nephin (@dnephin) From 0e30d0085bd66bbb91d597b63ce3c72785b9469e Mon Sep 17 00:00:00 2001 From: Alex Brandt Date: Sat, 28 Feb 2015 15:18:29 -0600 Subject: [PATCH 08/46] add bash completion to sdist When installing from the source distribution (for packaging or other purposes), it's convenient to be able to install the bash completion file as part of that process. --- MANIFEST.in | 1 + 1 file changed, 1 insertion(+) diff --git a/MANIFEST.in b/MANIFEST.in index ca9ecbd5b..2acd5ab6b 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -4,6 +4,7 @@ include requirements.txt include requirements-dev.txt include tox.ini include *.md +include contrib/completion/bash/docker-compose recursive-include tests * global-exclude *.pyc global-exclude *.pyo From f1fc1d7a16d2d35f172bd615873ad910ae4e47dc Mon Sep 17 00:00:00 2001 From: Igor Ch Date: Sat, 28 Feb 2015 23:26:20 +0200 Subject: [PATCH 09/46] Move several steps closer to python3 compatibility Signed-off-by: Igor Ch --- compose/cli/command.py | 2 +- compose/cli/log_printer.py | 2 +- compose/container.py | 1 + compose/project.py | 1 + compose/service.py | 4 ++-- requirements.txt | 2 +- tests/integration/service_test.py | 6 +++--- 7 files changed, 10 insertions(+), 8 deletions(-) diff --git a/compose/cli/command.py b/compose/cli/command.py index 67b77f31b..c26f3bc38 100644 --- a/compose/cli/command.py +++ b/compose/cli/command.py @@ -25,7 +25,7 @@ class Command(DocoptCommand): def dispatch(self, *args, **kwargs): try: super(Command, self).dispatch(*args, **kwargs) - except SSLError, e: + except SSLError as e: raise errors.UserError('SSL error: %s' % e) except ConnectionError: if call_silently(['which', 'docker']) != 0: diff --git a/compose/cli/log_printer.py b/compose/cli/log_printer.py index 77fa49ae3..ce7e10653 100644 --- a/compose/cli/log_printer.py +++ b/compose/cli/log_printer.py @@ -46,7 +46,7 @@ class LogPrinter(object): if monochrome: color_fn = no_color else: - color_fn = color_fns.next() + color_fn = next(color_fns) generators.append(self._make_log_generator(container, color_fn)) return generators diff --git a/compose/container.py b/compose/container.py index 692145988..1d044a421 100644 --- a/compose/container.py +++ b/compose/container.py @@ -2,6 +2,7 @@ from __future__ import unicode_literals from __future__ import absolute_import import six +from functools import reduce class Container(object): diff --git a/compose/project.py b/compose/project.py index 601604474..f2fa6a7ee 100644 --- a/compose/project.py +++ b/compose/project.py @@ -2,6 +2,7 @@ from __future__ import unicode_literals from __future__ import absolute_import import logging +from functools import reduce from .service import Service from .container import Container from docker.errors import APIError diff --git a/compose/service.py b/compose/service.py index eb6e86fb5..a21b78b76 100644 --- a/compose/service.py +++ b/compose/service.py @@ -482,7 +482,7 @@ class Service(object): try: all_events = stream_output(build_output, sys.stdout) - except StreamOutputError, e: + except StreamOutputError as e: raise BuildError(self, unicode(e)) image_id = None @@ -641,7 +641,7 @@ def merge_environment(options): else: env.update(options['environment']) - return dict(resolve_env(k, v) for k, v in env.iteritems()) + return dict(resolve_env(k, v) for k, v in env.items()) def split_env(env): diff --git a/requirements.txt b/requirements.txt index a31a19ae9..80b494562 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,5 +4,5 @@ dockerpty==0.3.2 docopt==0.6.1 requests==2.2.1 six==1.7.3 -texttable==0.8.1 +texttable==0.8.2 websocket-client==0.11.0 diff --git a/tests/integration/service_test.py b/tests/integration/service_test.py index 5904eb4ea..7b95b870f 100644 --- a/tests/integration/service_test.py +++ b/tests/integration/service_test.py @@ -472,13 +472,13 @@ class ServiceTest(DockerClientTestCase): def test_split_env(self): service = self.create_service('web', environment=['NORMAL=F1', 'CONTAINS_EQUALS=F=2', 'TRAILING_EQUALS=']) env = create_and_start_container(service).environment - for k,v in {'NORMAL': 'F1', 'CONTAINS_EQUALS': 'F=2', 'TRAILING_EQUALS': ''}.iteritems(): + for k,v in {'NORMAL': 'F1', 'CONTAINS_EQUALS': 'F=2', 'TRAILING_EQUALS': ''}.items(): self.assertEqual(env[k], v) def test_env_from_file_combined_with_env(self): service = self.create_service('web', environment=['ONE=1', 'TWO=2', 'THREE=3'], env_file=['tests/fixtures/env/one.env', 'tests/fixtures/env/two.env']) env = create_and_start_container(service).environment - for k,v in {'ONE': '1', 'TWO': '2', 'THREE': '3', 'FOO': 'baz', 'DOO': 'dah'}.iteritems(): + for k,v in {'ONE': '1', 'TWO': '2', 'THREE': '3', 'FOO': 'baz', 'DOO': 'dah'}.items(): self.assertEqual(env[k], v) def test_resolve_env(self): @@ -488,7 +488,7 @@ class ServiceTest(DockerClientTestCase): os.environ['ENV_DEF'] = 'E3' try: env = create_and_start_container(service).environment - for k,v in {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}.iteritems(): + for k,v in {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}.items(): self.assertEqual(env[k], v) finally: del os.environ['FILE_DEF'] From 8f38b288168a6f8ce4c7df8275cbebb5fdfa73df Mon Sep 17 00:00:00 2001 From: Zoltan Nagy Date: Fri, 20 Feb 2015 17:18:51 +0100 Subject: [PATCH 10/46] Use docker-py 1.0.0 Signed-off-by: Zoltan Nagy --- requirements.txt | 2 +- setup.py | 2 +- tests/unit/cli/docker_client_test.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/requirements.txt b/requirements.txt index a31a19ae9..ecfcd2c5b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ PyYAML==3.10 -docker-py==0.7.1 +docker-py==1.0.0 dockerpty==0.3.2 docopt==0.6.1 requests==2.2.1 diff --git a/setup.py b/setup.py index e1e29744b..4cbffd5f3 100644 --- a/setup.py +++ b/setup.py @@ -30,7 +30,7 @@ install_requires = [ 'requests >= 2.2.1, < 3', 'texttable >= 0.8.1, < 0.9', 'websocket-client >= 0.11.0, < 1.0', - 'docker-py >= 0.6.0, < 0.8', + 'docker-py >= 1.0.0, < 1.1.0', 'dockerpty >= 0.3.2, < 0.4', 'six >= 1.3.0, < 2', ] diff --git a/tests/unit/cli/docker_client_test.py b/tests/unit/cli/docker_client_test.py index abd40ef08..184aff4de 100644 --- a/tests/unit/cli/docker_client_test.py +++ b/tests/unit/cli/docker_client_test.py @@ -19,4 +19,4 @@ class DockerClientTestCase(unittest.TestCase): with mock.patch.dict(os.environ): os.environ['DOCKER_CLIENT_TIMEOUT'] = timeout = "300" client = docker_client.docker_client() - self.assertEqual(client._timeout, int(timeout)) + self.assertEqual(client.timeout, int(timeout)) From f47431d591fddd3a7791fc2e4fbafeae7cc6496a Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Sat, 14 Feb 2015 14:09:55 -0500 Subject: [PATCH 11/46] Resolves #927 - fix merging command line environment with a list in the config Signed-off-by: Daniel Nephin --- compose/cli/main.py | 24 +++++++-------- compose/service.py | 26 +++++++++++----- tests/unit/cli_test.py | 33 ++++++++++++++++++++- tests/unit/service_test.py | 61 +++++++++++++++++++++++++------------- tox.ini | 4 +-- 5 files changed, 103 insertions(+), 45 deletions(-) diff --git a/compose/cli/main.py b/compose/cli/main.py index 858f19479..eee59bb74 100644 --- a/compose/cli/main.py +++ b/compose/cli/main.py @@ -1,26 +1,25 @@ from __future__ import print_function from __future__ import unicode_literals +from inspect import getdoc +from operator import attrgetter import logging -import sys import re import signal -from operator import attrgetter +import sys -from inspect import getdoc +from docker.errors import APIError import dockerpty from .. import __version__ from ..project import NoSuchService, ConfigurationError -from ..service import BuildError, CannotBeScaledError +from ..service import BuildError, CannotBeScaledError, parse_environment from .command import Command +from .docopt_command import NoSuchCommand +from .errors import UserError from .formatter import Formatter from .log_printer import LogPrinter from .utils import yesno -from docker.errors import APIError -from .errors import UserError -from .docopt_command import NoSuchCommand - log = logging.getLogger(__name__) @@ -316,11 +315,10 @@ class TopLevelCommand(Command): } if options['-e']: - for option in options['-e']: - if 'environment' not in service.options: - service.options['environment'] = {} - k, v = option.split('=', 1) - service.options['environment'][k] = v + # Merge environment from config with -e command line + container_options['environment'] = dict( + parse_environment(service.options.get('environment')), + **parse_environment(options['-e'])) if options['--entrypoint']: container_options['entrypoint'] = options.get('--entrypoint') diff --git a/compose/service.py b/compose/service.py index a21b78b76..df6dd6ab5 100644 --- a/compose/service.py +++ b/compose/service.py @@ -8,6 +8,7 @@ from operator import attrgetter import sys from docker.errors import APIError +import six from .container import Container, get_container_name from .progress_stream import stream_output, StreamOutputError @@ -450,7 +451,7 @@ class Service(object): (parse_volume_spec(v).internal, {}) for v in container_options['volumes']) - container_options['environment'] = merge_environment(container_options) + container_options['environment'] = build_environment(container_options) if self.can_be_built(): container_options['image'] = self.full_name @@ -629,19 +630,28 @@ def get_env_files(options): return env_files -def merge_environment(options): +def build_environment(options): env = {} for f in get_env_files(options): env.update(env_vars_from_file(f)) - if 'environment' in options: - if isinstance(options['environment'], list): - env.update(dict(split_env(e) for e in options['environment'])) - else: - env.update(options['environment']) + env.update(parse_environment(options.get('environment'))) + return dict(resolve_env(k, v) for k, v in six.iteritems(env)) - return dict(resolve_env(k, v) for k, v in env.items()) + +def parse_environment(environment): + if not environment: + return {} + + if isinstance(environment, list): + return dict(split_env(e) for e in environment) + + if isinstance(environment, dict): + return environment + + raise ConfigError("environment \"%s\" must be a list or mapping," % + environment) def split_env(env): diff --git a/tests/unit/cli_test.py b/tests/unit/cli_test.py index 57e2f327f..d9a191ef0 100644 --- a/tests/unit/cli_test.py +++ b/tests/unit/cli_test.py @@ -6,12 +6,14 @@ import tempfile import shutil from .. import unittest +import docker import mock +from six import StringIO from compose.cli import main from compose.cli.main import TopLevelCommand from compose.cli.errors import ComposeFileNotFound -from six import StringIO +from compose.service import Service class CLITestCase(unittest.TestCase): @@ -103,6 +105,35 @@ class CLITestCase(unittest.TestCase): self.assertEqual(logging.getLogger().level, logging.DEBUG) self.assertEqual(logging.getLogger('requests').propagate, False) + @mock.patch('compose.cli.main.dockerpty', autospec=True) + def test_run_with_environment_merged_with_options_list(self, mock_dockerpty): + command = TopLevelCommand() + mock_client = mock.create_autospec(docker.Client) + mock_project = mock.Mock() + mock_project.get_service.return_value = Service( + 'service', + client=mock_client, + environment=['FOO=ONE', 'BAR=TWO'], + image='someimage') + + command.run(mock_project, { + 'SERVICE': 'service', + 'COMMAND': None, + '-e': ['BAR=NEW', 'OTHER=THREE'], + '--no-deps': None, + '--allow-insecure-ssl': None, + '-d': True, + '-T': None, + '--entrypoint': None, + '--service-ports': None, + '--rm': None, + }) + + _, _, call_kwargs = mock_client.create_container.mock_calls[0] + self.assertEqual( + call_kwargs['environment'], + {'FOO': 'ONE', 'BAR': 'NEW', 'OTHER': 'THREE'}) + def get_config_filename_for_files(filenames): project_dir = tempfile.mkdtemp() diff --git a/tests/unit/service_test.py b/tests/unit/service_test.py index c7b122fc2..012a51ab6 100644 --- a/tests/unit/service_test.py +++ b/tests/unit/service_test.py @@ -11,14 +11,15 @@ from requests import Response from compose import Service from compose.container import Container from compose.service import ( - ConfigError, - split_port, - build_port_bindings, - parse_volume_spec, - build_volume_binding, APIError, + ConfigError, + build_port_bindings, + build_volume_binding, get_container_name, + parse_environment, parse_repository_tag, + parse_volume_spec, + split_port, ) @@ -326,28 +327,47 @@ class ServiceEnvironmentTest(unittest.TestCase): self.mock_client = mock.create_autospec(docker.Client) self.mock_client.containers.return_value = [] - def test_parse_environment(self): - service = Service('foo', - environment=['NORMAL=F1', 'CONTAINS_EQUALS=F=2', 'TRAILING_EQUALS='], - client=self.mock_client, - image='image_name', - ) - options = service._get_container_create_options({}) + def test_parse_environment_as_list(self): + environment =[ + 'NORMAL=F1', + 'CONTAINS_EQUALS=F=2', + 'TRAILING_EQUALS=' + ] self.assertEqual( - options['environment'], - {'NORMAL': 'F1', 'CONTAINS_EQUALS': 'F=2', 'TRAILING_EQUALS': ''} - ) + parse_environment(environment), + {'NORMAL': 'F1', 'CONTAINS_EQUALS': 'F=2', 'TRAILING_EQUALS': ''}) + + def test_parse_environment_as_dict(self): + environment = { + 'NORMAL': 'F1', + 'CONTAINS_EQUALS': 'F=2', + 'TRAILING_EQUALS': None, + } + self.assertEqual(parse_environment(environment), environment) + + def test_parse_environment_invalid(self): + with self.assertRaises(ConfigError): + parse_environment('a=b') + + def test_parse_environment_empty(self): + self.assertEqual(parse_environment(None), {}) @mock.patch.dict(os.environ) def test_resolve_environment(self): os.environ['FILE_DEF'] = 'E1' os.environ['FILE_DEF_EMPTY'] = 'E2' os.environ['ENV_DEF'] = 'E3' - service = Service('foo', - environment={'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': None, 'NO_DEF': None}, - client=self.mock_client, - image='image_name', - ) + service = Service( + 'foo', + environment={ + 'FILE_DEF': 'F1', + 'FILE_DEF_EMPTY': '', + 'ENV_DEF': None, + 'NO_DEF': None + }, + client=self.mock_client, + image='image_name', + ) options = service._get_container_create_options({}) self.assertEqual( options['environment'], @@ -381,7 +401,6 @@ class ServiceEnvironmentTest(unittest.TestCase): def test_env_nonexistent_file(self): self.assertRaises(ConfigError, lambda: Service('foo', env_file='tests/fixtures/env/nonexistent.env')) - @mock.patch.dict(os.environ) def test_resolve_environment_from_file(self): os.environ['FILE_DEF'] = 'E1' diff --git a/tox.ini b/tox.ini index a20d984b7..6e83fc414 100644 --- a/tox.ini +++ b/tox.ini @@ -8,9 +8,9 @@ deps = -rrequirements-dev.txt commands = nosetests -v {posargs} - flake8 fig + flake8 compose [flake8] # ignore line-length for now ignore = E501,E203 -exclude = fig/packages +exclude = compose/packages From ac7a97f42039e941bdfe1f64d0246b380ef99ff8 Mon Sep 17 00:00:00 2001 From: Harald Albers Date: Mon, 2 Mar 2015 18:33:34 +0100 Subject: [PATCH 12/46] Add -f flag to bash completion for docker-compose rm Signed-off-by: Harald Albers --- contrib/completion/bash/docker-compose | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/completion/bash/docker-compose b/contrib/completion/bash/docker-compose index 03721f505..0f268b39f 100644 --- a/contrib/completion/bash/docker-compose +++ b/contrib/completion/bash/docker-compose @@ -203,7 +203,7 @@ _docker-compose_restart() { _docker-compose_rm() { case "$cur" in -*) - COMPREPLY=( $( compgen -W "--force -v" -- "$cur" ) ) + COMPREPLY=( $( compgen -W "--force -f -v" -- "$cur" ) ) ;; *) __docker-compose_services_stopped From 08f936b2e7ba91e3c6497e12ba4dd798de7158ba Mon Sep 17 00:00:00 2001 From: Ben Firshman Date: Wed, 4 Mar 2015 10:27:06 +0000 Subject: [PATCH 13/46] Fix missing space in rails docs From #1031 Signed-off-by: Ben Firshman --- docs/rails.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/rails.md b/docs/rails.md index 1fe484990..0671d0624 100644 --- a/docs/rails.md +++ b/docs/rails.md @@ -83,7 +83,7 @@ expects a database to be running on `localhost` - so you need to point it at the `db` container instead. You also need to change the database and username to align with the defaults set by the `postgres` image. -Open up your newly-generated `database.yml`file. Replace its contents with the +Open up your newly-generated `database.yml` file. Replace its contents with the following: development: &default From 95f4e2c7c3214c184aae2717417f273ddc6ad2ca Mon Sep 17 00:00:00 2001 From: Gil Clark Date: Fri, 6 Mar 2015 13:30:56 -0800 Subject: [PATCH 14/46] Make volumes_from and net containers first class dependencies and assure that starting order is correct. Added supporting unit and integration tests as well. Signed-off-by: Gil Clark --- compose/cli/main.py | 6 +- compose/project.py | 88 +++++++++++++----- compose/service.py | 35 +++++++- tests/integration/project_test.py | 145 +++++++++++++++++++++++++++--- tests/unit/project_test.py | 107 +++++++++++++++++++++- tests/unit/sort_service_test.py | 89 ++++++++++++++++++ 6 files changed, 429 insertions(+), 41 deletions(-) diff --git a/compose/cli/main.py b/compose/cli/main.py index eee59bb74..15c5e05f4 100644 --- a/compose/cli/main.py +++ b/compose/cli/main.py @@ -292,7 +292,7 @@ class TopLevelCommand(Command): if len(deps) > 0: project.up( service_names=deps, - start_links=True, + start_deps=True, recreate=False, insecure_registry=insecure_registry, detach=options['-d'] @@ -430,13 +430,13 @@ class TopLevelCommand(Command): monochrome = options['--no-color'] - start_links = not options['--no-deps'] + start_deps = not options['--no-deps'] recreate = not options['--no-recreate'] service_names = options['SERVICE'] project.up( service_names=service_names, - start_links=start_links, + start_deps=start_deps, recreate=recreate, insecure_registry=insecure_registry, detach=options['-d'], diff --git a/compose/project.py b/compose/project.py index f2fa6a7ee..794ef2b65 100644 --- a/compose/project.py +++ b/compose/project.py @@ -10,6 +10,17 @@ from docker.errors import APIError log = logging.getLogger(__name__) +def get_service_name_from_net(net_config): + if not net_config: + return + + if not net_config.startswith('container:'): + return + + _, net_name = net_config.split(':', 1) + return net_name + + def sort_service_dicts(services): # Topological sort (Cormen/Tarjan algorithm). unmarked = services[:] @@ -19,6 +30,15 @@ def sort_service_dicts(services): def get_service_names(links): return [link.split(':')[0] for link in links] + def get_service_dependents(service_dict, services): + name = service_dict['name'] + return [ + service for service in services + if (name in get_service_names(service.get('links', [])) or + name in service.get('volumes_from', []) or + name == get_service_name_from_net(service.get('net'))) + ] + def visit(n): if n['name'] in temporary_marked: if n['name'] in get_service_names(n.get('links', [])): @@ -29,8 +49,7 @@ def sort_service_dicts(services): raise DependencyError('Circular import between %s' % ' and '.join(temporary_marked)) if n in unmarked: temporary_marked.add(n['name']) - dependents = [m for m in services if (n['name'] in get_service_names(m.get('links', []))) or (n['name'] in m.get('volumes_from', []))] - for m in dependents: + for m in get_service_dependents(n, services): visit(m) temporary_marked.remove(n['name']) unmarked.remove(n) @@ -60,8 +79,10 @@ class Project(object): for service_dict in sort_service_dicts(service_dicts): links = project.get_links(service_dict) volumes_from = project.get_volumes_from(service_dict) + net = project.get_net(service_dict) - project.services.append(Service(client=client, project=name, links=links, volumes_from=volumes_from, **service_dict)) + project.services.append(Service(client=client, project=name, links=links, net=net, + volumes_from=volumes_from, **service_dict)) return project @classmethod @@ -85,31 +106,31 @@ class Project(object): raise NoSuchService(name) - def get_services(self, service_names=None, include_links=False): + def get_services(self, service_names=None, include_deps=False): """ Returns a list of this project's services filtered by the provided list of names, or all services if service_names is None or []. - If include_links is specified, returns a list including the links for + If include_deps is specified, returns a list including the dependencies for service_names, in order of dependency. Preserves the original order of self.services where possible, - reordering as needed to resolve links. + reordering as needed to resolve dependencies. Raises NoSuchService if any of the named services do not exist. """ if service_names is None or len(service_names) == 0: return self.get_services( service_names=[s.name for s in self.services], - include_links=include_links + include_deps=include_deps ) else: unsorted = [self.get_service(name) for name in service_names] services = [s for s in self.services if s in unsorted] - if include_links: - services = reduce(self._inject_links, services, []) + if include_deps: + services = reduce(self._inject_deps, services, []) uniques = [] [uniques.append(s) for s in services if s not in uniques] @@ -146,6 +167,28 @@ class Project(object): del service_dict['volumes_from'] return volumes_from + def get_net(self, service_dict): + if 'net' in service_dict: + net_name = get_service_name_from_net(service_dict.get('net')) + + if net_name: + try: + net = self.get_service(net_name) + except NoSuchService: + try: + net = Container.from_id(self.client, net_name) + except APIError: + raise ConfigurationError('Serivce "%s" is trying to use the network of "%s", which is not the name of a service or container.' % (service_dict['name'], net_name)) + else: + net = service_dict['net'] + + del service_dict['net'] + + else: + net = 'bridge' + + return net + def start(self, service_names=None, **options): for service in self.get_services(service_names): service.start(**options) @@ -171,13 +214,13 @@ class Project(object): def up(self, service_names=None, - start_links=True, + start_deps=True, recreate=True, insecure_registry=False, detach=False, do_build=True): running_containers = [] - for service in self.get_services(service_names, include_links=start_links): + for service in self.get_services(service_names, include_deps=start_deps): if recreate: for (_, container) in service.recreate_containers( insecure_registry=insecure_registry, @@ -194,7 +237,7 @@ class Project(object): return running_containers def pull(self, service_names=None, insecure_registry=False): - for service in self.get_services(service_names, include_links=True): + for service in self.get_services(service_names, include_deps=True): service.pull(insecure_registry=insecure_registry) def remove_stopped(self, service_names=None, **options): @@ -207,19 +250,22 @@ class Project(object): for service in self.get_services(service_names) if service.has_container(container, one_off=one_off)] - def _inject_links(self, acc, service): - linked_names = service.get_linked_names() + def _inject_deps(self, acc, service): + net_name = service.get_net_name() + dep_names = (service.get_linked_names() + + service.get_volumes_from_names() + + ([net_name] if net_name else [])) - if len(linked_names) > 0: - linked_services = self.get_services( - service_names=linked_names, - include_links=True + if len(dep_names) > 0: + dep_services = self.get_services( + service_names=list(set(dep_names)), + include_deps=True ) else: - linked_services = [] + dep_services = [] - linked_services.append(service) - return acc + linked_services + dep_services.append(service) + return acc + dep_services class NoSuchService(Exception): diff --git a/compose/service.py b/compose/service.py index df6dd6ab5..377198cf4 100644 --- a/compose/service.py +++ b/compose/service.py @@ -88,7 +88,7 @@ ServiceName = namedtuple('ServiceName', 'project service number') class Service(object): - def __init__(self, name, client=None, project='default', links=None, external_links=None, volumes_from=None, **options): + def __init__(self, name, client=None, project='default', links=None, external_links=None, volumes_from=None, net=None, **options): if not re.match('^%s+$' % VALID_NAME_CHARS, name): raise ConfigError('Invalid service name "%s" - only %s are allowed' % (name, VALID_NAME_CHARS)) if not re.match('^%s+$' % VALID_NAME_CHARS, project): @@ -116,6 +116,7 @@ class Service(object): self.links = links or [] self.external_links = external_links or [] self.volumes_from = volumes_from or [] + self.net = net or None self.options = options def containers(self, stopped=False, one_off=False): @@ -320,7 +321,6 @@ class Service(object): if ':' in volume) privileged = options.get('privileged', False) - net = options.get('net', 'bridge') dns = options.get('dns', None) dns_search = options.get('dns_search', None) cap_add = options.get('cap_add', None) @@ -334,7 +334,7 @@ class Service(object): binds=volume_bindings, volumes_from=self._get_volumes_from(intermediate_container), privileged=privileged, - network_mode=net, + network_mode=self._get_net(), dns=dns, dns_search=dns_search, restart_policy=restart, @@ -364,6 +364,15 @@ class Service(object): def get_linked_names(self): return [s.name for (s, _) in self.links] + def get_volumes_from_names(self): + return [s.name for s in self.volumes_from if isinstance(s, Service)] + + def get_net_name(self): + if isinstance(self.net, Service): + return self.net.name + else: + return + def _next_container_name(self, all_containers, one_off=False): bits = [self.project, self.name] if one_off: @@ -399,7 +408,6 @@ class Service(object): for volume_source in self.volumes_from: if isinstance(volume_source, Service): containers = volume_source.containers(stopped=True) - if not containers: volumes_from.append(volume_source.create_container().id) else: @@ -413,6 +421,25 @@ class Service(object): return volumes_from + def _get_net(self): + if not self.net: + return "bridge" + + if isinstance(self.net, Service): + containers = self.net.containers() + if len(containers) > 0: + net = 'container:' + containers[0].id + else: + log.warning("Warning: Service %s is trying to use reuse the network stack " + "of another service that is not running." % (self.net.name)) + net = None + elif isinstance(self.net, Container): + net = 'container:' + self.net.id + else: + net = self.net + + return net + def _get_container_create_options(self, override_options, one_off=False): container_options = dict( (k, self.options[k]) diff --git a/tests/integration/project_test.py b/tests/integration/project_test.py index 2577fd616..17b54daee 100644 --- a/tests/integration/project_test.py +++ b/tests/integration/project_test.py @@ -44,6 +44,63 @@ class ProjectTest(DockerClientTestCase): db = project.get_service('db') self.assertEqual(db.volumes_from, [data_container]) + project.kill() + project.remove_stopped() + + def test_net_from_service(self): + project = Project.from_config( + name='composetest', + config={ + 'net': { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"] + }, + 'web': { + 'image': 'busybox:latest', + 'net': 'container:net', + 'command': ["/bin/sleep", "300"] + }, + }, + client=self.client, + ) + + project.up() + + web = project.get_service('web') + net = project.get_service('net') + self.assertEqual(web._get_net(), 'container:'+net.containers()[0].id) + + project.kill() + project.remove_stopped() + + def test_net_from_container(self): + net_container = Container.create( + self.client, + image='busybox:latest', + name='composetest_net_container', + command='/bin/sleep 300' + ) + net_container.start() + + project = Project.from_config( + name='composetest', + config={ + 'web': { + 'image': 'busybox:latest', + 'net': 'container:composetest_net_container' + }, + }, + client=self.client, + ) + + project.up() + + web = project.get_service('web') + self.assertEqual(web._get_net(), 'container:'+net_container.id) + + project.kill() + project.remove_stopped() + def test_start_stop_kill_remove(self): web = self.create_service('web') db = self.create_service('db') @@ -199,20 +256,86 @@ class ProjectTest(DockerClientTestCase): project.kill() project.remove_stopped() - def test_project_up_with_no_deps(self): - console = self.create_service('console') - db = self.create_service('db', volumes=['/var/db']) - web = self.create_service('web', links=[(db, 'db')]) - - project = Project('composetest', [web, db, console], self.client) + def test_project_up_starts_depends(self): + project = Project.from_config( + name='composetest', + config={ + 'console': { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"], + }, + 'net' : { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"] + }, + 'app': { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"], + 'net': 'container:net' + }, + 'web': { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"], + 'net': 'container:net', + 'links': ['app'] + }, + }, + client=self.client, + ) project.start() self.assertEqual(len(project.containers()), 0) - project.up(['web'], start_links=False) - self.assertEqual(len(project.containers()), 1) - self.assertEqual(len(web.containers()), 1) - self.assertEqual(len(db.containers()), 0) - self.assertEqual(len(console.containers()), 0) + project.up(['web']) + self.assertEqual(len(project.containers()), 3) + self.assertEqual(len(project.get_service('web').containers()), 1) + self.assertEqual(len(project.get_service('app').containers()), 1) + self.assertEqual(len(project.get_service('net').containers()), 1) + self.assertEqual(len(project.get_service('console').containers()), 0) + + project.kill() + project.remove_stopped() + + def test_project_up_with_no_deps(self): + project = Project.from_config( + name='composetest', + config={ + 'console': { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"], + }, + 'net' : { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"] + }, + 'vol': { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"], + 'volumes': ["/tmp"] + }, + 'app': { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"], + 'net': 'container:net' + }, + 'web': { + 'image': 'busybox:latest', + 'command': ["/bin/sleep", "300"], + 'net': 'container:net', + 'links': ['app'], + 'volumes_from': ['vol'] + }, + }, + client=self.client, + ) + project.start() + self.assertEqual(len(project.containers()), 0) + + project.up(['web'], start_deps=False) + self.assertEqual(len(project.containers(stopped=True)), 2) + self.assertEqual(len(project.get_service('web').containers()), 1) + self.assertEqual(len(project.get_service('vol').containers(stopped=True)), 1) + self.assertEqual(len(project.get_service('net').containers()), 0) + self.assertEqual(len(project.get_service('console').containers()), 0) project.kill() project.remove_stopped() diff --git a/tests/unit/project_test.py b/tests/unit/project_test.py index d7aca64cf..b06e14e58 100644 --- a/tests/unit/project_test.py +++ b/tests/unit/project_test.py @@ -2,6 +2,10 @@ from __future__ import unicode_literals from .. import unittest from compose.service import Service from compose.project import Project, ConfigurationError +from compose.container import Container + +import mock +import docker class ProjectTest(unittest.TestCase): def test_from_dict(self): @@ -120,7 +124,7 @@ class ProjectTest(unittest.TestCase): ) project = Project('test', [web, db, cache, console], None) self.assertEqual( - project.get_services(['console'], include_links=True), + project.get_services(['console'], include_deps=True), [db, web, console] ) @@ -136,6 +140,105 @@ class ProjectTest(unittest.TestCase): ) project = Project('test', [web, db], None) self.assertEqual( - project.get_services(['web', 'db'], include_links=True), + project.get_services(['web', 'db'], include_deps=True), [db, web] ) + + def test_use_volumes_from_container(self): + container_id = 'aabbccddee' + container_dict = dict(Name='aaa', Id=container_id) + mock_client = mock.create_autospec(docker.Client) + mock_client.inspect_container.return_value = container_dict + project = Project.from_dicts('test', [ + { + 'name': 'test', + 'image': 'busybox:latest', + 'volumes_from': ['aaa'] + } + ], mock_client) + self.assertEqual(project.get_service('test')._get_volumes_from(), [container_id]) + + def test_use_volumes_from_service_no_container(self): + container_name = 'test_vol_1' + mock_client = mock.create_autospec(docker.Client) + mock_client.containers.return_value = [ + { + "Name": container_name, + "Names": [container_name], + "Id": container_name, + "Image": 'busybox:latest' + } + ] + project = Project.from_dicts('test', [ + { + 'name': 'vol', + 'image': 'busybox:latest' + }, + { + 'name': 'test', + 'image': 'busybox:latest', + 'volumes_from': ['vol'] + } + ], mock_client) + self.assertEqual(project.get_service('test')._get_volumes_from(), [container_name]) + + @mock.patch.object(Service, 'containers') + def test_use_volumes_from_service_container(self, mock_return): + container_ids = ['aabbccddee', '12345'] + mock_return.return_value = [ + mock.Mock(id=container_id, spec=Container) + for container_id in container_ids] + + project = Project.from_dicts('test', [ + { + 'name': 'vol', + 'image': 'busybox:latest' + }, + { + 'name': 'test', + 'image': 'busybox:latest', + 'volumes_from': ['vol'] + } + ], None) + self.assertEqual(project.get_service('test')._get_volumes_from(), container_ids) + + def test_use_net_from_container(self): + container_id = 'aabbccddee' + container_dict = dict(Name='aaa', Id=container_id) + mock_client = mock.create_autospec(docker.Client) + mock_client.inspect_container.return_value = container_dict + project = Project.from_dicts('test', [ + { + 'name': 'test', + 'image': 'busybox:latest', + 'net': 'container:aaa' + } + ], mock_client) + service = project.get_service('test') + self.assertEqual(service._get_net(), 'container:'+container_id) + + def test_use_net_from_service(self): + container_name = 'test_aaa_1' + mock_client = mock.create_autospec(docker.Client) + mock_client.containers.return_value = [ + { + "Name": container_name, + "Names": [container_name], + "Id": container_name, + "Image": 'busybox:latest' + } + ] + project = Project.from_dicts('test', [ + { + 'name': 'aaa', + 'image': 'busybox:latest' + }, + { + 'name': 'test', + 'image': 'busybox:latest', + 'net': 'container:aaa' + } + ], mock_client) + + service = project.get_service('test') + self.assertEqual(service._get_net(), 'container:'+container_name) diff --git a/tests/unit/sort_service_test.py b/tests/unit/sort_service_test.py index 420353c8a..f42a94748 100644 --- a/tests/unit/sort_service_test.py +++ b/tests/unit/sort_service_test.py @@ -65,6 +65,95 @@ class SortServiceTest(unittest.TestCase): self.assertEqual(sorted_services[1]['name'], 'parent') self.assertEqual(sorted_services[2]['name'], 'grandparent') + def test_sort_service_dicts_4(self): + services = [ + { + 'name': 'child' + }, + { + 'name': 'parent', + 'volumes_from': ['child'] + }, + { + 'links': ['parent'], + 'name': 'grandparent' + }, + ] + + sorted_services = sort_service_dicts(services) + self.assertEqual(len(sorted_services), 3) + self.assertEqual(sorted_services[0]['name'], 'child') + self.assertEqual(sorted_services[1]['name'], 'parent') + self.assertEqual(sorted_services[2]['name'], 'grandparent') + + def test_sort_service_dicts_5(self): + services = [ + { + 'links': ['parent'], + 'name': 'grandparent' + }, + { + 'name': 'parent', + 'net': 'container:child' + }, + { + 'name': 'child' + } + ] + + sorted_services = sort_service_dicts(services) + self.assertEqual(len(sorted_services), 3) + self.assertEqual(sorted_services[0]['name'], 'child') + self.assertEqual(sorted_services[1]['name'], 'parent') + self.assertEqual(sorted_services[2]['name'], 'grandparent') + + def test_sort_service_dicts_6(self): + services = [ + { + 'links': ['parent'], + 'name': 'grandparent' + }, + { + 'name': 'parent', + 'volumes_from': ['child'] + }, + { + 'name': 'child' + } + ] + + sorted_services = sort_service_dicts(services) + self.assertEqual(len(sorted_services), 3) + self.assertEqual(sorted_services[0]['name'], 'child') + self.assertEqual(sorted_services[1]['name'], 'parent') + self.assertEqual(sorted_services[2]['name'], 'grandparent') + + def test_sort_service_dicts_7(self): + services = [ + { + 'net': 'container:three', + 'name': 'four' + }, + { + 'links': ['two'], + 'name': 'three' + }, + { + 'name': 'two', + 'volumes_from': ['one'] + }, + { + 'name': 'one' + } + ] + + sorted_services = sort_service_dicts(services) + self.assertEqual(len(sorted_services), 4) + self.assertEqual(sorted_services[0]['name'], 'one') + self.assertEqual(sorted_services[1]['name'], 'two') + self.assertEqual(sorted_services[2]['name'], 'three') + self.assertEqual(sorted_services[3]['name'], 'four') + def test_sort_service_dicts_circular_imports(self): services = [ { From 74440b2f921e52746be20071ac5a451029a6d66d Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Mon, 23 Feb 2015 11:01:37 +0000 Subject: [PATCH 15/46] Run tests using Docker-in-Docker so we can test multiple versions Signed-off-by: Aanand Prasad --- CONTRIBUTING.md | 13 +++++++ Dockerfile | 15 +++++++- Dockerfile.tests | 5 +++ script/test | 20 ++++++++-- script/wrapdocker | 98 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 147 insertions(+), 4 deletions(-) create mode 100644 Dockerfile.tests create mode 100755 script/wrapdocker diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 00bb7f437..22cbdcf80 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -24,8 +24,21 @@ that should get you started. ## Running the test suite +Use the test script to run linting checks and then the full test suite: + $ script/test +Tests are run against a Docker daemon inside a container, so that we can test against multiple Docker versions. By default they'll run against only the latest Docker version - set the `DOCKER_VERSIONS` environment variable to "all" to run against all supported versions: + + $ DOCKER_VERSIONS=all script/test + +Arguments to `script/test` are passed through to the `nosetests` executable, so you can specify a test directory, file, module, class or method: + + $ script/test tests/unit + $ script/test tests/unit/cli_test.py + $ script/test tests.integration.service_test + $ script/test tests.integration.service_test:ServiceTest.test_containers + ## Building binaries Linux: diff --git a/Dockerfile b/Dockerfile index ee9fb4a2f..fc553cef0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,5 +1,18 @@ FROM debian:wheezy -RUN apt-get update -qq && apt-get install -qy python python-pip python-dev git && apt-get clean + +RUN apt-get update -qq + +# Compose dependencies +RUN apt-get install -qqy python python-pip python-dev git + +# Test dependencies +RUN apt-get install -qqy apt-transport-https ca-certificates curl lxc iptables +RUN curl https://get.docker.com/builds/Linux/x86_64/docker-1.3.3 > /usr/local/bin/docker-1.3.3 && chmod +x /usr/local/bin/docker-1.3.3 +RUN curl https://get.docker.com/builds/Linux/x86_64/docker-1.4.1 > /usr/local/bin/docker-1.4.1 && chmod +x /usr/local/bin/docker-1.4.1 +RUN curl https://get.docker.com/builds/Linux/x86_64/docker-1.5.0 > /usr/local/bin/docker-1.5.0 && chmod +x /usr/local/bin/docker-1.5.0 + +RUN apt-get clean + RUN useradd -d /home/user -m -s /bin/bash user WORKDIR /code/ diff --git a/Dockerfile.tests b/Dockerfile.tests new file mode 100644 index 000000000..f30564d2b --- /dev/null +++ b/Dockerfile.tests @@ -0,0 +1,5 @@ +FROM docker-compose + +ADD script/wrapdocker /usr/local/bin/wrapdocker +VOLUME /var/lib/docker +ENTRYPOINT ["/usr/local/bin/wrapdocker"] diff --git a/script/test b/script/test index fef16b807..461fe7d13 100755 --- a/script/test +++ b/script/test @@ -1,5 +1,19 @@ -#!/bin/sh +#!/bin/bash +# See CONTRIBUTING.md for usage. + set -ex + docker build -t docker-compose . -docker run -v /var/run/docker.sock:/var/run/docker.sock --rm --entrypoint flake8 docker-compose compose -docker run -v /var/run/docker.sock:/var/run/docker.sock --rm --entrypoint nosetests docker-compose $@ +docker run --privileged --rm --entrypoint flake8 docker-compose compose + +docker build -f Dockerfile.tests -t docker-compose-tests . + +if [ "$DOCKER_VERSIONS" == "" ]; then + DOCKER_VERSIONS="1.5.0" +elif [ "$DOCKER_VERSIONS" == "all" ]; then + DOCKER_VERSIONS="1.3.3 1.4.1 1.5.0" +fi + +for version in $DOCKER_VERSIONS; do + docker run --privileged --rm -e "DOCKER_VERSION=$version" docker-compose-tests nosetests "$@" +done diff --git a/script/wrapdocker b/script/wrapdocker new file mode 100755 index 000000000..e5bcbad6c --- /dev/null +++ b/script/wrapdocker @@ -0,0 +1,98 @@ +#!/bin/bash +# Adapted from https://github.com/jpetazzo/dind + +# First, make sure that cgroups are mounted correctly. +CGROUP=/sys/fs/cgroup +: {LOG:=stdio} + +[ -d $CGROUP ] || + mkdir $CGROUP + +mountpoint -q $CGROUP || + mount -n -t tmpfs -o uid=0,gid=0,mode=0755 cgroup $CGROUP || { + echo "Could not make a tmpfs mount. Did you use --privileged?" + exit 1 + } + +if [ -d /sys/kernel/security ] && ! mountpoint -q /sys/kernel/security +then + mount -t securityfs none /sys/kernel/security || { + echo "Could not mount /sys/kernel/security." + echo "AppArmor detection and --privileged mode might break." + } +fi + +# Mount the cgroup hierarchies exactly as they are in the parent system. +for SUBSYS in $(cut -d: -f2 /proc/1/cgroup) +do + [ -d $CGROUP/$SUBSYS ] || mkdir $CGROUP/$SUBSYS + mountpoint -q $CGROUP/$SUBSYS || + mount -n -t cgroup -o $SUBSYS cgroup $CGROUP/$SUBSYS + + # The two following sections address a bug which manifests itself + # by a cryptic "lxc-start: no ns_cgroup option specified" when + # trying to start containers withina container. + # The bug seems to appear when the cgroup hierarchies are not + # mounted on the exact same directories in the host, and in the + # container. + + # Named, control-less cgroups are mounted with "-o name=foo" + # (and appear as such under /proc//cgroup) but are usually + # mounted on a directory named "foo" (without the "name=" prefix). + # Systemd and OpenRC (and possibly others) both create such a + # cgroup. To avoid the aforementioned bug, we symlink "foo" to + # "name=foo". This shouldn't have any adverse effect. + echo $SUBSYS | grep -q ^name= && { + NAME=$(echo $SUBSYS | sed s/^name=//) + ln -s $SUBSYS $CGROUP/$NAME + } + + # Likewise, on at least one system, it has been reported that + # systemd would mount the CPU and CPU accounting controllers + # (respectively "cpu" and "cpuacct") with "-o cpuacct,cpu" + # but on a directory called "cpu,cpuacct" (note the inversion + # in the order of the groups). This tries to work around it. + [ $SUBSYS = cpuacct,cpu ] && ln -s $SUBSYS $CGROUP/cpu,cpuacct +done + +# Note: as I write those lines, the LXC userland tools cannot setup +# a "sub-container" properly if the "devices" cgroup is not in its +# own hierarchy. Let's detect this and issue a warning. +grep -q :devices: /proc/1/cgroup || + echo "WARNING: the 'devices' cgroup should be in its own hierarchy." +grep -qw devices /proc/1/cgroup || + echo "WARNING: it looks like the 'devices' cgroup is not mounted." + +# Now, close extraneous file descriptors. +pushd /proc/self/fd >/dev/null +for FD in * +do + case "$FD" in + # Keep stdin/stdout/stderr + [012]) + ;; + # Nuke everything else + *) + eval exec "$FD>&-" + ;; + esac +done +popd >/dev/null + +if [ "$DOCKER_VERSION" == "" ]; then + DOCKER_VERSION="1.5.0" +fi + +ln -s "/usr/local/bin/docker-$DOCKER_VERSION" "/usr/local/bin/docker" + +# If a pidfile is still around (for example after a container restart), +# delete it so that docker can start. +rm -rf /var/run/docker.pid +docker -d $DOCKER_DAEMON_ARGS &>/var/log/docker.log & + +>&2 echo "Waiting for Docker to start..." +while ! docker ps &>/dev/null; do + sleep 1 +done + +exec "$@" From 42e6296b0ee9e9c1ccb1c9ff249e33775108eba2 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Wed, 25 Feb 2015 13:52:55 +0000 Subject: [PATCH 16/46] Kick everything off from a single container Signed-off-by: Aanand Prasad --- Dockerfile | 30 ++++++++++----- Dockerfile.tests | 5 --- script/dind | 88 ++++++++++++++++++++++++++++++++++++++++++++ script/test | 23 +++++------- script/test-versions | 26 +++++++++++++ script/wrapdocker | 80 +--------------------------------------- 6 files changed, 145 insertions(+), 107 deletions(-) delete mode 100644 Dockerfile.tests create mode 100755 script/dind create mode 100755 script/test-versions diff --git a/Dockerfile b/Dockerfile index fc553cef0..d7a6019aa 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,17 +1,27 @@ FROM debian:wheezy -RUN apt-get update -qq +RUN set -ex; \ + apt-get update -qq; \ + apt-get install -y \ + python \ + python-pip \ + python-dev \ + git \ + apt-transport-https \ + ca-certificates \ + curl \ + lxc \ + iptables \ + ; \ + rm -rf /var/lib/apt/lists/* -# Compose dependencies -RUN apt-get install -qqy python python-pip python-dev git +ENV ALL_DOCKER_VERSIONS 1.3.3 1.4.1 1.5.0 -# Test dependencies -RUN apt-get install -qqy apt-transport-https ca-certificates curl lxc iptables -RUN curl https://get.docker.com/builds/Linux/x86_64/docker-1.3.3 > /usr/local/bin/docker-1.3.3 && chmod +x /usr/local/bin/docker-1.3.3 -RUN curl https://get.docker.com/builds/Linux/x86_64/docker-1.4.1 > /usr/local/bin/docker-1.4.1 && chmod +x /usr/local/bin/docker-1.4.1 -RUN curl https://get.docker.com/builds/Linux/x86_64/docker-1.5.0 > /usr/local/bin/docker-1.5.0 && chmod +x /usr/local/bin/docker-1.5.0 - -RUN apt-get clean +RUN set -ex; \ + for v in ${ALL_DOCKER_VERSIONS}; do \ + curl https://get.docker.com/builds/Linux/x86_64/docker-$v -o /usr/local/bin/docker-$v; \ + chmod +x /usr/local/bin/docker-$v; \ + done RUN useradd -d /home/user -m -s /bin/bash user WORKDIR /code/ diff --git a/Dockerfile.tests b/Dockerfile.tests deleted file mode 100644 index f30564d2b..000000000 --- a/Dockerfile.tests +++ /dev/null @@ -1,5 +0,0 @@ -FROM docker-compose - -ADD script/wrapdocker /usr/local/bin/wrapdocker -VOLUME /var/lib/docker -ENTRYPOINT ["/usr/local/bin/wrapdocker"] diff --git a/script/dind b/script/dind new file mode 100755 index 000000000..f8fae6379 --- /dev/null +++ b/script/dind @@ -0,0 +1,88 @@ +#!/bin/bash +set -e + +# DinD: a wrapper script which allows docker to be run inside a docker container. +# Original version by Jerome Petazzoni +# See the blog post: http://blog.docker.com/2013/09/docker-can-now-run-within-docker/ +# +# This script should be executed inside a docker container in privilieged mode +# ('docker run --privileged', introduced in docker 0.6). + +# Usage: dind CMD [ARG...] + +# apparmor sucks and Docker needs to know that it's in a container (c) @tianon +export container=docker + +# First, make sure that cgroups are mounted correctly. +CGROUP=/cgroup + +mkdir -p "$CGROUP" + +if ! mountpoint -q "$CGROUP"; then + mount -n -t tmpfs -o uid=0,gid=0,mode=0755 cgroup $CGROUP || { + echo >&2 'Could not make a tmpfs mount. Did you use --privileged?' + exit 1 + } +fi + +if [ -d /sys/kernel/security ] && ! mountpoint -q /sys/kernel/security; then + mount -t securityfs none /sys/kernel/security || { + echo >&2 'Could not mount /sys/kernel/security.' + echo >&2 'AppArmor detection and -privileged mode might break.' + } +fi + +# Mount the cgroup hierarchies exactly as they are in the parent system. +for SUBSYS in $(cut -d: -f2 /proc/1/cgroup); do + mkdir -p "$CGROUP/$SUBSYS" + if ! mountpoint -q $CGROUP/$SUBSYS; then + mount -n -t cgroup -o "$SUBSYS" cgroup "$CGROUP/$SUBSYS" + fi + + # The two following sections address a bug which manifests itself + # by a cryptic "lxc-start: no ns_cgroup option specified" when + # trying to start containers withina container. + # The bug seems to appear when the cgroup hierarchies are not + # mounted on the exact same directories in the host, and in the + # container. + + # Named, control-less cgroups are mounted with "-o name=foo" + # (and appear as such under /proc//cgroup) but are usually + # mounted on a directory named "foo" (without the "name=" prefix). + # Systemd and OpenRC (and possibly others) both create such a + # cgroup. To avoid the aforementioned bug, we symlink "foo" to + # "name=foo". This shouldn't have any adverse effect. + name="${SUBSYS#name=}" + if [ "$name" != "$SUBSYS" ]; then + ln -s "$SUBSYS" "$CGROUP/$name" + fi + + # Likewise, on at least one system, it has been reported that + # systemd would mount the CPU and CPU accounting controllers + # (respectively "cpu" and "cpuacct") with "-o cpuacct,cpu" + # but on a directory called "cpu,cpuacct" (note the inversion + # in the order of the groups). This tries to work around it. + if [ "$SUBSYS" = 'cpuacct,cpu' ]; then + ln -s "$SUBSYS" "$CGROUP/cpu,cpuacct" + fi +done + +# Note: as I write those lines, the LXC userland tools cannot setup +# a "sub-container" properly if the "devices" cgroup is not in its +# own hierarchy. Let's detect this and issue a warning. +if ! grep -q :devices: /proc/1/cgroup; then + echo >&2 'WARNING: the "devices" cgroup should be in its own hierarchy.' +fi +if ! grep -qw devices /proc/1/cgroup; then + echo >&2 'WARNING: it looks like the "devices" cgroup is not mounted.' +fi + +# Mount /tmp +mount -t tmpfs none /tmp + +if [ $# -gt 0 ]; then + exec "$@" +fi + +echo >&2 'ERROR: No command specified.' +echo >&2 'You probably want to run hack/make.sh, or maybe a shell?' diff --git a/script/test b/script/test index 461fe7d13..f278023a0 100755 --- a/script/test +++ b/script/test @@ -3,17 +3,14 @@ set -ex -docker build -t docker-compose . -docker run --privileged --rm --entrypoint flake8 docker-compose compose +TAG="docker-compose:$(git rev-parse --short HEAD)" -docker build -f Dockerfile.tests -t docker-compose-tests . - -if [ "$DOCKER_VERSIONS" == "" ]; then - DOCKER_VERSIONS="1.5.0" -elif [ "$DOCKER_VERSIONS" == "all" ]; then - DOCKER_VERSIONS="1.3.3 1.4.1 1.5.0" -fi - -for version in $DOCKER_VERSIONS; do - docker run --privileged --rm -e "DOCKER_VERSION=$version" docker-compose-tests nosetests "$@" -done +docker build -t "$TAG" . +docker run \ + --rm \ + --volume="/var/run/docker.sock:/var/run/docker.sock" \ + -e DOCKER_VERSIONS \ + -e "TAG=$TAG" \ + --entrypoint="script/test-versions" \ + "$TAG" \ + "$@" diff --git a/script/test-versions b/script/test-versions new file mode 100755 index 000000000..9f30eecaa --- /dev/null +++ b/script/test-versions @@ -0,0 +1,26 @@ +#!/bin/bash +# This should be run inside a container built from the Dockerfile +# at the root of the repo - script/test will do it automatically. + +set -e + +>&2 echo "Running lint checks" +flake8 compose + +if [ "$DOCKER_VERSIONS" == "" ]; then + DOCKER_VERSIONS="1.5.0" +elif [ "$DOCKER_VERSIONS" == "all" ]; then + DOCKER_VERSIONS="$ALL_DOCKER_VERSIONS" +fi + +for version in $DOCKER_VERSIONS; do + >&2 echo "Running tests against Docker $version" + docker-1.5.0 run \ + --rm \ + --privileged \ + --volume="/var/lib/docker" \ + -e "DOCKER_VERSION=$version" \ + --entrypoint="script/dind" \ + "$TAG" \ + script/wrapdocker nosetests "$@" +done diff --git a/script/wrapdocker b/script/wrapdocker index e5bcbad6c..20dc9e3ce 100755 --- a/script/wrapdocker +++ b/script/wrapdocker @@ -1,83 +1,4 @@ #!/bin/bash -# Adapted from https://github.com/jpetazzo/dind - -# First, make sure that cgroups are mounted correctly. -CGROUP=/sys/fs/cgroup -: {LOG:=stdio} - -[ -d $CGROUP ] || - mkdir $CGROUP - -mountpoint -q $CGROUP || - mount -n -t tmpfs -o uid=0,gid=0,mode=0755 cgroup $CGROUP || { - echo "Could not make a tmpfs mount. Did you use --privileged?" - exit 1 - } - -if [ -d /sys/kernel/security ] && ! mountpoint -q /sys/kernel/security -then - mount -t securityfs none /sys/kernel/security || { - echo "Could not mount /sys/kernel/security." - echo "AppArmor detection and --privileged mode might break." - } -fi - -# Mount the cgroup hierarchies exactly as they are in the parent system. -for SUBSYS in $(cut -d: -f2 /proc/1/cgroup) -do - [ -d $CGROUP/$SUBSYS ] || mkdir $CGROUP/$SUBSYS - mountpoint -q $CGROUP/$SUBSYS || - mount -n -t cgroup -o $SUBSYS cgroup $CGROUP/$SUBSYS - - # The two following sections address a bug which manifests itself - # by a cryptic "lxc-start: no ns_cgroup option specified" when - # trying to start containers withina container. - # The bug seems to appear when the cgroup hierarchies are not - # mounted on the exact same directories in the host, and in the - # container. - - # Named, control-less cgroups are mounted with "-o name=foo" - # (and appear as such under /proc//cgroup) but are usually - # mounted on a directory named "foo" (without the "name=" prefix). - # Systemd and OpenRC (and possibly others) both create such a - # cgroup. To avoid the aforementioned bug, we symlink "foo" to - # "name=foo". This shouldn't have any adverse effect. - echo $SUBSYS | grep -q ^name= && { - NAME=$(echo $SUBSYS | sed s/^name=//) - ln -s $SUBSYS $CGROUP/$NAME - } - - # Likewise, on at least one system, it has been reported that - # systemd would mount the CPU and CPU accounting controllers - # (respectively "cpu" and "cpuacct") with "-o cpuacct,cpu" - # but on a directory called "cpu,cpuacct" (note the inversion - # in the order of the groups). This tries to work around it. - [ $SUBSYS = cpuacct,cpu ] && ln -s $SUBSYS $CGROUP/cpu,cpuacct -done - -# Note: as I write those lines, the LXC userland tools cannot setup -# a "sub-container" properly if the "devices" cgroup is not in its -# own hierarchy. Let's detect this and issue a warning. -grep -q :devices: /proc/1/cgroup || - echo "WARNING: the 'devices' cgroup should be in its own hierarchy." -grep -qw devices /proc/1/cgroup || - echo "WARNING: it looks like the 'devices' cgroup is not mounted." - -# Now, close extraneous file descriptors. -pushd /proc/self/fd >/dev/null -for FD in * -do - case "$FD" in - # Keep stdin/stdout/stderr - [012]) - ;; - # Nuke everything else - *) - eval exec "$FD>&-" - ;; - esac -done -popd >/dev/null if [ "$DOCKER_VERSION" == "" ]; then DOCKER_VERSION="1.5.0" @@ -95,4 +16,5 @@ while ! docker ps &>/dev/null; do sleep 1 done +>&2 echo ">" "$@" exec "$@" From 2534a0964fbdf1e6b3d59fed81946c2fb7bb0b2a Mon Sep 17 00:00:00 2001 From: Paul Horn Date: Wed, 21 Jan 2015 00:22:30 +0100 Subject: [PATCH 17/46] Add timeout flag to stop, restart, and up The commands `stop`, `restart`, and `up` now support a flag `--timeout`. It represents the number of seconds to give the services to comply to the command. In case of `up`, this is only relevant if running in attached mode. Signed-off-by: Paul Horn --- compose/cli/main.py | 45 ++++++++++++++++++-------- contrib/completion/bash/docker-compose | 42 +++++++++++++++++++++--- 2 files changed, 68 insertions(+), 19 deletions(-) diff --git a/compose/cli/main.py b/compose/cli/main.py index eee59bb74..68c7da526 100644 --- a/compose/cli/main.py +++ b/compose/cli/main.py @@ -387,17 +387,29 @@ class TopLevelCommand(Command): They can be started again with `docker-compose start`. - Usage: stop [SERVICE...] + Usage: stop [options] [SERVICE...] + + Options: + -t, --timeout TIMEOUT Specify a shutdown timeout in seconds. + (default: 10) """ - project.stop(service_names=options['SERVICE']) + timeout = options.get('--timeout') + params = {} if timeout is None else {'timeout': int(timeout)} + project.stop(service_names=options['SERVICE'], **params) def restart(self, project, options): """ Restart running containers. - Usage: restart [SERVICE...] + Usage: restart [options] [SERVICE...] + + Options: + -t, --timeout TIMEOUT Specify a shutdown timeout in seconds. + (default: 10) """ - project.restart(service_names=options['SERVICE']) + timeout = options.get('--timeout') + params = {} if timeout is None else {'timeout': int(timeout)} + project.restart(service_names=options['SERVICE'], **params) def up(self, project, options): """ @@ -416,14 +428,17 @@ class TopLevelCommand(Command): Usage: up [options] [SERVICE...] Options: - --allow-insecure-ssl Allow insecure connections to the docker - registry - -d Detached mode: Run containers in the background, - print new container names. - --no-color Produce monochrome output. - --no-deps Don't start linked services. - --no-recreate If containers already exist, don't recreate them. - --no-build Don't build an image, even if it's missing + --allow-insecure-ssl Allow insecure connections to the docker + registry + -d Detached mode: Run containers in the background, + print new container names. + --no-color Produce monochrome output. + --no-deps Don't start linked services. + --no-recreate If containers already exist, don't recreate them. + --no-build Don't build an image, even if it's missing + -t, --timeout TIMEOUT When attached, use this timeout in seconds + for the shutdown. (default: 10) + """ insecure_registry = options['--allow-insecure-ssl'] detached = options['-d'] @@ -439,7 +454,7 @@ class TopLevelCommand(Command): start_links=start_links, recreate=recreate, insecure_registry=insecure_registry, - detach=options['-d'], + detach=detached, do_build=not options['--no-build'], ) @@ -458,7 +473,9 @@ class TopLevelCommand(Command): signal.signal(signal.SIGINT, handler) print("Gracefully stopping... (press Ctrl+C again to force)") - project.stop(service_names=service_names) + timeout = options.get('--timeout') + params = {} if timeout is None else {'timeout': int(timeout)} + project.stop(service_names=service_names, **params) def list_containers(containers): diff --git a/contrib/completion/bash/docker-compose b/contrib/completion/bash/docker-compose index 587d70a02..af3368036 100644 --- a/contrib/completion/bash/docker-compose +++ b/contrib/completion/bash/docker-compose @@ -1,7 +1,7 @@ #!bash # # bash completion for docker-compose -# +# # This work is based on the completion for the docker command. # # This script provides completion of: @@ -196,7 +196,20 @@ _docker-compose_pull() { _docker-compose_restart() { - __docker-compose_services_running + case "$prev" in + -t | --timeout) + return + ;; + esac + + case "$cur" in + -*) + COMPREPLY=( $( compgen -W "-t --timeout" -- "$cur" ) ) + ;; + *) + __docker-compose_services_running + ;; + esac } @@ -221,7 +234,7 @@ _docker-compose_run() { ;; --entrypoint) return - ;; + ;; esac case "$cur" in @@ -254,14 +267,33 @@ _docker-compose_start() { _docker-compose_stop() { - __docker-compose_services_running + case "$prev" in + -t | --timeout) + return + ;; + esac + + case "$cur" in + -*) + COMPREPLY=( $( compgen -W "-t --timeout" -- "$cur" ) ) + ;; + *) + __docker-compose_services_running + ;; + esac } _docker-compose_up() { + case "$prev" in + -t | --timeout) + return + ;; + esac + case "$cur" in -*) - COMPREPLY=( $( compgen -W "--allow-insecure-ssl -d --no-build --no-color --no-deps --no-recreate" -- "$cur" ) ) + COMPREPLY=( $( compgen -W "--allow-insecure-ssl -d --no-build --no-color --no-deps --no-recreate -t --timeout" -- "$cur" ) ) ;; *) __docker-compose_services_all From 86b723e2273e9bac1a6f5b3299ee1394bfa3ec9c Mon Sep 17 00:00:00 2001 From: Ian VanSchooten Date: Sat, 14 Feb 2015 21:08:47 -0500 Subject: [PATCH 18/46] Provide user override option on command line Allows overriding a user on the command line from the one specified in the docker-compose.yml The added tests verify that a specified user overrides a default user in the docker-compose.yml file. Based on commit f2f01e207b491866349db7168e3d48082d7abdda by @chmouel Signed-off-by: Ian VanSchooten --- compose/cli/main.py | 5 +++++ .../user-composefile/docker-compose.yml | 4 ++++ tests/integration/cli_test.py | 22 +++++++++++++++++++ tests/unit/cli_test.py | 1 + 4 files changed, 32 insertions(+) create mode 100644 tests/fixtures/user-composefile/docker-compose.yml diff --git a/compose/cli/main.py b/compose/cli/main.py index eee59bb74..cb6866a0e 100644 --- a/compose/cli/main.py +++ b/compose/cli/main.py @@ -275,6 +275,7 @@ class TopLevelCommand(Command): new container name. --entrypoint CMD Override the entrypoint of the image. -e KEY=VAL Set an environment variable (can be used multiple times) + -u, --user="" Run as specified username or uid --no-deps Don't start linked services. --rm Remove container after run. Ignored in detached mode. --service-ports Run command with the service's ports enabled and mapped @@ -322,6 +323,10 @@ class TopLevelCommand(Command): if options['--entrypoint']: container_options['entrypoint'] = options.get('--entrypoint') + + if options['--user']: + container_options['user'] = options.get('--user') + container = service.create_container( one_off=True, insecure_registry=insecure_registry, diff --git a/tests/fixtures/user-composefile/docker-compose.yml b/tests/fixtures/user-composefile/docker-compose.yml new file mode 100644 index 000000000..3eb7d3977 --- /dev/null +++ b/tests/fixtures/user-composefile/docker-compose.yml @@ -0,0 +1,4 @@ +service: + image: busybox:latest + user: notauser + command: id diff --git a/tests/integration/cli_test.py b/tests/integration/cli_test.py index 32c4294cc..7cf19be60 100644 --- a/tests/integration/cli_test.py +++ b/tests/integration/cli_test.py @@ -231,6 +231,28 @@ class CLITestCase(DockerClientTestCase): u'/bin/echo helloworld' ) + @patch('dockerpty.start') + def test_run_service_with_user_overridden(self, _): + self.command.base_dir = 'tests/fixtures/user-composefile' + name = 'service' + user = 'sshd' + args = ['run', '--user={}'.format(user), name] + self.command.dispatch(args, None) + service = self.project.get_service(name) + container = service.containers(stopped=True, one_off=True)[0] + self.assertEqual(user, container.get('Config.User')) + + @patch('dockerpty.start') + def test_run_service_with_user_overridden_short_form(self, _): + self.command.base_dir = 'tests/fixtures/user-composefile' + name = 'service' + user = 'sshd' + args = ['run', '-u', user, name] + self.command.dispatch(args, None) + service = self.project.get_service(name) + container = service.containers(stopped=True, one_off=True)[0] + self.assertEqual(user, container.get('Config.User')) + @patch('dockerpty.start') def test_run_service_with_environement_overridden(self, _): name = 'service' diff --git a/tests/unit/cli_test.py b/tests/unit/cli_test.py index d9a191ef0..0cb7a1d59 100644 --- a/tests/unit/cli_test.py +++ b/tests/unit/cli_test.py @@ -120,6 +120,7 @@ class CLITestCase(unittest.TestCase): 'SERVICE': 'service', 'COMMAND': None, '-e': ['BAR=NEW', 'OTHER=THREE'], + '--user': None, '--no-deps': None, '--allow-insecure-ssl': None, '-d': True, From 7b1f01bb520c37f4b18262f16ed15e06dbb98298 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Thu, 12 Mar 2015 14:02:14 +0000 Subject: [PATCH 19/46] Add script/shell Signed-off-by: Aanand Prasad --- script/shell | 4 ++++ 1 file changed, 4 insertions(+) create mode 100755 script/shell diff --git a/script/shell b/script/shell new file mode 100755 index 000000000..903be76fc --- /dev/null +++ b/script/shell @@ -0,0 +1,4 @@ +#!/bin/sh +set -ex +docker build -t docker-compose . +exec docker run -v /var/run/docker.sock:/var/run/docker.sock -v `pwd`:/code -ti --rm --entrypoint bash docker-compose From 81a32a266f998e144d97152cae953300d8956a78 Mon Sep 17 00:00:00 2001 From: Rotem Yaari Date: Fri, 13 Mar 2015 12:40:13 +0200 Subject: [PATCH 20/46] Remove restriction for requests version, update docker-py requirement --- requirements.txt | 4 ++-- setup.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/requirements.txt b/requirements.txt index 4c4113ab9..582aac1c2 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,8 +1,8 @@ PyYAML==3.10 -docker-py==1.0.0 +docker-py==1.1.0 dockerpty==0.3.2 docopt==0.6.1 -requests==2.2.1 +requests==2.5.3 six==1.7.3 texttable==0.8.2 websocket-client==0.11.0 diff --git a/setup.py b/setup.py index 0e25abd48..cce4d7cdf 100644 --- a/setup.py +++ b/setup.py @@ -27,10 +27,10 @@ def find_version(*file_paths): install_requires = [ 'docopt >= 0.6.1, < 0.7', 'PyYAML >= 3.10, < 4', - 'requests >= 2.2.1, < 2.5.0', + 'requests >= 2.5.0, < 2.6', 'texttable >= 0.8.1, < 0.9', 'websocket-client >= 0.11.0, < 1.0', - 'docker-py >= 1.0.0, < 1.1.0', + 'docker-py >= 1.1.0, < 1.2', 'dockerpty >= 0.3.2, < 0.4', 'six >= 1.3.0, < 2', ] From 4ecf5e01ff14b85e60eb2fe02aa15d41931528c2 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Fri, 27 Feb 2015 11:54:57 +0000 Subject: [PATCH 21/46] Extract YAML loading and parsing into config module Signed-off-by: Aanand Prasad --- compose/cli/command.py | 13 +-- compose/cli/main.py | 3 +- compose/config.py | 180 ++++++++++++++++++++++++++++++ compose/project.py | 19 +--- compose/service.py | 115 +------------------ tests/integration/project_test.py | 41 +++---- tests/integration/service_test.py | 15 +-- tests/integration/testcases.py | 8 +- tests/unit/config_test.py | 134 ++++++++++++++++++++++ tests/unit/project_test.py | 14 +-- tests/unit/service_test.py | 100 ----------------- 11 files changed, 358 insertions(+), 284 deletions(-) create mode 100644 compose/config.py create mode 100644 tests/unit/config_test.py diff --git a/compose/cli/command.py b/compose/cli/command.py index c26f3bc38..e829b25b2 100644 --- a/compose/cli/command.py +++ b/compose/cli/command.py @@ -4,9 +4,9 @@ from requests.exceptions import ConnectionError, SSLError import logging import os import re -import yaml import six +from .. import config from ..project import Project from ..service import ConfigError from .docopt_command import DocoptCommand @@ -69,18 +69,11 @@ class Command(DocoptCommand): return verbose_proxy.VerboseProxy('docker', client) return client - def get_config(self, config_path): - try: - with open(config_path, 'r') as fh: - return yaml.safe_load(fh) - except IOError as e: - raise errors.UserError(six.text_type(e)) - def get_project(self, config_path, project_name=None, verbose=False): try: - return Project.from_config( + return Project.from_dicts( self.get_project_name(config_path, project_name), - self.get_config(config_path), + config.load(config_path), self.get_client(verbose=verbose)) except ConfigError as e: raise errors.UserError(six.text_type(e)) diff --git a/compose/cli/main.py b/compose/cli/main.py index 434480b50..aafb199b7 100644 --- a/compose/cli/main.py +++ b/compose/cli/main.py @@ -12,7 +12,8 @@ import dockerpty from .. import __version__ from ..project import NoSuchService, ConfigurationError -from ..service import BuildError, CannotBeScaledError, parse_environment +from ..service import BuildError, CannotBeScaledError +from ..config import parse_environment from .command import Command from .docopt_command import NoSuchCommand from .errors import UserError diff --git a/compose/config.py b/compose/config.py new file mode 100644 index 000000000..4376d97cf --- /dev/null +++ b/compose/config.py @@ -0,0 +1,180 @@ +import os +import yaml +import six + + +DOCKER_CONFIG_KEYS = [ + 'cap_add', + 'cap_drop', + 'cpu_shares', + 'command', + 'detach', + 'dns', + 'dns_search', + 'domainname', + 'entrypoint', + 'env_file', + 'environment', + 'hostname', + 'image', + 'links', + 'mem_limit', + 'net', + 'ports', + 'privileged', + 'restart', + 'stdin_open', + 'tty', + 'user', + 'volumes', + 'volumes_from', + 'working_dir', +] + +ALLOWED_KEYS = DOCKER_CONFIG_KEYS + [ + 'build', + 'expose', + 'external_links', + 'name', +] + +DOCKER_CONFIG_HINTS = { + 'cpu_share' : 'cpu_shares', + 'link' : 'links', + 'port' : 'ports', + 'privilege' : 'privileged', + 'priviliged': 'privileged', + 'privilige' : 'privileged', + 'volume' : 'volumes', + 'workdir' : 'working_dir', +} + + +def load(filename): + return from_dictionary(load_yaml(filename)) + + +def load_yaml(filename): + try: + with open(filename, 'r') as fh: + return yaml.safe_load(fh) + except IOError as e: + raise ConfigurationError(six.text_type(e)) + + +def from_dictionary(dictionary): + service_dicts = [] + + for service_name, service_dict in list(dictionary.items()): + if not isinstance(service_dict, dict): + raise ConfigurationError('Service "%s" doesn\'t have any configuration options. All top level keys in your docker-compose.yml must map to a dictionary of configuration options.' % service_name) + service_dict = make_service_dict(service_name, service_dict) + service_dicts.append(service_dict) + + return service_dicts + + +def make_service_dict(name, options): + service_dict = options.copy() + service_dict['name'] = name + return process_container_options(service_dict) + + +def process_container_options(service_dict): + for k in service_dict: + if k not in ALLOWED_KEYS: + msg = "Unsupported config option for %s service: '%s'" % (service_dict['name'], k) + if k in DOCKER_CONFIG_HINTS: + msg += " (did you mean '%s'?)" % DOCKER_CONFIG_HINTS[k] + raise ConfigurationError(msg) + + for filename in get_env_files(service_dict): + if not os.path.exists(filename): + raise ConfigurationError("Couldn't find env file for service %s: %s" % (service_dict['name'], filename)) + + if 'environment' in service_dict or 'env_file' in service_dict: + service_dict['environment'] = build_environment(service_dict) + + return service_dict + + +def parse_links(links): + return dict(parse_link(l) for l in links) + + +def parse_link(link): + if ':' in link: + source, alias = link.split(':', 1) + return (alias, source) + else: + return (link, link) + + +def get_env_files(options): + env_files = options.get('env_file', []) + if not isinstance(env_files, list): + env_files = [env_files] + return env_files + + +def build_environment(options): + env = {} + + for f in get_env_files(options): + env.update(env_vars_from_file(f)) + + env.update(parse_environment(options.get('environment'))) + return dict(resolve_env(k, v) for k, v in six.iteritems(env)) + + +def parse_environment(environment): + if not environment: + return {} + + if isinstance(environment, list): + return dict(split_env(e) for e in environment) + + if isinstance(environment, dict): + return environment + + raise ConfigurationError( + "environment \"%s\" must be a list or mapping," % + environment + ) + + +def split_env(env): + if '=' in env: + return env.split('=', 1) + else: + return env, None + + +def resolve_env(key, val): + if val is not None: + return key, val + elif key in os.environ: + return key, os.environ[key] + else: + return key, '' + + +def env_vars_from_file(filename): + """ + Read in a line delimited file of environment variables. + """ + env = {} + for line in open(filename, 'r'): + line = line.strip() + if line and not line.startswith('#'): + k, v = split_env(line) + env[k] = v + return env + + +class ConfigurationError(Exception): + def __init__(self, msg): + self.msg = msg + + def __str__(self): + return self.msg diff --git a/compose/project.py b/compose/project.py index 794ef2b65..881d8eb0a 100644 --- a/compose/project.py +++ b/compose/project.py @@ -3,6 +3,7 @@ from __future__ import absolute_import import logging from functools import reduce +from .config import ConfigurationError from .service import Service from .container import Container from docker.errors import APIError @@ -85,16 +86,6 @@ class Project(object): volumes_from=volumes_from, **service_dict)) return project - @classmethod - def from_config(cls, name, config, client): - dicts = [] - for service_name, service in list(config.items()): - if not isinstance(service, dict): - raise ConfigurationError('Service "%s" doesn\'t have any configuration options. All top level keys in your docker-compose.yml must map to a dictionary of configuration options.' % service_name) - service['name'] = service_name - dicts.append(service) - return cls.from_dicts(name, dicts, client) - def get_service(self, name): """ Retrieve a service by name. Raises NoSuchService @@ -277,13 +268,5 @@ class NoSuchService(Exception): return self.msg -class ConfigurationError(Exception): - def __init__(self, msg): - self.msg = msg - - def __str__(self): - return self.msg - - class DependencyError(ConfigurationError): pass diff --git a/compose/service.py b/compose/service.py index 377198cf4..c65874c26 100644 --- a/compose/service.py +++ b/compose/service.py @@ -8,51 +8,14 @@ from operator import attrgetter import sys from docker.errors import APIError -import six +from .config import DOCKER_CONFIG_KEYS from .container import Container, get_container_name from .progress_stream import stream_output, StreamOutputError log = logging.getLogger(__name__) -DOCKER_CONFIG_KEYS = [ - 'cap_add', - 'cap_drop', - 'cpu_shares', - 'command', - 'detach', - 'dns', - 'dns_search', - 'domainname', - 'entrypoint', - 'env_file', - 'environment', - 'hostname', - 'image', - 'mem_limit', - 'net', - 'ports', - 'privileged', - 'restart', - 'stdin_open', - 'tty', - 'user', - 'volumes', - 'volumes_from', - 'working_dir', -] -DOCKER_CONFIG_HINTS = { - 'cpu_share' : 'cpu_shares', - 'link' : 'links', - 'port' : 'ports', - 'privilege' : 'privileged', - 'priviliged': 'privileged', - 'privilige' : 'privileged', - 'volume' : 'volumes', - 'workdir' : 'working_dir', -} - DOCKER_START_KEYS = [ 'cap_add', 'cap_drop', @@ -96,20 +59,6 @@ class Service(object): if 'image' in options and 'build' in options: raise ConfigError('Service %s has both an image and build path specified. A service can either be built to image or use an existing image, not both.' % name) - for filename in get_env_files(options): - if not os.path.exists(filename): - raise ConfigError("Couldn't find env file for service %s: %s" % (name, filename)) - - supported_options = DOCKER_CONFIG_KEYS + ['build', 'expose', - 'external_links'] - - for k in options: - if k not in supported_options: - msg = "Unsupported config option for %s service: '%s'" % (name, k) - if k in DOCKER_CONFIG_HINTS: - msg += " (did you mean '%s'?)" % DOCKER_CONFIG_HINTS[k] - raise ConfigError(msg) - self.name = name self.client = client self.project = project @@ -478,8 +427,6 @@ class Service(object): (parse_volume_spec(v).internal, {}) for v in container_options['volumes']) - container_options['environment'] = build_environment(container_options) - if self.can_be_built(): container_options['image'] = self.full_name else: @@ -648,63 +595,3 @@ def split_port(port): external_ip, external_port, internal_port = parts return internal_port, (external_ip, external_port or None) - - -def get_env_files(options): - env_files = options.get('env_file', []) - if not isinstance(env_files, list): - env_files = [env_files] - return env_files - - -def build_environment(options): - env = {} - - for f in get_env_files(options): - env.update(env_vars_from_file(f)) - - env.update(parse_environment(options.get('environment'))) - return dict(resolve_env(k, v) for k, v in six.iteritems(env)) - - -def parse_environment(environment): - if not environment: - return {} - - if isinstance(environment, list): - return dict(split_env(e) for e in environment) - - if isinstance(environment, dict): - return environment - - raise ConfigError("environment \"%s\" must be a list or mapping," % - environment) - - -def split_env(env): - if '=' in env: - return env.split('=', 1) - else: - return env, None - - -def resolve_env(key, val): - if val is not None: - return key, val - elif key in os.environ: - return key, os.environ[key] - else: - return key, '' - - -def env_vars_from_file(filename): - """ - Read in a line delimited file of environment variables. - """ - env = {} - for line in open(filename, 'r'): - line = line.strip() - if line and not line.startswith('#'): - k, v = split_env(line) - env[k] = v - return env diff --git a/tests/integration/project_test.py b/tests/integration/project_test.py index 17b54daee..a46fc2f5a 100644 --- a/tests/integration/project_test.py +++ b/tests/integration/project_test.py @@ -1,14 +1,15 @@ from __future__ import unicode_literals -from compose.project import Project, ConfigurationError +from compose import config +from compose.project import Project from compose.container import Container from .testcases import DockerClientTestCase class ProjectTest(DockerClientTestCase): def test_volumes_from_service(self): - project = Project.from_config( + project = Project.from_dicts( name='composetest', - config={ + service_dicts=config.from_dictionary({ 'data': { 'image': 'busybox:latest', 'volumes': ['/var/data'], @@ -17,7 +18,7 @@ class ProjectTest(DockerClientTestCase): 'image': 'busybox:latest', 'volumes_from': ['data'], }, - }, + }), client=self.client, ) db = project.get_service('db') @@ -31,14 +32,14 @@ class ProjectTest(DockerClientTestCase): volumes=['/var/data'], name='composetest_data_container', ) - project = Project.from_config( + project = Project.from_dicts( name='composetest', - config={ + service_dicts=config.from_dictionary({ 'db': { 'image': 'busybox:latest', 'volumes_from': ['composetest_data_container'], }, - }, + }), client=self.client, ) db = project.get_service('db') @@ -48,9 +49,9 @@ class ProjectTest(DockerClientTestCase): project.remove_stopped() def test_net_from_service(self): - project = Project.from_config( + project = Project.from_dicts( name='composetest', - config={ + service_dicts=config.from_dictionary({ 'net': { 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"] @@ -59,8 +60,8 @@ class ProjectTest(DockerClientTestCase): 'image': 'busybox:latest', 'net': 'container:net', 'command': ["/bin/sleep", "300"] - }, - }, + }, + }), client=self.client, ) @@ -82,14 +83,14 @@ class ProjectTest(DockerClientTestCase): ) net_container.start() - project = Project.from_config( + project = Project.from_dicts( name='composetest', - config={ + service_dicts=config.from_dictionary({ 'web': { 'image': 'busybox:latest', 'net': 'container:composetest_net_container' }, - }, + }), client=self.client, ) @@ -257,9 +258,9 @@ class ProjectTest(DockerClientTestCase): project.remove_stopped() def test_project_up_starts_depends(self): - project = Project.from_config( + project = Project.from_dicts( name='composetest', - config={ + service_dicts=config.from_dictionary({ 'console': { 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"], @@ -279,7 +280,7 @@ class ProjectTest(DockerClientTestCase): 'net': 'container:net', 'links': ['app'] }, - }, + }), client=self.client, ) project.start() @@ -296,9 +297,9 @@ class ProjectTest(DockerClientTestCase): project.remove_stopped() def test_project_up_with_no_deps(self): - project = Project.from_config( + project = Project.from_dicts( name='composetest', - config={ + service_dicts=config.from_dictionary({ 'console': { 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"], @@ -324,7 +325,7 @@ class ProjectTest(DockerClientTestCase): 'links': ['app'], 'volumes_from': ['vol'] }, - }, + }), client=self.client, ) project.start() diff --git a/tests/integration/service_test.py b/tests/integration/service_test.py index 7b95b870f..8008fbbca 100644 --- a/tests/integration/service_test.py +++ b/tests/integration/service_test.py @@ -2,6 +2,7 @@ from __future__ import unicode_literals from __future__ import absolute_import import os from os import path +import mock from compose import Service from compose.service import CannotBeScaledError @@ -481,16 +482,12 @@ class ServiceTest(DockerClientTestCase): for k,v in {'ONE': '1', 'TWO': '2', 'THREE': '3', 'FOO': 'baz', 'DOO': 'dah'}.items(): self.assertEqual(env[k], v) + @mock.patch.dict(os.environ) def test_resolve_env(self): - service = self.create_service('web', environment={'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': None, 'NO_DEF': None}) os.environ['FILE_DEF'] = 'E1' os.environ['FILE_DEF_EMPTY'] = 'E2' os.environ['ENV_DEF'] = 'E3' - try: - env = create_and_start_container(service).environment - for k,v in {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}.items(): - self.assertEqual(env[k], v) - finally: - del os.environ['FILE_DEF'] - del os.environ['FILE_DEF_EMPTY'] - del os.environ['ENV_DEF'] + service = self.create_service('web', environment={'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': None, 'NO_DEF': None}) + env = create_and_start_container(service).environment + for k,v in {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}.items(): + self.assertEqual(env[k], v) diff --git a/tests/integration/testcases.py b/tests/integration/testcases.py index 53882561b..4f49124cf 100644 --- a/tests/integration/testcases.py +++ b/tests/integration/testcases.py @@ -1,6 +1,7 @@ from __future__ import unicode_literals from __future__ import absolute_import from compose.service import Service +from compose.config import make_service_dict from compose.cli.docker_client import docker_client from compose.progress_stream import stream_output from .. import unittest @@ -21,14 +22,15 @@ class DockerClientTestCase(unittest.TestCase): self.client.remove_image(i) def create_service(self, name, **kwargs): + kwargs['image'] = "busybox:latest" + if 'command' not in kwargs: kwargs['command'] = ["/bin/sleep", "300"] + return Service( project='composetest', - name=name, client=self.client, - image="busybox:latest", - **kwargs + **make_service_dict(name, kwargs) ) def check_build(self, *args, **kwargs): diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py new file mode 100644 index 000000000..8f59694de --- /dev/null +++ b/tests/unit/config_test.py @@ -0,0 +1,134 @@ +import os +import mock +from .. import unittest + +from compose import config + +class ConfigTest(unittest.TestCase): + def test_from_dictionary(self): + service_dicts = config.from_dictionary({ + 'foo': {'image': 'busybox'}, + 'bar': {'environment': ['FOO=1']}, + }) + + self.assertEqual( + sorted(service_dicts, key=lambda d: d['name']), + sorted([ + { + 'name': 'bar', + 'environment': {'FOO': '1'}, + }, + { + 'name': 'foo', + 'image': 'busybox', + } + ]) + ) + + def test_from_dictionary_throws_error_when_not_dict(self): + with self.assertRaises(config.ConfigurationError): + config.from_dictionary({ + 'web': 'busybox:latest', + }) + + def test_config_validation(self): + self.assertRaises( + config.ConfigurationError, + lambda: config.make_service_dict('foo', {'port': ['8000']}) + ) + config.make_service_dict('foo', {'ports': ['8000']}) + + def test_parse_environment_as_list(self): + environment =[ + 'NORMAL=F1', + 'CONTAINS_EQUALS=F=2', + 'TRAILING_EQUALS=', + ] + self.assertEqual( + config.parse_environment(environment), + {'NORMAL': 'F1', 'CONTAINS_EQUALS': 'F=2', 'TRAILING_EQUALS': ''}, + ) + + def test_parse_environment_as_dict(self): + environment = { + 'NORMAL': 'F1', + 'CONTAINS_EQUALS': 'F=2', + 'TRAILING_EQUALS': None, + } + self.assertEqual(config.parse_environment(environment), environment) + + def test_parse_environment_invalid(self): + with self.assertRaises(config.ConfigurationError): + config.parse_environment('a=b') + + def test_parse_environment_empty(self): + self.assertEqual(config.parse_environment(None), {}) + + @mock.patch.dict(os.environ) + def test_resolve_environment(self): + os.environ['FILE_DEF'] = 'E1' + os.environ['FILE_DEF_EMPTY'] = 'E2' + os.environ['ENV_DEF'] = 'E3' + + service_dict = config.make_service_dict( + 'foo', + { + 'environment': { + 'FILE_DEF': 'F1', + 'FILE_DEF_EMPTY': '', + 'ENV_DEF': None, + 'NO_DEF': None + }, + }, + ) + + self.assertEqual( + service_dict['environment'], + {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}, + ) + + def test_env_from_file(self): + service_dict = config.make_service_dict( + 'foo', + {'env_file': 'tests/fixtures/env/one.env'}, + ) + self.assertEqual( + service_dict['environment'], + {'ONE': '2', 'TWO': '1', 'THREE': '3', 'FOO': 'bar'}, + ) + + def test_env_from_multiple_files(self): + service_dict = config.make_service_dict( + 'foo', + { + 'env_file': [ + 'tests/fixtures/env/one.env', + 'tests/fixtures/env/two.env', + ], + }, + ) + self.assertEqual( + service_dict['environment'], + {'ONE': '2', 'TWO': '1', 'THREE': '3', 'FOO': 'baz', 'DOO': 'dah'}, + ) + + def test_env_nonexistent_file(self): + options = {'env_file': 'tests/fixtures/env/nonexistent.env'} + self.assertRaises( + config.ConfigurationError, + lambda: config.make_service_dict('foo', options), + ) + + @mock.patch.dict(os.environ) + def test_resolve_environment_from_file(self): + os.environ['FILE_DEF'] = 'E1' + os.environ['FILE_DEF_EMPTY'] = 'E2' + os.environ['ENV_DEF'] = 'E3' + service_dict = config.make_service_dict( + 'foo', + {'env_file': 'tests/fixtures/env/resolve.env'}, + ) + self.assertEqual( + service_dict['environment'], + {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}, + ) diff --git a/tests/unit/project_test.py b/tests/unit/project_test.py index b06e14e58..c995d432f 100644 --- a/tests/unit/project_test.py +++ b/tests/unit/project_test.py @@ -1,8 +1,9 @@ from __future__ import unicode_literals from .. import unittest from compose.service import Service -from compose.project import Project, ConfigurationError +from compose.project import Project from compose.container import Container +from compose import config import mock import docker @@ -49,26 +50,21 @@ class ProjectTest(unittest.TestCase): self.assertEqual(project.services[2].name, 'web') def test_from_config(self): - project = Project.from_config('composetest', { + dicts = config.from_dictionary({ 'web': { 'image': 'busybox:latest', }, 'db': { 'image': 'busybox:latest', }, - }, None) + }) + project = Project.from_dicts('composetest', dicts, None) self.assertEqual(len(project.services), 2) self.assertEqual(project.get_service('web').name, 'web') self.assertEqual(project.get_service('web').options['image'], 'busybox:latest') self.assertEqual(project.get_service('db').name, 'db') self.assertEqual(project.get_service('db').options['image'], 'busybox:latest') - def test_from_config_throws_error_when_not_dict(self): - with self.assertRaises(ConfigurationError): - project = Project.from_config('composetest', { - 'web': 'busybox:latest', - }, None) - def test_get_service(self): web = Service( project='composetest', diff --git a/tests/unit/service_test.py b/tests/unit/service_test.py index 012a51ab6..c70c30bfa 100644 --- a/tests/unit/service_test.py +++ b/tests/unit/service_test.py @@ -16,7 +16,6 @@ from compose.service import ( build_port_bindings, build_volume_binding, get_container_name, - parse_environment, parse_repository_tag, parse_volume_spec, split_port, @@ -47,10 +46,6 @@ class ServiceTest(unittest.TestCase): self.assertRaises(ConfigError, lambda: Service(name='foo', project='_')) Service(name='foo', project='bar') - def test_config_validation(self): - self.assertRaises(ConfigError, lambda: Service(name='foo', port=['8000'])) - Service(name='foo', ports=['8000']) - def test_get_container_name(self): self.assertIsNone(get_container_name({})) self.assertEqual(get_container_name({'Name': 'myproject_db_1'}), 'myproject_db_1') @@ -321,98 +316,3 @@ class ServiceVolumesTest(unittest.TestCase): binding, ('/home/user', dict(bind='/home/user', ro=False))) -class ServiceEnvironmentTest(unittest.TestCase): - - def setUp(self): - self.mock_client = mock.create_autospec(docker.Client) - self.mock_client.containers.return_value = [] - - def test_parse_environment_as_list(self): - environment =[ - 'NORMAL=F1', - 'CONTAINS_EQUALS=F=2', - 'TRAILING_EQUALS=' - ] - self.assertEqual( - parse_environment(environment), - {'NORMAL': 'F1', 'CONTAINS_EQUALS': 'F=2', 'TRAILING_EQUALS': ''}) - - def test_parse_environment_as_dict(self): - environment = { - 'NORMAL': 'F1', - 'CONTAINS_EQUALS': 'F=2', - 'TRAILING_EQUALS': None, - } - self.assertEqual(parse_environment(environment), environment) - - def test_parse_environment_invalid(self): - with self.assertRaises(ConfigError): - parse_environment('a=b') - - def test_parse_environment_empty(self): - self.assertEqual(parse_environment(None), {}) - - @mock.patch.dict(os.environ) - def test_resolve_environment(self): - os.environ['FILE_DEF'] = 'E1' - os.environ['FILE_DEF_EMPTY'] = 'E2' - os.environ['ENV_DEF'] = 'E3' - service = Service( - 'foo', - environment={ - 'FILE_DEF': 'F1', - 'FILE_DEF_EMPTY': '', - 'ENV_DEF': None, - 'NO_DEF': None - }, - client=self.mock_client, - image='image_name', - ) - options = service._get_container_create_options({}) - self.assertEqual( - options['environment'], - {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''} - ) - - def test_env_from_file(self): - service = Service('foo', - env_file='tests/fixtures/env/one.env', - client=self.mock_client, - image='image_name', - ) - options = service._get_container_create_options({}) - self.assertEqual( - options['environment'], - {'ONE': '2', 'TWO': '1', 'THREE': '3', 'FOO': 'bar'} - ) - - def test_env_from_multiple_files(self): - service = Service('foo', - env_file=['tests/fixtures/env/one.env', 'tests/fixtures/env/two.env'], - client=self.mock_client, - image='image_name', - ) - options = service._get_container_create_options({}) - self.assertEqual( - options['environment'], - {'ONE': '2', 'TWO': '1', 'THREE': '3', 'FOO': 'baz', 'DOO': 'dah'} - ) - - def test_env_nonexistent_file(self): - self.assertRaises(ConfigError, lambda: Service('foo', env_file='tests/fixtures/env/nonexistent.env')) - - @mock.patch.dict(os.environ) - def test_resolve_environment_from_file(self): - os.environ['FILE_DEF'] = 'E1' - os.environ['FILE_DEF_EMPTY'] = 'E2' - os.environ['ENV_DEF'] = 'E3' - service = Service('foo', - env_file=['tests/fixtures/env/resolve.env'], - client=self.mock_client, - image='image_name', - ) - options = service._get_container_create_options({}) - self.assertEqual( - options['environment'], - {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''} - ) From 528bed9ef6921eeea219370d74ec0e6686e10636 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Thu, 12 Mar 2015 13:59:23 +0000 Subject: [PATCH 22/46] Fix environment resolution Signed-off-by: Aanand Prasad --- compose/config.py | 60 ++++++++++++++-------- docs/yml.md | 9 +++- tests/fixtures/env-file/docker-compose.yml | 4 ++ tests/fixtures/env-file/test.env | 1 + tests/integration/cli_test.py | 16 ++++++ tests/integration/testcases.py | 2 +- tests/unit/config_test.py | 18 +++---- 7 files changed, 77 insertions(+), 33 deletions(-) create mode 100644 tests/fixtures/env-file/docker-compose.yml create mode 100644 tests/fixtures/env-file/test.env diff --git a/compose/config.py b/compose/config.py index 4376d97cf..a4e3a991f 100644 --- a/compose/config.py +++ b/compose/config.py @@ -51,7 +51,8 @@ DOCKER_CONFIG_HINTS = { def load(filename): - return from_dictionary(load_yaml(filename)) + working_dir = os.path.dirname(filename) + return from_dictionary(load_yaml(filename), working_dir=working_dir) def load_yaml(filename): @@ -62,25 +63,26 @@ def load_yaml(filename): raise ConfigurationError(six.text_type(e)) -def from_dictionary(dictionary): +def from_dictionary(dictionary, working_dir=None): service_dicts = [] for service_name, service_dict in list(dictionary.items()): if not isinstance(service_dict, dict): raise ConfigurationError('Service "%s" doesn\'t have any configuration options. All top level keys in your docker-compose.yml must map to a dictionary of configuration options.' % service_name) - service_dict = make_service_dict(service_name, service_dict) + service_dict = make_service_dict(service_name, service_dict, working_dir=working_dir) service_dicts.append(service_dict) return service_dicts -def make_service_dict(name, options): +def make_service_dict(name, options, working_dir=None): service_dict = options.copy() service_dict['name'] = name - return process_container_options(service_dict) + service_dict = resolve_environment(service_dict, working_dir=working_dir) + return process_container_options(service_dict, working_dir=working_dir) -def process_container_options(service_dict): +def process_container_options(service_dict, working_dir=None): for k in service_dict: if k not in ALLOWED_KEYS: msg = "Unsupported config option for %s service: '%s'" % (service_dict['name'], k) @@ -88,13 +90,6 @@ def process_container_options(service_dict): msg += " (did you mean '%s'?)" % DOCKER_CONFIG_HINTS[k] raise ConfigurationError(msg) - for filename in get_env_files(service_dict): - if not os.path.exists(filename): - raise ConfigurationError("Couldn't find env file for service %s: %s" % (service_dict['name'], filename)) - - if 'environment' in service_dict or 'env_file' in service_dict: - service_dict['environment'] = build_environment(service_dict) - return service_dict @@ -110,21 +105,38 @@ def parse_link(link): return (link, link) -def get_env_files(options): +def get_env_files(options, working_dir=None): + if 'env_file' not in options: + return {} + + if working_dir is None: + raise Exception("No working_dir passed to get_env_files()") + env_files = options.get('env_file', []) if not isinstance(env_files, list): env_files = [env_files] - return env_files + + return [expand_path(working_dir, path) for path in env_files] -def build_environment(options): +def resolve_environment(service_dict, working_dir=None): + service_dict = service_dict.copy() + + if 'environment' not in service_dict and 'env_file' not in service_dict: + return service_dict + env = {} - for f in get_env_files(options): - env.update(env_vars_from_file(f)) + if 'env_file' in service_dict: + for f in get_env_files(service_dict, working_dir=working_dir): + env.update(env_vars_from_file(f)) + del service_dict['env_file'] - env.update(parse_environment(options.get('environment'))) - return dict(resolve_env(k, v) for k, v in six.iteritems(env)) + env.update(parse_environment(service_dict.get('environment'))) + env = dict(resolve_env_var(k, v) for k, v in six.iteritems(env)) + + service_dict['environment'] = env + return service_dict def parse_environment(environment): @@ -150,7 +162,7 @@ def split_env(env): return env, None -def resolve_env(key, val): +def resolve_env_var(key, val): if val is not None: return key, val elif key in os.environ: @@ -163,6 +175,8 @@ def env_vars_from_file(filename): """ Read in a line delimited file of environment variables. """ + if not os.path.exists(filename): + raise ConfigurationError("Couldn't find env file: %s" % filename) env = {} for line in open(filename, 'r'): line = line.strip() @@ -172,6 +186,10 @@ def env_vars_from_file(filename): return env +def expand_path(working_dir, path): + return os.path.abspath(os.path.join(working_dir, path)) + + class ConfigurationError(Exception): def __init__(self, msg): self.msg = msg diff --git a/docs/yml.md b/docs/yml.md index 035a99e92..52be706a0 100644 --- a/docs/yml.md +++ b/docs/yml.md @@ -158,11 +158,18 @@ environment: Add environment variables from a file. Can be a single value or a list. +If you have specified a Compose file with `docker-compose -f FILE`, paths in +`env_file` are relative to the directory that file is in. + Environment variables specified in `environment` override these values. ``` +env_file: .env + env_file: - - .env + - ./common.env + - ./apps/web.env + - /opt/secrets.env ``` ``` diff --git a/tests/fixtures/env-file/docker-compose.yml b/tests/fixtures/env-file/docker-compose.yml new file mode 100644 index 000000000..d9366ace2 --- /dev/null +++ b/tests/fixtures/env-file/docker-compose.yml @@ -0,0 +1,4 @@ +web: + image: busybox + command: /bin/true + env_file: ./test.env diff --git a/tests/fixtures/env-file/test.env b/tests/fixtures/env-file/test.env new file mode 100644 index 000000000..c9604dad5 --- /dev/null +++ b/tests/fixtures/env-file/test.env @@ -0,0 +1 @@ +FOO=1 \ No newline at end of file diff --git a/tests/integration/cli_test.py b/tests/integration/cli_test.py index 7cf19be60..a79b45cfb 100644 --- a/tests/integration/cli_test.py +++ b/tests/integration/cli_test.py @@ -1,5 +1,6 @@ from __future__ import absolute_import import sys +import os from six import StringIO from mock import patch @@ -23,6 +24,12 @@ class CLITestCase(DockerClientTestCase): @property def project(self): + # Hack: allow project to be overridden. This needs refactoring so that + # the project object is built exactly once, by the command object, and + # accessed by the test case object. + if hasattr(self, '_project'): + return self._project + return self.command.get_project(self.command.get_config_path()) def test_help(self): @@ -409,3 +416,12 @@ class CLITestCase(DockerClientTestCase): self.assertEqual(get_port(3000), container.get_local_port(3000)) self.assertEqual(get_port(3001), "0.0.0.0:9999") self.assertEqual(get_port(3002), "") + + def test_env_file_relative_to_compose_file(self): + config_path = os.path.abspath('tests/fixtures/env-file/docker-compose.yml') + self.command.dispatch(['-f', config_path, 'up', '-d'], None) + self._project = self.command.get_project(config_path) + + containers = self.project.containers(stopped=True) + self.assertEqual(len(containers), 1) + self.assertIn("FOO=1", containers[0].get('Config.Env')) diff --git a/tests/integration/testcases.py b/tests/integration/testcases.py index 4f49124cf..d5ca1debc 100644 --- a/tests/integration/testcases.py +++ b/tests/integration/testcases.py @@ -30,7 +30,7 @@ class DockerClientTestCase(unittest.TestCase): return Service( project='composetest', client=self.client, - **make_service_dict(name, kwargs) + **make_service_dict(name, kwargs, working_dir='.') ) def check_build(self, *args, **kwargs): diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py index 8f59694de..4ff08a9ef 100644 --- a/tests/unit/config_test.py +++ b/tests/unit/config_test.py @@ -90,7 +90,8 @@ class ConfigTest(unittest.TestCase): def test_env_from_file(self): service_dict = config.make_service_dict( 'foo', - {'env_file': 'tests/fixtures/env/one.env'}, + {'env_file': 'one.env'}, + 'tests/fixtures/env', ) self.assertEqual( service_dict['environment'], @@ -100,12 +101,8 @@ class ConfigTest(unittest.TestCase): def test_env_from_multiple_files(self): service_dict = config.make_service_dict( 'foo', - { - 'env_file': [ - 'tests/fixtures/env/one.env', - 'tests/fixtures/env/two.env', - ], - }, + {'env_file': ['one.env', 'two.env']}, + 'tests/fixtures/env', ) self.assertEqual( service_dict['environment'], @@ -113,10 +110,10 @@ class ConfigTest(unittest.TestCase): ) def test_env_nonexistent_file(self): - options = {'env_file': 'tests/fixtures/env/nonexistent.env'} + options = {'env_file': 'nonexistent.env'} self.assertRaises( config.ConfigurationError, - lambda: config.make_service_dict('foo', options), + lambda: config.make_service_dict('foo', options, 'tests/fixtures/env'), ) @mock.patch.dict(os.environ) @@ -126,7 +123,8 @@ class ConfigTest(unittest.TestCase): os.environ['ENV_DEF'] = 'E3' service_dict = config.make_service_dict( 'foo', - {'env_file': 'tests/fixtures/env/resolve.env'}, + {'env_file': 'resolve.env'}, + 'tests/fixtures/env', ) self.assertEqual( service_dict['environment'], From 3c8ef6a94c5d94446eabcbe27df33168ec52234d Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Fri, 13 Mar 2015 14:51:26 +0000 Subject: [PATCH 23/46] Fix Project.up() tests Signed-off-by: Aanand Prasad --- tests/integration/project_test.py | 37 +++++++++++++------------------ 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/tests/integration/project_test.py b/tests/integration/project_test.py index 17b54daee..73e8badc3 100644 --- a/tests/integration/project_test.py +++ b/tests/integration/project_test.py @@ -264,20 +264,19 @@ class ProjectTest(DockerClientTestCase): 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"], }, - 'net' : { + 'data' : { 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"] }, - 'app': { + 'db': { 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"], - 'net': 'container:net' + 'volumes_from': ['data'], }, 'web': { 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"], - 'net': 'container:net', - 'links': ['app'] + 'links': ['db'], }, }, client=self.client, @@ -288,8 +287,8 @@ class ProjectTest(DockerClientTestCase): project.up(['web']) self.assertEqual(len(project.containers()), 3) self.assertEqual(len(project.get_service('web').containers()), 1) - self.assertEqual(len(project.get_service('app').containers()), 1) - self.assertEqual(len(project.get_service('net').containers()), 1) + self.assertEqual(len(project.get_service('db').containers()), 1) + self.assertEqual(len(project.get_service('data').containers()), 1) self.assertEqual(len(project.get_service('console').containers()), 0) project.kill() @@ -303,26 +302,19 @@ class ProjectTest(DockerClientTestCase): 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"], }, - 'net' : { + 'data' : { 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"] }, - 'vol': { + 'db': { 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"], - 'volumes': ["/tmp"] - }, - 'app': { - 'image': 'busybox:latest', - 'command': ["/bin/sleep", "300"], - 'net': 'container:net' + 'volumes_from': ['data'], }, 'web': { 'image': 'busybox:latest', 'command': ["/bin/sleep", "300"], - 'net': 'container:net', - 'links': ['app'], - 'volumes_from': ['vol'] + 'links': ['db'], }, }, client=self.client, @@ -330,11 +322,12 @@ class ProjectTest(DockerClientTestCase): project.start() self.assertEqual(len(project.containers()), 0) - project.up(['web'], start_deps=False) + project.up(['db'], start_deps=False) self.assertEqual(len(project.containers(stopped=True)), 2) - self.assertEqual(len(project.get_service('web').containers()), 1) - self.assertEqual(len(project.get_service('vol').containers(stopped=True)), 1) - self.assertEqual(len(project.get_service('net').containers()), 0) + self.assertEqual(len(project.get_service('web').containers()), 0) + self.assertEqual(len(project.get_service('db').containers()), 1) + self.assertEqual(len(project.get_service('data').containers()), 0) + self.assertEqual(len(project.get_service('data').containers(stopped=True)), 1) self.assertEqual(len(project.get_service('console').containers()), 0) project.kill() From 198598c9364c02b47db54881e92515dbe39d1664 Mon Sep 17 00:00:00 2001 From: Ben Firshman Date: Fri, 13 Mar 2015 19:36:54 +0000 Subject: [PATCH 24/46] Remove wercker status from readme Because it doesn't really work anymore and looks scary when it's broken. Signed-off-by: Ben Firshman --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index c943c70d8..1c30420da 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,6 @@ Docker Compose ============== -[![wercker status](https://app.wercker.com/status/d5dbac3907301c3d5ce735e2d5e95a5b/s/master "wercker status")](https://app.wercker.com/project/bykey/d5dbac3907301c3d5ce735e2d5e95a5b) - *(Previously known as Fig)* Compose is a tool for defining and running complex applications with Docker. From 4c5a80f25344abac06de5fc0e88452990004dace Mon Sep 17 00:00:00 2001 From: funkyfuture Date: Tue, 17 Mar 2015 00:21:29 +0100 Subject: [PATCH 25/46] Change port in ports-composefile to 49152 This shall lower the propability to interfere with another service (e.g. the WebUI of an application) that is running on the machine where tests are run. Signed-off-by: funkyfuture --- tests/fixtures/ports-composefile/docker-compose.yml | 2 +- tests/integration/cli_test.py | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/fixtures/ports-composefile/docker-compose.yml b/tests/fixtures/ports-composefile/docker-compose.yml index 5ff08d339..2474087d0 100644 --- a/tests/fixtures/ports-composefile/docker-compose.yml +++ b/tests/fixtures/ports-composefile/docker-compose.yml @@ -4,4 +4,4 @@ simple: command: /bin/sleep 300 ports: - '3000' - - '9999:3001' + - '49152:3001' diff --git a/tests/integration/cli_test.py b/tests/integration/cli_test.py index a79b45cfb..2f961d2b2 100644 --- a/tests/integration/cli_test.py +++ b/tests/integration/cli_test.py @@ -300,6 +300,7 @@ class CLITestCase(DockerClientTestCase): @patch('dockerpty.start') def test_run_service_with_map_ports(self, __): + # create one off container self.command.base_dir = 'tests/fixtures/ports-composefile' self.command.dispatch(['run', '-d', '--service-ports', 'simple'], None) @@ -315,7 +316,7 @@ class CLITestCase(DockerClientTestCase): # check the ports self.assertNotEqual(port_random, None) self.assertIn("0.0.0.0", port_random) - self.assertEqual(port_assigned, "0.0.0.0:9999") + self.assertEqual(port_assigned, "0.0.0.0:49152") def test_rm(self): service = self.project.get_service('simple') @@ -404,6 +405,7 @@ class CLITestCase(DockerClientTestCase): self.assertEqual(len(project.get_service('another').containers()), 0) def test_port(self): + self.command.base_dir = 'tests/fixtures/ports-composefile' self.command.dispatch(['up', '-d'], None) container = self.project.get_service('simple').get_container() @@ -414,7 +416,7 @@ class CLITestCase(DockerClientTestCase): return mock_stdout.getvalue().rstrip() self.assertEqual(get_port(3000), container.get_local_port(3000)) - self.assertEqual(get_port(3001), "0.0.0.0:9999") + self.assertEqual(get_port(3001), "0.0.0.0:49152") self.assertEqual(get_port(3002), "") def test_env_file_relative_to_compose_file(self): From 85fb8956f3847d0e167e5bbe05ab698c04250fb7 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Tue, 17 Mar 2015 17:01:36 -0700 Subject: [PATCH 26/46] Validate DCO in script/test-versions Signed-off-by: Aanand Prasad --- script/test | 1 + script/test-versions | 3 +++ script/validate-dco | 2 ++ 3 files changed, 6 insertions(+) diff --git a/script/test b/script/test index f278023a0..ab0645fdc 100755 --- a/script/test +++ b/script/test @@ -9,6 +9,7 @@ docker build -t "$TAG" . docker run \ --rm \ --volume="/var/run/docker.sock:/var/run/docker.sock" \ + --volume="$(pwd):/code" \ -e DOCKER_VERSIONS \ -e "TAG=$TAG" \ --entrypoint="script/test-versions" \ diff --git a/script/test-versions b/script/test-versions index 9f30eecaa..a9e3bc4c7 100755 --- a/script/test-versions +++ b/script/test-versions @@ -4,6 +4,9 @@ set -e +>&2 echo "Validating DCO" +script/validate-dco + >&2 echo "Running lint checks" flake8 compose diff --git a/script/validate-dco b/script/validate-dco index 1c75d91bf..701ac5e46 100755 --- a/script/validate-dco +++ b/script/validate-dco @@ -1,5 +1,7 @@ #!/bin/bash +set -e + source "$(dirname "$BASH_SOURCE")/.validate" adds=$(validate_diff --numstat | awk '{ s += $1 } END { print s }') From 02266757663f659cb52105d4b78103c70d30a026 Mon Sep 17 00:00:00 2001 From: Jessica Frazelle Date: Thu, 19 Mar 2015 17:50:15 -0700 Subject: [PATCH 27/46] Add build status \o/ Signed-off-by: Jessica Frazelle --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1c30420da..e76431b06 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ Docker Compose ============== - +[![Build Status](http://jenkins.dockerproject.com/buildStatus/icon?job=Compose Master)](http://jenkins.dockerproject.com/job/Compose%20Master/) *(Previously known as Fig)* Compose is a tool for defining and running complex applications with Docker. From eef4bc39175913e82eb8d814f3f4b03df1f04f16 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Mon, 16 Feb 2015 14:30:31 +0000 Subject: [PATCH 28/46] Specify all HostConfig at create time This is required for Swarm integration: the cluster needs to know about config like `links` and `volumes_from` at create time so that it can co-schedule containers. Signed-off-by: Aanand Prasad --- compose/cli/docker_client.py | 2 +- compose/cli/main.py | 10 ++-- compose/service.py | 94 +++++++++++++++++++------------ tests/integration/service_test.py | 2 +- 4 files changed, 64 insertions(+), 44 deletions(-) diff --git a/compose/cli/docker_client.py b/compose/cli/docker_client.py index b27948446..20acbdebc 100644 --- a/compose/cli/docker_client.py +++ b/compose/cli/docker_client.py @@ -32,4 +32,4 @@ def docker_client(): ) timeout = int(os.environ.get('DOCKER_CLIENT_TIMEOUT', 60)) - return Client(base_url=base_url, tls=tls_config, version='1.14', timeout=timeout) + return Client(base_url=base_url, tls=tls_config, version='1.15', timeout=timeout) diff --git a/compose/cli/main.py b/compose/cli/main.py index 2fb853649..95dfb6cbd 100644 --- a/compose/cli/main.py +++ b/compose/cli/main.py @@ -328,20 +328,20 @@ class TopLevelCommand(Command): if options['--user']: container_options['user'] = options.get('--user') + if not options['--service-ports']: + container_options['ports'] = [] + container = service.create_container( one_off=True, insecure_registry=insecure_registry, **container_options ) - service_ports = None - if options['--service-ports']: - service_ports = service.options['ports'] if options['-d']: - service.start_container(container, ports=service_ports, one_off=True) + service.start_container(container) print(container.name) else: - service.start_container(container, ports=service_ports, one_off=True) + service.start_container(container) dockerpty.start(project.client, container.id, interactive=not options['-T']) exit_code = container.wait() if options['--rm']: diff --git a/compose/service.py b/compose/service.py index c65874c26..49e19cc56 100644 --- a/compose/service.py +++ b/compose/service.py @@ -8,6 +8,7 @@ from operator import attrgetter import sys from docker.errors import APIError +from docker.utils import create_host_config from .config import DOCKER_CONFIG_KEYS from .container import Container, get_container_name @@ -168,6 +169,7 @@ class Service(object): one_off=False, insecure_registry=False, do_build=True, + intermediate_container=None, **override_options): """ Create a container for this service. If the image doesn't exist, attempt to pull @@ -175,7 +177,9 @@ class Service(object): """ container_options = self._get_container_create_options( override_options, - one_off=one_off) + one_off=one_off, + intermediate_container=intermediate_container, + ) if (do_build and self.can_be_built() and @@ -240,56 +244,33 @@ class Service(object): entrypoint=['/bin/echo'], command=[], detach=True, + host_config=create_host_config(volumes_from=[container.id]), ) - intermediate_container.start(volumes_from=container.id) + intermediate_container.start() intermediate_container.wait() container.remove() options = dict(override_options) - new_container = self.create_container(do_build=False, **options) - self.start_container(new_container, intermediate_container=intermediate_container) + new_container = self.create_container( + do_build=False, + intermediate_container=intermediate_container, + **options + ) + self.start_container(new_container) intermediate_container.remove() return (intermediate_container, new_container) - def start_container_if_stopped(self, container, **options): + def start_container_if_stopped(self, container): if container.is_running: return container else: log.info("Starting %s..." % container.name) - return self.start_container(container, **options) + return self.start_container(container) - def start_container(self, container, intermediate_container=None, **override_options): - options = dict(self.options, **override_options) - port_bindings = build_port_bindings(options.get('ports') or []) - - volume_bindings = dict( - build_volume_binding(parse_volume_spec(volume)) - for volume in options.get('volumes') or [] - if ':' in volume) - - privileged = options.get('privileged', False) - dns = options.get('dns', None) - dns_search = options.get('dns_search', None) - cap_add = options.get('cap_add', None) - cap_drop = options.get('cap_drop', None) - - restart = parse_restart_spec(options.get('restart', None)) - - container.start( - links=self._get_links(link_to_self=options.get('one_off', False)), - port_bindings=port_bindings, - binds=volume_bindings, - volumes_from=self._get_volumes_from(intermediate_container), - privileged=privileged, - network_mode=self._get_net(), - dns=dns, - dns_search=dns_search, - restart_policy=restart, - cap_add=cap_add, - cap_drop=cap_drop, - ) + def start_container(self, container): + container.start() return container def start_or_create_containers( @@ -389,7 +370,7 @@ class Service(object): return net - def _get_container_create_options(self, override_options, one_off=False): + def _get_container_create_options(self, override_options, one_off=False, intermediate_container=None): container_options = dict( (k, self.options[k]) for k in DOCKER_CONFIG_KEYS if k in self.options) @@ -436,8 +417,47 @@ class Service(object): for key in DOCKER_START_KEYS: container_options.pop(key, None) + container_options['host_config'] = self._get_container_host_config(override_options, one_off=one_off, intermediate_container=intermediate_container) + return container_options + def _get_container_host_config(self, override_options, one_off=False, intermediate_container=None): + options = dict(self.options, **override_options) + port_bindings = build_port_bindings(options.get('ports') or []) + + volume_bindings = dict( + build_volume_binding(parse_volume_spec(volume)) + for volume in options.get('volumes') or [] + if ':' in volume) + + privileged = options.get('privileged', False) + cap_add = options.get('cap_add', None) + cap_drop = options.get('cap_drop', None) + + dns = options.get('dns', None) + if not isinstance(dns, list): + dns = [dns] + + dns_search = options.get('dns_search', None) + if not isinstance(dns_search, list): + dns_search = [dns_search] + + restart = parse_restart_spec(options.get('restart', None)) + + return create_host_config( + links=self._get_links(link_to_self=one_off), + port_bindings=port_bindings, + binds=volume_bindings, + volumes_from=self._get_volumes_from(intermediate_container), + privileged=privileged, + network_mode=self._get_net(), + dns=dns, + dns_search=dns_search, + restart_policy=restart, + cap_add=cap_add, + cap_drop=cap_drop, + ) + def _get_image_name(self, image): repo, tag = parse_repository_tag(image) if tag == "": diff --git a/tests/integration/service_test.py b/tests/integration/service_test.py index 8008fbbca..7c1695624 100644 --- a/tests/integration/service_test.py +++ b/tests/integration/service_test.py @@ -13,7 +13,7 @@ from .testcases import DockerClientTestCase def create_and_start_container(service, **override_options): container = service.create_container(**override_options) - return service.start_container(container, **override_options) + return service.start_container(container) class ServiceTest(DockerClientTestCase): From 4c582e4352f056c70b87abb4cb2d51bc231dec74 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Wed, 18 Mar 2015 13:51:27 -0700 Subject: [PATCH 29/46] Implement `extends` Signed-off-by: Aanand Prasad --- compose/config.py | 163 ++++++++++++++++-- compose/project.py | 13 +- docs/yml.md | 77 +++++++++ tests/fixtures/extends/circle-1.yml | 12 ++ tests/fixtures/extends/circle-2.yml | 12 ++ tests/fixtures/extends/common.yml | 6 + tests/fixtures/extends/docker-compose.yml | 16 ++ .../fixtures/extends/nested-intermediate.yml | 6 + tests/fixtures/extends/nested.yml | 6 + tests/integration/cli_test.py | 27 +++ tests/unit/config_test.py | 111 ++++++++++++ 11 files changed, 421 insertions(+), 28 deletions(-) create mode 100644 tests/fixtures/extends/circle-1.yml create mode 100644 tests/fixtures/extends/circle-2.yml create mode 100644 tests/fixtures/extends/common.yml create mode 100644 tests/fixtures/extends/docker-compose.yml create mode 100644 tests/fixtures/extends/nested-intermediate.yml create mode 100644 tests/fixtures/extends/nested.yml diff --git a/compose/config.py b/compose/config.py index a4e3a991f..cfa5ce44a 100644 --- a/compose/config.py +++ b/compose/config.py @@ -52,34 +52,110 @@ DOCKER_CONFIG_HINTS = { def load(filename): working_dir = os.path.dirname(filename) - return from_dictionary(load_yaml(filename), working_dir=working_dir) + return from_dictionary(load_yaml(filename), working_dir=working_dir, filename=filename) -def load_yaml(filename): - try: - with open(filename, 'r') as fh: - return yaml.safe_load(fh) - except IOError as e: - raise ConfigurationError(six.text_type(e)) - - -def from_dictionary(dictionary, working_dir=None): +def from_dictionary(dictionary, working_dir=None, filename=None): service_dicts = [] for service_name, service_dict in list(dictionary.items()): if not isinstance(service_dict, dict): raise ConfigurationError('Service "%s" doesn\'t have any configuration options. All top level keys in your docker-compose.yml must map to a dictionary of configuration options.' % service_name) - service_dict = make_service_dict(service_name, service_dict, working_dir=working_dir) + loader = ServiceLoader(working_dir=working_dir, filename=filename) + service_dict = loader.make_service_dict(service_name, service_dict) service_dicts.append(service_dict) return service_dicts -def make_service_dict(name, options, working_dir=None): - service_dict = options.copy() - service_dict['name'] = name - service_dict = resolve_environment(service_dict, working_dir=working_dir) - return process_container_options(service_dict, working_dir=working_dir) +def make_service_dict(name, service_dict, working_dir=None): + return ServiceLoader(working_dir=working_dir).make_service_dict(name, service_dict) + + +class ServiceLoader(object): + def __init__(self, working_dir, filename=None, already_seen=None): + self.working_dir = working_dir + self.filename = filename + self.already_seen = already_seen or [] + + def make_service_dict(self, name, service_dict): + if self.signature(name) in self.already_seen: + raise CircularReference(self.already_seen) + + service_dict = service_dict.copy() + service_dict['name'] = name + service_dict = resolve_environment(service_dict, working_dir=self.working_dir) + service_dict = self.resolve_extends(service_dict) + return process_container_options(service_dict, working_dir=self.working_dir) + + def resolve_extends(self, service_dict): + if 'extends' not in service_dict: + return service_dict + + extends_options = process_extends_options(service_dict['name'], service_dict['extends']) + + if self.working_dir is None: + raise Exception("No working_dir passed to ServiceLoader()") + + other_config_path = expand_path(self.working_dir, extends_options['file']) + other_working_dir = os.path.dirname(other_config_path) + other_already_seen = self.already_seen + [self.signature(service_dict['name'])] + other_loader = ServiceLoader( + working_dir=other_working_dir, + filename=other_config_path, + already_seen=other_already_seen, + ) + + other_config = load_yaml(other_config_path) + other_service_dict = other_config[extends_options['service']] + other_service_dict = other_loader.make_service_dict( + service_dict['name'], + other_service_dict, + ) + validate_extended_service_dict( + other_service_dict, + filename=other_config_path, + service=extends_options['service'], + ) + + return merge_service_dicts(other_service_dict, service_dict) + + def signature(self, name): + return (self.filename, name) + + +def process_extends_options(service_name, extends_options): + error_prefix = "Invalid 'extends' configuration for %s:" % service_name + + if not isinstance(extends_options, dict): + raise ConfigurationError("%s must be a dictionary" % error_prefix) + + if 'service' not in extends_options: + raise ConfigurationError( + "%s you need to specify a service, e.g. 'service: web'" % error_prefix + ) + + for k, _ in extends_options.items(): + if k not in ['file', 'service']: + raise ConfigurationError( + "%s unsupported configuration option '%s'" % (error_prefix, k) + ) + + return extends_options + + +def validate_extended_service_dict(service_dict, filename, service): + error_prefix = "Cannot extend service '%s' in %s:" % (service, filename) + + if 'links' in service_dict: + raise ConfigurationError("%s services with 'links' cannot be extended" % error_prefix) + + if 'volumes_from' in service_dict: + raise ConfigurationError("%s services with 'volumes_from' cannot be extended" % error_prefix) + + if 'net' in service_dict: + if get_service_name_from_net(service_dict['net']) is not None: + raise ConfigurationError("%s services with 'net: container' cannot be extended" % error_prefix) def process_container_options(service_dict, working_dir=None): @@ -93,6 +169,29 @@ def process_container_options(service_dict, working_dir=None): return service_dict +def merge_service_dicts(base, override): + d = base.copy() + + if 'environment' in base or 'environment' in override: + d['environment'] = merge_environment( + base.get('environment'), + override.get('environment'), + ) + + for k in ALLOWED_KEYS: + if k not in ['environment']: + if k in override: + d[k] = override[k] + + return d + + +def merge_environment(base, override): + env = parse_environment(base) + env.update(parse_environment(override)) + return env + + def parse_links(links): return dict(parse_link(l) for l in links) @@ -190,9 +289,41 @@ def expand_path(working_dir, path): return os.path.abspath(os.path.join(working_dir, path)) +def get_service_name_from_net(net_config): + if not net_config: + return + + if not net_config.startswith('container:'): + return + + _, net_name = net_config.split(':', 1) + return net_name + + +def load_yaml(filename): + try: + with open(filename, 'r') as fh: + return yaml.safe_load(fh) + except IOError as e: + raise ConfigurationError(six.text_type(e)) + + class ConfigurationError(Exception): def __init__(self, msg): self.msg = msg def __str__(self): return self.msg + + +class CircularReference(ConfigurationError): + def __init__(self, trail): + self.trail = trail + + @property + def msg(self): + lines = [ + "{} in {}".format(service_name, filename) + for (filename, service_name) in self.trail + ] + return "Circular reference:\n {}".format("\n extends ".join(lines)) diff --git a/compose/project.py b/compose/project.py index 881d8eb0a..7c0d19da3 100644 --- a/compose/project.py +++ b/compose/project.py @@ -3,7 +3,7 @@ from __future__ import absolute_import import logging from functools import reduce -from .config import ConfigurationError +from .config import get_service_name_from_net, ConfigurationError from .service import Service from .container import Container from docker.errors import APIError @@ -11,17 +11,6 @@ from docker.errors import APIError log = logging.getLogger(__name__) -def get_service_name_from_net(net_config): - if not net_config: - return - - if not net_config.startswith('container:'): - return - - _, net_name = net_config.split(':', 1) - return net_name - - def sort_service_dicts(services): # Topological sort (Cormen/Tarjan algorithm). unmarked = services[:] diff --git a/docs/yml.md b/docs/yml.md index 52be706a0..157ba4e67 100644 --- a/docs/yml.md +++ b/docs/yml.md @@ -176,6 +176,83 @@ env_file: RACK_ENV: development ``` +### extends + +Extend another service, in the current file or another, optionally overriding +configuration. + +Here's a simple example. Suppose we have 2 files - **common.yml** and +**development.yml**. We can use `extends` to define a service in +**development.yml** which uses configuration defined in **common.yml**: + +**common.yml** + +``` +webapp: + build: ./webapp + environment: + - DEBUG=false + - SEND_EMAILS=false +``` + +**development.yml** + +``` +web: + extends: + file: common.yml + service: webapp + ports: + - "8000:8000" + links: + - db + environment: + - DEBUG=true +db: + image: postgres +``` + +Here, the `web` service in **development.yml** inherits the configuration of +the `webapp` service in **common.yml** - the `build` and `environment` keys - +and adds `ports` and `links` configuration. It overrides one of the defined +environment variables (DEBUG) with a new value, and the other one +(SEND_EMAILS) is left untouched. It's exactly as if you defined `web` like +this: + +```yaml +web: + build: ./webapp + ports: + - "8000:8000" + links: + - db + environment: + - DEBUG=true + - SEND_EMAILS=false +``` + +The `extends` option is great for sharing configuration between different +apps, or for configuring the same app differently for different environments. +You could write a new file for a staging environment, **staging.yml**, which +binds to a different port and doesn't turn on debugging: + +``` +web: + extends: + file: common.yml + service: webapp + ports: + - "80:8000" + links: + - db +db: + image: postgres +``` + +> **Note:** When you extend a service, `links` and `volumes_from` +> configuration options are **not** inherited - you will have to define +> those manually each time you extend it. + ### net Networking mode. Use the same values as the docker client `--net` parameter. diff --git a/tests/fixtures/extends/circle-1.yml b/tests/fixtures/extends/circle-1.yml new file mode 100644 index 000000000..a034e9619 --- /dev/null +++ b/tests/fixtures/extends/circle-1.yml @@ -0,0 +1,12 @@ +foo: + image: busybox +bar: + image: busybox +web: + extends: + file: circle-2.yml + service: web +baz: + image: busybox +quux: + image: busybox diff --git a/tests/fixtures/extends/circle-2.yml b/tests/fixtures/extends/circle-2.yml new file mode 100644 index 000000000..fa6ddefcc --- /dev/null +++ b/tests/fixtures/extends/circle-2.yml @@ -0,0 +1,12 @@ +foo: + image: busybox +bar: + image: busybox +web: + extends: + file: circle-1.yml + service: web +baz: + image: busybox +quux: + image: busybox diff --git a/tests/fixtures/extends/common.yml b/tests/fixtures/extends/common.yml new file mode 100644 index 000000000..358ef5bcc --- /dev/null +++ b/tests/fixtures/extends/common.yml @@ -0,0 +1,6 @@ +web: + image: busybox + command: /bin/true + environment: + - FOO=1 + - BAR=1 diff --git a/tests/fixtures/extends/docker-compose.yml b/tests/fixtures/extends/docker-compose.yml new file mode 100644 index 000000000..0ae92d2a5 --- /dev/null +++ b/tests/fixtures/extends/docker-compose.yml @@ -0,0 +1,16 @@ +myweb: + extends: + file: common.yml + service: web + command: sleep 300 + links: + - "mydb:db" + environment: + # leave FOO alone + # override BAR + BAR: "2" + # add BAZ + BAZ: "2" +mydb: + image: busybox + command: sleep 300 diff --git a/tests/fixtures/extends/nested-intermediate.yml b/tests/fixtures/extends/nested-intermediate.yml new file mode 100644 index 000000000..c2dd8c943 --- /dev/null +++ b/tests/fixtures/extends/nested-intermediate.yml @@ -0,0 +1,6 @@ +webintermediate: + extends: + file: common.yml + service: web + environment: + - "FOO=2" diff --git a/tests/fixtures/extends/nested.yml b/tests/fixtures/extends/nested.yml new file mode 100644 index 000000000..6025e6d53 --- /dev/null +++ b/tests/fixtures/extends/nested.yml @@ -0,0 +1,6 @@ +myweb: + extends: + file: nested-intermediate.yml + service: webintermediate + environment: + - "BAR=2" diff --git a/tests/integration/cli_test.py b/tests/integration/cli_test.py index 2f961d2b2..df3eec66d 100644 --- a/tests/integration/cli_test.py +++ b/tests/integration/cli_test.py @@ -427,3 +427,30 @@ class CLITestCase(DockerClientTestCase): containers = self.project.containers(stopped=True) self.assertEqual(len(containers), 1) self.assertIn("FOO=1", containers[0].get('Config.Env')) + + def test_up_with_extends(self): + self.command.base_dir = 'tests/fixtures/extends' + self.command.dispatch(['up', '-d'], None) + + self.assertEqual( + set([s.name for s in self.project.services]), + set(['mydb', 'myweb']), + ) + + # Sort by name so we get [db, web] + containers = sorted( + self.project.containers(stopped=True), + key=lambda c: c.name, + ) + + self.assertEqual(len(containers), 2) + web = containers[1] + + self.assertEqual(set(web.links()), set(['db', 'mydb_1', 'extends_mydb_1'])) + + expected_env = set([ + "FOO=1", + "BAR=2", + "BAZ=2", + ]) + self.assertTrue(expected_env <= set(web.get('Config.Env'))) diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py index 4ff08a9ef..7e18e2e92 100644 --- a/tests/unit/config_test.py +++ b/tests/unit/config_test.py @@ -38,6 +38,8 @@ class ConfigTest(unittest.TestCase): ) config.make_service_dict('foo', {'ports': ['8000']}) + +class EnvTest(unittest.TestCase): def test_parse_environment_as_list(self): environment =[ 'NORMAL=F1', @@ -130,3 +132,112 @@ class ConfigTest(unittest.TestCase): service_dict['environment'], {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}, ) + + +class ExtendsTest(unittest.TestCase): + def test_extends(self): + service_dicts = config.load('tests/fixtures/extends/docker-compose.yml') + + service_dicts = sorted( + service_dicts, + key=lambda sd: sd['name'], + ) + + self.assertEqual(service_dicts, [ + { + 'name': 'mydb', + 'image': 'busybox', + 'command': 'sleep 300', + }, + { + 'name': 'myweb', + 'image': 'busybox', + 'command': 'sleep 300', + 'links': ['mydb:db'], + 'environment': { + "FOO": "1", + "BAR": "2", + "BAZ": "2", + }, + } + ]) + + def test_nested(self): + service_dicts = config.load('tests/fixtures/extends/nested.yml') + + self.assertEqual(service_dicts, [ + { + 'name': 'myweb', + 'image': 'busybox', + 'command': '/bin/true', + 'environment': { + "FOO": "2", + "BAR": "2", + }, + }, + ]) + + def test_circular(self): + try: + config.load('tests/fixtures/extends/circle-1.yml') + raise Exception("Expected config.CircularReference to be raised") + except config.CircularReference as e: + self.assertEqual( + [(os.path.basename(filename), service_name) for (filename, service_name) in e.trail], + [ + ('circle-1.yml', 'web'), + ('circle-2.yml', 'web'), + ('circle-1.yml', 'web'), + ], + ) + + + def test_extends_validation(self): + dictionary = {'extends': None} + load_config = lambda: config.make_service_dict('myweb', dictionary, working_dir='tests/fixtures/extends') + + self.assertRaisesRegexp(config.ConfigurationError, 'dictionary', load_config) + + dictionary['extends'] = {} + self.assertRaises(config.ConfigurationError, load_config) + + dictionary['extends']['file'] = 'common.yml' + self.assertRaisesRegexp(config.ConfigurationError, 'service', load_config) + + dictionary['extends']['service'] = 'web' + self.assertIsInstance(load_config(), dict) + + dictionary['extends']['what'] = 'is this' + self.assertRaisesRegexp(config.ConfigurationError, 'what', load_config) + + def test_blacklisted_options(self): + def load_config(): + return config.make_service_dict('myweb', { + 'extends': { + 'file': 'whatever', + 'service': 'web', + } + }, '.') + + with self.assertRaisesRegexp(config.ConfigurationError, 'links'): + other_config = {'web': {'links': ['db']}} + + with mock.patch.object(config, 'load_yaml', return_value=other_config): + print load_config() + + with self.assertRaisesRegexp(config.ConfigurationError, 'volumes_from'): + other_config = {'web': {'volumes_from': ['db']}} + + with mock.patch.object(config, 'load_yaml', return_value=other_config): + print load_config() + + with self.assertRaisesRegexp(config.ConfigurationError, 'net'): + other_config = {'web': {'net': 'container:db'}} + + with mock.patch.object(config, 'load_yaml', return_value=other_config): + print load_config() + + other_config = {'web': {'net': 'host'}} + + with mock.patch.object(config, 'load_yaml', return_value=other_config): + print load_config() From 37efdb1f8bfa5952fd15a3050f46c00c3fa582f0 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Wed, 18 Mar 2015 18:23:17 -0700 Subject: [PATCH 30/46] Make volume host paths relative to file, merge volumes when extending Signed-off-by: Aanand Prasad --- compose/config.py | 49 ++++++++++++++++++- .../fixtures/volume-path/common/services.yml | 5 ++ tests/fixtures/volume-path/docker-compose.yml | 6 +++ tests/integration/project_test.py | 21 ++++---- tests/unit/config_test.py | 11 ++++- 5 files changed, 80 insertions(+), 12 deletions(-) create mode 100644 tests/fixtures/volume-path/common/services.yml create mode 100644 tests/fixtures/volume-path/docker-compose.yml diff --git a/compose/config.py b/compose/config.py index cfa5ce44a..668d2b726 100644 --- a/compose/config.py +++ b/compose/config.py @@ -166,6 +166,11 @@ def process_container_options(service_dict, working_dir=None): msg += " (did you mean '%s'?)" % DOCKER_CONFIG_HINTS[k] raise ConfigurationError(msg) + service_dict = service_dict.copy() + + if 'volumes' in service_dict: + service_dict['volumes'] = resolve_host_paths(service_dict['volumes'], working_dir=working_dir) + return service_dict @@ -178,8 +183,14 @@ def merge_service_dicts(base, override): override.get('environment'), ) + if 'volumes' in base or 'volumes' in override: + d['volumes'] = merge_volumes( + base.get('volumes'), + override.get('volumes'), + ) + for k in ALLOWED_KEYS: - if k not in ['environment']: + if k not in ['environment', 'volumes']: if k in override: d[k] = override[k] @@ -285,6 +296,42 @@ def env_vars_from_file(filename): return env +def resolve_host_paths(volumes, working_dir=None): + if working_dir is None: + raise Exception("No working_dir passed to resolve_host_paths()") + + return [resolve_host_path(v, working_dir) for v in volumes] + + +def resolve_host_path(volume, working_dir): + container_path, host_path = split_volume(volume) + if host_path is not None: + return "%s:%s" % (expand_path(working_dir, host_path), container_path) + else: + return container_path + + +def merge_volumes(base, override): + d = dict_from_volumes(base) + d.update(dict_from_volumes(override)) + return volumes_from_dict(d) + + +def dict_from_volumes(volumes): + return dict(split_volume(v) for v in volumes) + + +def split_volume(volume): + if ':' in volume: + return reversed(volume.split(':', 1)) + else: + return (volume, None) + + +def volumes_from_dict(d): + return ["%s:%s" % (host_path, container_path) for (container_path, host_path) in d.items()] + + def expand_path(working_dir, path): return os.path.abspath(os.path.join(working_dir, path)) diff --git a/tests/fixtures/volume-path/common/services.yml b/tests/fixtures/volume-path/common/services.yml new file mode 100644 index 000000000..2dbf75961 --- /dev/null +++ b/tests/fixtures/volume-path/common/services.yml @@ -0,0 +1,5 @@ +db: + image: busybox + volumes: + - ./foo:/foo + - ./bar:/bar diff --git a/tests/fixtures/volume-path/docker-compose.yml b/tests/fixtures/volume-path/docker-compose.yml new file mode 100644 index 000000000..af433c52f --- /dev/null +++ b/tests/fixtures/volume-path/docker-compose.yml @@ -0,0 +1,6 @@ +db: + extends: + file: common/services.yml + service: db + volumes: + - ./bar:/bar diff --git a/tests/integration/project_test.py b/tests/integration/project_test.py index 3212b0be9..a585740f4 100644 --- a/tests/integration/project_test.py +++ b/tests/integration/project_test.py @@ -7,18 +7,19 @@ from .testcases import DockerClientTestCase class ProjectTest(DockerClientTestCase): def test_volumes_from_service(self): + service_dicts = config.from_dictionary({ + 'data': { + 'image': 'busybox:latest', + 'volumes': ['/var/data'], + }, + 'db': { + 'image': 'busybox:latest', + 'volumes_from': ['data'], + }, + }, working_dir='.') project = Project.from_dicts( name='composetest', - service_dicts=config.from_dictionary({ - 'data': { - 'image': 'busybox:latest', - 'volumes': ['/var/data'], - }, - 'db': { - 'image': 'busybox:latest', - 'volumes_from': ['data'], - }, - }), + service_dicts=service_dicts, client=self.client, ) db = project.get_service('db') diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py index 7e18e2e92..013ad5031 100644 --- a/tests/unit/config_test.py +++ b/tests/unit/config_test.py @@ -133,7 +133,6 @@ class EnvTest(unittest.TestCase): {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}, ) - class ExtendsTest(unittest.TestCase): def test_extends(self): service_dicts = config.load('tests/fixtures/extends/docker-compose.yml') @@ -241,3 +240,13 @@ class ExtendsTest(unittest.TestCase): with mock.patch.object(config, 'load_yaml', return_value=other_config): print load_config() + + def test_volume_path(self): + dicts = config.load('tests/fixtures/volume-path/docker-compose.yml') + + paths = [ + '%s:/foo' % os.path.abspath('tests/fixtures/volume-path/common/foo'), + '%s:/bar' % os.path.abspath('tests/fixtures/volume-path/bar'), + ] + + self.assertEqual(set(dicts[0]['volumes']), set(paths)) From d209ded13c9a7478fc47c41b2319fcf3c1935131 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Thu, 19 Mar 2015 11:21:58 -0700 Subject: [PATCH 31/46] Use dev version of Docker Signed-off-by: Aanand Prasad --- Dockerfile | 17 +++++++++++------ script/test-versions | 4 ++-- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/Dockerfile b/Dockerfile index d7a6019aa..1b016e9ee 100644 --- a/Dockerfile +++ b/Dockerfile @@ -15,13 +15,18 @@ RUN set -ex; \ ; \ rm -rf /var/lib/apt/lists/* -ENV ALL_DOCKER_VERSIONS 1.3.3 1.4.1 1.5.0 +# ENV ALL_DOCKER_VERSIONS 1.6.0 -RUN set -ex; \ - for v in ${ALL_DOCKER_VERSIONS}; do \ - curl https://get.docker.com/builds/Linux/x86_64/docker-$v -o /usr/local/bin/docker-$v; \ - chmod +x /usr/local/bin/docker-$v; \ - done +# RUN set -ex; \ +# for v in ${ALL_DOCKER_VERSIONS}; do \ +# curl https://get.docker.com/builds/Linux/x86_64/docker-$v -o /usr/local/bin/docker-$v; \ +# chmod +x /usr/local/bin/docker-$v; \ +# done + +# Temporarily use dev version of Docker +ENV ALL_DOCKER_VERSIONS dev +RUN curl https://master.dockerproject.com/linux/amd64/docker-1.5.0-dev > /usr/local/bin/docker-dev +RUN chmod +x /usr/local/bin/docker-dev RUN useradd -d /home/user -m -s /bin/bash user WORKDIR /code/ diff --git a/script/test-versions b/script/test-versions index a9e3bc4c7..d44cc9a7b 100755 --- a/script/test-versions +++ b/script/test-versions @@ -11,14 +11,14 @@ script/validate-dco flake8 compose if [ "$DOCKER_VERSIONS" == "" ]; then - DOCKER_VERSIONS="1.5.0" + DOCKER_VERSIONS="dev" elif [ "$DOCKER_VERSIONS" == "all" ]; then DOCKER_VERSIONS="$ALL_DOCKER_VERSIONS" fi for version in $DOCKER_VERSIONS; do >&2 echo "Running tests against Docker $version" - docker-1.5.0 run \ + docker-$version run \ --rm \ --privileged \ --volume="/var/lib/docker" \ From 721327110d922f807a225a572d772168c8c22641 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Thu, 19 Mar 2015 16:10:27 -0700 Subject: [PATCH 32/46] Add 'labels:' config option Signed-off-by: Aanand Prasad --- compose/cli/docker_client.py | 2 +- compose/config.py | 27 +++++++++++++++++++++++++++ docs/install.md | 2 +- docs/yml.md | 18 ++++++++++++++++++ requirements.txt | 2 +- tests/integration/service_test.py | 27 +++++++++++++++++++++++++++ 6 files changed, 75 insertions(+), 3 deletions(-) diff --git a/compose/cli/docker_client.py b/compose/cli/docker_client.py index 20acbdebc..e513182fb 100644 --- a/compose/cli/docker_client.py +++ b/compose/cli/docker_client.py @@ -32,4 +32,4 @@ def docker_client(): ) timeout = int(os.environ.get('DOCKER_CLIENT_TIMEOUT', 60)) - return Client(base_url=base_url, tls=tls_config, version='1.15', timeout=timeout) + return Client(base_url=base_url, tls=tls_config, version='1.18', timeout=timeout) diff --git a/compose/config.py b/compose/config.py index 668d2b726..0f7eec8b8 100644 --- a/compose/config.py +++ b/compose/config.py @@ -17,6 +17,7 @@ DOCKER_CONFIG_KEYS = [ 'environment', 'hostname', 'image', + 'labels', 'links', 'mem_limit', 'net', @@ -171,6 +172,9 @@ def process_container_options(service_dict, working_dir=None): if 'volumes' in service_dict: service_dict['volumes'] = resolve_host_paths(service_dict['volumes'], working_dir=working_dir) + if 'labels' in service_dict: + service_dict['labels'] = parse_labels(service_dict['labels']) + return service_dict @@ -332,6 +336,29 @@ def volumes_from_dict(d): return ["%s:%s" % (host_path, container_path) for (container_path, host_path) in d.items()] +def parse_labels(labels): + if not labels: + return {} + + if isinstance(labels, list): + return dict(split_label(e) for e in labels) + + if isinstance(labels, dict): + return labels + + raise ConfigurationError( + "labels \"%s\" must be a list or mapping" % + labels + ) + + +def split_label(label): + if '=' in label: + return label.split('=', 1) + else: + return label, '' + + def expand_path(working_dir, path): return os.path.abspath(os.path.join(working_dir, path)) diff --git a/docs/install.md b/docs/install.md index 0e60e1f18..00e4a3e3a 100644 --- a/docs/install.md +++ b/docs/install.md @@ -10,7 +10,7 @@ Compose with a `curl` command. ### Install Docker -First, install Docker version 1.3 or greater: +First, install Docker version 1.6 or greater: - [Instructions for Mac OS X](http://docs.docker.com/installation/mac/) - [Instructions for Ubuntu](http://docs.docker.com/installation/ubuntulinux/) diff --git a/docs/yml.md b/docs/yml.md index 157ba4e67..a85f0923f 100644 --- a/docs/yml.md +++ b/docs/yml.md @@ -253,6 +253,24 @@ db: > configuration options are **not** inherited - you will have to define > those manually each time you extend it. +### labels + +Add metadata to containers using [Docker labels](http://docs.docker.com/userguide/labels-custom-metadata/). You can use either an array or a dictionary. + +It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. + +``` +labels: + com.example.description: "Accounting webapp" + com.example.department: "Finance" + com.example.label-with-empty-value: "" + +labels: + - "com.example.description=Accounting webapp" + - "com.example.department=Finance" + - "com.example.label-with-empty-value" +``` + ### net Networking mode. Use the same values as the docker client `--net` parameter. diff --git a/requirements.txt b/requirements.txt index 582aac1c2..32c2b082b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ PyYAML==3.10 -docker-py==1.1.0 +-e git+https://github.com/docker/docker-py.git@70ce156e26d283d181e6ec10bd1309ddc1da1bbd#egg=docker-py dockerpty==0.3.2 docopt==0.6.1 requests==2.5.3 diff --git a/tests/integration/service_test.py b/tests/integration/service_test.py index 7c1695624..4e5ef8adb 100644 --- a/tests/integration/service_test.py +++ b/tests/integration/service_test.py @@ -491,3 +491,30 @@ class ServiceTest(DockerClientTestCase): env = create_and_start_container(service).environment for k,v in {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}.items(): self.assertEqual(env[k], v) + + def test_labels(self): + labels_dict = { + 'com.example.description': "Accounting webapp", + 'com.example.department': "Finance", + 'com.example.label-with-empty-value': "", + } + + service = self.create_service('web', labels=labels_dict) + labels = create_and_start_container(service).get('Config.Labels').items() + for pair in labels_dict.items(): + self.assertIn(pair, labels) + + labels_list = ["%s=%s" % pair for pair in labels_dict.items()] + + service = self.create_service('web', labels=labels_list) + labels = create_and_start_container(service).get('Config.Labels').items() + for pair in labels_dict.items(): + self.assertIn(pair, labels) + + def test_empty_labels(self): + labels_list = ['foo', 'bar'] + + service = self.create_service('web', labels=labels_list) + labels = create_and_start_container(service).get('Config.Labels').items() + for name in labels_list: + self.assertIn((name, ''), labels) From 965426e39b21e4ede9b4d9278cc8af50ca2bae35 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Mon, 23 Mar 2015 14:41:43 -0700 Subject: [PATCH 33/46] Revert "Add 'labels:' config option" This reverts commit 721327110d922f807a225a572d772168c8c22641. Signed-off-by: Aanand Prasad --- compose/cli/docker_client.py | 2 +- compose/config.py | 27 --------------------------- docs/install.md | 2 +- docs/yml.md | 18 ------------------ requirements.txt | 2 +- tests/integration/service_test.py | 27 --------------------------- 6 files changed, 3 insertions(+), 75 deletions(-) diff --git a/compose/cli/docker_client.py b/compose/cli/docker_client.py index e513182fb..20acbdebc 100644 --- a/compose/cli/docker_client.py +++ b/compose/cli/docker_client.py @@ -32,4 +32,4 @@ def docker_client(): ) timeout = int(os.environ.get('DOCKER_CLIENT_TIMEOUT', 60)) - return Client(base_url=base_url, tls=tls_config, version='1.18', timeout=timeout) + return Client(base_url=base_url, tls=tls_config, version='1.15', timeout=timeout) diff --git a/compose/config.py b/compose/config.py index 0f7eec8b8..668d2b726 100644 --- a/compose/config.py +++ b/compose/config.py @@ -17,7 +17,6 @@ DOCKER_CONFIG_KEYS = [ 'environment', 'hostname', 'image', - 'labels', 'links', 'mem_limit', 'net', @@ -172,9 +171,6 @@ def process_container_options(service_dict, working_dir=None): if 'volumes' in service_dict: service_dict['volumes'] = resolve_host_paths(service_dict['volumes'], working_dir=working_dir) - if 'labels' in service_dict: - service_dict['labels'] = parse_labels(service_dict['labels']) - return service_dict @@ -336,29 +332,6 @@ def volumes_from_dict(d): return ["%s:%s" % (host_path, container_path) for (container_path, host_path) in d.items()] -def parse_labels(labels): - if not labels: - return {} - - if isinstance(labels, list): - return dict(split_label(e) for e in labels) - - if isinstance(labels, dict): - return labels - - raise ConfigurationError( - "labels \"%s\" must be a list or mapping" % - labels - ) - - -def split_label(label): - if '=' in label: - return label.split('=', 1) - else: - return label, '' - - def expand_path(working_dir, path): return os.path.abspath(os.path.join(working_dir, path)) diff --git a/docs/install.md b/docs/install.md index 00e4a3e3a..0e60e1f18 100644 --- a/docs/install.md +++ b/docs/install.md @@ -10,7 +10,7 @@ Compose with a `curl` command. ### Install Docker -First, install Docker version 1.6 or greater: +First, install Docker version 1.3 or greater: - [Instructions for Mac OS X](http://docs.docker.com/installation/mac/) - [Instructions for Ubuntu](http://docs.docker.com/installation/ubuntulinux/) diff --git a/docs/yml.md b/docs/yml.md index a85f0923f..157ba4e67 100644 --- a/docs/yml.md +++ b/docs/yml.md @@ -253,24 +253,6 @@ db: > configuration options are **not** inherited - you will have to define > those manually each time you extend it. -### labels - -Add metadata to containers using [Docker labels](http://docs.docker.com/userguide/labels-custom-metadata/). You can use either an array or a dictionary. - -It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. - -``` -labels: - com.example.description: "Accounting webapp" - com.example.department: "Finance" - com.example.label-with-empty-value: "" - -labels: - - "com.example.description=Accounting webapp" - - "com.example.department=Finance" - - "com.example.label-with-empty-value" -``` - ### net Networking mode. Use the same values as the docker client `--net` parameter. diff --git a/requirements.txt b/requirements.txt index 32c2b082b..582aac1c2 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ PyYAML==3.10 --e git+https://github.com/docker/docker-py.git@70ce156e26d283d181e6ec10bd1309ddc1da1bbd#egg=docker-py +docker-py==1.1.0 dockerpty==0.3.2 docopt==0.6.1 requests==2.5.3 diff --git a/tests/integration/service_test.py b/tests/integration/service_test.py index 4e5ef8adb..7c1695624 100644 --- a/tests/integration/service_test.py +++ b/tests/integration/service_test.py @@ -491,30 +491,3 @@ class ServiceTest(DockerClientTestCase): env = create_and_start_container(service).environment for k,v in {'FILE_DEF': 'F1', 'FILE_DEF_EMPTY': '', 'ENV_DEF': 'E3', 'NO_DEF': ''}.items(): self.assertEqual(env[k], v) - - def test_labels(self): - labels_dict = { - 'com.example.description': "Accounting webapp", - 'com.example.department': "Finance", - 'com.example.label-with-empty-value': "", - } - - service = self.create_service('web', labels=labels_dict) - labels = create_and_start_container(service).get('Config.Labels').items() - for pair in labels_dict.items(): - self.assertIn(pair, labels) - - labels_list = ["%s=%s" % pair for pair in labels_dict.items()] - - service = self.create_service('web', labels=labels_list) - labels = create_and_start_container(service).get('Config.Labels').items() - for pair in labels_dict.items(): - self.assertIn(pair, labels) - - def test_empty_labels(self): - labels_list = ['foo', 'bar'] - - service = self.create_service('web', labels=labels_list) - labels = create_and_start_container(service).get('Config.Labels').items() - for name in labels_list: - self.assertIn((name, ''), labels) From 16495c577b673f16218b64e8547b4c5d2ae490ec Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Mon, 23 Mar 2015 14:41:53 -0700 Subject: [PATCH 34/46] Revert "Use dev version of Docker" This reverts commit d209ded13c9a7478fc47c41b2319fcf3c1935131. Signed-off-by: Aanand Prasad --- Dockerfile | 17 ++++++----------- script/test-versions | 4 ++-- 2 files changed, 8 insertions(+), 13 deletions(-) diff --git a/Dockerfile b/Dockerfile index 1b016e9ee..d7a6019aa 100644 --- a/Dockerfile +++ b/Dockerfile @@ -15,18 +15,13 @@ RUN set -ex; \ ; \ rm -rf /var/lib/apt/lists/* -# ENV ALL_DOCKER_VERSIONS 1.6.0 +ENV ALL_DOCKER_VERSIONS 1.3.3 1.4.1 1.5.0 -# RUN set -ex; \ -# for v in ${ALL_DOCKER_VERSIONS}; do \ -# curl https://get.docker.com/builds/Linux/x86_64/docker-$v -o /usr/local/bin/docker-$v; \ -# chmod +x /usr/local/bin/docker-$v; \ -# done - -# Temporarily use dev version of Docker -ENV ALL_DOCKER_VERSIONS dev -RUN curl https://master.dockerproject.com/linux/amd64/docker-1.5.0-dev > /usr/local/bin/docker-dev -RUN chmod +x /usr/local/bin/docker-dev +RUN set -ex; \ + for v in ${ALL_DOCKER_VERSIONS}; do \ + curl https://get.docker.com/builds/Linux/x86_64/docker-$v -o /usr/local/bin/docker-$v; \ + chmod +x /usr/local/bin/docker-$v; \ + done RUN useradd -d /home/user -m -s /bin/bash user WORKDIR /code/ diff --git a/script/test-versions b/script/test-versions index d44cc9a7b..a9e3bc4c7 100755 --- a/script/test-versions +++ b/script/test-versions @@ -11,14 +11,14 @@ script/validate-dco flake8 compose if [ "$DOCKER_VERSIONS" == "" ]; then - DOCKER_VERSIONS="dev" + DOCKER_VERSIONS="1.5.0" elif [ "$DOCKER_VERSIONS" == "all" ]; then DOCKER_VERSIONS="$ALL_DOCKER_VERSIONS" fi for version in $DOCKER_VERSIONS; do >&2 echo "Running tests against Docker $version" - docker-$version run \ + docker-1.5.0 run \ --rm \ --privileged \ --volume="/var/lib/docker" \ From f4ef2c09d66a0bf9fe5a82259e4a88af9f5c16e1 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Mon, 23 Mar 2015 14:42:21 -0700 Subject: [PATCH 35/46] Revert "Remove restriction for requests version, update docker-py requirement" This reverts commit 81a32a266f998e144d97152cae953300d8956a78. Signed-off-by: Aanand Prasad --- requirements.txt | 4 ++-- setup.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/requirements.txt b/requirements.txt index 582aac1c2..4c4113ab9 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,8 +1,8 @@ PyYAML==3.10 -docker-py==1.1.0 +docker-py==1.0.0 dockerpty==0.3.2 docopt==0.6.1 -requests==2.5.3 +requests==2.2.1 six==1.7.3 texttable==0.8.2 websocket-client==0.11.0 diff --git a/setup.py b/setup.py index cce4d7cdf..0e25abd48 100644 --- a/setup.py +++ b/setup.py @@ -27,10 +27,10 @@ def find_version(*file_paths): install_requires = [ 'docopt >= 0.6.1, < 0.7', 'PyYAML >= 3.10, < 4', - 'requests >= 2.5.0, < 2.6', + 'requests >= 2.2.1, < 2.5.0', 'texttable >= 0.8.1, < 0.9', 'websocket-client >= 0.11.0, < 1.0', - 'docker-py >= 1.1.0, < 1.2', + 'docker-py >= 1.0.0, < 1.1.0', 'dockerpty >= 0.3.2, < 0.4', 'six >= 1.3.0, < 2', ] From 1c14fc06da0d685b3af7f00eb97310e05174f88b Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Mon, 23 Mar 2015 14:44:28 -0700 Subject: [PATCH 36/46] Update docker-py and requests version ranges Leave the pinned versions in requirements.txt alone, as there's an incompatibility between PyInstaller and requests 2.5.2 and 2.5.3, and by extension with docker-py 1.1.0. Signed-off-by: Aanand Prasad --- setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index 0e25abd48..a8f7d98ea 100644 --- a/setup.py +++ b/setup.py @@ -27,10 +27,10 @@ def find_version(*file_paths): install_requires = [ 'docopt >= 0.6.1, < 0.7', 'PyYAML >= 3.10, < 4', - 'requests >= 2.2.1, < 2.5.0', + 'requests >= 2.2.1, < 2.6', 'texttable >= 0.8.1, < 0.9', 'websocket-client >= 0.11.0, < 1.0', - 'docker-py >= 1.0.0, < 1.1.0', + 'docker-py >= 1.0.0, < 1.2', 'dockerpty >= 0.3.2, < 0.4', 'six >= 1.3.0, < 2', ] From 276e43ca6b41917d2383199c870a2e73b7bbfa8e Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Mon, 23 Mar 2015 11:31:30 -0700 Subject: [PATCH 37/46] Fix service dict merging when only one dict has a volumes key Signed-off-by: Aanand Prasad --- compose/config.py | 5 ++++- tests/unit/config_test.py | 23 +++++++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/compose/config.py b/compose/config.py index 668d2b726..af30a702f 100644 --- a/compose/config.py +++ b/compose/config.py @@ -318,7 +318,10 @@ def merge_volumes(base, override): def dict_from_volumes(volumes): - return dict(split_volume(v) for v in volumes) + if volumes: + return dict(split_volume(v) for v in volumes) + else: + return {} def split_volume(volume): diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py index 013ad5031..323323d41 100644 --- a/tests/unit/config_test.py +++ b/tests/unit/config_test.py @@ -39,6 +39,29 @@ class ConfigTest(unittest.TestCase): config.make_service_dict('foo', {'ports': ['8000']}) +class MergeTest(unittest.TestCase): + def test_merge_volumes(self): + service_dict = config.merge_service_dicts({}, {}) + self.assertNotIn('volumes', service_dict) + + service_dict = config.merge_service_dicts({ + 'volumes': ['/foo:/data'], + }, {}) + self.assertEqual(service_dict['volumes'], ['/foo:/data']) + + service_dict = config.merge_service_dicts({}, { + 'volumes': ['/bar:/data'], + }) + self.assertEqual(service_dict['volumes'], ['/bar:/data']) + + service_dict = config.merge_service_dicts({ + 'volumes': ['/foo:/data'], + }, { + 'volumes': ['/bar:/data'], + }) + self.assertEqual(service_dict['volumes'], ['/bar:/data']) + + class EnvTest(unittest.TestCase): def test_parse_environment_as_list(self): environment =[ From 35c6e0314c07b9a6d2e97831a1bc4523526a6344 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Mon, 23 Mar 2015 11:40:44 -0700 Subject: [PATCH 38/46] Fix volume merging when there's no explicit host path Signed-off-by: Aanand Prasad --- compose/config.py | 25 +++++++++++++------- tests/unit/config_test.py | 48 +++++++++++++++++++++++++++------------ 2 files changed, 50 insertions(+), 23 deletions(-) diff --git a/compose/config.py b/compose/config.py index af30a702f..0cd7c1ae6 100644 --- a/compose/config.py +++ b/compose/config.py @@ -324,15 +324,24 @@ def dict_from_volumes(volumes): return {} -def split_volume(volume): - if ':' in volume: - return reversed(volume.split(':', 1)) - else: - return (volume, None) - - def volumes_from_dict(d): - return ["%s:%s" % (host_path, container_path) for (container_path, host_path) in d.items()] + return [join_volume(v) for v in d.items()] + + +def split_volume(string): + if ':' in string: + (host, container) = string.split(':', 1) + return (container, host) + else: + return (string, None) + + +def join_volume(pair): + (container, host) = pair + if host is None: + return container + else: + return ":".join((host, container)) def expand_path(working_dir, path): diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py index 323323d41..8deb457af 100644 --- a/tests/unit/config_test.py +++ b/tests/unit/config_test.py @@ -40,26 +40,44 @@ class ConfigTest(unittest.TestCase): class MergeTest(unittest.TestCase): - def test_merge_volumes(self): + def test_merge_volumes_empty(self): service_dict = config.merge_service_dicts({}, {}) self.assertNotIn('volumes', service_dict) - service_dict = config.merge_service_dicts({ - 'volumes': ['/foo:/data'], - }, {}) - self.assertEqual(service_dict['volumes'], ['/foo:/data']) + def test_merge_volumes_no_override(self): + service_dict = config.merge_service_dicts( + {'volumes': ['/foo:/code', '/data']}, + {}, + ) + self.assertEqual(set(service_dict['volumes']), set(['/foo:/code', '/data'])) - service_dict = config.merge_service_dicts({}, { - 'volumes': ['/bar:/data'], - }) - self.assertEqual(service_dict['volumes'], ['/bar:/data']) + def test_merge_volumes_no_base(self): + service_dict = config.merge_service_dicts( + {}, + {'volumes': ['/bar:/code']}, + ) + self.assertEqual(set(service_dict['volumes']), set(['/bar:/code'])) - service_dict = config.merge_service_dicts({ - 'volumes': ['/foo:/data'], - }, { - 'volumes': ['/bar:/data'], - }) - self.assertEqual(service_dict['volumes'], ['/bar:/data']) + def test_merge_volumes_override_explicit_path(self): + service_dict = config.merge_service_dicts( + {'volumes': ['/foo:/code', '/data']}, + {'volumes': ['/bar:/code']}, + ) + self.assertEqual(set(service_dict['volumes']), set(['/bar:/code', '/data'])) + + def test_merge_volumes_add_explicit_path(self): + service_dict = config.merge_service_dicts( + {'volumes': ['/foo:/code', '/data']}, + {'volumes': ['/bar:/code', '/quux:/data']}, + ) + self.assertEqual(set(service_dict['volumes']), set(['/bar:/code', '/quux:/data'])) + + def test_merge_volumes_remove_explicit_path(self): + service_dict = config.merge_service_dicts( + {'volumes': ['/foo:/code', '/quux:/data']}, + {'volumes': ['/bar:/code', '/data']}, + ) + self.assertEqual(set(service_dict['volumes']), set(['/bar:/code', '/data'])) class EnvTest(unittest.TestCase): From 83dcceacafa720e55efa499485495c9f9dc59d93 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Tue, 24 Mar 2015 16:17:05 -0700 Subject: [PATCH 39/46] Fix regression in Dns and DnsSearch settings Signed-off-by: Aanand Prasad --- compose/service.py | 5 +++-- tests/integration/service_test.py | 10 ++++++++++ 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/compose/service.py b/compose/service.py index 49e19cc56..936e3f9d0 100644 --- a/compose/service.py +++ b/compose/service.py @@ -6,6 +6,7 @@ import re import os from operator import attrgetter import sys +import six from docker.errors import APIError from docker.utils import create_host_config @@ -435,11 +436,11 @@ class Service(object): cap_drop = options.get('cap_drop', None) dns = options.get('dns', None) - if not isinstance(dns, list): + if isinstance(dns, six.string_types): dns = [dns] dns_search = options.get('dns_search', None) - if not isinstance(dns_search, list): + if isinstance(dns_search, six.string_types): dns_search = [dns_search] restart = parse_restart_spec(options.get('restart', None)) diff --git a/tests/integration/service_test.py b/tests/integration/service_test.py index 7c1695624..f0fb771d9 100644 --- a/tests/integration/service_test.py +++ b/tests/integration/service_test.py @@ -424,6 +424,11 @@ class ServiceTest(DockerClientTestCase): container = create_and_start_container(service) self.assertEqual(container.get('HostConfig.NetworkMode'), 'host') + def test_dns_no_value(self): + service = self.create_service('web') + container = create_and_start_container(service) + self.assertIsNone(container.get('HostConfig.Dns')) + def test_dns_single_value(self): service = self.create_service('web', dns='8.8.8.8') container = create_and_start_container(service) @@ -455,6 +460,11 @@ class ServiceTest(DockerClientTestCase): container = create_and_start_container(service) self.assertEqual(container.get('HostConfig.CapDrop'), ['SYS_ADMIN', 'NET_ADMIN']) + def test_dns_search_no_value(self): + service = self.create_service('web') + container = create_and_start_container(service) + self.assertIsNone(container.get('HostConfig.DnsSearch')) + def test_dns_search_single_value(self): service = self.create_service('web', dns_search='example.com') container = create_and_start_container(service) From 461b600068465fb9260f3918ba12e8d501986515 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Tue, 31 Mar 2015 15:23:34 -0400 Subject: [PATCH 40/46] Merge pull request #1225 from aanand/fix-1222 When extending, `build` replaces `image` and vice versa (cherry picked from commit 6dbe321a45dfd7539234f889825b54e1a026e46f) Signed-off-by: Aanand Prasad --- compose/config.py | 6 ++++++ tests/unit/config_test.py | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/compose/config.py b/compose/config.py index 0cd7c1ae6..8d6ffea7a 100644 --- a/compose/config.py +++ b/compose/config.py @@ -189,6 +189,12 @@ def merge_service_dicts(base, override): override.get('volumes'), ) + if 'image' in override and 'build' in d: + del d['build'] + + if 'build' in override and 'image' in d: + del d['image'] + for k in ALLOWED_KEYS: if k not in ['environment', 'volumes']: if k in override: diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py index 8deb457af..67f24a92d 100644 --- a/tests/unit/config_test.py +++ b/tests/unit/config_test.py @@ -79,6 +79,39 @@ class MergeTest(unittest.TestCase): ) self.assertEqual(set(service_dict['volumes']), set(['/bar:/code', '/data'])) + def test_merge_build_or_image_no_override(self): + self.assertEqual( + config.merge_service_dicts({'build': '.'}, {}), + {'build': '.'}, + ) + + self.assertEqual( + config.merge_service_dicts({'image': 'redis'}, {}), + {'image': 'redis'}, + ) + + def test_merge_build_or_image_override_with_same(self): + self.assertEqual( + config.merge_service_dicts({'build': '.'}, {'build': './web'}), + {'build': './web'}, + ) + + self.assertEqual( + config.merge_service_dicts({'image': 'redis'}, {'image': 'postgres'}), + {'image': 'postgres'}, + ) + + def test_merge_build_or_image_override_with_other(self): + self.assertEqual( + config.merge_service_dicts({'build': '.'}, {'image': 'redis'}), + {'image': 'redis'} + ) + + self.assertEqual( + config.merge_service_dicts({'image': 'redis'}, {'build': '.'}), + {'build': '.'} + ) + class EnvTest(unittest.TestCase): def test_parse_environment_as_list(self): From b24a60ba9fdbfc9d3bdade6efbc9b629cdd4872b Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Tue, 31 Mar 2015 16:01:22 -0400 Subject: [PATCH 41/46] Merge pull request #1226 from aanand/merge-multi-value-options Merge multi-value options when extending (cherry picked from commit e708f4f59dcb417e90a5bbdcadcee37e8c6b7802) Signed-off-by: Aanand Prasad --- compose/config.py | 30 ++++++++++++++--- tests/unit/config_test.py | 71 +++++++++++++++++++++++++++++++++++---- 2 files changed, 90 insertions(+), 11 deletions(-) diff --git a/compose/config.py b/compose/config.py index 8d6ffea7a..7f2e302b3 100644 --- a/compose/config.py +++ b/compose/config.py @@ -195,10 +195,23 @@ def merge_service_dicts(base, override): if 'build' in override and 'image' in d: del d['image'] - for k in ALLOWED_KEYS: - if k not in ['environment', 'volumes']: - if k in override: - d[k] = override[k] + list_keys = ['ports', 'expose', 'external_links'] + + for key in list_keys: + if key in base or key in override: + d[key] = base.get(key, []) + override.get(key, []) + + list_or_string_keys = ['dns', 'dns_search'] + + for key in list_or_string_keys: + if key in base or key in override: + d[key] = to_list(base.get(key)) + to_list(override.get(key)) + + already_merged_keys = ['environment', 'volumes'] + list_keys + list_or_string_keys + + for k in set(ALLOWED_KEYS) - set(already_merged_keys): + if k in override: + d[k] = override[k] return d @@ -354,6 +367,15 @@ def expand_path(working_dir, path): return os.path.abspath(os.path.join(working_dir, path)) +def to_list(value): + if value is None: + return [] + elif isinstance(value, six.string_types): + return [value] + else: + return value + + def get_service_name_from_net(net_config): if not net_config: return diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py index 67f24a92d..d95ba7838 100644 --- a/tests/unit/config_test.py +++ b/tests/unit/config_test.py @@ -39,40 +39,40 @@ class ConfigTest(unittest.TestCase): config.make_service_dict('foo', {'ports': ['8000']}) -class MergeTest(unittest.TestCase): - def test_merge_volumes_empty(self): +class MergeVolumesTest(unittest.TestCase): + def test_empty(self): service_dict = config.merge_service_dicts({}, {}) self.assertNotIn('volumes', service_dict) - def test_merge_volumes_no_override(self): + def test_no_override(self): service_dict = config.merge_service_dicts( {'volumes': ['/foo:/code', '/data']}, {}, ) self.assertEqual(set(service_dict['volumes']), set(['/foo:/code', '/data'])) - def test_merge_volumes_no_base(self): + def test_no_base(self): service_dict = config.merge_service_dicts( {}, {'volumes': ['/bar:/code']}, ) self.assertEqual(set(service_dict['volumes']), set(['/bar:/code'])) - def test_merge_volumes_override_explicit_path(self): + def test_override_explicit_path(self): service_dict = config.merge_service_dicts( {'volumes': ['/foo:/code', '/data']}, {'volumes': ['/bar:/code']}, ) self.assertEqual(set(service_dict['volumes']), set(['/bar:/code', '/data'])) - def test_merge_volumes_add_explicit_path(self): + def test_add_explicit_path(self): service_dict = config.merge_service_dicts( {'volumes': ['/foo:/code', '/data']}, {'volumes': ['/bar:/code', '/quux:/data']}, ) self.assertEqual(set(service_dict['volumes']), set(['/bar:/code', '/quux:/data'])) - def test_merge_volumes_remove_explicit_path(self): + def test_remove_explicit_path(self): service_dict = config.merge_service_dicts( {'volumes': ['/foo:/code', '/quux:/data']}, {'volumes': ['/bar:/code', '/data']}, @@ -113,6 +113,63 @@ class MergeTest(unittest.TestCase): ) +class MergeListsTest(unittest.TestCase): + def test_empty(self): + service_dict = config.merge_service_dicts({}, {}) + self.assertNotIn('ports', service_dict) + + def test_no_override(self): + service_dict = config.merge_service_dicts( + {'ports': ['10:8000', '9000']}, + {}, + ) + self.assertEqual(set(service_dict['ports']), set(['10:8000', '9000'])) + + def test_no_base(self): + service_dict = config.merge_service_dicts( + {}, + {'ports': ['10:8000', '9000']}, + ) + self.assertEqual(set(service_dict['ports']), set(['10:8000', '9000'])) + + def test_add_item(self): + service_dict = config.merge_service_dicts( + {'ports': ['10:8000', '9000']}, + {'ports': ['20:8000']}, + ) + self.assertEqual(set(service_dict['ports']), set(['10:8000', '9000', '20:8000'])) + + +class MergeStringsOrListsTest(unittest.TestCase): + def test_no_override(self): + service_dict = config.merge_service_dicts( + {'dns': '8.8.8.8'}, + {}, + ) + self.assertEqual(set(service_dict['dns']), set(['8.8.8.8'])) + + def test_no_base(self): + service_dict = config.merge_service_dicts( + {}, + {'dns': '8.8.8.8'}, + ) + self.assertEqual(set(service_dict['dns']), set(['8.8.8.8'])) + + def test_add_string(self): + service_dict = config.merge_service_dicts( + {'dns': ['8.8.8.8']}, + {'dns': '9.9.9.9'}, + ) + self.assertEqual(set(service_dict['dns']), set(['8.8.8.8', '9.9.9.9'])) + + def test_add_list(self): + service_dict = config.merge_service_dicts( + {'dns': '8.8.8.8'}, + {'dns': ['9.9.9.9']}, + ) + self.assertEqual(set(service_dict['dns']), set(['8.8.8.8', '9.9.9.9'])) + + class EnvTest(unittest.TestCase): def test_parse_environment_as_list(self): environment =[ From e4e802d1f86ceb16d45d0176f94906e799f90fc9 Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Tue, 31 Mar 2015 21:20:02 -0400 Subject: [PATCH 42/46] Merge pull request #1213 from moysesb/relative_build Make value of 'build:' relative to the yml file. (cherry picked from commit 0f70b8638ff7167e9755d24dc8dab1579662f72d) Signed-off-by: Aanand Prasad --- compose/config.py | 14 ++++++++ docs/yml.md | 5 +-- tests/fixtures/build-ctx/Dockerfile | 2 ++ tests/fixtures/build-path/docker-compose.yml | 2 ++ .../docker-compose.yml | 2 +- .../simple-dockerfile/docker-compose.yml | 2 +- tests/unit/config_test.py | 33 +++++++++++++++++++ 7 files changed, 56 insertions(+), 4 deletions(-) create mode 100644 tests/fixtures/build-ctx/Dockerfile create mode 100644 tests/fixtures/build-path/docker-compose.yml diff --git a/compose/config.py b/compose/config.py index 7f2e302b3..1dc64af25 100644 --- a/compose/config.py +++ b/compose/config.py @@ -171,6 +171,9 @@ def process_container_options(service_dict, working_dir=None): if 'volumes' in service_dict: service_dict['volumes'] = resolve_host_paths(service_dict['volumes'], working_dir=working_dir) + if 'build' in service_dict: + service_dict['build'] = resolve_build_path(service_dict['build'], working_dir=working_dir) + return service_dict @@ -330,6 +333,17 @@ def resolve_host_path(volume, working_dir): return container_path +def resolve_build_path(build_path, working_dir=None): + if working_dir is None: + raise Exception("No working_dir passed to resolve_build_path") + + _path = expand_path(working_dir, build_path) + if not os.path.exists(_path) or not os.access(_path, os.R_OK): + raise ConfigurationError("build path %s either does not exist or is not accessible." % _path) + else: + return _path + + def merge_volumes(base, override): d = dict_from_volumes(base) d.update(dict_from_volumes(override)) diff --git a/docs/yml.md b/docs/yml.md index 157ba4e67..a9909e816 100644 --- a/docs/yml.md +++ b/docs/yml.md @@ -29,8 +29,9 @@ image: a4bc65fd ### build -Path to a directory containing a Dockerfile. This directory is also the -build context that is sent to the Docker daemon. +Path to a directory containing a Dockerfile. When the value supplied is a +relative path, it is interpreted as relative to the location of the yml file +itself. This directory is also the build context that is sent to the Docker daemon. Compose will build and tag it with a generated name, and use that image thereafter. diff --git a/tests/fixtures/build-ctx/Dockerfile b/tests/fixtures/build-ctx/Dockerfile new file mode 100644 index 000000000..d1ceac6b7 --- /dev/null +++ b/tests/fixtures/build-ctx/Dockerfile @@ -0,0 +1,2 @@ +FROM busybox:latest +CMD echo "success" diff --git a/tests/fixtures/build-path/docker-compose.yml b/tests/fixtures/build-path/docker-compose.yml new file mode 100644 index 000000000..66e8916e9 --- /dev/null +++ b/tests/fixtures/build-path/docker-compose.yml @@ -0,0 +1,2 @@ +foo: + build: ../build-ctx/ diff --git a/tests/fixtures/dockerfile_with_entrypoint/docker-compose.yml b/tests/fixtures/dockerfile_with_entrypoint/docker-compose.yml index a10381187..786315020 100644 --- a/tests/fixtures/dockerfile_with_entrypoint/docker-compose.yml +++ b/tests/fixtures/dockerfile_with_entrypoint/docker-compose.yml @@ -1,2 +1,2 @@ service: - build: tests/fixtures/dockerfile_with_entrypoint + build: . diff --git a/tests/fixtures/simple-dockerfile/docker-compose.yml b/tests/fixtures/simple-dockerfile/docker-compose.yml index a3f56d46f..b0357541e 100644 --- a/tests/fixtures/simple-dockerfile/docker-compose.yml +++ b/tests/fixtures/simple-dockerfile/docker-compose.yml @@ -1,2 +1,2 @@ simple: - build: tests/fixtures/simple-dockerfile + build: . diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py index d95ba7838..f25f3a9d6 100644 --- a/tests/unit/config_test.py +++ b/tests/unit/config_test.py @@ -381,3 +381,36 @@ class ExtendsTest(unittest.TestCase): ] self.assertEqual(set(dicts[0]['volumes']), set(paths)) + + +class BuildPathTest(unittest.TestCase): + def setUp(self): + self.abs_context_path = os.path.join(os.getcwd(), 'tests/fixtures/build-ctx') + + def test_nonexistent_path(self): + options = {'build': 'nonexistent.path'} + self.assertRaises( + config.ConfigurationError, + lambda: config.make_service_dict('foo', options, 'tests/fixtures/build-path'), + ) + + def test_relative_path(self): + relative_build_path = '../build-ctx/' + service_dict = config.make_service_dict( + 'relpath', + {'build': relative_build_path}, + working_dir='tests/fixtures/build-path' + ) + self.assertEquals(service_dict['build'], self.abs_context_path) + + def test_absolute_path(self): + service_dict = config.make_service_dict( + 'abspath', + {'build': self.abs_context_path}, + working_dir='tests/fixtures/build-path' + ) + self.assertEquals(service_dict['build'], self.abs_context_path) + + def test_from_file(self): + service_dict = config.load('tests/fixtures/build-path/docker-compose.yml') + self.assertEquals(service_dict, [{'name': 'foo', 'build': self.abs_context_path}]) From 78227c3c068a3ca7be47d3104fceb8c1e065e078 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Fri, 27 Mar 2015 14:59:49 -0700 Subject: [PATCH 43/46] Merge pull request #1202 from aanand/jenkins-script WIP: Jenkins script (cherry picked from commit 853ce255eac5375562e399d3e105dc5a456dbb99) Signed-off-by: Aanand Prasad --- Dockerfile | 3 +++ script/build-linux | 18 +++++++++++------- script/build-linux-inner | 10 ++++++++++ script/ci | 18 ++++++++++++++++++ script/test-versions | 5 +---- script/wrapdocker | 2 +- 6 files changed, 44 insertions(+), 12 deletions(-) create mode 100755 script/build-linux-inner create mode 100755 script/ci diff --git a/Dockerfile b/Dockerfile index d7a6019aa..8ec05cc9b 100644 --- a/Dockerfile +++ b/Dockerfile @@ -23,6 +23,9 @@ RUN set -ex; \ chmod +x /usr/local/bin/docker-$v; \ done +# Set the default Docker to be run +RUN ln -s /usr/local/bin/docker-1.3.3 /usr/local/bin/docker + RUN useradd -d /home/user -m -s /bin/bash user WORKDIR /code/ diff --git a/script/build-linux b/script/build-linux index 07c9d7ec6..5e4a9470e 100755 --- a/script/build-linux +++ b/script/build-linux @@ -1,8 +1,12 @@ -#!/bin/sh +#!/bin/bash + set -ex -mkdir -p `pwd`/dist -chmod 777 `pwd`/dist -docker build -t docker-compose . -docker run -u user -v `pwd`/dist:/code/dist --rm --entrypoint pyinstaller docker-compose -F bin/docker-compose -mv dist/docker-compose dist/docker-compose-Linux-x86_64 -docker run -u user -v `pwd`/dist:/code/dist --rm --entrypoint dist/docker-compose-Linux-x86_64 docker-compose --version + +TAG="docker-compose" +docker build -t "$TAG" . +docker run \ + --rm \ + --user=user \ + --volume="$(pwd):/code" \ + --entrypoint="script/build-linux-inner" \ + "$TAG" diff --git a/script/build-linux-inner b/script/build-linux-inner new file mode 100755 index 000000000..34b0c06fd --- /dev/null +++ b/script/build-linux-inner @@ -0,0 +1,10 @@ +#!/bin/bash + +set -ex + +mkdir -p `pwd`/dist +chmod 777 `pwd`/dist + +pyinstaller -F bin/docker-compose +mv dist/docker-compose dist/docker-compose-Linux-x86_64 +dist/docker-compose-Linux-x86_64 --version diff --git a/script/ci b/script/ci new file mode 100755 index 000000000..a1391c627 --- /dev/null +++ b/script/ci @@ -0,0 +1,18 @@ +#!/bin/bash +# This should be run inside a container built from the Dockerfile +# at the root of the repo: +# +# $ TAG="docker-compose:$(git rev-parse --short HEAD)" +# $ docker build -t "$TAG" . +# $ docker run --rm --volume="/var/run/docker.sock:/var/run/docker.sock" --volume="$(pwd)/.git:/code/.git" -e "TAG=$TAG" --entrypoint="script/ci" "$TAG" + +set -e + +>&2 echo "Validating DCO" +script/validate-dco + +export DOCKER_VERSIONS=all +. script/test-versions + +>&2 echo "Building Linux binary" +su -c script/build-linux-inner user diff --git a/script/test-versions b/script/test-versions index a9e3bc4c7..a172b9a33 100755 --- a/script/test-versions +++ b/script/test-versions @@ -4,9 +4,6 @@ set -e ->&2 echo "Validating DCO" -script/validate-dco - >&2 echo "Running lint checks" flake8 compose @@ -18,7 +15,7 @@ fi for version in $DOCKER_VERSIONS; do >&2 echo "Running tests against Docker $version" - docker-1.5.0 run \ + docker run \ --rm \ --privileged \ --volume="/var/lib/docker" \ diff --git a/script/wrapdocker b/script/wrapdocker index 20dc9e3ce..7b699688a 100755 --- a/script/wrapdocker +++ b/script/wrapdocker @@ -4,7 +4,7 @@ if [ "$DOCKER_VERSION" == "" ]; then DOCKER_VERSION="1.5.0" fi -ln -s "/usr/local/bin/docker-$DOCKER_VERSION" "/usr/local/bin/docker" +ln -fs "/usr/local/bin/docker-$DOCKER_VERSION" "/usr/local/bin/docker" # If a pidfile is still around (for example after a container restart), # delete it so that docker can start. From a467a8a09486e9770a4d7de4f982aeb17d8439b2 Mon Sep 17 00:00:00 2001 From: Ben Firshman Date: Thu, 9 Apr 2015 14:44:07 +0100 Subject: [PATCH 44/46] Merge pull request #1261 from aanand/fix-vars-in-volume-paths Fix vars in volume paths (cherry picked from commit 4926f8aef629631032327542a56ae35099807005) Signed-off-by: Aanand Prasad Conflicts: tests/unit/service_test.py --- compose/config.py | 2 ++ compose/service.py | 4 +--- tests/integration/service_test.py | 18 ++++++++++++++++++ tests/unit/config_test.py | 14 ++++++++++++++ tests/unit/service_test.py | 16 ---------------- 5 files changed, 35 insertions(+), 19 deletions(-) diff --git a/compose/config.py b/compose/config.py index 1dc64af25..2dc59d231 100644 --- a/compose/config.py +++ b/compose/config.py @@ -328,6 +328,8 @@ def resolve_host_paths(volumes, working_dir=None): def resolve_host_path(volume, working_dir): container_path, host_path = split_volume(volume) if host_path is not None: + host_path = os.path.expanduser(host_path) + host_path = os.path.expandvars(host_path) return "%s:%s" % (expand_path(working_dir, host_path), container_path) else: return container_path diff --git a/compose/service.py b/compose/service.py index 936e3f9d0..86427a1ea 100644 --- a/compose/service.py +++ b/compose/service.py @@ -3,7 +3,6 @@ from __future__ import absolute_import from collections import namedtuple import logging import re -import os from operator import attrgetter import sys import six @@ -586,8 +585,7 @@ def parse_repository_tag(s): def build_volume_binding(volume_spec): internal = {'bind': volume_spec.internal, 'ro': volume_spec.mode == 'ro'} - external = os.path.expanduser(volume_spec.external) - return os.path.abspath(os.path.expandvars(external)), internal + return volume_spec.external, internal def build_port_bindings(ports): diff --git a/tests/integration/service_test.py b/tests/integration/service_test.py index f0fb771d9..a89fde97b 100644 --- a/tests/integration/service_test.py +++ b/tests/integration/service_test.py @@ -123,6 +123,24 @@ class ServiceTest(DockerClientTestCase): self.assertTrue(path.basename(actual_host_path) == path.basename(host_path), msg=("Last component differs: %s, %s" % (actual_host_path, host_path))) + @mock.patch.dict(os.environ) + def test_create_container_with_home_and_env_var_in_volume_path(self): + os.environ['VOLUME_NAME'] = 'my-volume' + os.environ['HOME'] = '/tmp/home-dir' + expected_host_path = os.path.join(os.environ['HOME'], os.environ['VOLUME_NAME']) + + host_path = '~/${VOLUME_NAME}' + container_path = '/container-path' + + service = self.create_service('db', volumes=['%s:%s' % (host_path, container_path)]) + container = service.create_container() + service.start_container(container) + + actual_host_path = container.get('Volumes')[container_path] + components = actual_host_path.split('/') + self.assertTrue(components[-2:] == ['home-dir', 'my-volume'], + msg="Last two components differ: %s, %s" % (actual_host_path, expected_host_path)) + def test_create_container_with_volumes_from(self): volume_service = self.create_service('data') volume_container_1 = volume_service.create_container() diff --git a/tests/unit/config_test.py b/tests/unit/config_test.py index f25f3a9d6..aa14a2a5e 100644 --- a/tests/unit/config_test.py +++ b/tests/unit/config_test.py @@ -39,6 +39,20 @@ class ConfigTest(unittest.TestCase): config.make_service_dict('foo', {'ports': ['8000']}) +class VolumePathTest(unittest.TestCase): + @mock.patch.dict(os.environ) + def test_volume_binding_with_environ(self): + os.environ['VOLUME_PATH'] = '/host/path' + d = config.make_service_dict('foo', {'volumes': ['${VOLUME_PATH}:/container/path']}, working_dir='.') + self.assertEqual(d['volumes'], ['/host/path:/container/path']) + + @mock.patch.dict(os.environ) + def test_volume_binding_with_home(self): + os.environ['HOME'] = '/home/user' + d = config.make_service_dict('foo', {'volumes': ['~:/container/path']}, working_dir='.') + self.assertEqual(d['volumes'], ['/home/user:/container/path']) + + class MergeVolumesTest(unittest.TestCase): def test_empty(self): service_dict = config.merge_service_dicts({}, {}) diff --git a/tests/unit/service_test.py b/tests/unit/service_test.py index c70c30bfa..24222dfe9 100644 --- a/tests/unit/service_test.py +++ b/tests/unit/service_test.py @@ -1,6 +1,5 @@ from __future__ import unicode_literals from __future__ import absolute_import -import os from .. import unittest import mock @@ -301,18 +300,3 @@ class ServiceVolumesTest(unittest.TestCase): self.assertEqual( binding, ('/outside', dict(bind='/inside', ro=False))) - - @mock.patch.dict(os.environ) - def test_build_volume_binding_with_environ(self): - os.environ['VOLUME_PATH'] = '/opt' - binding = build_volume_binding(parse_volume_spec('${VOLUME_PATH}:/opt')) - self.assertEqual(binding, ('/opt', dict(bind='/opt', ro=False))) - - @mock.patch.dict(os.environ) - def test_building_volume_binding_with_home(self): - os.environ['HOME'] = '/home/user' - binding = build_volume_binding(parse_volume_spec('~:/home/user')) - self.assertEqual( - binding, - ('/home/user', dict(bind='/home/user', ro=False))) - From b6acb3cd8cec598504a4f25a2f91383e71d61701 Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Tue, 14 Apr 2015 11:04:03 -0400 Subject: [PATCH 45/46] Merge pull request #1278 from albers/completion-run-user Add bash completion for docker-compose run --user (cherry picked from commit 3cd116b99d71f0e0da84e77797392e12070734e1) Signed-off-by: Aanand Prasad --- contrib/completion/bash/docker-compose | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contrib/completion/bash/docker-compose b/contrib/completion/bash/docker-compose index af3368036..548773d61 100644 --- a/contrib/completion/bash/docker-compose +++ b/contrib/completion/bash/docker-compose @@ -232,14 +232,14 @@ _docker-compose_run() { compopt -o nospace return ;; - --entrypoint) + --entrypoint|--user|-u) return ;; esac case "$cur" in -*) - COMPREPLY=( $( compgen -W "--allow-insecure-ssl -d --entrypoint -e --no-deps --rm --service-ports -T" -- "$cur" ) ) + COMPREPLY=( $( compgen -W "--allow-insecure-ssl -d --entrypoint -e --no-deps --rm --service-ports -T --user -u" -- "$cur" ) ) ;; *) __docker-compose_services_all From 39ae91c81c2dd8cddd2cbb3601dee8349a596340 Mon Sep 17 00:00:00 2001 From: Aanand Prasad Date: Mon, 23 Mar 2015 10:40:23 -0700 Subject: [PATCH 46/46] Bump 1.2.0 Signed-off-by: Aanand Prasad --- CHANGES.md | 23 +++++++++++++++++++++++ compose/__init__.py | 2 +- docs/completion.md | 2 +- docs/install.md | 2 +- 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index 75c130906..277a188a3 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,6 +1,29 @@ Change log ========== +1.2.0 (2015-04-16) +------------------ + +- `docker-compose.yml` now supports an `extends` option, which enables a service to inherit configuration from another service in another configuration file. This is really good for sharing common configuration between apps, or for configuring the same app for different environments. Here's the [documentation](https://github.com/docker/compose/blob/master/docs/yml.md#extends). + +- When using Compose with a Swarm cluster, containers that depend on one another will be co-scheduled on the same node. This means that most Compose apps will now work out of the box, as long as they don't use `build`. + +- Repeated invocations of `docker-compose up` when using Compose with a Swarm cluster now work reliably. + +- Directories passed to `build`, filenames passed to `env_file` and volume host paths passed to `volumes` are now treated as relative to the *directory of the configuration file*, not the directory that `docker-compose` is being run in. In the majority of cases, those are the same, but if you use the `-f|--file` argument to specify a configuration file in another directory, **this is a breaking change**. + +- A service can now share another service's network namespace with `net: container:`. + +- `volumes_from` and `net: container:` entries are taken into account when resolving dependencies, so `docker-compose up ` will correctly start all dependencies of ``. + +- `docker-compose run` now accepts a `--user` argument to specify a user to run the command as, just like `docker run`. + +- The `up`, `stop` and `restart` commands now accept a `--timeout` (or `-t`) argument to specify how long to wait when attempting to gracefully stop containers, just like `docker stop`. + +- `docker-compose rm` now accepts `-f` as a shorthand for `--force`, just like `docker rm`. + +Thanks, @abesto, @albers, @alunduil, @dnephin, @funkyfuture, @gilclark, @IanVS, @KingsleyKelly, @knutwalker, @thaJeztah and @vmalloc! + 1.1.0 (2015-02-25) ------------------ diff --git a/compose/__init__.py b/compose/__init__.py index c770b3950..2c426c781 100644 --- a/compose/__init__.py +++ b/compose/__init__.py @@ -1,4 +1,4 @@ from __future__ import unicode_literals from .service import Service # noqa:flake8 -__version__ = '1.1.0' +__version__ = '1.2.0' diff --git a/docs/completion.md b/docs/completion.md index d9b94f6cf..6ac95c2ef 100644 --- a/docs/completion.md +++ b/docs/completion.md @@ -17,7 +17,7 @@ On a Mac, install with `brew install bash-completion` Place the completion script in `/etc/bash_completion.d/` (`/usr/local/etc/bash_completion.d/` on a Mac), using e.g. - curl -L https://raw.githubusercontent.com/docker/compose/1.1.0/contrib/completion/bash/docker-compose > /etc/bash_completion.d/docker-compose + curl -L https://raw.githubusercontent.com/docker/compose/1.2.0/contrib/completion/bash/docker-compose > /etc/bash_completion.d/docker-compose Completion will be available upon next login. diff --git a/docs/install.md b/docs/install.md index 0e60e1f18..064ddc5f1 100644 --- a/docs/install.md +++ b/docs/install.md @@ -20,7 +20,7 @@ First, install Docker version 1.3 or greater: To install Compose, run the following commands: - curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose + curl -L https://github.com/docker/compose/releases/download/1.2.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose Optionally, you can also install [command completion](completion.md) for the