From 01522559f20b1a94618c534a29afa73d11aa99e8 Mon Sep 17 00:00:00 2001 From: yuvipanda Date: Wed, 7 Feb 2018 23:33:57 -0800 Subject: [PATCH 1/3] Add a CONTRIBUTING.md Thanks to @willingc for editing the release process notes --- .travis.yml | 4 +- CONTRIBUTING.md | 139 +++++++++++++++++++++++++++++++++++++++++++ dev-requirements.txt | 3 + 3 files changed, 144 insertions(+), 2 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 dev-requirements.txt diff --git a/.travis.yml b/.travis.yml index 5e5f687c..4f0e026f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -7,12 +7,12 @@ python: install: # Make a wheel and install it to test to catch possible # issues with MANIFEST.in -- pip install --no-cache-dir pyyaml pytest wheel +- pip install --no-cache-dir -r dev-requirements.txt - python setup.py bdist_wheel - pip install dist/*.whl script: - # cd into tests so CWD being repo2docker does not hide + # cd into tests so CWD being repo2docker does not hide # possible issues with MANIFEST.in - cd tests && pytest -s -v ${REPO_TYPE} diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..26a82dff --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,139 @@ +# Local Development + +To develop & test repo2docker locally, you need: + +1. Familiarity with using a command line terminal +2. A computer running macOS / Linux +3. Some knowledge of git +4. At least python 3.4 +5. Your favorite text editor +6. A recent version of [Docker Community Edition](https://www.docker.com/community-edition) + +## Clone the repository + +First, you need to get a copy of the repo2docker git repository on your local +disk. + +```bash +git clone https://github.com/jupyter/repo2docker +``` + +This will clone repo2docker into a directory called `repo2docker`. You can +make that your current directory with `cd repo2docker`. + +## Set up local virtual environment + +After cloning the repository (or your fork of the repo), you should set up an +isolated environment to install libraries required for running / developing +repo2docker. There are many ways to do this, and a `virtual environment` is +one of them. + +```bash +python3 -m venv . +source bin/activate +pip3 install -e . +pip3 install -r dev-requirements.txt +``` + +This should install all the libraries required for testing & running repo2docker! + +## Verify that docker is installed and running + +If you do not already have [Docker](https://www.docker.com/), you should be able +to download and install it for your operating system using the links from the +[official website](https://www.docker.com/community-edition). After you have +installed it, you can verify that it is working by running the following commands: + +```bash +docker version +``` + +It should output something like: + +``` +Client: + Version: 17.09.0-ce + API version: 1.32 + Go version: go1.8.3 + Git commit: afdb6d4 + Built: Tue Sep 26 22:42:45 2017 + OS/Arch: linux/amd64 + +Server: + Version: 17.09.0-ce + API version: 1.32 (minimum version 1.12) + Go version: go1.8.3 + Git commit: afdb6d4 + Built: Tue Sep 26 22:41:24 2017 + OS/Arch: linux/amd64 + Experimental: false +``` + +Then you are good to go! + +## Running tests + +We have a lot of tests for various cases supported by repo2docker in the `tests/` +subdirectory. These use [py.test](https://docs.pytest.org/). + +You can run all the tests with: + +```bash +py.test -s tests/* +``` + +If you want to run a specific test, you can do so with: + +```bash +py.test -s tests/ +``` + +# Release Process + +We try to make a release of repo2docker every few months if possible. + +## Access + +To release repo2docker, you will need proper access credentials prior to beginning the process. + +1. Access to the PyPI package for repo2docker +2. Access to push tags to the jupyter/repo2docker repository +3. Acess to push images to dockerhub on jupyter/repo2docker + +If you do not have access to any of these, please contact a current maintainer of the project! + +## Steps + +1. Make a PR bumping version number of repo2docker in the + `setup.py` file (like https://github.com/jupyter/repo2docker/pull/221), + get it merged, and make sure your local checkout is the + same as `master` on GitHub. + +2. In your environment, install packages needed to make releases: + ```bash + pip install wheel twine + ``` + +3. Clean out the `dist` directory and then build the `wheel` and `tar.gz` files: + ```bash + rm -f dist/* + python setup.py sdist bdist_wheel + ``` +4. Once tests pass, time to upload! + ```bash + twine upload dist/* + ``` + + This might ask for your PyPI username and password. + +5. Make a git tag and push it to GitHub: + ```bash + git tag -a v + git push official --tags + ``` + +6. Tag and push a docker image: + ```bash + docker build -t jupyter/repo2docker:v . + docker push jupyter/repo2docker:v + ``` diff --git a/dev-requirements.txt b/dev-requirements.txt new file mode 100644 index 00000000..4afcffba --- /dev/null +++ b/dev-requirements.txt @@ -0,0 +1,3 @@ +pyyaml +pytest +wheel From b6af51697e0ba97ac1028698a1bb07d053d3c26f Mon Sep 17 00:00:00 2001 From: yuvipanda Date: Thu, 8 Feb 2018 02:53:18 -0800 Subject: [PATCH 2/3] Fix trailing space in .travis.yml --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 4f0e026f..9e89916a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -12,7 +12,7 @@ install: - pip install dist/*.whl script: - # cd into tests so CWD being repo2docker does not hide + # cd into tests so CWD being repo2docker does not hide # possible issues with MANIFEST.in - cd tests && pytest -s -v ${REPO_TYPE} From 51d5c4d0cb560fdc25d619b8f9375d3546f3dd0e Mon Sep 17 00:00:00 2001 From: yuvipanda Date: Thu, 8 Feb 2018 11:15:53 -0800 Subject: [PATCH 3/3] Add docs on freezing dependencies --- CONTRIBUTING.md | 72 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 26a82dff..6338ef3e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -88,6 +88,78 @@ If you want to run a specific test, you can do so with: py.test -s tests/ ``` +# Updating libraries installed for all repos + +For both the conda and virtualenv base environments, we install specific +pinned versions of all dependencies. We explicitly list the dependencies +we want, then *freeze* them at commit time to explicitly list all the +transitive dependencies at current versions. This way, we know that +all dependencies will have the exact same version installed at all times. + +If you update a library in the base environment, you need to update it +for both the virtualenv and conda environments before submitting a PR. + +## Virtualenv dependencies + +There are two files related to virtualenv dependencies: + +1. `repo2docker/buildpacks/python/requirements.txt` + + Contains list of packages to install in Python3 virtualenvs, + which are the default. This should be where all Notebook versions & + notebook extensions (such as JupyterLab / nteract) go. + +2. `repo2docker/buildpacks/python/requirements2.txt` + + Contains list of packages to install in Python2 virtualenvs, which + can be specifically requested by users. This only needs `IPyKernel` + and kernel related libraries - Notebook / Notebook Extension need + not be installed here. + +Once you edit either of these files to add a new package / bump version on +an existing package, you should then run `./repo2docker/buildpacks/python/freeze.bash`. +This script will resolve dependencies and write them to the respective `.frozen.txt` +files. You will need Python3 and Python2 with virtualenv to run the script. + +All the .txt files in `repo2docker/buildpacks/python/` should be comitted to git. + +## Conda dependencies + +There are two files related to conda dependencies: + +1. `repo2docker/buildpacks/conda/environment.yml` + + Contains list of packages to install in Python3 conda environments, + which are the default. This should be where all Notebook versions & + notebook extensions (such as JupyterLab / nteract) go. + +2. `repo2docker/buildpacks/conda/environment.py-2.7.yml` + + Contains list of packages to install in Python2 conda environments, which + can be specifically requested by users. This only needs `IPyKernel` + and kernel related libraries - Notebook / Notebook Extension need + not be installed here. + +Once you edit either of these files to add a new package / bump version on +an existing package, you should then run `./repo2docker/buildpacks/conda/freeze.py`. +This script will resolve dependencies and write them to the respective `.frozen.yml` +files. You will need `docker` installed to run this script. + +The following files should be comitted to git: + +- repo2docker/buildpacks/conda/environment.frozen.yml +- repo2docker/buildpacks/conda/environment.py-2.7.yml +- repo2docker/buildpacks/conda/environment.py-3.6.frozen.yml +- repo2docker/buildpacks/conda/environment.py-2.7.frozen.yml +- repo2docker/buildpacks/conda/environment.py-3.5.frozen.yml +- repo2docker/buildpacks/conda/environment.yml + +## Make a Pull Request + +Once you've made the commit, please make a Pull Request to the `jupyter/repo2docker` +repository, with a description of what versions were bumped / what new packages were +added and why. + # Release Process We try to make a release of repo2docker every few months if possible.