diff --git a/docs/source/config_files.rst b/docs/source/config_files.rst index 6fcc5f0c..ceb0ee0f 100644 --- a/docs/source/config_files.rst +++ b/docs/source/config_files.rst @@ -1,9 +1,30 @@ .. _config-files: -Supported configuration files -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Configuration Files +~~~~~~~~~~~~~~~~~~~ -Below is a list of supported configuration files (roughly in the order of build priority). +``repo2docker`` looks for configuration files in the repository being built +to determine how to build it. In general, ``repo2docker`` uses the same +configuration files as other software installation tools, +rather than creating new custom configuration files. + +A number of ``repo2docker`` configuration files can be combined to compose more +complex setups. + +``repo2docker`` will look for configuration files in either: + +* A folder named ``binder/`` in the root of the repository. +* The root directory of the repository. + +If the folder ``binder/`` is located at the top level of the repository, + **only configuration files in the** ``binder/`` **folder will be considered**. + +The `binder examples `_ organization on +GitHub contains a list of sample repositories for common configurations +that ``repo2docker`` can build with various configuration files such as +Python and R installation in a repo. + +Below is a list of supported configuration files (roughly in the order of build priority): .. contents:: :local: @@ -22,14 +43,25 @@ With Dockerfiles, a regular Docker build will be performed. See the `Binder Documentation `_ for best-practices with Dockerfiles. +.. _environment-yml: + ``environment.yml`` ^^^^^^^^^^^^^^^^^^^ -This is a conda environment specification, that lets you install packages with conda. +``environment.yml`` is the standard configuration file used by Anaconda, conda, +and miniconda that lets you install Python packages. You can also install files from pip in your ``environment.yml`` as well. Our example `environment.yml `_ shows how one can specify a conda environment for repo2docker. +You can also specify which Python version to install in your built environment +with ``environment.yml``. By default, ``repo2docker`` **installs +Python 3.6** with your ``environment.yml`` unless you include the version of +Python in the file. ``conda`` supports Python versions 3.6, 3.5, 3.4, and 2.7. +``repo2docker`` support is best with Python 3.6, 3.5, and 2.7. If you include +a Python version in a ``runtime.txt`` file in addition to your +``environment.yml``, your ``runtime.txt`` **will be ignored**. + ``requirements.txt`` ^^^^^^^^^^^^^^^^^^^^ @@ -75,6 +107,8 @@ While one can specify dependencies in ``setup.py``, repo2docker **requires configuration files such as** ``environment.yml`` or ``requirements.txt`` to install dependencies during the build process. +.. _postBuild: + ``postBuild`` ^^^^^^^^^^^^^ @@ -96,6 +130,8 @@ To use python-2.7: add python-2.7 in runtime.txt file. The repository will run in a virtualenv with Python 2 installed. To see a full example repository, visit our `Python2 example `_. +**Python versions in ``runtime.txt`` are ignored when** ``environment.yml`` **is +present in the same folder**. repo2docker uses R libraries pinned to a specific snapshot on `MRAN `_. diff --git a/docs/source/design.md b/docs/source/design.md index 9b075137..6054a64a 100644 --- a/docs/source/design.md +++ b/docs/source/design.md @@ -1,5 +1,15 @@ # Design +The repo2docker buildpacks are inspired by +[Heroku's Build Packs](https://devcenter.heroku.com/articles/buildpacks). +The philosophy for the repo2docker buildpacks includes: + +- using common configuration files for familiar installation and packaging tools +- allowing configuration files to be combined to compose more complex setups +- specifying default locations for configuration files +(repo's root directory or .binder directory) + + When designing `repo2docker` and adding to it in the future, the developers are influenced by two primary use cases. The use cases for `repo2docker` which drive most design decisions are: @@ -9,9 +19,6 @@ The use cases for `repo2docker` which drive most design decisions are: 2. Manual image building and running the image from the command line client, `jupyter-repo2docker`, by users interactively on their workstations -We share our guiding design principles here. This is not an exhaustive -list :) - ## Deterministic output The core of `repo2docker` can be considered a diff --git a/docs/source/faq.md b/docs/source/faq.md index 053b9f35..dd555928 100644 --- a/docs/source/faq.md +++ b/docs/source/faq.md @@ -1,17 +1,15 @@ # Frequently Asked Questions (FAQ) -A collection of frequently asked questions with answers! - -If you have a question & have found an answer, send a PR to add it here! +A collection of frequently asked questions with answers. If you have a question +and have found an answer, send a PR to add it here! ## How should I specify another version of Python 3? -Currently the best way to do this is by using a conda ``environment.yml`` -file and setting the language to whichever version of Python you like. +One can specify a Python version in the ``environment.yml`` file of a repo. ## Can I add executable files to the user's PATH? -Yes! Using a ``postBuild`` file, you can place any files that should be called +Yes! With a :ref:`postBuild` file, you can place any files that should be called from the command line in the folder ``~/.local/``. This folder will be available in a user's PATH, and can be run from the command line (or as a subsequent build step.) @@ -36,4 +34,4 @@ Hence the output of `--debug` can not be built with a normal `docker build -t .` or similar traditional docker command. Check out the [binder-examples](http://github.com/binder-examples/) github -organization for example Dockerfiles you can copy & modify for your own use! +organization for example repositories you can copy & modify for your own use! diff --git a/docs/source/index.rst b/docs/source/index.rst index 7e8eb38d..63aeafbc 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -2,8 +2,11 @@ jupyter-repo2docker =================== **jupyter-repo2docker** is a tool to build, run, and push Docker -images from source code repositories. See the list below for various -ways in which you can use ``repo2docker``. +images from source code repositories. repo2docker fetches a repo +(e.g., from GitHub or other locations) and builds a container image based +on the configuration files found in the repo. It can be used to explore a +repository locally by building and executing the constructed image of the +repository. Please report `Bugs `_, `ask questions `_ or diff --git a/docs/source/install.rst b/docs/source/install.rst index 69647129..78266248 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -3,10 +3,24 @@ Installing ``repo2docker`` ========================== +repo2docker requires Python 3.4 and above on Linux and macOS. See +:ref:`below ` for more information about Windows support. + Prerequisite: docker -------------------- -Install `Docker `_ as it is required to build Docker images. +Install `Docker `_ as it is required +to build Docker images. The +`Community Edition `_, +is available for free. + +Recent versions of Docker are recommended. +The latest version of Docker, ``18.03``, successfully builds repositories from +`binder-examples `_. +The `BinderHub `_ helm chart uses version +``17.11.0-ce-dind``. See the +`helm chart `_ +for more details. Installing with ``pip`` ----------------------- @@ -30,13 +44,38 @@ i.e. if you are contributing back to this project:: That's it! For information on using ``repo2docker``, see :ref:`usage`. -Note about Windows support --------------------------- +.. _windows: -Windows support by ``repo2docker`` is still in the experimental stage. +Windows support +--------------- + +Windows support for ``repo2docker`` is still in the experimental stage. An article about `using Windows and the WSL`_ (Windows Subsytem for Linux or Bash on Windows) provides additional information about Windows and docker. .. _using Windows and the WSL: https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly + +.. _jupyterhub: + +JupyterHub-ready images +----------------------- + +`JupyterHub `_ allows multiple +users to collaborate on a shared Jupyter server. ``repo2docker`` can build +Docker images that can be shared within a JupyterHub deployment. For example, +`mybinder.org `_ uses JupyterHub and ``repo2docker`` +to allow anyone to build a Docker image of a git repository online and +share an executable version of the repository with a URL to the built image. + +To build `JupyterHub `_-ready +Docker images with ``repo2docker``, the version of your JupterHub deployment +must be included in the ``environment.yml`` or ``requiements.txt`` of the +git repositories you build. + +If your instance of JupyterHub uses ``DockerSpawner``, you will need to set its +command to run ``jupyterhub-singleuser`` by adding this line in your +configuration file:: + + c.DockerSpawner.cmd = ['jupyterhub-singleuser'] diff --git a/docs/source/usage.rst b/docs/source/usage.rst index bae9541f..5696f879 100644 --- a/docs/source/usage.rst +++ b/docs/source/usage.rst @@ -3,113 +3,86 @@ Using ``repo2docker`` ===================== -The core functionality of repo2docker is to fetch a repo (e.g., from GitHub or -other locations) and build a container image based on the specifications found in the -repo. Optionally, it can launch a local Jupyter Notebook which you can use to explore it. +`Docker `_ **must be running ** in +order to run ``repo2docker``. For more information on installing +``repo2docker``, see :ref:`install`. -This section describes the general ways in which you can use -``repo2docker``, including: - -.. contents:: - :depth: 1 - :local: - - -See the `Frequently Asked Questions `_ for additional information. - -Preparing your repository -------------------------- - -``repo2docker`` looks for configuration files in the repository being built -to determine how to build it. It is philosophically similar to -`Heroku Build Packs `_. - -In general, ``repo2docker`` uses the same configuration files as other software -installation tools, rather than creating new custom configuration files. -These files are described in :ref:`config-files`. - -``repo2docker`` configuration files are all **composable** - you can use any number -of them in the same repository. - -Locating and composing configuration files -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -``repo2docker`` will look for configuration files located in two places: - -* A folder named ``binder`` in the root of the repository. -* The root of the repository. - -There are a few notable rules for composition precedence and build priority: - -* If the folder ``binder`` is located at the top level of the repository, - **only configuration files in the** ``binder`` **folder will be considered**. -* If a Dockerfile is present, **all other files will be ignored**. -* ``environment.yml`` **takes precedent over** - ``requirements.txt``. If you wish to install ``pip`` packages - with ``environment.yml``, please use the - ``pip:`` key as described in the `conda documentation`_. - -For a list of repositories demonstrating various configurations, see -`binder examples `_. - -Preparing a repo to build JupyterHub-ready images -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It is possible to use ``repo2docker`` to build JupyterHub-ready -Docker images. For this to work properly, the version of the ``jupyterhub`` -package in your git repository must match the version in your JupyterHub -deployment. For example, if your JupyterHub deployment runs ``jupyterhub==0.8``, -you should put the following in ``requirements.txt`` or ``environment.yml``:: - - jupyterhub==0.8.* - -If your instance of JupyterHub uses ``DockerSpawner``, you will need to set its -command to run ``jupyterhub-singleuser`` by adding this line in your configuration file:: - - c.DockerSpawner.cmd = ['jupyterhub-singleuser'] - -Running ``repo2docker`` locally -------------------------------- - -`Docker `_ **must be running on your machine** in order to build images - with ``repo2docker``. - For more information on installing ``repo2docker``, see :ref:`install`. - - -The simplest invocation of ``repo2docker`` performs two steps: +``repo2docker`` performs two steps: 1. builds a Docker image from a git repo -2. runs a Jupyter server within the image +2. runs a Jupyter server within the image to explore the repo -This two step process enables you to build an image and run it so you can -explore the repository's contents. +To ensure you can run the software in your repository, you must -The **command** used is:: +repo2docker is called with this command:: jupyter-repo2docker -where ```` provides a URL or path to the source repository. +where ```` is a URL or path to the source repository. -For example, use the following to build an image and launch a Jupyter Notebook -server:: +For example, use the following to build an image of the +`Python Data Scinece Handbook `_:: jupyter-repo2docker https://github.com/jakevdp/PythonDataScienceHandbook -When the example completes building (which may take a few minutes), a message will -be output to your terminal:: +To build a particular branch and commit, use the argument ``--ref`` to +specify the ``branch-name/commit-hash``:: + + jupyter-repo2docker https://github.com/jakevdp/PythonDataScienceHandbook --ref master/599aa0fe3f882c0001670e676e5a8d43b92c35fc + +..tip:: + For reproducibile research, we recommend specifying a commit-hash to + deterministcally build a fixed version of a repository. Not specifying a + commit-hash will result in the latest commit of the repository being built. + +Building the image may take a few minutes. + +During building, ``repo2docker`` +clones the repository to obtain its contents and inspects the repo for +:ref:`configuration files `. + +By default, ``repo2docker`` will assume you are using +Python 3.6 unless you include the version of Python in your +:ref:`configuration files `. ``repo2docker`` support is best with +Python 2.7, 3.5, and 3.6. In the case of this repo, a Python version is not +specified in their configuation files and Python 3.6 is installed. + +`Python Data Scinece Handbook `_ +uses a `requirements.txt file `_ +to specify its Python environment. ``repo2docker`` uses ``pip`` to install +dependencies listed in the ``requirement.txt`` in the image. To learn more about +configuration files in ``repo2docker`` visit :ref:`config-files`. + +When the image is built, a message will be output to your terminal:: Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://0.0.0.0:36511/?token=f94f8fabb92e22f5bfab116c382b4707fc2cade56ad1ace0 -If you copy/paste that URL into your browser you will see a Jupyter Notebook with the -contents of the source repository which you have just built. +Pasting the URL into your browser will open Jupyter Notebook with the +dependencies and contents of the source repository in the built image. -Using the ``--debug`` and ``--no-build`` parameters ---------------------------------------------------- +Because JupyterLab is a server extension of the classic Jupyter Notebook server, +you can launch JupyterLab by opening Jupyter Notebook and visiting the +```/lab`` to the end of the URL: -If you want to debug and understand the details of the docker image being built, -you can pass the ``debug`` parameter to the commandline: +.. code-block:: none + +http(s):////lab + +To switch back to the classic notebook, add ``/tree`` to the URL: + +.. code-block:: none + +http(s):////tree + +To learn more about URLs in JupyterLab and Jupyter Notebook, visit +`starting JupyterLab `_. + +``--debug`` and ``--no-build`` +------------------------------ + +To debug the docker image being built, pass the ``--debug`` parameter: .. code-block:: bash @@ -125,22 +98,3 @@ be used by docker directly. .. code-block:: bash jupyter-repo2docker --no-build --debug https://github.com/jakevdp/PythonDataScienceHandbook - -Setting environment variables ------------------------------ - -If you want to define environment variables, you can pass the ``--env`` or ``-e`` parameter to the commandline: - - .. code-block:: bash - - jupyter-repo2docker -e VAR1=val1 -e VAR2=val2 ... - -Accessing help from the command line ------------------------------------- - -For a list of all the build configurations at your disposal, see the -CLI help:: - - jupyter-repo2docker -h - -.. _conda documentation: https://conda.io/docs/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually