Merge remote-tracking branch 'upstream/master' into hydroshare-content-provider

pull/800/head
Scott Black 2020-01-29 10:14:01 -07:00
commit 02d0107510
39 zmienionych plików z 225 dodań i 697 usunięć

Wyświetl plik

@ -3,6 +3,7 @@
[![Build Status](https://travis-ci.org/jupyter/repo2docker.svg?branch=master)](https://travis-ci.org/jupyter/repo2docker)
[![Documentation Status](https://readthedocs.org/projects/repo2docker/badge/?version=latest)](http://repo2docker.readthedocs.io/en/latest/?badge=latest)
[![Build Status](https://dev.azure.com/jupyter/repo2docker/_apis/build/status/jupyter.repo2docker?branchName=master)](https://dev.azure.com/jupyter/repo2docker/_build/latest?definitionId=1&branchName=master)
[![Contribute](https://img.shields.io/badge/I_want_to_contribute!-grey?logo=jupyter)](https://repo2docker.readthedocs.io/en/latest/contributing/contributing.html)
`repo2docker` fetches a git repository and builds a container image based on
the configuration files found in the repository.

Wyświetl plik

@ -1,7 +1,6 @@
# Note that there is also a Pipfile for this project if you are updating this
# Note that there is also a Pipfile for this project if you are updating this
# file do not forget to update the Pipfile accordingly
sphinx>=1.4, !=1.5.4
recommonmark
alabaster
alabaster_jupyterhub
sphinxcontrib-autoprogram
git+https://github.com/pandas-dev/pandas-sphinx-theme.git@master

Plik binarny nie jest wyświetlany.

Przed

Szerokość:  |  Wysokość:  |  Rozmiar: 1.2 KiB

Po

Szerokość:  |  Wysokość:  |  Rozmiar: 15 KiB

Plik binarny nie jest wyświetlany.

Po

Szerokość:  |  Wysokość:  |  Rozmiar: 13 KiB

Wyświetl plik

@ -1,8 +1,12 @@
# Architecture of repo2docker
# Architecture
This is a living document talking about the architecture of repo2docker
from various perspectives.
```eval_rst
.. _buildpacks:
```
## Buildpacks
The **buildpack** concept comes from [Heroku](https://devcenter.heroku.com/articles/buildpacks)
@ -39,9 +43,9 @@ It takes the following steps to determine this:
libraries can check for presence of an `environment.yml` file and say 'yes, I can handle this
repository' by returning `True`. Usually buildpacks look for presence of specific files
(`requirements.txt`, `environment.yml`, `install.R`, `manifest.xml` etc) to determine if they can handle a
repository or not. Buildpacks may also look into specific files to determine specifics of the
repository or not. Buildpacks may also look into specific files to determine specifics of the
required environment, such as the Stencila integration which extracts the required language-specific
executions contexts from an XML file (see base `BuildPack`). More than one buildpack may use such
executions contexts from an XML file (see base `BuildPack`). More than one buildpack may use such
information, as properties can be inherited (e.g. the R buildpack uses the list of required Stencila
contexts to see if R must be installed).
3. If no `BuildPack` returns true, then repo2docker will use the default `BuildPack` (defined in

Wyświetl plik

@ -19,7 +19,7 @@ API changes
Bug fixes
---------
- Fix for submodule check out in :pr:`809` by :user:`davidbrochart`.
Version 0.10.0

Wyświetl plik

@ -109,7 +109,7 @@ pygments_style = "sphinx"
todo_include_todos = False
# repo2docker Logo
html_logo = "_static/images/repo2docker.svg"
html_logo = "_static/images/logo.png"
html_favicon = "_static/images/favicon.ico"
# -- Options for HTML output ----------------------------------------------
@ -117,10 +117,7 @@ html_favicon = "_static/images/favicon.ico"
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
import alabaster_jupyterhub
html_theme = "alabaster_jupyterhub"
html_theme_path = [alabaster_jupyterhub.get_html_theme_path()]
html_theme = "pandas_sphinx_theme"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
@ -132,9 +129,7 @@ html_theme_path = [alabaster_jupyterhub.get_html_theme_path()]
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ["_static"]
html_sidebars = {
"**": ["globaltoc.html", "relations.html", "sourcelink.html", "searchbox.html"]
}
html_sidebars = {}
# -- Options for HTMLHelp output ------------------------------------------

Wyświetl plik

@ -17,12 +17,8 @@ GitHub contains a list of sample repositories for common configurations
that ``repo2docker`` can build with various configuration files such as
Python and R installation in a repository.
Below is a list of supported configuration files (roughly in the order of build priority):
.. contents::
:local:
:depth: 1
A list of supported configuration files (roughly in the order of build priority)
can be found on this page (and to the right).
.. _environment.yml:

Wyświetl plik

@ -0,0 +1,13 @@
===========================
Configuring your repository
===========================
Information about configuring your repository to work with repo2docker,
and controlling elements of the built environment using configuration files.
.. toctree::
:maxdepth: 2
:caption: Complete list of configuration files
../config_files
../specification

Wyświetl plik

@ -1,8 +1,10 @@
# Adding a new buildpack to repo2docker
# Add a new buildpack
A new buildpack is needed when a new language or a new package manager should be
supported. Existing buildpacks are a good model for how new buildpacks
should be structured.
supported. [Existing buildpacks](https://github.com/jupyter/repo2docker/tree/master/repo2docker/buildpacks)
are a good model for how new buildpacks should be structured.
See [the Buildpacks page](buildpacks) for more information about the
structure of a buildpack.
## Criteria to balance and consider
@ -17,7 +19,7 @@ Criteria to balance are:
this using one of the escape hatches in (2), so let us make it easy and add
native support".
## Adding libraries or UI to existing buildpacks
### Adding libraries or UI to existing buildpacks
Note that this doesn't apply to adding additional libraries / UI to existing
buildpacks. For example, if we had an R buildpack and it supported IRKernel,

Wyświetl plik

@ -0,0 +1,18 @@
==========================
Add a new content provider
==========================
Adding a new content provider allows repo2docker to grab repositories from new
locations on the internet. To do so, you should take the following steps:
#. Sub-class the `ContentProvider class <https://github.com/jupyter/repo2docker/blob/master/repo2docker/contentproviders/base.py#L17>`_.
This will give you a skeleton class you can modify to support your new
content provider.
#. Implement a **detect()** method for the class. This takes an input
string (e.g., a URL or path) and determines if it points to this particular
content provider. It should return a dictionary (called
``spec`` that will be passed to the ``fetch()`` method. `For example, see the ZenodoProvider detect method <https://github.com/jupyter/repo2docker/pull/693/files#diff-a96fcf624176b06e21c3ef7f6f6a425bR31>`_.
#. Implement a **fetch()** method for the class. This takes the dictionary ``spec`` as input, and
ensures the repository exists on disk (e.g., by downloading it) and
returns a path to it.
`For example, see the ZenodoProvider fetch method <https://github.com/jupyter/repo2docker/pull/693/files#diff-a96fcf624176b06e21c3ef7f6f6a425bR57>`_.

Wyświetl plik

@ -1,4 +1,4 @@
# Contributing to repo2docker development
# Contributing to repo2docker
Thank you for thinking about contributing to repo2docker!
This is an open source project that is developed and maintained entirely by volunteers.

Wyświetl plik

@ -0,0 +1,18 @@
============
Contributing
============
The repo2docker community is welcoming of all kinds of help and
participation from others. Below are a few ways that you can get involved,
as well as resources for understanding the structure and design of the
repo2docker package.
.. toctree::
contributing
roadmap
../architecture
../design
tasks
buildpack
contentprovider

Wyświetl plik

@ -1,4 +1,4 @@
# The repo2docker roadmap
# Roadmap
This roadmap collects "next steps" for repo2docker. It is about creating a
shared understanding of the project's vision and direction amongst

Wyświetl plik

@ -49,10 +49,8 @@ Many ingredients go into making an image from a repository:
1. versions of the libraries installed by the repository
`repo2docker` controls the first two, the user controls the third one. The current
policy for the version of the base image is that we will keep pace with Ubuntu
releases until we reach the next release with Long Term Support (LTS). We
currently use Artful Aardvark (17.10) and the next LTS version will be
Bionic Beaver (18.04).
policy for the version of the base image is that we will use the current LTS
version Bionic Beaver (18.04) for the foreseeable future.
The version of `repo2docker` used to build an image can influence which packages
are installed by default and which features are supported during the build
@ -73,7 +71,7 @@ that's separated out cleanly. This allows easy use by other projects (like
BinderHub).
There is additional (and very useful) design advice on this in
the [Art of Unix Programming](http://www.faqs.org/docs/artu/ch01s06.html) which
the [Art of Unix Programming](https://web.archive.org/web/20190921131144/http://www.faqs.org/docs/artu/ch01s06.html) which
is a highly recommended quick read.
## Composability

Wyświetl plik

@ -172,7 +172,7 @@ possible (this is why repo2docker buildpacks largely build off of patterns that
exist in the data analytics community). We try to perform due-diligence and search for
other communities to leverage and help, but sometimes it makes the most sense to build
our own new tool. In the case of repo2docker, we spent time integrating with a pre-existing
tool called [source2image](https://github.com/openshift/source-to-image).
tool called `source2image <https://github.com/openshift/source-to-image/>`_.
This is an excellent open tool for containerization, but we
ultimately decided that it did not fit the use-case we wanted to address. For more information,
[here's a short blog post about the decision and the reasoning behind it](https://github.com/yuvipanda/words/blob/fd096dd49d87e624acd8bdf6d13c0cecb930bb3f/content/post/why-not-s2i.md).
`here <https://github.com/yuvipanda/words/blob/fd096dd49d87e624acd8bdf6d13c0cecb930bb3f/content/post/why-not-s2i.md>`_ is a short blog post about the decision and the reasoning behind it.

Wyświetl plik

@ -0,0 +1,13 @@
===============
Getting Started
===============
Instructions and information on how to get started with repo2docker
on your own machine. Select from the pages listed below to begin.
.. toctree::
:maxdepth: 2
../install
../usage
../faq

Wyświetl plik

@ -0,0 +1,17 @@
=============
How-to Guides
=============
Short, actionable guides that cover specific topics with repo2docker.
Select from the pages listed below to get started.
.. toctree::
:maxdepth: 2
:caption: How-To guides
user_interface
languages
export_environment
lab_workspaces
jupyterhub_images
deploy

Wyświetl plik

@ -25,38 +25,15 @@ Please report `Bugs <https://github.com/jupyter/repo2docker/issues>`_,
:maxdepth: 2
:caption: Getting started with repo2docker
install
usage
faq
.. toctree::
:maxdepth: 2
:caption: How-To guides
howto/user_interface
howto/languages
howto/export_environment
howto/lab_workspaces
howto/jupyterhub_images
howto/deploy
.. toctree::
:maxdepth: 2
:caption: Complete list of configuration files
config_files
specification
getting-started/index
howto/index
configuration/index
.. toctree::
:maxdepth: 2
:caption: Contribute to repo2docker
contributing/contributing
contributing/roadmap
architecture
design
contributing/tasks
contributing/buildpack
contributing/index
.. toctree::
:maxdepth: 2

Wyświetl plik

@ -42,10 +42,17 @@ ARG NB_UID
ENV USER ${NB_USER}
ENV HOME /home/${NB_USER}
RUN adduser --disabled-password \
--gecos "Default user" \
--uid ${NB_UID} \
${NB_USER}
RUN groupadd \
--gid ${NB_UID} \
${NB_USER} && \
useradd \
--comment "Default user" \
--create-home \
--gid ${NB_UID} \
--no-log-init \
--shell /bin/bash \
--uid ${NB_UID} \
${NB_USER}
RUN wget --quiet -O - https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \
DISTRO="bionic" && \

Wyświetl plik

@ -4,7 +4,7 @@ set -ex
cd $(dirname $0)
MINICONDA_VERSION=4.7.10
CONDA_VERSION=4.7.10
CONDA_VERSION=4.7.12
# Only MD5 checksums are available for miniconda
# Can be obtained from https://repo.continuum.io/miniconda/
MD5SUM="1c945f2b3335c7b2b15130b1b2dc5cf4"

Wyświetl plik

@ -20,9 +20,12 @@ class JuliaProjectTomlBuildPack(PythonBuildPack):
"1.0.2",
"1.0.3",
"1.0.4",
"1.0.5",
"1.1.0",
"1.1.1",
"1.2.0",
"1.3.0",
"1.3.1",
]
@property

Wyświetl plik

@ -5,130 +5,34 @@ The Dockerfile is amended to add the contents of the repository
to the image and install a supported version of the notebook
and IPython kernel.
Note: This buildpack has been deprecated.
"""
import os
import shutil
from textwrap import dedent
from ..docker import DockerBuildPack
import logging
class LegacyBinderDockerBuildPack(DockerBuildPack):
"""Legacy build pack for compatibility to first version of Binder."""
class LegacyBinderDockerBuildPack:
"""Legacy build pack for compatibility to first version of Binder.
dockerfile = "._binder.Dockerfile"
legacy_prependix = dedent(
r"""
USER root
# update the source list now that jessie is archived
COPY apt-sources.list /etc/apt/sources.list
USER main
COPY python3.frozen.yml /tmp/python3.frozen.yml
COPY root.frozen.yml /tmp/root.frozen.yml
# update conda in two steps because the base image
# has very old conda that can't upgrade past 4.3
RUN conda install -yq conda>=4.3 && \
conda install -yq conda==4.4.11 && \
conda env update -n python3 -f /tmp/python3.frozen.yml && \
conda remove -yq -n python3 nb_conda_kernels _nb_ext_conf && \
conda env update -n root -f /tmp/root.frozen.yml && \
/home/main/anaconda2/envs/python3/bin/ipython kernel install --sys-prefix && \
/home/main/anaconda2/bin/ipython kernel install --prefix=/home/main/anaconda2/envs/python3 && \
/home/main/anaconda2/bin/ipython kernel install --sys-prefix
This buildpack has been deprecated.
"""
)
legacy_appendix = dedent(
r"""
USER root
COPY . /home/main/notebooks
RUN chown -R main:main /home/main/notebooks && \
rm /home/main/notebooks/root.frozen.yml && \
rm /home/main/notebooks/python3.frozen.yml && \
rm /home/main/notebooks/apt-sources.list
USER main
WORKDIR /home/main/notebooks
ENV PATH /home/main/anaconda2/envs/python3/bin:$PATH
ENV JUPYTER_PATH /home/main/anaconda2/share/jupyter:$JUPYTER_PATH
CMD jupyter notebook --ip 0.0.0.0
"""
)
def render(self):
"""Render buildpack into a Dockerfile.
Render legacy image source (andrewosh/binder-base at a specific commit)
and then prependix. Render appendix (post-build commands) at the end of
the Dockerfile.
"""
segments = [
"FROM andrewosh/binder-base@sha256:eabde24f4c55174832ed8795faa40cea62fc9e2a4a9f1ee1444f8a2e4f9710ee",
self.legacy_prependix,
]
with open("Dockerfile") as f:
for line in f:
if line.strip().startswith("FROM"):
break
segments.append(f.read())
segments.append(self.legacy_appendix)
return "\n".join(segments)
def get_build_script_files(self):
"""
Dict of files to be copied to the container image for use in building.
This is copied before the `build_scripts` & `assemble_scripts` are
run, so can be executed from either of them.
It's a dictionary where the key is the source file path in the host
system, and the value is the destination file path inside the
container image.
This currently adds a frozen set of Python requirements to the dict
of files.
"""
return {
"legacy/root.frozen.yml": "/tmp/root.frozen.yml",
"legacy/python3.frozen.yml": "/tmp/python3.frozen.yml",
"legacy/apt-sources.list": "/tmp/apt-sources.list",
}
def build(
self,
client,
image_spec,
memory_limit,
build_args,
cache_from,
extra_build_kwargs,
):
"""Build a legacy Docker image."""
with open(self.dockerfile, "w") as f:
f.write(self.render())
for env in ("root", "python3"):
env_file = env + ".frozen.yml"
src_path = os.path.join(os.path.dirname(__file__), env_file)
shutil.copy(src_path, env_file)
src_path = os.path.join(os.path.dirname(__file__), "apt-sources.list")
shutil.copy(src_path, "apt-sources.list")
return super().build(
client, image_spec, memory_limit, build_args, cache_from, extra_build_kwargs
)
def detect(self):
"""Check if current repo should be built with the Legacy BuildPack.
"""
"""Check if current repo should be built with the Legacy BuildPack."""
log = logging.getLogger("repo2docker")
try:
with open("Dockerfile", "r") as f:
for line in f:
if line.startswith("FROM"):
if "andrewosh/binder-base" in line.split("#")[0].lower():
return True
log.error(
"The legacy buildpack was removed in January 2020."
)
log.error(
"Please see https://repo2docker.readthedocs.io/en/"
"latest/configuration/index.html for alternative ways "
"of configuring your repository."
)
raise RuntimeError("The legacy buildpack has been removed.")
else:
return False
except FileNotFoundError:

Wyświetl plik

@ -1,2 +0,0 @@
deb http://archive.debian.org/debian jessie main
deb http://security.debian.org jessie/updates main

Wyświetl plik

@ -1,71 +0,0 @@
#!/usr/bin/env python3
"""
Freeze the conda environment.yml for legacy dockerfiles.
It runs the freeze in the andrewosh/binder-base image used for legacy dockerfiles.
Usage:
python freeze.py [3.5]
"""
from datetime import datetime
import os
import pathlib
import shutil
from subprocess import check_call
import sys
# need conda ≥ 4.4 to avoid bug adding spurious pip dependencies
CONDA_VERSION = "4.4.11"
HERE = pathlib.Path(os.path.dirname(os.path.abspath(__file__)))
def freeze(env_name, env_file, frozen_file):
"""Freeze a conda environment.yml
By running in docker:
conda env create
conda env export
Result will be stored in frozen_file
"""
print(f"Freezing {env_file} -> {frozen_file}")
with open(HERE / frozen_file, "w") as f:
f.write(f"# AUTO GENERATED FROM {env_file}, DO NOT MANUALLY MODIFY\n")
f.write(f"# Frozen on {datetime.utcnow():%Y-%m-%d %H:%M:%S UTC}\n")
check_call(
[
"docker",
"run",
"--rm",
"-v" f"{HERE}:/r2d",
"-it",
f"andrewosh/binder-base",
"sh",
"-c",
"; ".join(
[
"conda update -yq conda",
f"conda install -yq conda={CONDA_VERSION}",
"conda config --system --set auto_update_conda false",
f"conda env update -f /r2d/{env_file} -n {env_name}",
# exclude conda packages because we don't want to pin them
f"conda env export -n {env_name} | grep -v conda >> /r2d/{frozen_file}",
]
),
]
)
if __name__ == "__main__":
# allow specifying which env(s) to update on argv
env_names = sys.argv[1:] or ("root", "python3")
for env_name in env_names:
env_file = env_name + ".yml"
frozen_file = os.path.splitext(env_file)[0] + ".frozen.yml"
freeze(env_name, env_file, frozen_file)

Wyświetl plik

@ -1,197 +0,0 @@
# AUTO GENERATED FROM python3.yml, DO NOT MANUALLY MODIFY
# Frozen on 2018-03-20 11:41:06 UTC
name: python3
channels:
- defaults
dependencies:
- _license=1.1=py35_1
- _nb_ext_conf=0.3.0=py35_0
- alabaster=0.7.9=py35_0
- argcomplete=1.0.0=py35_1
- astroid=1.4.7=py35_0
- astropy=1.2.1=np111py35_0
- babel=2.3.4=py35_0
- backports=1.0=py35_0
- beautifulsoup4=4.5.1=py35_0
- bitarray=0.8.1=py35_0
- blaze=0.10.1=py35_0
- bokeh=0.12.2=py35_0
- boto=2.42.0=py35_0
- bottleneck=1.1.0=np111py35_0
- cairo=1.12.18=6
- cffi=1.7.0=py35_0
- chest=0.2.3=py35_0
- click=6.6=py35_0
- cloudpickle=0.2.1=py35_0
- clyent=1.2.2=py35_0
- colorama=0.3.7=py35_0
- configobj=5.0.6=py35_0
- contextlib2=0.5.3=py35_0
- cryptography=1.5=py35_0
- curl=7.49.0=1
- cycler=0.10.0=py35_0
- cython=0.24.1=py35_0
- cytoolz=0.8.0=py35_0
- dask=0.11.0=py35_0
- datashape=0.5.2=py35_0
- dbus=1.10.10=0
- decorator=4.0.10=py35_0
- dill=0.2.5=py35_0
- docutils=0.12=py35_2
- dynd-python=0.7.2=py35_0
- entrypoints=0.2.2=py35_0
- et_xmlfile=1.0.1=py35_0
- expat=2.1.0=0
- fastcache=1.0.2=py35_1
- filelock=2.0.6=py35_0
- flask=0.11.1=py35_0
- flask-cors=2.1.2=py35_0
- fontconfig=2.11.1=6
- freetype=2.5.5=1
- get_terminal_size=1.0.0=py35_0
- gevent=1.1.2=py35_0
- glib=2.43.0=1
- greenlet=0.4.10=py35_0
- gst-plugins-base=1.8.0=0
- gstreamer=1.8.0=0
- h5py=2.6.0=np111py35_2
- harfbuzz=0.9.39=1
- hdf5=1.8.17=1
- heapdict=1.0.0=py35_1
- icu=54.1=0
- idna=2.1=py35_0
- imagesize=0.7.1=py35_0
- ipykernel=4.5.0=py35_0
- ipython=5.1.0=py35_0
- ipython_genutils=0.1.0=py35_0
- ipywidgets=5.2.2=py35_0
- itsdangerous=0.24=py35_0
- jbig=2.1=0
- jdcal=1.2=py35_1
- jedi=0.9.0=py35_1
- jinja2=2.8=py35_1
- jpeg=8d=2
- jsonschema=2.5.1=py35_0
- jupyter=1.0.0=py35_3
- jupyter_client=4.4.0=py35_0
- jupyter_console=5.0.0=py35_0
- jupyter_core=4.2.0=py35_0
- lazy-object-proxy=1.2.1=py35_0
- libdynd=0.7.2=0
- libffi=3.2.1=0
- libgcc=4.8.5=2
- libgfortran=3.0.0=1
- libpng=1.6.22=0
- libsodium=1.0.10=0
- libtiff=4.0.6=2
- libxcb=1.12=0
- libxml2=2.9.2=0
- libxslt=1.1.28=0
- llvmlite=0.13.0=py35_0
- locket=0.2.0=py35_1
- lxml=3.6.4=py35_0
- markupsafe=0.23=py35_2
- matplotlib=1.5.3=np111py35_0
- mistune=0.7.3=py35_0
- mkl=11.3.3=0
- mkl-service=1.1.2=py35_2
- mpmath=0.19=py35_1
- multipledispatch=0.4.8=py35_0
- nbconvert=4.2.0=py35_0
- nbformat=4.1.0=py35_0
- nbpresent=3.0.2=py35_0
- networkx=1.11=py35_0
- nltk=3.2.1=py35_0
- nose=1.3.7=py35_1
- notebook=4.2.3=py35_0
- numba=0.28.1=np111py35_0
- numexpr=2.6.1=np111py35_0
- numpy=1.11.1=py35_0
- odo=0.5.0=py35_1
- openpyxl=2.3.2=py35_0
- openssl=1.0.2j=0
- pandas=0.18.1=np111py35_0
- partd=0.3.6=py35_0
- patchelf=0.9=0
- path.py=8.2.1=py35_0
- pathlib2=2.1.0=py35_0
- patsy=0.4.1=py35_0
- pep8=1.7.0=py35_0
- pexpect=4.0.1=py35_0
- pickleshare=0.7.4=py35_0
- pillow=3.3.1=py35_0
- pip=8.1.2=py35_0
- pixman=0.32.6=0
- pkginfo=1.3.2=py35_0
- ply=3.9=py35_0
- prompt_toolkit=1.0.3=py35_0
- psutil=4.3.1=py35_0
- ptyprocess=0.5.1=py35_0
- py=1.4.31=py35_0
- pyasn1=0.1.9=py35_0
- pycosat=0.6.1=py35_1
- pycparser=2.14=py35_1
- pycrypto=2.6.1=py35_4
- pycurl=7.43.0=py35_0
- pyflakes=1.3.0=py35_0
- pygments=2.1.3=py35_0
- pylint=1.5.4=py35_1
- pyopenssl=16.0.0=py35_0
- pyparsing=2.1.4=py35_0
- pyqt=5.6.0=py35_0
- pytables=3.2.3.1=np111py35_0
- pytest=2.9.2=py35_0
- python=3.5.2=0
- python-dateutil=2.5.3=py35_0
- pytz=2016.6.1=py35_0
- pyyaml=3.12=py35_0
- pyzmq=15.4.0=py35_0
- qt=5.6.0=0
- qtawesome=0.3.3=py35_0
- qtconsole=4.2.1=py35_1
- qtpy=1.1.2=py35_0
- readline=6.2=2
- redis=3.2.0=0
- redis-py=2.10.5=py35_0
- requests=2.11.1=py35_0
- rope=0.9.4=py35_1
- scikit-image=0.12.3=np111py35_1
- scikit-learn=0.17.1=np111py35_2
- scipy=0.18.1=np111py35_0
- setuptools=27.2.0=py35_0
- simplegeneric=0.8.1=py35_1
- singledispatch=3.4.0.3=py35_0
- sip=4.18=py35_0
- six=1.10.0=py35_0
- snowballstemmer=1.2.1=py35_0
- sockjs-tornado=1.0.3=py35_0
- sphinx=1.4.6=py35_0
- spyder=3.0.0=py35_0
- sqlalchemy=1.0.13=py35_0
- sqlite=3.13.0=0
- statsmodels=0.6.1=np111py35_1
- sympy=1.0=py35_0
- terminado=0.6=py35_0
- tk=8.5.18=0
- toolz=0.8.0=py35_0
- tornado=4.4.1=py35_0
- traitlets=4.3.0=py35_0
- unicodecsv=0.14.1=py35_0
- wcwidth=0.1.7=py35_0
- werkzeug=0.11.11=py35_0
- wheel=0.29.0=py35_0
- widgetsnbextension=1.2.6=py35_0
- wrapt=1.10.6=py35_0
- xlrd=1.0.0=py35_0
- xlsxwriter=0.9.3=py35_0
- xlwt=1.1.2=py35_0
- xz=5.2.2=0
- yaml=0.1.6=0
- zeromq=4.1.4=0
- zlib=1.2.8=3
- pip:
- backports.shutil-get-terminal-size==1.0.0
- dynd==0.7.3.dev1
- rope-py3k==0.9.4.post1
- tables==3.2.3.1

Wyświetl plik

@ -1,4 +0,0 @@
dependencies:
- notebook==5.4.1
- ipykernel==4.8.2
- tornado<5

Wyświetl plik

@ -1,183 +0,0 @@
# AUTO GENERATED FROM root.yml, DO NOT MANUALLY MODIFY
# Frozen on 2018-03-20 11:39:45 UTC
name: root
channels:
- defaults
dependencies:
- alabaster=0.7.7=py27_0
- argcomplete=1.0.0=py27_1
- astropy=1.1.2=np110py27_0
- babel=2.2.0=py27_0
- backports_abc=0.4=py27_0
- beautifulsoup4=4.4.1=py27_0
- bitarray=0.8.1=py27_0
- blaze=0.9.1=py27_0
- bokeh=0.11.1=py27_0
- boto=2.39.0=py27_0
- bottleneck=1.0.0=np110py27_0
- ca-certificates=2017.08.26=h1d4fec5_0
- cairo=1.12.18=6
- cdecimal=2.3=py27_0
- certifi=2018.1.18=py27_0
- cffi=1.5.2=py27_0
- chest=0.2.3=py27_0
- cloudpickle=0.1.1=py27_0
- clyent=1.2.1=py27_0
- colorama=0.3.7=py27_0
- configobj=5.0.6=py27_0
- cryptography=1.4=py27_0
- curl=7.45.0=0
- cycler=0.10.0=py27_0
- cython=0.23.4=py27_0
- cytoolz=0.7.5=py27_0
- dask=0.8.1=py27_0
- datashape=0.5.1=py27_0
- decorator=4.0.9=py27_0
- dill=0.2.4=py27_0
- docutils=0.12=py27_0
- dynd-python=0.7.2=py27_0
- enum34=1.1.2=py27_0
- et_xmlfile=1.0.1=py27_0
- fastcache=1.0.2=py27_0
- flask=0.10.1=py27_1
- flask-cors=2.1.2=py27_0
- fontconfig=2.11.1=5
- freetype=2.5.5=0
- funcsigs=0.4=py27_0
- futures=3.0.3=py27_0
- gevent=1.1.0=py27_0
- greenlet=0.4.9=py27_0
- grin=1.2.1=py27_1
- h5py=2.5.0=np110py27_4
- hdf5=1.8.15.1=2
- heapdict=1.0.0=py27_0
- idna=2.0=py27_0
- ipaddress=1.0.14=py27_0
- ipykernel=4.8.2=py27_0
- ipython=4.1.2=py27_1
- ipython_genutils=0.1.0=py27_0
- ipywidgets=4.1.1=py27_0
- itsdangerous=0.24=py27_0
- jbig=2.1=0
- jdcal=1.2=py27_0
- jedi=0.9.0=py27_0
- jinja2=2.8=py27_0
- jpeg=8d=0
- jsonschema=2.4.0=py27_0
- jupyter=1.0.0=py27_2
- jupyter_client=4.2.2=py27_0
- jupyter_console=4.1.1=py27_0
- jupyter_core=4.1.0=py27_0
- libdynd=0.7.2=0
- libffi=3.0.13=0
- libgcc-ng=7.2.0=hdf63c60_3
- libgfortran=3.0=0
- libpng=1.6.17=0
- libsodium=1.0.3=0
- libtiff=4.0.6=1
- libxml2=2.9.2=0
- libxslt=1.1.28=0
- llvmlite=0.9.0=py27_0
- locket=0.2.0=py27_0
- lxml=3.6.0=py27_0
- markupsafe=0.23=py27_0
- matplotlib=1.5.1=np110py27_0
- mistune=0.7.2=py27_0
- mkl=11.3.1=0
- mkl-service=1.1.2=py27_0
- mpmath=0.19=py27_0
- multipledispatch=0.4.8=py27_0
- nbconvert=4.1.0=py27_0
- nbformat=4.0.1=py27_0
- networkx=1.11=py27_0
- nltk=3.2=py27_0
- nose=1.3.7=py27_0
- notebook=4.1.0=py27_1
- numba=0.24.0=np110py27_0
- numexpr=2.5=np110py27_0
- numpy=1.10.4=py27_1
- odo=0.4.2=py27_0
- openpyxl=2.3.2=py27_0
- openssl=1.0.2n=hb7f436b_0
- pandas=0.18.0=np110py27_0
- partd=0.3.2=py27_1
- patchelf=0.8=0
- path.py=8.1.2=py27_1
- patsy=0.4.0=np110py27_0
- pep8=1.7.0=py27_0
- pexpect=4.0.1=py27_0
- pickleshare=0.5=py27_0
- pillow=3.1.1=py27_0
- pip=8.1.1=py27_1
- pixman=0.32.6=0
- ply=3.8=py27_0
- psutil=4.1.0=py27_0
- ptyprocess=0.5=py27_0
- py=1.4.31=py27_0
- pyasn1=0.1.9=py27_0
- pycairo=1.10.0=py27_0
- pycosat=0.6.3=py27ha4109ae_0
- pycparser=2.14=py27_0
- pycrypto=2.6.1=py27_0
- pycurl=7.19.5.3=py27_0
- pyflakes=1.1.0=py27_0
- pygments=2.1.1=py27_0
- pyopenssl=16.2.0=py27_0
- pyparsing=2.0.3=py27_0
- pyqt=4.11.4=py27_1
- pytables=3.2.2=np110py27_1
- pytest=2.8.5=py27_0
- python=2.7.13=0
- python-dateutil=2.5.1=py27_0
- pytz=2016.2=py27_0
- pyyaml=3.11=py27_1
- pyzmq=15.2.0=py27_0
- qt=4.8.7=1
- qtawesome=0.3.2=py27_0
- qtconsole=4.2.0=py27_0
- qtpy=1.0=py27_0
- readline=6.2=2
- redis=2.6.9=0
- redis-py=2.10.3=py27_0
- requests=2.12.4=py27_0
- rope=0.9.4=py27_1
- ruamel_yaml=0.11.14=py27_1
- scikit-image=0.12.3=np110py27_0
- scikit-learn=0.17.1=np110py27_0
- scipy=0.17.0=np110py27_2
- setuptools=20.3=py27_0
- simplegeneric=0.8.1=py27_0
- singledispatch=3.4.0.3=py27_0
- sip=4.16.9=py27_0
- six=1.10.0=py27_0
- snowballstemmer=1.2.1=py27_0
- sockjs-tornado=1.0.1=py27_0
- sphinx=1.3.5=py27_0
- sphinx_rtd_theme=0.1.9=py27_0
- spyder=2.3.8=py27_1
- sqlalchemy=1.0.12=py27_0
- sqlite=3.13.0=0
- ssl_match_hostname=3.4.0.2=py27_0
- statsmodels=0.6.1=np110py27_0
- sympy=1.0=py27_0
- terminado=0.5=py27_1
- tk=8.5.18=0
- toolz=0.7.4=py27_0
- tornado=4.5.3=py27_0
- traitlets=4.2.1=py27_0
- unicodecsv=0.14.1=py27_0
- util-linux=2.21=0
- werkzeug=0.11.4=py27_0
- wheel=0.29.0=py27_0
- xlrd=0.9.4=py27_0
- xlsxwriter=0.8.4=py27_0
- xlwt=1.0.0=py27_0
- xz=5.0.5=1
- yaml=0.1.6=0
- zeromq=4.1.3=0
- zlib=1.2.8=0
- pip:
- backports.ssl-match-hostname==3.4.0.2
- dynd==0.7.3.dev1
- tables==3.2.2

Wyświetl plik

@ -1,3 +0,0 @@
dependencies:
- ipykernel==4.8.2
- tornado<5

Wyświetl plik

@ -40,6 +40,8 @@ class RBuildPack(PythonBuildPack):
The `r-base` package from Ubuntu apt repositories is used to install
R itself, rather than any of the methods from https://cran.r-project.org/.
The `r-base-dev` package is installed as advised in RStudio instructions.
"""
@property
@ -178,6 +180,8 @@ class RBuildPack(PythonBuildPack):
# install from a different PPA
if V(self.r_version) < V("3.5"):
packages.append("r-base")
packages.append("r-base-dev")
packages.append("libclang-dev")
return super().get_packages().union(packages)
@ -201,13 +205,15 @@ class RBuildPack(PythonBuildPack):
We set the snapshot date used to install R libraries from based on the
contents of runtime.txt.
"""
rstudio_url = "https://download2.rstudio.org/rstudio-server-1.1.419-amd64.deb"
# Via https://rstudio.com/products/rstudio/download-server/debian-ubuntu/
rstudio_url = "https://download2.rstudio.org/server/bionic/amd64/rstudio-server-1.2.5001-amd64.deb"
# This is MD5, because that is what RStudio download page provides!
rstudio_checksum = "24cd11f0405d8372b4168fc9956e0386"
rstudio_checksum = "d33881b9ab786c09556c410e7dc477de"
# Via https://www.rstudio.com/products/shiny/download-server/
shiny_url = "https://download3.rstudio.org/ubuntu-14.04/x86_64/shiny-server-1.5.7.907-amd64.deb"
shiny_checksum = "78371a8361ba0e7fec44edd2b8e425ac"
shiny_url = "https://download3.rstudio.org/ubuntu-14.04/x86_64/shiny-server-1.5.12.933-amd64.deb"
shiny_checksum = "9aeef6613e7f58f21c97a4600921340e"
# Version of MRAN to pull devtools from.
devtools_version = "2018-02-01"
@ -242,12 +248,15 @@ class RBuildPack(PythonBuildPack):
"root",
r"""
apt-get update && \
apt-get install --yes r-base={} && \
apt-get install --yes r-base={R_version} \
r-base-dev={R_version} \
r-recommended={R_version} \
libclang-dev && \
apt-get -qq purge && \
apt-get -qq clean && \
rm -rf /var/lib/apt/lists/*
""".format(
self.r_version
R_version=self.r_version
),
),
]

Wyświetl plik

@ -21,15 +21,23 @@ class Git(ContentProvider):
# make a, possibly shallow, clone of the remote repository
try:
cmd = ["git", "clone", "--recursive"]
cmd = ["git", "clone"]
if ref is None:
# check out of HEAD is performed after the clone is complete
cmd.extend(["--depth", "1"])
else:
# don't check out HEAD, the given ref will be checked out later
# this prevents HEAD's submodules to be cloned if ref doesn't have them
cmd.extend(["--no-checkout"])
cmd.extend([repo, output_dir])
for line in execute_cmd(cmd, capture=yield_output):
yield line
except subprocess.CalledProcessError as e:
msg = "Failed to clone repository from {repo}.".format(repo=repo)
msg = "Failed to clone repository from {repo}".format(repo=repo)
if ref is not None:
msg += " (ref {ref})".format(ref=ref)
msg += "."
raise ContentProviderException(msg) from e
# check out the specific ref given by the user
@ -40,7 +48,9 @@ class Git(ContentProvider):
"Failed to check out ref %s", ref, extra=dict(phase="failed")
)
raise ValueError("Failed to check out ref {}".format(ref))
# If the hash is resolved above, we should be able to reset to it
# We don't need to explicitly checkout things as the reset will
# take of that. If the hash is resolved above, we should be
# able to reset to it
for line in execute_cmd(
["git", "reset", "--hard", hash], cwd=output_dir, capture=yield_output
):

Wyświetl plik

@ -5,6 +5,6 @@ from subprocess import check_output
assert sys.version_info[:2] == (3, 5), sys.version
out = check_output(["conda", "--version"]).decode("utf8").strip()
assert out == "conda 4.7.10", out
assert out == "conda 4.7.12", out
import numpy

Wyświetl plik

@ -145,6 +145,10 @@ def repo_with_submodule():
submod_sha1_b = _get_sha1(git_b_dir)
_add_content_to_git(git_b_dir)
# create a new branch in the parent without any submodule
subprocess.check_call(
["git", "checkout", "-b", "branch-without-submod"], cwd=git_a_dir
)
# create a new branch in the parent to add the submodule
subprocess.check_call(
["git", "checkout", "-b", "branch-with-submod"], cwd=git_a_dir

Wyświetl plik

@ -1,14 +0,0 @@
FROM andrewosh/binder-base
USER root
# Add Julia dependencies
RUN apt-get update
RUN apt-get install -y julia libnettle4 && apt-get clean
USER main
# Install Julia kernel
RUN julia -e 'Pkg.add("IJulia")'
ADD verify verify

Wyświetl plik

@ -1,8 +0,0 @@
Docker - Legacy Dockerfiles
---------------------------
This demonstrates the Dockerfile syntax that was often found in the first
version of Binder. It sources the ``andrewosh`` Docker image, which
contained many different dependencies, then installs Julia. We encourage
users to source one of the Jupyter base images as they are more streamlined,
reliable, and efficient.

Wyświetl plik

@ -1,7 +0,0 @@
#!/usr/bin/env python
import os
import sys
assert sys.version_info[:2] == (3, 5), sys.version
import jupyter

Wyświetl plik

@ -23,6 +23,18 @@ def test_submodule_clone(repo_with_submodule):
"""Test git clone containing a git submodule."""
upstream, expected_sha1_upstream, expected_sha1_submod = repo_with_submodule
# check that checking out a branch where there are no submodule
# indeed doesn't get any submodule, even though they are in master
with TemporaryDirectory() as clone_dir2:
submod_dir = os.path.join(clone_dir2, "submod") # set by fixture
spec = {"repo": upstream, "ref": "branch-without-submod"}
git_content = Git()
for _ in git_content.fetch(spec, clone_dir2):
pass
assert os.path.exists(os.path.join(clone_dir2, "test"))
assert not os.path.exists(os.path.join(submod_dir, "test"))
with TemporaryDirectory() as clone_dir:
submod_dir = os.path.join(clone_dir, "submod") # set by fixture
spec = {"repo": upstream}

Wyświetl plik

@ -0,0 +1,37 @@
from os.path import join as pjoin
import pytest
from tempfile import TemporaryDirectory
from repo2docker.buildpacks import LegacyBinderDockerBuildPack
from repo2docker.utils import chdir
def test_legacy_raises():
# check legacy buildpack raises on a repo that triggers it
with TemporaryDirectory() as repodir:
with open(pjoin(repodir, "Dockerfile"), "w") as d:
d.write("FROM andrewosh/binder-base")
with chdir(repodir):
bp = LegacyBinderDockerBuildPack()
with pytest.raises(RuntimeError):
bp.detect()
def test_legacy_doesnt_detect():
# check legacy buildpack doesn't trigger
with TemporaryDirectory() as repodir:
with open(pjoin(repodir, "Dockerfile"), "w") as d:
d.write("FROM andrewosh/some-image")
with chdir(repodir):
bp = LegacyBinderDockerBuildPack()
assert not bp.detect()
def test_legacy_on_repo_without_dockerfile():
# check legacy buildpack doesn't trigger on a repo w/o Dockerfile
with TemporaryDirectory() as repodir:
with chdir(repodir):
bp = LegacyBinderDockerBuildPack()
assert not bp.detect()

Wyświetl plik

@ -50,23 +50,3 @@ def test_cache_from_docker(tmpdir):
called_args, called_kwargs = fake_client.build.call_args
assert "cache_from" in called_kwargs
assert called_kwargs["cache_from"] == cache_from
def test_cache_from_legacy(tmpdir):
cache_from = ["image-1:latest"]
fake_log_value = {"stream": "fake"}
fake_client = MagicMock(spec=docker.APIClient)
fake_client.build.return_value = iter([fake_log_value])
extra_build_kwargs = {"somekey": "somevalue"}
# Test legacy docker image
with tmpdir.join("Dockerfile").open("w") as f:
f.write("FROM andrewosh/binder-base\n")
for line in LegacyBinderDockerBuildPack().build(
fake_client, "image-2", 100, {}, cache_from, extra_build_kwargs
):
assert line == fake_log_value
called_args, called_kwargs = fake_client.build.call_args
assert "cache_from" in called_kwargs
assert called_kwargs["cache_from"] == cache_from