Merge 4ae375e6ae into d2f467cef6

[pre-commit.ci] pre-commit autoupdate

.pre-commit-config.yaml

@@ -11,7 +11,7 @@
 repos:
   # Autoformat: Python code, syntax patterns are modernized
   - repo: https://github.com/asottile/pyupgrade
-    rev: v3.15.0
+    rev: v3.15.2
     hooks:
       - id: pyupgrade
         args:
@@ -23,7 +23,7 @@ repos:
 
   # Autoformat: Python code
   - repo: https://github.com/psf/black
-    rev: 24.1.1
+    rev: 24.3.0
     hooks:
       - id: black
         args:

docs/source/changelog.md

@@ -1,5 +1,63 @@
 # Changelog
 
+## Version 2024.03.0
+
+([full changelog](https://github.com/jupyterhub/repo2docker/compare/2023.06.0...2024.01.03))
+
+### New features added
+
+- Implement support for dockerignore and containerignore [#1205](https://github.com/jupyterhub/repo2docker/pull/1205) ([@sgaist](https://github.com/sgaist))
+
+### Enhancements made
+
+- rstudio: log-level info to stderr [#1317](https://github.com/jupyterhub/repo2docker/pull/1317) ([@manics](https://github.com/manics))
+
+### Bugs fixed
+
+- Get Zenodo working again [#1315](https://github.com/jupyterhub/repo2docker/pull/1315) ([@manics](https://github.com/manics))
+
+### Maintenance and upkeep improvements
+
+- Update mamba to 1.5.7 from 1.5.1, and conda to 24.3.0 from 23.7.4 [#1337](https://github.com/jupyterhub/repo2docker/pull/1337) ([@SylvainCorlay](https://github.com/SylvainCorlay))
+- Dockerfile: bump alpine from 3.17 to 3.19 and Python 3.10 to 3.11 [#1332](https://github.com/jupyterhub/repo2docker/pull/1332) ([@yuvipanda](https://github.com/yuvipanda))
+- Upgrade base image from to Ubuntu 22.04 from 18.04 [#1287](https://github.com/jupyterhub/repo2docker/pull/1287) ([@yuvipanda](https://github.com/yuvipanda))
+
+### Documentation improvements
+
+- Add Ubuntu 22.04 upgrade guide [#1309](https://github.com/jupyterhub/repo2docker/pull/1309) ([@manics](https://github.com/manics))
+- Update version of R available [#1288](https://github.com/jupyterhub/repo2docker/pull/1288) ([@yuvipanda](https://github.com/yuvipanda))
+- Add changelog for 2023.06.0 [#1286](https://github.com/jupyterhub/repo2docker/pull/1286) ([@yuvipanda](https://github.com/yuvipanda))
+
+### Other merged PRs
+
+- [pre-commit.ci] pre-commit autoupdate [#1333](https://github.com/jupyterhub/repo2docker/pull/1333) ([@pre-commit-ci](https://github.com/pre-commit-ci))
+- build(deps): bump codecov/codecov-action from 3 to 4 [#1331](https://github.com/jupyterhub/repo2docker/pull/1331) ([@dependabot](https://github.com/dependabot))
+- Support pytest=8 [#1330](https://github.com/jupyterhub/repo2docker/pull/1330) ([@manics](https://github.com/manics))
+- Update versioneer [#1329](https://github.com/jupyterhub/repo2docker/pull/1329) ([@TimoRoth](https://github.com/TimoRoth))
+- build(deps): bump actions/setup-python from 4 to 5 [#1328](https://github.com/jupyterhub/repo2docker/pull/1328) ([@dependabot](https://github.com/dependabot))
+- build(deps): bump actions/upload-artifact from 3 to 4 [#1327](https://github.com/jupyterhub/repo2docker/pull/1327) ([@dependabot](https://github.com/dependabot))
+- Add NYCU Dataverse [#1326](https://github.com/jupyterhub/repo2docker/pull/1326) ([@twtw](https://github.com/twtw))
+- [pre-commit.ci] pre-commit autoupdate [#1322](https://github.com/jupyterhub/repo2docker/pull/1322) ([@pre-commit-ci](https://github.com/pre-commit-ci))
+- New domain for Edmond MPG repository [#1321](https://github.com/jupyterhub/repo2docker/pull/1321) ([@haarli](https://github.com/haarli))
+- [pre-commit.ci] pre-commit autoupdate [#1319](https://github.com/jupyterhub/repo2docker/pull/1319) ([@pre-commit-ci](https://github.com/pre-commit-ci))
+- [MRG] docs: Add base_image parameter example. [#1318](https://github.com/jupyterhub/repo2docker/pull/1318) ([@hiroyuki-sato](https://github.com/hiroyuki-sato))
+- Upgrade mamba and refreeze [#1313](https://github.com/jupyterhub/repo2docker/pull/1313) ([@manics](https://github.com/manics))
+- r: Bump version of rsession-proxy [#1310](https://github.com/jupyterhub/repo2docker/pull/1310) ([@yuvipanda](https://github.com/yuvipanda))
+- build(deps): bump actions/checkout from 3 to 4 [#1308](https://github.com/jupyterhub/repo2docker/pull/1308) ([@dependabot](https://github.com/dependabot))
+- build(deps): bump docker/build-push-action from 4 to 5 [#1307](https://github.com/jupyterhub/repo2docker/pull/1307) ([@dependabot](https://github.com/dependabot))
+- build(deps): bump docker/setup-qemu-action from 2 to 3 [#1306](https://github.com/jupyterhub/repo2docker/pull/1306) ([@dependabot](https://github.com/dependabot))
+- build(deps): bump docker/setup-buildx-action from 2 to 3 [#1305](https://github.com/jupyterhub/repo2docker/pull/1305) ([@dependabot](https://github.com/dependabot))
+- Update conda and mamba [#1299](https://github.com/jupyterhub/repo2docker/pull/1299) ([@SylvainCorlay](https://github.com/SylvainCorlay))
+- Point to official documentation for handling JupyterLab workspace [#1294](https://github.com/jupyterhub/repo2docker/pull/1294) ([@fcollonval](https://github.com/fcollonval))
+- Fix rstudio-build selection [#1293](https://github.com/jupyterhub/repo2docker/pull/1293) ([@yamaton](https://github.com/yamaton))
+- [pre-commit.ci] pre-commit autoupdate [#1291](https://github.com/jupyterhub/repo2docker/pull/1291) ([@pre-commit-ci](https://github.com/pre-commit-ci))
+
+### Contributors to this release
+
+([GitHub contributors page for this release](https://github.com/jupyterhub/repo2docker/graphs/contributors?from=2023-06-13&to=2024-03-28&type=c))
+
+[@annakrystalli](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Aannakrystalli+updated%3A2023-06-13..2024-03-28&type=Issues) | [@betatim](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Abetatim+updated%3A2023-06-13..2024-03-28&type=Issues) | [@bollwyvl](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Abollwyvl+updated%3A2023-06-13..2024-03-28&type=Issues) | [@consideRatio](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3AconsideRatio+updated%3A2023-06-13..2024-03-28&type=Issues) | [@dependabot](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Adependabot+updated%3A2023-06-13..2024-03-28&type=Issues) | [@dolfinus](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Adolfinus+updated%3A2023-06-13..2024-03-28&type=Issues) | [@fcollonval](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Afcollonval+updated%3A2023-06-13..2024-03-28&type=Issues) | [@felder](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Afelder+updated%3A2023-06-13..2024-03-28&type=Issues) | [@haarli](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Ahaarli+updated%3A2023-06-13..2024-03-28&type=Issues) | [@hiroyuki-sato](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Ahiroyuki-sato+updated%3A2023-06-13..2024-03-28&type=Issues) | [@manics](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Amanics+updated%3A2023-06-13..2024-03-28&type=Issues) | [@mathieuboudreau](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Amathieuboudreau+updated%3A2023-06-13..2024-03-28&type=Issues) | [@minrk](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Aminrk+updated%3A2023-06-13..2024-03-28&type=Issues) | [@pre-commit-ci](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Apre-commit-ci+updated%3A2023-06-13..2024-03-28&type=Issues) | [@rgaiacs](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Argaiacs+updated%3A2023-06-13..2024-03-28&type=Issues) | [@ryanlovett](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Aryanlovett+updated%3A2023-06-13..2024-03-28&type=Issues) | [@sgaist](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Asgaist+updated%3A2023-06-13..2024-03-28&type=Issues) | [@SylvainCorlay](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3ASylvainCorlay+updated%3A2023-06-13..2024-03-28&type=Issues) | [@TimoRoth](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3ATimoRoth+updated%3A2023-06-13..2024-03-28&type=Issues) | [@twtw](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Atwtw+updated%3A2023-06-13..2024-03-28&type=Issues) | [@welcome](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Awelcome+updated%3A2023-06-13..2024-03-28&type=Issues) | [@yamaton](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Ayamaton+updated%3A2023-06-13..2024-03-28&type=Issues) | [@yuvipanda](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Ayuvipanda+updated%3A2023-06-13..2024-03-28&type=Issues)
+
 ## Version 2023.06.0
 
 ([full changelog](https://github.com/jupyterhub/repo2docker/compare/2022.10.0...2023.06.0))

repo2docker/app.py

@@ -154,6 +154,7 @@ class Repo2Docker(Application):
             contentproviders.Swhid,
             contentproviders.Mercurial,
             contentproviders.Git,
+            contentproviders.Meca,
         ],
         config=True,
         help="""

repo2docker/contentproviders/__init__.py

@@ -3,6 +3,7 @@ from .dataverse import Dataverse
 from .figshare import Figshare
 from .git import Git
 from .hydroshare import Hydroshare
+from .meca import Meca
 from .mercurial import Mercurial
 from .swhid import Swhid
 from .zenodo import Zenodo

repo2docker/contentproviders/meca.py

@@ -0,0 +1,130 @@
+import os
+import shutil
+import tempfile
+import xml.etree.ElementTree as ET
+from hashlib import md5
+from os import path
+from urllib.parse import urlparse, urlunparse
+from zipfile import ZipFile, is_zipfile
+
+from requests import Session
+
+from .base import ContentProvider
+
+
+def get_hashed_slug(url, changes_with_content):
+    """Returns a unique slug that is invariant to query parameters in the url"""
+    parsed_url = urlparse(url)
+    stripped_url = urlunparse(
+        (parsed_url.scheme, parsed_url.netloc, parsed_url.path, "", "", "")
+    )
+
+    return "meca-" + md5(f"{stripped_url}-{changes_with_content}".encode()).hexdigest()
+
+
+def fetch_zipfile(session, url, dst_dir):
+    resp = session.get(url, headers={"accept": "application/zip"}, stream=True)
+    resp.raise_for_status()
+
+    dst_filename = path.join(dst_dir, "meca.zip")
+    with open(dst_filename, "wb") as dst:
+        for chunk in resp.iter_content(chunk_size=128):
+            dst.write(chunk)
+
+    return dst_filename
+
+
+def extract_validate_and_identify_bundle(zip_filename, dst_dir):
+    if not os.path.exists(zip_filename):
+        raise RuntimeError("Downloaded MECA bundle not found")
+
+    if not is_zipfile(zip_filename):
+        raise RuntimeError("MECA bundle is not a zip file")
+
+    with ZipFile(zip_filename, "r") as zip_ref:
+        zip_ref.extractall(dst_dir)
+
+    try:
+        manifest = path.join(dst_dir, "manifest.xml")
+        if not os.path.exists(manifest):
+            raise RuntimeError("MECA bundle is missing manifest.xml")
+        article_source_dir = "bundle/"
+
+        tree = ET.parse(manifest)
+        root = tree.getroot()
+
+        bundle_instance = root.findall(
+            "{*}item[@item-type='article-source-directory']/{*}instance"
+        )
+        for attr in bundle_instance[0].attrib:
+            if attr.endswith("href"):
+                article_source_dir = bundle_instance[0].get(attr)
+
+        return True, path.join(dst_dir, article_source_dir)
+    except:
+        return False, dst_dir
+
+
+class Meca(ContentProvider):
+    """A repo2docker content provider for MECA bundles"""
+
+    def __init__(self):
+        super().__init__()
+        self.session = Session()
+        self.session.headers.update(
+            {
+                "user-agent": f"repo2docker MECA",
+            }
+        )
+
+    def detect(self, spec, ref=None, extra_args=None):
+        """`spec` contains a faux protocol of meca+http[s] for detection purposes
+        and we assume `spec` trusted as a reachable MECA bundle from an allowed origin
+        (binderhub RepoProvider class is already checking for this).
+
+        An other HEAD check in made here in order to get the content-length header
+        """
+        parsed = urlparse(spec)
+        if not parsed.scheme.endswith("+meca"):
+            return None
+        parsed = parsed._replace(scheme=parsed.scheme[:-5])
+        url = urlunparse(parsed)
+
+        headers = self.session.head(url).headers
+        changes_with_content = headers.get("ETag") or headers.get("Content-Length")
+
+        self.hashed_slug = get_hashed_slug(url, changes_with_content)
+
+        return {"url": url, "slug": self.hashed_slug}
+
+    def fetch(self, spec, output_dir, yield_output=False):
+        hashed_slug = spec["slug"]
+        url = spec["url"]
+
+        yield f"Creating temporary directory.\n"
+        with tempfile.TemporaryDirectory() as tmpdir:
+            yield f"Temporary directory created at {tmpdir}.\n"
+
+            yield f"Fetching MECA Bundle {url}.\n"
+            zip_filename = fetch_zipfile(self.session, url, tmpdir)
+
+            yield f"Extracting MECA Bundle {zip_filename}.\n"
+            is_meca, bundle_dir = extract_validate_and_identify_bundle(
+                zip_filename, tmpdir
+            )
+
+            if not is_meca:
+                yield f"This doesn't look like a meca bundle, extracting everything.\n"
+
+            yield f"Copying MECA Bundle at {bundle_dir} to {output_dir}.\n"
+            files = os.listdir(bundle_dir)
+            for f in files:
+                shutil.move(os.path.join(bundle_dir, f), output_dir)
+
+            yield f"Removing temporary directory.\n"
+
+        yield f"MECA Bundle {hashed_slug} fetched and unpacked.\n"
+
+    @property
+    def content_id(self):
+        return self.hashed_slug