ipydrawio/scripts/Drawio_Update.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# drawio file updater\n",
    "\n",
    "The [drawio](https://github.com/jgraph/drawio) distribution, submoduled into `src/drawio` is very large. We need to distribute parts of it, so that they are available on end user's computers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding the files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import re, json, IPython"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "> ### Packaging 0 of 3399 files"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ROOT = Path.cwd().parent\n",
    "SRC =  ROOT / \"src\"\n",
    "DIO = SRC / \"drawio\"\n",
    "DIO_APP = DIO /  \"src\" / \"main\" / \"webapp\"\n",
    "ALL_FILES = sorted(DIO_APP.rglob(\"*.*\"))\n",
    "WANT_FILES = set()\n",
    "\n",
    "def show_files():\n",
    "    IPython.display.display(IPython.display.Markdown(f\"> ### Packaging {len(WANT_FILES)} of {len(ALL_FILES)} files\"))\n",
    "show_files()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The manifest\n",
    "`drawio` can be shipped as a service worker. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "> ### Packaging 109 of 3399 files"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "CACHE_MANIFEST = DIO_APP / \"service-worker.js\"\n",
    "manifest = CACHE_MANIFEST.read_text()\n",
    "WANT_FILES |= {\n",
    "    DIO_APP / p[\"url\"] for p in \n",
    "    json.loads(manifest.split('workbox.precaching.precacheAndRoute(')[1].split(']')[0] + ']')\n",
    "}\n",
    "show_files()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sweep Up\n",
    "This results in a working app, but emits a few files `404`s... let's manually add some files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "> ### Packaging 1195 of 3399 files"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "WANT_FILES |= {\n",
    "    DIO_APP / \"js/PreConfig.js\",\n",
    "    DIO_APP / \"js/PostConfig.js\",\n",
    "    *(DIO_APP / \"math\").rglob(\"*.*\"),\n",
    "    *(DIO_APP / \"mxgraph/images\").rglob(\"*.png\"),\n",
    "    *(DIO_APP / \"mxgraph/images\").rglob(\"*.gif\"),\n",
    "    *(DIO_APP / \"images\").rglob(\"*.svg\"),\n",
    "    *(DIO_APP / \"images\").rglob(\"*.png\"),\n",
    "    *(DIO_APP / \"images\").rglob(\"*.gif\"),\n",
    "}\n",
    "show_files()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## WTF JSON\n",
    "\n",
    "Apparently, JSON files are not so good to use with file-loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "> ### Packaging 1191 of 3399 files"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "WANT_FILES = {wanted for wanted in WANT_FILES if not wanted.name.endswith(\".json\")}\n",
    "show_files()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the files into webpack\n",
    "\n",
    "When launched by JupyterLab, drawio is served in an `iframe`, and all of its assets need to be available locally:\n",
    "\n",
    "- exactly as they were originally named\n",
    "- in a predictable location relative to the main `index.html`\n",
    "\n",
    "This presents some challenges and opportunities:\n",
    "\n",
    "- Unfortunately, this conflicts with the JupyterLab/WebPack _weltanschauung_ of putting everything into a single file and munging the name with hashes\n",
    "- Fortunately, webpack's [`file-loader`](https://webpack.js.org/loaders/file-loader/) gives us a way to just copy a file to a predictable location\n",
    "- Unfortunately, `file-loader` can't handle copying entire folders\n",
    "\n",
    "So we need to: \n",
    "- add the list of files to `package.json`\n",
    "- tell webpack we actually want the files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "PACKAGE_JSON = ROOT / \"package.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our `package.json` needs to ship _our_ files, too!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "BASE_FILES = [\n",
    "    \"lib/**/*.{d.ts,eot,gif,html,jpg,js,js.map,json,png,svg,woff2,ttf,css}\",\n",
    "    \"style/**/*.{css,eot,gif,html,jpg,json,png,svg,woff2,ttf}\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "101380"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package_json = json.loads(PACKAGE_JSON.read_text())\n",
    "package_json[\"files\"] = [*BASE_FILES, *sorted([str(wanted.relative_to(ROOT)) for wanted in WANT_FILES])]\n",
    "PACKAGE_JSON.write_text(json.dumps(package_json, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "COPY_TMPL = \"\"\"import '!!file-loader?name=[path][name].[ext]&context=.!../{}';\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "164569"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "STATIC_TS = SRC / \"_static.ts\"\n",
    "STATIC_TS.write_text(\"\"\"/**\n",
    "    All files that should be copied to the jupyterlab static folder, available as:\n",
    "    \n",
    "    {{:base_url}}static/lab/node_modules/jupyterlab-drawio/src/{{:path}}\n",
    "    \n",
    "    This file generated from https://github.com/jgraph/drawio\n",
    "*/\n",
    "{}\n",
    "\"\"\".format(\"\\n\".join(COPY_TMPL.format(wanted.relative_to(ROOT)) for wanted in sorted(WANT_FILES))))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}