{ "cells": [ { "cell_type": "markdown", "id": "0f682c4f-2262-44d1-a17c-56427fd9e204", "metadata": {}, "source": [ "# Notebook to Diagram\n", "\n", "While the drag-and-drop behavior of MIME outputs works well, it is untenable at scale, and doesn't handle some features very well." ] }, { "cell_type": "markdown", "id": "18249538-b224-4d78-8f46-4f2841cb21f6", "metadata": {}, "source": [ "> # A Diagram Notebook\n", "\n", "> There _might_ be a compelling reason to store a drawio diagram in a notebook. This has one." ] }, { "cell_type": "code", "execution_count": null, "id": "46854767-3eef-4b17-9cf2-df639ca47ae1", "metadata": {}, "outputs": [], "source": [ "import pandas\n", "\n", "%config InlineBackend.figure_formats = ['svg']" ] }, { "cell_type": "code", "execution_count": null, "id": "279402d4-649c-4e18-9834-1d8b829fee84", "metadata": {}, "outputs": [], "source": [ "pandas.util.testing.makeDataFrame().plot();" ] }, { "cell_type": "code", "execution_count": null, "id": "1025d74e-d652-4e82-899f-c9923cc72688", "metadata": {}, "outputs": [], "source": [ "import copy\n", "import difflib\n", "import html\n", "import os\n", "from pathlib import Path\n", "from uuid import uuid4\n", "\n", "import black\n", "import bleach\n", "import IPython" ] }, { "cell_type": "code", "execution_count": 1, "id": "16e420c3-0402-4c19-9bb7-dd9922cbcd16", "metadata": {}, "outputs": [], "source": [ "import lxml.etree as ET" ] }, { "cell_type": "markdown", "id": "abf613e2-b543-4882-b8b1-f4be51b9c8a2", "metadata": {}, "source": [ "## What XML Library?\n", "\n", "`xml.etree` is part of the standard library, but `lxml.etree` is faster at scale. Let's prefer the latter, and offers some nice API improvements." ] }, { "cell_type": "code", "execution_count": null, "id": "34298b4b-f24d-4b14-908c-4c0482fd4ec5", "metadata": {}, "outputs": [], "source": [ "import nbformat\n", "import requests" ] }, { "cell_type": "markdown", "id": "c4f881b0-333a-40b9-8ca3-970263cbf629", "metadata": {}, "source": [ "# A Notebook\n", "\n", "Using this notebook is as good as any." ] }, { "cell_type": "code", "execution_count": null, "id": "921cb55f-7e53-40a0-a7d3-b710b15abb52", "metadata": {}, "outputs": [], "source": [ "from ipydrawio_widgets import Diagram\n", "from lxml.builder import E\n", "from nbconvert.exporters.html import TemplateExporter\n", "from nbconvert.filters import markdown2html_mistune\n", "from pygments import highlight\n", "from pygments.formatters import HtmlFormatter\n", "from pygments.lexers import get_lexer_by_name\n", "\n", "this_notebook = nbformat.reads(\n", " Path(os.environ.get(\"NOTEBOOK\", \"Notebook to Diagram.ipynb\")).read_text(\n", " encoding=\"utf-8\",\n", " ),\n", " as_version=4,\n", ")\n", "this_notebook.cells[0]" ] }, { "cell_type": "markdown", "id": "ef847468-7ab7-4c76-80b3-ed6bccf96b34", "metadata": {}, "source": [ "## Exporter\n", "\n", "We want to tie into the `nbconvert` pipeline pretty directly. We'll do it in-line for now." ] }, { "cell_type": "code", "execution_count": null, "id": "81f64be0-5e61-481c-9950-ec488762a2d9", "metadata": {}, "outputs": [], "source": [ "from ipydrawio.constants import A_SHORT_DRAWIO\n", "\n", "A_SHORT_DRAWIO" ] }, { "cell_type": "code", "execution_count": null, "id": "d1b716a4-3cdd-4a12-be25-ae52abbca012", "metadata": {}, "outputs": [], "source": [ "def a_style(**kwargs):\n", " return {\"style\": \"\".join([f\"{k}={v};\" for k, v in kwargs.items()])}" ] }, { "cell_type": "code", "execution_count": null, "id": "1de60028-0755-4b95-a0e9-5a0a065a464a", "metadata": {}, "outputs": [], "source": [ "wikimedia = \"https://tools-static.wmflabs.org/fontcdn/css?family=Architects+Daughter\"" ] }, { "cell_type": "code", "execution_count": null, "id": "2b7fbf0c-45c9-45d2-8c83-3955d955578c", "metadata": {}, "outputs": [], "source": [ "def a_note_style(**kwargs):\n", " _defaults_ = kwargs.pop(\n", " \"_defaults_\",\n", " {\n", " \"shape\": \"note\",\n", " \"backgroundOutline\": 1,\n", " \"darkOpacity\": 0.05,\n", " \"fillColor\": \"#FFF9B2\",\n", " \"strokeColor\": \"none\",\n", " \"fillStyle\": \"solid\",\n", " \"direction\": \"west\",\n", " \"gradientDirection\": \"north\",\n", " \"gradientColor\": \"#FFF2A1\",\n", " \"sketch\": 1,\n", " \"shadow\": 1,\n", " \"size\": 20,\n", " \"fontSize\": 24,\n", " \"jiggle\": 2,\n", " \"pointerEvents\": 1,\n", " \"hachureGap\": 4,\n", " \"whiteSpace\": \"wrap\",\n", " \"fontFamily\": \"Architects Daughter\",\n", " \"fontSource\": \"https%3A%2F%2Ftools-static.wmflabs.org%2Ffontcdn%2Fcss%3Ffamily%3DArchitects%2BDaughter\",\n", " },\n", " )\n", " style = dict(**_defaults_)\n", " style.update(**kwargs)\n", " return a_style(**style)" ] }, { "cell_type": "code", "execution_count": null, "id": "9776b2d7-b1b6-4796-97a7-8bd1ec5ed00a", "metadata": {}, "outputs": [], "source": [ "def a_note_geometry(**kwargs):\n", " _defaults_ = kwargs.get(\n", " \"_defaults_\",\n", " dict(x=\"0\", y=\"0\", width=\"500\", height=\"300\", **{\"as\": \"geometry\"}),\n", " )\n", " mx_kwargs = dict(_defaults_)\n", " mx_kwargs.pop(\"defaults\", None)\n", " mx_kwargs.update(**kwargs)\n", " return E.mxGeometry(**mx_kwargs)" ] }, { "cell_type": "code", "execution_count": null, "id": "686748a1-b11d-466f-b576-2b91cdc90a0e", "metadata": {}, "outputs": [], "source": [ "def a_note(value=None, geometry=None, vertex=\"1\", parent=\"1\", **kwargs):\n", " value = value or kwargs.pop(\"value\", None)\n", " _defaults_ = kwargs.pop(\n", " \"_defaults_\",\n", " {\n", " \"id\": f\"{uuid4()}\",\n", " \"value\": value,\n", " },\n", " )\n", " mx_kwargs = dict(_defaults_)\n", " mx_kwargs.update(**kwargs)\n", " style = mx_kwargs.pop(\"style\", {})\n", " if isinstance(style, dict):\n", " mx_kwargs.update(a_note_style(**style))\n", " mx_kwargs.update(\n", " vertex=kwargs.get(\"vertex\", \"1\"),\n", " parent=kwargs.get(\"parent\", \"1\"),\n", " )\n", " geometry = a_note_geometry(**(geometry or {}))\n", " return E.mxCell(geometry, **mx_kwargs)" ] }, { "cell_type": "code", "execution_count": null, "id": "65c28612-2cec-4200-a125-f5333658718f", "metadata": {}, "outputs": [], "source": [ "A_CARD = (\n", " \"\"\"\n", "\n", " \n", "\"\"\"\n", ").strip()" ] }, { "cell_type": "code", "execution_count": null, "id": "7ed371b0-5c47-4178-8438-cafde963629c", "metadata": {}, "outputs": [], "source": [ "def test_a_note():\n", " note = a_note(value=\"CARD A\", id=\"64\")\n", " xml = ET.tostring(note, encoding=str, pretty_print=True)\n", " lines = [xml.splitlines(), A_CARD.splitlines()]\n", " diff = \"\\n\".join([*difflib.unified_diff(*lines)])\n", " if diff:\n", " htmldiff = difflib.HtmlDiff()\n", " IPython.display.display(IPython.display.HTML(htmldiff.make_file(*lines)))\n", " assert not diff, diff" ] }, { "cell_type": "code", "execution_count": null, "id": "dc81e433-40bb-419d-adac-92772c5cdbb4", "metadata": {}, "outputs": [], "source": [ "test_a_note()" ] }, { "cell_type": "code", "execution_count": null, "id": "a910f0d5-119f-4c0d-8a93-2954706b266e", "metadata": {}, "outputs": [], "source": [ "ET.tostring(a_note(\"A Note\"))" ] }, { "cell_type": "code", "execution_count": null, "id": "94cdc65f-a1db-42b1-8005-c4e402a4c0d2", "metadata": {}, "outputs": [], "source": [ "class MXTools:\n", " page_height = 850\n", " page_width = 1100\n", "\n", " def empty_mx(self):\n", " return E.mxfile(\n", " E.diagram(\n", " E.mxGraphModel(\n", " E.root(\n", " E.mxCell(id=\"0\"),\n", " E.mxCell(id=\"1\", parent=\"0\"),\n", " ),\n", " dx=\"1450\",\n", " dy=\"467\",\n", " grid=\"1\",\n", " gridSize=\"10\",\n", " guides=\"1\",\n", " tooltips=\"1\",\n", " connect=\"1\",\n", " arrows=\"1\",\n", " fold=\"1\",\n", " page=\"1\",\n", " pageScale=\"1\",\n", " pageWidth=f\"{self.page_height}\",\n", " pageHeight=f\"{self.page_width}\",\n", " math=\"0\",\n", " shadow=\"0\",\n", " ),\n", " id=\"x\",\n", " name=\"Page-1\",\n", " ),\n", " version=\"14.6.11\",\n", " )\n", "\n", " def style(self, **kwargs):\n", " return {\"style\": \";\".join([f\"{k}={v}\" for k, v in kwargs.items()]) + \";\"}\n", "\n", " @property\n", " def text_style(self):\n", " return self.style(\n", " rounded=0,\n", " verticalAlign=\"top\",\n", " autosize=1,\n", " align=\"left\",\n", " whiteSpace=\"wrap\",\n", " fillColor=\"black\",\n", " strokeColor=\"none\",\n", " html=\"1\",\n", " )\n", "\n", " def svg_style(self, svg):\n", " return self.style(\n", " shape=\"image\",\n", " verticalLabelPosition=\"bottom\",\n", " labelBackgroundColor=\"#ffffff\",\n", " verticalAlign=\"top\",\n", " aspect=\"fixed\",\n", " imageAspect=0,\n", " image=f\"data:image/svg+xml,{svg}\",\n", " )" ] }, { "cell_type": "code", "execution_count": null, "id": "39d3f631-5553-4309-bbc3-f669f1d09e73", "metadata": {}, "outputs": [], "source": [ "DEFAULT_FORMATTER = HtmlFormatter(\n", " noclasses=True,\n", " style=\"monokai\",\n", " wrapcode=True,\n", " cssstyles=\"background: rgba(0,0,0,0.9); padding: 0.1em 0.5em; border-radius: 0.5em\",\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "6f060d3f-91c1-42e8-92f7-a53dde45bb32", "metadata": {}, "outputs": [], "source": [ "class NoteCellExporter(TemplateExporter, MXTools):\n", " export_from_notebook = \"Diagram\"\n", " formatter = DEFAULT_FORMATTER\n", " lexer = get_lexer_by_name(\"ipython\")\n", "\n", " def _file_extension_default(self):\n", " \"\"\"\n", " The new file extension is ``.test_ext``\n", " \"\"\"\n", " return \".dio\"\n", "\n", " def from_notebook_node(self, nb, resources=None, **kw):\n", " nb_copy = copy.deepcopy(nb)\n", " resources = self._init_resources(resources)\n", "\n", " if \"language\" in nb[\"metadata\"]:\n", " resources[\"language\"] = nb[\"metadata\"][\"language\"].lower()\n", "\n", " # Preprocess\n", " nb_copy, resources = self._preprocess(nb_copy, resources)\n", "\n", " tree = self.empty_mx()\n", " tree.xpath(\"//root\")\n", " prev = tree.xpath(\"//mxCell\")[-1]\n", " prev_id = 1\n", " col = 0\n", " row = 0\n", " w = 500\n", " h = 300\n", " p = 25\n", "\n", " def card_geo():\n", " return dict(\n", " x=f\"{col * (w + p)}\",\n", " y=f\"{row * (h + p)}\",\n", " width=f\"{w}\",\n", " height=f\"{h}\",\n", " **{\"as\": \"geometry\"},\n", " )\n", "\n", " for cell in nb_copy[\"cells\"]:\n", " if cell[\"cell_type\"] == \"markdown\":\n", " row += 1\n", " col = 0\n", " mxc = a_note(\n", " value=self.markdown_escaped(cell),\n", " id=f\"{prev_id + 1}\",\n", " style={\n", " \"html\": \"1\",\n", " \"fontFamily\": \"Architects Daughter\",\n", " \"fontSource\": \"https://tools-static.wmflabs.org/fontcdn/css?family=Architects+Daughter\",\n", " },\n", " geometry={\"y\": f\"{row * (h + p)}\", \"x\": f\"{col * (w + p)}\"},\n", " )\n", " prev.addnext(mxc)\n", " prev = mxc\n", " prev_id += 1\n", " col += 1\n", " elif cell[\"cell_type\"] == \"code\":\n", " mxc = E.mxCell(\n", " a_note_geometry(**card_geo()),\n", " id=f\"{prev_id + 1}\",\n", " parent=\"1\",\n", " vertex=\"1\",\n", " value=self.source_escaped(cell),\n", " **self.text_style,\n", " )\n", " prev.addnext(mxc)\n", " prev = mxc\n", " prev_id += 1\n", " col += 1\n", " for output in cell.outputs:\n", " if \"data\" not in output:\n", " continue\n", " svg = output.data.get(\"image/svg+xml\")\n", " if svg:\n", " mxc = E.mxCell(\n", " a_note_geometry(**card_geo()),\n", " id=f\"{prev_id + 1}\",\n", " parent=\"1\",\n", " vertex=\"1\",\n", " value=\"\",\n", " **self.svg_style(self.svg_escaped(svg)),\n", " )\n", " prev.addnext(mxc)\n", " prev = mxc\n", " prev_id += 1\n", " col += 1\n", " return (\n", " ET.tostring(tree, encoding=str, pretty_print=True).replace(\"&\", \"&\"),\n", " resources,\n", " )\n", "\n", " def source_escaped(self, cell):\n", " try:\n", " source = black.format_str(cell.source, mode=black.FileMode(line_length=60))\n", " except:\n", " source = cell.source\n", " return html.escape(highlight(source, self.lexer, self.formatter))\n", "\n", " allowed_tags = {\n", " *bleach.sanitizer.ALLOWED_TAGS,\n", " *{f\"h{i}\" for i in range(7)},\n", " \"div\",\n", " \"pre\",\n", " \"span\",\n", " \"blockquote\",\n", " \"strong\",\n", " } ^ {\"a\"}\n", "\n", " def markdown_escaped(self, cell):\n", " return html.escape(\n", " bleach.clean(\n", " markdown2html_mistune(cell.source),\n", " tags=self.allowed_tags,\n", " strip=True,\n", " ),\n", " )\n", "\n", " def svg_escaped(self, some_data):\n", " return requests.utils.quote(some_data, safe=\"!*()\")\n", "\n", " def svg_size(self, some_data):\n", " width = 100\n", " height = 100\n", " et = ET.fromstring(some_data.encode(\"utf-8\"))\n", " width = et.xpath(\"/@width\")\n", " height = et.xpath(\"/@height\")\n", " return f\"\"\"width=\"{width}\" height=\"{height}\" \"\"\"" ] }, { "cell_type": "code", "execution_count": null, "id": "e83d569e-9e6e-40b8-a652-dcbc1a75a049", "metadata": { "tags": [] }, "outputs": [], "source": [ "export_diagram = NoteCellExporter()\n", "(body, resources) = export_diagram.from_notebook_node(this_notebook)" ] }, { "cell_type": "code", "execution_count": null, "id": "3ef7fb6c-477d-4b44-a8ff-78d3cd681044", "metadata": { "tags": [] }, "outputs": [], "source": [ "diagram = Diagram(layout={\"height\": \"400px\", \"width\": \"100%\"})\n", "new_params = dict(**diagram.url_params)\n", "new_params.update(ui=\"sketch\", format=\"0\")\n", "diagram.url_params = new_params\n", "diagram" ] }, { "cell_type": "code", "execution_count": null, "id": "e0da8b9d-7248-48dd-88ec-012694718e2c", "metadata": {}, "outputs": [], "source": [ "diagram.source.value = body" ] }, { "cell_type": "code", "execution_count": null, "id": "110841eb-5ebe-47f4-8f70-0c15af693e59", "metadata": { "tags": [] }, "outputs": [], "source": [ "print(ET.tostring(ET.fromstring(diagram.source.value), encoding=str, pretty_print=True))" ] }, { "cell_type": "markdown", "id": "b654746b-256b-4761-886d-6a6abecb5a8a", "metadata": {}, "source": [ "## Packaging\n", "\n", "Not appearing here, but our `setup.cfg` should be upgraded to something like:\n", "\n", "\n", "```ini\n", "[options.entry_points]\n", "nbconvert.exporters =\n", " ipydrawio = ipydrawio.exporters:DiagramExporter\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.2" } }, "nbformat": 4, "nbformat_minor": 5 }