Merge branch 'master' into facet-by-comma

.travis.yml

@@ -5,7 +5,7 @@ dist: xenial
 python:
   - "3.6"
   - "3.7"
-  - "3.5"
+  - "3.8"
 
 # Executed for 3.5 AND 3.5 as the first "test" stage:
 script:

README.md

@@ -1,10 +1,12 @@
 # Datasette
 
 [![PyPI](https://img.shields.io/pypi/v/datasette.svg)](https://pypi.org/project/datasette/)
+[![Python 3.x](https://img.shields.io/pypi/pyversions/datasette.svg?logo=python&logoColor=white)](https://pypi.org/project/datasette/)
 [![Travis CI](https://travis-ci.org/simonw/datasette.svg?branch=master)](https://travis-ci.org/simonw/datasette)
 [![Documentation Status](https://readthedocs.org/projects/datasette/badge/?version=latest)](http://datasette.readthedocs.io/en/latest/?badge=latest)
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette/blob/master/LICENSE)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://black.readthedocs.io/en/stable/)
+[![docker: datasette](https://img.shields.io/badge/docker-datasette-blue)](https://hub.docker.com/r/datasetteproject/datasette)
 
 *A tool for exploring and publishing data*
 
@@ -20,6 +22,13 @@ Datasette is aimed at data journalists, museum curators, archivists, local gover
 
 ## News
 
+ * 14th November 2019: [Datasette 0.32](https://datasette.readthedocs.io/en/stable/changelog.html#v0-32) now uses asynchronous rendering in Jinja templates, which means template functions can perform asynchronous operations such as executing SQL queries. [datasette-template-sql](https://github.com/simonw/datasette-template-sql) is a new plugin uses this capability to add a new custom `sql(sql_query)` template function.
+ * 11th November 2019: [Datasette 0.31](https://datasette.readthedocs.io/en/stable/changelog.html#v0-31) - the first version of Datasette to support Python 3.8, which means dropping support for Python 3.5.
+ * 18th October 2019: [Datasette 0.30](https://datasette.readthedocs.io/en/stable/changelog.html#v0-30)
+ * 13th July 2019: [Single sign-on against GitHub using ASGI middleware](https://simonwillison.net/2019/Jul/14/sso-asgi/) talks about the implementation of [datasette-auth-github](https://github.com/simonw/datasette-auth-github) in more detail.
+ * 7th July 2019: [Datasette 0.29](https://datasette.readthedocs.io/en/stable/changelog.html#v0-29) - ASGI, new plugin hooks, facet by date and much, much more...
+   * [datasette-auth-github](https://github.com/simonw/datasette-auth-github) - a new plugin for Datasette 0.29 that lets you require users to authenticate against GitHub before accessing your Datasette instance. You can whitelist specific users, or you can restrict access to members of specific GitHub organizations or teams.
+   * [datasette-cors](https://github.com/simonw/datasette-cors) - a plugin that lets you configure CORS access from a list of domains (or a set of domain wildcards) so you can make JavaScript calls to a Datasette instance from a specific set of other hosts.
  * 23rd June 2019: [Porting Datasette to ASGI, and Turtles all the way down](https://simonwillison.net/2019/Jun/23/datasette-asgi/)
  * 21st May 2019: The anonymized raw data from [the Stack Overflow Developer Survey 2019](https://stackoverflow.blog/2019/05/21/public-data-release-of-stack-overflows-2019-developer-survey/) has been [published in partnership with Glitch](https://glitch.com/culture/discover-insights-explore-developer-survey-results-2019/), powered by Datasette.
  * 19th May 2019: [Datasette 0.28](https://datasette.readthedocs.io/en/stable/changelog.html#v0-28) - a salmagundi of new features!
@@ -63,7 +72,7 @@ sqlite-utils: a Python library and CLI tool for building SQLite databases](https
 
     pip3 install datasette
 
-Datasette requires Python 3.5 or higher. We also have [detailed installation instructions](https://datasette.readthedocs.io/en/stable/installation.html) covering other options such as Docker.
+Datasette requires Python 3.6 or higher. We also have [detailed installation instructions](https://datasette.readthedocs.io/en/stable/installation.html) covering other options such as Docker.
 
 ## Basic usage
 
@@ -83,26 +92,31 @@ Now visiting http://localhost:8001/History/downloads will show you a web interfa
 
 ## datasette serve options
 
-    $ datasette serve --help
-
     Usage: datasette serve [OPTIONS] [FILES]...
 
       Serve up specified SQLite database files with a web UI
 
     Options:
       -i, --immutable PATH      Database files to open in immutable mode
-      -h, --host TEXT           host for server, defaults to 127.0.0.1
-      -p, --port INTEGER        port for server, defaults to 8001
+      -h, --host TEXT           Host for server. Defaults to 127.0.0.1 which means
+                                only connections from the local machine will be
+                                allowed. Use 0.0.0.0 to listen to all IPs and
+                                allow access from other machines.
+      -p, --port INTEGER        Port for server, defaults to 8001
       --debug                   Enable debug mode - useful for development
-      --reload                  Automatically reload if database or code change detected -
-                                useful for development
-      --cors                    Enable CORS by serving Access-Control-Allow-Origin: *
+      --reload                  Automatically reload if database or code change
+                                detected - useful for development
+      --cors                    Enable CORS by serving Access-Control-Allow-
+                                Origin: *
       --load-extension PATH     Path to a SQLite extension to load
-      --inspect-file TEXT       Path to JSON file created using "datasette inspect"
-      -m, --metadata FILENAME   Path to JSON file containing license/source metadata
+      --inspect-file TEXT       Path to JSON file created using "datasette
+                                inspect"
+      -m, --metadata FILENAME   Path to JSON file containing license/source
+                                metadata
       --template-dir DIRECTORY  Path to directory containing custom templates
       --plugins-dir DIRECTORY   Path to directory containing custom plugins
-      --static STATIC MOUNT     mountpoint:path-to-directory for serving static files
+      --static STATIC MOUNT     mountpoint:path-to-directory for serving static
+                                files
       --memory                  Make :memory: database available
       --config CONFIG           Set config option using configname:value
                                 datasette.readthedocs.io/en/latest/config.html

datasette/__main__.py

@@ -0,0 +1,4 @@
+from datasette.cli import cli
+
+if __name__ == "__main__":
+    cli()

datasette/_version.py

@@ -409,7 +409,7 @@ def render_pep440_old(pieces):
 
     The ".dev0" means dirty.
 
-    Eexceptions:
+    Exceptions:
     1: no tags. 0.postDISTANCE[.dev0]
     """
     if pieces["closest-tag"]:

datasette/app.py

@@ -12,6 +12,7 @@ from pathlib import Path
 import click
 from markupsafe import Markup
 from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
+import uvicorn
 
 from .views.base import DatasetteError, ureg, AsgiRouter
 from .views.database import DatabaseDownload, DatabaseView
@@ -23,13 +24,11 @@ from .database import Database
 
 from .utils import (
     QueryInterrupted,
-    Results,
     escape_css_string,
     escape_sqlite,
     get_plugins,
     module_from_path,
     sqlite3,
-    sqlite_timelimit,
     to_css_class,
 )
 from .utils.asgi import (
@@ -41,13 +40,12 @@ from .utils.asgi import (
     asgi_send_json,
     asgi_send_redirect,
 )
-from .tracer import trace, AsgiTracer
+from .tracer import AsgiTracer
 from .plugins import pm, DEFAULT_PLUGINS
 from .version import __version__
 
 app_root = Path(__file__).parent.parent
 
-connections = threading.local()
 MEMORY = object()
 
 ConfigOption = collections.namedtuple("ConfigOption", ("name", "default", "help"))
@@ -159,7 +157,7 @@ class Datasette:
             self.files = [MEMORY]
         elif memory:
             self.files = (MEMORY,) + self.files
-        self.databases = {}
+        self.databases = collections.OrderedDict()
         self.inspect_data = inspect_data
         for file in self.files:
             path = file
@@ -335,6 +333,25 @@ class Datasette:
         # pylint: disable=no-member
         pm.hook.prepare_connection(conn=conn)
 
+    async def execute(
+        self,
+        db_name,
+        sql,
+        params=None,
+        truncate=False,
+        custom_time_limit=None,
+        page_size=None,
+        log_sql_errors=True,
+    ):
+        return await self.databases[db_name].execute(
+            sql,
+            params=params,
+            truncate=truncate,
+            custom_time_limit=custom_time_limit,
+            page_size=page_size,
+            log_sql_errors=log_sql_errors,
+        )
+
     async def expand_foreign_keys(self, database, table, column, values):
         "Returns dict mapping (column, value) -> label"
         labeled_fks = {}
@@ -433,6 +450,7 @@ class Datasette:
             },
             "datasette": datasette_version,
             "asgi": "3.0",
+            "uvicorn": uvicorn.__version__,
             "sqlite": {
                 "version": sqlite_version,
                 "fts_versions": fts_versions,
@@ -457,6 +475,15 @@ class Datasette:
             for p in ps
         ]
 
+    def threads(self):
+        threads = list(threading.enumerate())
+        return {
+            "num_threads": len(threads),
+            "threads": [
+                {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads
+            ],
+        }
+
     def table_metadata(self, database, table):
         "Fetch table-specific metadata."
         return (
@@ -466,85 +493,6 @@ class Datasette:
             .get(table, {})
         )
 
-    async def execute_against_connection_in_thread(self, db_name, fn):
-        def in_thread():
-            conn = getattr(connections, db_name, None)
-            if not conn:
-                db = self.databases[db_name]
-                if db.is_memory:
-                    conn = sqlite3.connect(":memory:")
-                else:
-                    # mode=ro or immutable=1?
-                    if db.is_mutable:
-                        qs = "mode=ro"
-                    else:
-                        qs = "immutable=1"
-                    conn = sqlite3.connect(
-                        "file:{}?{}".format(db.path, qs),
-                        uri=True,
-                        check_same_thread=False,
-                    )
-                self.prepare_connection(conn)
-                setattr(connections, db_name, conn)
-            return fn(conn)
-
-        return await asyncio.get_event_loop().run_in_executor(self.executor, in_thread)
-
-    async def execute(
-        self,
-        db_name,
-        sql,
-        params=None,
-        truncate=False,
-        custom_time_limit=None,
-        page_size=None,
-        log_sql_errors=True,
-    ):
-        """Executes sql against db_name in a thread"""
-        page_size = page_size or self.page_size
-
-        def sql_operation_in_thread(conn):
-            time_limit_ms = self.sql_time_limit_ms
-            if custom_time_limit and custom_time_limit < time_limit_ms:
-                time_limit_ms = custom_time_limit
-
-            with sqlite_timelimit(conn, time_limit_ms):
-                try:
-                    cursor = conn.cursor()
-                    cursor.execute(sql, params or {})
-                    max_returned_rows = self.max_returned_rows
-                    if max_returned_rows == page_size:
-                        max_returned_rows += 1
-                    if max_returned_rows and truncate:
-                        rows = cursor.fetchmany(max_returned_rows + 1)
-                        truncated = len(rows) > max_returned_rows
-                        rows = rows[:max_returned_rows]
-                    else:
-                        rows = cursor.fetchall()
-                        truncated = False
-                except sqlite3.OperationalError as e:
-                    if e.args == ("interrupted",):
-                        raise QueryInterrupted(e, sql, params)
-                    if log_sql_errors:
-                        print(
-                            "ERROR: conn={}, sql = {}, params = {}: {}".format(
-                                conn, repr(sql), params, e
-                            )
-                        )
-                    raise
-
-            if truncate:
-                return Results(rows, truncated, cursor.description)
-
-            else:
-                return Results(rows, False, cursor.description)
-
-        with trace("sql", database=db_name, sql=sql.strip(), params=params):
-            results = await self.execute_against_connection_in_thread(
-                db_name, sql_operation_in_thread
-            )
-        return results
-
     def register_renderers(self):
         """ Register output renderers which output data in custom formats. """
         # Built-in renderers
@@ -585,7 +533,9 @@ class Datasette:
                 ),
             ]
         )
-        self.jinja_env = Environment(loader=template_loader, autoescape=True)
+        self.jinja_env = Environment(
+            loader=template_loader, autoescape=True, enable_async=True
+        )
         self.jinja_env.filters["escape_css_string"] = escape_css_string
         self.jinja_env.filters["quote_plus"] = lambda u: urllib.parse.quote_plus(u)
         self.jinja_env.filters["escape_sqlite"] = escape_sqlite
@@ -616,8 +566,17 @@ class Datasette:
         # Mount any plugin static/ directories
         for plugin in get_plugins(pm):
             if plugin["static_path"]:
-                modpath = "/-/static-plugins/{}/(?P<path>.*)$".format(plugin["name"])
-                add_route(asgi_static(plugin["static_path"]), modpath)
+                add_route(
+                    asgi_static(plugin["static_path"]),
+                    "/-/static-plugins/{}/(?P<path>.*)$".format(plugin["name"]),
+                )
+                # Support underscores in name in addition to hyphens, see https://github.com/simonw/datasette/issues/611
+                add_route(
+                    asgi_static(plugin["static_path"]),
+                    "/-/static-plugins/{}/(?P<path>.*)$".format(
+                        plugin["name"].replace("-", "_")
+                    ),
+                )
         add_route(
             JsonDataView.as_asgi(self, "metadata.json", lambda: self._metadata),
             r"/-/metadata(?P<as_format>(\.json)?)$",
@@ -634,6 +593,10 @@ class Datasette:
             JsonDataView.as_asgi(self, "config.json", lambda: self._config),
             r"/-/config(?P<as_format>(\.json)?)$",
         )
+        add_route(
+            JsonDataView.as_asgi(self, "threads.json", self.threads),
+            r"/-/threads(?P<as_format>(\.json)?)$",
+        )
         add_route(
             JsonDataView.as_asgi(self, "databases.json", self.connected_databases),
             r"/-/databases(?P<as_format>(\.json)?)$",
@@ -719,5 +682,5 @@ class DatasetteRouter(AsgiRouter):
         else:
             template = self.ds.jinja_env.select_template(templates)
             await asgi_send_html(
-                send, template.render(info), status=status, headers=headers
+                send, await template.render_async(info), status=status, headers=headers
             )

datasette/cli.py

@@ -230,9 +230,16 @@ def package(
     multiple=True,
 )
 @click.option(
-    "-h", "--host", default="127.0.0.1", help="host for server, defaults to 127.0.0.1"
+    "-h",
+    "--host",
+    default="127.0.0.1",
+    help=(
+        "Host for server. Defaults to 127.0.0.1 which means only connections "
+        "from the local machine will be allowed. Use 0.0.0.0 to listen to "
+        "all IPs and allow access from other machines."
+    ),
 )
-@click.option("-p", "--port", default=8001, help="port for server, defaults to 8001")
+@click.option("-p", "--port", default=8001, help="Port for server, defaults to 8001")
 @click.option(
     "--debug", is_flag=True, help="Enable debug mode - useful for development"
 )

datasette/database.py

@@ -1,17 +1,25 @@
+import asyncio
+import contextlib
 from pathlib import Path
+import threading
 
+from .tracer import trace
 from .utils import (
     QueryInterrupted,
+    Results,
     detect_fts,
     detect_primary_keys,
     detect_spatialite,
     get_all_foreign_keys,
     get_outbound_foreign_keys,
+    sqlite_timelimit,
     sqlite3,
     table_columns,
 )
 from .inspect import inspect_hash
 
+connections = threading.local()
+
 
 class Database:
     def __init__(self, ds, path=None, is_mutable=False, is_memory=False):
@@ -33,6 +41,85 @@ class Database:
                     for key, value in self.ds.inspect_data[self.name]["tables"].items()
                 }
 
+    def connect(self):
+        if self.is_memory:
+            return sqlite3.connect(":memory:")
+        # mode=ro or immutable=1?
+        if self.is_mutable:
+            qs = "mode=ro"
+        else:
+            qs = "immutable=1"
+        return sqlite3.connect(
+            "file:{}?{}".format(self.path, qs), uri=True, check_same_thread=False
+        )
+
+    async def execute_against_connection_in_thread(self, fn):
+        def in_thread():
+            conn = getattr(connections, self.name, None)
+            if not conn:
+                conn = self.connect()
+                self.ds.prepare_connection(conn)
+                setattr(connections, self.name, conn)
+            return fn(conn)
+
+        return await asyncio.get_event_loop().run_in_executor(
+            self.ds.executor, in_thread
+        )
+
+    async def execute(
+        self,
+        sql,
+        params=None,
+        truncate=False,
+        custom_time_limit=None,
+        page_size=None,
+        log_sql_errors=True,
+    ):
+        """Executes sql against db_name in a thread"""
+        page_size = page_size or self.ds.page_size
+
+        def sql_operation_in_thread(conn):
+            time_limit_ms = self.ds.sql_time_limit_ms
+            if custom_time_limit and custom_time_limit < time_limit_ms:
+                time_limit_ms = custom_time_limit
+
+            with sqlite_timelimit(conn, time_limit_ms):
+                try:
+                    cursor = conn.cursor()
+                    cursor.execute(sql, params or {})
+                    max_returned_rows = self.ds.max_returned_rows
+                    if max_returned_rows == page_size:
+                        max_returned_rows += 1
+                    if max_returned_rows and truncate:
+                        rows = cursor.fetchmany(max_returned_rows + 1)
+                        truncated = len(rows) > max_returned_rows
+                        rows = rows[:max_returned_rows]
+                    else:
+                        rows = cursor.fetchall()
+                        truncated = False
+                except sqlite3.OperationalError as e:
+                    if e.args == ("interrupted",):
+                        raise QueryInterrupted(e, sql, params)
+                    if log_sql_errors:
+                        print(
+                            "ERROR: conn={}, sql = {}, params = {}: {}".format(
+                                conn, repr(sql), params, e
+                            )
+                        )
+                    raise
+
+            if truncate:
+                return Results(rows, truncated, cursor.description)
+
+            else:
+                return Results(rows, False, cursor.description)
+
+        with trace("sql", database=self.name, sql=sql.strip(), params=params):
+            results = await self.execute_against_connection_in_thread(
+                sql_operation_in_thread
+            )
+        return results
+
     @property
     def size(self):
         if self.is_memory:
@@ -50,8 +137,7 @@ class Database:
         for table in await self.table_names():
             try:
                 table_count = (
-                    await self.ds.execute(
-                        self.name,
+                    await self.execute(
                         "select count(*) from [{}]".format(table),
                         custom_time_limit=limit,
                     )
@@ -77,32 +163,30 @@ class Database:
             return Path(self.path).stem
 
     async def table_exists(self, table):
-        results = await self.ds.execute(
-            self.name,
-            "select 1 from sqlite_master where type='table' and name=?",
-            params=(table,),
+        results = await self.execute(
+            "select 1 from sqlite_master where type='table' and name=?", params=(table,)
         )
         return bool(results.rows)
 
     async def table_names(self):
-        results = await self.ds.execute(
-            self.name, "select name from sqlite_master where type='table'"
+        results = await self.execute(
+            "select name from sqlite_master where type='table'"
         )
         return [r[0] for r in results.rows]
 
     async def table_columns(self, table):
-        return await self.ds.execute_against_connection_in_thread(
-            self.name, lambda conn: table_columns(conn, table)
+        return await self.execute_against_connection_in_thread(
+            lambda conn: table_columns(conn, table)
         )
 
     async def primary_keys(self, table):
-        return await self.ds.execute_against_connection_in_thread(
-            self.name, lambda conn: detect_primary_keys(conn, table)
+        return await self.execute_against_connection_in_thread(
+            lambda conn: detect_primary_keys(conn, table)
         )
 
     async def fts_table(self, table):
-        return await self.ds.execute_against_connection_in_thread(
-            self.name, lambda conn: detect_fts(conn, table)
+        return await self.execute_against_connection_in_thread(
+            lambda conn: detect_fts(conn, table)
         )
 
     async def label_column_for_table(self, table):
@@ -112,8 +196,8 @@ class Database:
         if explicit_label_column:
             return explicit_label_column
         # If a table has two columns, one of which is ID, then label_column is the other one
-        column_names = await self.ds.execute_against_connection_in_thread(
-            self.name, lambda conn: table_columns(conn, table)
+        column_names = await self.execute_against_connection_in_thread(
+            lambda conn: table_columns(conn, table)
         )
         # Is there a name or title column?
         name_or_title = [c for c in column_names if c in ("name", "title")]
@@ -129,8 +213,8 @@ class Database:
         return None
 
     async def foreign_keys_for_table(self, table):
-        return await self.ds.execute_against_connection_in_thread(
-            self.name, lambda conn: get_outbound_foreign_keys(conn, table)
+        return await self.execute_against_connection_in_thread(
+            lambda conn: get_outbound_foreign_keys(conn, table)
         )
 
     async def hidden_table_names(self):
@@ -138,18 +222,17 @@ class Database:
         hidden_tables = [
             r[0]
             for r in (
-                await self.ds.execute(
-                    self.name,
+                await self.execute(
                     """
                 select name from sqlite_master
                 where rootpage = 0
                 and sql like '%VIRTUAL TABLE%USING FTS%'
-            """,
+            """
                 )
             ).rows
         ]
-        has_spatialite = await self.ds.execute_against_connection_in_thread(
-            self.name, detect_spatialite
+        has_spatialite = await self.execute_against_connection_in_thread(
+            detect_spatialite
         )
         if has_spatialite:
             # Also hide Spatialite internal tables
@@ -166,13 +249,12 @@ class Database:
             ] + [
                 r[0]
                 for r in (
-                    await self.ds.execute(
-                        self.name,
+                    await self.execute(
                         """
                         select name from sqlite_master
                         where name like "idx_%"
                         and type = "table"
-                    """,
+                    """
                     )
                 ).rows
             ]
@@ -195,32 +277,38 @@ class Database:
         return hidden_tables
 
     async def view_names(self):
-        results = await self.ds.execute(
-            self.name, "select name from sqlite_master where type='view'"
-        )
+        results = await self.execute("select name from sqlite_master where type='view'")
         return [r[0] for r in results.rows]
 
     async def get_all_foreign_keys(self):
-        return await self.ds.execute_against_connection_in_thread(
-            self.name, get_all_foreign_keys
-        )
+        return await self.execute_against_connection_in_thread(get_all_foreign_keys)
 
     async def get_outbound_foreign_keys(self, table):
-        return await self.ds.execute_against_connection_in_thread(
-            self.name, lambda conn: get_outbound_foreign_keys(conn, table)
+        return await self.execute_against_connection_in_thread(
+            lambda conn: get_outbound_foreign_keys(conn, table)
         )
 
     async def get_table_definition(self, table, type_="table"):
         table_definition_rows = list(
-            await self.ds.execute(
-                self.name,
+            await self.execute(
                 "select sql from sqlite_master where name = :n and type=:t",
                 {"n": table, "t": type_},
             )
         )
         if not table_definition_rows:
             return None
-        return table_definition_rows[0][0]
+        bits = [table_definition_rows[0][0] + ";"]
+        # Add on any indexes
+        index_rows = list(
+            await self.ds.execute(
+                self.name,
+                "select sql from sqlite_master where tbl_name = :n and type='index' and sql is not null",
+                {"n": table},
+            )
+        )
+        for index_row in index_rows:
+            bits.append(index_row[0] + ";")
+        return "\n".join(bits)
 
     async def get_view_definition(self, view):
         return await self.get_table_definition(view, "view")

datasette/facets.py

@@ -60,7 +60,7 @@ def load_facet_configs(request, table_metadata):
 
 @hookimpl
 def register_facet_classes():
-    classes = [ColumnFacet, DateFacet, ManyToManyFacet]
+    classes = [ColumnFacet, DateFacet]
     if detect_json1():
         classes.append(ArrayFacet)
     return classes
@@ -271,6 +271,16 @@ class ColumnFacet(Facet):
 class ArrayFacet(Facet):
     type = "array"
 
+    def _is_json_array_of_strings(self, json_string):
+        try:
+            array = json.loads(json_string)
+        except ValueError:
+            return False
+        for item in array:
+            if not isinstance(item, str):
+                return False
+        return True
+
     async def suggest(self):
         columns = await self.get_columns(self.sql, self.params)
         suggested_facets = []
@@ -300,18 +310,37 @@ class ArrayFacet(Facet):
                 )
                 types = tuple(r[0] for r in results.rows)
                 if types in (("array",), ("array", None)):
-                    suggested_facets.append(
-                        {
-                            "name": column,
-                            "type": "array",
-                            "toggle_url": self.ds.absolute_url(
-                                self.request,
-                                path_with_added_args(
-                                    self.request, {"_facet_array": column}
-                                ),
+                    # Now sanity check that first 100 arrays contain only strings
+                    first_100 = [
+                        v[0]
+                        for v in await self.ds.execute(
+                            self.database,
+                            "select {column} from ({sql}) where {column} is not null and json_array_length({column}) > 0 limit 100".format(
+                                column=escape_sqlite(column), sql=self.sql
                             ),
-                        }
-                    )
+                            self.params,
+                            truncate=False,
+                            custom_time_limit=self.ds.config(
+                                "facet_suggest_time_limit_ms"
+                            ),
+                            log_sql_errors=False,
+                        )
+                    ]
+                    if first_100 and all(
+                        self._is_json_array_of_strings(r) for r in first_100
+                    ):
+                        suggested_facets.append(
+                            {
+                                "name": column,
+                                "type": "array",
+                                "toggle_url": self.ds.absolute_url(
+                                    self.request,
+                                    path_with_added_args(
+                                        self.request, {"_facet_array": column}
+                                    ),
+                                ),
+                            }
+                        )
             except (QueryInterrupted, sqlite3.OperationalError):
                 continue
         return suggested_facets
@@ -500,190 +529,3 @@ class DateFacet(Facet):
                 facets_timed_out.append(column)
 
         return facet_results, facets_timed_out
-
-
-class ManyToManyFacet(Facet):
-    type = "m2m"
-
-    async def suggest(self):
-        # This is calculated based on foreign key relationships to this table
-        # Are there any many-to-many tables pointing here?
-        suggested_facets = []
-        db = self.ds.databases[self.database]
-        all_foreign_keys = await db.get_all_foreign_keys()
-        if not all_foreign_keys.get(self.table):
-            # It's probably a view
-            return []
-        args = set(self.get_querystring_pairs())
-        incoming = all_foreign_keys[self.table]["incoming"]
-        # Do any of these incoming tables have exactly two outgoing keys?
-        for fk in incoming:
-            other_table = fk["other_table"]
-            other_table_outgoing_foreign_keys = all_foreign_keys[other_table][
-                "outgoing"
-            ]
-            if len(other_table_outgoing_foreign_keys) == 2:
-                destination_table = [
-                    t
-                    for t in other_table_outgoing_foreign_keys
-                    if t["other_table"] != self.table
-                ][0]["other_table"]
-                # Only suggest if it's not selected already
-                if ("_facet_m2m", destination_table) in args:
-                    continue
-                suggested_facets.append(
-                    {
-                        "name": destination_table,
-                        "type": "m2m",
-                        "toggle_url": self.ds.absolute_url(
-                            self.request,
-                            path_with_added_args(
-                                self.request, {"_facet_m2m": destination_table}
-                            ),
-                        ),
-                    }
-                )
-        return suggested_facets
-
-    async def facet_results(self):
-        facet_results = {}
-        facets_timed_out = []
-        args = set(self.get_querystring_pairs())
-        facet_size = self.ds.config("default_facet_size")
-        db = self.ds.databases[self.database]
-        all_foreign_keys = await db.get_all_foreign_keys()
-        if not all_foreign_keys.get(self.table):
-            return [], []
-        # We care about three tables: self.table, middle_table and destination_table
-        incoming = all_foreign_keys[self.table]["incoming"]
-        for source_and_config in self.get_configs():
-            config = source_and_config["config"]
-            source = source_and_config["source"]
-            # The destination_table is specified in the _facet_m2m=xxx parameter
-            destination_table = config.get("column") or config["simple"]
-            # Find middle table - it has fks to self.table AND destination_table
-            fks = None
-            middle_table = None
-            for fk in incoming:
-                other_table = fk["other_table"]
-                other_table_outgoing_foreign_keys = all_foreign_keys[other_table][
-                    "outgoing"
-                ]
-                if (
-                    any(
-                        o
-                        for o in other_table_outgoing_foreign_keys
-                        if o["other_table"] == destination_table
-                    )
-                    and len(other_table_outgoing_foreign_keys) == 2
-                ):
-                    fks = other_table_outgoing_foreign_keys
-                    middle_table = other_table
-                    break
-            if middle_table is None or fks is None:
-                return [], []
-            # Now that we have determined the middle_table, we need to figure out the three
-            # columns on that table which are relevant to us. These are:
-            #    column_to_table - the middle_table column with a foreign key to self.table
-            #    table_pk - the primary key column on self.table that is referenced
-            #    column_to_destination - the column with a foreign key to destination_table
-            #
-            # It turns out we don't actually need the fourth obvious column:
-            #    destination_pk = the primary key column on destination_table which is referenced
-            #
-            # These are both in the fks array - which now contains 2 foreign key relationships, e.g:
-            # [
-            #   {'other_table': 'characteristic', 'column': 'characteristic_id', 'other_column': 'pk'},
-            #   {'other_table': 'attractions', 'column': 'attraction_id', 'other_column': 'pk'}
-            # ]
-            column_to_table = None
-            table_pk = None
-            column_to_destination = None
-            for fk in fks:
-                if fk["other_table"] == self.table:
-                    table_pk = fk["other_column"]
-                    column_to_table = fk["column"]
-                elif fk["other_table"] == destination_table:
-                    column_to_destination = fk["column"]
-            assert all((column_to_table, table_pk, column_to_destination))
-            facet_sql = """
-                select
-                    {middle_table}.{column_to_destination} as value,
-                    count(distinct {middle_table}.{column_to_table}) as count
-                from {middle_table}
-                where {middle_table}.{column_to_table} in (
-                    select {table_pk} from ({sql})
-                )
-                group by {middle_table}.{column_to_destination}
-                order by count desc limit {limit}
-            """.format(
-                sql=self.sql,
-                limit=facet_size + 1,
-                middle_table=escape_sqlite(middle_table),
-                column_to_destination=escape_sqlite(column_to_destination),
-                column_to_table=escape_sqlite(column_to_table),
-                table_pk=escape_sqlite(table_pk),
-            )
-            try:
-                facet_rows_results = await self.ds.execute(
-                    self.database,
-                    facet_sql,
-                    self.params,
-                    truncate=False,
-                    custom_time_limit=self.ds.config("facet_time_limit_ms"),
-                )
-                facet_results_values = []
-                facet_results[destination_table] = {
-                    "name": destination_table,
-                    "type": self.type,
-                    "results": facet_results_values,
-                    "hideable": source != "metadata",
-                    "toggle_url": path_with_removed_args(
-                        self.request, {"_facet_m2m": destination_table}
-                    ),
-                    "truncated": len(facet_rows_results) > facet_size,
-                }
-                facet_rows = facet_rows_results.rows[:facet_size]
-
-                # Attempt to expand foreign keys into labels
-                values = [row["value"] for row in facet_rows]
-                expanded = await self.ds.expand_foreign_keys(
-                    self.database, middle_table, column_to_destination, values
-                )
-
-                for row in facet_rows:
-                    through = json.dumps(
-                        {
-                            "table": middle_table,
-                            "column": column_to_destination,
-                            "value": str(row["value"]),
-                        },
-                        separators=(",", ":"),
-                        sort_keys=True,
-                    )
-                    selected = ("_through", through) in args
-                    if selected:
-                        toggle_path = path_with_removed_args(
-                            self.request, {"_through": through}
-                        )
-                    else:
-                        toggle_path = path_with_added_args(
-                            self.request, {"_through": through}
-                        )
-                    facet_results_values.append(
-                        {
-                            "value": row["value"],
-                            "label": expanded.get(
-                                (column_to_destination, row["value"]), row["value"]
-                            ),
-                            "count": row["count"],
-                            "toggle_url": self.ds.absolute_url(
-                                self.request, toggle_path
-                            ),
-                            "selected": selected,
-                        }
-                    )
-            except QueryInterrupted:
-                facets_timed_out.append(destination_table)
-
-        return facet_results, facets_timed_out

datasette/filters.py

@@ -77,6 +77,20 @@ class InFilter(Filter):
         return "{} in {}".format(column, json.dumps(self.split_value(value)))
 
 
+class NotInFilter(InFilter):
+    key = "notin"
+    display = "not in"
+
+    def where_clause(self, table, column, value, param_counter):
+        values = self.split_value(value)
+        params = [":p{}".format(param_counter + i) for i in range(len(values))]
+        sql = "{} not in ({})".format(escape_sqlite(column), ", ".join(params))
+        return sql, values
+
+    def human_clause(self, column, value):
+        return "{} not in {}".format(column, json.dumps(self.split_value(value)))
+
+
 class Filters:
     _filters = (
         [
@@ -125,6 +139,7 @@ class Filters:
             TemplatedFilter("like", "like", '"{c}" like :{p}', '{c} like "{v}"'),
             TemplatedFilter("glob", "glob", '"{c}" glob :{p}', '{c} glob "{v}"'),
             InFilter(),
+            NotInFilter(),
         ]
         + (
             [

datasette/publish/cloudrun.py

@@ -1,6 +1,7 @@
 from datasette import hookimpl
 import click
 import json
+import os
 from subprocess import check_call, check_output
 
 from .common import (
@@ -24,6 +25,11 @@ def publish_subcommand(publish):
         "--service", default="", help="Cloud Run service to deploy (or over-write)"
     )
     @click.option("--spatialite", is_flag=True, help="Enable SpatialLite extension")
+    @click.option(
+        "--show-files",
+        is_flag=True,
+        help="Output the generated Dockerfile and metadata.json",
+    )
     def cloudrun(
         files,
         metadata,
@@ -33,6 +39,7 @@ def publish_subcommand(publish):
         plugins_dir,
         static,
         install,
+        plugin_secret,
         version_note,
         title,
         license,
@@ -44,6 +51,7 @@ def publish_subcommand(publish):
         name,
         service,
         spatialite,
+        show_files,
     ):
         fail_if_publish_binary_not_installed(
             "gcloud", "Google Cloud", "https://cloud.google.com/sdk/"
@@ -52,6 +60,47 @@ def publish_subcommand(publish):
             "gcloud config get-value project", shell=True, universal_newlines=True
         ).strip()
 
+        if not service:
+            # Show the user their current services, then prompt for one
+            click.echo("Please provide a service name for this deployment\n")
+            click.echo("Using an existing service name will over-write it")
+            click.echo("")
+            existing_services = get_existing_services()
+            if existing_services:
+                click.echo("Your existing services:\n")
+                for existing_service in existing_services:
+                    click.echo(
+                        "  {name} - created {created} - {url}".format(
+                            **existing_service
+                        )
+                    )
+                click.echo("")
+            service = click.prompt("Service name", type=str)
+
+        extra_metadata = {
+            "title": title,
+            "license": license,
+            "license_url": license_url,
+            "source": source,
+            "source_url": source_url,
+            "about": about,
+            "about_url": about_url,
+        }
+
+        environment_variables = {}
+        if plugin_secret:
+            extra_metadata["plugins"] = {}
+            for plugin_name, plugin_setting, setting_value in plugin_secret:
+                environment_variable = (
+                    "{}_{}".format(plugin_name, plugin_setting)
+                    .upper()
+                    .replace("-", "_")
+                )
+                environment_variables[environment_variable] = setting_value
+                extra_metadata["plugins"].setdefault(plugin_name, {})[
+                    plugin_setting
+                ] = {"$env": environment_variable}
+
         with temporary_docker_directory(
             files,
             name,
@@ -64,21 +113,40 @@ def publish_subcommand(publish):
             install,
             spatialite,
             version_note,
-            {
-                "title": title,
-                "license": license,
-                "license_url": license_url,
-                "source": source,
-                "source_url": source_url,
-                "about": about,
-                "about_url": about_url,
-            },
+            extra_metadata,
+            environment_variables,
         ):
+            if show_files:
+                if os.path.exists("metadata.json"):
+                    print("=== metadata.json ===\n")
+                    print(open("metadata.json").read())
+                print("\n==== Dockerfile ====\n")
+                print(open("Dockerfile").read())
+                print("\n====================\n")
+
             image_id = "gcr.io/{project}/{name}".format(project=project, name=name)
             check_call("gcloud builds submit --tag {}".format(image_id), shell=True)
         check_call(
-            "gcloud beta run deploy --allow-unauthenticated --image {}{}".format(
-                image_id, " {}".format(service) if service else ""
+            "gcloud beta run deploy --allow-unauthenticated --platform=managed --image {} {}".format(
+                image_id, service,
             ),
             shell=True,
         )
+
+
+def get_existing_services():
+    services = json.loads(
+        check_output(
+            "gcloud beta run services list --platform=managed --format json",
+            shell=True,
+            universal_newlines=True,
+        )
+    )
+    return [
+        {
+            "name": service["metadata"]["name"],
+            "created": service["metadata"]["creationTimestamp"],
+            "url": service["status"]["address"]["url"],
+        }
+        for service in services
+    ]

datasette/publish/common.py

@@ -41,6 +41,14 @@ def add_common_publish_arguments_and_options(subcommand):
                 help="Additional packages (e.g. plugins) to install",
                 multiple=True,
             ),
+            click.option(
+                "--plugin-secret",
+                nargs=3,
+                type=(str, str, str),
+                callback=validate_plugin_secret,
+                multiple=True,
+                help="Secrets to pass to plugins, e.g. --plugin-secret datasette-auth-github client_id xxx",
+            ),
             click.option(
                 "--version-note", help="Additional note to show on /-/versions"
             ),
@@ -76,3 +84,10 @@ def fail_if_publish_binary_not_installed(binary, publish_target, install_link):
             err=True,
         )
         sys.exit(1)
+
+
+def validate_plugin_secret(ctx, param, value):
+    for plugin_name, plugin_setting, setting_value in value:
+        if "'" in setting_value:
+            raise click.BadParameter("--plugin-secret cannot contain single quotes")
+    return value

datasette/publish/heroku.py

@@ -33,6 +33,7 @@ def publish_subcommand(publish):
         plugins_dir,
         static,
         install,
+        plugin_secret,
         version_note,
         title,
         license,
@@ -61,6 +62,33 @@ def publish_subcommand(publish):
             )
             call(["heroku", "plugins:install", "heroku-builds"])
 
+        extra_metadata = {
+            "title": title,
+            "license": license,
+            "license_url": license_url,
+            "source": source,
+            "source_url": source_url,
+            "about": about,
+            "about_url": about_url,
+        }
+
+        environment_variables = {
+            # Avoid uvicorn error: https://github.com/simonw/datasette/issues/633
+            "WEB_CONCURRENCY": "1"
+        }
+        if plugin_secret:
+            extra_metadata["plugins"] = {}
+            for plugin_name, plugin_setting, setting_value in plugin_secret:
+                environment_variable = (
+                    "{}_{}".format(plugin_name, plugin_setting)
+                    .upper()
+                    .replace("-", "_")
+                )
+                environment_variables[environment_variable] = setting_value
+                extra_metadata["plugins"].setdefault(plugin_name, {})[
+                    plugin_setting
+                ] = {"$env": environment_variable}
+
         with temporary_heroku_directory(
             files,
             name,
@@ -72,15 +100,7 @@ def publish_subcommand(publish):
             static,
             install,
             version_note,
-            {
-                "title": title,
-                "license": license,
-                "license_url": license_url,
-                "source": source,
-                "source_url": source_url,
-                "about": about,
-                "about_url": about_url,
-            },
+            extra_metadata,
         ):
             app_name = None
             if name:
@@ -104,6 +124,11 @@ def publish_subcommand(publish):
                 create_output = check_output(cmd).decode("utf8")
                 app_name = json.loads(create_output)["name"]
 
+            for key, value in environment_variables.items():
+                call(
+                    ["heroku", "config:set", "-a", app_name, "{}={}".format(key, value)]
+                )
+
             call(["heroku", "builds:create", "-a", app_name, "--include-vcs-ignore"])
 
 
@@ -142,7 +167,7 @@ def temporary_heroku_directory(
         if metadata_content:
             open("metadata.json", "w").write(json.dumps(metadata_content, indent=2))
 
-        open("runtime.txt", "w").write("python-3.6.8")
+        open("runtime.txt", "w").write("python-3.8.0")
 
         if branch:
             install = [

datasette/publish/now.py

@@ -1,6 +1,7 @@
 from datasette import hookimpl
 import click
 import json
+import os
 from subprocess import run, PIPE
 
 from .common import (
@@ -24,6 +25,11 @@ def publish_subcommand(publish):
     @click.option("--token", help="Auth token to use for deploy")
     @click.option("--alias", multiple=True, help="Desired alias e.g. yoursite.now.sh")
     @click.option("--spatialite", is_flag=True, help="Enable SpatialLite extension")
+    @click.option(
+        "--show-files",
+        is_flag=True,
+        help="Output the generated Dockerfile and metadata.json",
+    )
     def nowv1(
         files,
         metadata,
@@ -33,6 +39,7 @@ def publish_subcommand(publish):
         plugins_dir,
         static,
         install,
+        plugin_secret,
         version_note,
         title,
         license,
@@ -46,6 +53,7 @@ def publish_subcommand(publish):
         token,
         alias,
         spatialite,
+        show_files,
     ):
         fail_if_publish_binary_not_installed("now", "Zeit Now", "https://zeit.co/now")
         if extra_options:
@@ -54,6 +62,30 @@ def publish_subcommand(publish):
             extra_options = ""
         extra_options += "--config force_https_urls:on"
 
+        extra_metadata = {
+            "title": title,
+            "license": license,
+            "license_url": license_url,
+            "source": source,
+            "source_url": source_url,
+            "about": about,
+            "about_url": about_url,
+        }
+
+        environment_variables = {}
+        if plugin_secret:
+            extra_metadata["plugins"] = {}
+            for plugin_name, plugin_setting, setting_value in plugin_secret:
+                environment_variable = (
+                    "{}_{}".format(plugin_name, plugin_setting)
+                    .upper()
+                    .replace("-", "_")
+                )
+                environment_variables[environment_variable] = setting_value
+                extra_metadata["plugins"].setdefault(plugin_name, {})[
+                    plugin_setting
+                ] = {"$env": environment_variable}
+
         with temporary_docker_directory(
             files,
             name,
@@ -66,15 +98,8 @@ def publish_subcommand(publish):
             install,
             spatialite,
             version_note,
-            {
-                "title": title,
-                "license": license,
-                "license_url": license_url,
-                "source": source,
-                "source_url": source_url,
-                "about": about,
-                "about_url": about_url,
-            },
+            extra_metadata,
+            environment_variables,
         ):
             now_json = {"version": 1}
             open("now.json", "w").write(json.dumps(now_json, indent=4))
@@ -88,6 +113,13 @@ def publish_subcommand(publish):
             else:
                 done = run("now", stdout=PIPE)
             deployment_url = done.stdout
+            if show_files:
+                if os.path.exists("metadata.json"):
+                    print("=== metadata.json ===\n")
+                    print(open("metadata.json").read())
+                print("\n==== Dockerfile ====\n")
+                print(open("Dockerfile").read())
+                print("\n====================\n")
             if alias:
                 # I couldn't get --target=production working, so I call
                 # 'now alias' with arguments directly instead - but that

datasette/static/app.css

@@ -90,6 +90,7 @@ table a:visited {
     background-color: #eee;
     overflow: hidden;
     box-sizing: border-box;
+    min-height: 2rem;
 }
 .hd p {
     margin: 0;
@@ -165,22 +166,32 @@ form input[type=search] {
         width: 95%;
     }
 }
-form input[type=submit] {
-    color: #fff;
-    background-color: #007bff;
-    border-color: #007bff;
+form input[type=submit], form button[type=button] {
     font-weight: 400;
     cursor: pointer;
     text-align: center;
     vertical-align: middle;
-    border: 1px solid blue;
+    border-width: 1px;
+    border-style: solid;
     padding: .5em 0.8em;
     font-size: 0.9rem;
     line-height: 1;
     border-radius: .25rem;
+}
+
+form input[type=submit] {
+    color: #fff;
+    background-color: #007bff;
+    border-color: #007bff;
     -webkit-appearance: button;
 }
 
+form button[type=button] {
+    color: #007bff;
+    background-color: #fff;
+    border-color: #007bff;
+}
+
 .filter-row {
     margin-bottom: 0.6em;
 }
@@ -312,3 +323,7 @@ a.not-underlined {
     font-style: normal;
     font-size: 0.8em;
 }
+
+pre.wrapped-sql {
+    white-space: pre-wrap;
+}

datasette/templates/500.html

@@ -2,8 +2,14 @@
 
 {% block title %}{% if title %}{{ title }}{% else %}Error {{ status }}{% endif %}{% endblock %}
 
+{% block nav %}
+    <p class="crumbs">
+        <a href="/">home</a>
+    </p>
+    {{ super() }}
+{% endblock %}
+
 {% block content %}
-<div class="hd"><a href="/">home</a></div>
 
 <h1>{% if title %}{{ title }}{% else %}Error {{ status }}{% endif %}</h1>
 

datasette/templates/_codemirror.html

@@ -1,3 +1,4 @@
+<script src="/-/static/sql-formatter-2.3.3.min.js" defer></script>
 <script src="/-/static/codemirror-5.31.0.js"></script>
 <link rel="stylesheet" href="/-/static/codemirror-5.31.0-min.css" />
 <script src="/-/static/codemirror-5.31.0-sql.min.js"></script>

datasette/templates/_codemirror_foot.html

@@ -1,13 +1,37 @@
 <script>
-    var editor = CodeMirror.fromTextArea(document.getElementById("sql-editor"), {
-      lineNumbers: true,
-      mode: "text/x-sql",
-      lineWrapping: true,
-    });
-    editor.setOption("extraKeys", {
-      "Shift-Enter": function() {
-        document.getElementsByClassName("sql")[0].submit();
-      },
-      Tab: false
-    });
+window.onload = () => {
+    const sqlFormat = document.querySelector("button#sql-format");
+    const readOnly = document.querySelector("pre#sql-query");
+    const sqlInput = document.querySelector("textarea#sql-editor");
+    if (sqlFormat && !readOnly) {
+        sqlFormat.hidden = false;
+    }
+    if (sqlInput) {
+        var editor = CodeMirror.fromTextArea(sqlInput, {
+          lineNumbers: true,
+          mode: "text/x-sql",
+          lineWrapping: true,
+        });
+        editor.setOption("extraKeys", {
+          "Shift-Enter": function() {
+            document.getElementsByClassName("sql")[0].submit();
+          },
+          Tab: false
+        });
+        if (sqlFormat) {
+            sqlFormat.addEventListener("click", ev => {
+                editor.setValue(sqlFormatter.format(editor.getValue()));
+            })
+        }
+    }
+    if (sqlFormat && readOnly) {
+        const formatted = sqlFormatter.format(readOnly.innerHTML);
+        if (formatted != readOnly.innerHTML) {
+            sqlFormat.hidden = false;
+            sqlFormat.addEventListener("click", ev => {
+                readOnly.innerHTML = formatted;
+            })
+        }
+    }
+}
 </script>

datasette/templates/_description_source_license.html

@@ -21,7 +21,7 @@
             <a href="{{ metadata.source_url }}">
         {% endif %}{{ metadata.source or metadata.source_url }}{% if metadata.source_url %}</a>{% endif %}
     {% endif %}
-    {% if metadata.about or metadata.about_url %}{% if metadata.license or metadata.license_url or metadata.source or metadat.source_url %}&middot;{% endif %}
+    {% if metadata.about or metadata.about_url %}{% if metadata.license or metadata.license_url or metadata.source or metadata.source_url %}&middot;{% endif %}
         About: {% if metadata.about_url %}
             <a href="{{ metadata.about_url }}">
         {% endif %}{{ metadata.about or metadata.about_url }}{% if metadata.about_url %}</a>{% endif %}

datasette/templates/database.html

@@ -25,8 +25,11 @@
 {% if config.allow_sql %}
     <form class="sql" action="{{ database_url(database) }}" method="get">
         <h3>Custom SQL query</h3>
-        <p><textarea name="sql">{% if tables %}select * from {{ tables[0].name|escape_sqlite }}{% else %}select sqlite_version(){% endif %}</textarea></p>
-        <p><input type="submit" value="Run SQL"></p>
+        <p><textarea id="sql-editor" name="sql">{% if tables %}select * from {{ tables[0].name|escape_sqlite }}{% else %}select sqlite_version(){% endif %}</textarea></p>
+        <p>
+            <button id="sql-format" type="button" hidden>Format SQL</button>
+            <input type="submit" value="Run SQL">
+        </p>
     </form>
 {% endif %}
 

datasette/templates/query.html

@@ -18,9 +18,15 @@
 
 {% block body_class %}query db-{{ database|to_css_class }}{% endblock %}
 
-{% block content %}
-<div class="hd"><a href="/">home</a> / <a href="{{ database_url(database) }}">{{ database }}</a></div>
+{% block nav %}
+    <p class="crumbs">
+        <a href="/">home</a> /
+        <a href="{{ database_url(database) }}">{{ database }}</a>
+    </p>
+    {{ super() }}
+{% endblock %}
 
+{% block content %}
 <h1 style="padding-left: 10px; border-left: 10px solid #{{ database_color(database) }}">{{ metadata.title or database }}</h1>
 
 {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %}
@@ -31,7 +37,7 @@
         {% if editable and config.allow_sql %}
             <p><textarea id="sql-editor" name="sql">{% if query and query.sql %}{{ query.sql }}{% else %}select * from {{ tables[0].name|escape_sqlite }}{% endif %}</textarea></p>
         {% else %}
-            <pre>{% if query %}{{ query.sql }}{% endif %}</pre>
+            <pre id="sql-query">{% if query %}{{ query.sql }}{% endif %}</pre>
         {% endif %}
     {% else %}
         <input type="hidden" name="sql" value="{% if query and query.sql %}{{ query.sql }}{% else %}select * from {{ tables[0].name|escape_sqlite }}{% endif %}">
@@ -43,7 +49,10 @@
             <p><label for="qp{{ loop.index }}">{{ name }}</label> <input type="text" id="qp{{ loop.index }}" name="{{ name }}" value="{{ value }}"></p>
         {% endfor %}
     {% endif %}
-    <p><input type="submit" value="Run SQL"></p>
+    <p>
+        <button id="sql-format" type="button" hidden>Format SQL</button>
+        <input type="submit" value="Run SQL">
+    </p>
 </form>
 
 {% if display_rows %}

datasette/templates/table.html

@@ -184,11 +184,11 @@
 {% endif %}
 
 {% if table_definition %}
-    <pre>{{ table_definition }}</pre>
+    <pre class="wrapped-sql">{{ table_definition }}</pre>
 {% endif %}
 
 {% if view_definition %}
-    <pre>{{ view_definition }}</pre>
+    <pre class="wrapped-sql">{{ view_definition }}</pre>
 {% endif %}
 
 {% endblock %}

datasette/tracer.py

@@ -9,12 +9,19 @@ tracers = {}
 TRACE_RESERVED_KEYS = {"type", "start", "end", "duration_ms", "traceback"}
 
 
+# asyncio.current_task was introduced in Python 3.7:
+for obj in (asyncio, asyncio.Task):
+    current_task = getattr(obj, "current_task", None)
+    if current_task is not None:
+        break
+
+
 def get_task_id():
     try:
         loop = asyncio.get_event_loop()
     except RuntimeError:
         return None
-    return id(asyncio.Task.current_task(loop=loop))
+    return id(current_task(loop=loop))
 
 
 @contextmanager

datasette/utils/__init__.py

@@ -167,6 +167,8 @@ allowed_sql_res = [
     re.compile(r"^explain select\b"),
     re.compile(r"^explain query plan select\b"),
     re.compile(r"^with\b"),
+    re.compile(r"^explain with\b"),
+    re.compile(r"^explain query plan with\b"),
 ]
 disallawed_sql_res = [(re.compile("pragma"), "Statement may not contain PRAGMA")]
 
@@ -272,6 +274,7 @@ def make_dockerfile(
     install,
     spatialite,
     version_note,
+    environment_variables=None,
 ):
     cmd = ["datasette", "serve", "--host", "0.0.0.0"]
     for filename in files:
@@ -303,15 +306,22 @@ def make_dockerfile(
         install = ["datasette"] + list(install)
 
     return """
-FROM python:3.6
+FROM python:3.8
 COPY . /app
 WORKDIR /app
 {spatialite_extras}
+{environment_variables}
 RUN pip install -U {install_from}
 RUN datasette inspect {files} --inspect-file inspect-data.json
 ENV PORT 8001
 EXPOSE 8001
 CMD {cmd}""".format(
+        environment_variables="\n".join(
+            [
+                "ENV {} '{}'".format(key, value)
+                for key, value in (environment_variables or {}).items()
+            ]
+        ),
         files=" ".join(files),
         cmd=cmd,
         install_from=" ".join(install),
@@ -333,6 +343,7 @@ def temporary_docker_directory(
     spatialite,
     version_note,
     extra_metadata=None,
+    environment_variables=None,
 ):
     extra_metadata = extra_metadata or {}
     tmp = tempfile.TemporaryDirectory()
@@ -361,6 +372,7 @@ def temporary_docker_directory(
             install,
             spatialite,
             version_note,
+            environment_variables,
         )
         os.chdir(datasette_dir)
         if metadata_content:
@@ -459,6 +471,7 @@ def detect_fts_sql(table):
             where rootpage = 0
             and (
                 sql like '%VIRTUAL TABLE%USING FTS%content="{table}"%'
+                or sql like '%VIRTUAL TABLE%USING FTS%content=[{table}]%'
                 or (
                     tbl_name = "{table}"
                     and sql like '%VIRTUAL TABLE%USING FTS%'
@@ -620,6 +633,7 @@ def get_plugins(pm):
         distinfo = plugin_to_distinfo.get(plugin)
         if distinfo:
             plugin_info["version"] = distinfo.version
+            plugin_info["name"] = distinfo.project_name
         plugins.append(plugin_info)
     return plugins
 
@@ -725,7 +739,8 @@ class StaticMount(click.ParamType):
                 param,
                 ctx,
             )
-        path, dirpath = value.split(":")
+        path, dirpath = value.split(":", 1)
+        dirpath = os.path.abspath(dirpath)
         if not os.path.exists(dirpath) or not os.path.isdir(dirpath):
             self.fail("%s is not a valid directory path" % value, param, ctx)
         return path, dirpath

datasette/utils/asgi.py

@@ -45,9 +45,10 @@ class Request:
 
     @property
     def path(self):
-        return (
-            self.scope.get("raw_path", self.scope["path"].encode("latin-1"))
-        ).decode("latin-1")
+        if "raw_path" in self.scope:
+            return self.scope["raw_path"].decode("latin-1")
+        else:
+            return self.scope["path"].decode("utf-8")
 
     @property
     def query_string(self):
@@ -216,7 +217,7 @@ class AsgiWriter:
         await self.send(
             {
                 "type": "http.response.body",
-                "body": chunk.encode("latin-1"),
+                "body": chunk.encode("utf-8"),
                 "more_body": True,
             }
         )
@@ -300,14 +301,17 @@ async def asgi_send_file(
 def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):
     async def inner_static(scope, receive, send):
         path = scope["url_route"]["kwargs"]["path"]
-        full_path = (Path(root_path) / path).absolute()
+        try:
+            full_path = (Path(root_path) / path).resolve().absolute()
+        except FileNotFoundError:
+            await asgi_send_html(send, "404", 404)
+            return
         # Ensure full_path is within root_path to avoid weird "../" tricks
         try:
             full_path.relative_to(root_path)
         except ValueError:
             await asgi_send_html(send, "404", 404)
             return
-        first = True
         try:
             await asgi_send_file(send, full_path, chunk_size=chunk_size)
         except FileNotFoundError:

datasette/views/base.py

@@ -139,7 +139,7 @@ class BaseView(AsgiView):
             extra_template_vars.update(extra_vars)
 
         return Response.html(
-            template.render(
+            await template.render_async(
                 {
                     **context,
                     **{
@@ -193,22 +193,23 @@ class DataView(BaseView):
     async def resolve_db_name(self, request, db_name, **kwargs):
         hash = None
         name = None
-        if "-" in db_name:
-            # Might be name-and-hash, or might just be
-            # a name with a hyphen in it
-            name, hash = db_name.rsplit("-", 1)
-            if name not in self.ds.databases:
-                # Try the whole name
-                name = db_name
-                hash = None
+        if db_name not in self.ds.databases and "-" in db_name:
+            # No matching DB found, maybe it's a name-hash?
+            name_bit, hash_bit = db_name.rsplit("-", 1)
+            if name_bit not in self.ds.databases:
+                raise NotFound("Database not found: {}".format(name))
+            else:
+                name = name_bit
+                hash = hash_bit
         else:
             name = db_name
-        # Verify the hash
+        name = urllib.parse.unquote_plus(name)
         try:
             db = self.ds.databases[name]
         except KeyError:
             raise NotFound("Database not found: {}".format(name))
 
+        # Verify the hash
         expected = "000"
         if db.hash is not None:
             expected = db.hash[:HASH_LENGTH]
@@ -257,9 +258,12 @@ class DataView(BaseView):
         assert NotImplemented
 
     async def get(self, request, db_name, **kwargs):
-        database, hash, correct_hash_provided, should_redirect = await self.resolve_db_name(
-            request, db_name, **kwargs
-        )
+        (
+            database,
+            hash,
+            correct_hash_provided,
+            should_redirect,
+        ) = await self.resolve_db_name(request, db_name, **kwargs)
         if should_redirect:
             return self.redirect(request, should_redirect, remove_args={"_hash"})
 
@@ -327,10 +331,14 @@ class DataView(BaseView):
                         else:
                             # Look for {"value": "label": } dicts and expand
                             new_row = []
-                            for cell in row:
-                                if isinstance(cell, dict):
-                                    new_row.append(cell["value"])
-                                    new_row.append(cell["label"])
+                            for heading, cell in zip(data["columns"], row):
+                                if heading in expanded_columns:
+                                    if cell is None:
+                                        new_row.extend(("", ""))
+                                    else:
+                                        assert isinstance(cell, dict)
+                                        new_row.append(cell["value"])
+                                        new_row.append(cell["label"])
                                 else:
                                     new_row.append(cell)
                             await writer.writerow(new_row)
@@ -362,6 +370,8 @@ class DataView(BaseView):
         _format = request.args.get("_format", None)
         if not _format:
             _format = (args.pop("as_format", None) or "").lstrip(".")
+        else:
+            args.pop("as_format", None)
         if "table_and_format" in args:
             db = self.ds.databases[database]
 
@@ -387,7 +397,7 @@ class DataView(BaseView):
             return await self.as_csv(request, database, hash, **kwargs)
 
         if _format is None:
-            # HTML views default to expanding all foriegn key labels
+            # HTML views default to expanding all foreign key labels
             kwargs["default_labels"] = True
 
         extra_template_data = {}

datasette/views/database.py

@@ -10,12 +10,17 @@ class DatabaseView(DataView):
     name = "database"
 
     async def data(self, request, database, hash, default_labels=False, _size=None):
+        metadata = (self.ds.metadata("databases") or {}).get(database, {})
+        self.ds.update_with_inherited_metadata(metadata)
+
         if request.args.get("sql"):
             if not self.ds.config("allow_sql"):
                 raise DatasetteError("sql= is not allowed", status=400)
             sql = request.raw_args.pop("sql")
             validate_sql_select(sql)
-            return await self.custom_sql(request, database, hash, sql, _size=_size)
+            return await self.custom_sql(
+                request, database, hash, sql, _size=_size, metadata=metadata
+            )
 
         db = self.ds.databases[database]
 
@@ -24,9 +29,6 @@ class DatabaseView(DataView):
         hidden_table_names = set(await db.hidden_table_names())
         all_foreign_keys = await db.get_all_foreign_keys()
 
-        metadata = (self.ds.metadata("databases") or {}).get(database, {})
-        self.ds.update_with_inherited_metadata(metadata)
-
         tables = []
         for table in table_counts:
             table_columns = await db.table_columns(table)

datasette/views/index.py

@@ -97,8 +97,6 @@ class IndexView(BaseView):
                 }
             )
 
-        databases.sort(key=lambda database: database["name"])
-
         if as_format:
             headers = {}
             if self.ds.cors:

datasette/views/table.py

@@ -235,13 +235,17 @@ class TableView(RowTableShared):
             raise NotFound("Table not found: {}".format(table))
 
         pks = await db.primary_keys(table)
+        table_columns = await db.table_columns(table)
+
+        select_columns = ", ".join(escape_sqlite(t) for t in table_columns)
+
         use_rowid = not pks and not is_view
         if use_rowid:
-            select = "rowid, *"
+            select = "rowid, {}".format(select_columns)
             order_by = "rowid"
             order_by_pks = "rowid"
         else:
-            select = "*"
+            select = select_columns
             order_by_pks = ", ".join([escape_sqlite(pk) for pk in pks])
             order_by = order_by_pks
 
@@ -257,12 +261,10 @@ class TableView(RowTableShared):
         # That's so if there is a column that starts with _
         # it can still be queried using ?_col__exact=blah
         special_args = {}
-        special_args_lists = {}
         other_args = []
         for key, value in args.items():
             if key.startswith("_") and "__" not in key:
                 special_args[key] = value[0]
-                special_args_lists[key] = value
             else:
                 for v in value:
                     other_args.append((key, v))
@@ -495,18 +497,6 @@ class TableView(RowTableShared):
         if order_by:
             order_by = "order by {} ".format(order_by)
 
-        # _group_count=col1&_group_count=col2
-        group_count = special_args_lists.get("_group_count") or []
-        if group_count:
-            sql = 'select {group_cols}, count(*) as "count" from {table_name} {where} group by {group_cols} order by "count" desc limit 100'.format(
-                group_cols=", ".join(
-                    '"{}"'.format(group_count_col) for group_count_col in group_count
-                ),
-                table_name=escape_sqlite(table),
-                where=where_clause,
-            )
-            return await self.custom_sql(request, database, hash, sql, editable=True)
-
         extra_args = {}
         # Handle ?_size=500
         page_size = _size or request.raw_args.get("_size")
@@ -586,9 +576,10 @@ class TableView(RowTableShared):
             )
 
         for facet in facet_instances:
-            instance_facet_results, instance_facets_timed_out = (
-                await facet.facet_results()
-            )
+            (
+                instance_facet_results,
+                instance_facets_timed_out,
+            ) = await facet.facet_results()
             facet_results.update(instance_facet_results)
             facets_timed_out.extend(instance_facets_timed_out)
 
@@ -636,7 +627,7 @@ class TableView(RowTableShared):
                     new_row = CustomRow(columns)
                     for column in row.keys():
                         value = row[column]
-                        if (column, value) in expanded_labels:
+                        if (column, value) in expanded_labels and value is not None:
                             new_row[column] = {
                                 "value": value,
                                 "label": expanded_labels[(column, value)],
@@ -720,6 +711,9 @@ class TableView(RowTableShared):
             for arg in ("_fts_table", "_fts_pk"):
                 if arg in special_args:
                     form_hidden_args.append((arg, special_args[arg]))
+            if request.args.get("_where"):
+                for where_text in request.args["_where"]:
+                    form_hidden_args.append(("_where", where_text))
             return {
                 "supports_search": bool(fts_table),
                 "search": search or "",

docs/changelog.rst

@@ -4,6 +4,204 @@
 Changelog
 =========
 
+.. _v0_32:
+
+0.32 (2019-11-14)
+-----------------
+
+Datasette now renders templates using `Jinja async mode <https://jinja.palletsprojects.com/en/2.10.x/api/#async-support>`__. This makes it easy for plugins to provide custom template functions that perform asynchronous actions, for example the new `datasette-template-sql <https://github.com/simonw/datasette-template-sql>`__ plugin which allows custom templates to directly execute SQL queries and render their results. (`#628 <https://github.com/simonw/datasette/issues/628>`__)
+
+.. _v0_31_2:
+
+0.31.2 (2019-11-13)
+-------------------
+
+- Fixed a bug where ``datasette publish heroku`` applications failed to start (`#633 <https://github.com/simonw/datasette/issues/633>`__)
+- Fix for ``datasette publish`` with just ``--source_url`` - thanks, Stanley Zheng (`#572 <https://github.com/simonw/datasette/issues/572>`__)
+- Deployments to Heroku now use Python 3.8.0 (`#632 <https://github.com/simonw/datasette/issues/632>`__)
+
+.. _v0_31_1:
+
+0.31.1 (2019-11-12)
+-------------------
+
+- Deployments created using ``datasette publish``  now use ``python:3.8`` base Docker image (`#629 <https://github.com/simonw/datasette/pull/629>`__)
+
+.. _v0_31:
+
+0.31 (2019-11-11)
+-----------------
+
+This version adds compatibility with Python 3.8 and breaks compatibility with Python 3.5.
+
+If you are still running Python 3.5 you should stick with ``0.30.2``, which you can install like this::
+
+    pip install datasette==0.30.2
+
+- Format SQL button now works with read-only SQL queries - thanks, Tobias Kunze (`#602 <https://github.com/simonw/datasette/pull/602>`__)
+- New ``?column__notin=x,y,z`` filter for table views (`#614 <https://github.com/simonw/datasette/issues/614>`__)
+- Table view now uses ``select col1, col2, col3`` instead of ``select *``
+- Database filenames can now contain spaces - thanks, Tobias Kunze (`#590 <https://github.com/simonw/datasette/pull/590>`__)
+- Removed obsolete ``?_group_count=col`` feature (`#504 <https://github.com/simonw/datasette/issues/504>`__)
+- Improved user interface and documentation for ``datasette publish cloudrun`` (`#608 <https://github.com/simonw/datasette/issues/608>`__)
+- Tables with indexes now show the ``CREATE INDEX`` statements on the table page (`#618 <https://github.com/simonw/datasette/issues/618>`__)
+- Current version of `uvicorn <https://www.uvicorn.org/>`__ is now shown on ``/-/versions``
+- Python 3.8 is now supported! (`#622 <https://github.com/simonw/datasette/issues/622>`__)
+- Python 3.5 is no longer supported.
+
+.. _v0_30_2:
+
+0.30.2 (2019-11-02)
+-------------------
+
+- ``/-/plugins`` page now uses distribution name e.g. ``datasette-cluster-map`` instead of the name of the underlying Python package (``datasette_cluster_map``) (`#606 <https://github.com/simonw/datasette/issues/606>`__)
+- Array faceting is now only suggested for columns that contain arrays of strings (`#562 <https://github.com/simonw/datasette/issues/562>`__)
+- Better documentation for the ``--host`` argument (`#574 <https://github.com/simonw/datasette/issues/574>`__)
+- Don't show ``None`` with a broken link for the label on a nullable foreign key (`#406 <https://github.com/simonw/datasette/issues/406>`__)
+
+.. _v0_30_1:
+
+0.30.1 (2019-10-30)
+-------------------
+
+- Fixed bug where ``?_where=`` parameter was not persisted in hidden form fields (`#604 <https://github.com/simonw/datasette/issues/604>`__)
+- Fixed bug with .JSON representation of row pages - thanks, Chris Shaw (`#603 <https://github.com/simonw/datasette/issues/603>`__)
+
+.. _v0_30:
+
+
+0.30 (2019-10-18)
+-----------------
+
+- Added ``/-/threads`` debugging page
+- Allow ``EXPLAIN WITH...`` (`#583 <https://github.com/simonw/datasette/issues/583>`__)
+- Button to format SQL - thanks, Tobias Kunze (`#136 <https://github.com/simonw/datasette/issues/136>`__)
+- Sort databases on homepage by argument order - thanks, Tobias Kunze (`#585 <https://github.com/simonw/datasette/issues/585>`__)
+- Display metadata footer on custom SQL queries - thanks, Tobias Kunze (`#589 <https://github.com/simonw/datasette/pull/589>`__)
+- Use ``--platform=managed`` for ``publish cloudrun`` (`#587 <https://github.com/simonw/datasette/issues/587>`__)
+- Fixed bug returning non-ASCII characters in CSV (`#584 <https://github.com/simonw/datasette/issues/584>`__)
+- Fix for ``/foo`` v.s. ``/foo-bar`` bug (`#601 <https://github.com/simonw/datasette/issues/601>`__)
+
+.. _v0_29_3:
+
+0.29.3 (2019-09-02)
+-------------------
+
+- Fixed implementation of CodeMirror on database page (`#560 <https://github.com/simonw/datasette/issues/560>`__)
+- Documentation typo fixes - thanks, Min ho Kim (`#561 <https://github.com/simonw/datasette/pull/561>`__)
+- Mechanism for detecting if a table has FTS enabled now works if the table name used alternative escaping mechanisms (`#570 <https://github.com/simonw/datasette/issues/570>`__) - for compatibility with `a recent change to sqlite-utils <https://github.com/simonw/sqlite-utils/pull/57>`__.
+
+.. _v0_29_2:
+
+0.29.2 (2019-07-13)
+-------------------
+
+- Bumped `Uvicorn <https://www.uvicorn.org/>`__ to 0.8.4, fixing a bug where the querystring was not included in the server logs. (`#559 <https://github.com/simonw/datasette/issues/559>`__)
+- Fixed bug where the navigation breadcrumbs were not displayed correctly on the page for a custom query. (`#558 <https://github.com/simonw/datasette/issues/558>`__)
+- Fixed bug where custom query names containing unicode characters caused errors.
+
+.. _v0_29_1:
+
+0.29.1 (2019-07-11)
+-------------------
+
+- Fixed bug with static mounts using relative paths which could lead to traversal exploits (`#555 <https://github.com/simonw/datasette/issues/555>`__) - thanks Abdussamet Kocak!
+- Datasette can now be run as a module: ``python -m datasette`` (`#556 <https://github.com/simonw/datasette/issues/556>`__) - thanks, Abdussamet Kocak!
+
+.. _v0_29:
+
+0.29 (2019-07-07)
+-----------------
+
+ASGI, new plugin hooks, facet by date and much, much more...
+
+ASGI
+~~~~
+
+`ASGI <https://asgi.readthedocs.io/>`__ is the Asynchronous Server Gateway Interface standard. I've been wanting to convert Datasette into an ASGI application for over a year - `Port Datasette to ASGI #272 <https://github.com/simonw/datasette/issues/272>`__ tracks thirteen months of intermittent development - but with Datasette 0.29 the change is finally released. This also means Datasette now runs on top of `Uvicorn <https://www.uvicorn.org/>`__ and no longer depends on `Sanic <https://github.com/huge-success/sanic>`__.
+
+I wrote about the significance of this change in `Porting Datasette to ASGI, and Turtles all the way down <https://simonwillison.net/2019/Jun/23/datasette-asgi/>`__.
+
+The most exciting consequence of this change is that Datasette plugins can now take advantage of the ASGI standard.
+
+New plugin hook: asgi_wrapper
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :ref:`plugin_asgi_wrapper` plugin hook allows plugins to entirely wrap the Datasette ASGI application in their own ASGI middleware. (`#520 <https://github.com/simonw/datasette/issues/520>`__)
+
+Two new plugins take advantage of this hook:
+
+* `datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ adds a authentication layer: users will have to sign in using their GitHub account before they can view data or interact with Datasette. You can also use it to restrict access to specific GitHub users, or to members of specified GitHub `organizations <https://help.github.com/en/articles/about-organizations>`__ or `teams <https://help.github.com/en/articles/organizing-members-into-teams>`__.
+
+* `datasette-cors <https://github.com/simonw/datasette-cors>`__ allows you to configure `CORS headers <https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS>`__ for your Datasette instance. You can use this to enable JavaScript running on a whitelisted set of domains to make ``fetch()`` calls to the JSON API provided by your Datasette instance.
+
+New plugin hook: extra_template_vars
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :ref:`plugin_hook_extra_template_vars` plugin hook allows plugins to inject their own additional variables into the Datasette template context. This can be used in conjunction with custom templates to customize the Datasette interface. `datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ uses this hook to add custom HTML to the new top navigation bar (which is designed to be modified by plugins, see `#540 <https://github.com/simonw/datasette/issues/540>`__).
+
+Secret plugin configuration options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Plugins like `datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ need a safe way to set secret configuration options. Since the default mechanism for configuring plugins exposes those settings in ``/-/metadata`` a new mechanism was needed. :ref:`plugins_configuration_secret` describes how plugins can now specify that their settings should be read from a file or an environment variable::
+
+    {
+        "plugins": {
+            "datasette-auth-github": {
+                "client_secret": {
+                    "$env": "GITHUB_CLIENT_SECRET"
+                }
+            }
+        }
+    }
+
+These plugin secrets can be set directly using ``datasette publish``. See :ref:`publish_custom_metadata_and_plugins` for details. (`#538 <https://github.com/simonw/datasette/issues/538>`__ and `#543 <https://github.com/simonw/datasette/issues/543>`__)
+
+Facet by date
+~~~~~~~~~~~~~
+
+If a column contains datetime values, Datasette can now facet that column by date. (`#481 <https://github.com/simonw/datasette/issues/481>`__)
+
+.. _v0_29_medium_changes:
+
+Easier custom templates for table rows
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to customize the display of individual table rows, you can do so using a ``_table.html`` template include that looks something like this::
+
+    {% for row in display_rows %}
+        <div>
+            <h2>{{ row["title"] }}</h2>
+            <p>{{ row["description"] }}<lp>
+            <p>Category: {{ row.display("category_id") }}</p>
+        </div>
+    {% endfor %}
+
+This is a **backwards incompatible change**. If you previously had a custom template called ``_rows_and_columns.html`` you need to rename it to ``_table.html``.
+
+See :ref:`customization_custom_templates` for full details.
+
+?_through= for joins through many-to-many tables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The new ``?_through={json}`` argument to the Table view allows records to be filtered based on a many-to-many relationship. See :ref:`json_api_table_arguments` for full documentation - here's `an example <https://latest.datasette.io/fixtures/roadside_attractions?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22}>`__. (`#355 <https://github.com/simonw/datasette/issues/355>`__)
+
+This feature was added to help support `facet by many-to-many <https://github.com/simonw/datasette/issues/551>`__, which isn't quite ready yet but will be coming in the next Datasette release.
+
+Small changes
+~~~~~~~~~~~~~
+
+* Databases published using ``datasette publish`` now open in :ref:`performance_immutable_mode`. (`#469 <https://github.com/simonw/datasette/issues/469>`__)
+* ``?col__date=`` now works for columns containing spaces
+* Automatic label detection (for deciding which column to show when linking to a foreign key) has been improved. (`#485 <https://github.com/simonw/datasette/issues/485>`__)
+* Fixed bug where pagination broke when combined with an expanded foreign key. (`#489 <https://github.com/simonw/datasette/issues/489>`__)
+* Contributors can now run ``pip install -e .[docs]`` to get all of the dependencies needed to build the documentation, including ``cd docs && make livehtml`` support.
+* Datasette's dependencies are now all specified using the ``~=`` match operator. (`#532 <https://github.com/simonw/datasette/issues/532>`__)
+* ``white-space: pre-wrap`` now used for table creation SQL. (`#505 <https://github.com/simonw/datasette/issues/505>`__)
+
+
+`Full list of commits <https://github.com/simonw/datasette/compare/0.28...0.29>`__ between 0.28 and 0.29.
+
 .. _v0_28:
 
 0.28 (2019-05-19)
@@ -31,7 +229,7 @@ Datasette can still run against immutable files and gains numerous performance b
 Faceting improvements, and faceting plugins
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Datasette :ref:`facets` provide an intuitive way to quickly summarize and interact with data. Previously the only supported faceting technique was column faceting, but 0.28 introduces two powerful new capibilities: facet-by-JSON-array and the ability to define further facet types using plugins.
+Datasette :ref:`facets` provide an intuitive way to quickly summarize and interact with data. Previously the only supported faceting technique was column faceting, but 0.28 introduces two powerful new capabilities: facet-by-JSON-array and the ability to define further facet types using plugins.
 
 Facet by array (`#359 <https://github.com/simonw/datasette/issues/359>`__) is only available if your SQLite installation provides the ``json1`` extension. Datasette will automatically detect columns that contain JSON arrays of values and offer a faceting interface against those columns - useful for modelling things like tags without needing to break them out into a new table. See :ref:`facet_by_json_array` for more.
 
@@ -42,7 +240,7 @@ The new :ref:`plugin_register_facet_classes` plugin hook (`#445 <https://github.
 datasette publish cloudrun
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-`Google Cloud Run <https://cloud.google.com/run/>`__ is a brand new serverless hosting platform from Google, which allows you to build a Docker container which will run only when HTTP traffic is recieved and will shut down (and hence cost you nothing) the rest of the time. It's similar to Zeit's Now v1 Docker hosting platform which sadly is `no longer accepting signups <https://hyperion.alpha.spectrum.chat/zeit/now/cannot-create-now-v1-deployments~d206a0d4-5835-4af5-bb5c-a17f0171fb25?m=MTU0Njk2NzgwODM3OA==>`__ from new users.
+`Google Cloud Run <https://cloud.google.com/run/>`__ is a brand new serverless hosting platform from Google, which allows you to build a Docker container which will run only when HTTP traffic is received and will shut down (and hence cost you nothing) the rest of the time. It's similar to Zeit's Now v1 Docker hosting platform which sadly is `no longer accepting signups <https://hyperion.alpha.spectrum.chat/zeit/now/cannot-create-now-v1-deployments~d206a0d4-5835-4af5-bb5c-a17f0171fb25?m=MTU0Njk2NzgwODM3OA==>`__ from new users.
 
 The new ``datasette publish cloudrun`` command was contributed by Romain Primet (`#434 <https://github.com/simonw/datasette/pull/434>`__) and publishes selected databases to a new Datasette instance running on Google Cloud Run.
 
@@ -481,7 +679,7 @@ Mostly new work on the :ref:`plugins` mechanism: plugins can now bundle static a
 - Longer time limit for test_paginate_compound_keys
 
   It was failing intermittently in Travis - see `#209 <https://github.com/simonw/datasette/issues/209>`_
-- Use application/octet-stream for downloadable databses
+- Use application/octet-stream for downloadable databases
 - Updated PyPI classifiers
 - Updated PyPI link to pypi.org
 

docs/contributing.rst

@@ -18,7 +18,7 @@ General guidelines
 Setting up a development environment
 ------------------------------------
 
-If you have Python 3.5 or higher installed on your computer (on OS X the easiest way to do this `is using homebrew <https://docs.python-guide.org/starting/install3/osx/>`__) you can install an editable copy of Datasette using the following steps.
+If you have Python 3.6 or higher installed on your computer (on OS X the easiest way to do this `is using homebrew <https://docs.python-guide.org/starting/install3/osx/>`__) you can install an editable copy of Datasette using the following steps.
 
 If you want to use GitHub to publish your changes, first `create a fork of datasette <https://github.com/simonw/datasette/fork>`__ under your own GitHub account.
 
@@ -150,4 +150,7 @@ Wait long enough for Travis to build and deploy the demo version of that commit
     git tag 0.25.2
     git push --tags
 
-Once the release is out, you can manually update https://github.com/simonw/datasette/releases
+Final steps once the release has deployed to https://pypi.org/project/datasette/
+
+* Manually post the new release to GitHub releases: https://github.com/simonw/datasette/releases
+* Manually kick off a build of the `stable` branch on Read The Docs: https://readthedocs.org/projects/datasette/builds/

docs/custom_templates.rst

@@ -102,6 +102,8 @@ database column they are representing, for example::
         </tbody>
     </table>
 
+.. _customization_custom_templates:
+
 Custom templates
 ----------------
 

docs/datasette-publish-cloudrun-help.txt

@@ -3,22 +3,26 @@ $ datasette publish cloudrun --help
 Usage: datasette publish cloudrun [OPTIONS] [FILES]...
 
 Options:
-  -m, --metadata FILENAME   Path to JSON file containing metadata to publish
-  --extra-options TEXT      Extra options to pass to datasette serve
-  --branch TEXT             Install datasette from a GitHub branch e.g. master
-  --template-dir DIRECTORY  Path to directory containing custom templates
-  --plugins-dir DIRECTORY   Path to directory containing custom plugins
-  --static STATIC MOUNT     mountpoint:path-to-directory for serving static files
-  --install TEXT            Additional packages (e.g. plugins) to install
-  --version-note TEXT       Additional note to show on /-/versions
-  --title TEXT              Title for metadata
-  --license TEXT            License label for metadata
-  --license_url TEXT        License URL for metadata
-  --source TEXT             Source label for metadata
-  --source_url TEXT         Source URL for metadata
-  --about TEXT              About label for metadata
-  --about_url TEXT          About URL for metadata
-  -n, --name TEXT           Application name to use when building
-  --service TEXT            Cloud Run service to deploy (or over-write)
-  --spatialite              Enable SpatialLite extension
-  --help                    Show this message and exit.
+  -m, --metadata FILENAME         Path to JSON file containing metadata to publish
+  --extra-options TEXT            Extra options to pass to datasette serve
+  --branch TEXT                   Install datasette from a GitHub branch e.g. master
+  --template-dir DIRECTORY        Path to directory containing custom templates
+  --plugins-dir DIRECTORY         Path to directory containing custom plugins
+  --static STATIC MOUNT           mountpoint:path-to-directory for serving static files
+  --install TEXT                  Additional packages (e.g. plugins) to install
+  --plugin-secret <TEXT TEXT TEXT>...
+                                  Secrets to pass to plugins, e.g. --plugin-secret
+                                  datasette-auth-github client_id xxx
+  --version-note TEXT             Additional note to show on /-/versions
+  --title TEXT                    Title for metadata
+  --license TEXT                  License label for metadata
+  --license_url TEXT              License URL for metadata
+  --source TEXT                   Source label for metadata
+  --source_url TEXT               Source URL for metadata
+  --about TEXT                    About label for metadata
+  --about_url TEXT                About URL for metadata
+  -n, --name TEXT                 Application name to use when building
+  --service TEXT                  Cloud Run service to deploy (or over-write)
+  --spatialite                    Enable SpatialLite extension
+  --show-files                    Output the generated Dockerfile and metadata.json
+  --help                          Show this message and exit.

docs/datasette-publish-heroku-help.txt

@@ -3,20 +3,23 @@ $ datasette publish heroku --help
 Usage: datasette publish heroku [OPTIONS] [FILES]...
 
 Options:
-  -m, --metadata FILENAME   Path to JSON file containing metadata to publish
-  --extra-options TEXT      Extra options to pass to datasette serve
-  --branch TEXT             Install datasette from a GitHub branch e.g. master
-  --template-dir DIRECTORY  Path to directory containing custom templates
-  --plugins-dir DIRECTORY   Path to directory containing custom plugins
-  --static STATIC MOUNT     mountpoint:path-to-directory for serving static files
-  --install TEXT            Additional packages (e.g. plugins) to install
-  --version-note TEXT       Additional note to show on /-/versions
-  --title TEXT              Title for metadata
-  --license TEXT            License label for metadata
-  --license_url TEXT        License URL for metadata
-  --source TEXT             Source label for metadata
-  --source_url TEXT         Source URL for metadata
-  --about TEXT              About label for metadata
-  --about_url TEXT          About URL for metadata
-  -n, --name TEXT           Application name to use when deploying
-  --help                    Show this message and exit.
+  -m, --metadata FILENAME         Path to JSON file containing metadata to publish
+  --extra-options TEXT            Extra options to pass to datasette serve
+  --branch TEXT                   Install datasette from a GitHub branch e.g. master
+  --template-dir DIRECTORY        Path to directory containing custom templates
+  --plugins-dir DIRECTORY         Path to directory containing custom plugins
+  --static STATIC MOUNT           mountpoint:path-to-directory for serving static files
+  --install TEXT                  Additional packages (e.g. plugins) to install
+  --plugin-secret <TEXT TEXT TEXT>...
+                                  Secrets to pass to plugins, e.g. --plugin-secret
+                                  datasette-auth-github client_id xxx
+  --version-note TEXT             Additional note to show on /-/versions
+  --title TEXT                    Title for metadata
+  --license TEXT                  License label for metadata
+  --license_url TEXT              License URL for metadata
+  --source TEXT                   Source label for metadata
+  --source_url TEXT               Source URL for metadata
+  --about TEXT                    About label for metadata
+  --about_url TEXT                About URL for metadata
+  -n, --name TEXT                 Application name to use when deploying
+  --help                          Show this message and exit.

docs/datasette-publish-nowv1-help.txt

@@ -3,24 +3,28 @@ $ datasette publish nowv1 --help
 Usage: datasette publish nowv1 [OPTIONS] [FILES]...
 
 Options:
-  -m, --metadata FILENAME   Path to JSON file containing metadata to publish
-  --extra-options TEXT      Extra options to pass to datasette serve
-  --branch TEXT             Install datasette from a GitHub branch e.g. master
-  --template-dir DIRECTORY  Path to directory containing custom templates
-  --plugins-dir DIRECTORY   Path to directory containing custom plugins
-  --static STATIC MOUNT     mountpoint:path-to-directory for serving static files
-  --install TEXT            Additional packages (e.g. plugins) to install
-  --version-note TEXT       Additional note to show on /-/versions
-  --title TEXT              Title for metadata
-  --license TEXT            License label for metadata
-  --license_url TEXT        License URL for metadata
-  --source TEXT             Source label for metadata
-  --source_url TEXT         Source URL for metadata
-  --about TEXT              About label for metadata
-  --about_url TEXT          About URL for metadata
-  -n, --name TEXT           Application name to use when deploying
-  --force                   Pass --force option to now
-  --token TEXT              Auth token to use for deploy
-  --alias TEXT              Desired alias e.g. yoursite.now.sh
-  --spatialite              Enable SpatialLite extension
-  --help                    Show this message and exit.
+  -m, --metadata FILENAME         Path to JSON file containing metadata to publish
+  --extra-options TEXT            Extra options to pass to datasette serve
+  --branch TEXT                   Install datasette from a GitHub branch e.g. master
+  --template-dir DIRECTORY        Path to directory containing custom templates
+  --plugins-dir DIRECTORY         Path to directory containing custom plugins
+  --static STATIC MOUNT           mountpoint:path-to-directory for serving static files
+  --install TEXT                  Additional packages (e.g. plugins) to install
+  --plugin-secret <TEXT TEXT TEXT>...
+                                  Secrets to pass to plugins, e.g. --plugin-secret
+                                  datasette-auth-github client_id xxx
+  --version-note TEXT             Additional note to show on /-/versions
+  --title TEXT                    Title for metadata
+  --license TEXT                  License label for metadata
+  --license_url TEXT              License URL for metadata
+  --source TEXT                   Source label for metadata
+  --source_url TEXT               Source URL for metadata
+  --about TEXT                    About label for metadata
+  --about_url TEXT                About URL for metadata
+  -n, --name TEXT                 Application name to use when deploying
+  --force                         Pass --force option to now
+  --token TEXT                    Auth token to use for deploy
+  --alias TEXT                    Desired alias e.g. yoursite.now.sh
+  --spatialite                    Enable SpatialLite extension
+  --show-files                    Output the generated Dockerfile and metadata.json
+  --help                          Show this message and exit.

docs/datasette-serve-help.txt

@@ -6,8 +6,11 @@ Usage: datasette serve [OPTIONS] [FILES]...
 
 Options:
   -i, --immutable PATH      Database files to open in immutable mode
-  -h, --host TEXT           host for server, defaults to 127.0.0.1
-  -p, --port INTEGER        port for server, defaults to 8001
+  -h, --host TEXT           Host for server. Defaults to 127.0.0.1 which means only
+                            connections from the local machine will be allowed. Use
+                            0.0.0.0 to listen to all IPs and allow access from other
+                            machines.
+  -p, --port INTEGER        Port for server, defaults to 8001
   --debug                   Enable debug mode - useful for development
   --reload                  Automatically reload if database or code change detected -
                             useful for development

docs/ecosystem.rst

@@ -70,6 +70,11 @@ datasette-vega
 
 `datasette-vega <https://github.com/simonw/datasette-vega>`__ exposes the powerful  `Vega <https://vega.github.io/vega/>`__ charting library, allowing you to construct line, bar and scatter charts against your data and share links to your visualizations.
 
+datasette-auth-github
+---------------------
+
+`datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ adds an authentication layer to Datasette. Users will have to sign in using their GitHub account before they can view data or interact with Datasette. You can also use it to restrict access to specific GitHub users, or to members of specified GitHub `organizations <https://help.github.com/en/articles/about-organizations>`__ or `teams <https://help.github.com/en/articles/organizing-members-into-teams>`__.
+
 datasette-json-html
 -------------------
 
@@ -114,3 +119,8 @@ datasette-bplist
 ----------------
 
 `datasette-bplist <https://github.com/simonw/datasette-bplist>`__ provides tools for working with Apple's binary plist format embedded in SQLite database tables. If you use OS X you already have dozens of SQLite databases hidden away in your ``~/Library`` folder that include data in this format - this plugin allows you to view the decoded data and run SQL queries against embedded values using a ``bplist_to_json(value)`` custom SQL function.
+
+datasette-cors
+--------------
+
+`datasette-cors <https://github.com/simonw/datasette-cors>`__ allows you to configure `CORS headers <https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS>`__ for your Datasette instance. You can use this to enable JavaScript running on a whitelisted set of domains to make ``fetch()`` calls to the JSON API provided by your Datasette instance.

docs/facets.rst

@@ -129,17 +129,6 @@ The performance of facets can be greatly improved by adding indexes on the colum
     Enter ".help" for usage hints.
     sqlite> CREATE INDEX Food_Trucks_state ON Food_Trucks("state");
 
-.. _facet_by_m2m:
-
-Facet by many-to-many
----------------------
-
-Datasette can detect many-to-many SQL tables - defined as SQL tables which have foreign key relationships to two other tables.
-
-If a many-to-many table exists pointing at the table you are currently viewing, Datasette will suggest you facet the table based on that relationship.
-
-Example here: `latest.datasette.io/fixtures/roadside_attractions?_facet_m2m=attraction_characteristic <https://latest.datasette.io/fixtures/roadside_attractions?_facet_m2m=attraction_characteristic>`__
-
 .. _facet_by_json_array:
 
 Facet by JSON array

docs/getting_started.rst

@@ -25,7 +25,7 @@ Glitch allows you to "remix" any project to create your own copy and start editi
 .. image:: https://cdn.glitch.com/2703baf2-b643-4da7-ab91-7ee2a2d00b5b%2Fremix-button.svg
    :target: https://glitch.com/edit/#!/remix/datasette-csvs
 
-Find a CSV file and drag it onto the Glitch file explorer panel - ``datasette-csvs`` will automatically convert it to a SQLite database (using `csvs-to-sqlite <https://github.com/simonw/csvs-to-sqlite>`__) and allow you to start exploring it using Datasette.
+Find a CSV file and drag it onto the Glitch file explorer panel - ``datasette-csvs`` will automatically convert it to a SQLite database (using `sqlite-utils <https://github.com/simonw/sqlite-utils>`__) and allow you to start exploring it using Datasette.
 
 If your CSV file has a ``latitude`` and ``longitude`` column you can visualize it on a map by uncommenting the ``datasette-cluster-map`` line in the ``requirements.txt`` file using the Glitch file editor.
 

docs/installation.rst

@@ -69,16 +69,19 @@ You can now run the new custom image like so::
 You can confirm that the plugins are installed by visiting
 http://127.0.0.1:8001/-/plugins
 
-
 Install using pip
 -----------------
 
-To run Datasette without Docker you will need Python 3.5 or higher.
+To run Datasette without Docker you will need Python 3.6 or higher.
 
 You can install Datasette and its dependencies using ``pip``::
 
     pip install datasette
 
+The last version to support Python 3.5 was 0.30.2 - you can install that version like so::
+
+    pip install datasette==0.30.2
+
 If you want to install Datasette in its own virtual environment, use this::
 
     python -mvenv datasette-venv

docs/introspection.rst

@@ -90,6 +90,8 @@ Shows the :ref:`config` options for this instance of Datasette. `Config example
         "sql_time_limit_ms": 1000
     }
 
+.. _JsonDataView_databases:
+
 /-/databases
 ------------
 
@@ -105,3 +107,26 @@ Shows currently attached databases. `Databases example <https://latest.datasette
             "size": 225280
         }
     ]
+
+.. _JsonDataView_threads:
+
+/-/threads
+----------
+
+Shows details of threads. `Threads example <https://latest.datasette.io/-/threads>`_::
+
+    {
+        "num_threads": 2,
+        "threads": [
+            {
+                "daemon": false,
+                "ident": 4759197120,
+                "name": "MainThread"
+            },
+            {
+                "daemon": true,
+                "ident": 123145319682048,
+                "name": "Thread-1"
+            },
+        ]
+    }

docs/json_api.rst

@@ -228,6 +228,9 @@ You can filter the data returned by the table based on column values using a que
 
     ``?column__in=["value","value,with,commas"]``
 
+``?column__notin=value1,value2,value3``
+    Rows where column does not match any of the provided values. The inverse of ``__in=``. Also supports JSON arrays.
+
 ``?column__arraycontains=value``
     Works against columns that contain JSON arrays - matches if any of the values in that array match.
 
@@ -318,15 +321,6 @@ Special table arguments
 
     Here's `an example <https://latest.datasette.io/fixtures/roadside_attractions?_through={%22table%22:%22roadside_attraction_characteristics%22,%22column%22:%22characteristic_id%22,%22value%22:%221%22}>`__.
 
-
-``?_group_count=COLUMN``
-    Executes a SQL query that returns a count of the number of rows matching
-    each unique value in that column, with the most common ordered first.
-
-``?_group_count=COLUMN1&_group_count=column2``
-    You can pass multiple ``_group_count`` columns to return counts against
-    unique combinations of those columns.
-
 ``?_next=TOKEN``
     Pagination by continuation token - pass the token that was returned in the
     ``"next"`` property by the previous page.

docs/metadata.rst

@@ -62,7 +62,7 @@ Each of the top-level metadata fields can be used at the database and table leve
 Source, license and about
 -------------------------
 
-The three visible metadata fields you can apply to everything, specific databases or specific tables are source, license and about. All three are optionaly.
+The three visible metadata fields you can apply to everything, specific databases or specific tables are source, license and about. All three are optional.
 
 **source** and **source_url** should be used to indicate where the underlying data came from.
 

docs/performance.rst

@@ -7,6 +7,8 @@ Datasette runs on top of SQLite, and SQLite has excellent performance.  For smal
 
 That said, there are a number of tricks you can use to improve Datasette's performance.
 
+.. _performance_immutable_mode:
+
 Immutable mode
 --------------
 
@@ -37,7 +39,7 @@ Then later you can start Datasette against the ``counts.json`` file and use it t
 
     datasette -i data.db --inspect-file=counts.json
 
-You need to use the ``-i`` immutable mode agaist the databse file here or the counts from the JSON file will be ignored.
+You need to use the ``-i`` immutable mode against the databse file here or the counts from the JSON file will be ignored.
 
 You will rarely need to use this optimization in every-day use, but several of the ``datasette publish`` commands described in :ref:`publishing` use this optimization for better performance when deploying a database file to a hosting provider.
 

docs/plugins.rst

@@ -219,6 +219,8 @@ Here is an example of some plugin configuration for a specific table::
 
 This tells the ``datasette-cluster-map`` column which latitude and longitude columns should be used for a table called ``Street_Tree_List`` inside a database file called ``sf-trees.db``.
 
+.. _plugins_configuration_secret:
+
 Secret configuration values
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -236,7 +238,6 @@ Any values embedded in ``metadata.json`` will be visible to anyone who views the
         }
     }
 
-
 **As values in separate files**. Your secrets can also live in files on disk. To specify a secret should be read from a file, provide the full file path like this::
 
     {
@@ -249,6 +250,14 @@ Any values embedded in ``metadata.json`` will be visible to anyone who views the
         }
     }
 
+If you are publishing your data using the :ref:`datasette publish <cli_publish>` family of commands, you can use the ``--plugin-secret`` option to set these secrets at publish time. For example, using Heroku you might run the following command::
+
+    $ datasette publish heroku my_database.db \
+        --name my-heroku-app-demo \
+        --install=datasette-auth-github \
+        --plugin-secret datasette-auth-github client_id your_client_id \
+        --plugin-secret datasette-auth-github client_secret your_client_secret
+
 Writing plugins that accept configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -433,7 +442,7 @@ you have one:
     @hookimpl
     def extra_js_urls():
         return [
-            '/-/static-plugins/your_plugin/app.js'
+            '/-/static-plugins/your-plugin/app.js'
         ]
 
 .. _plugin_hook_publish_subcommand:
@@ -620,7 +629,9 @@ Function that returns a dictionary
     If you return a function it will be executed. If it returns a dictionary those values will will be merged into the template context.
 
 Function that returns an awaitable function that returns a dictionary
-    You can also return a function which returns an awaitable function which returns a dictionary. This means you can execute additional SQL queries using ``datasette.execute()``.
+    You can also return a function which returns an awaitable function which returns a dictionary.
+
+Datasette runs Jinja2 in `async mode <https://jinja.palletsprojects.com/en/2.10.x/api/#async-support>`__, which means you can add awaitable functions to the template scope and they will be automatically awaited when they are rendered by the template.
 
 Here's an example plugin that returns an authentication object from the ASGI scope:
 
@@ -632,20 +643,19 @@ Here's an example plugin that returns an authentication object from the ASGI sco
             "auth": request.scope.get("auth")
         }
 
-And here's an example which returns the current version of SQLite:
+And here's an example which adds a ``sql_first(sql_query)`` function which executes a SQL statement and returns the first column of the first row of results:
 
 .. code-block:: python
 
     @hookimpl
-    def extra_template_vars(datasette):
-        async def inner():
-            first_db = list(datasette.databases.keys())[0]
-            return {
-                "sqlite_version": (
-                    await datasette.execute(first_db, "select sqlite_version()")
-                ).rows[0][0]
-            }
-        return inner
+    def extra_template_vars(datasette, database):
+        async def sql_first(sql, dbname=None):
+            dbname = dbname or database or next(iter(datasette.databases.keys()))
+            return (await datasette.execute(dbname, sql)).rows[0][0]
+
+You can then use the new function in a template like so::
+
+    SQLite version: {{ sql_first("select sqlite_version()") }}
 
 .. _plugin_register_output_renderer:
 
@@ -712,9 +722,9 @@ Each Facet subclass implements a new type of facet operation. The class should l
         # This key must be unique across all facet classes:
         type = "special"
 
-        async def suggest(self, sql, params, filtered_table_rows_count):
+        async def suggest(self):
+            # Use self.sql and self.params to suggest some facets
             suggested_facets = []
-            # Perform calculations to suggest facets
             suggested_facets.append({
                 "name": column, # Or other unique name
                 # Construct the URL that will enable this facet:
@@ -726,8 +736,9 @@ Each Facet subclass implements a new type of facet operation. The class should l
             })
             return suggested_facets
 
-        async def facet_results(self, sql, params):
-            # This should execute the facet operation and return results
+        async def facet_results(self):
+            # This should execute the facet operation and return results, again
+            # using self.sql and self.params as the starting point
             facet_results = {}
             facets_timed_out = []
             # Do some calculations here...
@@ -752,7 +763,7 @@ Each Facet subclass implements a new type of facet operation. The class should l
 
             return facet_results, facets_timed_out
 
-See ``datasette/facets.py`` for examples of how these classes can work.
+See `datasette/facets.py <https://github.com/simonw/datasette/blob/master/datasette/facets.py>`__ for examples of how these classes can work.
 
 The plugin hook can then be used to register the new facet class like this:
 

docs/publish.rst

@@ -6,6 +6,8 @@
 
 Datasette includes tools for publishing and deploying your data to the internet. The ``datasette publish`` command will deploy a new Datasette instance containing your databases directly to a Heroku, Google Cloud or Zeit Now hosting account. You can also use ``datasette package`` to create a Docker image that bundles your databases together with the datasette application that is used to serve them.
 
+.. _cli_publish:
+
 datasette publish
 =================
 
@@ -41,14 +43,16 @@ You will first need to install and configure the Google Cloud CLI tools by follo
 
 You can then publish a database to Google Cloud Run using the following command::
 
-    datasette publish cloudrun mydatabase.db
+    datasette publish cloudrun mydatabase.db --service=my-database
+
+A Cloud Run **service** is a single hosted application. The service name you specify will be used as part of the Cloud Run URL. If you deploy to a service name that you have used in the past your new deployment will replace the previous one.
+
+If you omit the ``--service`` option you will be asked to pick a service name interactively during the deploy.
 
 You may need to interact with prompts from the tool. Once it has finished it will output a URL like this one::
 
-    Service [datasette] revision [datasette-00001] has been deployed
-    and is serving traffic at https://datasette-j7hipcg4aq-uc.a.run.app
-
-During the deployment the tool will prompt you for the name of your service. You can reuse an existing name to replace your previous deployment with your new version, or pick a new name to deploy to a new URL.
+    Service [my-service] revision [my-service-00001] has been deployed
+    and is serving traffic at https://my-service-j7hipcg4aq-uc.a.run.app
 
 .. literalinclude:: datasette-publish-cloudrun-help.txt
 
@@ -79,6 +83,8 @@ You can also use custom domains, if you `first register them with Zeit Now <http
 
 .. literalinclude:: datasette-publish-nowv1-help.txt
 
+.. _publish_custom_metadata_and_plugins:
+
 Custom metadata and plugins
 ---------------------------
 
@@ -86,19 +92,26 @@ Custom metadata and plugins
 
 You can define your own :ref:`metadata` and deploy that with your instance like so::
 
-    datasette publish nowv1 mydatabase.db -m metadata.json
+    datasette publish cloudrun --service=my-service mydatabase.db -m metadata.json
 
 If you just want to set the title, license or source information you can do that directly using extra options to ``datasette publish``::
 
-    datasette publish nowv1 mydatabase.db \
+    datasette publish cloudrun mydatabase.db --service=my-service \
         --title="Title of my database" \
         --source="Where the data originated" \
         --source_url="http://www.example.com/"
 
 You can also specify plugins you would like to install. For example, if you want to include the `datasette-vega <https://github.com/simonw/datasette-vega>`_ visualization plugin you can use the following::
 
-    datasette publish nowv1 mydatabase.db --install=datasette-vega
+    datasette publish cloudrun mydatabase.db --service=my-service --install=datasette-vega
 
+If a plugin has any :ref:`plugins_configuration_secret` you can use the ``--plugin-secret`` option to set those secrets at publish time. For example, using Heroku with `datasette-auth-github <https://github.com/simonw/datasette-auth-github>`__ you might run the following command::
+
+    $ datasette publish heroku my_database.db \
+        --name my-heroku-app-demo \
+        --install=datasette-auth-github \
+        --plugin-secret datasette-auth-github client_id your_client_id \
+        --plugin-secret datasette-auth-github client_secret your_client_secret
 
 datasette package
 =================

setup.py

@@ -22,11 +22,6 @@ def get_version():
     return g["__version__"]
 
 
-# Only install black on Python 3.6 or higher
-maybe_black = []
-if sys.version_info > (3, 6):
-    maybe_black = ["black"]
-
 setup(
     name="datasette",
     version=versioneer.get_version(),
@@ -42,12 +37,12 @@ setup(
     include_package_data=True,
     install_requires=[
         "click~=7.0",
-        "click-default-group~=1.2.1",
-        "Jinja2~=2.10.1",
-        "hupper~=1.0",
-        "pint~=0.8.1",
-        "pluggy~=0.12.0",
-        "uvicorn~=0.8.1",
+        "click-default-group~=1.2.2",
+        "Jinja2~=2.10.3",
+        "hupper~=1.9",
+        "pint~=0.9",
+        "pluggy~=0.13.0",
+        "uvicorn~=0.10.4",
         "aiofiles~=0.4.0",
     ],
     entry_points="""
@@ -58,13 +53,13 @@ setup(
     extras_require={
         "docs": ["sphinx_rtd_theme", "sphinx-autobuild"],
         "test": [
-            "pytest~=5.0.0",
+            "pytest~=5.2.2",
             "pytest-asyncio~=0.10.0",
-            "aiohttp~=3.5.3",
-            "beautifulsoup4~=4.6.1",
-            "asgiref~=3.1.2",
-        ]
-        + maybe_black,
+            "aiohttp~=3.6.2",
+            "beautifulsoup4~=4.8.1",
+            "asgiref~=3.2.3",
+            "black~=19.10b0",
+        ],
     },
     tests_require=["datasette[test]"],
     classifiers=[
@@ -74,8 +69,8 @@ setup(
         "Intended Audience :: End Users/Desktop",
         "Topic :: Database",
         "License :: OSI Approved :: Apache Software License",
+        "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.6",
-        "Programming Language :: Python :: 3.5",
     ],
 )

tests/fixtures.py

@@ -12,7 +12,7 @@ import sys
 import string
 import tempfile
 import time
-from urllib.parse import unquote
+from urllib.parse import unquote, quote
 
 
 # This temp file is used by one of the plugin config tests
@@ -49,18 +49,20 @@ class TestClient:
         if "?" in path:
             path, _, query_string = path.partition("?")
             query_string = query_string.encode("utf8")
-        instance = ApplicationCommunicator(
-            self.asgi_app,
-            {
-                "type": "http",
-                "http_version": "1.0",
-                "method": method,
-                "path": unquote(path),
-                "raw_path": path.encode("ascii"),
-                "query_string": query_string,
-                "headers": [[b"host", b"localhost"]],
-            },
-        )
+        if "%" in path:
+            raw_path = path.encode("latin-1")
+        else:
+            raw_path = quote(path, safe="/:,").encode("latin-1")
+        scope = {
+            "type": "http",
+            "http_version": "1.0",
+            "method": method,
+            "path": unquote(path),
+            "raw_path": raw_path,
+            "query_string": query_string,
+            "headers": [[b"host", b"localhost"]],
+        }
+        instance = ApplicationCommunicator(self.asgi_app, scope)
         await instance.send_input({"type": "http.request"})
         # First message back should be response.start with headers and status
         messages = []
@@ -172,14 +174,21 @@ def app_client_no_files():
 @pytest.fixture(scope="session")
 def app_client_two_attached_databases():
     yield from make_app_client(
-        extra_databases={"extra_database.db": EXTRA_DATABASE_SQL}
+        extra_databases={"extra database.db": EXTRA_DATABASE_SQL}
+    )
+
+
+@pytest.fixture(scope="session")
+def app_client_conflicting_database_names():
+    yield from make_app_client(
+        extra_databases={"foo.db": EXTRA_DATABASE_SQL, "foo-bar.db": EXTRA_DATABASE_SQL}
     )
 
 
 @pytest.fixture(scope="session")
 def app_client_two_attached_databases_one_immutable():
     yield from make_app_client(
-        is_immutable=True, extra_databases={"extra_database.db": EXTRA_DATABASE_SQL}
+        is_immutable=True, extra_databases={"extra database.db": EXTRA_DATABASE_SQL}
     )
 
 
@@ -291,6 +300,7 @@ METADATA = {
                 },
             },
             "queries": {
+                "𝐜𝐢𝐭𝐢𝐞𝐬": "select id, name from facet_cities order by id limit 1;",
                 "pragma_cache_size": "PRAGMA cache_size;",
                 "neighborhood_search": {
                     "sql": """
@@ -436,13 +446,19 @@ def render_cell(value, database):
 
 @hookimpl
 def extra_template_vars(template, database, table, view_name, request, datasette):
+    async def query_database(sql):
+        first_db = list(datasette.databases.keys())[0]
+        return (
+            await datasette.execute(first_db, sql)
+        ).rows[0][0]
     async def inner():
         return {
             "extra_template_vars_from_awaitable": json.dumps({
                 "template": template,
                 "scope_path": request.scope["path"],
                 "awaitable": True,
-            }, default=lambda b: b.decode("utf8"))
+            }, default=lambda b: b.decode("utf8")),
+            "query_database": query_database,
         }
     return inner
 
@@ -504,6 +520,7 @@ CREATE TABLE compound_three_primary_keys (
   content text,
   PRIMARY KEY (pk1, pk2, pk3)
 );
+CREATE INDEX idx_compound_three_primary_keys_content ON compound_three_primary_keys(content);
 
 CREATE TABLE foreign_key_references (
   pk varchar(30) primary key,
@@ -651,26 +668,27 @@ CREATE TABLE facetable (
     city_id integer,
     neighborhood text,
     tags text,
+    complex_array text,
     FOREIGN KEY ("city_id") REFERENCES [facet_cities](id)
 );
 INSERT INTO facetable
-    (created, planet_int, on_earth, state, city_id, neighborhood, tags)
+    (created, planet_int, on_earth, state, city_id, neighborhood, tags, complex_array)
 VALUES
-    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Mission', '["tag1", "tag2"]'),
-    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Dogpatch', '["tag1", "tag3"]'),
-    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'SOMA', '[]'),
-    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Tenderloin', '[]'),
-    ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Bernal Heights', '[]'),
-    ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Hayes Valley', '[]'),
-    ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Hollywood', '[]'),
-    ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Downtown', '[]'),
-    ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Los Feliz', '[]'),
-    ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Koreatown', '[]'),
-    ("2019-01-16 08:00:00", 1, 1, 'MI', 3, 'Downtown', '[]'),
-    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Greektown', '[]'),
-    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Corktown', '[]'),
-    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Mexicantown', '[]'),
-    ("2019-01-17 08:00:00", 2, 0, 'MC', 4, 'Arcadia Planitia', '[]')
+    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Mission', '["tag1", "tag2"]', '[{"foo": "bar"}]'),
+    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Dogpatch', '["tag1", "tag3"]', '[]'),
+    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'SOMA', '[]', '[]'),
+    ("2019-01-14 08:00:00", 1, 1, 'CA', 1, 'Tenderloin', '[]', '[]'),
+    ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Bernal Heights', '[]', '[]'),
+    ("2019-01-15 08:00:00", 1, 1, 'CA', 1, 'Hayes Valley', '[]', '[]'),
+    ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Hollywood', '[]', '[]'),
+    ("2019-01-15 08:00:00", 1, 1, 'CA', 2, 'Downtown', '[]', '[]'),
+    ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Los Feliz', '[]', '[]'),
+    ("2019-01-16 08:00:00", 1, 1, 'CA', 2, 'Koreatown', '[]', '[]'),
+    ("2019-01-16 08:00:00", 1, 1, 'MI', 3, 'Downtown', '[]', '[]'),
+    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Greektown', '[]', '[]'),
+    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Corktown', '[]', '[]'),
+    ("2019-01-17 08:00:00", 1, 1, 'MI', 3, 'Mexicantown', '[]', '[]'),
+    ("2019-01-17 08:00:00", 2, 0, 'MC', 4, 'Arcadia Planitia', '[]', '[]')
 ;
 
 CREATE TABLE binary_data (
@@ -743,6 +761,7 @@ INSERT INTO primary_key_multiple_columns VALUES (1, 'hey', 'world');
 INSERT INTO primary_key_multiple_columns_explicit_label VALUES (1, 'hey', 'world2');
 
 INSERT INTO foreign_key_references VALUES (1, 1, 1);
+INSERT INTO foreign_key_references VALUES (2, null, null);
 
 INSERT INTO complex_foreign_keys VALUES (1, 1, 2, 1);
 INSERT INTO custom_foreign_key_label VALUES (1, 1);

tests/test_api.py

@@ -6,7 +6,9 @@ from .fixtures import (  # noqa
     app_client_shorter_time_limit,
     app_client_larger_cache_size,
     app_client_returned_rows_matches_page_size,
+    app_client_two_attached_databases,
     app_client_two_attached_databases_one_immutable,
+    app_client_conflicting_database_names,
     app_client_with_cors,
     app_client_with_dot,
     generate_compound_rows,
@@ -194,6 +196,7 @@ def test_database_page(app_client):
                 "city_id",
                 "neighborhood",
                 "tags",
+                "complex_array",
             ],
             "primary_keys": ["pk"],
             "count": 15,
@@ -214,7 +217,7 @@ def test_database_page(app_client):
             "name": "foreign_key_references",
             "columns": ["pk", "foreign_key_with_label", "foreign_key_with_no_label"],
             "primary_keys": ["pk"],
-            "count": 1,
+            "count": 2,
             "hidden": False,
             "fts_table": None,
             "foreign_keys": {
@@ -608,7 +611,8 @@ def test_table_json(app_client):
     assert response.status == 200
     data = response.json
     assert (
-        data["query"]["sql"] == "select * from simple_primary_key order by id limit 51"
+        data["query"]["sql"]
+        == "select id, content from simple_primary_key order by id limit 51"
     )
     assert data["query"]["params"] == {}
     assert data["rows"] == [
@@ -1028,15 +1032,25 @@ def test_table_filter_queries_multiple_of_same_type(app_client):
 def test_table_filter_json_arraycontains(app_client):
     response = app_client.get("/fixtures/facetable.json?tags__arraycontains=tag1")
     assert [
-        [1, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Mission", '["tag1", "tag2"]'],
-        [2, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Dogpatch", '["tag1", "tag3"]'],
+        [
+            1,
+            "2019-01-14 08:00:00",
+            1,
+            1,
+            "CA",
+            1,
+            "Mission",
+            '["tag1", "tag2"]',
+            '[{"foo": "bar"}]',
+        ],
+        [2, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Dogpatch", '["tag1", "tag3"]', "[]"],
     ] == response.json["rows"]
 
 
 def test_table_filter_extra_where(app_client):
     response = app_client.get("/fixtures/facetable.json?_where=neighborhood='Dogpatch'")
     assert [
-        [2, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Dogpatch", '["tag1", "tag3"]']
+        [2, "2019-01-14 08:00:00", 1, 1, "CA", 1, "Dogpatch", '["tag1", "tag3"]', "[]"]
     ] == response.json["rows"]
 
 
@@ -1106,6 +1120,15 @@ def test_row(app_client):
     assert [{"id": "1", "content": "hello"}] == response.json["rows"]
 
 
+def test_row_format_in_querystring(app_client):
+    # regression test for https://github.com/simonw/datasette/issues/563
+    response = app_client.get(
+        "/fixtures/simple_primary_key/1?_format=json&_shape=objects"
+    )
+    assert response.status == 200
+    assert [{"id": "1", "content": "hello"}] == response.json["rows"]
+
+
 def test_row_strange_table_name(app_client):
     response = app_client.get(
         "/fixtures/table%2Fwith%2Fslashes.csv/3.json?_shape=objects"
@@ -1166,7 +1189,7 @@ def test_databases_json(app_client_two_attached_databases_one_immutable):
     databases = response.json
     assert 2 == len(databases)
     extra_database, fixtures_database = databases
-    assert "extra_database" == extra_database["name"]
+    assert "extra database" == extra_database["name"]
     assert None == extra_database["hash"]
     assert True == extra_database["is_mutable"]
     assert False == extra_database["is_memory"]
@@ -1235,7 +1258,7 @@ def test_config_json(app_client):
 
 
 def test_page_size_matching_max_returned_rows(
-    app_client_returned_rows_matches_page_size
+    app_client_returned_rows_matches_page_size,
 ):
     fetched = []
     path = "/fixtures/no_primary_key.json"
@@ -1443,6 +1466,7 @@ def test_suggested_facets(app_client):
         {"name": "city_id", "querystring": "_facet=city_id"},
         {"name": "neighborhood", "querystring": "_facet=neighborhood"},
         {"name": "tags", "querystring": "_facet=tags"},
+        {"name": "complex_array", "querystring": "_facet=complex_array"},
         {"name": "created", "querystring": "_facet_date=created"},
     ]
     if detect_json1():
@@ -1478,6 +1502,7 @@ def test_expand_labels(app_client):
             "city_id": {"value": 1, "label": "San Francisco"},
             "neighborhood": "Dogpatch",
             "tags": '["tag1", "tag3"]',
+            "complex_array": "[]",
         },
         "13": {
             "pk": 13,
@@ -1488,6 +1513,7 @@ def test_expand_labels(app_client):
             "city_id": {"value": 3, "label": "Detroit"},
             "neighborhood": "Corktown",
             "tags": "[]",
+            "complex_array": "[]",
         },
     } == response.json
 
@@ -1495,7 +1521,7 @@ def test_expand_labels(app_client):
 def test_expand_label(app_client):
     response = app_client.get(
         "/fixtures/foreign_key_references.json?_shape=object"
-        "&_label=foreign_key_with_label"
+        "&_label=foreign_key_with_label&_size=1"
     )
     assert {
         "1": {
@@ -1613,6 +1639,11 @@ def test_infinity_returned_as_invalid_json_if_requested(app_client):
     ] == response.json
 
 
+def test_custom_query_with_unicode_characters(app_client):
+    response = app_client.get("/fixtures/𝐜𝐢𝐭𝐢𝐞𝐬.json?_shape=array")
+    assert [{"id": 1, "name": "San Francisco"}] == response.json
+
+
 def test_trace(app_client):
     response = app_client.get("/fixtures/simple_primary_key.json?_trace=1")
     data = response.json
@@ -1647,3 +1678,50 @@ def test_cors(app_client_with_cors, path, status_code):
     response = app_client_with_cors.get(path)
     assert response.status == status_code
     assert "*" == response.headers["Access-Control-Allow-Origin"]
+
+
+@pytest.mark.parametrize(
+    "path",
+    (
+        "/",
+        ".json",
+        "/searchable",
+        "/searchable.json",
+        "/searchable_view",
+        "/searchable_view.json",
+    ),
+)
+def test_database_with_space_in_name(app_client_two_attached_databases, path):
+    response = app_client_two_attached_databases.get("/extra database" + path)
+    assert response.status == 200
+
+
+def test_common_prefix_database_names(app_client_conflicting_database_names):
+    # https://github.com/simonw/datasette/issues/597
+    assert ["fixtures", "foo", "foo-bar"] == [
+        d["name"]
+        for d in json.loads(
+            app_client_conflicting_database_names.get("/-/databases.json").body.decode(
+                "utf8"
+            )
+        )
+    ]
+    for db_name, path in (("foo", "/foo.json"), ("foo-bar", "/foo-bar.json")):
+        data = json.loads(
+            app_client_conflicting_database_names.get(path).body.decode("utf8")
+        )
+        assert db_name == data["database"]
+
+
+def test_null_foreign_keys_are_not_expanded(app_client):
+    response = app_client.get(
+        "/fixtures/foreign_key_references.json?_shape=array&_labels=on"
+    )
+    assert [
+        {
+            "pk": "1",
+            "foreign_key_with_label": {"value": "1", "label": "hello"},
+            "foreign_key_with_no_label": {"value": "1", "label": "1"},
+        },
+        {"pk": "2", "foreign_key_with_label": None, "foreign_key_with_no_label": None,},
+    ] == response.json

tests/test_black.py

@@ -1,3 +1,4 @@
+import black
 from click.testing import CliRunner
 from pathlib import Path
 import pytest
@@ -6,13 +7,7 @@ import sys
 code_root = Path(__file__).parent.parent
 
 
-@pytest.mark.skipif(
-    sys.version_info[:2] < (3, 6), reason="Black requires Python 3.6 or later"
-)
 def test_black():
-    # Do not import at top of module because Python 3.5 will not have it installed
-    import black
-
     runner = CliRunner()
     result = runner.invoke(
         black.main, [str(code_root / "tests"), str(code_root / "datasette"), "--check"]

tests/test_csv.py

@@ -21,22 +21,30 @@ world
 )
 
 EXPECTED_TABLE_WITH_LABELS_CSV = """
-pk,created,planet_int,on_earth,state,city_id,city_id_label,neighborhood,tags
-1,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Mission,"[""tag1"", ""tag2""]"
-2,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Dogpatch,"[""tag1"", ""tag3""]"
-3,2019-01-14 08:00:00,1,1,CA,1,San Francisco,SOMA,[]
-4,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Tenderloin,[]
-5,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Bernal Heights,[]
-6,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Hayes Valley,[]
-7,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Hollywood,[]
-8,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Downtown,[]
-9,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Los Feliz,[]
-10,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Koreatown,[]
-11,2019-01-16 08:00:00,1,1,MI,3,Detroit,Downtown,[]
-12,2019-01-17 08:00:00,1,1,MI,3,Detroit,Greektown,[]
-13,2019-01-17 08:00:00,1,1,MI,3,Detroit,Corktown,[]
-14,2019-01-17 08:00:00,1,1,MI,3,Detroit,Mexicantown,[]
-15,2019-01-17 08:00:00,2,0,MC,4,Memnonia,Arcadia Planitia,[]
+pk,created,planet_int,on_earth,state,city_id,city_id_label,neighborhood,tags,complex_array
+1,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Mission,"[""tag1"", ""tag2""]","[{""foo"": ""bar""}]"
+2,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Dogpatch,"[""tag1"", ""tag3""]",[]
+3,2019-01-14 08:00:00,1,1,CA,1,San Francisco,SOMA,[],[]
+4,2019-01-14 08:00:00,1,1,CA,1,San Francisco,Tenderloin,[],[]
+5,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Bernal Heights,[],[]
+6,2019-01-15 08:00:00,1,1,CA,1,San Francisco,Hayes Valley,[],[]
+7,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Hollywood,[],[]
+8,2019-01-15 08:00:00,1,1,CA,2,Los Angeles,Downtown,[],[]
+9,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Los Feliz,[],[]
+10,2019-01-16 08:00:00,1,1,CA,2,Los Angeles,Koreatown,[],[]
+11,2019-01-16 08:00:00,1,1,MI,3,Detroit,Downtown,[],[]
+12,2019-01-17 08:00:00,1,1,MI,3,Detroit,Greektown,[],[]
+13,2019-01-17 08:00:00,1,1,MI,3,Detroit,Corktown,[],[]
+14,2019-01-17 08:00:00,1,1,MI,3,Detroit,Mexicantown,[],[]
+15,2019-01-17 08:00:00,2,0,MC,4,Memnonia,Arcadia Planitia,[],[]
+""".lstrip().replace(
+    "\n", "\r\n"
+)
+
+EXPECTED_TABLE_WITH_NULLABLE_LABELS_CSV = """
+pk,foreign_key_with_label,foreign_key_with_label_label,foreign_key_with_no_label,foreign_key_with_no_label_label
+1,1,hello,1,1
+2,,,,
 """.lstrip().replace(
     "\n", "\r\n"
 )
@@ -63,6 +71,13 @@ def test_table_csv_with_labels(app_client):
     assert EXPECTED_TABLE_WITH_LABELS_CSV == response.text
 
 
+def test_table_csv_with_nullable_labels(app_client):
+    response = app_client.get("/fixtures/foreign_key_references.csv?_labels=1")
+    assert response.status == 200
+    assert "text/plain; charset=utf-8" == response.headers["content-type"]
+    assert EXPECTED_TABLE_WITH_NULLABLE_LABELS_CSV == response.text
+
+
 def test_custom_sql_csv(app_client):
     response = app_client.get(
         "/fixtures.csv?sql=select+content+from+simple_primary_key+limit+2"
@@ -80,6 +95,15 @@ def test_table_csv_download(app_client):
     assert expected_disposition == response.headers["Content-Disposition"]
 
 
+def test_csv_with_non_ascii_characters(app_client):
+    response = app_client.get(
+        "/fixtures.csv?sql=select%0D%0A++%27%F0%9D%90%9C%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%A2%F0%9D%90%9E%F0%9D%90%AC%27+as+text%2C%0D%0A++1+as+number%0D%0Aunion%0D%0Aselect%0D%0A++%27bob%27+as+text%2C%0D%0A++2+as+number%0D%0Aorder+by%0D%0A++number"
+    )
+    assert response.status == 200
+    assert "text/plain; charset=utf-8" == response.headers["content-type"]
+    assert "text,number\r\n𝐜𝐢𝐭𝐢𝐞𝐬,1\r\nbob,2\r\n" == response.body.decode("utf8")
+
+
 def test_max_csv_mb(app_client_csv_max_mb_one):
     response = app_client_csv_max_mb_one.get(
         "/fixtures.csv?sql=select+randomblob(10000)+"

tests/test_facets.py

@@ -1,8 +1,7 @@
-from datasette.facets import ColumnFacet, ArrayFacet, DateFacet, ManyToManyFacet
+from datasette.facets import ColumnFacet, ArrayFacet, DateFacet
 from datasette.utils import detect_json1
 from .fixtures import app_client  # noqa
 from .utils import MockRequest
-from collections import namedtuple
 import pytest
 
 
@@ -24,6 +23,10 @@ async def test_column_facet_suggest(app_client):
         {"name": "city_id", "toggle_url": "http://localhost/?_facet=city_id"},
         {"name": "neighborhood", "toggle_url": "http://localhost/?_facet=neighborhood"},
         {"name": "tags", "toggle_url": "http://localhost/?_facet=tags"},
+        {
+            "name": "complex_array",
+            "toggle_url": "http://localhost/?_facet=complex_array",
+        },
     ] == suggestions
 
 
@@ -58,6 +61,10 @@ async def test_column_facet_suggest_skip_if_already_selected(app_client):
             "name": "tags",
             "toggle_url": "http://localhost/?_facet=planet_int&_facet=on_earth&_facet=tags",
         },
+        {
+            "name": "complex_array",
+            "toggle_url": "http://localhost/?_facet=planet_int&_facet=on_earth&_facet=complex_array",
+        },
     ] == suggestions
 
 
@@ -79,6 +86,7 @@ async def test_column_facet_suggest_skip_if_enabled_by_metadata(app_client):
         "state",
         "neighborhood",
         "tags",
+        "complex_array",
     ] == suggestions
 
 
@@ -207,6 +215,20 @@ async def test_array_facet_suggest(app_client):
     ] == suggestions
 
 
+@pytest.mark.asyncio
+@pytest.mark.skipif(not detect_json1(), reason="Requires the SQLite json1 module")
+async def test_array_facet_suggest_not_if_all_empty_arrays(app_client):
+    facet = ArrayFacet(
+        app_client.ds,
+        MockRequest("http://localhost/"),
+        database="fixtures",
+        sql="select * from facetable where tags = '[]'",
+        table="facetable",
+    )
+    suggestions = await facet.suggest()
+    assert [] == suggestions
+
+
 @pytest.mark.asyncio
 @pytest.mark.skipif(not detect_json1(), reason="Requires the SQLite json1 module")
 async def test_array_facet_results(app_client):
@@ -303,60 +325,3 @@ async def test_date_facet_results(app_client):
             "truncated": False,
         }
     } == buckets
-
-
-@pytest.mark.asyncio
-async def test_m2m_facet_suggest(app_client):
-    facet = ManyToManyFacet(
-        app_client.ds,
-        MockRequest("http://localhost/"),
-        database="fixtures",
-        sql="select * from roadside_attractions",
-        table="roadside_attractions",
-    )
-    suggestions = await facet.suggest()
-    assert [
-        {
-            "name": "attraction_characteristic",
-            "type": "m2m",
-            "toggle_url": "http://localhost/?_facet_m2m=attraction_characteristic",
-        }
-    ] == suggestions
-
-
-@pytest.mark.asyncio
-async def test_m2m_facet_results(app_client):
-    facet = ManyToManyFacet(
-        app_client.ds,
-        MockRequest("http://localhost/?_facet_m2m=attraction_characteristic"),
-        database="fixtures",
-        sql="select * from roadside_attractions",
-        table="roadside_attractions",
-    )
-    buckets, timed_out = await facet.facet_results()
-    assert [] == timed_out
-    assert {
-        "attraction_characteristic": {
-            "name": "attraction_characteristic",
-            "type": "m2m",
-            "results": [
-                {
-                    "value": 2,
-                    "label": "Paranormal",
-                    "count": 3,
-                    "toggle_url": "http://localhost/?_facet_m2m=attraction_characteristic&_through=%7B%22column%22%3A%22characteristic_id%22%2C%22table%22%3A%22roadside_attraction_characteristics%22%2C%22value%22%3A%222%22%7D",
-                    "selected": False,
-                },
-                {
-                    "value": 1,
-                    "label": "Museum",
-                    "count": 2,
-                    "toggle_url": "http://localhost/?_facet_m2m=attraction_characteristic&_through=%7B%22column%22%3A%22characteristic_id%22%2C%22table%22%3A%22roadside_attraction_characteristics%22%2C%22value%22%3A%221%22%7D",
-                    "selected": False,
-                },
-            ],
-            "hideable": True,
-            "toggle_url": "/",
-            "truncated": False,
-        }
-    } == buckets

tests/test_filters.py

@@ -47,6 +47,9 @@ import pytest
             ["foo in (:p0, :p1)"],
             ["dog,cat", "cat[dog]"],
         ),
+        # Not in, and JSON array not in
+        ((("foo__notin", "1,2,3"),), ["foo not in (:p0, :p1, :p2)"], ["1", "2", "3"]),
+        ((("foo__notin", "[1,2,3]"),), ["foo not in (:p0, :p1, :p2)"], [1, 2, 3]),
     ],
 )
 def test_build_where(args, expected_where, expected_params):

tests/test_html.py

@@ -26,12 +26,12 @@ def test_homepage(app_client_two_attached_databases):
     )
     # Should be two attached databases
     assert [
-        {"href": "/extra_database", "text": "extra_database"},
         {"href": "/fixtures", "text": "fixtures"},
+        {"href": "/extra database", "text": "extra database"},
     ] == [{"href": a["href"], "text": a.text.strip()} for a in soup.select("h2 a")]
     # The first attached database should show count text and attached tables
-    h2 = soup.select("h2")[0]
-    assert "extra_database" == h2.text.strip()
+    h2 = soup.select("h2")[1]
+    assert "extra database" == h2.text.strip()
     counts_p, links_p = h2.find_all_next("p")[:2]
     assert (
         "2 rows in 1 table, 5 rows in 4 hidden tables, 1 view" == counts_p.text.strip()
@@ -41,8 +41,8 @@ def test_homepage(app_client_two_attached_databases):
         {"href": a["href"], "text": a.text.strip()} for a in links_p.findAll("a")
     ]
     assert [
-        {"href": "/extra_database/searchable", "text": "searchable"},
-        {"href": "/extra_database/searchable_view", "text": "searchable_view"},
+        {"href": "/extra database/searchable", "text": "searchable"},
+        {"href": "/extra database/searchable_view", "text": "searchable_view"},
     ] == table_links
 
 
@@ -67,6 +67,8 @@ def test_static_mounts():
         assert response.status == 200
         response = client.get("/custom-static/not_exists.py")
         assert response.status == 404
+        response = client.get("/custom-static/../LICENSE")
+        assert response.status == 404
 
 
 def test_memory_database_page():
@@ -117,6 +119,39 @@ def test_row_strange_table_name_with_url_hash(app_client_with_hash):
     assert response.status == 200
 
 
+@pytest.mark.parametrize(
+    "path,expected_definition_sql",
+    [
+        (
+            "/fixtures/facet_cities",
+            """
+CREATE TABLE facet_cities (
+    id integer primary key,
+    name text
+);
+        """.strip(),
+        ),
+        (
+            "/fixtures/compound_three_primary_keys",
+            """
+CREATE TABLE compound_three_primary_keys (
+  pk1 varchar(30),
+  pk2 varchar(30),
+  pk3 varchar(30),
+  content text,
+  PRIMARY KEY (pk1, pk2, pk3)
+);
+CREATE INDEX idx_compound_three_primary_keys_content ON compound_three_primary_keys(content);
+            """.strip(),
+        ),
+    ],
+)
+def test_definition_sql(path, expected_definition_sql, app_client):
+    response = app_client.get(path)
+    pre = Soup(response.body, "html.parser").select_one("pre.wrapped-sql")
+    assert expected_definition_sql == pre.string
+
+
 def test_table_cell_truncation():
     for client in make_app_client(config={"truncate_cells_html": 5}):
         response = client.get("/fixtures/facetable")
@@ -601,7 +636,12 @@ def test_table_html_foreign_key_links(app_client):
             '<td class="col-pk"><a href="/fixtures/foreign_key_references/1">1</a></td>',
             '<td class="col-foreign_key_with_label"><a href="/fixtures/simple_primary_key/1">hello</a>\xa0<em>1</em></td>',
             '<td class="col-foreign_key_with_no_label"><a href="/fixtures/primary_key_multiple_columns/1">1</a></td>',
-        ]
+        ],
+        [
+            '<td class="col-pk"><a href="/fixtures/foreign_key_references/2">2</a></td>',
+            '<td class="col-foreign_key_with_label">\xa0</td>',
+            '<td class="col-foreign_key_with_no_label">\xa0</td>',
+        ],
     ]
     assert expected == [
         [str(td) for td in tr.select("td")] for tr in table.select("tbody tr")
@@ -609,7 +649,7 @@ def test_table_html_foreign_key_links(app_client):
 
 
 def test_table_html_disable_foreign_key_links_with_labels(app_client):
-    response = app_client.get("/fixtures/foreign_key_references?_labels=off")
+    response = app_client.get("/fixtures/foreign_key_references?_labels=off&_size=1")
     assert response.status == 200
     table = Soup(response.body, "html.parser").find("table")
     expected = [
@@ -735,6 +775,18 @@ def test_database_metadata(app_client):
     assert_footer_links(soup)
 
 
+def test_database_metadata_with_custom_sql(app_client):
+    response = app_client.get("/fixtures?sql=select+*+from+simple_primary_key")
+    assert response.status == 200
+    soup = Soup(response.body, "html.parser")
+    # Page title should be the default
+    assert "fixtures" == soup.find("h1").text
+    # Description should be custom
+    assert "Custom SQL query returning" in soup.find("h3").text
+    # The source/license should be inherited
+    assert_footer_links(soup)
+
+
 def test_table_metadata(app_client):
     response = app_client.get("/fixtures/simple_primary_key")
     assert response.status == 200
@@ -941,6 +993,12 @@ def test_extra_where_clauses(app_client):
         "/fixtures/facetable?_where=city_id%3D1",
         "/fixtures/facetable?_where=neighborhood%3D%27Dogpatch%27",
     ] == hrefs
+    # These should also be persisted as hidden fields
+    inputs = soup.find("form").findAll("input")
+    hiddens = [i for i in inputs if i["type"] == "hidden"]
+    assert [("_where", "neighborhood='Dogpatch'"), ("_where", "city_id=1")] == [
+        (hidden["name"], hidden["value"]) for hidden in hiddens
+    ]
 
 
 def test_binary_data_display(app_client):

tests/test_plugins.py

@@ -1,5 +1,6 @@
 from bs4 import BeautifulSoup as Soup
 from .fixtures import app_client, make_app_client, TEMP_PLUGIN_SECRET_FILE  # noqa
+from datasette.utils import sqlite3
 import base64
 import json
 import os
@@ -214,3 +215,20 @@ def test_plugins_extra_template_vars(restore_working_directory):
             "awaitable": True,
             "scope_path": "/-/metadata",
         } == extra_template_vars_from_awaitable
+
+
+def test_plugins_async_template_function(restore_working_directory):
+    for client in make_app_client(
+        template_dir=str(pathlib.Path(__file__).parent / "test_templates")
+    ):
+        response = client.get("/-/metadata")
+        assert response.status == 200
+        extra_from_awaitable_function = (
+            Soup(response.body, "html.parser")
+            .select("pre.extra_from_awaitable_function")[0]
+            .text
+        )
+        expected = (
+            sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0]
+        )
+        assert expected == extra_from_awaitable_function

tests/test_publish_cloudrun.py

@@ -1,6 +1,7 @@
 from click.testing import CliRunner
 from datasette import cli
 from unittest import mock
+import json
 
 
 @mock.patch("shutil.which")
@@ -26,23 +27,127 @@ def test_publish_cloudrun_invalid_database(mock_which):
 @mock.patch("shutil.which")
 @mock.patch("datasette.publish.cloudrun.check_output")
 @mock.patch("datasette.publish.cloudrun.check_call")
-def test_publish_cloudrun(mock_call, mock_output, mock_which):
+@mock.patch("datasette.publish.cloudrun.get_existing_services")
+def test_publish_cloudrun_prompts_for_service(
+    mock_get_existing_services, mock_call, mock_output, mock_which
+):
+    mock_get_existing_services.return_value = [
+        {"name": "existing", "created": "2019-01-01", "url": "http://www.example.com/"}
+    ]
     mock_output.return_value = "myproject"
     mock_which.return_value = True
     runner = CliRunner()
     with runner.isolated_filesystem():
         open("test.db", "w").write("data")
-        result = runner.invoke(cli.cli, ["publish", "cloudrun", "test.db"])
+        result = runner.invoke(
+            cli.cli, ["publish", "cloudrun", "test.db"], input="input-service"
+        )
+        assert (
+            """
+Please provide a service name for this deployment
+
+Using an existing service name will over-write it
+
+Your existing services:
+
+  existing - created 2019-01-01 - http://www.example.com/
+
+Service name: input-service
+""".strip()
+            == result.output.strip()
+        )
         assert 0 == result.exit_code
-        tag = "gcr.io/{}/datasette".format(mock_output.return_value)
+        tag = "gcr.io/myproject/datasette"
         mock_call.assert_has_calls(
             [
                 mock.call("gcloud builds submit --tag {}".format(tag), shell=True),
                 mock.call(
-                    "gcloud beta run deploy --allow-unauthenticated --image {}".format(
+                    "gcloud beta run deploy --allow-unauthenticated --platform=managed --image {} input-service".format(
                         tag
                     ),
                     shell=True,
                 ),
             ]
         )
+
+
+@mock.patch("shutil.which")
+@mock.patch("datasette.publish.cloudrun.check_output")
+@mock.patch("datasette.publish.cloudrun.check_call")
+def test_publish_cloudrun(mock_call, mock_output, mock_which):
+    mock_output.return_value = "myproject"
+    mock_which.return_value = True
+    runner = CliRunner()
+    with runner.isolated_filesystem():
+        open("test.db", "w").write("data")
+        result = runner.invoke(
+            cli.cli, ["publish", "cloudrun", "test.db", "--service", "test"]
+        )
+        assert 0 == result.exit_code
+        tag = "gcr.io/{}/datasette".format(mock_output.return_value)
+        mock_call.assert_has_calls(
+            [
+                mock.call("gcloud builds submit --tag {}".format(tag), shell=True),
+                mock.call(
+                    "gcloud beta run deploy --allow-unauthenticated --platform=managed --image {} test".format(
+                        tag
+                    ),
+                    shell=True,
+                ),
+            ]
+        )
+
+
+@mock.patch("shutil.which")
+@mock.patch("datasette.publish.cloudrun.check_output")
+@mock.patch("datasette.publish.cloudrun.check_call")
+def test_publish_cloudrun_plugin_secrets(mock_call, mock_output, mock_which):
+    mock_which.return_value = True
+    mock_output.return_value = "myproject"
+
+    runner = CliRunner()
+    with runner.isolated_filesystem():
+        open("test.db", "w").write("data")
+        result = runner.invoke(
+            cli.cli,
+            [
+                "publish",
+                "cloudrun",
+                "test.db",
+                "--service",
+                "datasette",
+                "--plugin-secret",
+                "datasette-auth-github",
+                "client_id",
+                "x-client-id",
+                "--show-files",
+            ],
+        )
+        dockerfile = (
+            result.output.split("==== Dockerfile ====\n")[1]
+            .split("\n====================\n")[0]
+            .strip()
+        )
+        expected = """FROM python:3.8
+COPY . /app
+WORKDIR /app
+
+ENV DATASETTE_AUTH_GITHUB_CLIENT_ID 'x-client-id'
+RUN pip install -U datasette
+RUN datasette inspect test.db --inspect-file inspect-data.json
+ENV PORT 8001
+EXPOSE 8001
+CMD datasette serve --host 0.0.0.0 -i test.db --cors --inspect-file inspect-data.json --metadata metadata.json --port $PORT""".strip()
+        assert expected == dockerfile
+        metadata = (
+            result.output.split("=== metadata.json ===\n")[1]
+            .split("\n==== Dockerfile ====\n")[0]
+            .strip()
+        )
+        assert {
+            "plugins": {
+                "datasette-auth-github": {
+                    "client_id": {"$env": "DATASETTE_AUTH_GITHUB_CLIENT_ID"}
+                }
+            }
+        } == json.loads(metadata)

tests/test_publish_heroku.py

@@ -46,7 +46,7 @@ def test_publish_heroku_invalid_database(mock_which):
 @mock.patch("datasette.publish.heroku.check_output")
 @mock.patch("datasette.publish.heroku.call")
 def test_publish_heroku(mock_call, mock_check_output, mock_which):
-    mock_which.return_varue = True
+    mock_which.return_value = True
     mock_check_output.side_effect = lambda s: {
         "['heroku', 'plugins']": b"heroku-builds",
         "['heroku', 'apps:list', '--json']": b"[]",
@@ -57,6 +57,55 @@ def test_publish_heroku(mock_call, mock_check_output, mock_which):
         open("test.db", "w").write("data")
         result = runner.invoke(cli.cli, ["publish", "heroku", "test.db"])
         assert 0 == result.exit_code, result.output
-        mock_call.assert_called_once_with(
-            ["heroku", "builds:create", "-a", "f", "--include-vcs-ignore"]
+        mock_call.assert_has_calls(
+            [
+                mock.call(["heroku", "config:set", "-a", "f", "WEB_CONCURRENCY=1",]),
+                mock.call(
+                    ["heroku", "builds:create", "-a", "f", "--include-vcs-ignore"]
+                ),
+            ]
+        )
+
+
+@mock.patch("shutil.which")
+@mock.patch("datasette.publish.heroku.check_output")
+@mock.patch("datasette.publish.heroku.call")
+def test_publish_heroku_plugin_secrets(mock_call, mock_check_output, mock_which):
+    mock_which.return_value = True
+    mock_check_output.side_effect = lambda s: {
+        "['heroku', 'plugins']": b"heroku-builds",
+        "['heroku', 'apps:list', '--json']": b"[]",
+        "['heroku', 'apps:create', 'datasette', '--json']": b'{"name": "f"}',
+    }[repr(s)]
+    runner = CliRunner()
+    with runner.isolated_filesystem():
+        open("test.db", "w").write("data")
+        result = runner.invoke(
+            cli.cli,
+            [
+                "publish",
+                "heroku",
+                "test.db",
+                "--plugin-secret",
+                "datasette-auth-github",
+                "client_id",
+                "x-client-id",
+            ],
+        )
+        assert 0 == result.exit_code, result.output
+        mock_call.assert_has_calls(
+            [
+                mock.call(
+                    [
+                        "heroku",
+                        "config:set",
+                        "-a",
+                        "f",
+                        "DATASETTE_AUTH_GITHUB_CLIENT_ID=x-client-id",
+                    ]
+                ),
+                mock.call(
+                    ["heroku", "builds:create", "-a", "f", "--include-vcs-ignore"]
+                ),
+            ]
         )

tests/test_publish_now.py

@@ -1,6 +1,7 @@
 from click.testing import CliRunner
 from datasette import cli
 from unittest import mock
+import json
 import subprocess
 
 
@@ -105,3 +106,58 @@ def test_publish_now_multiple_aliases(mock_run, mock_which):
                 ),
             ]
         )
+
+
+@mock.patch("shutil.which")
+@mock.patch("datasette.publish.now.run")
+def test_publish_now_plugin_secrets(mock_run, mock_which):
+    mock_which.return_value = True
+    mock_run.return_value = mock.Mock(0)
+    mock_run.return_value.stdout = b"https://demo.example.com/"
+
+    runner = CliRunner()
+    with runner.isolated_filesystem():
+        open("test.db", "w").write("data")
+        result = runner.invoke(
+            cli.cli,
+            [
+                "publish",
+                "now",
+                "test.db",
+                "--token",
+                "XXX",
+                "--plugin-secret",
+                "datasette-auth-github",
+                "client_id",
+                "x-client-id",
+                "--show-files",
+            ],
+        )
+        dockerfile = (
+            result.output.split("==== Dockerfile ====\n")[1]
+            .split("\n====================\n")[0]
+            .strip()
+        )
+        expected = """FROM python:3.8
+COPY . /app
+WORKDIR /app
+
+ENV DATASETTE_AUTH_GITHUB_CLIENT_ID 'x-client-id'
+RUN pip install -U datasette
+RUN datasette inspect test.db --inspect-file inspect-data.json
+ENV PORT 8001
+EXPOSE 8001
+CMD datasette serve --host 0.0.0.0 -i test.db --cors --inspect-file inspect-data.json --metadata metadata.json --config force_https_urls:on --port $PORT""".strip()
+        assert expected == dockerfile
+        metadata = (
+            result.output.split("=== metadata.json ===\n")[1]
+            .split("\n==== Dockerfile ====\n")[0]
+            .strip()
+        )
+        assert {
+            "plugins": {
+                "datasette-auth-github": {
+                    "client_id": {"$env": "DATASETTE_AUTH_GITHUB_CLIENT_ID"}
+                }
+            }
+        } == json.loads(metadata)

tests/test_templates/show_json.html

@@ -5,4 +5,5 @@
 Test data for extra_template_vars:
 <pre class="extra_template_vars">{{ extra_template_vars|safe }}</pre>
 <pre class="extra_template_vars_from_awaitable">{{ extra_template_vars_from_awaitable|safe }}</pre>
+<pre class="extra_from_awaitable_function">{{ query_database("select sqlite_version();") }}</pre>
 {% endblock %}

tests/test_utils.py

@@ -151,15 +151,20 @@ def test_validate_sql_select_bad(bad_sql):
         "select count(*) from airports",
         "select foo from bar",
         "select 1 + 1",
+        "explain select 1 + 1",
+        "explain query plan select 1 + 1",
         "SELECT\nblah FROM foo",
         "WITH RECURSIVE cnt(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM cnt LIMIT 10) SELECT x FROM cnt;",
+        "explain WITH RECURSIVE cnt(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM cnt LIMIT 10) SELECT x FROM cnt;",
+        "explain query plan WITH RECURSIVE cnt(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM cnt LIMIT 10) SELECT x FROM cnt;",
     ],
 )
 def test_validate_sql_select_good(good_sql):
     utils.validate_sql_select(good_sql)
 
 
-def test_detect_fts():
+@pytest.mark.parametrize("open_quote,close_quote", [('"', '"'), ("[", "]")])
+def test_detect_fts(open_quote, close_quote):
     sql = """
     CREATE TABLE "Dumb_Table" (
       "TreeID" INTEGER,
@@ -175,9 +180,11 @@ def test_detect_fts():
       "qCaretaker" TEXT
     );
     CREATE VIEW Test_View AS SELECT * FROM Dumb_Table;
-    CREATE VIRTUAL TABLE "Street_Tree_List_fts" USING FTS4 ("qAddress", "qCaretaker", "qSpecies", content="Street_Tree_List");
+    CREATE VIRTUAL TABLE {open}Street_Tree_List_fts{close} USING FTS4 ("qAddress", "qCaretaker", "qSpecies", content={open}Street_Tree_List{close});
     CREATE VIRTUAL TABLE r USING rtree(a, b, c);
-    """
+    """.format(
+        open=open_quote, close=close_quote
+    )
     conn = utils.sqlite3.connect(":memory:")
     conn.executescript(sql)
     assert None is utils.detect_fts(conn, "Dumb_Table")

versioneer.py

@@ -180,7 +180,7 @@ two common reasons why `setup.py` might not be in the root:
   `setup.cfg`, and `tox.ini`. Projects like these produce multiple PyPI
   distributions (and upload multiple independently-installable tarballs).
 * Source trees whose main purpose is to contain a C library, but which also
-  provide bindings to Python (and perhaps other langauges) in subdirectories.
+  provide bindings to Python (and perhaps other languages) in subdirectories.
 
 Versioneer will look for `.git` in parent directories, and most operations
 should get the right version string. However `pip` and `setuptools` have bugs
@@ -805,7 +805,7 @@ def render_pep440_old(pieces):
 
     The ".dev0" means dirty.
 
-    Eexceptions:
+    Exceptions:
     1: no tags. 0.postDISTANCE[.dev0]
     """
     if pieces["closest-tag"]:
@@ -1306,7 +1306,7 @@ def render_pep440_old(pieces):
 
     The ".dev0" means dirty.
 
-    Eexceptions:
+    Exceptions:
     1: no tags. 0.postDISTANCE[.dev0]
     """
     if pieces["closest-tag"]: