The extra_css_urls and extra_js_urls hooks now take additional optional
parameters.
Also refactored them out of the Datasette class and into RenderMixin.
Plus improved plugin documentation to explicitly list parameters.
Also removed xfail from test_view_classes_are_documented, so any future *View
classes that are added without documentation will cause the tests to fail.
More documentation unit tests. These ones check that every single **View class
imported into the datasette/app.py module are covered by our documentation.
Just one problem: they aren't documented yet. So I'm using the xfail pytest
decorator to mark these tests as allowed-to-fail. When you run the test suite
you now get a report of how many views still need to be documented, but it
doesn't fail the tests.
The output looks something like this:
$ pytest tests/test_docs.py
collected 31 items
tests/test_docs.py ..........................XXXxx. [100%]
============ 26 passed, 2 xfailed, 3 xpassed in 1.06 seconds ============
Once I have documented all the views I will remove the xfail so any future
views that are added without documentation will cause a test failure.
We can detect that a view is documented by looking for ReST label in the docs,
for example:
.. _IndexView:
Some view classes can be used to power multiple URLs - the JsonDataView class
for example is used to power /-/metadata and /-/config and /-/plugins
In this case, the second part of the label can indicate the variety of page, e.g:
.. _JsonDataView_metadata:
The test will pass as long as there is at least one label that starts with
_JsonDataView.
This change introduces a new plugin hook, publish_subcommand, which can be
used to implement new subcommands for the "datasette publish" command family.
I've used this new hook to refactor out the "publish now" and "publish heroku"
implementations into separate modules. I've also added unit tests for these
two publishers, mocking the subprocess.call and subprocess.check_output
functions.
As part of this, I introduced a mechanism for loading default plugins. These
are defined in the new "default_plugins" list inside datasette/app.py
Closes#217 (Plugin support for datasette publish)
Closes#348 (Unit tests for "datasette publish")
Refs #14, #59, #102, #103, #146, #236, #347
Unit tests now check that docs/*.txt help examples are all up-to-date.
I ran into a problem here in that the terminal_width needed to be more
accurately defined - so I replaced update-docs-help.sh with update-docs-
help.py which hard-codes the terminal width.
* table.csv?_stream=1 to download all rows - refs #266
This option causes Datasette to serve ALL rows in the table, by internally
following the _next= pagination links and serving everything out as a stream.
Also added new config option, allow_csv_stream, which can be used to disable
this feature.
* New config option max_csv_mb limiting size of CSV export
This is a relatively obscure new command-line argument that helps solve the
problem of showing accurate version information in deployed instances of
Datasette even if they were deployed directly from source code.
You can pass --version-note to datasette publish and package and it will then
in turn be passed to datasette when it starts:
datasette --version-note=hello fixtures.db
Now if you visit /-/versions.json you will see this:
{
"datasette": {
"note": "hello",
"version": "0+unknown"
},
"python": {
"full": "3.6.5 (default, Jun 6 2018, 19:19:24) \n[GCC 6.3.0 20170516]",
"version": "3.6.5"
},
...
}
I plan to use this in some Travis CI configuration, refs #313
The fixtures database created by our unit tests makes for a good "live" demo
of Datasette in action.
I've improved the metadata it ships with to better support this use-case.
I've also improved the mechanism for writing out fixtures: you can do this:
python tests/fixtures.py fixtures.db
To get just the fixtures database written out... or you can do this:
python tests/fixtures.py fixtures.db fixtures.json
To get metadata which you can then serve like so:
datasette fixtures.db -m fixtures.json
Refs #313
These new querystring arguments can be used to request expanded foreign keys
in both JSON and CSV formats.
?_labels=on turns on expansions for ALL foreign key columns
?_label=COLUMN1&_label=COLUMN2 can be used to pick specific columns to expand
e.g. `Street_Tree_List.json?_label=qSpecies&_label=qLegalStatus`
{
"rowid": 233,
"TreeID": 121240,
"qLegalStatus": {
"value" 2,
"label": "Private"
}
"qSpecies": {
"value": 16,
"label": "Sycamore"
}
"qAddress": "91 Commonwealth Ave",
...
}
The labels option also works for the HTML and CSV views.
HTML defaults to `?_labels=on`, so if you pass `?_labels=off` you can disable
foreign key expansion entirely - or you can use `?_label=COLUMN` to request
just specific columns.
If you expand labels on CSV you get additional columns in the output:
`/Street_Tree_List.csv?_label=qLegalStatus`
rowid,TreeID,qLegalStatus,qLegalStatus_label...
1,141565,1,Permitted Site...
2,232565,2,Undocumented...
I also refactored the existing foreign key expansion code.
Closes#233. Refs #266.
The test used to expect CSV to come back like this:
hello
world
""
With the final blank value encoded in quotes.
Judging by Travis failures, this behaviour changed between Python 3.6.3 and
3.6.5:
https://travis-ci.org/simonw/datasette/jobs/392586661
Tables and custom SQL query results can now be exported as CSV.
The easiest way to do this is to use the .csv extension, e.g.
/test_tables/facet_cities.csv
By default this is served as Content-Type: text/plain so you can see it in
your browser. If you want to download the file (using text/csv and with an
appropriate Content-Disposition: attachment header) you can do so like this:
/test_tables/facet_cities.csv?_dl=1
We link to the CSV and downloadable CSV URLs from the table and query pages.
The links use ?_size=max and so by default will return 1,000 rows.
Also fixes#303 - table names ending in .json or .csv are now detected and
URLs are generated that look like this instead:
/test_tables/table%2Fwith%2Fslashes.csv?_format=csv
The ?_format= option is available for everything else too, but we link to the
.csv / .json versions in most cases because they are aesthetically pleasing.
https://github.com/pytest-dev/pytest/issues/1875 made it impossible to declare
a function as a fixture multiple times, which we were doing across different
modules. The fix was to move our @pytest.fixture calls into decorators in the
tests/fixtures.py module.
Simon Willison