kopia lustrzana https://github.com/simonw/datasette
Documentation for SQLite full-text search support, closes #253
rodzic
04d21ccd08
commit
694f845715
|
@ -0,0 +1,99 @@
|
|||
.. _full_text_search:
|
||||
|
||||
Full-text search
|
||||
================
|
||||
|
||||
SQLite includes `a powerful mechanism for enabling full-text search <https://www.sqlite.org/fts3.html>`_ against SQLite records. Datasette can detect if a table has had full-text search configured for it in the underlying database and display a search interface for filtering that table.
|
||||
|
||||
Datasette detects which tables have been configured for full-text search when it first inspects the database on startup (or via the ``datasette inspect`` command). You can visit the ``/-/inspect`` page on your Datasette instance to see the results of this inspection. Tables that have been configured for full-text search will have their ``fts_table`` property set to the name of another table (tables without full-text search will have this property set to ``null``).
|
||||
|
||||
FTS versions
|
||||
------------
|
||||
|
||||
There are three different versions of the SQLite FTS module: FTS3, FTS4 and FTS5. You can tell which versions are supported by your instance of Datasette by checking the ``/-/versions`` page.
|
||||
|
||||
FTS5 is the most advanced module, but is usually not available is the SQLite version that is bundled with Python. If in doubt, you should use FTS4.
|
||||
|
||||
Adding full-text search to a SQLite table
|
||||
-----------------------------------------
|
||||
|
||||
Datasette takes advantage of the `external content <https://www.sqlite.org/fts3.html#_external_content_fts4_tables_>`_ mechanism in SQLite, which allows a full-text search virtual table to be associated with the contents of another SQLite table.
|
||||
|
||||
To set up full-text search for a table, you need to do two things:
|
||||
|
||||
* Create a new FTS virtual table associated with your table
|
||||
* Populate that FTS table with the data that you would like to be able to run searches against
|
||||
|
||||
To enable full-text search for a table called ``items`` that works against the ``name`` and ``description`` columns, you would run the following SQL to create a new ``items_fts`` FTS virtual table::
|
||||
|
||||
CREATE VIRTUAL TABLE "items_fts" USING FTS4 (
|
||||
name,
|
||||
description,
|
||||
content="items"
|
||||
);
|
||||
|
||||
This creates a set of tables to power full-text search against ``items``. The new ``items_fts`` table will be detected by Datasette as the ``fts_table`` for the ``items`` table.
|
||||
|
||||
Creating the table is not enough: you also need to populate it with a copy of the data that you wish to make searchable. You can do that using the following SQL::
|
||||
|
||||
INSERT INTO "items_fts" (rowid, name, description)
|
||||
SELECT rowid, name, description FROM items;
|
||||
|
||||
If your table has columns that are foreign key references to other tables you can include that data in your full-text search index using a join. Imagine the ``items`` table has a foreign key column called ``category_id`` which refers to a ``categories`` table - you could create a full-text search table like this::
|
||||
|
||||
CREATE VIRTUAL TABLE "items_fts" USING FTS4 (
|
||||
name,
|
||||
description,
|
||||
category_name,
|
||||
content="items"
|
||||
);
|
||||
|
||||
And then populate it like this::
|
||||
|
||||
INSERT INTO "items_fts" (rowid, name, description, category_name)
|
||||
SELECT items.rowid,
|
||||
items.name,
|
||||
items.description,
|
||||
categories.name
|
||||
FROM items JOIN categories ON items.category_id=categories.id;
|
||||
|
||||
You can use this technique to populate the full-text search index from any combination of tables and joins that makes sense for your project.
|
||||
|
||||
Setting up full-text search using csvs-to-sqlite
|
||||
------------------------------------------------
|
||||
|
||||
If your data starts out in CSV files, you can use Datasette's companion tool `csvs-to-sqlite <https://github.com/simonw/csvs-to-sqlite>`_ to convert that file into a SQLite database and enable full-text search on specific columns. For a file called ``items.csv`` where you want full-text search to operate against the ``name`` and ``description`` columns you would run the following::
|
||||
|
||||
csvs-to-sqlite items.csv items.db -f name -f description
|
||||
|
||||
The table view API
|
||||
------------------
|
||||
|
||||
Table views that support full-text search can be queried using the ``?_search=TERMS`` querystring parameter. This will run the search against content from all of the columns that have been included in the index.
|
||||
|
||||
SQLite full-text search supports wildcards. This means you can easily implement prefix auto-complete by including an asterisk at the end of the search term - for example::
|
||||
|
||||
/dbname/tablename/?_search=rob*
|
||||
|
||||
This will return all records containing at least one word that starts with the letters ``rob``.
|
||||
|
||||
You can also run searches against just the content of a specific named column by using ``_search_COLNAME=TERMS`` - for example, this would search for just rows where the ``name`` column in the FTS index mentions ``Sarah``::
|
||||
|
||||
/dbname/tablename/?_search_name=Sarah
|
||||
|
||||
Searches using custom SQL
|
||||
-------------------------
|
||||
|
||||
You can include full-text search results in custom SQL queries. The general pattern with SQLite search is to run the search as a sub-select that returns rowid values, then include those rowids in another part of the query.
|
||||
|
||||
You can see the syntax for a basic search by running that search on a table page and then clicking "View and edit SQL" to see the underlying SQL. For example, consider this search for `cherry trees in San Francisco <https://san-francisco.datasettes.com/sf-trees/Street_Tree_List?_search=cherry>`_::
|
||||
|
||||
/sf-trees/Street_Tree_List?_search=cherry
|
||||
|
||||
If you click `View and edit SQL <https://san-francisco.datasettes.com/sf-trees?sql=select+rowid%2C+*+from+Street_Tree_List+where+rowid+in+(select+rowid+from+[Street_Tree_List_fts]+where+[Street_Tree_List_fts]+match+%3Asearch)+order+by+rowid+limit+101&search=cherry>`_ you'll see that the underlying SQL looks like this::
|
||||
|
||||
select rowid, * from Street_Tree_List
|
||||
where rowid in (
|
||||
select rowid from [Street_Tree_List_fts]
|
||||
where [Street_Tree_List_fts] match "cherry"
|
||||
) order by rowid limit 101
|
|
@ -19,6 +19,7 @@ Contents
|
|||
getting_started
|
||||
json_api
|
||||
sql_queries
|
||||
full_text_search
|
||||
metadata
|
||||
custom_templates
|
||||
plugins
|
||||
|
|
Ładowanie…
Reference in New Issue