Extra ES params passed through new OPTIONS key

Closes #2778

Extra ES params are now passed through new OPTIONS key in the
WAGTAILSEARCH_BACKENDS setting.

It's backward compatible: if no OPTIONS key is found and some parameters
still exist, those parameters are used for the ES constructor..

CHANGELOG.txt

@@ -24,6 +24,8 @@ Changelog
  * Redundant action buttons are now omitted from the root page in the explorer (Nick Smith)
  * Locked pages are now disabled from editing at the browser level (Edd Baldry)
  * Added `in_site` method for filtering page querysets to pages within the specified site (Chris Rogers)
+ * Added the ability to override the default index settings for Elasticsearch (PyMan Claudio Marinozzi)
+ * Extra options for the Elasticsearch constructor should be now defined with the new key `OPTIONS` of the `WAGTAILSEARCH_BACKENDS` setting (PyMan Claudio Marinozzi)
  * Fix: `AbstractForm` now respects custom `get_template` methods on the page model (Gagaro)
  * Fix: Use specific page model for the parent page in the explore index (Gagaro)
  * Fix: Remove responsive styles in embed when there is no ratio available (Gagaro)

CONTRIBUTORS.rst

@@ -189,6 +189,7 @@ Contributors
 * Diederik van der Boor
 * Sean Hoefler
 * Edd Baldry
+* PyMan Claudio Marinozzi
 
 Translators
 ===========

docs/releases/1.8.rst

@@ -55,6 +55,8 @@ Minor features
  * Redundant action buttons are now omitted from the root page in the explorer (Nick Smith)
  * Locked pages are now disabled from editing at the browser level (Edd Baldry)
  * Added :meth:`wagtail.wagtailcore.query.PageQuerySet.in_site` method for filtering page querysets to pages within the specified site (Chris Rogers)
+ * Added the ability to override the default index settings for Elasticsearch. See :ref:`wagtailsearch_backends_elasticsearch` (PyMan Claudio Marinozzi)
+ * Extra options for the Elasticsearch constructor should be now defined with the new key ``OPTIONS`` of the ``WAGTAILSEARCH_BACKENDS`` setting (PyMan Claudio Marinozzi)
 
 
 Bug fixes
@@ -141,3 +143,8 @@ The ``wagtail.contrib.wagtailfrontendcache.backends.CloudflareBackend`` module h
     }
 
 For details of how to obtain the zone identifier, see `the Cloudflare API documentation <https://api.cloudflare.com/#getting-started-resource-ids>`_.
+
+Extra options for the Elasticsearch constructor should be now defined with the new key ``OPTIONS`` of the ``WAGTAILSEARCH_BACKENDS`` setting
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For the Elasticsearch backend, all extra keys defined in ``WAGTAILSEARCH_BACKENDS`` are passed directly to the Elasticsearch constructor. All these keys now should be moved inside the new ``OPTIONS`` dictionary. The old behaviour is still supported, but deprecated.

docs/topics/search/backends.rst

@@ -118,10 +118,36 @@ The backend is configured in settings:
           'URLS': ['http://localhost:9200'],
           'INDEX': 'wagtail',
           'TIMEOUT': 5,
+          'OPTIONS': {},
+          'INDEX_SETTINGS': {},
       }
   }
 
-Other than ``BACKEND``, the keys are optional and default to the values shown. In addition, any other keys are passed directly to the Elasticsearch constructor as case-sensitive keyword arguments (e.g. ``'max_retries': 1``).
+Other than ``BACKEND``, the keys are optional and default to the values shown. Any defined key in ``OPTIONS`` is passed directly to the Elasticsearch constructor as case-sensitive keyword argument (e.g. ``'max_retries': 1``).
+
+``INDEX_SETTINGS`` is a dictionary used to override the default settings to create the index. The default settings are defined inside the ``ElasticsearchSearchBackend`` class in the module ``wagtail/wagtail/wagtailsearch/backends/elasticsearch.py``. Any new key is added, any existing key, if not a dictionary, is replaced with the new value. Here's a sample on how to configure the number of shards and setting the italian LanguageAnalyzer as the default analyzer:
+
+.. code-block:: python
+
+  WAGTAILSEARCH_BACKENDS = {
+      'default': {
+          ...,
+          'INDEX_SETTINGS': {
+              'settings': {
+                  'number_of_shards': 2,
+                      'index': {
+                          'analysis': {
+                              'analyzer': {
+                                  'default': {
+                                      'type': 'italian'
+                                  }
+                              }
+                          }
+                      }
+                  }
+              },
+          }
+      }
 
 If you prefer not to run an Elasticsearch server in development or production, there are many hosted services available, including `Bonsai`_, who offer a free account suitable for testing and development. To use Bonsai:
 

wagtail/utils/utils.py

@@ -0,0 +1,23 @@
+from __future__ import absolute_import, unicode_literals
+
+import collections
+import sys
+
+
+def deep_update(source, overrides):
+    """Update a nested dictionary or similar mapping.
+
+    Modify ``source`` in place.
+    """
+    if sys.version_info >= (3, 0):
+        items = overrides.items()
+    else:
+        items = overrides.iteritems()
+
+    for key, value in items:
+        if isinstance(value, collections.Mapping) and value:
+            returned = deep_update(source.get(key, {}), value)
+            source[key] = returned
+        else:
+            source[key] = overrides[key]
+    return source

wagtail/wagtailsearch/backends/elasticsearch.py

@@ -1,6 +1,7 @@
 from __future__ import absolute_import, unicode_literals
 
 import json
+import warnings
 
 from django.db import models
 from django.utils.crypto import get_random_string
@@ -8,6 +9,8 @@ from django.utils.six.moves.urllib.parse import urlparse
 from elasticsearch import Elasticsearch, NotFoundError
 from elasticsearch.helpers import bulk
 
+from wagtail.utils.deprecation import RemovedInWagtail110Warning
+from wagtail.utils.utils import deep_update
 from wagtail.wagtailsearch.backends.base import (
     BaseSearchBackend, BaseSearchQuery, BaseSearchResults)
 from wagtail.wagtailsearch.index import (
@@ -760,12 +763,24 @@ class ElasticsearchSearchBackend(BaseSearchBackend):
                     'http_auth': http_auth,
                 })
 
+        self.settings = self.settings.copy()  # Make the class settings attribute as instance settings attribute
+        self.settings = deep_update(self.settings, params.pop("INDEX_SETTINGS", {}))
+
         # Get Elasticsearch interface
         # Any remaining params are passed into the Elasticsearch constructor
+        options = params.pop('OPTIONS', {})
+        if not options and params:
+            options = params
+
+            warnings.warn(
+                "Any extra parameter for the ElasticSearch constructor must be passed through the OPTIONS dictionary.",
+                category=RemovedInWagtail110Warning, stacklevel=2
+            )
+
         self.es = Elasticsearch(
             hosts=self.hosts,
             timeout=self.timeout,
-            **params)
+            **options)
 
     def get_index_for_model(self, model):
         return self.index_class(self, self.index_name)

wagtail/wagtailsearch/tests/test_elasticsearch_backend.py

@@ -981,6 +981,33 @@ class TestBackendConfiguration(TestCase):
         self.assertEqual(backend.hosts[3]['use_ssl'], True)
         self.assertEqual(backend.hosts[3]['url_prefix'], '/hello')
 
+    def test_default_index_settings_override(self):
+        backend = ElasticsearchSearchBackend(params={
+            'INDEX_SETTINGS': {
+                "settings": {  # Already existing key
+                    "number_of_shards": 2,  # New key
+                    "analysis": {  # Already existing key
+                        "analyzer": {  # Already existing key
+                            "edgengram_analyzer": {  # Already existing key
+                                "tokenizer": "standard"  # Key redefinition
+                            }
+                        }
+                    }
+                }
+            }
+        })
+
+        # Check structure
+        self.assertIn("number_of_shards", backend.settings["settings"].keys())
+        self.assertIn("analysis", backend.settings["settings"].keys())
+        self.assertIn("analyzer", backend.settings["settings"]["analysis"].keys())
+        self.assertIn("edgengram_analyzer", backend.settings["settings"]["analysis"]["analyzer"].keys())
+        self.assertIn("tokenizer", backend.settings["settings"]["analysis"]["analyzer"]["edgengram_analyzer"].keys())
+        # Check values
+        self.assertEqual(backend.settings["settings"]["number_of_shards"], 2)
+        self.assertEqual(backend.settings["settings"]["analysis"]["analyzer"]["edgengram_analyzer"]["tokenizer"], "standard")
+        self.assertEqual(backend.settings["settings"]["analysis"]["analyzer"]["edgengram_analyzer"]["type"], "custom")  # Check if a default setting still exists
+
 
 @unittest.skipUnless(os.environ.get('ELASTICSEARCH_URL', False), "ELASTICSEARCH_URL not set")
 @unittest.skipUnless(os.environ.get('ELASTICSEARCH_VERSION', '1') == '1', "ELASTICSEARCH_VERSION not set to 1")