Added support for AWS CloudFront in Frontend cache invalidation module (#1845)

* added base cloudfrontbackend and testcase

* added boto3 cloudfront client

* implemented create invalidation method
added error handling botocore

* added aws docs

* fixed typo

* flake8 fixes

* added boto3 configuration docs

* removed return

* purge path instead of full url

* added multisite hostname mapping

* added validation of DISTRIBUTION_ID

* renamed Cloudfront to CloudFront

* added note to include www in mapping
added tests for cloudfront site mapping

* removed deprecated has_key, used in
fixed _create_invalidation

* changed type checking of dict
removed debug line of code to check hostname

* fixed dict type checking condition
added assert t make sure no invalid cache is being purged

* changed import order

* fixed isort error

* more detailed error message for cloudfront
pep8 fixes 120 chars per line

* Log missing cloudfront distribution id as info

Was logging as error, but it may be possible that a developer wants cloudfront on only specific hostnames.

* , => .

* Docs edits

* Removed hard-dependency on boto3

docs/reference/contrib/api/configuration.rst

@@ -59,7 +59,7 @@ This list also supports child relations (which will be nested inside the returne
 Frontend cache invalidation
 ---------------------------
 
-If you have a Varnish, Squid or Cloudflare instance in front of your API, the ``wagtailapi`` module can automatically invalidate cached responses for you whenever they are updated in the database.
+If you have a Varnish, Squid, Cloudflare or CloudFront instance in front of your API, the ``wagtailapi`` module can automatically invalidate cached responses for you whenever they are updated in the database.
 
 To enable it, firstly configure the ``wagtail.contrib.wagtailfrontendcache`` module within your project (see [Wagtail frontend cache docs](http://docs.wagtail.io/en/latest/contrib_components/frontendcache.html) for more information).
 

docs/reference/contrib/frontendcache.rst

@@ -8,7 +8,11 @@ Frontend cache invalidator
    * Multiple backend support added
    * Cloudflare support added
 
-Many websites use a frontend cache such as Varnish, Squid or Cloudflare to gain extra performance. The downside of using a frontend cache though is that they don't respond well to updating content and will often keep an old version of a page cached after it has been updated.
+.. versionchanged:: 1.6
+
+   * Amazon CloudFront support added
+
+Many websites use a frontend cache such as Varnish, Squid, Cloudflare or CloudFront to gain extra performance. The downside of using a frontend cache though is that they don't respond well to updating content and will often keep an old version of a page cached after it has been updated.
 
 This document describes how to configure Wagtail to purge old versions of pages from a frontend cache whenever a page gets updated.
 
@@ -76,6 +80,41 @@ Add an item into the ``WAGTAILFRONTENDCACHE`` and set the ``BACKEND`` parameter
     }
 
 
+Amazon CloudFront
+^^^^^^^^^^^^^^^^^
+
+Within Amazon Web Services you will need at least one CloudFront web distribution. If you don't have one, you can get one here: `CloudFront getting started <https://aws.amazon.com/cloudfront/>`_
+
+Add an item into the ``WAGTAILFRONTENDCACHE`` and set the ``BACKEND`` parameter to ``wagtail.contrib.wagtailfrontendcache.backends.CloudfrontBackend``. This backend requires one extra parameter, ``DISTRIBUTION_ID`` (your CloudFront generated distrubition id).
+
+.. code-block:: python
+
+    WAGTAILFRONTENDCACHE = {
+        'cloudfront': {
+            'BACKEND': 'wagtail.contrib.wagtailfrontendcache.backends.CloudfrontBackend',
+            'DISTRIBUTION_ID': 'your-distribution-id',
+        },
+    }
+
+Configuration of credentials can done in multiple ways. You won't need to store them in your Django settings file. You can read more about this here: `Boto 3 Docs <http://boto3.readthedocs.org/en/latest/guide/configuration.html>`_
+
+In case you run multiple sites with Wagtail and each site has its CloudFront distribution, provide a mapping instead of a single distribution. Make sure the mapping matches with the hostnames provided in your site settings.
+
+.. code-block:: python
+
+    WAGTAILFRONTENDCACHE = {
+        'cloudfront': {
+            'BACKEND': 'wagtail.contrib.wagtailfrontendcache.backends.CloudfrontBackend',
+            'DISTRIBUTION_ID': {
+                'www.wagtail.io': 'your-distribution-id',
+                'www.madewithwagtail.org': 'your-distribution-id',
+            },
+        },
+    }
+
+  .. note::
+    In most cases, absolute URLs with ``www`` prefixed domain names should be used in your mapping. Only drop the ``www`` prefix if you're absolutely sure you're not using it (e.g. a subdomain).
+
 Advanced usage
 --------------
 

docs/reference/contrib/index.rst

@@ -46,7 +46,7 @@ Provides a view that generates a Google XML sitemap of your public Wagtail conte
 :doc:`frontendcache`
 --------------------
 
-A module for automatically purging pages from a cache (Varnish, Squid or Cloudflare) when their content is changed.
+A module for automatically purging pages from a cache (Varnish, Squid, Cloudflare or Cloudfront) when their content is changed.
 
 
 :doc:`routablepage`

docs/spelling_wordlist.txt

@@ -4,6 +4,7 @@ backends
 callable
 callables
 Cloudflare
+Cloudfront
 contrib
 Django
 Elasticsearch

wagtail/contrib/wagtailfrontendcache/backends.py

@@ -2,7 +2,9 @@ from __future__ import absolute_import, unicode_literals
 
 import json
 import logging
+import uuid
 
+from django.core.exceptions import ImproperlyConfigured
 from django.utils.six.moves.urllib.error import HTTPError, URLError
 from django.utils.six.moves.urllib.parse import urlencode, urlparse, urlunparse
 from django.utils.six.moves.urllib.request import Request, urlopen
@@ -84,3 +86,56 @@ class CloudflareBackend(BaseBackend):
         if response_json['result'] == 'error':
             logger.error("Couldn't purge '%s' from Cloudflare. Cloudflare error '%s'", url, response_json['msg'])
             return
+
+
+class CloudfrontBackend(BaseBackend):
+    def __init__(self, params):
+        import boto3
+
+        self.client = boto3.client('cloudfront')
+        try:
+            self.cloudfront_distribution_id = params.pop('DISTRIBUTION_ID')
+        except KeyError:
+            raise ImproperlyConfigured(
+                "The setting 'WAGTAILFRONTENDCACHE' requires the object 'DISTRIBUTION_ID'."
+            )
+
+    def purge(self, url):
+        url_parsed = urlparse(url)
+        distribution_id = None
+
+        if isinstance(self.cloudfront_distribution_id, dict):
+            host = url_parsed.hostname
+            if host in self.cloudfront_distribution_id:
+                distribution_id = self.cloudfront_distribution_id.get(host)
+            else:
+                logger.info(
+                    "Couldn't purge '%s' from CloudFront. Hostname '%s' not found in the DISTRIBUTION_ID mapping",
+                    url, host)
+        else:
+            distribution_id = self.cloudfront_distribution_id
+
+        if distribution_id:
+            path = url_parsed.path
+            self._create_invalidation(distribution_id, path)
+
+    def _create_invalidation(self, distribution_id, path):
+        import botocore
+
+        try:
+            self.client.create_invalidation(
+                DistributionId=distribution_id,
+                InvalidationBatch={
+                    'Paths': {
+                        'Quantity': 1,
+                        'Items': [
+                            path,
+                        ]
+                    },
+                    'CallerReference': str(uuid.uuid4())
+                }
+            )
+        except botocore.exceptions.ClientError as e:
+            logger.error(
+                "Couldn't purge '%s' from CloudFront. ClientError: %s %s", path, e.response['Error']['Code'],
+                e.response['Error']['Message'])

wagtail/contrib/wagtailfrontendcache/tests.py

@@ -1,10 +1,13 @@
 from __future__ import absolute_import, unicode_literals
 
+import mock
+
+from django.core.exceptions import ImproperlyConfigured
 from django.test import TestCase
 from django.test.utils import override_settings
 
 from wagtail.contrib.wagtailfrontendcache.backends import (
-    BaseBackend, CloudflareBackend, HTTPBackend)
+    BaseBackend, CloudflareBackend, CloudfrontBackend, HTTPBackend)
 from wagtail.contrib.wagtailfrontendcache.utils import get_backends
 from wagtail.tests.testapp.models import EventIndex
 from wagtail.wagtailcore.models import Page
@@ -45,6 +48,42 @@ class TestBackendConfiguration(TestCase):
         self.assertEqual(backends['cloudflare'].cloudflare_email, 'test@test.com')
         self.assertEqual(backends['cloudflare'].cloudflare_token, 'this is the token')
 
+    def test_cloudfront(self):
+        backends = get_backends(backend_settings={
+            'cloudfront': {
+                'BACKEND': 'wagtail.contrib.wagtailfrontendcache.backends.CloudfrontBackend',
+                'DISTRIBUTION_ID': 'frontend',
+            },
+        })
+
+        self.assertEqual(set(backends.keys()), set(['cloudfront']))
+        self.assertIsInstance(backends['cloudfront'], CloudfrontBackend)
+
+        self.assertEqual(backends['cloudfront'].cloudfront_distribution_id, 'frontend')
+
+    def test_cloudfront_validate_distribution_id(self):
+        with self.assertRaises(ImproperlyConfigured):
+            get_backends(backend_settings={
+                'cloudfront': {
+                    'BACKEND': 'wagtail.contrib.wagtailfrontendcache.backends.CloudfrontBackend',
+                },
+            })
+
+    @mock.patch('wagtail.contrib.wagtailfrontendcache.backends.CloudfrontBackend._create_invalidation')
+    def test_cloudfront_distribution_id_mapping(self, _create_invalidation):
+        backends = get_backends(backend_settings={
+            'cloudfront': {
+                'BACKEND': 'wagtail.contrib.wagtailfrontendcache.backends.CloudfrontBackend',
+                'DISTRIBUTION_ID': {
+                    'www.wagtail.io': 'frontend',
+                }
+            },
+        })
+        backends.get('cloudfront').purge('http://www.wagtail.io/home/events/christmas/')
+        backends.get('cloudfront').purge('http://torchbox.com/blog/')
+
+        _create_invalidation.assert_called_once_with('frontend', '/home/events/christmas/')
+
     def test_multiple(self):
         backends = get_backends(backend_settings={
             'varnish': {