Add date & datetime formatting to AbstractEmailForm + split out render_email method

* Rewrite AbstractEmailForm render to use cleaned_data
* Add date time formatting to AbstractEmailForm's render method according to Django settings SHORT_DATE_FORMAT and
SHORT_DATETIME_FORMAT
* Previously it was iterating over `form` which left no place to remove
data from the submission without removing the field from the form (which
is inadvisable, as discussed on #4313)
* Preserve order of fields in form emails using cleaned data
* Add a test for consistent date rendering in email forms
* update form customisation examples and add note about date / time formatting in email form usage (docs)
* resolves #3733
* resolves #4313

CHANGELOG.txt

@@ -9,7 +9,10 @@ Changelog
  * Added filtering to locked pages report (Karl Hobley)
  * Adds ability to view a group's users via standalone admin URL and a link to this on the group edit view (Karran Besen)
  * Redirect to previous url when deleting/copying/unpublish a page and modify this url via the relevant hooks (Ascani Carlo)
+ * `AbstractEmailForm` will use `SHORT_DATETIME_FORMAT` and `SHORT_DATE_FORMAT` Django settings to format date/time values in email (Haydn Greatnews)
+ * `AbstractEmailForm` now has a separate method (`render_email`) to build up email content on submission emails (Haydn Greatnews)
  * Fix: Ensure link to add a new user works when no users are visible in the users list (LB (Ben Johnston))
+ * Fix: `AbstractEmailForm` saved submission fields are now aligned with the email content fields, `form.cleaned_data` will be used instead of `form.fields` (Haydn Greatnews)
 
 
 2.9 (xx.xx.xxxx) - IN DEVELOPMENT

docs/reference/contrib/forms/customisation.rst

@@ -707,10 +707,57 @@ Example:
         form_builder = CustomFormBuilder
 
 
+.. _form_builder_render_email:
+
+Custom ``render_email`` method
+------------------------------
+
+If you want to change the content of the email that is sent when a form submits you can override the ``render_email`` method.
+
+
+To do this, you need to:
+
+* Ensure you have your form model defined that extends ``wagtail.contrib.forms.models.AbstractEmailForm``.
+* Override the ``render_email`` method in your page model.
+
+Example:
+
+.. code-block:: python
+
+    from datetime import date
+    # ... additional wagtail imports
+    from wagtail.contrib.forms.models import AbstractEmailForm
+
+
+    class FormPage(AbstractEmailForm):
+        # ... fields, content_panels, etc
+
+        def render_email(self, form):
+            # Get the original content (string)
+            email_content = super().render_email(form)
+
+            # Add a title (not part of original method)
+            title = '{}: {}'.format('Form', self.title)
+
+            content = [title, '', email_content, '']
+
+            # Add a link to the form page
+            content.append('{}: {}'.format('Submitted Via', self.full_url))
+
+            # Add the date the form was submitted
+            submitted_date_str = date.today().strftime('%x')
+            content.append('{}: {}'.format('Submitted on', submitted_date_str))
+
+            # Content is joined with a new line to separate each text line
+            content = '\n'.join(content)
+
+            return content
+
+
 Custom ``send_mail`` method
 ---------------------------
 
-If you want to change the content of the email that is sent when a form submits you can override the ``send_mail`` method.
+If you want to change the subject or some other part of how an email is sent when a form submits you can override the ``send_mail`` method.
 
 
 To do this, you need to:
@@ -719,6 +766,7 @@ To do this, you need to:
 * In your models.py file, import the ``wagtail.admin.mail.send_mail`` function.
 * Override the ``send_mail`` method in your page model.
 
+
 Example:
 
 .. code-block:: python
@@ -738,31 +786,9 @@ Example:
             # Email addresses are parsed from the FormPage's addresses field
             addresses = [x.strip() for x in self.to_address.split(',')]
 
-            # Subject can be adjusted, be sure to include the form's defined subject field
+            # Subject can be adjusted (adding submitted date), be sure to include the form's defined subject field
             submitted_date_str = date.today().strftime('%x')
-            subject = self.subject + " - " + submitted_date_str # add date to email subject
+            subject = f"{self.subject} - {submitted_date_str}"
 
-            content = []
+            send_mail(subject, self.render_email(form), addresses, self.from_address,)
 
-            # Add a title (not part of original method)
-            content.append('{}: {}'.format('Form', self.title))
-
-            for field in form:
-                # add the value of each field as a new line
-                value = field.value()
-                if isinstance(value, list):
-                    value = ', '.join(value)
-                content.append('{}: {}'.format(field.label, value))
-
-            # Add a link to the form page
-            content.append('{}: {}'.format('Submitted Via', self.full_url))
-
-            # Add the date the form was submitted
-            content.append('{}: {}'.format('Submitted on', submitted_date_str))
-
-            # Content is joined with a new line to separate each text line
-            content = '\n'.join(content)
-
-            # wagtail.admin.mail - send_mail function is called
-            # This function extends the Django default send_mail function
-            send_mail(subject, content, addresses, self.from_address)

docs/reference/contrib/forms/index.rst

@@ -60,6 +60,8 @@ Within the ``models.py`` of one of your apps, create a model that extends ``wagt
 
 ``AbstractEmailForm`` defines the fields ``to_address``, ``from_address`` and ``subject``, and expects ``form_fields`` to be defined. Any additional fields are treated as ordinary page content - note that ``FormPage`` is responsible for serving both the form page itself and the landing page after submission, so the model definition should include all necessary content fields for both of those views.
 
+Date and datetime values in a form response will be formatted with the `SHORT_DATE_FORMAT <https://docs.djangoproject.com/en/3.0/ref/settings/#short-date-format>`_ and `SHORT_DATETIME_FORMAT <https://docs.djangoproject.com/en/3.0/ref/settings/#short-datetime-format>`_ respectively. (see :ref:`form_builder_render_email` for how to customise the email content).
+
 If you do not want your form page type to offer form-to-email functionality, you can inherit from AbstractForm instead of ``AbstractEmailForm``, and omit the ``to_address``, ``from_address`` and ``subject`` fields from the ``content_panels`` definition.
 
 You now need to create two templates named ``form_page.html`` and ``form_page_landing.html`` (where ``form_page`` is the underscore-formatted version of the class name). ``form_page.html`` differs from a standard Wagtail template in that it is passed a variable ``form``, containing a Django ``Form`` object, in addition to the usual ``page`` variable. A very basic template for the form would thus be:

docs/releases/2.10.rst

@@ -18,12 +18,15 @@ Other features
  * Added filtering to locked pages report (Karl Hobley)
  * Adds ability to view a group's users via standalone admin URL and a link to this on the group edit view (Karran Besen)
  * Redirect to previous url when deleting/copying/unpublish a page and modify this url via the relevant hooks (Ascani Carlo)
+ * ``AbstractEmailForm`` will use ``SHORT_DATETIME_FORMAT`` and ``SHORT_DATE_FORMAT`` Django settings to format date/time values in email (Haydn Greatnews)
+ * ``AbstractEmailForm`` now has a separate method (``render_email``) to build up email content on submission emails. See :ref:`form_builder_render_email`. (Haydn Greatnews)
 
 
 Bug fixes
 ~~~~~~~~~
 
  * Ensure link to add a new user works when no users are visible in the users list (LB (Ben Johnston))
+ * ``AbstractEmailForm`` saved submission fields are now aligned with the email content fields, ``form.cleaned_data`` will be used instead of ``form.fields`` (Haydn Greatnews)
 
 
 Upgrade considerations

wagtail/contrib/forms/models.py

@@ -1,9 +1,12 @@
+import datetime
 import json
 import os
 
+from django.conf import settings
 from django.core.serializers.json import DjangoJSONEncoder
 from django.db import models
 from django.template.response import TemplateResponse
+from django.utils.formats import date_format
 from django.utils.text import slugify
 from django.utils.translation import gettext_lazy as _
 from unidecode import unidecode
@@ -282,14 +285,30 @@ class AbstractEmailForm(AbstractForm):
 
     def send_mail(self, form):
         addresses = [x.strip() for x in self.to_address.split(',')]
+        send_mail(self.subject, self.render_email(form), addresses, self.from_address,)
+
+    def render_email(self, form):
         content = []
+
+        cleaned_data = form.cleaned_data
         for field in form:
-            value = field.value()
+            if field.name not in cleaned_data:
+                continue
+
+            value = cleaned_data.get(field.name)
+
             if isinstance(value, list):
                 value = ', '.join(value)
+
+            # Format dates and datetimes with SHORT_DATE(TIME)_FORMAT
+            if isinstance(value, datetime.datetime):
+                value = date_format(value, settings.SHORT_DATETIME_FORMAT)
+            elif isinstance(value, datetime.date):
+                value = date_format(value, settings.SHORT_DATE_FORMAT)
+
             content.append('{}: {}'.format(field.label, value))
-        content = '\n'.join(content)
-        send_mail(self.subject, content, addresses, self.from_address,)
+
+        return '\n'.join(content)
 
     class Meta:
         abstract = True

wagtail/contrib/forms/tests/test_models.py

@@ -2,11 +2,12 @@
 import json
 
 from django.core import mail
-from django.test import TestCase
+from django.test import TestCase, override_settings
 
 from wagtail.contrib.forms.models import FormSubmission
 from wagtail.contrib.forms.tests.utils import (
-    make_form_page, make_form_page_with_custom_submission, make_form_page_with_redirect)
+    make_form_page, make_form_page_with_custom_submission, make_form_page_with_redirect,
+    make_types_test_form_page)
 from wagtail.core.models import Page
 from wagtail.tests.testapp.models import (
     CustomFormPageSubmission, ExtendedFormField, FormField, FormPageWithCustomFormBuilder,
@@ -476,6 +477,99 @@ class TestFormPageWithCustomFormBuilder(TestCase, WagtailTestUtils):
         self.assertTemplateUsed(response, 'tests/form_page_with_custom_form_builder_landing.html')
 
 
+class TestCleanedDataEmails(TestCase):
+    def setUp(self):
+        # Create a form page
+        self.form_page = make_types_test_form_page()
+
+    def test_empty_field_presence(self):
+        self.client.post('/contact-us/', {})
+
+        # Check the email
+        self.assertEqual(len(mail.outbox), 1)
+        self.assertIn("Single line text: ", mail.outbox[0].body)
+        self.assertIn("Multiline: ", mail.outbox[0].body)
+        self.assertIn("Email: ", mail.outbox[0].body)
+        self.assertIn("Number: ", mail.outbox[0].body)
+        self.assertIn("URL: ", mail.outbox[0].body)
+        self.assertIn("Checkbox: ", mail.outbox[0].body)
+        self.assertIn("Checkboxes: ", mail.outbox[0].body)
+        self.assertIn("Drop down: ", mail.outbox[0].body)
+        self.assertIn("Multiple select: ", mail.outbox[0].body)
+        self.assertIn("Radio buttons: ", mail.outbox[0].body)
+        self.assertIn("Date: ", mail.outbox[0].body)
+        self.assertIn("Datetime: ", mail.outbox[0].body)
+
+    def test_email_field_order(self):
+        self.client.post('/contact-us/', {})
+
+        line_beginnings = [
+            "Single line text: ",
+            "Multiline: ",
+            "Email: ",
+            "Number: ",
+            "URL: ",
+            "Checkbox: ",
+            "Checkboxes: ",
+            "Drop down: ",
+            "Multiple select: ",
+            "Radio buttons: ",
+            "Date: ",
+            "Datetime: ",
+        ]
+
+        # Check the email
+        self.assertEqual(len(mail.outbox), 1)
+        email_lines = mail.outbox[0].body.split('\n')
+
+        for beginning in line_beginnings:
+            message_line = email_lines.pop(0)
+            self.assertTrue(message_line.startswith(beginning))
+
+    @override_settings(SHORT_DATE_FORMAT='m/d/Y')
+    def test_date_normalization(self):
+        self.client.post('/contact-us/', {
+            'date': '12/31/17',
+        })
+
+        # Check the email
+        self.assertEqual(len(mail.outbox), 1)
+        self.assertIn("Date: 12/31/2017", mail.outbox[0].body)
+
+        self.client.post('/contact-us/', {
+            'date': '12/31/1917',
+        })
+
+        # Check the email
+        self.assertEqual(len(mail.outbox), 2)
+        self.assertIn("Date: 12/31/1917", mail.outbox[1].body)
+
+
+    @override_settings(SHORT_DATETIME_FORMAT='m/d/Y P')
+    def test_datetime_normalization(self):
+        self.client.post('/contact-us/', {
+            'datetime': '12/31/17 4:00:00',
+        })
+
+        self.assertEqual(len(mail.outbox), 1)
+        self.assertIn("Datetime: 12/31/2017 4 a.m.", mail.outbox[0].body)
+
+        self.client.post('/contact-us/', {
+            'datetime': '12/31/1917 21:19',
+        })
+
+        self.assertEqual(len(mail.outbox), 2)
+        self.assertIn("Datetime: 12/31/1917 9:19 p.m.", mail.outbox[1].body)
+
+        self.client.post('/contact-us/', {
+            'datetime': '1910-12-21 21:19:12',
+        })
+
+        self.assertEqual(len(mail.outbox), 3)
+        self.assertIn("Datetime: 12/21/1910 9:19 p.m.", mail.outbox[2].body)
+
+
+
 class TestIssue798(TestCase):
     fixtures = ['test.json']
 

wagtail/contrib/forms/tests/utils.py

@@ -116,3 +116,105 @@ def make_form_page_with_redirect(**kwargs):
     )
 
     return form_page
+
+
+def make_types_test_form_page(**kwargs):
+    kwargs.setdefault('title', "Contact us")
+    kwargs.setdefault('slug', "contact-us")
+    kwargs.setdefault('to_address', "to@email.com")
+    kwargs.setdefault('from_address', "from@email.com")
+    kwargs.setdefault('subject', "The subject")
+
+    home_page = Page.objects.get(url_path='/home/')
+    form_page = home_page.add_child(instance=FormPage(**kwargs))
+
+    FormField.objects.create(
+        page=form_page,
+        sort_order=1,
+        label="Single line text",
+        field_type='singleline',
+        required=False,
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=2,
+        label="Multiline",
+        field_type='multiline',
+        required=False,
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=3,
+        label="Email",
+        field_type='email',
+        required=False,
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=4,
+        label="Number",
+        field_type='number',
+        required=False,
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=5,
+        label="URL",
+        field_type='url',
+        required=False,
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=6,
+        label="Checkbox",
+        field_type='checkbox',
+        required=False,
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=7,
+        label="Checkboxes",
+        field_type='checkboxes',
+        required=False,
+        choices='foo,bar,baz',
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=8,
+        label="Drop down",
+        field_type='dropdown',
+        required=False,
+        choices='spam,ham,eggs',
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=9,
+        label="Multiple select",
+        field_type='multiselect',
+        required=False,
+        choices='qux,quux,quuz,corge',
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=10,
+        label="Radio buttons",
+        field_type='radio',
+        required=False,
+        choices='wibble,wobble,wubble',
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=11,
+        label="Date",
+        field_type='date',
+        required=False,
+    )
+    FormField.objects.create(
+        page=form_page,
+        sort_order=12,
+        label="Datetime",
+        field_type='datetime',
+        required=False,
+    )
+
+    return form_page