mirror
/
bridgy-fed
kopia lustrzana https://github.com/snarfed/bridgy-fed
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność

migrate to the app engine standard python 3 runtime! 

https://cloud.google.com/appengine/docs/standard/python3/python-differences
https://cloud.google.com/appengine/docs/standard/python/migrate-to-python3/
python3
Ryan Barrett 2019-12-25 22:20:57 -08:00
rodzic ab55c9d62c
commit 02d36b3b1f
Nie znaleziono w bazie danych klucza dla tego podpisu
ID klucza GPG: 6BE31FDF4776E9D4
30 zmienionych plików z 438 dodań i 599 usunięć
Pokaż statystyki Ściągnij plik aktualizacji Ściągnij plik porównania Expand all files Collapse all files

37
.circleci/config.yml
Wyświetl plik

@ -9,38 +9,45 @@ jobs:
docker:
# https://circleci.com/docs/2.0/google-container-engine/#selecting-a-base-image
- image: google/cloud-sdk
- image: python:2.7.12
environment:
- PYTHONPATH: /usr/lib/google-cloud-sdk/platform/google_appengine
# https://github.com/GoogleCloudPlatform/google-cloud-datastore/issues/168#issuecomment-294418422
- APPLICATION_ID: dev~bridgy-federated
steps:
- checkout
- restore_cache:
key: venv-{{ .Branch }}-{{ checksum "requirements.freeze.txt" }}
key: venv-{{ .Branch }}-{{ checksum "requirements.txt" }}
- run:
name: Python 2 dependencies
name: Base dependencies
command: |
apt-get install -y python-virtualenv google-cloud-sdk-app-engine-python-extras
virtualenv --python=python2 local
. local/bin/activate
pip install -r requirements.freeze.txt
pip install coverage coveralls # for https://coveralls.io/
apt-get update
apt-get install -y python3
- run:
name: Python 3 dependencies
command: |
apt-get install -y python3-venv python3-dev
python3 -m venv local3
. local3/bin/activate
pip install -U -r requirements.txt
pip install mox3
pip install coverage coveralls
- run:
name: Build and test
command: |
. local/bin/activate
python2 -m coverage run --source=. --omit=appengine_config.py,local/\*,oauth-dropins/\*,tests/\* -m unittest discover -v
python2 -m coverage html -d /tmp/coverage_html
# send coverage data to coveralls
. local3/bin/activate
CLOUDSDK_CORE_PROJECT=bridgy-federated gcloud beta emulators datastore start --no-store-on-disk --consistency=1.0 --host-port=localhost:8089 < /dev/null >& /dev/null &
sleep 5s
python -m coverage run --source=. --omit=appengine_config.py,local3/\*,logs.py,tests/\* -m unittest discover -v
python -m coverage html -d /tmp/coverage_html
if [ "$COVERALLS_REPO_TOKEN" != "" ]; then coveralls || true; fi
- save_cache:
key: venv-{{ .Branch }}-{{ checksum "requirements.freeze.txt" }}
key: venv-{{ .Branch }}-{{ checksum "requirements.txt" }}
paths:
- local

24
README.md
Wyświetl plik

@ -15,22 +15,11 @@ License: This project is placed in the public domain.
Development
---
First, install the [Google Cloud SDK](https://cloud.google.com/sdk/gcloud/) (aka `gcloud`) with the `gcloud-appengine-python` and `gcloud-appengine-python-extras` [components](https://cloud.google.com/sdk/docs/components#additional_components).
Once you've done that, run this to find the App Engine libraries directory:
You'll need Python 3. Install the [Google Cloud SDK](https://cloud.google.com/sdk/gcloud/) (aka `gcloud`) with the `gcloud-appengine-python` and `gcloud-appengine-python-extras` [components](https://cloud.google.com/sdk/docs/components#additional_components). Then, run:
```sh
STATE instead use Installation Root: [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk]
gcloud info | grep -o -E '/[^:]+google_appengine'
```
(If that doesn't output anything, try just `gcloud info`, look in the _Python PATH_ section, and try to find the App Engine directory.)
Add that directory to your `$PYTHONPATH`, e.g. `export PYTHONPATH=$PYTHONPATH:/opt/homebrew-cask/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine`. Then, run:
```sh
virtualenv local
source local/bin/activate
python3 -m venv local3
source local3/bin/activate
pip install -r requirements.txt
python -m unittest discover
```
@ -60,17 +49,12 @@ You may need to change [granary](https://github.com/snarfed/granary), [oauth-dro
```sh
pip uninstall -y granary
pip install -e <path to granary>
ln -s <path to granary>/granary \
local/lib/python2.7/site-packages/granary
```
The symlinks are necessary because App Engine's `vendor` module evidently
doesn't follow `.egg-link` or `.pth` files. :/
To deploy to the production instance on App Engine - if @snarfed has added you as an owner - run:
```sh
gcloud -q app deploy --project bridgy-federated *.yaml
gcloud -q beta app deploy --no-cache --project bridgy-federated *.yaml
```

26
activitypub.py
Wyświetl plik

@ -3,12 +3,10 @@
import datetime
import logging
import appengine_config
from google.appengine.ext import ndb
from google.cloud import ndb
from granary import as2, microformats2
import mf2util
from oauth_dropins.webutil import util
from oauth_dropins.webutil import appengine_info, util
from oauth_dropins.webutil.handlers import cache_response
from oauth_dropins.webutil.util import json_dumps, json_loads
import webapp2
@ -86,10 +84,10 @@ Couldn't find a representative h-card (http://microformats.org/wiki/representati
obj = common.postprocess_as2(as2.from_as1(microformats2.json_to_object(hcard)),
key=key)
obj.update({
'inbox': '%s/%s/inbox' % (appengine_config.HOST_URL, domain),
'outbox': '%s/%s/outbox' % (appengine_config.HOST_URL, domain),
'following': '%s/%s/following' % (appengine_config.HOST_URL, domain),
'followers': '%s/%s/followers' % (appengine_config.HOST_URL, domain),
'inbox': '%s/%s/inbox' % (appengine_info.HOST_URL, domain),
'outbox': '%s/%s/outbox' % (appengine_info.HOST_URL, domain),
'following': '%s/%s/following' % (appengine_info.HOST_URL, domain),
'followers': '%s/%s/followers' % (appengine_info.HOST_URL, domain),
})
logging.info('Returning: %s', json_dumps(obj, indent=2))
@ -114,7 +112,7 @@ class InboxHandler(webapp2.RequestHandler):
common.error(self, "Couldn't parse body as JSON", exc_info=True)
obj = activity.get('object') or {}
if isinstance(obj, basestring):
if isinstance(obj, str):
obj = {'id': obj}
type = activity.get('type')
@ -135,7 +133,7 @@ class InboxHandler(webapp2.RequestHandler):
# fetch actor if necessary so we have name, profile photo, etc
for elem in obj, activity:
actor = elem.get('actor')
if actor and isinstance(actor, basestring):
if actor and isinstance(actor, str):
elem['actor'] = common.get_as2(actor).json()
activity_unwrapped = common.redirect_unwrap(activity)
@ -174,7 +172,7 @@ class InboxHandler(webapp2.RequestHandler):
# send AP Accept
accept = {
'@context': 'https://www.w3.org/ns/activitystreams',
'id': util.tag_uri(appengine_config.HOST, 'accept/%s/%s' % (
'id': util.tag_uri(appengine_info.HOST, 'accept/%s/%s' % (
(user_domain, follow.get('id')))),
'type': 'Accept',
'actor': followee,
@ -193,7 +191,7 @@ class InboxHandler(webapp2.RequestHandler):
self, as2.to_as1(follow), proxy=True, protocol='activitypub',
source_as2=json_dumps(follow_unwrapped))
@ndb.transactional
@ndb.transactional()
def undo_follow(self, undo_unwrapped):
"""Replies to an AP Follow request with an Accept request.
@ -222,7 +220,7 @@ class InboxHandler(webapp2.RequestHandler):
# TODO send webmention with 410 of u-follow
app = webapp2.WSGIApplication([
ROUTES = [
(r'/%s/?' % common.DOMAIN_RE, ActorHandler),
(r'/%s/inbox' % common.DOMAIN_RE, InboxHandler),
], debug=appengine_config.DEBUG)
]

13
add_webmention.py
Wyświetl plik

@ -1,10 +1,9 @@
"""HTTP proxy that injects our webmention endpoint.
"""
import datetime
import urllib
import appengine_config
import urllib.parse
from oauth_dropins.webutil import appengine_info
from oauth_dropins.webutil.handlers import cache_response
import requests
import webapp2
@ -21,7 +20,7 @@ class AddWebmentionHandler(webapp2.RequestHandler):
@cache_response(CACHE_TIME)
def get(self, url):
url = urllib.unquote(url)
url = urllib.parse.unquote(url)
if not url.startswith('http://') and not url.startswith('https://'):
common.error(self, 'URL must start with http:// or https://')
@ -36,12 +35,12 @@ class AddWebmentionHandler(webapp2.RequestHandler):
self.response.write(resp.content)
endpoint = LINK_HEADER % (str(self.request.get('endpoint')) or
appengine_config.HOST_URL + '/webmention')
appengine_info.HOST_URL + '/webmention')
self.response.headers.clear()
self.response.headers.update(resp.headers)
self.response.headers.add('Link', endpoint)
app = webapp2.WSGIApplication([
ROUTES = [
('/wm/(.+)', AddWebmentionHandler),
], debug=appengine_config.DEBUG)
]

24
app.py 100644
Wyświetl plik

@ -0,0 +1,24 @@
"""Main WSGI application. Just URL routes to the other modules."""
import importlib
from oauth_dropins.webutil import appengine_info, appengine_config, handlers
import webapp2
routes = []
for module in (
'activitypub',
'add_webmention',
'logs',
'redirect',
'render',
'salmon',
'superfeedr',
'webfinger',
'webmention',
):
routes += importlib.import_module(module).ROUTES
application = handlers.ndb_context_middleware(
webapp2.WSGIApplication(routes, debug=appengine_info.DEBUG),
client=appengine_config.ndb_client)

66
app.yaml
Wyświetl plik

@ -1,28 +1,12 @@
# https://cloud.google.com/appengine/docs/standard/python/config/appref
# application: bridgy-federated
# version: 1
runtime: python27
threadsafe: yes
api_version: 1
default_expiration: 1h
runtime: python37
# default_expiration: 1h
includes:
- local/lib/python2.7/site-packages/oauth_dropins/webutil/app.common.yaml
libraries:
- name: jinja2
version: latest
- name: lxml
version: latest
- name: pycrypto
version: latest
- name: ssl
version: latest
- name: ujson
version: latest
- name: webob
version: latest
# https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
# https://docs.gunicorn.org/en/latest/settings.html#timeout
entrypoint: gunicorn --workers 1 --threads 10 --timeout 60 -b :$PORT app:application
handlers:
@ -61,42 +45,6 @@ handlers:
secure: always
# dynamic
- url: /render
script: render.app
secure: always
- url: /(log|responses)
script: logs.app
secure: always
- url: /r/.+
script: redirect.app
secure: always
- url: /superfeedr/.*
script: superfeedr.app
secure: always
- url: /wm/.+
script: add_webmention.app
secure: always
- url: /webmention
script: webmention.app
secure: always
- url: /[^@/]+/?(inbox)?
script: activitypub.app
secure: always
- url: /[^/]+/salmon/?
script: salmon.app
secure: always
- url: /(acct:)?[^/]+/?
script: webfinger.app
secure: always
- url: /.well-known/.*
script: webfinger.app
- url: .*
script: auto
secure: always

38
appengine_config.py
Wyświetl plik

@ -1,44 +1,14 @@
"""Bridgy App Engine config.
"""
import os
# Load packages from virtualenv
# https://cloud.google.com/appengine/docs/python/tools/libraries27#vendoring
from google.appengine.ext import vendor
try:
vendor.add('local')
except ValueError as e:
import logging
logging.warning("Couldn't set up App Engine vendor virtualenv! %s", e)
from granary.appengine_config import *
if os.environ.get('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
HOST = 'fed.brid.gy'
HOST_URL = '%s://%s' % (SCHEME, HOST)
# Stub out the multiprocessing module. it's not supported on App Engine
# Standard, but humanfriendly uses it for some terminal animation thing that we
# don't need.
import sys
from types import ModuleType
class DummyProcessing(ModuleType):
pass
sys.modules['multiprocessing'] = DummyProcessing
# Make requests and urllib3 play nice with App Engine.
# https://github.com/snarfed/bridgy/issues/396
# http://stackoverflow.com/questions/34574740
from requests_toolbelt.adapters import appengine
appengine.monkeypatch()
# import os
# if os.environ.get('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
# HOST = 'fed.brid.gy'
# HOST_URL = '%s://%s' % (SCHEME, HOST)
# suppresses these INFO logs:
# Sandbox prevented access to file "/usr/local/Caskroom/google-cloud-sdk"
# If it is a static file, check that `application_readable: true` is set in your app.yaml
import logging
class StubsFilter(logging.Filter):

37
common.py
Wyświetl plik

@ -1,29 +1,28 @@
# coding=utf-8
"""Misc common utilities.
"""
from __future__ import unicode_literals
import itertools
import logging
import re
import urlparse
import urllib.parse
from granary import as2
from oauth_dropins.webutil import appengine_info
from oauth_dropins.webutil import handlers, util
import requests
from webmentiontools import send
from webob import exc
import appengine_config
import common
from models import Response
DOMAIN_RE = r'([^/]+\.[^/]+)'
DOMAIN_RE = r'([^/:]+\.[^/:]+)'
ACCT_RE = r'(?:acct:)?([^@]+)@' + DOMAIN_RE
HEADERS = {
'User-Agent': 'Bridgy Fed (https://fed.brid.gy/)',
}
# see redirect_wrap() and redirect_unwrap()
REDIRECT_PREFIX = urlparse.urljoin(appengine_config.HOST_URL, '/r/')
REDIRECT_PREFIX = urllib.parse.urljoin(appengine_info.HOST_URL, '/r/')
XML_UTF8 = "<?xml version='1.0' encoding='UTF-8'?>\n"
# USERNAME = 'me'
# USERNAME_EMOJI = '🌎' # globe
@ -35,12 +34,12 @@ AS2_PUBLIC_AUDIENCE = 'https://www.w3.org/ns/activitystreams#Public'
#
# ActivityPub Content-Type details:
# https://www.w3.org/TR/activitypub/#retrieving-objects
CONTENT_TYPE_AS2_LD = b'application/ld+json; profile="https://www.w3.org/ns/activitystreams"'
CONTENT_TYPE_AS2 = b'application/activity+json'
CONTENT_TYPE_AS1 = b'application/stream+json'
CONTENT_TYPE_HTML = b'text/html; charset=utf-8'
CONTENT_TYPE_ATOM = b'application/atom+xml'
CONTENT_TYPE_MAGIC_ENVELOPE = b'application/magic-envelope+xml'
CONTENT_TYPE_AS2_LD = 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"'
CONTENT_TYPE_AS2 = 'application/activity+json'
CONTENT_TYPE_AS1 = 'application/stream+json'
CONTENT_TYPE_HTML = 'text/html; charset=utf-8'
CONTENT_TYPE_ATOM = 'application/atom+xml'
CONTENT_TYPE_MAGIC_ENVELOPE = 'application/magic-envelope+xml'
CONNEG_HEADERS_AS2 = {
'Accept': '%s; q=0.9, %s; q=0.8' % (CONTENT_TYPE_AS2, CONTENT_TYPE_AS2_LD),
@ -129,7 +128,7 @@ def get_as2(url):
if not (as2 and as2['href']):
_error(resp)
resp = requests_get(urlparse.urljoin(resp.url, as2['href']),
resp = requests_get(urllib.parse.urljoin(resp.url, as2['href']),
headers=CONNEG_HEADERS_AS2)
if content_type(resp) in (CONTENT_TYPE_AS2, CONTENT_TYPE_AS2_LD):
return resp
@ -182,7 +181,7 @@ def send_webmentions(handler, activity_wrapped, proxy=None, **response_props):
for tag in tags:
if tag.get('objectType') == 'mention':
url = tag.get('url')
if url and url.startswith(appengine_config.HOST_URL):
if url and url.startswith(appengine_info.HOST_URL):
targets.append(redirect_unwrap(url))
if verb in ('follow', 'like', 'share'):
@ -245,7 +244,7 @@ def postprocess_as2(activity, target=None, key=None):
# https://github.com/tootsuite/mastodon/blob/bc2c263504e584e154384ecc2d804aeb1afb1ba3/app/services/activitypub/process_account_service.rb#L77
activity['publicKey'] = {
'id': activity.get('preferredUsername'),
'publicKeyPem': key.public_pem(),
'publicKeyPem': key.public_pem().decode(),
}
return activity
@ -337,9 +336,9 @@ def postprocess_as2_actor(actor):
"""
url = actor.get('url')
if url:
domain = urlparse.urlparse(url).netloc
domain = urllib.parse.urlparse(url).netloc
actor.setdefault('preferredUsername', domain)
actor['id'] = '%s/%s' % (appengine_config.HOST_URL, domain)
actor['id'] = '%s/%s' % (appengine_info.HOST_URL, domain)
actor['url'] = redirect_wrap(url)
# required by pixelfed. https://github.com/snarfed/bridgy-fed/issues/39
@ -375,11 +374,11 @@ def redirect_unwrap(val):
elif isinstance(val, list):
return [redirect_unwrap(v) for v in val]
elif isinstance(val, basestring):
elif isinstance(val, str):
if val.startswith(REDIRECT_PREFIX):
return val[len(REDIRECT_PREFIX):]
elif val.startswith(appengine_config.HOST_URL):
elif val.startswith(appengine_info.HOST_URL):
return util.follow_redirects(
util.domain_from_link(urlparse.urlparse(val).path.strip('/'))).url
util.domain_from_link(urllib.parse.urlparse(val).path.strip('/'))).url
return val

2
index.yaml 100644
Wyświetl plik

@ -0,0 +1,2 @@
indexes:
# AUTOGENERATED

10
logs.py
Wyświetl plik

@ -1,9 +1,7 @@
"""Render recent responses and logs."""
import calendar
import datetime
import urllib
import appengine_config
import urllib.parse
from oauth_dropins.webutil import util
from oauth_dropins.webutil.handlers import TemplateHandler
@ -31,7 +29,7 @@ class ResponsesHandler(TemplateHandler):
for r in responses:
r.source_link = util.pretty_link(r.source())
r.target_link = util.pretty_link(r.target())
r.log_url_path = '/log?' + urllib.urlencode({
r.log_url_path = '/log?' + urllib.parse.urlencode({
'key': r.key.id(),
'start_time': calendar.timegm(r.updated.timetuple()),
})
@ -41,7 +39,7 @@ class ResponsesHandler(TemplateHandler):
}
app = webapp2.WSGIApplication([
ROUTES = [
('/log', LogHandler),
('/responses', ResponsesHandler),
], debug=appengine_config.DEBUG)
]

13
models.py
Wyświetl plik

@ -3,15 +3,14 @@
Based on webfinger-unofficial/user.py.
"""
import logging
import urllib
import urllib.parse
from Crypto.PublicKey import RSA
from django_salmon import magicsigs
from google.appengine.ext import ndb
from google.cloud import ndb
from oauth_dropins.webutil import appengine_info
from oauth_dropins.webutil.models import StringIdModel
import appengine_config
class MagicKey(StringIdModel):
"""Stores a user's public/private key pair used for Magic Signatures.
@ -31,7 +30,7 @@ class MagicKey(StringIdModel):
private_exponent = ndb.StringProperty(required=True)
@staticmethod
@ndb.transactional
@ndb.transactional()
def get_or_create(domain):
"""Loads and returns a MagicKey. Creates it if necessary."""
key = MagicKey.get_by_id(domain)
@ -51,11 +50,13 @@ class MagicKey(StringIdModel):
self.mod, self.public_exponent)
def public_pem(self):
"""Returns: bytes"""
rsa = RSA.construct((magicsigs.base64_to_long(str(self.mod)),
magicsigs.base64_to_long(str(self.public_exponent))))
return rsa.exportKey(format='PEM')
def private_pem(self):
"""Returns: bytes"""
rsa = RSA.construct((magicsigs.base64_to_long(str(self.mod)),
magicsigs.base64_to_long(str(self.public_exponent)),
magicsigs.base64_to_long(str(self.private_exponent))))
@ -107,7 +108,7 @@ class Response(StringIdModel):
"""Returns the Bridgy Fed proxy URL to render this response as HTML."""
if self.source_mf2 or self.source_as2 or self.source_atom:
source, target = self.key.id().split(' ')
return '%s/render?%s' % (appengine_config.HOST_URL, urllib.urlencode({
return '%s/render?%s' % (appengine_info.HOST_URL, urllib.parse.urlencode({
'source': source,
'target': target,
}))

5
redirect.py
Wyświetl plik

@ -19,7 +19,6 @@ from oauth_dropins.webutil.util import json_dumps
import ujson as json
import webapp2
import appengine_config
import common
CACHE_TIME = datetime.timedelta(seconds=15)
@ -68,6 +67,6 @@ class RedirectHandler(webapp2.RequestHandler):
self.response.write(json_dumps(obj, indent=2))
app = webapp2.WSGIApplication([
ROUTES = [
(r'/r/.+', RedirectHandler),
], debug=appengine_config.DEBUG)
]

8
render.py
Wyświetl plik

@ -2,8 +2,6 @@
"""Renders mf2 proxy pages based on stored Responses."""
import datetime
import appengine_config
from granary import as2, atom, microformats2
from oauth_dropins.webutil.handlers import cache_response, ModernHandler
from oauth_dropins.webutil import util
@ -47,6 +45,6 @@ class RenderHandler(ModernHandler):
self.response.write(html)
app = webapp2.WSGIApplication([
('/render', RenderHandler),
], debug=appengine_config.DEBUG)
ROUTES = [
('/render', RenderHandler),
]

64
requirements.freeze.txt
Wyświetl plik

@ -1,64 +0,0 @@
backports.functools-lru-cache==1.5
beautifulsoup4==4.8.0
brevity==0.2.17
bs4==0.0.1
cachetools==3.1.1
certifi==2019.9.11
chardet==3.0.4
coverage==4.0.3
-e git+https://github.com/snarfed/django-salmon.git@3869986d640222ffe30b9678686b04500ac1de27#egg=django_salmon
docopt==0.6.2
extras==1.0.0
feedgen==0.8.0
feedparser==5.2.1
fixtures==3.0.0
funcsigs==1.0.2
future==0.17.1
gdata==2.0.18
google-api-python-client==1.7.11
google-auth==1.6.3
google-auth-httplib2==0.0.3
-e git+git@github.com:snarfed/granary.git#egg=granary
html2text==2019.8.11
html5lib==1.0.1
httplib2==0.14.0
-e git+git@github.com:snarfed/httpsig.git@6f0ebfe00af39fb75d50c7a29f46da49e8190d70#egg=httpsig
humanfriendly==4.18
humanize==0.5.1
idna==2.8
Jinja2==2.10.3
linecache2==1.0.0
lxml==3.7.3
MarkupSafe==1.1.1
-e git+git@github.com:snarfed/mf2py.git#egg=mf2py
mf2util==0.5.1
mock==2.0.0
mox3==0.28.0
-e git+git@github.com:snarfed/oauth-dropins.git#egg=oauth_dropins
oauth2client==4.1.3
oauthlib==3.1.0
pbr==5.4.3
pyasn1==0.4.7
pyasn1-modules==0.2.6
pycrypto==2.6.1
PySocks==1.7.1
python-dateutil==2.8.0
python-mimeparse==1.6.0
python-tumblpy==1.1.4
requests==2.22.0
requests-oauthlib==1.2.0
requests-toolbelt==0.9.1
rsa==4.0
six==1.12.0
soupsieve==1.9.4
testtools==2.3.0
traceback2==1.4.0
tweepy==3.8.0
ujson==1.35
unittest2==1.1.0
uritemplate==3.0.0
urllib3==1.25.6
webapp2==3.0.0b1
webencodings==0.5.1
-e git+https://github.com/snarfed/webmention-tools.git@a67be86024868a6e99f51be568f78505a79a1e2f#egg=webmentiontools
WebOb==1.8.5

26
requirements.txt
Wyświetl plik

@ -1,16 +1,10 @@
-e git+https://github.com/snarfed/django-salmon.git#egg=django_salmon
-e git+https://github.com/snarfed/webmention-tools.git#egg=webmentiontools
beautifulsoup4
feedparser
granary>=1.14
httpsig
jinja2>=2.10
mf2py>=1.0.4
mf2util>=0.5.0
mock
mox3>=0.24.0
pycrypto
requests==2.10.0
requests-toolbelt==0.6.2
ujson>=1.35
urllib3>=1.14
git+https://github.com/snarfed/django-salmon.git#egg=django_salmon
git+https://github.com/snarfed/httpsig.git@signature_header#egg=httpsig
git+https://github.com/snarfed/webmention-tools.git@python3#egg=webmentiontools
git+https://github.com/snarfed/oauth-dropins.git@master#egg=oauth_dropins
git+https://github.com/snarfed/granary.git@master#egg=granary
feedparser~=5.2
google-cloud-logging~=1.14
mf2util~=0.5.0
pycrypto~=2.6
requests~=2.22

8
salmon.py
Wyświetl plik

@ -6,8 +6,6 @@ https://github.com/salmon-protocol/salmon-protocol/blob/master/draft-panzer-magi
import logging
from xml.etree.ElementTree import ParseError
import appengine_config
from django_salmon import magicsigs, utils
from granary import atom
from oauth_dropins.webutil import util
@ -41,7 +39,7 @@ class SlapHandler(webapp2.RequestHandler):
parsed = utils.parse_magic_envelope(self.request.body)
except ParseError as e:
common.error(self, 'Could not parse POST body as XML', exc_info=True)
data = utils.decode(parsed['data'])
data = parsed['data']
logging.info('Decoded: %s', data)
# check that we support this activity type
@ -80,7 +78,7 @@ class SlapHandler(webapp2.RequestHandler):
common.send_webmentions(self, activity, protocol='ostatus', source_atom=data)
app = webapp2.WSGIApplication([
ROUTES = [
(r'/%s/salmon' % common.ACCT_RE, SlapHandler),
(r'/()%s/salmon' % common.DOMAIN_RE, SlapHandler),
], debug=appengine_config.DEBUG)
]

6
superfeedr.py
Wyświetl plik

@ -8,8 +8,6 @@ import logging
import webapp2
import appengine_config
class SuperfeedrHandler(webapp2.RequestHandler):
"""Superfeedr subscription callback handler.
@ -24,6 +22,6 @@ class SuperfeedrHandler(webapp2.RequestHandler):
get = post
app = webapp2.WSGIApplication([
ROUTES = [
(r'/superfeedr/.*', SuperfeedrHandler),
], debug=appengine_config.DEBUG)
]

3
tests/__init__.py
Wyświetl plik

@ -1,3 +0,0 @@
# webutil/tests/__init__.py has setup code that makes App Engine SDK's
# bundled libraries importable.
import oauth_dropins.webutil.tests

95
tests/test_activitypub.py
Wyświetl plik

@ -3,21 +3,19 @@
TODO: test error handling
"""
from __future__ import unicode_literals
import copy
import urllib
from unittest.mock import call, patch
from mock import call, patch
from oauth_dropins.webutil import util
from oauth_dropins.webutil.testutil import requests_response
from oauth_dropins.webutil.util import json_dumps, json_loads
import requests
import activitypub
from activitypub import ActorHandler, app
from app import application
import common
from models import Follower, MagicKey, Response
import testutil
from . import testutil
REPLY_OBJECT = {
@ -140,7 +138,7 @@ class ActivityPubTest(testutil.TestCase):
def setUp(self):
super(ActivityPubTest, self).setUp()
ActorHandler.get.cache_clear()
activitypub.ActorHandler.get.cache_clear()
def test_actor_handler(self, _, mock_get, __):
mock_get.return_value = requests_response("""
@ -149,12 +147,13 @@ class ActivityPubTest(testutil.TestCase):
</body>
""", url='https://foo.com/', content_type=common.CONTENT_TYPE_HTML)
got = app.get_response('/foo.com')
got = application.get_response('/foo.com')
mock_get.assert_called_once_with('http://foo.com/', headers=common.HEADERS,
stream=True, timeout=util.HTTP_TIMEOUT)
self.assertEquals(200, got.status_int)
self.assertEquals(common.CONTENT_TYPE_AS2, got.headers['Content-Type'])
self.assertEquals({
self.assertEqual(200, got.status_int)
type = got.headers['Content-Type']
self.assertTrue(type.startswith(common.CONTENT_TYPE_AS2), type)
self.assertEqual({
'@context': 'https://www.w3.org/ns/activitystreams',
'type' : 'Person',
'name': 'Mrs. ☕ Foo',
@ -168,7 +167,7 @@ class ActivityPubTest(testutil.TestCase):
'followers': 'http://localhost/foo.com/followers',
'publicKey': {
'id': 'foo.com',
'publicKeyPem': MagicKey.get_by_id('foo.com').public_pem(),
'publicKeyPem': MagicKey.get_by_id('foo.com').public_pem().decode(),
},
}, json_loads(got.body))
@ -181,11 +180,11 @@ class ActivityPubTest(testutil.TestCase):
</body>
""")
got = app.get_response('/foo.com')
got = application.get_response('/foo.com')
mock_get.assert_called_once_with('http://foo.com/', headers=common.HEADERS,
stream=True, timeout=util.HTTP_TIMEOUT)
self.assertEquals(400, got.status_int)
self.assertIn('representative h-card', got.body)
self.assertEqual(400, got.status_int)
self.assertIn('representative h-card', got.body.decode())
def test_inbox_reply_object(self, *mocks):
self._test_inbox_reply(REPLY_OBJECT, REPLY_OBJECT, *mocks)
@ -202,9 +201,9 @@ class ActivityPubTest(testutil.TestCase):
'<html><head><link rel="webmention" href="/webmention"></html>')
mock_post.return_value = requests_response()
got = app.get_response('/foo.com/inbox', method='POST',
body=json_dumps(as2))
self.assertEquals(200, got.status_int, got.body)
got = application.get_response('/foo.com/inbox', method='POST',
body=json_dumps(as2).encode())
self.assertEqual(200, got.status_int, got.body)
mock_get.assert_called_once_with(
'http://orig/post', headers=common.HEADERS, verify=False)
@ -233,15 +232,15 @@ class ActivityPubTest(testutil.TestCase):
mock_head.return_value = requests_response(url='http://this/')
got = app.get_response('/foo.com/inbox', method='POST',
body=json_dumps(reply))
self.assertEquals(200, got.status_int, got.body)
got = application.get_response('/foo.com/inbox', method='POST',
body=json_dumps(reply).encode())
self.assertEqual(200, got.status_int, got.body)
mock_head.assert_called_once_with(
'http://this', allow_redirects=True, stream=True, timeout=15)
mock_get.assert_not_called()
mock_post.assert_not_called()
self.assertEquals(0, Response.query().count())
self.assertEqual(0, Response.query().count())
def test_inbox_mention_object(self, *mocks):
self._test_inbox_mention(MENTION_OBJECT, *mocks)
@ -255,9 +254,9 @@ class ActivityPubTest(testutil.TestCase):
'<html><head><link rel="webmention" href="/webmention"></html>')
mock_post.return_value = requests_response()
got = app.get_response('/foo.com/inbox', method='POST',
body=json_dumps(as2))
self.assertEquals(200, got.status_int, got.body)
got = application.get_response('/foo.com/inbox', method='POST',
body=json_dumps(as2).encode())
self.assertEqual(200, got.status_int, got.body)
mock_get.assert_called_once_with(
'http://target/', headers=common.HEADERS, verify=False)
@ -290,9 +289,9 @@ class ActivityPubTest(testutil.TestCase):
]
mock_post.return_value = requests_response()
got = app.get_response('/foo.com/inbox', method='POST',
body=json_dumps(LIKE_WRAPPED))
self.assertEquals(200, got.status_int)
got = application.get_response('/foo.com/inbox', method='POST',
body=json_dumps(LIKE_WRAPPED).encode())
self.assertEqual(200, got.status_int)
as2_headers = copy.deepcopy(common.HEADERS)
as2_headers.update(common.CONNEG_HEADERS_AS2_HTML)
@ -302,8 +301,8 @@ class ActivityPubTest(testutil.TestCase):
))
args, kwargs = mock_post.call_args
self.assertEquals(('http://orig/webmention',), args)
self.assertEquals({
self.assertEqual(('http://orig/webmention',), args)
self.assertEqual({
# TODO
'source': 'http://localhost/render?source=http%3A%2F%2Fthis%2Flike__ok&target=http%3A%2F%2Forig%2Fpost',
'target': 'http://orig/post',
@ -327,9 +326,9 @@ class ActivityPubTest(testutil.TestCase):
]
mock_post.return_value = requests_response()
got = app.get_response('/foo.com/inbox', method='POST',
body=json_dumps(FOLLOW_WRAPPED))
self.assertEquals(200, got.status_int)
got = application.get_response('/foo.com/inbox', method='POST',
body=json_dumps(FOLLOW_WRAPPED).encode())
self.assertEqual(200, got.status_int)
as2_headers = copy.deepcopy(common.HEADERS)
as2_headers.update(common.CONNEG_HEADERS_AS2_HTML)
@ -340,13 +339,13 @@ class ActivityPubTest(testutil.TestCase):
# check AP Accept
self.assertEqual(2, len(mock_post.call_args_list))
args, kwargs = mock_post.call_args_list[0]
self.assertEquals(('http://follower/inbox',), args)
self.assertEquals(ACCEPT, kwargs['json'])
self.assertEqual(('http://follower/inbox',), args)
self.assertEqual(ACCEPT, kwargs['json'])
# check webmention
args, kwargs = mock_post.call_args_list[1]
self.assertEquals(('https://realize.be/webmention',), args)
self.assertEquals({
self.assertEqual(('https://realize.be/webmention',), args)
self.assertEqual({
'source': 'http://localhost/render?source=https%3A%2F%2Fmastodon.social%2F6d1a&target=https%3A%2F%2Frealize.be%2F',
'target': 'https://realize.be/',
}, kwargs['data'])
@ -367,9 +366,9 @@ class ActivityPubTest(testutil.TestCase):
Follower(id=Follower._id('realize.be', FOLLOW['actor'])).put()
got = app.get_response('/foo.com/inbox', method='POST',
body=json_dumps(UNDO_FOLLOW_WRAPPED))
self.assertEquals(200, got.status_int)
got = application.get_response('/foo.com/inbox', method='POST',
body=json_dumps(UNDO_FOLLOW_WRAPPED).encode())
self.assertEqual(200, got.status_int)
follower = Follower.get_by_id('realize.be %s' % FOLLOW['actor'])
self.assertEqual('inactive', follower.status)
@ -377,25 +376,25 @@ class ActivityPubTest(testutil.TestCase):
def test_inbox_undo_follow_doesnt_exist(self, mock_head, mock_get, mock_post):
mock_head.return_value = requests_response(url='https://realize.be/')
got = app.get_response('/foo.com/inbox', method='POST',
body=json_dumps(UNDO_FOLLOW_WRAPPED))
self.assertEquals(200, got.status_int)
got = application.get_response('/foo.com/inbox', method='POST',
body=json_dumps(UNDO_FOLLOW_WRAPPED).encode())
self.assertEqual(200, got.status_int)
def test_inbox_undo_follow_inactive(self, mock_head, mock_get, mock_post):
mock_head.return_value = requests_response(url='https://realize.be/')
Follower(id=Follower._id('realize.be', 'https://mastodon.social/users/swentel'),
status='inactive').put()
got = app.get_response('/foo.com/inbox', method='POST',
body=json_dumps(UNDO_FOLLOW_WRAPPED))
self.assertEquals(200, got.status_int)
got = application.get_response('/foo.com/inbox', method='POST',
body=json_dumps(UNDO_FOLLOW_WRAPPED).encode())
self.assertEqual(200, got.status_int)
def test_inbox_unsupported_type(self, *_):
got = app.get_response('/foo.com/inbox', method='POST', body=json_dumps({
got = application.get_response('/foo.com/inbox', method='POST', body=json_dumps({
'@context': ['https://www.w3.org/ns/activitystreams'],
'id': 'https://xoxo.zone/users/aaronpk#follows/40',
'type': 'Block',
'actor': 'https://xoxo.zone/users/aaronpk',
'object': 'http://snarfed.org/',
}))
self.assertEquals(501, got.status_int)
}).encode())
self.assertEqual(501, got.status_int)

12
tests/test_add_webmention.py
Wyświetl plik

@ -1,14 +1,12 @@
# coding=utf-8
"""Unit tests for add_webmention.py.
"""
from __future__ import unicode_literals
from unittest import mock
import mock
from app import application
from oauth_dropins.webutil.testutil import requests_response
import requests
from add_webmention import app
import testutil
from . import testutil
@mock.patch('requests.get')
@ -25,7 +23,7 @@ class AddWebmentionTest(testutil.TestCase):
self.resp.status_code = 202
mock_get.return_value = self.resp
got = app.get_response('/wm/http://url')
got = application.get_response('/wm/http://url')
self.assertEqual(202, got.status_int)
self.assertEqual(self.resp._content, got.body)
self.assertEqual(['bar'], got.headers.getall('Foo'))
@ -35,6 +33,6 @@ class AddWebmentionTest(testutil.TestCase):
def test_endpoint_param(self, mock_get):
mock_get.return_value = self.resp
got = app.get_response('/wm/http://url?endpoint=https://end/point')
got = application.get_response('/wm/http://url?endpoint=https://end/point')
self.assertEqual(200, got.status_int)
self.assertEqual('<https://end/point>; rel="webmention"', got.headers['Link'])

5
tests/test_common.py
Wyświetl plik

@ -1,16 +1,15 @@
# coding=utf-8
"""Unit tests for common.py."""
from __future__ import unicode_literals
import logging
from unittest import mock
import mock
from oauth_dropins.webutil import util
from oauth_dropins.webutil.testutil import requests_response
import requests
from webob import exc
import common
import testutil
from . import testutil
HTML = requests_response('<html></html>', headers={
'Content-Type': common.CONTENT_TYPE_HTML,

26
tests/test_models.py
Wyświetl plik

@ -1,9 +1,7 @@
# coding=utf-8
"""Unit tests for models.py."""
from __future__ import unicode_literals
from models import MagicKey, Response
import testutil
from . import testutil
class MagicKeyTest(testutil.TestCase):
@ -18,7 +16,7 @@ class MagicKeyTest(testutil.TestCase):
assert self.key.private_exponent
same = MagicKey.get_or_create('y.z')
self.assertEquals(same, self.key)
self.assertEqual(same, self.key)
def test_href(self):
href = self.key.href()
@ -28,35 +26,35 @@ class MagicKeyTest(testutil.TestCase):
def test_public_pem(self):
pem = self.key.public_pem()
self.assertTrue(pem.startswith('-----BEGIN PUBLIC KEY-----\n'), pem)
self.assertTrue(pem.endswith('-----END PUBLIC KEY-----'), pem)
self.assertTrue(pem.decode().startswith('-----BEGIN PUBLIC KEY-----\n'), pem)
self.assertTrue(pem.decode().endswith('-----END PUBLIC KEY-----'), pem)
def test_public_pem(self):
def test_private_pem(self):
pem = self.key.private_pem()
self.assertTrue(pem.startswith('-----BEGIN RSA PRIVATE KEY-----\n'), pem)
self.assertTrue(pem.endswith('-----END RSA PRIVATE KEY-----'), pem)
self.assertTrue(pem.decode().startswith('-----BEGIN RSA PRIVATE KEY-----\n'), pem)
self.assertTrue(pem.decode().endswith('-----END RSA PRIVATE KEY-----'), pem)
class ResponseTest(testutil.TestCase):
def test_constructor(self):
resp = Response('abc', 'xyz')
self.assertEquals('abc xyz', resp.key.id())
self.assertEqual('abc xyz', resp.key.id())
resp = Response('abc#1', 'xyz#Z')
self.assertEquals('abc__1 xyz__Z', resp.key.id())
self.assertEqual('abc__1 xyz__Z', resp.key.id())
def test_get_or_create(self):
resp = Response.get_or_create('abc', 'xyz')
self.assertEquals('abc xyz', resp.key.id())
self.assertEqual('abc xyz', resp.key.id())
resp = Response.get_or_create('abc#1', 'xyz#Z')
self.assertEquals('abc__1 xyz__Z', resp.key.id())
self.assertEqual('abc__1 xyz__Z', resp.key.id())
def test_proxy_url(self):
resp = Response.get_or_create('abc', 'xyz')
self.assertIsNone(resp.proxy_url())
resp.source_as2 = 'as2'
self.assertEquals('http://localhost/render?source=abc&target=xyz',
self.assertEqual('http://localhost/render?source=abc&target=xyz',
resp.proxy_url())

17
tests/test_redirect.py
Wyświetl plik

@ -1,14 +1,15 @@
"""Unit tests for redirect.py.
"""
import copy
from unittest.mock import patch
from mock import patch
from oauth_dropins.webutil.testutil import requests_response
from app import application
import common
from redirect import app, RedirectHandler
from test_webmention import REPOST_HTML, REPOST_AS2
import testutil
from redirect import RedirectHandler
from .test_webmention import REPOST_HTML, REPOST_AS2
from . import testutil
class RedirectTest(testutil.TestCase):
@ -18,16 +19,16 @@ class RedirectTest(testutil.TestCase):
RedirectHandler.get.cache_clear()
def test_redirect(self):
got = app.get_response('/r/https://foo.com/bar?baz=baj&biff')
got = application.get_response('/r/https://foo.com/bar?baz=baj&biff')
self.assertEqual(302, got.status_int)
self.assertEqual('https://foo.com/bar?baz=baj&biff', got.headers['Location'])
def test_redirect_scheme_missing(self):
got = app.get_response('/r/asdf.com')
got = application.get_response('/r/asdf.com')
self.assertEqual(400, got.status_int)
def test_redirect_url_missing(self):
got = app.get_response('/r/')
got = application.get_response('/r/')
self.assertEqual(404, got.status_int)
def test_as2(self):
@ -51,7 +52,7 @@ class RedirectTest(testutil.TestCase):
mock_get.return_value = requests_response(
REPOST_HTML, content_type=common.CONTENT_TYPE_HTML)
got = app.get_response('/r/https://foo.com/bar', headers={'Accept': accept})
got = application.get_response('/r/https://foo.com/bar', headers={'Accept': accept})
args, kwargs = mock_get.call_args
self.assertEqual(('https://foo.com/bar',), args)

37
tests/test_render.py
Wyświetl plik

@ -1,12 +1,11 @@
# coding=utf-8
"""Unit tests for render.py."""
from __future__ import unicode_literals
from oauth_dropins.webutil.util import json_dumps
from app import application
from models import Response
import testutil
from render import app, RenderHandler
from render import RenderHandler
from . import testutil
class RenderTest(testutil.TestCase):
@ -61,35 +60,35 @@ class RenderTest(testutil.TestCase):
def test_render_errors(self):
for source, target in ('', ''), ('abc', ''), ('', 'xyz'):
resp = app.get_response('/render?source=%s&target=%s' % (source, target))
self.assertEquals(400, resp.status_int, resp.body)
resp = application.get_response('/render?source=%s&target=%s' % (source, target))
self.assertEqual(400, resp.status_int, resp.body)
# no Response
resp = app.get_response('/render?source=abc&target=xyz')
self.assertEquals(404, resp.status_int)
resp = application.get_response('/render?source=abc&target=xyz')
self.assertEqual(404, resp.status_int)
# no source data
Response(id='abc xyz').put()
resp = app.get_response('/render?source=abc&target=xyz')
self.assertEquals(404, resp.status_int)
resp = application.get_response('/render?source=abc&target=xyz')
self.assertEqual(404, resp.status_int)
def test_render_as2(self):
Response(id='abc xyz', source_as2=json_dumps(self.as2)).put()
resp = app.get_response('/render?source=abc&target=xyz')
self.assertEquals(200, resp.status_int)
self.assert_multiline_equals(self.html, resp.body.decode('utf-8'),
resp = application.get_response('/render?source=abc&target=xyz')
self.assertEqual(200, resp.status_int)
self.assert_multiline_equals(self.html, resp.body.decode(),
ignore_blanks=True)
def test_render_mf2(self):
Response(id='abc xyz', source_mf2=json_dumps(self.mf2)).put()
resp = app.get_response('/render?source=abc&target=xyz')
self.assertEquals(200, resp.status_int)
self.assert_multiline_equals(self.html, resp.body.decode('utf-8'),
resp = application.get_response('/render?source=abc&target=xyz')
self.assertEqual(200, resp.status_int)
self.assert_multiline_equals(self.html, resp.body.decode(),
ignore_blanks=True)
def test_render_atom(self):
Response(id='abc xyz', source_atom=self.atom).put()
resp = app.get_response('/render?source=abc&target=xyz')
self.assertEquals(200, resp.status_int)
self.assert_multiline_equals(self.html, resp.body.decode('utf-8'),
resp = application.get_response('/render?source=abc&target=xyz')
self.assertEqual(200, resp.status_int)
self.assert_multiline_equals(self.html, resp.body.decode(),
ignore_blanks=True)

33
tests/test_salmon.py
Wyświetl plik

@ -3,26 +3,24 @@
TODO: test error handling
"""
from __future__ import unicode_literals
import copy
import datetime
import urllib
from unittest import mock
from django_salmon import magicsigs
import mock
from oauth_dropins.webutil.testutil import requests_response, UrlopenResult
import requests
from app import application
import common
from models import MagicKey, Response
from salmon import app
import testutil
from . import testutil
@mock.patch('requests.post')
@mock.patch('requests.get')
@mock.patch('requests.head')
@mock.patch('urllib2.urlopen')
@mock.patch('urllib.request.urlopen')
class SalmonTest(testutil.TestCase):
def setUp(self):
@ -53,8 +51,9 @@ class SalmonTest(testutil.TestCase):
mock_post.return_value = requests_response()
slap = magicsigs.magic_envelope(atom_slap, common.CONTENT_TYPE_ATOM, self.key)
got = app.get_response('/foo.com@foo.com/salmon', method='POST', body=slap)
self.assertEquals(200, got.status_int)
got = application.get_response('/foo.com@foo.com/salmon', method='POST',
body=slap)
self.assertEqual(200, got.status_int)
# check salmon magic key discovery
mock_urlopen.assert_has_calls((
@ -84,7 +83,7 @@ class SalmonTest(testutil.TestCase):
<content>I hereby reply.</content>
<title>My Reply</title>
<updated>%s</updated>
</entry>""" % datetime.datetime.now().isoformat(b'T')
</entry>""" % datetime.datetime.now().isoformat('T')
self.send_slap(mock_urlopen, mock_head, mock_get, mock_post, atom_reply)
# check webmention post
@ -115,7 +114,7 @@ class SalmonTest(testutil.TestCase):
<activity:verb>http://activitystrea.ms/schema/1.0/like</activity:verb>
<activity:object>http://orig/post</activity:object>
<updated>%s</updated>
</entry>""" % datetime.datetime.now().isoformat(b'T')
</entry>""" % datetime.datetime.now().isoformat('T')
self.send_slap(mock_urlopen, mock_head, mock_get, mock_post, atom_like)
# check webmention post
@ -137,14 +136,14 @@ class SalmonTest(testutil.TestCase):
self.assertEqual(atom_like, resp.source_atom)
def test_bad_envelope(self, *mocks):
got = app.get_response('/foo.com/salmon', method='POST',
body='not xml'.encode('utf-8'))
self.assertEquals(400, got.status_int)
got = application.get_response('/foo.com/salmon', method='POST',
body=b'not xml')
self.assertEqual(400, got.status_int)
def test_bad_inner_xml(self, *mocks):
slap = magicsigs.magic_envelope('not xml', common.CONTENT_TYPE_ATOM, self.key)
got = app.get_response('/foo.com/salmon', method='POST', body=slap)
self.assertEquals(400, got.status_int)
got = application.get_response('/foo.com/salmon', method='POST', body=slap)
self.assertEqual(400, got.status_int)
def test_rsvp_not_supported(self, *mocks):
slap = magicsigs.magic_envelope("""\
@ -155,5 +154,5 @@ class SalmonTest(testutil.TestCase):
<activity:verb>http://activitystrea.ms/schema/1.0/rsvp</activity:verb>
<activity:object>http://orig/event</activity:object>
</entry>""", common.CONTENT_TYPE_ATOM, self.key)
got = app.get_response('/foo.com/salmon', method='POST', body=slap)
self.assertEquals(501, got.status_int)
got = application.get_response('/foo.com/salmon', method='POST', body=slap)
self.assertEqual(501, got.status_int)

104
tests/test_webfinger.py
Wyświetl plik

@ -3,19 +3,19 @@
TODO: test error handling
"""
from __future__ import unicode_literals
import urllib
from unittest import mock
import urllib.parse
import mock
from oauth_dropins.webutil import util
from oauth_dropins.webutil.testutil import requests_response
from oauth_dropins.webutil.util import json_loads
import requests
from app import application
import common
import models
import testutil
from webfinger import app, UserHandler, WebfingerHandler
from webfinger import UserHandler, WebfingerHandler
from . import testutil
USER = 'foo.com@foo.com'
@ -68,7 +68,7 @@ class WebFingerTest(testutil.TestCase):
}, {
'rel': 'http://schemas.google.com/g/2010#updates-from',
'type': 'application/atom+xml',
'href': 'https://granary.io/url?url=https%3A%2F%2Ffoo.com%2F&input=html&hub=https%3A%2F%2Ffoo.com%2F&output=atom',
'href': 'https://granary.io/url?input=html&output=atom&url=https%3A%2F%2Ffoo.com%2F&hub=https%3A%2F%2Ffoo.com%2F',
}, {
'rel': 'hub',
'href': 'https://bridgy-fed.superfeedr.com/'
@ -85,46 +85,50 @@ class WebFingerTest(testutil.TestCase):
}
def test_host_meta_handler_xrd(self):
got = app.get_response('/.well-known/host-meta')
self.assertEquals(200, got.status_int)
self.assertEquals('application/xrd+xml; charset=utf-8',
got = application.get_response('/.well-known/host-meta')
self.assertEqual(200, got.status_int)
self.assertEqual('application/xrd+xml; charset=utf-8',
got.headers['Content-Type'])
self.assertTrue(got.body.startswith('<?xml'), got.body)
body = got.body.decode()
self.assertTrue(body.startswith('<?xml'), body)
def test_host_meta_handler_xrds(self):
got = app.get_response('/.well-known/host-meta.xrds')
self.assertEquals(200, got.status_int)
self.assertEquals('application/xrds+xml; charset=utf-8',
got = application.get_response('/.well-known/host-meta.xrds')
self.assertEqual(200, got.status_int)
self.assertEqual('application/xrds+xml; charset=utf-8',
got.headers['Content-Type'])
self.assertTrue(got.body.startswith('<XRDS'), got.body)
body = got.body.decode()
self.assertTrue(body.startswith('<XRDS'), body)
def test_host_meta_handler_jrd(self):
got = app.get_response('/.well-known/host-meta.json')
self.assertEquals(200, got.status_int)
self.assertEquals('application/json; charset=utf-8',
got = application.get_response('/.well-known/host-meta.json')
self.assertEqual(200, got.status_int)
self.assertEqual('application/json; charset=utf-8',
got.headers['Content-Type'])
self.assertTrue(got.body.startswith('{'), got.body)
body = got.body.decode()
self.assertTrue(body.startswith('{'), body)
@mock.patch('requests.get')
def test_user_handler(self, mock_get):
mock_get.return_value = requests_response(self.html, url = 'https://foo.com/')
got = app.get_response('/foo.com', headers={'Accept': 'application/json'})
self.assertEquals(200, got.status_int)
self.assertEquals('application/json; charset=utf-8',
got = application.get_response('/acct:foo.com',
headers={'Accept': 'application/json'})
self.assertEqual(200, got.status_int)
self.assertEqual('application/json; charset=utf-8',
got.headers['Content-Type'])
mock_get.assert_called_once_with('http://foo.com/', headers=common.HEADERS,
stream=True, timeout=util.HTTP_TIMEOUT)
self.assertEquals(self.expected_webfinger, json_loads(got.body))
self.assertEqual(self.expected_webfinger, json_loads(got.body.decode()))
# check that magic key is persistent
again = json_loads(app.get_response(
'/foo.com', headers={'Accept': 'application/json'}).body)
self.assertEquals(self.key.href(), again['magic_keys'][0]['value'])
again = json_loads(application.get_response(
'/acct:foo.com', headers={'Accept': 'application/json'}).body.decode())
self.assertEqual(self.key.href(), again['magic_keys'][0]['value'])
links = {l['rel']: l['href'] for l in again['links']}
self.assertEquals(self.key.href(), links['magic-public-key'])
self.assertEqual(self.key.href(), links['magic-public-key'])
@mock.patch('requests.get')
def test_user_handler_with_atom_feed(self, mock_get):
@ -138,13 +142,14 @@ class WebFingerTest(testutil.TestCase):
""" + self.html
mock_get.return_value = requests_response(html, url = 'https://foo.com/')
got = app.get_response('/foo.com', headers={'Accept': 'application/json'})
self.assertEquals(200, got.status_int)
got = application.get_response('/acct:foo.com',
headers={'Accept': 'application/json'})
self.assertEqual(200, got.status_int)
self.assertIn({
'rel': 'http://schemas.google.com/g/2010#updates-from',
'type': 'application/atom+xml',
'href': 'https://foo.com/use-this',
}, json_loads(got.body)['links'])
}, json_loads(got.body.decode())['links'])
@mock.patch('requests.get')
def test_user_handler_with_push_header(self, mock_get):
@ -155,12 +160,13 @@ class WebFingerTest(testutil.TestCase):
'<http://a.custom.hub/>; rel="hub"',
})
got = app.get_response('/foo.com', headers={'Accept': 'application/json'})
self.assertEquals(200, got.status_int)
got = application.get_response('/acct:foo.com',
headers={'Accept': 'application/json'})
self.assertEqual(200, got.status_int)
self.assertIn({
'rel': 'hub',
'href': 'http://a.custom.hub/',
}, json_loads(got.body)['links'])
}, json_loads(got.body.decode())['links'])
@mock.patch('requests.get')
def test_user_handler_no_hcard(self, mock_get):
@ -171,16 +177,16 @@ class WebFingerTest(testutil.TestCase):
</div>
</body>
""")
got = app.get_response('/foo.com')
got = application.get_response('/acct:foo.com')
mock_get.assert_called_once_with('http://foo.com/', headers=common.HEADERS,
stream=True, timeout=util.HTTP_TIMEOUT)
self.assertEquals(400, got.status_int)
self.assertIn('representative h-card', got.body)
self.assertEqual(400, got.status_int)
self.assertIn('representative h-card', got.body.decode())
def test_user_handler_bad_tld(self):
got = app.get_response('/foo.json')
self.assertEquals(404, got.status_int)
self.assertIn("doesn't look like a domain", got.body)
got = application.get_response('/acct:foo.json')
self.assertEqual(404, got.status_int)
self.assertIn("doesn't look like a domain", got.body.decode())
@mock.patch('requests.get')
def test_webfinger_handler(self, mock_get):
@ -188,13 +194,14 @@ class WebFingerTest(testutil.TestCase):
for resource in ('foo.com@foo.com', 'acct:foo.com@foo.com', 'xyz@foo.com',
'foo.com', 'http://foo.com/', 'https://foo.com/'):
url = '/.well-known/webfinger?%s' % urllib.urlencode(
url = '/.well-known/webfinger?%s' % urllib.parse.urlencode(
{'resource': resource})
got = app.get_response(url, headers={'Accept': 'application/json'})
self.assertEquals(200, got.status_int, got.body)
self.assertEquals('application/json; charset=utf-8',
got = application.get_response(url, headers={'Accept': 'application/json'})
body = got.body.decode()
self.assertEqual(200, got.status_int, body)
self.assertEqual('application/json; charset=utf-8',
got.headers['Content-Type'])
self.assertEquals(self.expected_webfinger, json_loads(got.body))
self.assertEqual(self.expected_webfinger, json_loads(body))
@mock.patch('requests.get')
def test_webfinger_handler_custom_username(self, mock_get):
@ -217,10 +224,11 @@ class WebFingerTest(testutil.TestCase):
for resource in ('customuser@foo.com', 'acct:customuser@foo.com',
'foo.com', 'http://foo.com/', 'https://foo.com/'):
url = '/.well-known/webfinger?%s' % urllib.urlencode(
url = '/.well-known/webfinger?%s' % urllib.parse.urlencode(
{'resource': resource})
got = app.get_response(url, headers={'Accept': 'application/json'})
self.assertEquals(200, got.status_int, got.body)
self.assertEquals('application/json; charset=utf-8',
got = application.get_response(url, headers={'Accept': 'application/json'})
body = got.body.decode()
self.assertEqual(200, got.status_int, body)
self.assertEqual('application/json; charset=utf-8',
got.headers['Content-Type'])
self.assertEquals(self.expected_webfinger, json_loads(got.body))
self.assertEqual(self.expected_webfinger, json_loads(body))

214
tests/test_webmention.py
Wyświetl plik

@ -3,23 +3,21 @@
TODO: test error handling
"""
from __future__ import unicode_literals
import copy
import urllib
import urllib2
from unittest import mock
from urllib.parse import urlencode
from django_salmon import magicsigs, utils
import feedparser
from granary import atom, microformats2
from httpsig.sign import HeaderSigner
import mock
from mock import call
from oauth_dropins.webutil import util
from oauth_dropins.webutil.testutil import requests_response
from oauth_dropins.webutil.util import json_dumps, json_loads
import requests
import activitypub
from app import application
from common import (
AS2_PUBLIC_AUDIENCE,
CONNEG_HEADERS_AS2,
@ -31,9 +29,8 @@ from common import (
HEADERS,
)
from models import Follower, MagicKey, Response
import testutil
import webmention
from webmention import app
from . import testutil
REPOST_HTML = """\
<html>
@ -277,19 +274,18 @@ class WebmentionTest(testutil.TestCase):
kwargs['headers']['Content-Type'])
env = utils.parse_magic_envelope(kwargs['data'])
data = utils.decode(env['data'])
assert magicsigs.verify(None, data, env['sig'], key=self.key)
assert magicsigs.verify(None, env['data'], env['sig'].encode(), key=self.key)
return data
return env['data']
def test_bad_source_url(self, mock_get, mock_post):
got = app.get_response('/webmention', method='POST', body=b'')
self.assertEquals(400, got.status_int)
got = application.get_response('/webmention', method='POST', body=b'')
self.assertEqual(400, got.status_int)
mock_get.side_effect = ValueError('foo bar')
got = app.get_response('/webmention', method='POST',
body=urllib.urlencode({'source': 'bad'}))
self.assertEquals(400, got.status_int)
got = application.get_response('/webmention', method='POST',
body=urlencode({'source': 'bad'}).encode())
self.assertEqual(400, got.status_int)
def test_no_source_entry(self, mock_get, mock_post):
mock_get.return_value = requests_response("""
@ -299,12 +295,12 @@ class WebmentionTest(testutil.TestCase):
</body>
</html>""", content_type=CONTENT_TYPE_HTML)
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/post',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(400, got.status_int)
}).encode())
self.assertEqual(400, got.status_int)
mock_get.assert_has_calls((self.req('http://a/post'),))
@ -316,12 +312,12 @@ class WebmentionTest(testutil.TestCase):
</body>
</html>""", content_type=CONTENT_TYPE_HTML)
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/post',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_has_calls((self.req('http://a/post'),))
@ -331,9 +327,10 @@ class WebmentionTest(testutil.TestCase):
content_type=CONTENT_TYPE_HTML),
ValueError('foo bar'))
got = app.get_response('/webmention', method='POST',
body=urllib.urlencode({'source': 'http://a/post'}))
self.assertEquals(400, got.status_int)
got = application.get_response(
'/webmention', method='POST',
body=urlencode({'source': 'http://a/post'}).encode())
self.assertEqual(400, got.status_int)
def test_target_fetch_fails(self, mock_get, mock_post):
mock_get.side_effect = (
@ -341,21 +338,22 @@ class WebmentionTest(testutil.TestCase):
content_type=CONTENT_TYPE_HTML),
requests.Timeout('foo bar'))
got = app.get_response('/webmention', method='POST',
body=urllib.urlencode({'source': 'http://a/post'}))
self.assertEquals(502, got.status_int)
got = application.get_response(
'/webmention', method='POST',
body=urlencode({'source': 'http://a/post'}).encode())
self.assertEqual(502, got.status_int)
def test_no_backlink(self, mock_get, mock_post):
mock_get.return_value = requests_response(
self.reply_html.replace('<a href="http://localhost/"></a>', ''),
content_type=CONTENT_TYPE_HTML)
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/post',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(400, got.status_int)
}).encode())
self.assertEqual(400, got.status_int)
mock_get.assert_has_calls((self.req('http://a/post'),))
@ -363,12 +361,12 @@ class WebmentionTest(testutil.TestCase):
mock_get.side_effect = self.activitypub_gets
mock_post.return_value = requests_response('abc xyz', status=203)
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(203, got.status_int)
}).encode())
self.assertEqual(203, got.status_int)
mock_get.assert_has_calls((
self.req('http://a/reply'),
@ -404,12 +402,12 @@ class WebmentionTest(testutil.TestCase):
mock_get.side_effect = self.activitypub_gets
mock_post.return_value = requests_response('abc xyz')
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
args, kwargs = mock_post.call_args
self.assertEqual(('https://foo.com/inbox',), args)
@ -431,12 +429,12 @@ class WebmentionTest(testutil.TestCase):
mock_get.side_effect = [self.reply, orig_as2_resp, self.actor]
mock_post.return_value = requests_response('abc xyz', status=203)
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(203, got.status_int)
}).encode())
self.assertEqual(203, got.status_int)
mock_get.assert_has_calls((
self.req('http://a/reply'),
@ -454,12 +452,12 @@ class WebmentionTest(testutil.TestCase):
mock_get.side_effect = self.activitypub_gets
mock_post.return_value = requests_response('abc xyz')
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
args, kwargs = mock_post.call_args
self.assertEqual(('https://foo.com/inbox',), args)
@ -469,12 +467,12 @@ class WebmentionTest(testutil.TestCase):
mock_get.side_effect = [self.repost, self.orig_as2, self.actor]
mock_post.return_value = requests_response('abc xyz')
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/repost',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_has_calls((
self.req('http://a/repost'),
@ -503,12 +501,12 @@ class WebmentionTest(testutil.TestCase):
self.actor]
mock_post.return_value = requests_response('abc xyz')
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_has_calls((
self.req('http://a/reply'),
@ -535,11 +533,11 @@ class WebmentionTest(testutil.TestCase):
mock_get.side_effect = [missing_url, self.orig_as2, self.actor]
mock_post.return_value = requests_response('abc xyz', status=203)
got = app.get_response('/webmention', method='POST', body=urllib.urlencode({
got = application.get_response('/webmention', method='POST', body=urlencode({
'source': 'http://a/repost',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(203, got.status_int)
}).encode())
self.assertEqual(203, got.status_int)
args, kwargs = mock_post.call_args
self.assertEqual(('https://foo.com/inbox',), args)
@ -570,11 +568,11 @@ class WebmentionTest(testutil.TestCase):
mock_get.side_effect = [repost, author, self.orig_as2, self.actor]
mock_post.return_value = requests_response('abc xyz', status=201)
got = app.get_response('/webmention', method='POST', body=urllib.urlencode({
got = application.get_response('/webmention', method='POST', body=urlencode({
'source': 'http://a/repost',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(201, got.status_int)
}).encode())
self.assertEqual(201, got.status_int)
args, kwargs = mock_post.call_args
self.assertEqual(('https://foo.com/inbox',), args)
@ -609,24 +607,23 @@ class WebmentionTest(testutil.TestCase):
last_follow=json_dumps({'actor': {
'inbox': 'https://unused/2',
}}))
self.datastore_stub.Flush()
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://orig/post',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_has_calls((
self.req('http://orig/post'),
))
inboxes = ('https://public/inbox', 'https://shared/inbox', 'https://inbox')
self.assertEquals(len(inboxes), len(mock_post.call_args_list))
self.assertEqual(len(inboxes), len(mock_post.call_args_list))
for call, inbox in zip(mock_post.call_args_list, inboxes):
self.assertEquals((inbox,), call[0])
self.assertEquals(self.create_as2, call[1]['json'])
self.assertEqual((inbox,), call[0])
self.assertEqual(self.create_as2, call[1]['json'])
for inbox in inboxes:
resp = Response.get_by_id('http://orig/post %s' % inbox)
@ -647,33 +644,32 @@ class WebmentionTest(testutil.TestCase):
Follower.get_or_create(
'orig', 'https://mastodon/aaa',
last_follow=json_dumps({'actor': {'inbox': 'https://inbox'}}))
self.datastore_stub.Flush()
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://orig/post',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
self.assertEquals(('https://inbox',), mock_post.call_args[0])
self.assertEqual(('https://inbox',), mock_post.call_args[0])
create = copy.deepcopy(self.create_as2)
create['object'].update({
'image': [{'url': 'http://im/age', 'type': 'Image'}],
'attachment': [{'url': 'http://im/age', 'type': 'Image'}],
})
self.assertEquals(create, mock_post.call_args[1]['json'])
self.assertEqual(create, mock_post.call_args[1]['json'])
def test_activitypub_follow(self, mock_get, mock_post):
mock_get.side_effect = [self.follow, self.actor]
mock_post.return_value = requests_response('abc xyz')
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/follow',
'target': 'https://fed.brid.gy/',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_has_calls((
self.req('http://a/follow'),
@ -699,12 +695,12 @@ class WebmentionTest(testutil.TestCase):
def test_salmon_reply(self, mock_get, mock_post):
mock_get.side_effect = [self.reply, self.orig_html_atom, self.orig_atom]
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'http://orig/post',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_has_calls((
self.req('http://a/reply'),
@ -716,19 +712,19 @@ class WebmentionTest(testutil.TestCase):
parsed = feedparser.parse(data)
entry = parsed.entries[0]
self.assertEquals('http://a/reply', entry['id'])
self.assertEqual('http://a/reply', entry['id'])
self.assertIn({
'rel': 'alternate',
'href': 'http://a/reply',
'type': 'text/html',
}, entry['links'])
self.assertEquals({
self.assertEqual({
'type': 'text/html',
'href': 'http://orig/post',
'ref': 'tag:fed.brid.gy,2017-08-22:orig-post'
}, entry['thr_in-reply-to'])
self.assertEquals(
'<a class="u-in-reply-to" href="http://orig/post">foo ☕ bar</a><br />\n<a href="http://localhost/"></a>',
self.assertEqual(
'<a class="u-in-reply-to" href="http://orig/post">foo ☕ bar</a><br></br>\n<a href="http://localhost/"></a>',
entry.content[0]['value'])
resp = Response.get_by_id('http://a/reply http://orig/post')
@ -740,12 +736,12 @@ class WebmentionTest(testutil.TestCase):
def test_salmon_like(self, mock_get, mock_post):
mock_get.side_effect = [self.like, self.orig_html_atom, self.orig_atom]
got = app.get_response(
'/webmention', method='POST', body=urllib.urlencode({
got = application.get_response(
'/webmention', method='POST', body=urlencode({
'source': 'http://a/like',
'target': 'http://orig/post',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_has_calls((
self.req('http://a/like'),
@ -757,13 +753,13 @@ class WebmentionTest(testutil.TestCase):
parsed = feedparser.parse(data)
entry = parsed.entries[0]
self.assertEquals('http://a/like', entry['id'])
self.assertEqual('http://a/like', entry['id'])
self.assertIn({
'rel': 'alternate',
'href': 'http://a/like',
'type': 'text/html',
}, entry['links'])
self.assertEquals('http://orig/post', entry['activity_object'])
self.assertEqual('http://orig/post', entry['activity_object'])
resp = Response.get_by_id('http://a/like http://orig/post')
self.assertEqual('out', resp.direction)
@ -791,11 +787,11 @@ class WebmentionTest(testutil.TestCase):
})
mock_get.side_effect = [self.reply, self.orig_html_atom, orig_atom, webfinger]
got = app.get_response('/webmention', method='POST', body=urllib.urlencode({
got = application.get_response('/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'http://orig/post',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_any_call(
'http://orig/.well-known/webfinger?resource=acct:ryan@orig',
@ -809,12 +805,12 @@ class WebmentionTest(testutil.TestCase):
</html>""", 'http://orig/url')
mock_get.side_effect = [self.reply, orig_no_atom]
got = app.get_response('/webmention', method='POST', body=urllib.urlencode({
got = application.get_response('/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'http://orig/post',
}))
self.assertEquals(400, got.status_int)
self.assertIn('Target post http://orig/url has no Atom link', got.body)
}).encode())
self.assertEqual(400, got.status_int)
self.assertIn('Target post http://orig/url has no Atom link', got.body.decode())
self.assertIsNone(Response.get_by_id('http://a/reply http://orig/post'))
@ -827,11 +823,11 @@ class WebmentionTest(testutil.TestCase):
</html>""", 'http://orig/url')
mock_get.side_effect = [self.reply, orig_relative, self.orig_atom]
got = app.get_response('/webmention', method='POST', body=urllib.urlencode({
got = application.get_response('/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'http://orig/post',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_any_call('http://orig/atom/1', headers=HEADERS,
stream=True, timeout=util.HTTP_TIMEOUT)
@ -847,11 +843,11 @@ class WebmentionTest(testutil.TestCase):
</html>""", 'http://orig/url')
mock_get.side_effect = [self.reply, orig_base, self.orig_atom]
got = app.get_response('/webmention', method='POST', body=urllib.urlencode({
got = application.get_response('/webmention', method='POST', body=urlencode({
'source': 'http://a/reply',
'target': 'http://orig/post',
}))
self.assertEquals(200, got.status_int)
}).encode())
self.assertEqual(200, got.status_int)
mock_get.assert_any_call('http://orig/base/atom/1', headers=HEADERS,
stream=True, timeout=util.HTTP_TIMEOUT)

21
tests/testutil.py
Wyświetl plik

@ -2,31 +2,28 @@
"""
import copy
import unittest
from unittest.mock import ANY, call
from google.appengine.datastore import datastore_stub_util
from google.appengine.ext import testbed
from mock import ANY, call
from oauth_dropins.webutil import testutil, util
from oauth_dropins.webutil.appengine_config import ndb_client
import requests
import common
class TestCase(unittest.TestCase, testutil.Asserts):
maxDiff = None
def setUp(self):
super(TestCase, self).setUp()
self.testbed = testbed.Testbed()
self.testbed.activate()
hrd_policy = datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=.5)
self.testbed.init_datastore_v3_stub(consistency_policy=hrd_policy)
self.datastore_stub = self.testbed.get_stub('datastore_v3')
self.testbed.init_memcache_stub()
self.testbed.init_mail_stub()
# clear datastore
requests.post('http://%s/reset' % ndb_client.host)
self.ndb_context = ndb_client.context()
self.ndb_context.__enter__()
def tearDown(self):
self.testbed.deactivate()
self.ndb_context.__exit__(None, None, None)
super(TestCase, self).tearDown()
def req(self, url, **kwargs):

33
webfinger.py
Wyświetl plik

@ -10,16 +10,13 @@ TODO: test:
* acct: URI handling
* user URL that redirects
"""
from __future__ import unicode_literals
import datetime
import logging
import urllib
import urlparse
import appengine_config
import urllib.parse
import urllib.parse
import mf2util
from oauth_dropins.webutil import handlers, util
from oauth_dropins.webutil import appengine_info, handlers, util
from oauth_dropins.webutil.util import json_dumps
import webapp2
@ -50,7 +47,7 @@ class UserHandler(handlers.XrdOrJrdHandler):
# find representative h-card. try url, then url's home page, then domain
urls = ['http://%s/' % domain]
if url:
urls = [url, urlparse.urljoin(url, '/')] + urls
urls = [url, urllib.parse.urljoin(url, '/')] + urls
for candidate in urls:
resp = common.requests_get(candidate)
@ -83,9 +80,9 @@ Couldn't find a representative h-card (http://microformats.org/wiki/representati
# discover atom feed, if any
atom = parsed.find('link', rel='alternate', type=common.CONTENT_TYPE_ATOM)
if atom and atom['href']:
atom = urlparse.urljoin(resp.url, atom['href'])
atom = urllib.parse.urljoin(resp.url, atom['href'])
else:
atom = 'https://granary.io/url?' + urllib.urlencode({
atom = 'https://granary.io/url?' + urllib.parse.urlencode({
'input': 'html',
'output': 'atom',
'url': resp.url,
@ -122,15 +119,15 @@ Couldn't find a representative h-card (http://microformats.org/wiki/representati
# ActivityPub
{
'rel': 'self',
'type': 'application/activity+json',
'type': common.CONTENT_TYPE_AS2,
# use HOST_URL instead of e.g. request.host_url because it
# sometimes lost port, e.g. http://localhost:8080 would become
# just http://localhost. no clue how or why.
'href': '%s/%s' % (appengine_config.HOST_URL, domain),
'href': '%s/%s' % (appengine_info.HOST_URL, domain),
}, {
'rel': 'inbox',
'type': 'application/activity+json',
'href': '%s/%s/inbox' % (appengine_config.HOST_URL, domain),
'type': common.CONTENT_TYPE_AS2,
'href': '%s/%s/inbox' % (appengine_info.HOST_URL, domain),
},
# OStatus
@ -146,7 +143,7 @@ Couldn't find a representative h-card (http://microformats.org/wiki/representati
'href': key.href(),
}, {
'rel': 'salmon',
'href': '%s/%s/salmon' % (appengine_config.HOST_URL, domain),
'href': '%s/%s/salmon' % (appengine_info.HOST_URL, domain),
}]
})
logging.info('Returning WebFinger data: %s', json_dumps(data, indent=2))
@ -163,7 +160,7 @@ class WebfingerHandler(UserHandler):
try:
_, domain = util.parse_acct_uri(resource)
except ValueError:
domain = urlparse.urlparse(resource).netloc or resource
domain = urllib.parse.urlparse(resource).netloc or resource
url = None
if resource.startswith('http://') or resource.startswith('https://'):
@ -172,7 +169,7 @@ class WebfingerHandler(UserHandler):
return super(WebfingerHandler, self).template_vars(domain, url=url)
app = webapp2.WSGIApplication([
(r'/%s/?' % common.DOMAIN_RE, UserHandler),
ROUTES = [
(r'/acct:%s/?' % common.DOMAIN_RE, UserHandler),
('/.well-known/webfinger', WebfingerHandler),
] + handlers.HOST_META_ROUTES, debug=appengine_config.DEBUG)
] + handlers.HOST_META_ROUTES

30
webmention.py
Wyświetl plik

@ -5,15 +5,13 @@ TODO tests:
* salmon rel via webfinger via author.name + domain
"""
import logging
import urllib
import urlparse
import appengine_config
import urllib.parse
from urllib.parse import urlencode
import django_salmon
from django_salmon import magicsigs
import feedparser
from google.appengine.ext.ndb import Key
from google.cloud.ndb import Key
from granary import as2, atom, microformats2, source
import mf2util
from oauth_dropins.webutil import util
@ -44,7 +42,7 @@ class WebmentionHandler(webapp2.RequestHandler):
source = util.get_required_param(self, 'source')
source_resp = common.requests_get(source)
self.source_url = source_resp.url or source
self.source_domain = urlparse.urlparse(self.source_url).netloc.split(':')[0]
self.source_domain = urllib.parse.urlparse(self.source_url).netloc.split(':')[0]
self.source_mf2 = util.parse_mf2(source_resp)
# logging.debug('Parsed mf2 for %s: %s', source_resp.url, json_dumps(self.source_mf2 indent=2))
@ -52,7 +50,7 @@ class WebmentionHandler(webapp2.RequestHandler):
# check for backlink to bridgy fed (for webmention spec and to confirm
# source's intent to federate to mastodon)
if (self.request.host_url not in source_resp.text and
urllib.quote(self.request.host_url, safe='') not in source_resp.text):
urllib.parse.quote(self.request.host_url, safe='') not in source_resp.text):
common.error(self, "Couldn't find link to %s" % self.request.host_url)
# convert source page to ActivityStreams
@ -190,7 +188,7 @@ class WebmentionHandler(webapp2.RequestHandler):
actor = actor.get('url') or actor.get('id')
if not inbox_url and not actor:
common.error(self, 'Target object has no actor or attributedTo with URL or id.')
elif not isinstance(actor, basestring):
elif not isinstance(actor, str):
common.error(self, 'Target actor or attributedTo has unexpected url or id object: %r' % actor)
if not inbox_url:
@ -204,7 +202,7 @@ class WebmentionHandler(webapp2.RequestHandler):
# common.error(self, 'Target actor has no inbox')
return []
inbox_url = urlparse.urljoin(target_url, inbox_url)
inbox_url = urllib.parse.urljoin(target_url, inbox_url)
return [(resp, inbox_url)]
def try_salmon(self):
@ -250,8 +248,8 @@ class WebmentionHandler(webapp2.RequestHandler):
if base and base.get('href'):
base_url = base['href']
atom_link = parsed.find('link', rel='alternate', type=common.CONTENT_TYPE_ATOM)
atom_url = urlparse.urljoin(
resp.target(), urlparse.urljoin(base_url, atom_link['href']))
atom_url = urllib.parse.urljoin(
resp.target(), urllib.parse.urljoin(base_url, atom_link['href']))
feed = common.requests_get(atom_url).text
parsed = feedparser.parse(feed)
@ -281,7 +279,7 @@ class WebmentionHandler(webapp2.RequestHandler):
if not endpoint:
# try webfinger
parsed = urlparse.urlparse(resp.target())
parsed = urllib.parse.urlparse(resp.target())
# TODO: test missing email
email = entry.author_detail.get('email') or '@'.join(
(entry.author_detail.name, parsed.netloc))
@ -307,11 +305,11 @@ class WebmentionHandler(webapp2.RequestHandler):
logging.info('Converted %s to Atom:\n%s', self.source_url, entry)
# sign reply and wrap in magic envelope
domain = urlparse.urlparse(self.source_url).netloc
domain = urllib.parse.urlparse(self.source_url).netloc
key = MagicKey.get_or_create(domain)
logging.info('Using key for %s: %s', domain, key)
magic_envelope = magicsigs.magic_envelope(
entry, common.CONTENT_TYPE_ATOM, key)
entry, common.CONTENT_TYPE_ATOM, key).decode()
logging.info('Sending Salmon slap to %s', endpoint)
common.requests_post(
@ -320,6 +318,6 @@ class WebmentionHandler(webapp2.RequestHandler):
return True
app = webapp2.WSGIApplication([
ROUTES = [
('/webmention', WebmentionHandler),
], debug=appengine_config.DEBUG)
]