Wykres commitów

311 Commity (a48ce4f7aa64d250328072c870ab69dca3e755d2)

Autor SHA1 Wiadomość Data
Ryan Barrett b20bbf0564
memcache config: try to only use prod Memorystore on prod GAE 2024-06-19 11:01:46 -07:00
Ryan Barrett 4e489a4ee0
activitypub.inbox: store seen activity ids in memcache, short circuit on seen ids earlier 2024-06-13 13:54:37 -07:00
Ryan Barrett 3c06739889
common.report_exception bug fix 2024-06-13 12:02:54 -07:00
Ryan Barrett ba00850840
turn on memorystore ndb global cache in hub and atproto_firehose 2024-06-04 19:27:52 -07:00
Ryan Barrett 3ccce1bc89
start using memorystore memcached for ndb global cache 2024-06-04 15:40:32 -07:00
Ryan Barrett 07517f1685
ndb global cache: limit to users and profile objects, not other objects/activities 2024-06-04 14:27:04 -07:00
Ryan Barrett 83893d0ea7
HTTP caching headers: switch to flask_util.headers
for #432, fixes https://console.cloud.google.com/errors/detail/CMK10__x4ZbgbQ;time=P7D;refresh=true?project=bridgy-federated
2024-06-04 14:19:39 -07:00
Ryan Barrett 712da61181
auth error reporting: standardize messages, include HTTP context
trying to collect them into two top-level Error Reporting errors, with details for each one in the user field

for #566
2024-06-04 11:27:17 -07:00
Ryan Barrett e8e2c9692d
common.report_error bug fix
fixes https://console.cloud.google.com/errors/detail/COyxtrLf5r2XowE;time=P30D?project=bridgy-federated
2024-06-03 16:20:53 -07:00
Ryan Barrett cc564a2b19
turn on authorization checks! HTTP sig matches actor, actor is allowed to modify object
for #566
2024-06-03 14:12:08 -07:00
Ryan Barrett 9c4888b02e
switch from flask_util.cached to HTTP caching, suppress some verbose logging 2024-05-30 14:55:35 -07:00
Ryan Barrett 6ffc4a3c3b
infra: add new router flex service, switch tasks to run on it
receive, send, webmention, and poll-feed
2024-05-23 20:40:54 -07:00
Ryan Barrett 348987f0b4
add Protocol.REQUIRES_OLD_ACCOUNT, enable for ActivityPub 2024-05-11 19:20:33 -07:00
Ryan Barrett f2b5f79489
ATProto firehose: add handle wrapper that catches exceptions
also refactor and simplify load_dids, error reporting. #978
2024-05-10 19:59:12 -07:00
Ryan Barrett b49fe13e59
ATProto firehose: use report_exception() so we get stack traces 2024-05-09 21:31:41 -07:00
Ryan Barrett 940c0e8cae
ATProto firehose: don't assume required fields, handle error reporting errors
for #978
2024-05-09 13:26:24 -07:00
Ryan Barrett a690bc0115
add bsky.social, x.com to domain blocklist
for https://github.com/snarfed/bridgy-fed/discussions/1035 . thanks for the nudge @lucajet!
2024-05-09 07:08:34 -07:00
Ryan Barrett 0f35a2e1a1
ATProto firehose: convert to function we can run in a thread
for #978
2024-05-07 16:58:52 -07:00
Ryan Barrett c608ecaa36
noop: misc constants cleanup 2024-05-05 16:35:23 -07:00
Ryan Barrett 0781a4c695
AP inbox delivery: return 422 when a user with a bad handle follows the bot user
for #982
2024-05-04 13:39:39 -07:00
Ryan Barrett b6845246e9
fix protocol bot users' image URLs
re: #967
2024-04-28 08:14:55 -07:00
Ryan Barrett 3c62f7cfcc
give up on email for now, still can't get SMTP to work
Gmail SMTP is unhappy with auth, even though I got it to work on https://shell-py3.appspot.com/ , and Gandi SMTP seems to block Google Cloud IPs. sigh.
2024-04-24 15:30:20 -07:00
Ryan Barrett a79dc45b28
email: update from address 2024-04-24 15:07:42 -07:00
Ryan Barrett b543fdb1d5
switch to Gmail SMTP, other minor tweaks 2024-04-24 14:26:20 -07:00
Ryan Barrett ece168fac1
email me when someone enables or disables a protocol 2024-04-24 11:15:28 -07:00
Ryan Barrett 11eb082190
tighten common.unwrap so it doesn't remove protocol bot user URLs
...like https://bsky.brid.gy/ . this hopefully fixes following bot users in eg AP to enable protocols.
2024-04-23 12:00:39 -07:00
Ryan Barrett ed78090d2c
expand User.ap_subdomain to allow protocol subdomains like bsky
part of setting up per-protocol bot users for #880
2024-04-22 11:12:03 -07:00
Ryan Barrett f357ea1698
ActivityPub: accept non-public DMs to protocol bot users
for #880
2024-04-21 08:36:03 -07:00
Ryan Barrett 2886ae180d
remove common.ENABLED_PROTOCOLS, use Protocol.DEFAULT_ENABLED_PROTOCOLS instead
also use is_enabled_to in user page template
2024-04-18 16:39:15 -07:00
Ryan Barrett d36885728f
Protocol.receive: blocking protocol user disables that protocol
for #880
2024-04-18 16:03:51 -07:00
Ryan Barrett 39a641e000
remove USER_ALLOWLIST in favor of User.enabled_protocols 2024-04-17 17:02:17 -07:00
Ryan Barrett 259b7d72dd
start on conditional opt in
* add Protocol.DEFAULT_ENABLED_PROTOCOLS
* add User.enabled_protocols
* move common.is_enabled to Protocol.is_enabled_to, include opt out/in
2024-04-17 16:43:10 -07:00
Ryan Barrett 056644d19e
add handle_or_id kwarg to common.is_enabled
for allowlist of test users while testing a given pair of protocols
2024-04-10 11:40:17 -07:00
Ryan Barrett 95e46c5ebb
Revert "cache outbound HTTP request responses, locally to each inbound request"
This reverts commit 30debfc8fa.

seemed promising, but broke in production. Saw a lot of `IncompleteRead`s on both GETs and POSTs. Rolled back for now.

```
('Connection broken: IncompleteRead(9172 bytes read, -4586 more expected)', IncompleteRead(9172 bytes read, -4586 more expected))
...
File "oauth_dropins/webutil/util.py", line 1673, in call
  resp = getattr((session or requests), fn)(url, *args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 102, in get
  return self.request('GET', url, params=params, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 158, in request
  return super().request(method, url, *args, headers=headers, **kwargs)  # type: ignore
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests/sessions.py", line 589, in request
  resp = self.send(prep, **send_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 205, in send
  response = self._send_and_cache(request, actions, cached_response, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 233, in _send_and_cache
  self.cache.save_response(response, actions.cache_key, actions.expires)
File "requests_cache/backends/base.py", line 89, in save_response
  cached_response = CachedResponse.from_response(response, expires=expires)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/models/response.py", line 102, in from_response
  obj.raw = CachedHTTPResponse.from_response(response)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/models/raw_response.py", line 69, in from_response
  _ = response.content  # This property reads, decodes, and stores response content
      ^^^^^^^^^^^^^^^^
File "requests/models.py", line 899, in content
  self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests/models.py", line 818, in generate
  raise ChunkedEncodingError(e)
```
2024-03-11 15:31:30 -07:00
Ryan Barrett 30debfc8fa
cache outbound HTTP request responses, locally to each inbound request
using requests-cache. pretty straightforward! for #588
2024-03-08 12:34:49 -08:00
Ryan Barrett d2865fdb86
add common.ENABLED_BRIDGES, check before conversion and /bridge-user 2024-02-28 12:07:06 -08:00
Ryan Barrett 2c938cd287
switch aaronparecki.com from manually blocklisted to Web.manual_opt_out 2024-01-22 08:43:33 -08:00
Ryan Barrett 4529f30dce
switch from opt_outs.txt to new User.manual_opt_out property
opt_outs.txt wasn't included in continuous deploy from Circle, so those deploys were ignoring ids in that file :/
2024-01-14 21:02:39 -08:00
Ryan Barrett beb865bc07
web feed fetching: add next poll task
use average of time between posts in feed as ETA

for #550
2024-01-01 20:12:33 -10:00
Ryan Barrett 8dab5a8a13
make unit tests run without manual opt outs file
for aaf115347e
2023-11-29 10:39:13 -08:00
Ryan Barrett aaf115347e
add manual opt-out list 2023-11-29 10:25:45 -08:00
Ryan Barrett 6091cd4063
drop the rest of g.user!
for #690
2023-11-25 20:38:28 -08:00
Ryan Barrett f97b837960
cache follower/following counts for 2h 2023-11-08 12:03:56 -08:00
Ryan Barrett db646909e2
delivery: switch from threads to tasks, one per send
for #652
2023-10-31 12:49:27 -07:00
Ryan Barrett 87edfd69d5
noop: common: move flask_app import to local (it's for running tasks inline) 2023-10-26 16:55:01 -07:00
Ryan Barrett f74be7b0e5
add Protocol.convert_wrap 2023-10-26 13:53:31 -07:00
Ryan Barrett 422a240183
Resolve protocol-subdomain-wrapped ids/URLs by stripping the subdomain wrapping
Renames Object.replace_copies_with_originals => resolve_ids. As a side effect, also fixes https://console.cloud.google.com/errors/detail/CK3U3PONxv5Q;time=P30D?project=bridgy-federated
2023-10-24 10:47:44 -07:00
Ryan Barrett d12fd99b03
noop: move Protocol.subdomain_url to common.subdomain_wrap 2023-10-23 15:44:32 -07:00
Ryan Barrett 12375e0111
protect webmention and receive task handlers with @cloud_tasks_only 2023-10-18 11:03:21 -07:00
Ryan Barrett 1b78e86e6c
add Protocol.LOGO_HTML constant, change user page to link to external user
for #442
2023-10-11 12:25:38 -07:00
Ryan Barrett 62302b9105
more docstring formatting tweaks 2023-10-06 08:22:50 -07:00
Ryan Barrett db29ad7757
docs: fix docstring formatting, other tweaks 2023-10-05 23:32:31 -07:00
Ryan Barrett e89730e68f
drop /_ah/ path prefix on task handlers
...since it's not supported in dispatch.yaml: https://cloud.google.com/appengine/docs/flexible/reference/dispatch-yaml?tab=python#url
2023-09-29 13:49:17 -07:00
Ryan Barrett a823dd1d65
use protocol subdomains in AP inbox
...and other misc protocol subdomain fixes
2023-09-27 13:55:16 -07:00
Ryan Barrett 0b592ace35
update common.redirect_[un]wrap to handle all domains
including protocol subdomains
2023-09-27 13:10:28 -07:00
Ryan Barrett c6a5dd42b6
expand DOMAIN_RE to allow more than two segments 2023-09-22 11:42:08 -07:00
Ryan Barrett f7792f81c8
tasks: add user param to receive, make create_task run inline in local server 2023-09-19 11:15:49 -07:00
Ryan Barrett 55f96c428e
common.create_task: run task inline if we're in a local server 2023-09-18 19:02:15 -07:00
Ryan Barrett afda648a4a
tests: extract out assert_task() method 2023-09-13 14:36:24 -07:00
Ryan Barrett efcd2d89b4
common.create_task: only log in local server, don't actually create task 2023-09-09 07:51:54 -07:00
Ryan Barrett 8b242e65ba
move common.is_blacklisted to Protocol so that subclasses can override it 2023-09-06 17:35:15 -07:00
Ryan Barrett f91486db75
add new atproto-commit task queue, enqueue tasks for new commits
hub will consume these and emit them to subscribeRepos subscribers
2023-09-05 20:10:11 -07:00
Ryan Barrett e18dabf510
implement ATProto.target_for, .fetch for at:// URIs 2023-08-31 10:48:28 -07:00
Ryan Barrett e194ccb3ad
Protocol.load/fetch returns True/False (or None) to mean "not this protocol"
ie, nothing failed, but the given id eitehr doesn't below to the given protocol, or fetched successfully but its data belongs to a different protocol.

vs raising exceptions for fetches that fail or error, eg network connection problems, Web got HTML with microformats2 but no h-entry, ActivityPub got non-AS2, etc.

used in Protocol.for_id: when a given Protocol's fetch doesn't work, we move on and try the next protocol _if_ it returned False (ie wasn't fetchable with that protocol, but we fail fast if it raises an exception (ie fetch failed).
2023-07-14 12:58:47 -07:00
Ryan Barrett b3a3de73f2
extend Web/ActivityPub.owns_id() to return False if blocklisted 2023-07-11 13:25:29 -07:00
Ryan Barrett 0caca9243e
add common.add to add to repeated properties only if not already there
tried a bunch of other more sophisticated ways to do this in snarfed/webutil@280a93e092 (plus snarfed/webutil@6cb9c1f719), tried a custom repeated property class that acted like a set instead of a list, but couldn't get it to work reliably or compatibly enough.
2023-06-29 22:15:07 -07:00
Ryan Barrett ab1c28ee4d
noop, lint fixes from flake8
remaining:

$ flake8 --extend-ignore=E501 *.py tests/*.py
"pyflakes" failed during execution due to "'FlakesChecker' object has no attribute 'NAMEDEXPR'"
Run flake8 with greater verbosity to see more details
activitypub.py:15:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
activitypub.py:36:1: F401 'web' imported but unused
activitypub.py:48:1: E302 expected 2 blank lines, found 1
activitypub.py:51:9: F811 redefinition of unused 'web' from line 36
app.py:6:1: F401 'flask_app.app' imported but unused
app.py:9:1: F401 'activitypub' imported but unused
app.py:9:1: F401 'convert' imported but unused
app.py:9:1: F401 'follow' imported but unused
app.py:9:1: F401 'pages' imported but unused
app.py:9:1: F401 'redirect' imported but unused
app.py:9:1: F401 'superfeedr' imported but unused
app.py:9:1: F401 'ui' imported but unused
app.py:9:1: F401 'webfinger' imported but unused
app.py:9:1: F401 'web' imported but unused
app.py:9:1: F401 'xrpc_actor' imported but unused
app.py:9:1: F401 'xrpc_feed' imported but unused
app.py:9:1: F401 'xrpc_graph' imported but unused
app.py:9:19: E401 multiple imports on one line
models.py:19:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
models.py:364:31: E114 indentation is not a multiple of four (comment)
models.py:364:31: E116 unexpected indentation (comment)
protocol.py:17:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
redirect.py:26:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
web.py:18:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
webfinger.py:13:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
webfinger.py:110:13: E122 continuation line missing indentation or outdented
webfinger.py:111:13: E122 continuation line missing indentation or outdented
webfinger.py:131:13: E122 continuation line missing indentation or outdented
webfinger.py:132:13: E122 continuation line missing indentation or outdented
webfinger.py:133:13: E122 continuation line missing indentation or outdented
webfinger.py:134:13: E122 continuation line missing indentation or outdented
tests/__init__.py:2:1: F401 'oauth_dropins.webutil.tests' imported but unused
tests/test_follow.py:11:1: F401 'oauth_dropins.webutil.util.json_dumps' imported but unused
tests/test_follow.py:14:1: F401 '.testutil.Fake' imported but unused
tests/test_models.py:156:15: E122 continuation line missing indentation or outdented
tests/test_models.py:157:15: E122 continuation line missing indentation or outdented
tests/test_models.py:158:11: E122 continuation line missing indentation or outdented
tests/test_web.py:12:1: F401 'oauth_dropins.webutil.util.json_dumps' imported but unused
tests/test_web.py:17:1: F401 '.testutil' imported but unused
tests/test_web.py:1513:13: E128 continuation line under-indented for visual indent
tests/test_web.py:1514:9: E124 closing bracket does not match visual indentation
tests/testutil.py:106:1: E402 module level import not at top of file
tests/testutil.py:107:1: E402 module level import not at top of file
tests/testutil.py:108:1: E402 module level import not at top of file
tests/testutil.py:109:1: E402 module level import not at top of file
tests/testutil.py:110:1: E402 module level import not at top of file
tests/testutil.py:301:24: E203 whitespace before ':'
tests/testutil.py:301:25: E701 multiple statements on one line (colon)
tests/testutil.py:301:25: E231 missing whitespace after ':'
2023-06-20 11:29:31 -07:00
Ryan Barrett 8e14ce003f
Web: relax domain validation, allow underscores in all segments except TLD
we already have one Web user with an underscore, brandon_swoop.ngrok.io
2023-06-16 12:05:55 -07:00
Ryan Barrett d3c4415cd3
migrate User.actor_as2 to obj key pointing to Object
for #547
2023-06-15 21:23:31 -07:00
Ryan Barrett 0902994af0
webfinger: infer protocol from brid.gy subdomain in request or resource
first commit for inferring protocol. exciting! for #512
2023-06-12 15:50:00 -07:00
Ryan Barrett 510cd036bf
AP users: add Protocol.for_domain and for_request
getting ready to serve per-protocol subdomains, eg nostr.brid.gy, ap.brid.gy, etc

for #512
2023-06-10 19:55:08 -07:00
Ryan Barrett 4909046506
add new brid.gy subdomains and todo to canonicalize them 2023-06-10 16:29:58 -07:00
Ryan Barrett 6796b47a9f
noop: move web.NoMicroformats to common, misc other minor cleanup 2023-06-10 07:53:07 -07:00
Ryan Barrett 7f6cc61683
improve domain validation for Web key ids, normalize to lower case 2023-06-09 10:58:28 -07:00
Ryan Barrett 958f81ddd1
AP users: rename [is_]homepage => [is_]web_url, move into Protocol subclasses
for #512
2023-05-31 18:47:40 -07:00
Ryan Barrett b7bebb4075
move base64_to_long, long_to_base64 to common 2023-05-27 14:58:22 -07:00
Ryan Barrett c8b2103e13
rename Webmention class => Web, webmention.py => web.py 2023-05-26 17:40:29 -07:00
Ryan Barrett 0dbe18aae4
remove common.actor() 2023-04-04 22:19:35 -07:00
Ryan Barrett 4c9744cd65
common.error bug fix, accept exc_info kwarg 2023-04-04 07:14:31 -07:00
Ryan Barrett 88dd51b05b
continue abstracting logic in Protocol class
lots more to do.

also convert Object.as1 to AS2 in ActivityPub.send if necessary.
2023-03-20 19:17:55 -07:00
Ryan Barrett fb5f7b3fb0
move current user into Flask g request-global 2023-03-20 14:28:14 -07:00
Ryan Barrett 5e3c623552
add User.actor_id() 2023-03-19 09:34:42 -07:00
Ryan Barrett d8932d4e76
add mf2 to user page dashboard to make a notifications feed
for #442
2023-03-14 11:27:10 -07:00
Ryan Barrett 9cc8451182
fix webmention endpoint discovery cache key
#423
2023-03-11 12:14:48 -08:00
Ryan Barrett 76e26a7c2c
cache webmention endpoint discovery for 2h 2023-03-10 22:24:58 -08:00
Ryan Barrett ffd8810b44
start to separate logic from protocols with new Protocol/ActivityPub classes
for #388
2023-03-10 07:36:50 -08:00
Ryan Barrett 85b8d86b17
common.get_as2: handle empty and non/bad JSON responses
fixes https://console.cloud.google.com/errors/detail/CKH_9oGur7T1ag;time=P30D?project=bridgy-federated
2023-02-28 16:21:33 -08:00
Ryan Barrett fd27dabe61
switch JSON properties to custom JSONProperty that works in web console UI
https://github.com/googleapis/python-ndb/issues/874#issuecomment-1442753255
2023-02-24 07:25:29 -06:00
Ryan Barrett 91a60c7e67
switch Object.as1 to be a ComputedProperty 2023-02-23 21:17:26 -06:00
Ryan Barrett 1f3bd41dc5
common.redirect_unwrap: stop following redirects when unwrapping actor URLs 2023-02-23 08:35:06 -06:00
Ryan Barrett 74b3b3b689
refactoring, move Object creation out of common.send_webmentions
ugh this was painful
2023-02-23 07:44:54 -06:00
Ryan Barrett 3d3335e860
AP => webmention repost bug fix, merge follower and wm domains 2023-02-23 07:39:26 -06:00
Ryan Barrett 65bbc6751e
HTTP Signature verification: bug fix, use actual HTTP method 2023-02-16 07:34:12 -08:00
Ryan Barrett 48a7720f88
common.get_object: normalize ids by removing fragments
this is still underspecified and a bit inconsistent across fediverse implementations:

https://socialhub.activitypub.rocks/t/problems-posting-to-mastodon-inbox/801/11

> The reason might also be that your IDs aren’t permanent, as in, they contain a #fragment. Posts and their corresponding Create activities are supposed to be resolvable — which means one should be able to send a GET request to the ID URL and get the object back. This can’t be done with an URL that contains a fragment as the fragment is not a part of the HTTP exchange, it’s processed on the client.

https://socialhub.activitypub.rocks/t/problems-posting-to-mastodon-inbox/801/23

> I ran into this object id #fragment problem as well. It seems because of some URL normalization, Mastodon will remove the fragment, and drop any additional posts with different fragments (because they become the same url).

https://socialhub.activitypub.rocks/t/s2s-create-activity/1647/5
https://github.com/mastodon/mastodon/issues/13879 (open!)
https://github.com/w3c/activitypub/issues/224

nothing in the http sig spec, example key ids aren't even URLs there:
https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-message-signatures-16
2023-02-15 20:00:12 -08:00
Ryan Barrett 63d0e59c2c
AP: verify incoming signatures!
for #315
2023-02-15 14:10:25 -08:00
Ryan Barrett e423bde6d0
unify logging for writing datastore entities into post put hook
and other logging tweaks
2023-02-14 15:38:42 -08:00
Ryan Barrett 47b83c4e51
common.get_object: update existing object if available 2023-02-14 15:26:25 -08:00
Ryan Barrett f44aa3b39e
common.get_object: fall back to HTTP fetch if stored entity has no as2
also add more logging
2023-02-14 15:17:03 -08:00
Ryan Barrett 588598c5ff
switch most uses of common.get_as2 to common.get_object 2023-02-14 14:56:27 -08:00
Ryan Barrett c2e6174330
add common.get_object() 2023-02-14 14:30:00 -08:00