Wykres commitów

319 Commity (b045262bb5399bb76e7c57a8391b678d99bd05a1)

Autor SHA1 Wiadomość Data
Ryan Barrett 88cbe3b7b4
cache models.get_originals in memcache with new memcache_memoize decorator
for #1149
2024-07-30 14:50:33 -07:00
Ryan Barrett 33e0d0b14a
common.memcache_key: encode Unicode chars as UTF-8 2024-07-30 14:49:50 -07:00
Ryan Barrett 075c33cade
memcache: encode spaces in keys as %20
fixes https://console.cloud.google.com/errors/detail/CNPzqYzWg4_TyQE;time=P7D?project=bridgy-federated
2024-07-16 13:48:52 -07:00
Ryan Barrett 7b0151b0cf
memcache: truncate keys at 250 chars
max memcache key len:
https://github.com/memcached/memcached/wiki/Commands#standard-protocol
https://pymemcache.readthedocs.io/en/latest/apidoc/pymemcache.client.base.html

fixes https://console.cloud.google.com/errors/detail/CNPzqYzWg4_TyQE;time=P30D?project=bridgy-federated
2024-07-12 07:50:54 -07:00
Ryan Barrett 98d21ae0d9
common.report_error/exception bug fix, handle running outside request context 2024-07-05 12:15:06 -07:00
Ryan Barrett 866f6593e3
add a few domains to the blocklist 2024-06-27 13:20:07 -07:00
Ryan Barrett 550e1030c4
drop unused TLD_BLOCKLIST
subsumed by using brevity.TLDs in 84949f265b
2024-06-23 09:22:50 -07:00
Ryan Barrett fc0cc1118a
activitypub.inbox: short circuit on activity or actor id on opted out domain 2024-06-23 09:20:22 -07:00
Ryan Barrett b20bbf0564
memcache config: try to only use prod Memorystore on prod GAE 2024-06-19 11:01:46 -07:00
Ryan Barrett 4e489a4ee0
activitypub.inbox: store seen activity ids in memcache, short circuit on seen ids earlier 2024-06-13 13:54:37 -07:00
Ryan Barrett 3c06739889
common.report_exception bug fix 2024-06-13 12:02:54 -07:00
Ryan Barrett ba00850840
turn on memorystore ndb global cache in hub and atproto_firehose 2024-06-04 19:27:52 -07:00
Ryan Barrett 3ccce1bc89
start using memorystore memcached for ndb global cache 2024-06-04 15:40:32 -07:00
Ryan Barrett 07517f1685
ndb global cache: limit to users and profile objects, not other objects/activities 2024-06-04 14:27:04 -07:00
Ryan Barrett 83893d0ea7
HTTP caching headers: switch to flask_util.headers
for #432, fixes https://console.cloud.google.com/errors/detail/CMK10__x4ZbgbQ;time=P7D;refresh=true?project=bridgy-federated
2024-06-04 14:19:39 -07:00
Ryan Barrett 712da61181
auth error reporting: standardize messages, include HTTP context
trying to collect them into two top-level Error Reporting errors, with details for each one in the user field

for #566
2024-06-04 11:27:17 -07:00
Ryan Barrett e8e2c9692d
common.report_error bug fix
fixes https://console.cloud.google.com/errors/detail/COyxtrLf5r2XowE;time=P30D?project=bridgy-federated
2024-06-03 16:20:53 -07:00
Ryan Barrett cc564a2b19
turn on authorization checks! HTTP sig matches actor, actor is allowed to modify object
for #566
2024-06-03 14:12:08 -07:00
Ryan Barrett 9c4888b02e
switch from flask_util.cached to HTTP caching, suppress some verbose logging 2024-05-30 14:55:35 -07:00
Ryan Barrett 6ffc4a3c3b
infra: add new router flex service, switch tasks to run on it
receive, send, webmention, and poll-feed
2024-05-23 20:40:54 -07:00
Ryan Barrett 348987f0b4
add Protocol.REQUIRES_OLD_ACCOUNT, enable for ActivityPub 2024-05-11 19:20:33 -07:00
Ryan Barrett f2b5f79489
ATProto firehose: add handle wrapper that catches exceptions
also refactor and simplify load_dids, error reporting. #978
2024-05-10 19:59:12 -07:00
Ryan Barrett b49fe13e59
ATProto firehose: use report_exception() so we get stack traces 2024-05-09 21:31:41 -07:00
Ryan Barrett 940c0e8cae
ATProto firehose: don't assume required fields, handle error reporting errors
for #978
2024-05-09 13:26:24 -07:00
Ryan Barrett a690bc0115
add bsky.social, x.com to domain blocklist
for https://github.com/snarfed/bridgy-fed/discussions/1035 . thanks for the nudge @lucajet!
2024-05-09 07:08:34 -07:00
Ryan Barrett 0f35a2e1a1
ATProto firehose: convert to function we can run in a thread
for #978
2024-05-07 16:58:52 -07:00
Ryan Barrett c608ecaa36
noop: misc constants cleanup 2024-05-05 16:35:23 -07:00
Ryan Barrett 0781a4c695
AP inbox delivery: return 422 when a user with a bad handle follows the bot user
for #982
2024-05-04 13:39:39 -07:00
Ryan Barrett b6845246e9
fix protocol bot users' image URLs
re: #967
2024-04-28 08:14:55 -07:00
Ryan Barrett 3c62f7cfcc
give up on email for now, still can't get SMTP to work
Gmail SMTP is unhappy with auth, even though I got it to work on https://shell-py3.appspot.com/ , and Gandi SMTP seems to block Google Cloud IPs. sigh.
2024-04-24 15:30:20 -07:00
Ryan Barrett a79dc45b28
email: update from address 2024-04-24 15:07:42 -07:00
Ryan Barrett b543fdb1d5
switch to Gmail SMTP, other minor tweaks 2024-04-24 14:26:20 -07:00
Ryan Barrett ece168fac1
email me when someone enables or disables a protocol 2024-04-24 11:15:28 -07:00
Ryan Barrett 11eb082190
tighten common.unwrap so it doesn't remove protocol bot user URLs
...like https://bsky.brid.gy/ . this hopefully fixes following bot users in eg AP to enable protocols.
2024-04-23 12:00:39 -07:00
Ryan Barrett ed78090d2c
expand User.ap_subdomain to allow protocol subdomains like bsky
part of setting up per-protocol bot users for #880
2024-04-22 11:12:03 -07:00
Ryan Barrett f357ea1698
ActivityPub: accept non-public DMs to protocol bot users
for #880
2024-04-21 08:36:03 -07:00
Ryan Barrett 2886ae180d
remove common.ENABLED_PROTOCOLS, use Protocol.DEFAULT_ENABLED_PROTOCOLS instead
also use is_enabled_to in user page template
2024-04-18 16:39:15 -07:00
Ryan Barrett d36885728f
Protocol.receive: blocking protocol user disables that protocol
for #880
2024-04-18 16:03:51 -07:00
Ryan Barrett 39a641e000
remove USER_ALLOWLIST in favor of User.enabled_protocols 2024-04-17 17:02:17 -07:00
Ryan Barrett 259b7d72dd
start on conditional opt in
* add Protocol.DEFAULT_ENABLED_PROTOCOLS
* add User.enabled_protocols
* move common.is_enabled to Protocol.is_enabled_to, include opt out/in
2024-04-17 16:43:10 -07:00
Ryan Barrett 056644d19e
add handle_or_id kwarg to common.is_enabled
for allowlist of test users while testing a given pair of protocols
2024-04-10 11:40:17 -07:00
Ryan Barrett 95e46c5ebb
Revert "cache outbound HTTP request responses, locally to each inbound request"
This reverts commit 30debfc8fa.

seemed promising, but broke in production. Saw a lot of `IncompleteRead`s on both GETs and POSTs. Rolled back for now.

```
('Connection broken: IncompleteRead(9172 bytes read, -4586 more expected)', IncompleteRead(9172 bytes read, -4586 more expected))
...
File "oauth_dropins/webutil/util.py", line 1673, in call
  resp = getattr((session or requests), fn)(url, *args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 102, in get
  return self.request('GET', url, params=params, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 158, in request
  return super().request(method, url, *args, headers=headers, **kwargs)  # type: ignore
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests/sessions.py", line 589, in request
  resp = self.send(prep, **send_kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 205, in send
  response = self._send_and_cache(request, actions, cached_response, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 233, in _send_and_cache
  self.cache.save_response(response, actions.cache_key, actions.expires)
File "requests_cache/backends/base.py", line 89, in save_response
  cached_response = CachedResponse.from_response(response, expires=expires)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/models/response.py", line 102, in from_response
  obj.raw = CachedHTTPResponse.from_response(response)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/models/raw_response.py", line 69, in from_response
  _ = response.content  # This property reads, decodes, and stores response content
      ^^^^^^^^^^^^^^^^
File "requests/models.py", line 899, in content
  self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests/models.py", line 818, in generate
  raise ChunkedEncodingError(e)
```
2024-03-11 15:31:30 -07:00
Ryan Barrett 30debfc8fa
cache outbound HTTP request responses, locally to each inbound request
using requests-cache. pretty straightforward! for #588
2024-03-08 12:34:49 -08:00
Ryan Barrett d2865fdb86
add common.ENABLED_BRIDGES, check before conversion and /bridge-user 2024-02-28 12:07:06 -08:00
Ryan Barrett 2c938cd287
switch aaronparecki.com from manually blocklisted to Web.manual_opt_out 2024-01-22 08:43:33 -08:00
Ryan Barrett 4529f30dce
switch from opt_outs.txt to new User.manual_opt_out property
opt_outs.txt wasn't included in continuous deploy from Circle, so those deploys were ignoring ids in that file :/
2024-01-14 21:02:39 -08:00
Ryan Barrett beb865bc07
web feed fetching: add next poll task
use average of time between posts in feed as ETA

for #550
2024-01-01 20:12:33 -10:00
Ryan Barrett 8dab5a8a13
make unit tests run without manual opt outs file
for aaf115347e
2023-11-29 10:39:13 -08:00
Ryan Barrett aaf115347e
add manual opt-out list 2023-11-29 10:25:45 -08:00
Ryan Barrett 6091cd4063
drop the rest of g.user!
for #690
2023-11-25 20:38:28 -08:00
Ryan Barrett f97b837960
cache follower/following counts for 2h 2023-11-08 12:03:56 -08:00
Ryan Barrett db646909e2
delivery: switch from threads to tasks, one per send
for #652
2023-10-31 12:49:27 -07:00
Ryan Barrett 87edfd69d5
noop: common: move flask_app import to local (it's for running tasks inline) 2023-10-26 16:55:01 -07:00
Ryan Barrett f74be7b0e5
add Protocol.convert_wrap 2023-10-26 13:53:31 -07:00
Ryan Barrett 422a240183
Resolve protocol-subdomain-wrapped ids/URLs by stripping the subdomain wrapping
Renames Object.replace_copies_with_originals => resolve_ids. As a side effect, also fixes https://console.cloud.google.com/errors/detail/CK3U3PONxv5Q;time=P30D?project=bridgy-federated
2023-10-24 10:47:44 -07:00
Ryan Barrett d12fd99b03
noop: move Protocol.subdomain_url to common.subdomain_wrap 2023-10-23 15:44:32 -07:00
Ryan Barrett 12375e0111
protect webmention and receive task handlers with @cloud_tasks_only 2023-10-18 11:03:21 -07:00
Ryan Barrett 1b78e86e6c
add Protocol.LOGO_HTML constant, change user page to link to external user
for #442
2023-10-11 12:25:38 -07:00
Ryan Barrett 62302b9105
more docstring formatting tweaks 2023-10-06 08:22:50 -07:00
Ryan Barrett db29ad7757
docs: fix docstring formatting, other tweaks 2023-10-05 23:32:31 -07:00
Ryan Barrett e89730e68f
drop /_ah/ path prefix on task handlers
...since it's not supported in dispatch.yaml: https://cloud.google.com/appengine/docs/flexible/reference/dispatch-yaml?tab=python#url
2023-09-29 13:49:17 -07:00
Ryan Barrett a823dd1d65
use protocol subdomains in AP inbox
...and other misc protocol subdomain fixes
2023-09-27 13:55:16 -07:00
Ryan Barrett 0b592ace35
update common.redirect_[un]wrap to handle all domains
including protocol subdomains
2023-09-27 13:10:28 -07:00
Ryan Barrett c6a5dd42b6
expand DOMAIN_RE to allow more than two segments 2023-09-22 11:42:08 -07:00
Ryan Barrett f7792f81c8
tasks: add user param to receive, make create_task run inline in local server 2023-09-19 11:15:49 -07:00
Ryan Barrett 55f96c428e
common.create_task: run task inline if we're in a local server 2023-09-18 19:02:15 -07:00
Ryan Barrett afda648a4a
tests: extract out assert_task() method 2023-09-13 14:36:24 -07:00
Ryan Barrett efcd2d89b4
common.create_task: only log in local server, don't actually create task 2023-09-09 07:51:54 -07:00
Ryan Barrett 8b242e65ba
move common.is_blacklisted to Protocol so that subclasses can override it 2023-09-06 17:35:15 -07:00
Ryan Barrett f91486db75
add new atproto-commit task queue, enqueue tasks for new commits
hub will consume these and emit them to subscribeRepos subscribers
2023-09-05 20:10:11 -07:00
Ryan Barrett e18dabf510
implement ATProto.target_for, .fetch for at:// URIs 2023-08-31 10:48:28 -07:00
Ryan Barrett e194ccb3ad
Protocol.load/fetch returns True/False (or None) to mean "not this protocol"
ie, nothing failed, but the given id eitehr doesn't below to the given protocol, or fetched successfully but its data belongs to a different protocol.

vs raising exceptions for fetches that fail or error, eg network connection problems, Web got HTML with microformats2 but no h-entry, ActivityPub got non-AS2, etc.

used in Protocol.for_id: when a given Protocol's fetch doesn't work, we move on and try the next protocol _if_ it returned False (ie wasn't fetchable with that protocol, but we fail fast if it raises an exception (ie fetch failed).
2023-07-14 12:58:47 -07:00
Ryan Barrett b3a3de73f2
extend Web/ActivityPub.owns_id() to return False if blocklisted 2023-07-11 13:25:29 -07:00
Ryan Barrett 0caca9243e
add common.add to add to repeated properties only if not already there
tried a bunch of other more sophisticated ways to do this in snarfed/webutil@280a93e092 (plus snarfed/webutil@6cb9c1f719), tried a custom repeated property class that acted like a set instead of a list, but couldn't get it to work reliably or compatibly enough.
2023-06-29 22:15:07 -07:00
Ryan Barrett ab1c28ee4d
noop, lint fixes from flake8
remaining:

$ flake8 --extend-ignore=E501 *.py tests/*.py
"pyflakes" failed during execution due to "'FlakesChecker' object has no attribute 'NAMEDEXPR'"
Run flake8 with greater verbosity to see more details
activitypub.py:15:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
activitypub.py:36:1: F401 'web' imported but unused
activitypub.py:48:1: E302 expected 2 blank lines, found 1
activitypub.py:51:9: F811 redefinition of unused 'web' from line 36
app.py:6:1: F401 'flask_app.app' imported but unused
app.py:9:1: F401 'activitypub' imported but unused
app.py:9:1: F401 'convert' imported but unused
app.py:9:1: F401 'follow' imported but unused
app.py:9:1: F401 'pages' imported but unused
app.py:9:1: F401 'redirect' imported but unused
app.py:9:1: F401 'superfeedr' imported but unused
app.py:9:1: F401 'ui' imported but unused
app.py:9:1: F401 'webfinger' imported but unused
app.py:9:1: F401 'web' imported but unused
app.py:9:1: F401 'xrpc_actor' imported but unused
app.py:9:1: F401 'xrpc_feed' imported but unused
app.py:9:1: F401 'xrpc_graph' imported but unused
app.py:9:19: E401 multiple imports on one line
models.py:19:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
models.py:364:31: E114 indentation is not a multiple of four (comment)
models.py:364:31: E116 unexpected indentation (comment)
protocol.py:17:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
redirect.py:26:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
web.py:18:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
webfinger.py:13:1: F401 'oauth_dropins.webutil.util.json_loads' imported but unused
webfinger.py:110:13: E122 continuation line missing indentation or outdented
webfinger.py:111:13: E122 continuation line missing indentation or outdented
webfinger.py:131:13: E122 continuation line missing indentation or outdented
webfinger.py:132:13: E122 continuation line missing indentation or outdented
webfinger.py:133:13: E122 continuation line missing indentation or outdented
webfinger.py:134:13: E122 continuation line missing indentation or outdented
tests/__init__.py:2:1: F401 'oauth_dropins.webutil.tests' imported but unused
tests/test_follow.py:11:1: F401 'oauth_dropins.webutil.util.json_dumps' imported but unused
tests/test_follow.py:14:1: F401 '.testutil.Fake' imported but unused
tests/test_models.py:156:15: E122 continuation line missing indentation or outdented
tests/test_models.py:157:15: E122 continuation line missing indentation or outdented
tests/test_models.py:158:11: E122 continuation line missing indentation or outdented
tests/test_web.py:12:1: F401 'oauth_dropins.webutil.util.json_dumps' imported but unused
tests/test_web.py:17:1: F401 '.testutil' imported but unused
tests/test_web.py:1513:13: E128 continuation line under-indented for visual indent
tests/test_web.py:1514:9: E124 closing bracket does not match visual indentation
tests/testutil.py:106:1: E402 module level import not at top of file
tests/testutil.py:107:1: E402 module level import not at top of file
tests/testutil.py:108:1: E402 module level import not at top of file
tests/testutil.py:109:1: E402 module level import not at top of file
tests/testutil.py:110:1: E402 module level import not at top of file
tests/testutil.py:301:24: E203 whitespace before ':'
tests/testutil.py:301:25: E701 multiple statements on one line (colon)
tests/testutil.py:301:25: E231 missing whitespace after ':'
2023-06-20 11:29:31 -07:00
Ryan Barrett 8e14ce003f
Web: relax domain validation, allow underscores in all segments except TLD
we already have one Web user with an underscore, brandon_swoop.ngrok.io
2023-06-16 12:05:55 -07:00
Ryan Barrett d3c4415cd3
migrate User.actor_as2 to obj key pointing to Object
for #547
2023-06-15 21:23:31 -07:00
Ryan Barrett 0902994af0
webfinger: infer protocol from brid.gy subdomain in request or resource
first commit for inferring protocol. exciting! for #512
2023-06-12 15:50:00 -07:00
Ryan Barrett 510cd036bf
AP users: add Protocol.for_domain and for_request
getting ready to serve per-protocol subdomains, eg nostr.brid.gy, ap.brid.gy, etc

for #512
2023-06-10 19:55:08 -07:00
Ryan Barrett 4909046506
add new brid.gy subdomains and todo to canonicalize them 2023-06-10 16:29:58 -07:00
Ryan Barrett 6796b47a9f
noop: move web.NoMicroformats to common, misc other minor cleanup 2023-06-10 07:53:07 -07:00
Ryan Barrett 7f6cc61683
improve domain validation for Web key ids, normalize to lower case 2023-06-09 10:58:28 -07:00
Ryan Barrett 958f81ddd1
AP users: rename [is_]homepage => [is_]web_url, move into Protocol subclasses
for #512
2023-05-31 18:47:40 -07:00
Ryan Barrett b7bebb4075
move base64_to_long, long_to_base64 to common 2023-05-27 14:58:22 -07:00
Ryan Barrett c8b2103e13
rename Webmention class => Web, webmention.py => web.py 2023-05-26 17:40:29 -07:00
Ryan Barrett 0dbe18aae4
remove common.actor() 2023-04-04 22:19:35 -07:00
Ryan Barrett 4c9744cd65
common.error bug fix, accept exc_info kwarg 2023-04-04 07:14:31 -07:00
Ryan Barrett 88dd51b05b
continue abstracting logic in Protocol class
lots more to do.

also convert Object.as1 to AS2 in ActivityPub.send if necessary.
2023-03-20 19:17:55 -07:00
Ryan Barrett fb5f7b3fb0
move current user into Flask g request-global 2023-03-20 14:28:14 -07:00
Ryan Barrett 5e3c623552
add User.actor_id() 2023-03-19 09:34:42 -07:00
Ryan Barrett d8932d4e76
add mf2 to user page dashboard to make a notifications feed
for #442
2023-03-14 11:27:10 -07:00
Ryan Barrett 9cc8451182
fix webmention endpoint discovery cache key
#423
2023-03-11 12:14:48 -08:00
Ryan Barrett 76e26a7c2c
cache webmention endpoint discovery for 2h 2023-03-10 22:24:58 -08:00
Ryan Barrett ffd8810b44
start to separate logic from protocols with new Protocol/ActivityPub classes
for #388
2023-03-10 07:36:50 -08:00
Ryan Barrett 85b8d86b17
common.get_as2: handle empty and non/bad JSON responses
fixes https://console.cloud.google.com/errors/detail/CKH_9oGur7T1ag;time=P30D?project=bridgy-federated
2023-02-28 16:21:33 -08:00
Ryan Barrett fd27dabe61
switch JSON properties to custom JSONProperty that works in web console UI
https://github.com/googleapis/python-ndb/issues/874#issuecomment-1442753255
2023-02-24 07:25:29 -06:00
Ryan Barrett 91a60c7e67
switch Object.as1 to be a ComputedProperty 2023-02-23 21:17:26 -06:00
Ryan Barrett 1f3bd41dc5
common.redirect_unwrap: stop following redirects when unwrapping actor URLs 2023-02-23 08:35:06 -06:00
Ryan Barrett 74b3b3b689
refactoring, move Object creation out of common.send_webmentions
ugh this was painful
2023-02-23 07:44:54 -06:00
Ryan Barrett 3d3335e860
AP => webmention repost bug fix, merge follower and wm domains 2023-02-23 07:39:26 -06:00