Ryan Barrett
11eb082190
tighten common.unwrap so it doesn't remove protocol bot user URLs
...
...like https://bsky.brid.gy/ . this hopefully fixes following bot users in eg AP to enable protocols.
2024-04-23 12:00:39 -07:00
Ryan Barrett
d4a56127d9
incoming webmentions: stop converting home pages to update activities
...
pass Protocol.receive the bare actor object, let it convert to update activity, like with posts etc
2024-04-23 06:50:55 -07:00
Ryan Barrett
34692abc60
handle protocol bot users in webfinger, ids.translate_handle, Web.owns_handle
...
for #880
2024-04-22 14:01:09 -07:00
Ryan Barrett
b9551c4de7
adjust Web.owns_id to say it owns protocol bot user domains but not their pages
...
eg bsky.brid.gy True, https://bsky.brid.gy/ True, https://bsky.brid.gy/foo False.
also move our internal synthetic UI-initiated follow ids from https://fed.brid.gy/web/ ... to under the user's own domain. hopefully this won't break anything 🤞
2024-04-22 11:58:01 -07:00
Ryan Barrett
ed78090d2c
expand User.ap_subdomain to allow protocol subdomains like bsky
...
part of setting up per-protocol bot users for #880
2024-04-22 11:12:03 -07:00
Ryan Barrett
e1f9021696
AP instance actor: move AS2 JSON files, start adding per-protocol bot users
2024-04-21 16:35:17 -07:00
Ryan Barrett
393605bde9
change ATProto.ABBREV to bsky
...
🤞 , for #961
2024-04-17 06:54:16 -07:00
Ryan Barrett
5767ffabb5
add new Object.normalize_ids method, use in Protocol.receive
...
eg https://bsky.app/ ... URLs to DIDs for actors or at:// URIs for objects
this hopefully fixes web => ATProto follows of https://bsky.app/ ... profile URLs.
2024-04-09 13:07:57 -07:00
Ryan Barrett
7e7556bb02
Web.load: support domain ids, convert to homepage URLs
2024-03-12 21:09:34 -07:00
Ryan Barrett
95e46c5ebb
Revert "cache outbound HTTP request responses, locally to each inbound request"
...
This reverts commit 30debfc8fa
.
seemed promising, but broke in production. Saw a lot of `IncompleteRead`s on both GETs and POSTs. Rolled back for now.
```
('Connection broken: IncompleteRead(9172 bytes read, -4586 more expected)', IncompleteRead(9172 bytes read, -4586 more expected))
...
File "oauth_dropins/webutil/util.py", line 1673, in call
resp = getattr((session or requests), fn)(url, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 102, in get
return self.request('GET', url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 158, in request
return super().request(method, url, *args, headers=headers, **kwargs) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 205, in send
response = self._send_and_cache(request, actions, cached_response, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/session.py", line 233, in _send_and_cache
self.cache.save_response(response, actions.cache_key, actions.expires)
File "requests_cache/backends/base.py", line 89, in save_response
cached_response = CachedResponse.from_response(response, expires=expires)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/models/response.py", line 102, in from_response
obj.raw = CachedHTTPResponse.from_response(response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests_cache/models/raw_response.py", line 69, in from_response
_ = response.content # This property reads, decodes, and stores response content
^^^^^^^^^^^^^^^^
File "requests/models.py", line 899, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
```
2024-03-11 15:31:30 -07:00
Ryan Barrett
30debfc8fa
cache outbound HTTP request responses, locally to each inbound request
...
using requests-cache. pretty straightforward! for #588
2024-03-08 12:34:49 -08:00
Ryan Barrett
01e5ac51ab
fix crash in Web.owns_id
...
fixes https://console.cloud.google.com/errors/detail/CPOsl7usgr3VUQ;time=P30D?project=bridgy-federated
2024-03-06 14:04:44 -08:00
Ryan Barrett
9489204d64
AP: add profile to Content-Type: application/ld+json
...
for #895
2024-02-27 11:38:00 -08:00
Ryan Barrett
fce24c0645
/web-site: don't allow URLs with paths
...
for #882
2024-02-27 11:19:14 -08:00
Ryan Barrett
42b4541c8d
Web.get_or_create: bug fix for normalizing domain case
...
fixes https://console.cloud.google.com/errors/detail/CKOs59zx0rSXQg;time=P30D?project=bridgy-federated
2024-02-26 19:26:47 -08:00
Ryan Barrett
ac4b7fb4e5
web.poll_feed_task: handle more HTTP and parse errors
...
fixes https://console.cloud.google.com/errors/detail/CNj81tKbzNSzwAE;time=P30D?project=bridgy-federated
2024-02-20 12:23:20 -08:00
Ryan Barrett
6a63820cd8
tighten Object.owns_id to reject blocklisted domains
...
fixes https://console.cloud.google.com/errors/detail/CLSnttKfy4v90wE;time=P30D?project=bridgy-federated
2024-02-08 14:41:15 -08:00
Ryan Barrett
9818b39b16
Web.owns_id bug fix: return None for opted out domain, not False
2024-01-27 13:45:02 -08:00
Ryan Barrett
fcef6c21ab
Revert "heuristic: assume no AP actor id is the root path on its host"
...
This reverts commit b7e890b4bb
.
it was a nice idea, but we're now hitting AP actor ids that are the root path on their host. this is evidently how microblog.pub does AP; 0x3b0b's site https://bw3.dev/ is an example. general info on https://docs.microblog.pub/user_guide.html#activitypub but not this specifically.
fixes https://console.cloud.google.com/errors/detail/CLSnttKfy4v90wE;time=P30D?project=bridgy-federated
2024-01-26 12:59:54 -08:00
Ryan Barrett
114ba69d3b
test fix for 86079c3684
2024-01-22 13:20:03 -08:00
Ryan Barrett
4e5021e2b0
/web-site bug fix, handle opted out users
...
fixes https://console.cloud.google.com/errors/detail/CN7a9KGAicelyAE;time=P30D?project=bridgy-federated
2024-01-22 08:43:33 -08:00
Ryan Barrett
3a8d33ee52
web feed polling: start using ETag/If-None-Match and Last-Modified/If-Modified-Since
...
for #791
2024-01-19 21:07:48 -08:00
Ryan Barrett
09dcabc708
feed polling: bug fix, calculate posting frequency independent of order
2024-01-19 11:33:58 -08:00
Ryan Barrett
7fcbf89d97
feed polling: add Web.feed_last_item, skip parts of feeds we've already seen
...
for #791
2024-01-19 11:33:58 -08:00
Ryan Barrett
a832a4edf6
follow UI: error on bridged accounts
...
also make ActivityPub.owns_handle return False for bridged accounts
fixes https://console.cloud.google.com/errors/detail/CLSnttKfy4v90wE;time=P30D?project=bridgy-federated
2024-01-15 07:57:03 -08:00
Ryan Barrett
b6989a4f2e
if a web user has the webfinger redirects, use AS2 type Person, not Application
2024-01-14 13:53:28 -08:00
Ryan Barrett
7941b632ca
Object._pre_put_hook: require that protocol owns id
2024-01-12 19:53:01 -08:00
Ryan Barrett
e9194b0a66
AP: default actors to type Application, add disclaimer with links to profile bio
...
thanks for the nudge @fromjasonstuff!
2024-01-11 16:37:10 -08:00
Ryan Barrett
c8e2fafba4
web.poll_feed_task: use URL as id since some feeds use non-URL (eg tag URI) ids
...
example: https://www.producthunt.com/feed , Atom. excerpt:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<feed xml:lang="en-US" xmlns="http://www.w3.org/2005/Atom ">
<id>tag:www.producthunt.com,2005:/feed</id>
<link rel="alternate" type="text/html" href="https://www.producthunt.com "/>
<link rel="self" type="application/atom+xml" href="https://www.producthunt.com/feed "/>
<title>Product Hunt — The best new products, every day</title>
<updated>2024-01-10T02:57:06-08:00</updated>
<entry>
<id>tag:www.producthunt.com,2005:Post/432225</id>
<published>2023-12-31T18:15:55-08:00</published>
<updated>2024-01-10T12:54:42-08:00</updated>
<link rel="alternate" type="text/html" href="https://www.producthunt.com/posts/seemless "/>
<title>Seemless</title>
...
```
fixes https://console.cloud.google.com/errors/detail/CKLuk-v4x8X0NQ;time=P30D?project=bridgy-federated
2024-01-11 13:01:45 -08:00
Ryan Barrett
91876a7d22
web.poll_feed_task: handle when feed content and Content-Type don't match
...
fixes https://console.cloud.google.com/errors/detail/CIX--Oz-w4D_Xg;time=P30D?project=bridgy-federated
2024-01-11 13:01:45 -08:00
Ryan Barrett
ab4427c182
web.poll_feed_task: handle missing Content-Type header
...
fixes https://console.cloud.google.com/errors/detail/CKiskJSXi5z9gwE;time=P30D?project=bridgy-federated
2024-01-11 13:01:45 -08:00
Ryan Barrett
81e675f79b
web.poll_feed_task: default actor/author to user
2024-01-09 16:02:41 -08:00
Ryan Barrett
aac85cedca
web feed polling: if we get Content-Type application/xml, use the link's type
...
fixes eg https://simonwillison.net/atom/everything/
2024-01-09 12:00:44 -08:00
Ryan Barrett
bab5270c3b
start clearing Object.our_as1 again on updates
...
fixes #778 . bug was that when we fetched a web object, we'd populate the new `mf2` value, but still keep the old `our_as1`, which then overrode it in `as1`. ugh.
partially reverts commit 62ca4022fb
for #721 .
2024-01-08 13:03:44 -08:00
Ryan Barrett
e82555ad91
Web: add poll feed task for new users, drop task if they've sent a webmention
2024-01-06 13:57:11 -10:00
Ryan Barrett
1db5358688
Web: test a couple poll feed errors
2024-01-06 13:04:22 -10:00
Ryan Barrett
5f1c521500
Web: add last_polled_feed property
2024-01-06 12:56:37 -10:00
Ryan Barrett
26de4097ae
AP: switch default signing actor to real fed.brid.gy instance actor
...
https://seb.jambor.dev/posts/understanding-activitypub-part-4-threads/#the-instance-actor
2024-01-06 11:59:31 -10:00
Ryan Barrett
6025925c26
web: start removing superfeedr code
2024-01-04 16:32:39 -10:00
Ryan Barrett
72df4d7d42
web: enforce https (SSL) requirement
...
fixes https://console.cloud.google.com/errors/detail/CP2isf7noqyZjAE;time=P30D?project=bridgy-federated . thanks for finding @tantek!
2024-01-03 16:32:57 -10:00
Ryan Barrett
beb865bc07
web feed fetching: add next poll task
...
use average of time between posts in feed as ETA
for #550
2024-01-01 20:12:33 -10:00
Ryan Barrett
9347651349
web feed fetching: test RSS, store feed index in AS1
...
for #550
2024-01-01 20:12:33 -10:00
Ryan Barrett
7e702305bf
Web: add poll_feed_task for ingesting Atom/RSS feeds
...
also demotes Web.atom to informational only, we're switching to populate feed entries into our_as1 because we can't easily extract individual items from feedparser.
for #550
2024-01-01 20:12:33 -10:00
Ryan Barrett
79af59b951
upgrade mf2py to 2.0.1, add test for resolving relative URLs in content
...
fixes #390 , re https://github.com/microformats/mf2py/issues/181 . thank you again @angelogladding!
2023-12-08 10:02:07 -08:00
Ryan Barrett
cf17c5ce47
disable superfeedr subscribe/unsubscribe
...
for #550
2023-12-05 13:07:21 -08:00
Ryan Barrett
8f3b4c1c9b
Web: ignore superfeedr subscribe failures
...
it's pretty unreliable right now 😕
for #550
2023-12-05 13:07:17 -08:00
Ryan Barrett
0e5866dbd9
Object.as1 from atom: override more ids with urls when necessary
2023-12-05 12:39:40 -08:00
Ryan Barrett
b7129703cd
Web: don't subscribe to Superfeedr if user has webfinger redirects
...
for #550
2023-12-05 09:28:19 -08:00
Ryan Barrett
9a088f5b7f
Web: unsubscribe from superfeedr on first webmention
...
for #550
2023-12-05 08:59:29 -08:00
Ryan Barrett
d14b1bc04c
add Web.maybe_superfeedr_unsubscribe
...
for #550
2023-12-05 08:51:27 -08:00