e2b6662776
atproto_firehose.handle: use stop-following AS1 verb for deletes of follows
...
for #1669
2025-01-07 08:02:11 -08:00
57fa6a1294
atproto_firehose load_dids: don't load disabled ATProto users
...
for #1669
2025-01-06 20:45:55 -08:00
d957fe0c7b
atproto_firehose.subscribe: don't log #handle, #tombstone
2025-01-05 20:06:48 -08:00
d95724ff67
atproto_firehose subscriber: bug fix for #account/identity events for non-bridged repos
2025-01-05 19:54:41 -08:00
aa4f9e5bf8
atproto_firehose: handle #account and #identity events
...
for #1201
2025-01-05 13:39:17 -08:00
ae0a7920a0
atproto_firehose: enqueue #account and #identity events
...
for #1201
2025-01-05 13:39:17 -08:00
91b268fa4f
unify ndb_client.context() kwargs into common.NDB_CONTEXT_KWARGS
2024-12-18 15:44:35 -08:00
38b0f70841
ATProto.create_for: don't set DNS if the user has a custom domain handle
...
fixes https://console.cloud.google.com/errors/detail/COSMr7uQuqb2zQE;locations=global;time=P30D?project=bridgy-federated
2024-12-11 15:09:56 -08:00
6b923572bd
atproto_firehose.subscribe: prune logging
2024-12-06 10:12:35 -08:00
20dc814c80
add 2m delay to receive tasks for deletes
...
for #1361
2024-11-28 21:39:30 -08:00
aea12c1e39
atproto_firehose._load_dids: if we crash, make sure we re-fetch everything
...
handles (but doesn't fix) https://console.cloud.google.com/errors/detail/CNezhfr029Lq9QE;time=P1D;locations=global?inv=1&invt=AbiGWw&project=bridgy-federated
2024-11-21 15:55:09 -08:00
9e5d7fa3ac
atproto_firehose subscriber: delay reconnecting on exception as well as disconnect
2024-11-20 19:59:29 -08:00
915d63a723
noop: atproto_firehose log message fix
2024-11-18 14:39:13 -08:00
b4ac98c84e
move common.add, remove to webutil.util
...
in snarfed/webutil@46ef97e4fe
2024-10-31 09:41:28 -07:00
fa8149165b
receive task: fix delay logging
...
for #1378
2024-10-23 21:11:21 -07:00
788b37279a
pass Object data in receive task HTTP payload instead of through datastore
...
trying to cut down datastore load and costs. this switches receive tasks from storing and fetching Objects from the datastore to passing their properties - notably as2, our_as1, bsky, etc - in the HTTP request body, as serialized JSON inside form-encoded params.
for #1354 , #1149
2024-10-01 21:44:12 -07:00
6147753e07
atproto_firehose.handle: when we get an ndb ContextError, recreate the context
...
for #1315 , https://console.cloud.google.com/errors/detail/CIvwj_7MmsfOWw;time=PT1H;locations=global?project=bridgy-federated
😠
2024-10-01 08:53:48 -07:00
f2539775d2
atproto_firehose.load_dids: initialize protocol_bot_dids inside NDB context
...
fixes https://console.cloud.google.com/errors/detail/CMv5nZnKiIjJ7gE;locations=global;time=P30D?project=bridgy-federated
2024-09-23 15:33:21 -07:00
f83e3725ec
atproto_firehose.subscribe bug fix: allow follows of protocol bot users from anyone
...
oops! re https://bsky.app/profile/yamakyu.bsky.social/post/3l4rotguqe42h , #1329
2024-09-22 21:50:25 -07:00
f928251efe
atproto_firehose: use decode=False with subscribeRepos
...
for snarfed/lexrpc@677cd0cc1c
2024-09-21 17:34:35 -07:00
776d2b91e0
ATProto firehose subscriber: narrow filtering
...
this drastically reduces the volume of inbound records that we enqueue receive tasks for, from 3-6/s to .2-.4/s. before, we were enqueueing more or less everything from a bridged ATProto user *and* everything that referenced a bridged non-ATProto user. now, it's more like the intersection.
for #1329 , #1149
2024-09-21 07:27:24 -07:00
c5809e4de9
atproto_firehose.subscribe: skip record types we don't support
...
for #1329
2024-09-13 11:10:51 -07:00
a8b4333759
atproto_firehose: add missing import
2024-09-10 12:42:49 -07:00
72877b7f11
downgrade libipld from 2.0.0 back to 1.2.3
...
we're not quite ready for raw bytes CIDs :/ see #1316
also updates a test to match 59b207cd8b
2024-09-10 12:18:56 -07:00
9ba6bdf37d
deps: upgrade libipld to v2.0.0
...
https://github.com/MarshalX/python-libipld/releases/tag/v2.0.0
main breaking change is that CIDs returned by decode_dag_cbor* and decode_car are now raw bytes, not base-32 encoded str. surprisingly we didn't have to change anything special in atproto_firehose to make this work...? still suspicious about that, but tests and local runs work fine.
also other minor atproto_firehose tweaks and comments
2024-09-09 14:20:02 -07:00
156fda24eb
atproto_firehose: move ndb contexts outside outer subscribe/handle loops
...
flailing around blindly to try to fix https://console.cloud.google.com/errors/detail/CIvwj_7MmsfOWw;time=P1D;locations=global?project=bridgy-federated . not very hopeful for this one.
2024-09-09 13:21:42 -07:00
9496957819
atproto_firehose: move CPU-bound filtering back onto main subscribe thread
...
effectively reverts 5b280dc235e7c6ced26a#1295
2024-09-06 18:45:40 -07:00
41aa8c0544
atproto_firehose consumer: add logging metric for events/s, backlog
...
for #1295
2024-09-06 12:28:51 -07:00
b0de078732
atproto_firehose: delay parsing payload until we know it's a commit
...
again, this time for libipld
for #1295
2024-09-06 12:28:51 -07:00
48c9b5c7e7
atproto_firehose: switch CAR decoding from carbox to libipld
...
for #1295
2024-09-06 12:28:50 -07:00
3823379334
atproto_firehose.handle: delay parsing payload until after we've checked header
...
for #1295
2024-09-06 12:28:50 -07:00
e7c6ced26a
atproto_firehose: parallelize CBOR parsing
...
switch to putting raw websocket frame bytes onto queue, then threads parse it. for #1266
2024-09-03 12:15:25 -07:00
5b280dc235
atproto_firehose: move most logic from subscribe to handle
...
trying to offload more CPU from the firehose client. for #1266
2024-09-03 12:15:25 -07:00
1ecad1fcc8
atproto_firehose: add instructions for running locally
...
ie what to comment out to prevent creating receive tasks or updating cursor in the datastore
2024-08-30 13:34:52 -07:00
43c94ad2e0
atproto_firehose.subscribe: handle commits on unsupported collections without record
...
fixes https://console.cloud.google.com/errors/detail/CJT6x7XMiNLobg;time=PT1H;locations=global?project=bridgy-federated
2024-08-30 13:29:39 -07:00
5dfb17428e
atproto_firehose: move storing cursor from handle to subscribe
...
...so that I can run multiple handle threads for scaling
2024-08-30 13:16:17 -07:00
85dd01d0d9
atproto_firehose: log queue size when fetching new DIDs
2024-08-30 09:35:07 -07:00
8ad8565295
atproto_firehose: don't log #account, #identity, #handle events
2024-08-30 09:17:58 -07:00
c1dcc03ee3
atproto_firehose.handle: handle missing $type
...
fixes https://console.cloud.google.com/errors/detail/CLal7vX3yJ6-1QE;locations=global;time=P30D?project=bridgy-federated
2024-08-29 22:02:02 -07:00
88033f1e73
atproto_firehose stability improvement: limit max size of new_commits queue
...
for #1091
2024-08-29 21:51:21 -07:00
9c0ad8c00c
ATProto: poll for new chat messages to protocol bot users
...
for #966
2024-08-27 22:44:37 -07:00
bb9e2d1f95
atproto_firehose.handle: bug fix for encoding record with CID as JSON
2024-08-21 07:55:09 -07:00
e9dc38c75c
actually turn on ndb global cache with global_cache_policy
...
🤦
2024-08-01 12:42:13 -07:00
4097fe71fb
use ndb in-memory context cache for DID doc Objects
...
for #1149
2024-08-01 12:09:37 -07:00
755997e5f7
atproto_firehose: don't load DIDs for tombstoned AtpRepos
...
for #1193
2024-07-22 22:24:13 -07:00
5423857774
atproto_firehose: bug fix for loading ATProto DIDs by updated instead of created
...
for f2be280016
2024-07-20 15:19:18 -07:00
88666cb2d4
atproto_firehose: noop TODO
2024-07-20 11:17:08 -07:00
13da9becf6
atproto_firehose.handle: map delete of activity to undo, not delete
...
for #1162
2024-07-06 15:26:20 -07:00
df75e97880
atproto_firehose.handle bug fix: deletes don't have records
...
for #1162
2024-07-06 14:47:14 -07:00
b606924d63
refactor SUPPORTED_AS1_TYPES usage into new Protocol.check_supported method
2024-07-04 21:11:38 -07:00
Ryan Barrett