Ryan Barrett
e9dc38c75c
actually turn on ndb global cache with global_cache_policy
...
🤦
2024-08-01 12:42:13 -07:00
Ryan Barrett
4097fe71fb
use ndb in-memory context cache for DID doc Objects
...
for #1149
2024-08-01 12:09:37 -07:00
Ryan Barrett
755997e5f7
atproto_firehose: don't load DIDs for tombstoned AtpRepos
...
for #1193
2024-07-22 22:24:13 -07:00
Ryan Barrett
5423857774
atproto_firehose: bug fix for loading ATProto DIDs by updated instead of created
...
for f2be280016
2024-07-20 15:19:18 -07:00
Ryan Barrett
88666cb2d4
atproto_firehose: noop TODO
2024-07-20 11:17:08 -07:00
Ryan Barrett
13da9becf6
atproto_firehose.handle: map delete of activity to undo, not delete
...
for #1162
2024-07-06 15:26:20 -07:00
Ryan Barrett
df75e97880
atproto_firehose.handle bug fix: deletes don't have records
...
for #1162
2024-07-06 14:47:14 -07:00
Ryan Barrett
b606924d63
refactor SUPPORTED_AS1_TYPES usage into new Protocol.check_supported method
2024-07-04 21:11:38 -07:00
Ryan Barrett
850d681f83
add Protocol.SUPPORTED_AS1_TYPES
2024-07-04 16:58:06 -07:00
Ryan Barrett
616073f788
merge hub and atproto_firehose services into a single service, atproto_hub
...
atproto_firehose is averaging 20-40% CPU right now, hub 5-10%, and both are 500-600M memory steady state, so they can easily fit together.
2024-06-13 12:02:54 -07:00
Ryan Barrett
ba00850840
turn on memorystore ndb global cache in hub and atproto_firehose
2024-06-04 19:27:52 -07:00
Ryan Barrett
441849a22a
Revert "Revert "start using an ndb in memory global cache in the frontends and router""
...
This reverts commit 0cbcf81ecc
.
trying again!
2024-06-04 14:27:02 -07:00
Ryan Barrett
0cbcf81ecc
Revert "start using an ndb in memory global cache in the frontends and router"
...
This reverts commit 7e77d7a463
.
This slows down receive task processing, tasks all start taking longer and longer, can't figure out why. some kind of lock starvation in the global cache?
will experiment with this more in the ndb-global-cache branch.
2024-06-01 07:04:38 -07:00
Ryan Barrett
7e77d7a463
start using an ndb in memory global cache in the frontends and router
...
lol yolo 982f8a70df/google/cloud/ndb/global_cache.py (L200-L202)
> Not intended for production use. Uses a single process wide dictionary to
> keep an in memory cache. For use in testing and to have an easily grokkable
> reference implementation. Thread safety is potentially a little sketchy.
2024-05-31 13:08:25 -07:00
Ryan Barrett
f2a64c8b5c
Object.get_or_create: rename actor => authed_as, add to all calls
...
for #566
2024-05-28 16:07:32 -07:00
Ryan Barrett
f2be280016
atproto_firehose: load ATProto users by updated, not by created
...
fixes #1082
2024-05-24 18:40:22 -07:00
Ryan Barrett
4cf0959080
atproto_firehose: oops, still need to skip commits from our bridged users
...
add a test, should have had one originally!
2024-05-22 22:08:25 -07:00
Ryan Barrett
f6575cca4f
atproto_firehose.subscribe: skip bad commits
...
fixes #1061
2024-05-19 13:47:13 -07:00
Ryan Barrett
8e670dadc2
atproto_firehose: handle errors decoding bad DAG-CBOR blocks
...
for https://github.com/hashberg-io/dag-cbor/issues/14
2024-05-18 23:17:07 -07:00
Ryan Barrett
8ee4c3b2ea
atproto_firehose.handle: switch dag_json from compact=True to dialect='atproto'
...
for AT Protocol's DAG-JSON variant with $link and $bytes: https://atproto.com/specs/data-model ; for https://github.com/snarfed/bridgy-fed/issues/1000
2024-05-15 21:00:42 -07:00
Ryan Barrett
031d8b9cd2
atproto_firehose: re-enable handler
2024-05-14 16:53:15 -07:00
Ryan Barrett
8eeb588d11
atproto_firehose: make its own GAE service, lots of other refactoring
...
optimize loading DIDs, move around threads and exception handling, etc.
for #978
2024-05-13 22:02:09 -07:00
Ryan Barrett
f2b5f79489
ATProto firehose: add handle wrapper that catches exceptions
...
also refactor and simplify load_dids, error reporting. #978
2024-05-10 19:59:12 -07:00
Ryan Barrett
6bc9269ad0
ATProto firehose: load bridged dids from AtpRepo instead of users
...
hopefully cuts down on datastore load a bit
for #978
2024-05-10 11:31:47 -07:00
Ryan Barrett
741146eb8d
ATProto firehose: encode Object.bsky as compact DAG-JSON
...
fixes https://console.cloud.google.com/errors/detail/CIGJnbnCq7fm1gE;time=P30D?project=bridgy-federated and similar
2024-05-10 06:51:58 -07:00
Ryan Barrett
b49fe13e59
ATProto firehose: use report_exception() so we get stack traces
2024-05-09 21:31:41 -07:00
Ryan Barrett
989cd29860
ATProto firehose: bug fix for loading DIDs
...
fixes https://console.cloud.google.com/errors/detail/CKKTu6mM0sunXQ;time=P30D?project=bridgy-federated
2024-05-09 21:25:13 -07:00
Ryan Barrett
d56f405c96
ATProto firehose: fix race conditions in loading DIDs
...
for #978
2024-05-09 15:39:18 -07:00
Ryan Barrett
5ef4111b4d
ATProto firehose: reload our DIDs every 20s
...
for #978
2024-05-09 14:31:57 -07:00
Ryan Barrett
66b8846760
ATProto firehose: refactor and extract out loading our DIDs
2024-05-09 14:05:05 -07:00
Ryan Barrett
940c0e8cae
ATProto firehose: don't assume required fields, handle error reporting errors
...
for #978
2024-05-09 13:26:24 -07:00
Ryan Barrett
eea8779872
ATProto firehose: subscribe at stored cursor + 1
...
for #978
2024-05-09 08:44:30 -07:00
Ryan Barrett
87165497f0
misc noop langauge changes in comments and docs, eg BGS => relay
2024-05-09 06:43:51 -07:00
Ryan Barrett
1ecdaf6b34
ATProto firehose: store cursor
2024-05-08 23:11:27 -07:00
Ryan Barrett
b788105a71
ATProto firehose: start implementing cursor
...
for #978
2024-05-08 21:35:03 -07:00
Ryan Barrett
1e5fcfda0b
ATProto firehose: error handling, reconnect in subscribe
2024-05-08 20:43:10 -07:00
Ryan Barrett
b8e0fecc3b
atproto_firehose.handle: implement delete
...
for #978
2024-05-08 20:19:18 -07:00
Ryan Barrett
b4841f3e60
ATProto firehose: noop refactoring, add Op namedtuple
...
for #978
2024-05-08 15:48:39 -07:00
Ryan Barrett
0d24fafefc
ATProto firehose: tests for updates
...
for #978
2024-05-08 13:31:08 -07:00
Ryan Barrett
7867326ebe
ATProto firehose: implement deletes
...
for #978
2024-05-08 13:26:36 -07:00
Ryan Barrett
5c77f02dba
ATProto firehose: bug fix for quote posts with media embeds
...
for #978
2024-05-08 12:13:26 -07:00
Ryan Barrett
20afb59809
ATProto firehose: implement quote posts
...
for #978
2024-05-08 11:39:45 -07:00
Ryan Barrett
72ae63f4a0
ATProto firehose: test replies, mentions
...
#978
2024-05-08 11:31:31 -07:00
Ryan Barrett
44763ca8ab
ATProto firehose: more tests
...
for #978
2024-05-08 11:18:50 -07:00
Ryan Barrett
e214bc0a43
ATProto firehose: more tests
...
for #978
2024-05-08 10:39:03 -07:00
Ryan Barrett
7ecf3dc381
ATProto firehose: start tests
...
for #978
2024-05-07 17:36:34 -07:00
Ryan Barrett
0f35a2e1a1
ATProto firehose: convert to function we can run in a thread
...
for #978
2024-05-07 16:58:52 -07:00
Ryan Barrett
622d07a984
ATProto firehose: move and rename file
2024-05-07 16:24:16 -07:00