bridgy-fed

Wykres commitów

Autor	SHA1	Wiadomość	Data
Ryan Barrett	df75e97880	atproto_firehose.handle bug fix: deletes don't have records for #1162	2024-07-06 14:47:14 -07:00
Ryan Barrett	b606924d63	refactor SUPPORTED_AS1_TYPES usage into new Protocol.check_supported method	2024-07-04 21:11:38 -07:00
Ryan Barrett	850d681f83	add Protocol.SUPPORTED_AS1_TYPES	2024-07-04 16:58:06 -07:00
Ryan Barrett	616073f788	merge hub and atproto_firehose services into a single service, atproto_hub atproto_firehose is averaging 20-40% CPU right now, hub 5-10%, and both are 500-600M memory steady state, so they can easily fit together.	2024-06-13 12:02:54 -07:00
Ryan Barrett	ba00850840	turn on memorystore ndb global cache in hub and atproto_firehose	2024-06-04 19:27:52 -07:00
Ryan Barrett	441849a22a	Revert "Revert "start using an ndb in memory global cache in the frontends and router"" This reverts commit `0cbcf81ecc`. trying again!	2024-06-04 14:27:02 -07:00
Ryan Barrett	0cbcf81ecc	Revert "start using an ndb in memory global cache in the frontends and router" This reverts commit `7e77d7a463`. This slows down receive task processing, tasks all start taking longer and longer, can't figure out why. some kind of lock starvation in the global cache? will experiment with this more in the ndb-global-cache branch.	2024-06-01 07:04:38 -07:00
Ryan Barrett	7e77d7a463	start using an ndb in memory global cache in the frontends and router lol yolo `982f8a70df/google/cloud/ndb/global_cache.py (L200-L202)` > Not intended for production use. Uses a single process wide dictionary to > keep an in memory cache. For use in testing and to have an easily grokkable > reference implementation. Thread safety is potentially a little sketchy.	2024-05-31 13:08:25 -07:00
Ryan Barrett	f2a64c8b5c	Object.get_or_create: rename actor => authed_as, add to all calls for #566	2024-05-28 16:07:32 -07:00
Ryan Barrett	f2be280016	atproto_firehose: load ATProto users by updated, not by created fixes #1082	2024-05-24 18:40:22 -07:00
Ryan Barrett	4cf0959080	atproto_firehose: oops, still need to skip commits from our bridged users add a test, should have had one originally!	2024-05-22 22:08:25 -07:00
Ryan Barrett	f6575cca4f	atproto_firehose.subscribe: skip bad commits fixes #1061	2024-05-19 13:47:13 -07:00
Ryan Barrett	8e670dadc2	atproto_firehose: handle errors decoding bad DAG-CBOR blocks for https://github.com/hashberg-io/dag-cbor/issues/14	2024-05-18 23:17:07 -07:00
Ryan Barrett	8ee4c3b2ea	atproto_firehose.handle: switch dag_json from compact=True to dialect='atproto' for AT Protocol's DAG-JSON variant with $link and $bytes: https://atproto.com/specs/data-model ; for https://github.com/snarfed/bridgy-fed/issues/1000	2024-05-15 21:00:42 -07:00
Ryan Barrett	031d8b9cd2	atproto_firehose: re-enable handler	2024-05-14 16:53:15 -07:00
Ryan Barrett	8eeb588d11	atproto_firehose: make its own GAE service, lots of other refactoring optimize loading DIDs, move around threads and exception handling, etc. for #978	2024-05-13 22:02:09 -07:00
Ryan Barrett	f2b5f79489	ATProto firehose: add handle wrapper that catches exceptions also refactor and simplify load_dids, error reporting. #978	2024-05-10 19:59:12 -07:00
Ryan Barrett	6bc9269ad0	ATProto firehose: load bridged dids from AtpRepo instead of users hopefully cuts down on datastore load a bit for #978	2024-05-10 11:31:47 -07:00
Ryan Barrett	741146eb8d	ATProto firehose: encode Object.bsky as compact DAG-JSON fixes https://console.cloud.google.com/errors/detail/CIGJnbnCq7fm1gE;time=P30D?project=bridgy-federated and similar	2024-05-10 06:51:58 -07:00
Ryan Barrett	b49fe13e59	ATProto firehose: use report_exception() so we get stack traces	2024-05-09 21:31:41 -07:00
Ryan Barrett	989cd29860	ATProto firehose: bug fix for loading DIDs fixes https://console.cloud.google.com/errors/detail/CKKTu6mM0sunXQ;time=P30D?project=bridgy-federated	2024-05-09 21:25:13 -07:00
Ryan Barrett	d56f405c96	ATProto firehose: fix race conditions in loading DIDs for #978	2024-05-09 15:39:18 -07:00
Ryan Barrett	5ef4111b4d	ATProto firehose: reload our DIDs every 20s for #978	2024-05-09 14:31:57 -07:00
Ryan Barrett	66b8846760	ATProto firehose: refactor and extract out loading our DIDs	2024-05-09 14:05:05 -07:00
Ryan Barrett	940c0e8cae	ATProto firehose: don't assume required fields, handle error reporting errors for #978	2024-05-09 13:26:24 -07:00
Ryan Barrett	eea8779872	ATProto firehose: subscribe at stored cursor + 1 for #978	2024-05-09 08:44:30 -07:00
Ryan Barrett	87165497f0	misc noop langauge changes in comments and docs, eg BGS => relay	2024-05-09 06:43:51 -07:00
Ryan Barrett	1ecdaf6b34	ATProto firehose: store cursor	2024-05-08 23:11:27 -07:00
Ryan Barrett	b788105a71	ATProto firehose: start implementing cursor for #978	2024-05-08 21:35:03 -07:00
Ryan Barrett	1e5fcfda0b	ATProto firehose: error handling, reconnect in subscribe	2024-05-08 20:43:10 -07:00
Ryan Barrett	b8e0fecc3b	atproto_firehose.handle: implement delete for #978	2024-05-08 20:19:18 -07:00
Ryan Barrett	b4841f3e60	ATProto firehose: noop refactoring, add Op namedtuple for #978	2024-05-08 15:48:39 -07:00
Ryan Barrett	0d24fafefc	ATProto firehose: tests for updates for #978	2024-05-08 13:31:08 -07:00
Ryan Barrett	7867326ebe	ATProto firehose: implement deletes for #978	2024-05-08 13:26:36 -07:00
Ryan Barrett	5c77f02dba	ATProto firehose: bug fix for quote posts with media embeds for #978	2024-05-08 12:13:26 -07:00
Ryan Barrett	20afb59809	ATProto firehose: implement quote posts for #978	2024-05-08 11:39:45 -07:00
Ryan Barrett	72ae63f4a0	ATProto firehose: test replies, mentions #978	2024-05-08 11:31:31 -07:00
Ryan Barrett	44763ca8ab	ATProto firehose: more tests for #978	2024-05-08 11:18:50 -07:00
Ryan Barrett	e214bc0a43	ATProto firehose: more tests for #978	2024-05-08 10:39:03 -07:00
Ryan Barrett	7ecf3dc381	ATProto firehose: start tests for #978	2024-05-07 17:36:34 -07:00
Ryan Barrett	0f35a2e1a1	ATProto firehose: convert to function we can run in a thread for #978	2024-05-07 16:58:52 -07:00
Ryan Barrett	622d07a984	ATProto firehose: move and rename file	2024-05-07 16:24:16 -07:00

42 Commity (df75e9788028cb50a4f500a14aa4c4f60adff719)