60 Commity (3823379334236db185d5a0527e63be2f17f1b509)

Autor	SHA1	Wiadomość	Data
Ryan Barrett	3823379334	atproto_firehose.handle: delay parsing payload until after we've checked header ... for #1295	2024-09-06 12:28:50 -07:00
Ryan Barrett	e7c6ced26a	atproto_firehose: parallelize CBOR parsing ... switch to putting raw websocket frame bytes onto queue, then threads parse it. for #1266	2024-09-03 12:15:25 -07:00
Ryan Barrett	5b280dc235	atproto_firehose: move most logic from subscribe to handle ... trying to offload more CPU from the firehose client. for #1266	2024-09-03 12:15:25 -07:00
Ryan Barrett	1ecad1fcc8	atproto_firehose: add instructions for running locally ... ie what to comment out to prevent creating receive tasks or updating cursor in the datastore	2024-08-30 13:34:52 -07:00
Ryan Barrett	43c94ad2e0	atproto_firehose.subscribe: handle commits on unsupported collections without record ... fixes https://console.cloud.google.com/errors/detail/CJT6x7XMiNLobg;time=PT1H;locations=global?project=bridgy-federated	2024-08-30 13:29:39 -07:00
Ryan Barrett	5dfb17428e	atproto_firehose: move storing cursor from handle to subscribe ... ...so that I can run multiple handle threads for scaling	2024-08-30 13:16:17 -07:00
Ryan Barrett	85dd01d0d9	atproto_firehose: log queue size when fetching new DIDs	2024-08-30 09:35:07 -07:00
Ryan Barrett	8ad8565295	atproto_firehose: don't log #account, #identity, #handle events	2024-08-30 09:17:58 -07:00
Ryan Barrett	c1dcc03ee3	atproto_firehose.handle: handle missing $type ... fixes https://console.cloud.google.com/errors/detail/CLal7vX3yJ6-1QE;locations=global;time=P30D?project=bridgy-federated	2024-08-29 22:02:02 -07:00
Ryan Barrett	88033f1e73	atproto_firehose stability improvement: limit max size of new_commits queue ... for #1091	2024-08-29 21:51:21 -07:00
Ryan Barrett	9c0ad8c00c	ATProto: poll for new chat messages to protocol bot users ... for #966	2024-08-27 22:44:37 -07:00
Ryan Barrett	bb9e2d1f95	atproto_firehose.handle: bug fix for encoding record with CID as JSON	2024-08-21 07:55:09 -07:00
Ryan Barrett	e9dc38c75c	actually turn on ndb global cache with global_cache_policy ... 🤦	2024-08-01 12:42:13 -07:00
Ryan Barrett	4097fe71fb	use ndb in-memory context cache for DID doc Objects ... for #1149	2024-08-01 12:09:37 -07:00
Ryan Barrett	755997e5f7	atproto_firehose: don't load DIDs for tombstoned AtpRepos ... for #1193	2024-07-22 22:24:13 -07:00
Ryan Barrett	5423857774	atproto_firehose: bug fix for loading ATProto DIDs by updated instead of created ... for f2be280016	2024-07-20 15:19:18 -07:00
Ryan Barrett	88666cb2d4	atproto_firehose: noop TODO	2024-07-20 11:17:08 -07:00
Ryan Barrett	13da9becf6	atproto_firehose.handle: map delete of activity to undo, not delete ... for #1162	2024-07-06 15:26:20 -07:00
Ryan Barrett	df75e97880	atproto_firehose.handle bug fix: deletes don't have records ... for #1162	2024-07-06 14:47:14 -07:00
Ryan Barrett	b606924d63	refactor SUPPORTED_AS1_TYPES usage into new Protocol.check_supported method	2024-07-04 21:11:38 -07:00
Ryan Barrett	850d681f83	add Protocol.SUPPORTED_AS1_TYPES	2024-07-04 16:58:06 -07:00
Ryan Barrett	616073f788	merge hub and atproto_firehose services into a single service, atproto_hub ... atproto_firehose is averaging 20-40% CPU right now, hub 5-10%, and both are 500-600M memory steady state, so they can easily fit together.	2024-06-13 12:02:54 -07:00
Ryan Barrett	ba00850840	turn on memorystore ndb global cache in hub and atproto_firehose	2024-06-04 19:27:52 -07:00
Ryan Barrett	441849a22a	Revert "Revert "start using an ndb in memory global cache in the frontends and router"" ... This reverts commit 0cbcf81ecc. trying again!	2024-06-04 14:27:02 -07:00
Ryan Barrett	0cbcf81ecc	Revert "start using an ndb in memory global cache in the frontends and router" ... This reverts commit 7e77d7a463. This slows down receive task processing, tasks all start taking longer and longer, can't figure out why. some kind of lock starvation in the global cache? will experiment with this more in the ndb-global-cache branch.	2024-06-01 07:04:38 -07:00
Ryan Barrett	7e77d7a463	start using an ndb in memory global cache in the frontends and router ... lol yolo 982f8a70df/google/cloud/ndb/global_cache.py (L200-L202) > Not intended for production use. Uses a single process wide dictionary to > keep an in memory cache. For use in testing and to have an easily grokkable > reference implementation. Thread safety is potentially a little sketchy.	2024-05-31 13:08:25 -07:00
Ryan Barrett	f2a64c8b5c	Object.get_or_create: rename actor => authed_as, add to all calls ... for #566	2024-05-28 16:07:32 -07:00
Ryan Barrett	f2be280016	atproto_firehose: load ATProto users by updated, not by created ... fixes #1082	2024-05-24 18:40:22 -07:00
Ryan Barrett	4cf0959080	atproto_firehose: oops, still need to skip commits from our bridged users ... add a test, should have had one originally!	2024-05-22 22:08:25 -07:00
Ryan Barrett	f6575cca4f	atproto_firehose.subscribe: skip bad commits ... fixes #1061	2024-05-19 13:47:13 -07:00
Ryan Barrett	8e670dadc2	atproto_firehose: handle errors decoding bad DAG-CBOR blocks ... for https://github.com/hashberg-io/dag-cbor/issues/14	2024-05-18 23:17:07 -07:00
Ryan Barrett	8ee4c3b2ea	atproto_firehose.handle: switch dag_json from compact=True to dialect='atproto' ... for AT Protocol's DAG-JSON variant with $link and $bytes: https://atproto.com/specs/data-model ; for https://github.com/snarfed/bridgy-fed/issues/1000	2024-05-15 21:00:42 -07:00
Ryan Barrett	031d8b9cd2	atproto_firehose: re-enable handler	2024-05-14 16:53:15 -07:00
Ryan Barrett	8eeb588d11	atproto_firehose: make its own GAE service, lots of other refactoring ... optimize loading DIDs, move around threads and exception handling, etc. for #978	2024-05-13 22:02:09 -07:00
Ryan Barrett	f2b5f79489	ATProto firehose: add handle wrapper that catches exceptions ... also refactor and simplify load_dids, error reporting. #978	2024-05-10 19:59:12 -07:00
Ryan Barrett	6bc9269ad0	ATProto firehose: load bridged dids from AtpRepo instead of users ... hopefully cuts down on datastore load a bit for #978	2024-05-10 11:31:47 -07:00
Ryan Barrett	741146eb8d	ATProto firehose: encode Object.bsky as compact DAG-JSON ... fixes https://console.cloud.google.com/errors/detail/CIGJnbnCq7fm1gE;time=P30D?project=bridgy-federated and similar	2024-05-10 06:51:58 -07:00
Ryan Barrett	b49fe13e59	ATProto firehose: use report_exception() so we get stack traces	2024-05-09 21:31:41 -07:00
Ryan Barrett	989cd29860	ATProto firehose: bug fix for loading DIDs ... fixes https://console.cloud.google.com/errors/detail/CKKTu6mM0sunXQ;time=P30D?project=bridgy-federated	2024-05-09 21:25:13 -07:00
Ryan Barrett	d56f405c96	ATProto firehose: fix race conditions in loading DIDs ... for #978	2024-05-09 15:39:18 -07:00
Ryan Barrett	5ef4111b4d	ATProto firehose: reload our DIDs every 20s ... for #978	2024-05-09 14:31:57 -07:00
Ryan Barrett	66b8846760	ATProto firehose: refactor and extract out loading our DIDs	2024-05-09 14:05:05 -07:00
Ryan Barrett	940c0e8cae	ATProto firehose: don't assume required fields, handle error reporting errors ... for #978	2024-05-09 13:26:24 -07:00
Ryan Barrett	eea8779872	ATProto firehose: subscribe at stored cursor + 1 ... for #978	2024-05-09 08:44:30 -07:00
Ryan Barrett	87165497f0	misc noop langauge changes in comments and docs, eg BGS => relay	2024-05-09 06:43:51 -07:00
Ryan Barrett	1ecdaf6b34	ATProto firehose: store cursor	2024-05-08 23:11:27 -07:00
Ryan Barrett	b788105a71	ATProto firehose: start implementing cursor ... for #978	2024-05-08 21:35:03 -07:00
Ryan Barrett	1e5fcfda0b	ATProto firehose: error handling, reconnect in subscribe	2024-05-08 20:43:10 -07:00
Ryan Barrett	b8e0fecc3b	atproto_firehose.handle: implement delete ... for #978	2024-05-08 20:19:18 -07:00
Ryan Barrett	b4841f3e60	ATProto firehose: noop refactoring, add Op namedtuple ... for #978	2024-05-08 15:48:39 -07:00