bridgy-fed/queue.yaml

# https://cloud.google.com/appengine/docs/standard/python/config/queueref
# https://cloud.google.com/tasks/docs/queue-yaml
queue:
- name: atproto-commit
  rate: 300/s
  max_concurrent_requests: 1
  retry_parameters:
    task_retry_limit: 2
    min_backoff_seconds: 5
    max_doublings: 5

- name: webmention
  rate: 500/s
  max_concurrent_requests: 10
  retry_parameters:
    task_retry_limit: 2
    min_backoff_seconds: 300
    max_doublings: 2

- name: receive
  rate: 500/s
  max_concurrent_requests: 160
  retry_parameters:
    task_retry_limit: 2
    min_backoff_seconds: 300
    max_doublings: 2

- name: send
  rate: 500/s
  max_concurrent_requests: 40
  retry_parameters:
    task_retry_limit: 2
    min_backoff_seconds: 300
    max_doublings: 2

- name: poll-feed
  rate: 5/s
  max_concurrent_requests: 1
  retry_parameters:
    task_retry_limit: 2
    min_backoff_seconds: 300
    max_doublings: 2
add new atproto-commit task queue, enqueue tasks for new commits hub will consume these and emit them to subscribeRepos subscribers 2023-09-06 03:10:11 +00:00			`# https://cloud.google.com/appengine/docs/standard/python/config/queueref`
			`# https://cloud.google.com/tasks/docs/queue-yaml`
			`queue:`
			`- name: atproto-commit`
infra: add new router flex service, switch tasks to run on it receive, send, webmention, and poll-feed 2024-05-24 03:40:54 +00:00			`rate: 300/s`
			`max_concurrent_requests: 1`
add new atproto-commit task queue, enqueue tasks for new commits hub will consume these and emit them to subscribeRepos subscribers 2023-09-06 03:10:11 +00:00			`retry_parameters:`
bump up task queue rates to handle new volume 2024-05-03 18:00:28 +00:00			`task_retry_limit: 2`
add new atproto-commit task queue, enqueue tasks for new commits hub will consume these and emit them to subscribeRepos subscribers 2023-09-06 03:10:11 +00:00			`min_backoff_seconds: 5`
			`max_doublings: 5`

			`- name: webmention`
bump up router threads from 40 to 50, task queue rate limits router is hitting 100% CPU sustained during bursts right now, probably need to either rethink its architecture or just add another core soon :/ 2024-06-12 22:48:11 +00:00			`rate: 500/s`
infra: add new router flex service, switch tasks to run on it receive, send, webmention, and poll-feed 2024-05-24 03:40:54 +00:00			`max_concurrent_requests: 10`
add new atproto-commit task queue, enqueue tasks for new commits hub will consume these and emit them to subscribeRepos subscribers 2023-09-06 03:10:11 +00:00			`retry_parameters:`
bump up task queue rates to handle new volume 2024-05-03 18:00:28 +00:00			`task_retry_limit: 2`
task queues: bump up first retry delay from 2m to 5m 2024-05-31 20:31:45 +00:00			`min_backoff_seconds: 300`
bump up task queue rates to handle new volume 2024-05-03 18:00:28 +00:00			`max_doublings: 2`
add receive task queue 2023-09-19 18:34:41 +00:00
			`- name: receive`
bump up router threads from 40 to 50, task queue rate limits router is hitting 100% CPU sustained during bursts right now, probably need to either rethink its architecture or just add another core soon :/ 2024-06-12 22:48:11 +00:00			`rate: 500/s`
bump up router to 4 cores, bump up receive and send queues to match for #1138 2024-06-25 20:39:42 +00:00			`max_concurrent_requests: 160`
add receive task queue 2023-09-19 18:34:41 +00:00			`retry_parameters:`
bump up task queue rates to handle new volume 2024-05-03 18:00:28 +00:00			`task_retry_limit: 2`
task queues: bump up first retry delay from 2m to 5m 2024-05-31 20:31:45 +00:00			`min_backoff_seconds: 300`
bump up task queue rates to handle new volume 2024-05-03 18:00:28 +00:00			`max_doublings: 2`
delivery: switch from threads to tasks, one per send for #652 2023-10-31 19:49:15 +00:00
			`- name: send`
bump up router threads from 40 to 50, task queue rate limits router is hitting 100% CPU sustained during bursts right now, probably need to either rethink its architecture or just add another core soon :/ 2024-06-12 22:48:11 +00:00			`rate: 500/s`
bump up router to 4 cores, bump up receive and send queues to match for #1138 2024-06-25 20:39:42 +00:00			`max_concurrent_requests: 40`
delivery: switch from threads to tasks, one per send for #652 2023-10-31 19:49:15 +00:00			`retry_parameters:`
bump up task queue rates to handle new volume 2024-05-03 18:00:28 +00:00			`task_retry_limit: 2`
task queues: bump up first retry delay from 2m to 5m 2024-05-31 20:31:45 +00:00			`min_backoff_seconds: 300`
bump up task queue rates to handle new volume 2024-05-03 18:00:28 +00:00			`max_doublings: 2`
add poll-feed task queue 2024-01-06 22:44:09 +00:00
			`- name: poll-feed`
			`rate: 5/s`
			`max_concurrent_requests: 1`
			`retry_parameters:`
bump up task queue rates to handle new volume 2024-05-03 18:00:28 +00:00			`task_retry_limit: 2`
task queues: bump up first retry delay from 2m to 5m 2024-05-31 20:31:45 +00:00			`min_backoff_seconds: 300`
bump up task queue rates to handle new volume 2024-05-03 18:00:28 +00:00			`max_doublings: 2`