mirror
/
moonstream
kopia lustrzana https://github.com/bugout-dev/moonstream
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność
3 451 Commity
108 Gałęzie
40 Tagi
21 MiB
Wykres commitów

872 Commity (main)

Autor SHA1 Wiadomość Data
Andrey 0436ab784a init state. 2022-10-27 13:00:08 +03:00
Andrey 15126c5086 Add autoscale batch size. 2022-10-26 14:48:30 +03:00
Andrey c580a7d3c2 Add jobs. 2022-10-20 21:20:01 +03:00
gnawali 777414c7db
Update README.md 
The site is now available only with www prefix
 2022-10-20 11:36:17 -05:00
Andrey 94e98a3059 Add logging and thread poolexecutor. 2022-10-20 17:40:06 +03:00
kompotkot accb5caf03 Mumbai sync and missing crawlers 2022-10-19 09:22:06 +00:00
Sergei Sumarokov a8470441cb
Merge pull request #680 from zomglings/fix-crawler-tx-rollback 
Added transaction rollback when there is a violation of unique constraint when crawling blocks
 2022-10-18 17:13:44 +03:00
Andrey 9f441373fe Add covert args address to checksum address. 2022-10-18 16:39:44 +03:00
Andrey b8ce45470b Add logging. 2022-10-18 16:28:03 +03:00
Neeraj Kashyap 3d7bbb03ce Added a rollback when unique constraint is violated 
when attempting to add blocks to database.
 2022-10-14 21:05:49 -07:00
Andrey 2f5147a84f Add timeout for HTTPProvider. 2022-10-05 18:18:12 +03:00
Andrey 20cda62a1b Fix missing rollback. 2022-10-05 14:26:56 +03:00
Andrey e5f2c4a550 Blocks cache was increased. 2022-10-03 18:43:40 +03:00
Andrey 03af327f8e Move log under if. 2022-10-03 17:26:58 +03:00
Andrey dd9b462c4e Request not full block db raw. Move log under if. 2022-10-03 17:25:39 +03:00
Andrey cccb42156e Remove unnecessary log. 2022-09-21 19:09:47 +03:00
Andrey 8c6d22c88b Replace moonworm==0.5.1 -> moonworm[moonstream]==0.5.1. 
Change version.
 2022-09-21 19:01:04 +03:00
Andrey 3e3288a55f Add remove comments. 2022-09-21 18:41:42 +03:00
Andrey 3f35b47a91 Add autoscale to historical crawl. 2022-09-21 18:38:05 +03:00
Sergei Sumarokov 01d0b8dd03
Merge pull request #670 from bugout-dev/fix-timer-on-boot 
Different timer on boot start time
 2022-09-20 15:35:03 +03:00
Andrey bffe318cd9 Remove address requirements for historical_crawler. 2022-09-20 14:14:37 +03:00
kompotkot a4c04a472a Different timer on boot start time 2022-09-15 13:22:10 +00:00
kompotkot 66ac8c9d5e Clean state service and timer 2022-09-14 14:24:37 +00:00
Andrey 2b8e1f4b7b Add commit in that commit. 2022-09-14 15:51:38 +03:00
Andrey e8f787dcb0 Fix. 2022-09-14 13:29:06 +03:00
Andrey 7074eb9923 Add orm version of query. 2022-09-14 13:23:35 +03:00
Andrey d31ef8b909 Add initial version. 2022-09-13 19:03:48 +03:00
kompotkot 7dc045c3bf Metadata services and deploy 2022-09-08 19:17:30 +00:00
Andrey d200e82460 Fix user MOONSTREAM_DB_URI_READ_ONLY -> MOONSTREAM_DB_URI. 2022-09-08 18:34:50 +03:00
Andrey ec718f0e4c Move table name from parameters to format. 2022-09-08 17:51:01 +03:00
Andrey 14dedab479 Merge branch 'main' into metadata-crawler 2022-09-08 17:42:40 +03:00
Andrey 31aa042dfa Fix mypy. Add logger. Query use table name. 2022-09-08 17:39:28 +03:00
Andrey b0a605cf90 Add logging. 2022-09-08 13:26:14 +03:00
Andrey Dolgolev 7252d06cad
Merge pull request #659 from bugout-dev/cu-state-crawler 
Cu state crawler
 2022-09-08 13:04:50 +03:00
Andrey 4a22a593e0 Fix moonstream db user MOONSTREAM_DB_URI_READ_ONLY -> MOONSTREAM_DB_URI. 2022-09-08 13:01:44 +03:00
Andrey 808c604299 Change veesion. 2022-09-08 12:53:00 +03:00
Andrey 95ab437a22 Merge branch 'cu-state-crawler' of github.com:bugout-dev/moonstream into cu-state-crawler 2022-09-08 12:52:16 +03:00
Andrey 88d1c5a77b Add fixes. 2022-09-08 12:51:41 +03:00
kompotkot 6582d12759 Deployment for state crawler 2022-09-08 09:39:38 +00:00
Andrey def1dc1527 Add sanityze for label data. 2022-09-07 15:15:15 +03:00
Andrey 7d073c3e63 Fix lint issues. 2022-09-07 14:48:33 +03:00
Andrey 8855097728 Switch using web3py instead brownie. 2022-09-06 19:05:02 +03:00
Andrey 12cbe3e797 Add replace of r"\u0000". 2022-09-05 17:05:11 +03:00
Andrey d9c80e566c Add general interface. 2022-09-05 13:35:48 +03:00
Andrey 59f8f22b2e Clean up code. 2022-09-01 17:26:57 +03:00
Andrey 42ebb66e4d Switch to urlopen lib. 2022-09-01 16:04:07 +03:00
Andrey aaa6c95953 Add init metadata parser. 2022-08-31 19:26:30 +03:00
Andrey c2d790d08d Add settings. 2022-08-31 16:50:36 +03:00
Andrey 7fe587fdbc Merge branch 'main' into cu-state-crawler 2022-08-30 17:38:47 +03:00
Andrey 54870630c4 Add inital version. 2022-08-30 17:37:38 +03:00
Andrey b3eba087f0 fix try. 2022-08-25 20:28:57 +03:00
Andrey af8e114515 Update version in setup.py add pool_pre_ping. 2022-08-25 20:25:26 +03:00
Andrey 5646dbf442 Add fixes. 2022-08-25 19:58:02 +03:00
Andrey 77c3e7e2f8 Add worked tasks resolver without write to database. 2022-08-25 19:09:55 +03:00
Andrey 9d8566cb59 Black formating. 2022-08-25 15:22:01 +03:00
Andrey 935ebe11a2 remove print. black formating. 2022-08-25 15:21:09 +03:00
Andrey e6a730d760 Add more complex typecheck. 
Tested new created session with custom timeout..
Add reporter for stats execution.
 2022-08-25 15:13:46 +03:00
kompotkot 1cfd22adb4 Mypy fix 2022-08-17 14:56:35 +00:00
kompotkot 6a140e7cac Naming and try catch for moonstream bugout client api call 2022-08-17 14:50:47 +00:00
Andrey b14eae475c Black formating. 2022-08-17 17:42:37 +03:00
Andrey 1339de073b Fix name. 2022-08-17 17:38:54 +03:00
Andrey 870faf15d7 Add timeout for bugout in crawlers. 2022-08-17 17:25:00 +03:00
kompotkot 70f96ab457 Merge branch 'main' into mumbai-support-crawlers 2022-08-11 14:28:30 +00:00
Andrey 3534456ede Black formating. 2022-08-10 23:25:30 +03:00
Andrey a0b3418bf6 All not string and int columns transform to string. 2022-08-10 21:06:48 +03:00
kompotkot 5b912ff0d9 Bumped version 2022-08-10 17:03:04 +00:00
kompotkot de879a7adf Mumbai chain support 2022-08-10 16:55:49 +00:00
kompotkot 9496214d58 Modifications of crawlers for mumbai 2022-08-09 16:29:19 +00:00
Yhtyyar Sahatov df4f11ef48
Merge pull request #634 from bugout-dev/moonworm-historical-crawler 
Moonworm historical crawler
 2022-07-25 15:49:47 +03:00
kompotkot ab0ce50cf9 There is no need of data key at tx 2022-07-16 15:04:24 +00:00
kompotkot 3b965522d4 Better with exception 2022-06-22 19:01:06 +00:00
kompotkot c2b8a55778 Pointing to correct network at moonworm_crawler 2022-06-22 18:45:53 +00:00
Yhtyyar Sahatov da78bffbd0 removed select 1 2022-06-22 18:19:23 +03:00
Yhtyyar Sahatov 0f110bf16a removed unneeded log 2022-06-22 18:18:02 +03:00
Yhtyyar Sahatov 2336dba7c9 fixed some issues 2022-06-22 15:34:02 +03:00
Yhtyyar Sahatov 4166d71d41 added help message for `crawl` command 2022-06-21 14:33:45 +03:00
Yhtyyar Sahatov a5a9df8695 added 2022-06-21 14:31:26 +03:00
Andrey Dolgolev 648931d2b1 remove comments. 2022-06-21 13:35:11 +03:00
Andrey Dolgolev 96e570039c Black formating. 2022-06-21 13:24:41 +03:00
Andrey Dolgolev ab9622b2c3 Add fix. 2022-06-21 13:21:01 +03:00
Andrey Dolgolev 1b106a08e0 Merge branch 'main' into dashboard-stats-use-replica 2022-06-21 13:12:05 +03:00
Andrey Dolgolev d135a0710c Remove moonworm condition. 2022-06-21 13:00:05 +03:00
Andrey Dolgolev b32a993bd7 Fix. 2022-06-21 12:58:05 +03:00
Andrey Dolgolev 257bcc7728 Remove sqlalchemy logger. 2022-06-21 12:51:31 +03:00
Yhtyyar Sahatov a3f2418820 isort 2022-06-16 15:54:19 +03:00
Yhtyyar Sahatov 14ef9372a2 working hisotic crawler 2022-06-16 15:53:19 +03:00
kompotkot b3fe75ae34 Removed start block for moonworm xdai crawler 2022-06-11 12:07:20 +00:00
kompotkot 2cfdc2dc99 Applied CPU Weight 2022-06-11 12:06:01 +00:00
kompotkot 5cf74404f9 Merge branch 'main' into fix-systemd-assignment-out 2022-06-11 09:51:51 +00:00
Andrey Dolgolev cdfc970f37 Add try select 1; 2022-06-02 23:39:11 +03:00
kompotkot 4f2ccb9830 XDai moonworm crawler service 2022-05-26 14:32:27 +00:00
kompotkot 60e3c078fb Merge branch 'moonworm-prod-xdai-support' into xdai-subscription 2022-05-26 14:23:59 +00:00
Yhtyyar Sahatov 7c80be419b moved commit back to interval 2022-05-26 17:22:45 +03:00
Yhtyyar Sahatov d25b4d6866 moved commit_session out of heart beat interval 2022-05-26 17:19:48 +03:00
kompotkot 3e7b566c63 Black synt fix 2022-05-26 13:43:49 +00:00
kompotkot 9776db79d6 Update moonstreamdb version 2022-05-26 13:31:00 +00:00
kompotkot 5fd98a3b8a Added access id to work with nb 2022-05-26 13:10:55 +00:00
kompotkot ea62342b07 Merge branch 'main' into xdai-subscription 2022-05-26 13:00:47 +00:00
Yhtyyar Sahatov 5dc7716343 added xdai support 2022-05-26 16:00:35 +03:00
Yhtyyar Sahatov 23e4ec367c isort 2022-05-26 15:41:58 +03:00
Yhtyyar Sahatov 5641fe7119 linting fixes 2022-05-26 15:41:27 +03:00
Yhtyyar Sahatov b085bb9049 Merge branch 'main' into nft-dataset-v2 2022-05-26 15:32:26 +03:00
Yhtyyar Sahatov 181d1769a0 fixed cli choices, added help messages 2022-05-26 15:22:09 +03:00
Yhtyyar Sahatov 5d31b8d58a fixed choices in cli 2022-05-26 15:15:12 +03:00
Andrey Dolgolev d04b877621 Add cte condition. 2022-05-26 13:35:19 +03:00
kompotkot d09570b5dc Moonworm version updated in setup dependencies 2022-05-25 14:23:28 +00:00
kompotkot f72a99295e XDai statistics dashboard deployment 2022-05-25 14:10:52 +00:00
kompotkot a2e4565c45 Updated version and moonstreamdb dependence for mooncrawl package 2022-05-25 13:48:42 +00:00
kompotkot b38455826c XDai deployment services and scripts 2022-05-25 10:08:25 +00:00
kompotkot 942eca1330 XDai blockchain support 2022-05-24 08:21:53 +00:00
kompotkot 2bfec90e87 With backslash systemd throwing error with assignment outside 2022-05-17 20:31:01 +00:00
kompotkot 3d97c4cb25 Moonworm crawler update to work with nodebalancer access id 2022-05-17 14:49:11 +00:00
kompotkot 4e358326af Updated version of mooncrawl 2022-05-16 13:46:11 +00:00
kompotkot 0d6f6203b8 Fixed mypy 2022-05-16 13:42:24 +00:00
kompotkot 6fba3957c8 Services with access id for node balancer 2022-05-16 13:31:34 +00:00
kompotkot 668c9b6484 Merge branch 'main' into crawlers-access-id 2022-05-16 12:45:21 +00:00
kompotkot 97567e9de9 Auto deploy of moonworm cu mainnet crawler 2022-05-16 12:35:24 +00:00
kompotkot 7a3b7c15d7 Moonworm crawler optimized to work with access id 2022-05-16 11:53:35 +00:00
kompotkot a25e149e1d Merge branch 'main' into crawlers-access-id 2022-05-16 11:20:09 +00:00
Neeraj Kashyap 78da81790c
Merge pull request #585 from bugout-dev/moonworm-job-creation-fix 
fix v2.
 2022-05-05 09:16:55 -07:00
Andrey Dolgolev a9b0831740 Add black formating. 2022-05-05 13:24:00 +03:00
Andrey Dolgolev 5df12db95c Add fix. 2022-05-05 13:12:33 +03:00
Andrey Dolgolev e7f3a8717b Add fixes for mypy. 2022-05-05 12:52:54 +03:00
Andrey Dolgolev b623ff96aa Merge branch 'main' into dashboard-stats-use-replica 2022-05-05 11:42:42 +03:00
Andrey Dolgolev 49f19e5dd7 Clean up code from prints and traceback. 2022-05-05 11:29:48 +03:00
Andrey Dolgolev a1ce774b99 Add worked merged subscription. 2022-05-03 16:53:34 +03:00
kompotkot a0e3165cf2 Sys log identifiers 2022-05-03 12:39:33 +00:00
Andrey Dolgolev 6a7217a70a Init version of merging in stats generation. 2022-05-02 17:49:55 +03:00
Andrey Dolgolev b58cfd34b8 Switch stats generation to use replica. 2022-04-19 17:02:44 +03:00
Yhtyyar Sahatov 330fcceb51 mypy fixes 2022-04-18 23:36:19 +03:00
Yhtyyar Sahatov a76721fdde recuded min block batch and max block batch 2022-04-18 17:54:35 +03:00
Yhtyyar Sahatov dbbe24fb21 revert settings.py 2022-04-18 17:28:12 +03:00
Yhtyyar Sahatov 440fc0fbef fix v2. 2022-04-18 17:25:13 +03:00
Neeraj Kashyap 84cbfcc38a
Revert "checking function abi before creating job" 2022-04-14 16:00:06 -07:00
Yhtyyar Sahatov 22228a7627 checking function abi before creating job 2022-04-14 19:22:49 +03:00
Yhtyyar Sahatov e82e255709 slightly time optimization 2022-03-22 17:26:59 +03:00
Yhtyyar Sahatov ee0f22cff2 more correct logic for even poppulation 2022-03-21 22:16:24 +03:00
Yhtyyar Sahatov ba541b80af generic crawler fix 2022-03-21 22:05:02 +03:00
kompotkot 6c1143de8e updated txpool services and sample env 2022-03-18 19:19:03 +00:00
kompotkot 631555ab94 specified header for rpc dial if access id is set 2022-03-18 19:12:07 +00:00
Yhtyyar Sahatov 87ccc82049 update erc20 populate 2022-03-18 20:41:28 +03:00
kompotkot d76098718b content type header for rpc connect 2022-03-18 15:23:26 +00:00
kompotkot 3a1de527a4 access id default from env variable for all mooncrawl crawlers 2022-03-17 20:07:28 +00:00
kompotkot 9435568c18 crawler work with access id 2022-03-17 19:19:48 +00:00
Sergei Sumarokov 665a857b29
Merge pull request #557 from bugout-dev/ldb-crawler 2022-03-11 19:37:07 +03:00
Yhtyyar Sahatov 3a8d1bbd70 added continuous crawling 2022-03-10 18:25:21 +03:00
Yhtyyar Sahatov c1729dd836 added generic crawler 2022-03-09 22:35:27 +03:00
kompotkot d375438668 Query validator with tests 2022-03-09 15:51:49 +00:00
kompotkot b408c20b0b Added missed env variables in sample.env 2022-03-09 12:44:54 +00:00
kompotkot 29e84431a6 Bumped db version, extended sample.end, docker config and mypy fix 2022-03-09 12:08:31 +00:00
kompotkot cf14df0333 Merge branch 'main' into add_sql_execution_endpoint 2022-03-09 11:47:34 +00:00
Andrey Dolgolev 2077ea2c81 Add black formating. 2022-03-08 17:10:06 +02:00
Andrey Dolgolev 542f411449 Add using read only user. 2022-03-08 17:06:50 +02:00
Andrey Dolgolev 3c59048459 Fix issue with items. 2022-03-08 16:00:39 +02:00
Andrey Dolgolev f5b2550505 Add fixes. 2022-03-08 15:18:27 +02:00
kompotkot 1473ca3b40 Fixes 2022-03-06 23:33:28 +00:00
kompotkot d55209df42 Code structure organization 2022-03-06 23:21:24 +00:00
kompotkot 838c75ae29 Verification command in goroutines 2022-03-06 22:59:47 +00:00
kompotkot 39c391dfa0 Final 1 thread version 2022-03-06 19:04:01 +00:00
kompotkot c33d033ad9 Smart verification humbug report counter 2022-03-06 18:51:36 +00:00
kompotkot cb8bb515aa Blocks range generator 2022-03-06 18:10:55 +00:00
kompotkot 392bdf801d Insert block 2022-03-06 00:11:13 +00:00
kompotkot e8ae125dd1 Blockchain flag 2022-03-05 19:49:53 +00:00
kompotkot 4085d11226 Respect geth structure due work, optimised sql query to db 2022-03-05 19:13:42 +00:00
kompotkot b7578838c8 Parse uint fix 2022-03-03 21:32:34 +00:00
kompotkot 1642ed9e0a Functionality to report at humbug journal with corrupt blocks 2022-03-03 21:05:33 +00:00
kompotkot 45cc528582 ldb blockchain database crawler 2022-03-03 20:11:40 +00:00
Yhtyyar Sahatov 301f3bb745 added value when saving 2022-02-24 18:32:16 +03:00
Yhtyyar Sahatov 07efbd44fd fucking breakpoint 2022-02-24 18:30:44 +03:00
Yhtyyar Sahatov 14ddc78bac added values to tx 2022-02-24 18:27:57 +03:00
Yhtyyar Sahatov 3cba4d3fb3 fixed sleep time bug 2022-02-22 18:46:17 +03:00
Yhtyyar Sahatov 0bab8d30dd tx value 2022-02-22 18:37:39 +03:00
Andrey Dolgolev d031a71013 Add temp solution for show block and timestamp. 2022-02-17 17:26:48 +02:00
Andrey Dolgolev 1498d1bac6 remove testing. 2022-02-17 12:24:14 +02:00
Andrey Dolgolev 1c741d5084 Add create update get endpoints. 2022-02-17 12:22:20 +02:00
Yhtyyar Sahatov 169dc78563 working crawler 2022-02-17 00:58:37 +03:00
Andrey Dolgolev 4dd101eb43 Add lint fixes. 2022-02-16 18:23:29 +02:00
Andrey Dolgolev 025baeb1c4 Add test workable version. 2022-02-16 18:12:42 +02:00
Andrey Dolgolev 5485a73b98 Add black formating. 2022-02-16 03:00:49 +02:00
Andrey Dolgolev 373f768a3f Quries worker. 2022-02-16 02:57:39 +02:00
Andrey Dolgolev 0cbd177442 Add fixes. 2022-02-16 02:55:14 +02:00
Andrey Dolgolev c41f8d1616 black formating crawlers. 2022-02-16 02:49:23 +02:00
Andrey Dolgolev 3d1108e569 Add json response from drones. 2022-02-16 02:39:49 +02:00
Andrey Dolgolev bc45e5cc76 Add init queries drone api. 2022-02-16 02:29:12 +02:00
kompotkot 3798f39946 In deploy scripts point to go root dir $HOME/ -> /root/ 2022-02-08 08:41:13 +00:00
kompotkot 1d54c8758b Dasboard updated modified from 15m -> 6h 2022-01-31 13:05:33 +00:00
Sergei Sumarokov 7e0c7b52ba
Revert "Client id to work with node balancer" 2022-01-24 20:35:46 +03:00
kompotkot 89caa99213 Services for crawlers with client id 2022-01-24 17:17:44 +00:00
kompotkot deabed9837 Token->client_id 2022-01-24 17:16:58 +00:00
kompotkot c8b458bf8b CLI client token arg for mooncrawl 2022-01-24 14:00:49 +00:00
kompotkot 4830d28a55 Sync crawlers restart automaticly with systemd 2022-01-21 13:13:48 +00:00
kompotkot 30c5f6da9b Ethereum sync service in multi threads 2022-01-20 11:38:58 +00:00
Andrey Dolgolev 4206012192 Add fix of keys. 2022-01-19 13:25:32 +02:00
Andrey Dolgolev 8f2d494039 Add fix. 2022-01-19 01:21:24 +02:00
Andrey Dolgolev 9510cecc17 Remove address from special rule. 2022-01-18 18:36:46 +02:00
kompotkot bb30fe44c1 Work with env var port for mooncrawl service 2022-01-18 09:22:18 +00:00
Andrey Dolgolev 7947d6f3f7 Increase service timer. 2022-01-18 11:18:40 +02:00
Andrey Dolgolev 261065f91a Add function for extention data which go to cards componener. 2022-01-17 14:44:58 +02:00
Andrey Dolgolev 6dfbb986a0 Fix. 2022-01-17 10:46:29 +02:00
Andrey Dolgolev 2ddb6a49c7 Romove unused variables. 2022-01-17 10:32:32 +02:00
Andrey Dolgolev 04f7e10cd0 Add fix. 2022-01-14 12:01:57 +02:00
Andrey Dolgolev d221ffd6be Merge branch 'main' into refresh-dashboard-stats 2022-01-14 12:00:27 +02:00
kompotkot 606b965cd4 --no-block for statistic timer crawler 2022-01-13 18:01:18 +00:00
Andrey Dolgolev abb5415c8e Add fix. 2022-01-13 20:01:05 +02:00
Andrey Dolgolev d3c8b85cc8 Add mypy ignore. 2022-01-13 19:26:39 +02:00
Andrey Dolgolev e0b5b00eab Remove and test generation. 2022-01-13 17:31:25 +02:00
Andrey Dolgolev a1678aa08c Add logger instead prints. 2022-01-13 12:29:36 +02:00
Andrey Dolgolev 40e6e6c99b Add rollback. 2022-01-13 00:36:21 +02:00
Andrey Dolgolev 098c4de6e8 Add fixes. 2022-01-12 10:50:53 +02:00
Andrey Dolgolev 15f19583fc Merge branch 'main' into refresh-dashboard-stats 2022-01-10 12:52:30 +02:00
Andrey Dolgolev ef064827fd Add correct refreshing status. 2022-01-10 12:46:46 +02:00
Sergei Sumarokov 2419d95f5b
Merge pull request #512 from bugout-dev/mooncrawl-service-api 
Moonstream crawlers service for HTTP API server
 2021-12-23 17:41:32 +03:00
Yhtyyar Sahatov 2e297aa299
Merge pull request #510 from bugout-dev/continuous-crawler-batch-saving 
Continuous crawler batch saving
 2021-12-23 15:31:53 +03:00
Andrey Dolgolev 95b6ccdb4e All updates is working but require extend presignUrl responce. 2021-12-22 18:55:42 +02:00
kompotkot 6fd9319fd3 Moonstream crawlers service for HTTP API server 2021-12-22 15:14:50 +00:00
kompotkot 0c1e67a45b Mypy fix 2021-12-22 12:36:06 +00:00
kompotkot 6529e19bc6 Retrieve local ip for crawlers 2021-12-22 10:57:11 +00:00
kompotkot 64e0bf2645 Mooncrawl ready to go with WEB3_URI instread of ips and ports 2021-12-22 09:34:18 +00:00
kompotkot 4989e3afc1 Txpool now works with WEB3_URI env variable 2021-12-22 09:23:23 +00:00
kompotkot 0016fbf8ed Modified deploy for crawlers to work with checkenv 2021-12-22 09:15:18 +00:00
Andrey Dolgolev 968a1ddf45 Merge branch 'main' into refresh-dashboard-stats 2021-12-21 17:37:57 +02:00
yhtiyar 874b330846 bump of version 2021-12-21 18:35:04 +03:00
yhtiyar 6be687b1ed update of continuous crawler to batch save session 2021-12-21 18:33:31 +03:00
yhtiyar 3228e5e740 update of moonworm 2021-12-21 18:32:38 +03:00
Andrey Dolgolev d5b997e935 pre init 2021-12-21 17:23:26 +02:00
Neeraj Kashyap 6f2813710a
Merge pull request #502 from bugout-dev/job-pickup-fix 
fixed deduplication of abi
 2021-12-16 15:39:53 -08:00
yhtiyar aa112952c0 fixed deduplication of abi 2021-12-17 02:33:00 +03:00
Neeraj Kashyap 954f6014f7
Merge pull request #501 from bugout-dev/job-pickup-fix 
fixed job pickup
 2021-12-16 15:09:32 -08:00
yhtiyar 3a22e60226 fixed job pickup 2021-12-17 02:04:25 +03:00
kompotkot ddf33b9c92 Polygon syncronize crawler works in multi threads 2021-12-16 22:33:08 +00:00
kompotkot b1e828cf8e Merge branch 'moonworm-crawler' of github.com:bugout-dev/moonstream into moonworm-crawler 2021-12-16 21:49:20 +00:00
kompotkot 471a81277a Fixed moonworm service entrypoint 2021-12-16 21:48:47 +00:00
yhtiyar b8b8e01787 crawler begins from 1000 behind 2021-12-17 00:48:36 +03:00
yhtiyar 16a272ab71 black fix 2021-12-17 00:44:31 +03:00
yhtiyar 8e8c1c1569 Merge branch 'moonworm-crawler' of github.com:bugout-dev/moonstream into moonworm-crawler 2021-12-17 00:39:58 +03:00
yhtiyar c9ea6b85ff working crawler 2021-12-17 00:39:35 +03:00
kompotkot 364775dc94 Version, deployment service, setup requirements update for moonworm 2021-12-16 21:32:17 +00:00
yhtiyar 8a7413500c Merge branch 'moonworm-crawler' of github.com:bugout-dev/moonstream into moonworm-crawler 2021-12-16 23:18:14 +03:00
yhtiyar 034c20c913 added --force arg 2021-12-16 23:17:37 +03:00
yhtiyar ee91fadf40 fixed cli 2021-12-16 23:13:00 +03:00
yhtiyar 5df739b00e fixed import 2021-12-16 23:12:39 +03:00
Andrey Dolgolev ccc5cf42c9 fix label 2021-12-16 21:51:23 +02:00
yhtiyar 49454a87cc isort and mypy fixes 2021-12-16 22:46:44 +03:00
yhtiyar 83eaeebe1a [WIP] crawler 2021-12-16 22:43:32 +03:00
Andrey Dolgolev 489bb66e57 Merge branch 'main' into moonworm-crawler 2021-12-16 21:34:05 +02:00
Andrey Dolgolev ee476c5b46 Add fixes. 2021-12-16 21:25:24 +02:00
Andrey Dolgolev 3c79aaf23a Working state. 2021-12-16 19:17:13 +02:00
Tim Pechersky fd8ee961f1 Merge branch 'main' into customized-dashboard 2021-12-16 16:37:57 +00:00
Andrey Dolgolev 4d4032e274 Revert changes abi processing. 2021-12-16 17:15:23 +02:00
Andrey Dolgolev 9783690f8b isort and init state of api tasks. 2021-12-16 15:26:04 +02:00
yhtiyar 6d87050067 mixed continuous crawler 2021-12-14 18:39:04 +03:00
Andrey Dolgolev 2aad256de2 Now both subscription types blockchain and smartcontract support dashboard generation. 2021-12-12 15:41:15 +02:00
Andrey Dolgolev 3f48463330 Apply fixes. 2021-12-11 17:26:56 +02:00
Andrey Dolgolev 7e651795ac Swich keys subscription type for stats worker. 2021-12-09 21:00:34 +02:00
Andrey Dolgolev 57e766501f Remove reading methods from abi. 2021-12-09 20:20:36 +02:00
Andrey Dolgolev 099426bd68 Merge branch 'main' into customized-dashboard 2021-12-09 19:23:25 +02:00
yhtiyar cc5532370a small fixes 2021-12-09 20:05:21 +03:00
yhtiyar 11acf6540e working version of event crawler 2021-12-09 15:56:38 +03:00
yhtiyar abe981beb9 fix type: abi_metod_hash -> abi_method_hash 2021-12-09 15:08:17 +03:00
yhtiyar f4f7c4b393 Merge branch 'Add-moonworm-tasks-to-journal' into moonworm-crawler 2021-12-07 17:41:49 +03:00
yhtiyar 20291871a5 WIP moonworm crawler 2021-12-07 17:39:33 +03:00
Andrey Dolgolev e6abbfbb60 Add fixes. 2021-12-07 13:45:24 +02:00
Andrey Dolgolev f154dd15aa Change keys and variables names. 2021-12-06 22:09:25 +02:00
Andrey Dolgolev 0353c49035 Add type annotationю 2021-12-05 14:25:03 +02:00
Andrey Dolgolev da2540c2ed Add required fixes. 2021-12-05 14:11:51 +02:00
Andrey Dolgolev e6dc7bf0b3 Add init version. 2021-12-04 18:34:29 +02:00
Andrey Dolgolev a3bce62707 Merge branch 'main' into fix-dashbord-worker-filters 2021-12-02 22:03:37 +02:00
Andrey Dolgolev e7a6c00371 Fix dashboard_subscriptions to subscription_settings. 2021-12-02 22:02:03 +02:00
Andrey Dolgolev 0338530e9c Add fixes. 2021-12-02 15:22:16 +02:00
kompotkot 3458186a1e Organized sample, renamed s3 variable and web3_provider env changed 
Func in web3_provider checks if env variable starts with http then
it use it, otherwise it call route53 to fetch ip address of node.
 2021-11-30 14:23:13 +00:00
kompotkot 7305b38243 Small fixes and large pack pack temp solution 2021-11-25 15:49:07 +00:00
Sergei Sumarokov 8f459fbee9
Merge pull request #460 from bugout-dev/txpool-polygon 
Txpool working with both blockchains
 2021-11-25 16:14:23 +03:00
Andrey Dolgolev 2b44b2180f Remove empty grafs generations. 2021-11-25 15:09:01 +02:00
Andrey Dolgolev f589fc9c21 Fix import. 2021-11-25 14:29:57 +02:00
Andrey Dolgolev 5a6b7bd998 Add reporter. 2021-11-25 14:26:19 +02:00
Andrey Dolgolev 0916610f9a Add fixes. 2021-11-25 13:23:34 +02:00
kompotkot 5063ee4e8b Commit for now polygon txpool crawler in deploy 2021-11-24 17:18:14 +00:00
kompotkot e6f650624b Txpool working with both blockchains 2021-11-24 17:16:19 +00:00
Andrey Dolgolev d81026df03 Merge branch 'main' into dashboards-worker-apply-names-filter 2021-11-24 16:06:11 +02:00
Andrey Dolgolev 6c3a0e2c65 Remove comments. 2021-11-24 13:52:12 +02:00
Andrey Dolgolev 6e3c09e2bb Remove imports. 2021-11-23 20:26:14 +02:00
Andrey Dolgolev d4731771f8 Add custom tracking of hatched events. 2021-11-23 20:23:33 +02:00
Andrey Dolgolev 74a8f97fcc Remove filterout labels by dates. 2021-11-23 17:22:47 +02:00
Andrey Dolgolev 41285d5a78 Remove already processed check. 2021-11-23 17:18:11 +02:00
Andrey Dolgolev 20f02899fa Add fixes. 2021-11-23 17:16:18 +02:00
Andrey Dolgolev 4f3326c05b Add apply of filters wich in general depends on names of dashboard subscriptions. 
Reformat code for get support of subscription wich not contain abi and only are generic methods.
 2021-11-23 16:59:52 +02:00
Andrey Dolgolev b548a5d19c Add correct datetime formating on dashboard worker. 2021-11-16 00:15:42 +02:00
Andrey Dolgolev f4e3c0addd Fix missing import. 2021-11-15 20:46:49 +02:00
Andrey Dolgolev ee5ee33285 Add unique users metrics. 2021-11-15 20:41:18 +02:00
Andrey Dolgolev 8f6e7ea01f Remove debuging prints. 2021-11-15 18:39:33 +02:00
yhtiyar 2bc565cebd fixed func_abi 2021-11-15 18:14:28 +03:00
yhtiyar 10a7ab8748 fixed func_abi 2021-11-15 18:07:28 +03:00
yhtiyar 44cc1562bb fixed func_abi 2021-11-15 18:04:56 +03:00
yhtiyar 845b768c45 Merge branch 'Extentions-metrics' of github.com:bugout-dev/moonstream into Extentions-metrics 2021-11-15 18:01:18 +03:00
yhtiyar edcde4f930 fix of comment 2021-11-15 18:01:14 +03:00
Andrey Dolgolev 8bab2340c6 Fixes 2021-11-15 17:00:29 +02:00
yhtiyar 14f0f572d7 added calls to web3 2021-11-15 17:29:19 +03:00
Andrey Dolgolev 1213ad874d Ad init state. 2021-11-15 15:42:20 +02:00
kompotkot b67d4563df Fixed typo in deploy 2021-11-15 11:00:42 +00:00
kompotkot 2f6f63ac52 Increased confirmations for polygon sync crawler 2021-11-15 10:52:33 +00:00
Sergei Sumarokov b9cbae95bc
Merge pull request #323 from bugout-dev/ABI-Defined-Dashboars 
ABI Defined dashboards
 2021-11-15 10:35:12 +00:00
Andrey Dolgolev ea0a81476f Add blockchain name to s3 path for contract abi. 2021-11-15 12:27:45 +02:00
Andrey Dolgolev ea4a2685b5 Add fix for resolve blockchain on s3 path in contracts_data. 2021-11-15 11:53:14 +02:00
kompotkot dfe0d8e5e8 Service file with timer at deploy script for statistics 2021-11-14 18:49:15 +00:00
kompotkot 27b693daa3 Service files for stats_worker.dashboard generate command 2021-11-14 18:44:58 +00:00
Andrey Dolgolev 03df4853dd Add remove enviroment vriable and swich month to hors resolution. 2021-11-14 19:18:30 +02:00
Andrey Dolgolev 1e74b4f77e Add functions calls. 2021-11-14 16:25:19 +02:00
Andrey Dolgolev 729e7faa01 Change queries to database. 2021-11-14 16:15:07 +02:00
Andrey Dolgolev 394f9217e1 Add bucket routs fixes. 2021-11-14 14:58:15 +02:00
Andrey Dolgolev bece7be92e Add filter for blockchains by subscription_id. Add polygon subscription. 2021-11-14 13:18:52 +02:00
kompotkot 4a7fea6acf Confirmations for missing blocks 150 2021-11-13 18:57:20 +00:00
kompotkot c88f972870 Removed api s3 bucket link and prefix 2021-11-13 16:47:38 +00:00
kompotkot b8ce0b7cc3 For polygon use 40 confirmations 2021-11-13 16:39:08 +00:00
kompotkot 08d7991056 dev.sh file for dev server api testing 2021-11-13 15:54:37 +00:00
kompotkot 11aa45690f API endpoint for crawlers 2021-11-13 15:51:32 +00:00
kompotkot 42e764d1b2 Removed duplication 2021-11-13 15:32:11 +00:00
kompotkot 02699c0bc0 Merge branch 'stats_data_worker' of github.com:bugout-dev/moonstream into stats_data_worker 2021-11-13 15:27:50 +00:00
kompotkot bf6a94a17e Optimized to work with both blockchains, added cli to setup 2021-11-13 15:21:41 +00:00
Andrey Dolgolev 42d69ca7c6 Add cli. 2021-11-13 16:49:22 +02:00
kompotkot 77dd49405d Merge branch 'main' into stats_data_worker 2021-11-13 14:49:15 +00:00
Andrey Dolgolev f9b5eb57b1 Delete testing block. 2021-11-13 16:45:09 +02:00
Andrey Dolgolev 46ab3d72b0 Fix typo. 2021-11-13 16:44:08 +02:00
Andrey Dolgolev 8c54d9b4fc Rename file. 2021-11-13 16:34:31 +02:00
Andrey Dolgolev d87054773d Add fixes. 2021-11-13 16:32:55 +02:00
Andrey Dolgolev 362b266496 Add metrics timeseries. 2021-11-13 16:30:13 +02:00
kompotkot 326b1615b5 Merge branch 'main' into nodes-deployment-update 2021-11-13 09:02:52 +00:00
kompotkot 0e208c97d5 Moved crawlers status server, now it is node server 2021-11-13 09:01:15 +00:00
kompotkot 1756b24bfb Blocks missing instead of missing 2021-11-13 08:09:11 +00:00
kompotkot 172a7557c2 Crawlers status server now nodes status server 2021-11-12 18:24:36 +00:00
kompotkot 1b7ac6b263 Service and timer files 2021-11-12 18:13:01 +00:00
kompotkot 273e6f0ef9 Polygon service and timer variables 2021-11-12 18:12:07 +00:00
kompotkot 35b899bf38 Missing block finder automatization 2021-11-12 18:09:55 +00:00
Andrey Dolgolev e7fe8f9003 Add worker code. 2021-11-11 17:16:21 +02:00
kompotkot b6ab715be1 Removed specific versions at setup.py 2021-11-10 18:10:38 +00:00
kompotkot 29584107a6 Added lost package chardet 2021-11-10 15:05:58 +00:00
kompotkot 034236ce1e Renamed backend moonstream -> moonstreamapi 2021-11-10 14:17:36 +00:00
kompotkot dd8e8d3db2 Removed requirements.txt and used setup.py instead 2021-11-10 14:05:13 +00:00
kompotkot 683ca3b2a9 Fixed typo in polygon crawlers 2021-11-09 17:41:33 +00:00
kompotkot 4b9018f6de Fixed typo and extra_data statement 2021-11-09 16:48:12 +00:00
kompotkot 3ce14460e3 Polygon synchronize service file 2021-11-09 13:56:25 +00:00
kompotkot 381a4cef40 Blockchain model dynamic chose for crawlers 2021-11-09 13:46:33 +00:00
kompotkot 526aa4635e Fixed deploy system files with new flag for ethereum 2021-11-09 11:25:52 +00:00
kompotkot baea584a9e Tested with ethereum crawlers 2021-11-09 11:14:03 +00:00
kompotkot 3be05c454b Removed wrong isort config file 2021-11-09 10:23:23 +00:00
kompotkot e80c211748 try except for init py to ignore humbug import 2021-11-09 10:22:13 +00:00
kompotkot b4809ce825 Merge branch 'main' into polygon-crawlers 
And updated version for mooncrawl
 2021-11-08 17:44:47 +00:00
kompotkot 725a1054a4 Crawler can be executed with flag -t to chose what type of blockchain to use 2021-11-08 15:08:57 +00:00
kompotkot 8aa1734bdc Updated hash to work with polygon models, env sample and entrypoint 2021-11-08 12:57:16 +00:00
yhtiyar 5610deb367 black formatting 2021-11-07 18:30:21 +03:00
yhtiyar 8320d6e78f run isort 2021-11-07 17:17:11 +03:00