mirror
/
moonstream
kopia lustrzana https://github.com/bugout-dev/moonstream
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność
3 462 Commity
121 Gałęzie
76 Tagi
29 MiB
Wykres commitów

872 Commity (add-address-selector-index)

Autor SHA1 Wiadomość Data
yhtiyar 6d87050067 mixed continuous crawler 2021-12-14 18:39:04 +03:00
Andrey Dolgolev 2aad256de2 Now both subscription types blockchain and smartcontract support dashboard generation. 2021-12-12 15:41:15 +02:00
Andrey Dolgolev 3f48463330 Apply fixes. 2021-12-11 17:26:56 +02:00
Andrey Dolgolev 7e651795ac Swich keys subscription type for stats worker. 2021-12-09 21:00:34 +02:00
Andrey Dolgolev 57e766501f Remove reading methods from abi. 2021-12-09 20:20:36 +02:00
Andrey Dolgolev 099426bd68 Merge branch 'main' into customized-dashboard 2021-12-09 19:23:25 +02:00
yhtiyar cc5532370a small fixes 2021-12-09 20:05:21 +03:00
yhtiyar 11acf6540e working version of event crawler 2021-12-09 15:56:38 +03:00
yhtiyar abe981beb9 fix type: abi_metod_hash -> abi_method_hash 2021-12-09 15:08:17 +03:00
yhtiyar f4f7c4b393 Merge branch 'Add-moonworm-tasks-to-journal' into moonworm-crawler 2021-12-07 17:41:49 +03:00
yhtiyar 20291871a5 WIP moonworm crawler 2021-12-07 17:39:33 +03:00
Andrey Dolgolev e6abbfbb60 Add fixes. 2021-12-07 13:45:24 +02:00
Andrey Dolgolev f154dd15aa Change keys and variables names. 2021-12-06 22:09:25 +02:00
Andrey Dolgolev 0353c49035 Add type annotationю 2021-12-05 14:25:03 +02:00
Andrey Dolgolev da2540c2ed Add required fixes. 2021-12-05 14:11:51 +02:00
Andrey Dolgolev e6dc7bf0b3 Add init version. 2021-12-04 18:34:29 +02:00
Andrey Dolgolev a3bce62707 Merge branch 'main' into fix-dashbord-worker-filters 2021-12-02 22:03:37 +02:00
Andrey Dolgolev e7a6c00371 Fix dashboard_subscriptions to subscription_settings. 2021-12-02 22:02:03 +02:00
Andrey Dolgolev 0338530e9c Add fixes. 2021-12-02 15:22:16 +02:00
kompotkot 3458186a1e Organized sample, renamed s3 variable and web3_provider env changed 
Func in web3_provider checks if env variable starts with http then
it use it, otherwise it call route53 to fetch ip address of node.
 2021-11-30 14:23:13 +00:00
kompotkot 7305b38243 Small fixes and large pack pack temp solution 2021-11-25 15:49:07 +00:00
Sergei Sumarokov 8f459fbee9
Merge pull request #460 from bugout-dev/txpool-polygon 
Txpool working with both blockchains
 2021-11-25 16:14:23 +03:00
Andrey Dolgolev 2b44b2180f Remove empty grafs generations. 2021-11-25 15:09:01 +02:00
Andrey Dolgolev f589fc9c21 Fix import. 2021-11-25 14:29:57 +02:00
Andrey Dolgolev 5a6b7bd998 Add reporter. 2021-11-25 14:26:19 +02:00
Andrey Dolgolev 0916610f9a Add fixes. 2021-11-25 13:23:34 +02:00
kompotkot 5063ee4e8b Commit for now polygon txpool crawler in deploy 2021-11-24 17:18:14 +00:00
kompotkot e6f650624b Txpool working with both blockchains 2021-11-24 17:16:19 +00:00
Andrey Dolgolev d81026df03 Merge branch 'main' into dashboards-worker-apply-names-filter 2021-11-24 16:06:11 +02:00
Andrey Dolgolev 6c3a0e2c65 Remove comments. 2021-11-24 13:52:12 +02:00
Andrey Dolgolev 6e3c09e2bb Remove imports. 2021-11-23 20:26:14 +02:00
Andrey Dolgolev d4731771f8 Add custom tracking of hatched events. 2021-11-23 20:23:33 +02:00
Andrey Dolgolev 74a8f97fcc Remove filterout labels by dates. 2021-11-23 17:22:47 +02:00
Andrey Dolgolev 41285d5a78 Remove already processed check. 2021-11-23 17:18:11 +02:00
Andrey Dolgolev 20f02899fa Add fixes. 2021-11-23 17:16:18 +02:00
Andrey Dolgolev 4f3326c05b Add apply of filters wich in general depends on names of dashboard subscriptions. 
Reformat code for get support of subscription wich not contain abi and only are generic methods.
 2021-11-23 16:59:52 +02:00
Andrey Dolgolev b548a5d19c Add correct datetime formating on dashboard worker. 2021-11-16 00:15:42 +02:00
Andrey Dolgolev f4e3c0addd Fix missing import. 2021-11-15 20:46:49 +02:00
Andrey Dolgolev ee5ee33285 Add unique users metrics. 2021-11-15 20:41:18 +02:00
Andrey Dolgolev 8f6e7ea01f Remove debuging prints. 2021-11-15 18:39:33 +02:00
yhtiyar 2bc565cebd fixed func_abi 2021-11-15 18:14:28 +03:00
yhtiyar 10a7ab8748 fixed func_abi 2021-11-15 18:07:28 +03:00
yhtiyar 44cc1562bb fixed func_abi 2021-11-15 18:04:56 +03:00
yhtiyar 845b768c45 Merge branch 'Extentions-metrics' of github.com:bugout-dev/moonstream into Extentions-metrics 2021-11-15 18:01:18 +03:00
yhtiyar edcde4f930 fix of comment 2021-11-15 18:01:14 +03:00
Andrey Dolgolev 8bab2340c6 Fixes 2021-11-15 17:00:29 +02:00
yhtiyar 14f0f572d7 added calls to web3 2021-11-15 17:29:19 +03:00
Andrey Dolgolev 1213ad874d Ad init state. 2021-11-15 15:42:20 +02:00
kompotkot b67d4563df Fixed typo in deploy 2021-11-15 11:00:42 +00:00
kompotkot 2f6f63ac52 Increased confirmations for polygon sync crawler 2021-11-15 10:52:33 +00:00
Sergei Sumarokov b9cbae95bc
Merge pull request #323 from bugout-dev/ABI-Defined-Dashboars 
ABI Defined dashboards
 2021-11-15 10:35:12 +00:00
Andrey Dolgolev ea0a81476f Add blockchain name to s3 path for contract abi. 2021-11-15 12:27:45 +02:00
Andrey Dolgolev ea4a2685b5 Add fix for resolve blockchain on s3 path in contracts_data. 2021-11-15 11:53:14 +02:00
kompotkot dfe0d8e5e8 Service file with timer at deploy script for statistics 2021-11-14 18:49:15 +00:00
kompotkot 27b693daa3 Service files for stats_worker.dashboard generate command 2021-11-14 18:44:58 +00:00
Andrey Dolgolev 03df4853dd Add remove enviroment vriable and swich month to hors resolution. 2021-11-14 19:18:30 +02:00
Andrey Dolgolev 1e74b4f77e Add functions calls. 2021-11-14 16:25:19 +02:00
Andrey Dolgolev 729e7faa01 Change queries to database. 2021-11-14 16:15:07 +02:00
Andrey Dolgolev 394f9217e1 Add bucket routs fixes. 2021-11-14 14:58:15 +02:00
Andrey Dolgolev bece7be92e Add filter for blockchains by subscription_id. Add polygon subscription. 2021-11-14 13:18:52 +02:00
kompotkot 4a7fea6acf Confirmations for missing blocks 150 2021-11-13 18:57:20 +00:00
kompotkot c88f972870 Removed api s3 bucket link and prefix 2021-11-13 16:47:38 +00:00
kompotkot b8ce0b7cc3 For polygon use 40 confirmations 2021-11-13 16:39:08 +00:00
kompotkot 08d7991056 dev.sh file for dev server api testing 2021-11-13 15:54:37 +00:00
kompotkot 11aa45690f API endpoint for crawlers 2021-11-13 15:51:32 +00:00
kompotkot 42e764d1b2 Removed duplication 2021-11-13 15:32:11 +00:00
kompotkot 02699c0bc0 Merge branch 'stats_data_worker' of github.com:bugout-dev/moonstream into stats_data_worker 2021-11-13 15:27:50 +00:00
kompotkot bf6a94a17e Optimized to work with both blockchains, added cli to setup 2021-11-13 15:21:41 +00:00
Andrey Dolgolev 42d69ca7c6 Add cli. 2021-11-13 16:49:22 +02:00
kompotkot 77dd49405d Merge branch 'main' into stats_data_worker 2021-11-13 14:49:15 +00:00
Andrey Dolgolev f9b5eb57b1 Delete testing block. 2021-11-13 16:45:09 +02:00
Andrey Dolgolev 46ab3d72b0 Fix typo. 2021-11-13 16:44:08 +02:00
Andrey Dolgolev 8c54d9b4fc Rename file. 2021-11-13 16:34:31 +02:00
Andrey Dolgolev d87054773d Add fixes. 2021-11-13 16:32:55 +02:00
Andrey Dolgolev 362b266496 Add metrics timeseries. 2021-11-13 16:30:13 +02:00
kompotkot 326b1615b5 Merge branch 'main' into nodes-deployment-update 2021-11-13 09:02:52 +00:00
kompotkot 0e208c97d5 Moved crawlers status server, now it is node server 2021-11-13 09:01:15 +00:00
kompotkot 1756b24bfb Blocks missing instead of missing 2021-11-13 08:09:11 +00:00
kompotkot 172a7557c2 Crawlers status server now nodes status server 2021-11-12 18:24:36 +00:00
kompotkot 1b7ac6b263 Service and timer files 2021-11-12 18:13:01 +00:00
kompotkot 273e6f0ef9 Polygon service and timer variables 2021-11-12 18:12:07 +00:00
kompotkot 35b899bf38 Missing block finder automatization 2021-11-12 18:09:55 +00:00
Andrey Dolgolev e7fe8f9003 Add worker code. 2021-11-11 17:16:21 +02:00
kompotkot b6ab715be1 Removed specific versions at setup.py 2021-11-10 18:10:38 +00:00
kompotkot 29584107a6 Added lost package chardet 2021-11-10 15:05:58 +00:00
kompotkot 034236ce1e Renamed backend moonstream -> moonstreamapi 2021-11-10 14:17:36 +00:00
kompotkot dd8e8d3db2 Removed requirements.txt and used setup.py instead 2021-11-10 14:05:13 +00:00
kompotkot 683ca3b2a9 Fixed typo in polygon crawlers 2021-11-09 17:41:33 +00:00
kompotkot 4b9018f6de Fixed typo and extra_data statement 2021-11-09 16:48:12 +00:00
kompotkot 3ce14460e3 Polygon synchronize service file 2021-11-09 13:56:25 +00:00
kompotkot 381a4cef40 Blockchain model dynamic chose for crawlers 2021-11-09 13:46:33 +00:00
kompotkot 526aa4635e Fixed deploy system files with new flag for ethereum 2021-11-09 11:25:52 +00:00
kompotkot baea584a9e Tested with ethereum crawlers 2021-11-09 11:14:03 +00:00
kompotkot 3be05c454b Removed wrong isort config file 2021-11-09 10:23:23 +00:00
kompotkot e80c211748 try except for init py to ignore humbug import 2021-11-09 10:22:13 +00:00
kompotkot b4809ce825 Merge branch 'main' into polygon-crawlers 
And updated version for mooncrawl
 2021-11-08 17:44:47 +00:00
kompotkot 725a1054a4 Crawler can be executed with flag -t to chose what type of blockchain to use 2021-11-08 15:08:57 +00:00
kompotkot 8aa1734bdc Updated hash to work with polygon models, env sample and entrypoint 2021-11-08 12:57:16 +00:00
yhtiyar 5610deb367 black formatting 2021-11-07 18:30:21 +03:00
yhtiyar 8320d6e78f run isort 2021-11-07 17:17:11 +03:00
yhtiyar 2e1e33c624 changed sleep default time 2021-11-07 17:15:05 +03:00
yhtiyar 5187d30257 fixes after review 2021-11-07 17:04:58 +03:00
yhtiyar 5ed9772852 working version of contract deploy crawler 2021-11-07 01:27:26 +03:00
kompotkot 28f16438cc Removed unnecessary flag at service crawler 2021-11-04 13:42:54 +00:00
kompotkot 0df96e053c Go fmt 2021-11-04 12:55:54 +00:00
kompotkot de922e8e83 Work with checkenv repo for crawlers deployment 2021-11-04 12:50:36 +00:00
kompotkot 40ffdd6ee5 Working go status servers 2021-11-03 11:18:49 +00:00
kompotkot e8e0f480ac Crawlers work with transaction type column 2021-10-28 13:14:48 +00:00
kompotkot 7cf1ef411f Block and tx crawlers works with new fields 2021-10-26 12:02:55 +00:00
Andrey Dolgolev 5b2de0d20c Remove import EthereumAddresses from etherscan and fix join on nft derive. 2021-10-21 11:39:16 +03:00
Andrey Dolgolev 689a8f2c31 Remove imports. 2021-10-20 20:50:27 +03:00
Andrey Dolgolev a4d103dc44 Change commit hash. 2021-10-20 20:27:58 +03:00
Neeraj Kashyap 6b5b6049b5 Merge branch 'main' into modify-label-table 2021-10-20 09:36:05 -07:00
Andrey Dolgolev 90cd175721 Remove all EthereumAddresses usage from cli and some imports. 2021-10-20 17:16:46 +03:00
Andrey Dolgolev 2f4c995687 Add fixes. 2021-10-19 23:02:10 +03:00
Sergei Sumarokov a7e172f8fe
Merge pull request #315 from bugout-dev/timeout-node-shutdown 
Timeout for node service shutdown
 2021-10-15 13:49:48 +03:00
kompotkot 68357ec2c4 Typo fix 2021-10-15 09:36:22 +00:00
kompotkot 55c5e89bbe Logic if block difference to large to catch up with crawler 2021-10-15 09:33:56 +00:00
kompotkot 31d9ff6f48 Fix sync blocks logic 2021-10-15 09:03:07 +00:00
kompotkot f795306afb Nodes deployment module 2021-10-15 08:38:28 +00:00
Andrey Dolgolev cfaa72e043 Add fixes. 2021-10-14 18:06:35 +03:00
Andrey Dolgolev 6890d38097 Add changes for labels table. 2021-10-14 18:01:00 +03:00
kompotkot 9e9462c519 x3 glob queue txpool value for geth 2021-10-13 16:03:05 +00:00
kompotkot 1443a15c66 Reduced glob tx pool size 2021-10-13 16:02:18 +00:00
kompotkot 600bba36bb Timeout for node serivce stop 2021-10-12 18:57:02 +00:00
kompotkot 704f89c7ad Fixed CORS check in go status servers 2021-10-01 14:56:07 +00:00
kompotkot 5a522b7926 Cors settings for status servers 2021-10-01 13:26:18 +00:00
kompotkot 10206840d2 Geth status returns block number 2021-09-29 14:25:41 +00:00
kompotkot b3412a3e36 Crawlers server for health checks 2021-09-28 13:44:24 +00:00
Neeraj Kashyap 6ae67d7a75
Merge pull request #283 from bugout-dev/fixing_nft_crawler 
added reporter to nft crawler, removed global cache for addresses
 2021-09-25 08:57:36 -07:00
Neeraj Kashyap d5eecf100f Fixed mypy issue 2021-09-25 08:56:22 -07:00
Neeraj Kashyap 126090ab54 Handled unicode code points that can't be stored in Postgres 
Also applied MOONSTREAM_SPIRE_API_URL configuration to mooncrawl
reporter so that we can report to non-production Spire APIs.
 2021-09-25 08:51:03 -07:00
yhtiyar 4bf2b5311d fix typo 2021-09-24 23:00:10 +03:00
yhtiyar 5f83b7edf4 added reporter 2021-09-24 22:58:17 +03:00
kompotkot 7d6853d0c8 tqdm bar for block crawlers 2021-09-24 14:31:34 +00:00
kompotkot 7e4f0f6ce0 Threads instead of pools, missing block cmd check num of transactions 
While we have a troubles with Pool execution calls with psycopg2,
using Threads executor instead.
 2021-09-24 13:28:41 +00:00
yhtiyar 9430d383d4 fixed required changes 2021-09-23 21:37:55 +03:00
yhtiyar a7c785e751 upgrade of moonstreamdb version 2021-09-23 21:28:36 +03:00
Sergei Sumarokov 1d3c4c5d41
Merge pull request #263 from bugout-dev/nft-explorer 
Improvements to "nft ethereum" crawlers
 2021-09-23 13:15:35 +03:00
yhtiyar 20a7189706 removed nft service 2021-09-23 13:14:01 +03:00
yhtiyar 1e7a070335 NFT_HUMBUG_TOKEN 2021-09-23 13:13:06 +03:00
yhtiyar 54ebdd922b MOONSTREAM_HUMBUG_TOKEN -> NFT_HUMBUG_TOKEN 2021-09-23 13:08:06 +03:00
yhtiyar 711319fbad added humbug reporter 2021-09-23 12:02:14 +03:00
kompotkot eafbc91a14 Fixed missing params and added logger for sync crawler 2021-09-22 14:17:25 +00:00
yhtiyar e75fcb7c22 removed casting for env variables. 2021-09-22 16:53:11 +03:00
yhtiyar f27d828d5a small fixes 2021-09-22 16:37:27 +03:00
kompotkot fbbeae957d Fix for lost go.mod during crawler build 2021-09-22 12:27:36 +00:00
kompotkot cddf7c07b6 Ethereum block sync service in deploy.bash 2021-09-22 11:29:37 +00:00
kompotkot 02f753564b Added geth node service for deployment 2021-09-22 10:29:00 +00:00
yhtiyar 310b23501e added bugout to setup.py 2021-09-22 13:08:29 +03:00
Sergei Sumarokov b93a08c48f
Merge pull request #269 from bugout-dev/sync-from-current-block 
Option to run block sync from current block at Ethereum
 2021-09-22 12:21:57 +03:00
yhtiyar db9e35e954 added new env variables to samle.env 2021-09-22 11:43:41 +03:00
yhtiyar d8cfa5a996 moved env variables to settings.py 2021-09-22 11:37:07 +03:00
kompotkot c3cc51e2e0 Block sync now in ProcessingOrder.ASCENDING by default 2021-09-21 16:02:40 +00:00
yhtiyar 94e7069c8a changed default time for sync to 1st jan of 20201 2021-09-21 18:47:13 +03:00
yhtiyar 8d11e1c8f0 removed comment and extended sleep time to 10 minutes 2021-09-21 18:32:32 +03:00
yhtiyar b173b617ee made synchronize work properly 2021-09-21 18:17:49 +03:00
yhtiyar e5f9a6c194 removed totalSupply 2021-09-21 18:17:24 +03:00
yhtiyar 79287e1fd2 fixed sync -> synchronize 2021-09-21 18:15:36 +03:00
yhtiyar f3e9559da9 added bugout dependency 2021-09-21 18:14:39 +03:00
kompotkot e9d84f0a07 Option to run block sync from current block at Ethereum 2021-09-21 14:06:21 +00:00
kompotkot fa3c1de288 Fixed path in deploy for crawlers 2021-09-21 10:04:46 +00:00
yhtiyar 42f6d893cc added sync functionality 2021-09-16 18:52:07 +03:00
Neeraj Kashyap c134d4f6ac Added support for created_at in Humbug reports 
`nft ethereum summary` crawler reports the `"date_range.end_time"` as
its `created_at`. This allows us to query the Humbug journal for all
summaries involving end blocks that were mined during a given stream
boundary.
 2021-09-16 02:14:22 -07:00
Neeraj Kashyap 33fa888117 Small updates on CLI 
Improved help for --humbug option for nft summaries and removed
`-humbug` form for that argument.
 2021-09-16 00:40:19 -07:00
yhtiyar a093946a7a changed summary schema, now it requires start and end blocks 2021-09-16 10:32:49 +03:00
yhtiyar 5b6d06cf99 added ethereum-nft service 2021-09-15 12:55:03 +03:00
kompotkot 8527932788 Deployment script for crawlers 2021-09-13 12:35:35 +00:00
Neeraj Kashyap 439d019164
Merge pull request #226 from bugout-dev/nft-explorer 
Nft explorer
 2021-09-08 08:39:20 -07:00
yhtiyar e51e008164 added sync command, that will continiously work 2021-09-08 17:00:02 +03:00
yhtiyar ed8096b813 Merge branch 'nft-explorer' of github.com:bugout-dev/moonstream into nft-explorer 2021-09-07 17:41:54 +03:00
yhtiyar e294cfa926 now, logs querries to node is done with batches 2021-09-07 17:40:59 +03:00
Neeraj Kashyap 8f541f84e3 NFT purchaser and minter statistics 2021-09-07 04:48:06 -07:00
Neeraj Kashyap 36233447c5 [WIP] Trying to get number of owners calculation to work 
DISTINCT ON query seems to be returning incorrect results.
 2021-09-07 04:08:22 -07:00
Neeraj Kashyap 0eb757d492 Fixed imports, added logging 2021-09-06 09:49:38 -07:00
Neeraj Kashyap 741383b503 Fix Etherscan crawler so that we don't manually specify autogenerated id 2021-09-06 09:46:50 -07:00
Neeraj Kashyap 1f771a650e "mints" now points to an object with an "amounts" key 
Makes it consistent with the other statistics.
 2021-09-06 00:01:18 -07:00
Neeraj Kashyap 1013d5cb25 First version of "nft ethereum summary" crawler 2021-09-05 22:59:10 -07:00
Neeraj Kashyap 5cc7def647 Working summaries involving nft mints, transfers, value 
This is currently only done for the past hour. Next step is to modify
the CLI to do it for:
1. Last hour
2. Last 24 hours
3. Last week
 2021-09-05 21:14:01 -07:00
Neeraj Kashyap 19c0dfeeb3 [WIP] Working prototypes for "nft ethereum label" and "nft ethereum summary" 
TODO:
- `nft ethereum label` should have reasonable defaults for `--start` and
`--end`. It should use the currenty block for `--end` and the last
labelled block for `--start`?
- Keep on plugging away at `nft ethereum summary` as per:
https://github.com/bugout-dev/moonstream/issues/223
 2021-09-03 14:10:13 -07:00
Neeraj Kashyap d0cbff9277 Superficial progress bar updates 2021-09-03 13:33:23 -07:00
Neeraj Kashyap 506bad759d Fixed some bugs in ethcrawler and nft ethereum crawler 
Tested that the "nft ethereum label" is working correctly.
 2021-09-03 13:30:13 -07:00
Neeraj Kashyap d2ff8d46ee [WIP] nft_mint and nft_transfer labels 
Not yet tested.
 2021-09-03 10:50:13 -07:00
yhtiyar 7031991661 added erc721 token 2021-09-03 20:10:35 +03:00
Neeraj Kashyap 60aa3006ed [WIP] Adding "erc721" label to unlabelled NFT contracts 2021-09-03 01:02:09 -07:00
Neeraj Kashyap 9d5421f866 Bumped moonstreamdb dependency in mooncrawl 2021-09-02 23:05:06 -07:00
Neeraj Kashyap 79d97afc9d Merge branch 'main' into nft-explorer 2021-09-02 22:49:54 -07:00
Neeraj Kashyap 5a997ad772 Few improvements to "nft ethereum" 2021-09-02 22:42:32 -07:00
Neeraj Kashyap a564737dbc summary function for Ethereum NFTs 2021-09-02 22:37:40 -07:00
Neeraj Kashyap f06f2e3fad Added Humbug publication functionality for "nft ethereum" command 2021-09-02 22:16:42 -07:00
Neeraj Kashyap 0694e7d45e Added nft CLI and bumped version to 0.0.4 2021-09-02 21:54:33 -07:00
Neeraj Kashyap 6ab99a03b2 Moved nft crawler into its own submodule 2021-09-02 21:53:18 -07:00
Neeraj Kashyap 738bc20b5e Cleaned up NFT crawler code a bit 
Hooked it up to comamnd line
 2021-09-01 15:01:24 -07:00
yhtiyar a80898a6a9 added raw nft transfer 2021-09-01 23:54:17 +03:00
yhtiyar 8cfbd716b3 not important fix 2021-09-01 22:29:53 +03:00
Neeraj Kashyap 05781726fd
Merge pull request #222 from bugout-dev/humbug-integration-crawlers 
Humbug reporter for moonstream crawlers
 2021-09-01 10:40:27 -07:00
kompotkot 6388716156 Hardcoded version in setup.py 2021-09-01 17:34:02 +00:00
kompotkot 8bbfd1e911 Humbug crash reporter for moonstream crawlers 2021-09-01 14:45:05 +00:00
yhtiyar a1acc7aaed initial version of nft explorer 2021-09-01 17:33:44 +03:00
kompotkot ec19d9e739 Specified a name for report for crawlers 2021-09-01 13:56:11 +00:00
kompotkot 44fb2bc8c7 Humbug reporter for moonstream python crawlers 2021-09-01 13:41:31 +00:00
kompotkot 11766df9cd Added value to tags in ether 2021-08-30 20:28:29 +00:00
kompotkot 39b66c217f CMC parser checks address and label exists 2021-08-30 13:43:08 +00:00
Neeraj Kashyap 0ae71fbd58 Merge branch 'main' of github.com:bugout-dev/moonstream into main 2021-08-27 09:38:54 -07:00
Neeraj Kashyap 9036f4dd6b Removed virtualenv directory that was accidentally commited in mooncrawl 2021-08-27 09:38:33 -07:00
kompotkot 84f0a66e75 Just a string 2021-08-27 12:09:38 +00:00
kompotkot fa763e923e Added additional tag for txpool crawler 2021-08-27 10:23:21 +00:00
kompotkot c73417a778 Updated version of humbug with bulk 2021-08-24 16:35:35 +00:00
kompotkot a6c71fb401 Modified deployment script for python crawlers 2021-08-24 16:07:18 +00:00
kompotkot d14be3e5d8 Merge branch 'main' into go-txpool-crawler 2021-08-24 15:57:55 +00:00
kompotkot eea2ce8f8b Using ValueError 2021-08-24 15:43:04 +00:00
kompotkot 6c7d0a783b Using exception instead of empty return with print 2021-08-24 15:42:00 +00:00
kompotkot 6f0a224562 Fixes for crawler mypy 2021-08-24 14:13:20 +00:00
kompotkot e54ea8bddb Finilized crawler to work with bulk endpoints 2021-08-24 12:52:45 +00:00
kompotkot 86c6d418a3 Working version of ethtxpool crawler 2021-08-23 11:01:49 +00:00
Neeraj Kashyap cca6be2922 Updated ethtxpool so that it only triggers publishing workflow for new 
pending transactions
 2021-08-20 07:36:11 -07:00
kompotkot 0eed78ba16 Some work 2021-08-20 13:39:02 +00:00
Neeraj Kashyap c075b94c51 Defer closing of gethClient 2021-08-18 12:46:17 -07:00
Neeraj Kashyap 69be51d8e3 Added TODOs 2021-08-18 12:44:12 -07:00
Neeraj Kashyap 5b45b95727 Fixed JSON marshalling for PendingTransaction 
Changed `Transaction` to `interface{}` because the Ethereum
`types.Transaction` struct does not have `json` annotations and so it
was serializing as an empty `{}`.
 2021-08-18 12:30:03 -07:00
Neeraj Kashyap 15d0836353 Initial draft of ethtxpool crawler 2021-08-18 12:25:30 -07:00
Neeraj Kashyap a6c6898b83 Moved mooncrawl Python package into subdirectory of crawlers/ 2021-08-18 11:14:47 -07:00
kompotkot 2d2f33f076 Initial go CLI 2021-08-18 18:05:49 +00:00
yhtiyar 0ef9f5e06c removed AWS_BUCKET_PREFIX 2021-08-18 19:18:28 +03:00
yhtiyar 44e95bb008 fixed index printing 2021-08-18 18:37:22 +03:00
yhtiyar 2d37e71a89 fixed typo 2021-08-18 18:03:17 +03:00
yhtiyar 21d48bc81d fixed typo 2021-08-18 17:51:23 +03:00
yhtiyar b21eb73376 updated version 2021-08-18 17:50:12 +03:00
yhtiyar c642742a7a Fixed required changes in PR 2021-08-18 17:24:09 +03:00
yhtiyar d54c6d854f Changet ETHSCAN_TOKEN name to MOONSTREAM_ETHSCAN_TOKEN 2021-08-18 14:16:56 +03:00
yhtiyar 57260b7ea4 Made crawler work 2021-08-18 14:12:53 +03:00
yhtiyar df39564db9 Merge branch 'main' into ethscan-crawler 2021-08-18 11:52:48 +03:00
yhtiyar c7794a964c removed unneeded crawl call 2021-08-18 11:52:18 +03:00
kompotkot 1ff4a887ce cmc url for each label in parsed token from cmc 2021-08-16 15:37:39 +00:00
kompotkot fcdd501261 Changed labele name for cmc parser 2021-08-16 15:19:11 +00:00
yhtiyar e929a3e72f Merge branch 'main' into ethscan-crawler 2021-08-16 17:26:33 +03:00
yhtiyar b0d545eb32 using datetime.now() instead of time.time() 2021-08-16 17:24:24 +03:00
yhtiyar 24253f8bb7 removed csv from repo, added it to s3 bucket 2021-08-16 17:09:23 +03:00
kompotkot 257405fd47 Fixed parser to work with labels properly to get coinmarketcap 2021-08-16 11:16:54 +00:00
yhtiyar 068941c163 Merge branch 'main' into ethscan-crawler 2021-08-10 17:21:49 +03:00
yhtiyar 5b90ed8832 temporary fix 2021-08-10 17:20:03 +03:00
Neeraj Kashyap 70adccdebb Updated "ethcrawler trending" result body 
Added `crawler_version:<version>` tag to Humbug report.
 2021-08-09 19:45:20 -07:00
Neeraj Kashyap ca6a5b2b70 "ethcrawler trending": default start, end 2021-08-09 19:37:57 -07:00
Neeraj Kashyap 0e98794db9 "ethcrawler trending" humbug token from env 
This makes life easier when using it as a systemd service.

Also added systemd service and timer for trending addresses crawler.
 2021-08-09 18:58:43 -07:00
Neeraj Kashyap bd49b2956b Slight renaming for clarity of internal args 2021-08-09 09:30:08 -07:00
Neeraj Kashyap 5084f7f32f Merge branch 'main' into crawlers-ethereum-trending 2021-08-09 09:17:40 -07:00
Neeraj Kashyap 272fabfd18 Black formatting 2021-08-09 09:13:08 -07:00
kompotkot de972d6722 Modified git commit hash to work with new labels table 2021-08-09 16:06:17 +00:00
yhtiyar b3e3c0d85f Merge branch 'main' into ethscan-crawler 2021-08-09 17:46:57 +03:00
yhtiyar d5aa127843 added ethscan smart contract source crawler 2021-08-09 17:41:13 +03:00
yhtiyar fb5b6fd56b added boto3 requirment 2021-08-09 17:40:02 +03:00
yhtiyar 3c479a37b4 added env variables for ethscan crawler 2021-08-09 17:39:28 +03:00
kompotkot 55ac27689c Working migration smart_contr -> address 2021-08-09 14:04:17 +00:00
Neeraj Kashyap d9d863e742 Functionality to publish JSON results to Humbug 
For `mooncrawl ethcrawler trending`
 2021-08-09 06:33:56 -07:00
Neeraj Kashyap 2d513800aa Displaying WEI values as ints (not floats) 
Also changed "metric" to "statistic" in the output for `mooncrawl
ethcrawler trending`.
 2021-08-09 06:01:28 -07:00
Neeraj Kashyap e5226b8a8b Working implementation of "ethcrawlers trending" 
The 24 hour queries take a LONG time but the 1 hour queries execute in 4
seconds.
 2021-08-09 05:55:10 -07:00
Neeraj Kashyap ea1a714e9b Scaffolding for "mooncrawl ethcrawler trending" 2021-08-09 04:58:47 -07:00
kompotkot ce1d114e14 Merge branch 'main' into address-identity 2021-08-09 11:39:51 +00:00
Neeraj Kashyap b0dc2c9e9c Added MOONSTREAM_HUMBUG_TOKEN environment variable 
The ethereum trending addresses crawler will be the first one to report
data through a Humbug journal. This requires us to accept a Humbug token
as configuration.
 2021-08-09 04:36:52 -07:00
kompotkot 26573fecf8 Initial version of eth address model with labels 2021-08-06 12:05:59 +00:00
Neeraj Kashyap 019da38ed2 Improved exception reporting for ethereum crawler 
Introduced `EthereumBlockCrawlError` and raising it whenever there is an
issue crawling blocks.
 2021-08-05 17:11:07 -07:00
Neeraj Kashyap 2f481c0c81 --confirmations argument to "ethcrawl synchronize" 2021-08-05 16:11:10 -07:00
Neeraj Kashyap 82b425da01 Had to save setup.py 
Also updated README to reflect name change.
 2021-08-05 15:54:19 -07:00
Neeraj Kashyap 44cbbc7a17 Version number import fixed 
Renamed version variable.

Bumped version to 0.0.2
 2021-08-05 15:53:03 -07:00
Neeraj Kashyap 4438324e05 Renamed moonstreamcrawlers -> mooncrawl 
Other name was a mouthful.
 2021-08-05 15:52:10 -07:00
Sergei Sumarokov b29e1b04da
Merge pull request #45 from bugout-dev/improved-missing-check 
Improved speed for missing blocks command check
 2021-08-03 16:14:52 +03:00
kompotkot 3c5a2929b7 Fixed args after review 2021-08-03 16:13:32 +03:00
kompotkot b969e01b1c Black format 2021-08-03 12:31:46 +00:00
kompotkot dc08573174 Github workflows 2021-08-03 12:22:06 +00:00
kompotkot 68d5c3755a Simplified bool args 2021-08-03 12:17:08 +03:00
kompotkot 45455a6ac2 Improved speed for missing blocks command check 2021-08-03 11:41:51 +03:00
Neeraj Kashyap 8955c29a64 Removed sleep after processing block 2021-08-02 12:04:09 -07:00
Neeraj Kashyap 6801fb5019 Changed sleeps to 5 seconds 2021-08-02 12:01:19 -07:00
Neeraj Kashyap 84a7a6acaf Added --jobs argument to "blocks synchronize" 2021-08-02 11:57:08 -07:00
Neeraj Kashyap 5ebb4491d0 Fixed CLI 2021-08-02 11:50:56 -07:00
Neeraj Kashyap 66a9b9a80f Actually use --order argument when running sync jobs 
Was not hooked up the the `crawl_blocks_executor` before.

Also removed branching for whether to run `crawl_blocks` or
`crawl_blocks_executor` in the CLI handler. Added `num_processes`
arguments to `crawl_blocks_executor` that callers can use to specify
whether or not to create subprocesses.
 2021-08-02 11:46:50 -07:00
Neeraj Kashyap 50cad0f17f Added --order argument to "blocks synchronize" 
This specifies if you want to crawl blocks in ascending or descending
order.
 2021-08-02 11:36:57 -07:00
Neeraj Kashyap b9d878a033 Fixed get_latest_block 2021-08-02 07:15:01 -07:00
Neeraj Kashyap 04383c11c3 Added support for HTTPProviders 2021-08-02 06:21:37 -07:00
Neeraj Kashyap 7f3fb3b8f9 Added --start argument to "blocks synchronize" 
Also black formatting
 2021-08-02 06:15:33 -07:00
kompotkot 39ff5e0b10 Small fix for CLI crawler sync command 2021-08-02 15:38:25 +03:00
kompotkot a7e2035273 Merge branch 'main' into clean-ethcrawler 2021-07-30 19:16:16 +03:00
Neeraj Kashyap 56e19b153b Merge branch 'main' into crawl-smart-contract-addresses-from-transaction-receipts 2021-07-30 05:39:53 -07:00
kompotkot 7950c04e41 Added TODO for handling errors for crawler 2021-07-29 22:08:26 +03:00
kompotkot 71d1f6c9b8 Added len check 2021-07-29 22:06:55 +03:00
Neeraj Kashyap d809d9b3d5 Error handling for crawlers 2021-07-29 11:52:37 -07:00
Neeraj Kashyap cbd9c2ccf1 Wait for crawling futures to return 2021-07-29 11:41:37 -07:00
Neeraj Kashyap 5ec29305d7 Cleaned up a little the worker submissions for ethcrawler 2021-07-29 11:36:19 -07:00
Neeraj Kashyap c6ddfb464c Updated .gitignore 2021-07-29 11:32:59 -07:00
kompotkot 4f8f589df2 Small updates 2021-07-29 21:09:53 +03:00
Neeraj Kashyap d2c30256b3 Merge branch 'main' into crawl-smart-contract-addresses-from-transaction-receipts 2021-07-29 09:42:55 -07:00
kompotkot 163c8ff350 Added verbose option 2021-07-29 17:01:39 +03:00
Neeraj Kashyap ad37dbd464 Made contract update crawler idempotent 
Had forgotten to add a check before so that it wouldn't reprocess old
transactions. Fixed now.
 2021-07-28 23:26:39 -07:00
Neeraj Kashyap 67e3135386 Latest commit for moonstreamdb requirement 2021-07-28 21:37:12 -07:00
Neeraj Kashyap 03a9295681 Ethereum smart contract registry updater 2021-07-28 21:35:49 -07:00
Neeraj Kashyap 1a749fe736 Added most recent revision of moonstreamdb 2021-07-28 21:00:10 -07:00
Neeraj Kashyap c5422c920d More prep work 2021-07-28 20:45:19 -07:00
Neeraj Kashyap 0f2d50a475 Updated commit hash for moonstreamdb 2021-07-28 20:26:50 -07:00
Neeraj Kashyap f027f6bc6b Moved esd.py into the moonstreamcrawlers package 
Also removed requirements.txt - it had windows-specific pywin32 that
just failed to load on Linux.

`moonstreamcrawlers` environments should be set up using `setup.py`.
 2021-07-28 20:22:25 -07:00
Neeraj Kashyap 5c6dc77392 Made esd a submodule of the crawlers 2021-07-28 20:14:10 -07:00
kompotkot c1c554d759 Fixes for transactions writing to db 2021-07-29 01:49:31 +03:00
kompotkot 048ff0fe23 Break for unnecessary synchronization 2021-07-29 01:02:33 +03:00
kompotkot c6f50a6f61 Lazy crawl block during missing check 2021-07-29 00:57:30 +03:00
kompotkot 73da2e430a Sync latest blocks and optimized blocks list gen 2021-07-29 00:43:35 +03:00
Neeraj Kashyap b9c828fc7f Merge branch 'main' into smart-contract-crawlers 2021-07-28 10:34:55 -07:00
Neeraj Kashyap 1a99d6b778 Merge branch 'main' into smart-contract-crawlers 2021-07-28 10:02:53 -07:00
kompotkot 3420d6cf86 Missing block check for crawlers cli 2021-07-28 18:03:06 +03:00
kompotkot 728122aba7 Updated hash of moonstreamdb in requirements 2021-07-28 14:20:11 +03:00
kompotkot dbf9e7f15c Updated setup and requirements for db and crawlers 2021-07-28 13:17:39 +03:00
kompotkot 59c7ba68e0 Added moonstreamdb at crawlers module 2021-07-28 13:07:24 +03:00
kompotkot ca64ebee56 Environment variables for windows powershell 2021-07-28 00:47:01 +03:00
kompotkot d5ac1e9074 Fixed env vars for eth crawler 2021-07-28 00:28:45 +03:00
Neeraj Kashyap a0d642f4c6 Merge branch 'main' into smart-contract-crawlers 2021-07-27 09:56:26 -07:00
kompotkot 19b2ed19aa Small fixes in crawler 2021-07-27 17:46:13 +03:00
kompotkot bb16b28a60 Added check argument for crawler 2021-07-27 17:12:02 +03:00
yhtiyar 3c87f5269f fixed repository path 2021-07-27 16:21:19 +03:00
Neeraj Kashyap a59e6545e8 README for esd crawler 2021-07-27 00:58:30 -07:00
Neeraj Kashyap ef9c73c554 Python implementation of Ethereum Signature Database crawler 2021-07-27 00:52:14 -07:00
Neeraj Kashyap 11980328dc Documentation (init) 2021-07-26 23:07:57 -07:00
kompotkot a20e581b52 Working crawler to parse blocks and transactions at Ethereum blockchain 2021-07-27 01:15:50 +03:00
kompotkot 4b7d0cb35c Init of ethereum crawler folder 2021-07-26 22:16:59 +03:00
yhtiyar 9a2552d45f Added function/event crawler for 4byte.directory 2021-07-26 13:02:09 +03:00