mirror
/
moonstream
kopia lustrzana https://github.com/bugout-dev/moonstream
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność
3 462 Commity
121 Gałęzie
76 Tagi
29 MiB
Wykres commitów

872 Commity (add-address-selector-index)

Autor SHA1 Wiadomość Data
yhtiyar 2e1e33c624 changed sleep default time 2021-11-07 17:15:05 +03:00
yhtiyar 5187d30257 fixes after review 2021-11-07 17:04:58 +03:00
yhtiyar 5ed9772852 working version of contract deploy crawler 2021-11-07 01:27:26 +03:00
kompotkot 28f16438cc Removed unnecessary flag at service crawler 2021-11-04 13:42:54 +00:00
kompotkot 0df96e053c Go fmt 2021-11-04 12:55:54 +00:00
kompotkot de922e8e83 Work with checkenv repo for crawlers deployment 2021-11-04 12:50:36 +00:00
kompotkot 40ffdd6ee5 Working go status servers 2021-11-03 11:18:49 +00:00
kompotkot e8e0f480ac Crawlers work with transaction type column 2021-10-28 13:14:48 +00:00
kompotkot 7cf1ef411f Block and tx crawlers works with new fields 2021-10-26 12:02:55 +00:00
Andrey Dolgolev 5b2de0d20c Remove import EthereumAddresses from etherscan and fix join on nft derive. 2021-10-21 11:39:16 +03:00
Andrey Dolgolev 689a8f2c31 Remove imports. 2021-10-20 20:50:27 +03:00
Andrey Dolgolev a4d103dc44 Change commit hash. 2021-10-20 20:27:58 +03:00
Neeraj Kashyap 6b5b6049b5 Merge branch 'main' into modify-label-table 2021-10-20 09:36:05 -07:00
Andrey Dolgolev 90cd175721 Remove all EthereumAddresses usage from cli and some imports. 2021-10-20 17:16:46 +03:00
Andrey Dolgolev 2f4c995687 Add fixes. 2021-10-19 23:02:10 +03:00
Sergei Sumarokov a7e172f8fe
Merge pull request #315 from bugout-dev/timeout-node-shutdown 
Timeout for node service shutdown
 2021-10-15 13:49:48 +03:00
kompotkot 68357ec2c4 Typo fix 2021-10-15 09:36:22 +00:00
kompotkot 55c5e89bbe Logic if block difference to large to catch up with crawler 2021-10-15 09:33:56 +00:00
kompotkot 31d9ff6f48 Fix sync blocks logic 2021-10-15 09:03:07 +00:00
kompotkot f795306afb Nodes deployment module 2021-10-15 08:38:28 +00:00
Andrey Dolgolev cfaa72e043 Add fixes. 2021-10-14 18:06:35 +03:00
Andrey Dolgolev 6890d38097 Add changes for labels table. 2021-10-14 18:01:00 +03:00
kompotkot 9e9462c519 x3 glob queue txpool value for geth 2021-10-13 16:03:05 +00:00
kompotkot 1443a15c66 Reduced glob tx pool size 2021-10-13 16:02:18 +00:00
kompotkot 600bba36bb Timeout for node serivce stop 2021-10-12 18:57:02 +00:00
kompotkot 704f89c7ad Fixed CORS check in go status servers 2021-10-01 14:56:07 +00:00
kompotkot 5a522b7926 Cors settings for status servers 2021-10-01 13:26:18 +00:00
kompotkot 10206840d2 Geth status returns block number 2021-09-29 14:25:41 +00:00
kompotkot b3412a3e36 Crawlers server for health checks 2021-09-28 13:44:24 +00:00
Neeraj Kashyap 6ae67d7a75
Merge pull request #283 from bugout-dev/fixing_nft_crawler 
added reporter to nft crawler, removed global cache for addresses
 2021-09-25 08:57:36 -07:00
Neeraj Kashyap d5eecf100f Fixed mypy issue 2021-09-25 08:56:22 -07:00
Neeraj Kashyap 126090ab54 Handled unicode code points that can't be stored in Postgres 
Also applied MOONSTREAM_SPIRE_API_URL configuration to mooncrawl
reporter so that we can report to non-production Spire APIs.
 2021-09-25 08:51:03 -07:00
yhtiyar 4bf2b5311d fix typo 2021-09-24 23:00:10 +03:00
yhtiyar 5f83b7edf4 added reporter 2021-09-24 22:58:17 +03:00
kompotkot 7d6853d0c8 tqdm bar for block crawlers 2021-09-24 14:31:34 +00:00
kompotkot 7e4f0f6ce0 Threads instead of pools, missing block cmd check num of transactions 
While we have a troubles with Pool execution calls with psycopg2,
using Threads executor instead.
 2021-09-24 13:28:41 +00:00
yhtiyar 9430d383d4 fixed required changes 2021-09-23 21:37:55 +03:00
yhtiyar a7c785e751 upgrade of moonstreamdb version 2021-09-23 21:28:36 +03:00
Sergei Sumarokov 1d3c4c5d41
Merge pull request #263 from bugout-dev/nft-explorer 
Improvements to "nft ethereum" crawlers
 2021-09-23 13:15:35 +03:00
yhtiyar 20a7189706 removed nft service 2021-09-23 13:14:01 +03:00
yhtiyar 1e7a070335 NFT_HUMBUG_TOKEN 2021-09-23 13:13:06 +03:00
yhtiyar 54ebdd922b MOONSTREAM_HUMBUG_TOKEN -> NFT_HUMBUG_TOKEN 2021-09-23 13:08:06 +03:00
yhtiyar 711319fbad added humbug reporter 2021-09-23 12:02:14 +03:00
kompotkot eafbc91a14 Fixed missing params and added logger for sync crawler 2021-09-22 14:17:25 +00:00
yhtiyar e75fcb7c22 removed casting for env variables. 2021-09-22 16:53:11 +03:00
yhtiyar f27d828d5a small fixes 2021-09-22 16:37:27 +03:00
kompotkot fbbeae957d Fix for lost go.mod during crawler build 2021-09-22 12:27:36 +00:00
kompotkot cddf7c07b6 Ethereum block sync service in deploy.bash 2021-09-22 11:29:37 +00:00
kompotkot 02f753564b Added geth node service for deployment 2021-09-22 10:29:00 +00:00
yhtiyar 310b23501e added bugout to setup.py 2021-09-22 13:08:29 +03:00
Sergei Sumarokov b93a08c48f
Merge pull request #269 from bugout-dev/sync-from-current-block 
Option to run block sync from current block at Ethereum
 2021-09-22 12:21:57 +03:00
yhtiyar db9e35e954 added new env variables to samle.env 2021-09-22 11:43:41 +03:00
yhtiyar d8cfa5a996 moved env variables to settings.py 2021-09-22 11:37:07 +03:00
kompotkot c3cc51e2e0 Block sync now in ProcessingOrder.ASCENDING by default 2021-09-21 16:02:40 +00:00
yhtiyar 94e7069c8a changed default time for sync to 1st jan of 20201 2021-09-21 18:47:13 +03:00
yhtiyar 8d11e1c8f0 removed comment and extended sleep time to 10 minutes 2021-09-21 18:32:32 +03:00
yhtiyar b173b617ee made synchronize work properly 2021-09-21 18:17:49 +03:00
yhtiyar e5f9a6c194 removed totalSupply 2021-09-21 18:17:24 +03:00
yhtiyar 79287e1fd2 fixed sync -> synchronize 2021-09-21 18:15:36 +03:00
yhtiyar f3e9559da9 added bugout dependency 2021-09-21 18:14:39 +03:00
kompotkot e9d84f0a07 Option to run block sync from current block at Ethereum 2021-09-21 14:06:21 +00:00
kompotkot fa3c1de288 Fixed path in deploy for crawlers 2021-09-21 10:04:46 +00:00
yhtiyar 42f6d893cc added sync functionality 2021-09-16 18:52:07 +03:00
Neeraj Kashyap c134d4f6ac Added support for created_at in Humbug reports 
`nft ethereum summary` crawler reports the `"date_range.end_time"` as
its `created_at`. This allows us to query the Humbug journal for all
summaries involving end blocks that were mined during a given stream
boundary.
 2021-09-16 02:14:22 -07:00
Neeraj Kashyap 33fa888117 Small updates on CLI 
Improved help for --humbug option for nft summaries and removed
`-humbug` form for that argument.
 2021-09-16 00:40:19 -07:00
yhtiyar a093946a7a changed summary schema, now it requires start and end blocks 2021-09-16 10:32:49 +03:00
yhtiyar 5b6d06cf99 added ethereum-nft service 2021-09-15 12:55:03 +03:00
kompotkot 8527932788 Deployment script for crawlers 2021-09-13 12:35:35 +00:00
Neeraj Kashyap 439d019164
Merge pull request #226 from bugout-dev/nft-explorer 
Nft explorer
 2021-09-08 08:39:20 -07:00
yhtiyar e51e008164 added sync command, that will continiously work 2021-09-08 17:00:02 +03:00
yhtiyar ed8096b813 Merge branch 'nft-explorer' of github.com:bugout-dev/moonstream into nft-explorer 2021-09-07 17:41:54 +03:00
yhtiyar e294cfa926 now, logs querries to node is done with batches 2021-09-07 17:40:59 +03:00
Neeraj Kashyap 8f541f84e3 NFT purchaser and minter statistics 2021-09-07 04:48:06 -07:00
Neeraj Kashyap 36233447c5 [WIP] Trying to get number of owners calculation to work 
DISTINCT ON query seems to be returning incorrect results.
 2021-09-07 04:08:22 -07:00
Neeraj Kashyap 0eb757d492 Fixed imports, added logging 2021-09-06 09:49:38 -07:00
Neeraj Kashyap 741383b503 Fix Etherscan crawler so that we don't manually specify autogenerated id 2021-09-06 09:46:50 -07:00
Neeraj Kashyap 1f771a650e "mints" now points to an object with an "amounts" key 
Makes it consistent with the other statistics.
 2021-09-06 00:01:18 -07:00
Neeraj Kashyap 1013d5cb25 First version of "nft ethereum summary" crawler 2021-09-05 22:59:10 -07:00
Neeraj Kashyap 5cc7def647 Working summaries involving nft mints, transfers, value 
This is currently only done for the past hour. Next step is to modify
the CLI to do it for:
1. Last hour
2. Last 24 hours
3. Last week
 2021-09-05 21:14:01 -07:00
Neeraj Kashyap 19c0dfeeb3 [WIP] Working prototypes for "nft ethereum label" and "nft ethereum summary" 
TODO:
- `nft ethereum label` should have reasonable defaults for `--start` and
`--end`. It should use the currenty block for `--end` and the last
labelled block for `--start`?
- Keep on plugging away at `nft ethereum summary` as per:
https://github.com/bugout-dev/moonstream/issues/223
 2021-09-03 14:10:13 -07:00
Neeraj Kashyap d0cbff9277 Superficial progress bar updates 2021-09-03 13:33:23 -07:00
Neeraj Kashyap 506bad759d Fixed some bugs in ethcrawler and nft ethereum crawler 
Tested that the "nft ethereum label" is working correctly.
 2021-09-03 13:30:13 -07:00
Neeraj Kashyap d2ff8d46ee [WIP] nft_mint and nft_transfer labels 
Not yet tested.
 2021-09-03 10:50:13 -07:00
yhtiyar 7031991661 added erc721 token 2021-09-03 20:10:35 +03:00
Neeraj Kashyap 60aa3006ed [WIP] Adding "erc721" label to unlabelled NFT contracts 2021-09-03 01:02:09 -07:00
Neeraj Kashyap 9d5421f866 Bumped moonstreamdb dependency in mooncrawl 2021-09-02 23:05:06 -07:00
Neeraj Kashyap 79d97afc9d Merge branch 'main' into nft-explorer 2021-09-02 22:49:54 -07:00
Neeraj Kashyap 5a997ad772 Few improvements to "nft ethereum" 2021-09-02 22:42:32 -07:00
Neeraj Kashyap a564737dbc summary function for Ethereum NFTs 2021-09-02 22:37:40 -07:00
Neeraj Kashyap f06f2e3fad Added Humbug publication functionality for "nft ethereum" command 2021-09-02 22:16:42 -07:00
Neeraj Kashyap 0694e7d45e Added nft CLI and bumped version to 0.0.4 2021-09-02 21:54:33 -07:00
Neeraj Kashyap 6ab99a03b2 Moved nft crawler into its own submodule 2021-09-02 21:53:18 -07:00
Neeraj Kashyap 738bc20b5e Cleaned up NFT crawler code a bit 
Hooked it up to comamnd line
 2021-09-01 15:01:24 -07:00
yhtiyar a80898a6a9 added raw nft transfer 2021-09-01 23:54:17 +03:00
yhtiyar 8cfbd716b3 not important fix 2021-09-01 22:29:53 +03:00
Neeraj Kashyap 05781726fd
Merge pull request #222 from bugout-dev/humbug-integration-crawlers 
Humbug reporter for moonstream crawlers
 2021-09-01 10:40:27 -07:00
kompotkot 6388716156 Hardcoded version in setup.py 2021-09-01 17:34:02 +00:00
kompotkot 8bbfd1e911 Humbug crash reporter for moonstream crawlers 2021-09-01 14:45:05 +00:00
yhtiyar a1acc7aaed initial version of nft explorer 2021-09-01 17:33:44 +03:00
kompotkot ec19d9e739 Specified a name for report for crawlers 2021-09-01 13:56:11 +00:00
kompotkot 44fb2bc8c7 Humbug reporter for moonstream python crawlers 2021-09-01 13:41:31 +00:00
kompotkot 11766df9cd Added value to tags in ether 2021-08-30 20:28:29 +00:00
kompotkot 39b66c217f CMC parser checks address and label exists 2021-08-30 13:43:08 +00:00
Neeraj Kashyap 0ae71fbd58 Merge branch 'main' of github.com:bugout-dev/moonstream into main 2021-08-27 09:38:54 -07:00
Neeraj Kashyap 9036f4dd6b Removed virtualenv directory that was accidentally commited in mooncrawl 2021-08-27 09:38:33 -07:00
kompotkot 84f0a66e75 Just a string 2021-08-27 12:09:38 +00:00
kompotkot fa763e923e Added additional tag for txpool crawler 2021-08-27 10:23:21 +00:00
kompotkot c73417a778 Updated version of humbug with bulk 2021-08-24 16:35:35 +00:00
kompotkot a6c71fb401 Modified deployment script for python crawlers 2021-08-24 16:07:18 +00:00
kompotkot d14be3e5d8 Merge branch 'main' into go-txpool-crawler 2021-08-24 15:57:55 +00:00
kompotkot eea2ce8f8b Using ValueError 2021-08-24 15:43:04 +00:00
kompotkot 6c7d0a783b Using exception instead of empty return with print 2021-08-24 15:42:00 +00:00
kompotkot 6f0a224562 Fixes for crawler mypy 2021-08-24 14:13:20 +00:00
kompotkot e54ea8bddb Finilized crawler to work with bulk endpoints 2021-08-24 12:52:45 +00:00
kompotkot 86c6d418a3 Working version of ethtxpool crawler 2021-08-23 11:01:49 +00:00
Neeraj Kashyap cca6be2922 Updated ethtxpool so that it only triggers publishing workflow for new 
pending transactions
 2021-08-20 07:36:11 -07:00
kompotkot 0eed78ba16 Some work 2021-08-20 13:39:02 +00:00
Neeraj Kashyap c075b94c51 Defer closing of gethClient 2021-08-18 12:46:17 -07:00
Neeraj Kashyap 69be51d8e3 Added TODOs 2021-08-18 12:44:12 -07:00
Neeraj Kashyap 5b45b95727 Fixed JSON marshalling for PendingTransaction 
Changed `Transaction` to `interface{}` because the Ethereum
`types.Transaction` struct does not have `json` annotations and so it
was serializing as an empty `{}`.
 2021-08-18 12:30:03 -07:00
Neeraj Kashyap 15d0836353 Initial draft of ethtxpool crawler 2021-08-18 12:25:30 -07:00
Neeraj Kashyap a6c6898b83 Moved mooncrawl Python package into subdirectory of crawlers/ 2021-08-18 11:14:47 -07:00
kompotkot 2d2f33f076 Initial go CLI 2021-08-18 18:05:49 +00:00
yhtiyar 0ef9f5e06c removed AWS_BUCKET_PREFIX 2021-08-18 19:18:28 +03:00
yhtiyar 44e95bb008 fixed index printing 2021-08-18 18:37:22 +03:00
yhtiyar 2d37e71a89 fixed typo 2021-08-18 18:03:17 +03:00
yhtiyar 21d48bc81d fixed typo 2021-08-18 17:51:23 +03:00
yhtiyar b21eb73376 updated version 2021-08-18 17:50:12 +03:00
yhtiyar c642742a7a Fixed required changes in PR 2021-08-18 17:24:09 +03:00
yhtiyar d54c6d854f Changet ETHSCAN_TOKEN name to MOONSTREAM_ETHSCAN_TOKEN 2021-08-18 14:16:56 +03:00
yhtiyar 57260b7ea4 Made crawler work 2021-08-18 14:12:53 +03:00
yhtiyar df39564db9 Merge branch 'main' into ethscan-crawler 2021-08-18 11:52:48 +03:00
yhtiyar c7794a964c removed unneeded crawl call 2021-08-18 11:52:18 +03:00
kompotkot 1ff4a887ce cmc url for each label in parsed token from cmc 2021-08-16 15:37:39 +00:00
kompotkot fcdd501261 Changed labele name for cmc parser 2021-08-16 15:19:11 +00:00
yhtiyar e929a3e72f Merge branch 'main' into ethscan-crawler 2021-08-16 17:26:33 +03:00
yhtiyar b0d545eb32 using datetime.now() instead of time.time() 2021-08-16 17:24:24 +03:00
yhtiyar 24253f8bb7 removed csv from repo, added it to s3 bucket 2021-08-16 17:09:23 +03:00
kompotkot 257405fd47 Fixed parser to work with labels properly to get coinmarketcap 2021-08-16 11:16:54 +00:00
yhtiyar 068941c163 Merge branch 'main' into ethscan-crawler 2021-08-10 17:21:49 +03:00
yhtiyar 5b90ed8832 temporary fix 2021-08-10 17:20:03 +03:00
Neeraj Kashyap 70adccdebb Updated "ethcrawler trending" result body 
Added `crawler_version:<version>` tag to Humbug report.
 2021-08-09 19:45:20 -07:00
Neeraj Kashyap ca6a5b2b70 "ethcrawler trending": default start, end 2021-08-09 19:37:57 -07:00
Neeraj Kashyap 0e98794db9 "ethcrawler trending" humbug token from env 
This makes life easier when using it as a systemd service.

Also added systemd service and timer for trending addresses crawler.
 2021-08-09 18:58:43 -07:00
Neeraj Kashyap bd49b2956b Slight renaming for clarity of internal args 2021-08-09 09:30:08 -07:00
Neeraj Kashyap 5084f7f32f Merge branch 'main' into crawlers-ethereum-trending 2021-08-09 09:17:40 -07:00
Neeraj Kashyap 272fabfd18 Black formatting 2021-08-09 09:13:08 -07:00
kompotkot de972d6722 Modified git commit hash to work with new labels table 2021-08-09 16:06:17 +00:00
yhtiyar b3e3c0d85f Merge branch 'main' into ethscan-crawler 2021-08-09 17:46:57 +03:00
yhtiyar d5aa127843 added ethscan smart contract source crawler 2021-08-09 17:41:13 +03:00
yhtiyar fb5b6fd56b added boto3 requirment 2021-08-09 17:40:02 +03:00
yhtiyar 3c479a37b4 added env variables for ethscan crawler 2021-08-09 17:39:28 +03:00
kompotkot 55ac27689c Working migration smart_contr -> address 2021-08-09 14:04:17 +00:00
Neeraj Kashyap d9d863e742 Functionality to publish JSON results to Humbug 
For `mooncrawl ethcrawler trending`
 2021-08-09 06:33:56 -07:00
Neeraj Kashyap 2d513800aa Displaying WEI values as ints (not floats) 
Also changed "metric" to "statistic" in the output for `mooncrawl
ethcrawler trending`.
 2021-08-09 06:01:28 -07:00
Neeraj Kashyap e5226b8a8b Working implementation of "ethcrawlers trending" 
The 24 hour queries take a LONG time but the 1 hour queries execute in 4
seconds.
 2021-08-09 05:55:10 -07:00
Neeraj Kashyap ea1a714e9b Scaffolding for "mooncrawl ethcrawler trending" 2021-08-09 04:58:47 -07:00
kompotkot ce1d114e14 Merge branch 'main' into address-identity 2021-08-09 11:39:51 +00:00
Neeraj Kashyap b0dc2c9e9c Added MOONSTREAM_HUMBUG_TOKEN environment variable 
The ethereum trending addresses crawler will be the first one to report
data through a Humbug journal. This requires us to accept a Humbug token
as configuration.
 2021-08-09 04:36:52 -07:00
kompotkot 26573fecf8 Initial version of eth address model with labels 2021-08-06 12:05:59 +00:00
Neeraj Kashyap 019da38ed2 Improved exception reporting for ethereum crawler 
Introduced `EthereumBlockCrawlError` and raising it whenever there is an
issue crawling blocks.
 2021-08-05 17:11:07 -07:00
Neeraj Kashyap 2f481c0c81 --confirmations argument to "ethcrawl synchronize" 2021-08-05 16:11:10 -07:00
Neeraj Kashyap 82b425da01 Had to save setup.py 
Also updated README to reflect name change.
 2021-08-05 15:54:19 -07:00
Neeraj Kashyap 44cbbc7a17 Version number import fixed 
Renamed version variable.

Bumped version to 0.0.2
 2021-08-05 15:53:03 -07:00
Neeraj Kashyap 4438324e05 Renamed moonstreamcrawlers -> mooncrawl 
Other name was a mouthful.
 2021-08-05 15:52:10 -07:00
Sergei Sumarokov b29e1b04da
Merge pull request #45 from bugout-dev/improved-missing-check 
Improved speed for missing blocks command check
 2021-08-03 16:14:52 +03:00
kompotkot 3c5a2929b7 Fixed args after review 2021-08-03 16:13:32 +03:00
kompotkot b969e01b1c Black format 2021-08-03 12:31:46 +00:00
kompotkot dc08573174 Github workflows 2021-08-03 12:22:06 +00:00
kompotkot 68d5c3755a Simplified bool args 2021-08-03 12:17:08 +03:00
kompotkot 45455a6ac2 Improved speed for missing blocks command check 2021-08-03 11:41:51 +03:00
Neeraj Kashyap 8955c29a64 Removed sleep after processing block 2021-08-02 12:04:09 -07:00
Neeraj Kashyap 6801fb5019 Changed sleeps to 5 seconds 2021-08-02 12:01:19 -07:00
Neeraj Kashyap 84a7a6acaf Added --jobs argument to "blocks synchronize" 2021-08-02 11:57:08 -07:00
Neeraj Kashyap 5ebb4491d0 Fixed CLI 2021-08-02 11:50:56 -07:00
Neeraj Kashyap 66a9b9a80f Actually use --order argument when running sync jobs 
Was not hooked up the the `crawl_blocks_executor` before.

Also removed branching for whether to run `crawl_blocks` or
`crawl_blocks_executor` in the CLI handler. Added `num_processes`
arguments to `crawl_blocks_executor` that callers can use to specify
whether or not to create subprocesses.
 2021-08-02 11:46:50 -07:00
Neeraj Kashyap 50cad0f17f Added --order argument to "blocks synchronize" 
This specifies if you want to crawl blocks in ascending or descending
order.
 2021-08-02 11:36:57 -07:00
Neeraj Kashyap b9d878a033 Fixed get_latest_block 2021-08-02 07:15:01 -07:00
Neeraj Kashyap 04383c11c3 Added support for HTTPProviders 2021-08-02 06:21:37 -07:00
Neeraj Kashyap 7f3fb3b8f9 Added --start argument to "blocks synchronize" 
Also black formatting
 2021-08-02 06:15:33 -07:00
kompotkot 39ff5e0b10 Small fix for CLI crawler sync command 2021-08-02 15:38:25 +03:00
kompotkot a7e2035273 Merge branch 'main' into clean-ethcrawler 2021-07-30 19:16:16 +03:00
Neeraj Kashyap 56e19b153b Merge branch 'main' into crawl-smart-contract-addresses-from-transaction-receipts 2021-07-30 05:39:53 -07:00
kompotkot 7950c04e41 Added TODO for handling errors for crawler 2021-07-29 22:08:26 +03:00
kompotkot 71d1f6c9b8 Added len check 2021-07-29 22:06:55 +03:00
Neeraj Kashyap d809d9b3d5 Error handling for crawlers 2021-07-29 11:52:37 -07:00
Neeraj Kashyap cbd9c2ccf1 Wait for crawling futures to return 2021-07-29 11:41:37 -07:00
Neeraj Kashyap 5ec29305d7 Cleaned up a little the worker submissions for ethcrawler 2021-07-29 11:36:19 -07:00
Neeraj Kashyap c6ddfb464c Updated .gitignore 2021-07-29 11:32:59 -07:00
kompotkot 4f8f589df2 Small updates 2021-07-29 21:09:53 +03:00
Neeraj Kashyap d2c30256b3 Merge branch 'main' into crawl-smart-contract-addresses-from-transaction-receipts 2021-07-29 09:42:55 -07:00
kompotkot 163c8ff350 Added verbose option 2021-07-29 17:01:39 +03:00
Neeraj Kashyap ad37dbd464 Made contract update crawler idempotent 
Had forgotten to add a check before so that it wouldn't reprocess old
transactions. Fixed now.
 2021-07-28 23:26:39 -07:00
Neeraj Kashyap 67e3135386 Latest commit for moonstreamdb requirement 2021-07-28 21:37:12 -07:00
Neeraj Kashyap 03a9295681 Ethereum smart contract registry updater 2021-07-28 21:35:49 -07:00
Neeraj Kashyap 1a749fe736 Added most recent revision of moonstreamdb 2021-07-28 21:00:10 -07:00
Neeraj Kashyap c5422c920d More prep work 2021-07-28 20:45:19 -07:00
Neeraj Kashyap 0f2d50a475 Updated commit hash for moonstreamdb 2021-07-28 20:26:50 -07:00
Neeraj Kashyap f027f6bc6b Moved esd.py into the moonstreamcrawlers package 
Also removed requirements.txt - it had windows-specific pywin32 that
just failed to load on Linux.

`moonstreamcrawlers` environments should be set up using `setup.py`.
 2021-07-28 20:22:25 -07:00
Neeraj Kashyap 5c6dc77392 Made esd a submodule of the crawlers 2021-07-28 20:14:10 -07:00
kompotkot c1c554d759 Fixes for transactions writing to db 2021-07-29 01:49:31 +03:00
kompotkot 048ff0fe23 Break for unnecessary synchronization 2021-07-29 01:02:33 +03:00
kompotkot c6f50a6f61 Lazy crawl block during missing check 2021-07-29 00:57:30 +03:00
kompotkot 73da2e430a Sync latest blocks and optimized blocks list gen 2021-07-29 00:43:35 +03:00
Neeraj Kashyap b9c828fc7f Merge branch 'main' into smart-contract-crawlers 2021-07-28 10:34:55 -07:00
Neeraj Kashyap 1a99d6b778 Merge branch 'main' into smart-contract-crawlers 2021-07-28 10:02:53 -07:00
kompotkot 3420d6cf86 Missing block check for crawlers cli 2021-07-28 18:03:06 +03:00
kompotkot 728122aba7 Updated hash of moonstreamdb in requirements 2021-07-28 14:20:11 +03:00
kompotkot dbf9e7f15c Updated setup and requirements for db and crawlers 2021-07-28 13:17:39 +03:00
kompotkot 59c7ba68e0 Added moonstreamdb at crawlers module 2021-07-28 13:07:24 +03:00
kompotkot ca64ebee56 Environment variables for windows powershell 2021-07-28 00:47:01 +03:00
kompotkot d5ac1e9074 Fixed env vars for eth crawler 2021-07-28 00:28:45 +03:00
Neeraj Kashyap a0d642f4c6 Merge branch 'main' into smart-contract-crawlers 2021-07-27 09:56:26 -07:00
kompotkot 19b2ed19aa Small fixes in crawler 2021-07-27 17:46:13 +03:00
kompotkot bb16b28a60 Added check argument for crawler 2021-07-27 17:12:02 +03:00
yhtiyar 3c87f5269f fixed repository path 2021-07-27 16:21:19 +03:00
Neeraj Kashyap a59e6545e8 README for esd crawler 2021-07-27 00:58:30 -07:00
Neeraj Kashyap ef9c73c554 Python implementation of Ethereum Signature Database crawler 2021-07-27 00:52:14 -07:00
Neeraj Kashyap 11980328dc Documentation (init) 2021-07-26 23:07:57 -07:00
kompotkot a20e581b52 Working crawler to parse blocks and transactions at Ethereum blockchain 2021-07-27 01:15:50 +03:00
kompotkot 4b7d0cb35c Init of ethereum crawler folder 2021-07-26 22:16:59 +03:00
yhtiyar 9a2552d45f Added function/event crawler for 4byte.directory 2021-07-26 13:02:09 +03:00