mirror
/
moonstream
kopia lustrzana https://github.com/bugout-dev/moonstream
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność
1 293 Commity
111 Gałęzie
50 Tagi
21 MiB
Wykres commitów

249 Commity (77dd49405d6f673918c4474956a22e81a3836bcf)

Autor SHA1 Wiadomość Data
Neeraj Kashyap 36233447c5 [WIP] Trying to get number of owners calculation to work 
DISTINCT ON query seems to be returning incorrect results.
 2021-09-07 04:08:22 -07:00
Neeraj Kashyap 0eb757d492 Fixed imports, added logging 2021-09-06 09:49:38 -07:00
Neeraj Kashyap 741383b503 Fix Etherscan crawler so that we don't manually specify autogenerated id 2021-09-06 09:46:50 -07:00
Neeraj Kashyap 1f771a650e "mints" now points to an object with an "amounts" key 
Makes it consistent with the other statistics.
 2021-09-06 00:01:18 -07:00
Neeraj Kashyap 1013d5cb25 First version of "nft ethereum summary" crawler 2021-09-05 22:59:10 -07:00
Neeraj Kashyap 5cc7def647 Working summaries involving nft mints, transfers, value 
This is currently only done for the past hour. Next step is to modify
the CLI to do it for:
1. Last hour
2. Last 24 hours
3. Last week
 2021-09-05 21:14:01 -07:00
Neeraj Kashyap 19c0dfeeb3 [WIP] Working prototypes for "nft ethereum label" and "nft ethereum summary" 
TODO:
- `nft ethereum label` should have reasonable defaults for `--start` and
`--end`. It should use the currenty block for `--end` and the last
labelled block for `--start`?
- Keep on plugging away at `nft ethereum summary` as per:
https://github.com/bugout-dev/moonstream/issues/223
 2021-09-03 14:10:13 -07:00
Neeraj Kashyap d0cbff9277 Superficial progress bar updates 2021-09-03 13:33:23 -07:00
Neeraj Kashyap 506bad759d Fixed some bugs in ethcrawler and nft ethereum crawler 
Tested that the "nft ethereum label" is working correctly.
 2021-09-03 13:30:13 -07:00
Neeraj Kashyap d2ff8d46ee [WIP] nft_mint and nft_transfer labels 
Not yet tested.
 2021-09-03 10:50:13 -07:00
yhtiyar 7031991661 added erc721 token 2021-09-03 20:10:35 +03:00
Neeraj Kashyap 60aa3006ed [WIP] Adding "erc721" label to unlabelled NFT contracts 2021-09-03 01:02:09 -07:00
Neeraj Kashyap 9d5421f866 Bumped moonstreamdb dependency in mooncrawl 2021-09-02 23:05:06 -07:00
Neeraj Kashyap 79d97afc9d Merge branch 'main' into nft-explorer 2021-09-02 22:49:54 -07:00
Neeraj Kashyap 5a997ad772 Few improvements to "nft ethereum" 2021-09-02 22:42:32 -07:00
Neeraj Kashyap a564737dbc summary function for Ethereum NFTs 2021-09-02 22:37:40 -07:00
Neeraj Kashyap f06f2e3fad Added Humbug publication functionality for "nft ethereum" command 2021-09-02 22:16:42 -07:00
Neeraj Kashyap 0694e7d45e Added nft CLI and bumped version to 0.0.4 2021-09-02 21:54:33 -07:00
Neeraj Kashyap 6ab99a03b2 Moved nft crawler into its own submodule 2021-09-02 21:53:18 -07:00
Neeraj Kashyap 738bc20b5e Cleaned up NFT crawler code a bit 
Hooked it up to comamnd line
 2021-09-01 15:01:24 -07:00
yhtiyar a80898a6a9 added raw nft transfer 2021-09-01 23:54:17 +03:00
yhtiyar 8cfbd716b3 not important fix 2021-09-01 22:29:53 +03:00
Neeraj Kashyap 05781726fd
Merge pull request #222 from bugout-dev/humbug-integration-crawlers 
Humbug reporter for moonstream crawlers
 2021-09-01 10:40:27 -07:00
kompotkot 6388716156 Hardcoded version in setup.py 2021-09-01 17:34:02 +00:00
kompotkot 8bbfd1e911 Humbug crash reporter for moonstream crawlers 2021-09-01 14:45:05 +00:00
yhtiyar a1acc7aaed initial version of nft explorer 2021-09-01 17:33:44 +03:00
kompotkot ec19d9e739 Specified a name for report for crawlers 2021-09-01 13:56:11 +00:00
kompotkot 44fb2bc8c7 Humbug reporter for moonstream python crawlers 2021-09-01 13:41:31 +00:00
kompotkot 11766df9cd Added value to tags in ether 2021-08-30 20:28:29 +00:00
kompotkot 39b66c217f CMC parser checks address and label exists 2021-08-30 13:43:08 +00:00
Neeraj Kashyap 0ae71fbd58 Merge branch 'main' of github.com:bugout-dev/moonstream into main 2021-08-27 09:38:54 -07:00
Neeraj Kashyap 9036f4dd6b Removed virtualenv directory that was accidentally commited in mooncrawl 2021-08-27 09:38:33 -07:00
kompotkot 84f0a66e75 Just a string 2021-08-27 12:09:38 +00:00
kompotkot fa763e923e Added additional tag for txpool crawler 2021-08-27 10:23:21 +00:00
kompotkot c73417a778 Updated version of humbug with bulk 2021-08-24 16:35:35 +00:00
kompotkot a6c71fb401 Modified deployment script for python crawlers 2021-08-24 16:07:18 +00:00
kompotkot d14be3e5d8 Merge branch 'main' into go-txpool-crawler 2021-08-24 15:57:55 +00:00
kompotkot eea2ce8f8b Using ValueError 2021-08-24 15:43:04 +00:00
kompotkot 6c7d0a783b Using exception instead of empty return with print 2021-08-24 15:42:00 +00:00
kompotkot 6f0a224562 Fixes for crawler mypy 2021-08-24 14:13:20 +00:00
kompotkot e54ea8bddb Finilized crawler to work with bulk endpoints 2021-08-24 12:52:45 +00:00
kompotkot 86c6d418a3 Working version of ethtxpool crawler 2021-08-23 11:01:49 +00:00
Neeraj Kashyap cca6be2922 Updated ethtxpool so that it only triggers publishing workflow for new 
pending transactions
 2021-08-20 07:36:11 -07:00
kompotkot 0eed78ba16 Some work 2021-08-20 13:39:02 +00:00
Neeraj Kashyap c075b94c51 Defer closing of gethClient 2021-08-18 12:46:17 -07:00
Neeraj Kashyap 69be51d8e3 Added TODOs 2021-08-18 12:44:12 -07:00
Neeraj Kashyap 5b45b95727 Fixed JSON marshalling for PendingTransaction 
Changed `Transaction` to `interface{}` because the Ethereum
`types.Transaction` struct does not have `json` annotations and so it
was serializing as an empty `{}`.
 2021-08-18 12:30:03 -07:00
Neeraj Kashyap 15d0836353 Initial draft of ethtxpool crawler 2021-08-18 12:25:30 -07:00
Neeraj Kashyap a6c6898b83 Moved mooncrawl Python package into subdirectory of crawlers/ 2021-08-18 11:14:47 -07:00
kompotkot 2d2f33f076 Initial go CLI 2021-08-18 18:05:49 +00:00
yhtiyar 0ef9f5e06c removed AWS_BUCKET_PREFIX 2021-08-18 19:18:28 +03:00
yhtiyar 44e95bb008 fixed index printing 2021-08-18 18:37:22 +03:00
yhtiyar 2d37e71a89 fixed typo 2021-08-18 18:03:17 +03:00
yhtiyar 21d48bc81d fixed typo 2021-08-18 17:51:23 +03:00
yhtiyar b21eb73376 updated version 2021-08-18 17:50:12 +03:00
yhtiyar c642742a7a Fixed required changes in PR 2021-08-18 17:24:09 +03:00
yhtiyar d54c6d854f Changet ETHSCAN_TOKEN name to MOONSTREAM_ETHSCAN_TOKEN 2021-08-18 14:16:56 +03:00
yhtiyar 57260b7ea4 Made crawler work 2021-08-18 14:12:53 +03:00
yhtiyar df39564db9 Merge branch 'main' into ethscan-crawler 2021-08-18 11:52:48 +03:00
yhtiyar c7794a964c removed unneeded crawl call 2021-08-18 11:52:18 +03:00
kompotkot 1ff4a887ce cmc url for each label in parsed token from cmc 2021-08-16 15:37:39 +00:00
kompotkot fcdd501261 Changed labele name for cmc parser 2021-08-16 15:19:11 +00:00
yhtiyar e929a3e72f Merge branch 'main' into ethscan-crawler 2021-08-16 17:26:33 +03:00
yhtiyar b0d545eb32 using datetime.now() instead of time.time() 2021-08-16 17:24:24 +03:00
yhtiyar 24253f8bb7 removed csv from repo, added it to s3 bucket 2021-08-16 17:09:23 +03:00
kompotkot 257405fd47 Fixed parser to work with labels properly to get coinmarketcap 2021-08-16 11:16:54 +00:00
yhtiyar 068941c163 Merge branch 'main' into ethscan-crawler 2021-08-10 17:21:49 +03:00
yhtiyar 5b90ed8832 temporary fix 2021-08-10 17:20:03 +03:00
Neeraj Kashyap 70adccdebb Updated "ethcrawler trending" result body 
Added `crawler_version:<version>` tag to Humbug report.
 2021-08-09 19:45:20 -07:00
Neeraj Kashyap ca6a5b2b70 "ethcrawler trending": default start, end 2021-08-09 19:37:57 -07:00
Neeraj Kashyap 0e98794db9 "ethcrawler trending" humbug token from env 
This makes life easier when using it as a systemd service.

Also added systemd service and timer for trending addresses crawler.
 2021-08-09 18:58:43 -07:00
Neeraj Kashyap bd49b2956b Slight renaming for clarity of internal args 2021-08-09 09:30:08 -07:00
Neeraj Kashyap 5084f7f32f Merge branch 'main' into crawlers-ethereum-trending 2021-08-09 09:17:40 -07:00
Neeraj Kashyap 272fabfd18 Black formatting 2021-08-09 09:13:08 -07:00
kompotkot de972d6722 Modified git commit hash to work with new labels table 2021-08-09 16:06:17 +00:00
yhtiyar b3e3c0d85f Merge branch 'main' into ethscan-crawler 2021-08-09 17:46:57 +03:00
yhtiyar d5aa127843 added ethscan smart contract source crawler 2021-08-09 17:41:13 +03:00
yhtiyar fb5b6fd56b added boto3 requirment 2021-08-09 17:40:02 +03:00
yhtiyar 3c479a37b4 added env variables for ethscan crawler 2021-08-09 17:39:28 +03:00
kompotkot 55ac27689c Working migration smart_contr -> address 2021-08-09 14:04:17 +00:00
Neeraj Kashyap d9d863e742 Functionality to publish JSON results to Humbug 
For `mooncrawl ethcrawler trending`
 2021-08-09 06:33:56 -07:00
Neeraj Kashyap 2d513800aa Displaying WEI values as ints (not floats) 
Also changed "metric" to "statistic" in the output for `mooncrawl
ethcrawler trending`.
 2021-08-09 06:01:28 -07:00
Neeraj Kashyap e5226b8a8b Working implementation of "ethcrawlers trending" 
The 24 hour queries take a LONG time but the 1 hour queries execute in 4
seconds.
 2021-08-09 05:55:10 -07:00
Neeraj Kashyap ea1a714e9b Scaffolding for "mooncrawl ethcrawler trending" 2021-08-09 04:58:47 -07:00
kompotkot ce1d114e14 Merge branch 'main' into address-identity 2021-08-09 11:39:51 +00:00
Neeraj Kashyap b0dc2c9e9c Added MOONSTREAM_HUMBUG_TOKEN environment variable 
The ethereum trending addresses crawler will be the first one to report
data through a Humbug journal. This requires us to accept a Humbug token
as configuration.
 2021-08-09 04:36:52 -07:00
kompotkot 26573fecf8 Initial version of eth address model with labels 2021-08-06 12:05:59 +00:00
Neeraj Kashyap 019da38ed2 Improved exception reporting for ethereum crawler 
Introduced `EthereumBlockCrawlError` and raising it whenever there is an
issue crawling blocks.
 2021-08-05 17:11:07 -07:00
Neeraj Kashyap 2f481c0c81 --confirmations argument to "ethcrawl synchronize" 2021-08-05 16:11:10 -07:00
Neeraj Kashyap 82b425da01 Had to save setup.py 
Also updated README to reflect name change.
 2021-08-05 15:54:19 -07:00
Neeraj Kashyap 44cbbc7a17 Version number import fixed 
Renamed version variable.

Bumped version to 0.0.2
 2021-08-05 15:53:03 -07:00
Neeraj Kashyap 4438324e05 Renamed moonstreamcrawlers -> mooncrawl 
Other name was a mouthful.
 2021-08-05 15:52:10 -07:00
Sergei Sumarokov b29e1b04da
Merge pull request #45 from bugout-dev/improved-missing-check 
Improved speed for missing blocks command check
 2021-08-03 16:14:52 +03:00
kompotkot 3c5a2929b7 Fixed args after review 2021-08-03 16:13:32 +03:00
kompotkot b969e01b1c Black format 2021-08-03 12:31:46 +00:00
kompotkot dc08573174 Github workflows 2021-08-03 12:22:06 +00:00
kompotkot 68d5c3755a Simplified bool args 2021-08-03 12:17:08 +03:00
kompotkot 45455a6ac2 Improved speed for missing blocks command check 2021-08-03 11:41:51 +03:00
Neeraj Kashyap 8955c29a64 Removed sleep after processing block 2021-08-02 12:04:09 -07:00
Neeraj Kashyap 6801fb5019 Changed sleeps to 5 seconds 2021-08-02 12:01:19 -07:00
Neeraj Kashyap 84a7a6acaf Added --jobs argument to "blocks synchronize" 2021-08-02 11:57:08 -07:00
Neeraj Kashyap 5ebb4491d0 Fixed CLI 2021-08-02 11:50:56 -07:00
Neeraj Kashyap 66a9b9a80f Actually use --order argument when running sync jobs 
Was not hooked up the the `crawl_blocks_executor` before.

Also removed branching for whether to run `crawl_blocks` or
`crawl_blocks_executor` in the CLI handler. Added `num_processes`
arguments to `crawl_blocks_executor` that callers can use to specify
whether or not to create subprocesses.
 2021-08-02 11:46:50 -07:00
Neeraj Kashyap 50cad0f17f Added --order argument to "blocks synchronize" 
This specifies if you want to crawl blocks in ascending or descending
order.
 2021-08-02 11:36:57 -07:00
Neeraj Kashyap b9d878a033 Fixed get_latest_block 2021-08-02 07:15:01 -07:00
Neeraj Kashyap 04383c11c3 Added support for HTTPProviders 2021-08-02 06:21:37 -07:00
Neeraj Kashyap 7f3fb3b8f9 Added --start argument to "blocks synchronize" 
Also black formatting
 2021-08-02 06:15:33 -07:00
kompotkot 39ff5e0b10 Small fix for CLI crawler sync command 2021-08-02 15:38:25 +03:00
kompotkot a7e2035273 Merge branch 'main' into clean-ethcrawler 2021-07-30 19:16:16 +03:00
Neeraj Kashyap 56e19b153b Merge branch 'main' into crawl-smart-contract-addresses-from-transaction-receipts 2021-07-30 05:39:53 -07:00
kompotkot 7950c04e41 Added TODO for handling errors for crawler 2021-07-29 22:08:26 +03:00
kompotkot 71d1f6c9b8 Added len check 2021-07-29 22:06:55 +03:00
Neeraj Kashyap d809d9b3d5 Error handling for crawlers 2021-07-29 11:52:37 -07:00
Neeraj Kashyap cbd9c2ccf1 Wait for crawling futures to return 2021-07-29 11:41:37 -07:00
Neeraj Kashyap 5ec29305d7 Cleaned up a little the worker submissions for ethcrawler 2021-07-29 11:36:19 -07:00
Neeraj Kashyap c6ddfb464c Updated .gitignore 2021-07-29 11:32:59 -07:00
kompotkot 4f8f589df2 Small updates 2021-07-29 21:09:53 +03:00
Neeraj Kashyap d2c30256b3 Merge branch 'main' into crawl-smart-contract-addresses-from-transaction-receipts 2021-07-29 09:42:55 -07:00
kompotkot 163c8ff350 Added verbose option 2021-07-29 17:01:39 +03:00
Neeraj Kashyap ad37dbd464 Made contract update crawler idempotent 
Had forgotten to add a check before so that it wouldn't reprocess old
transactions. Fixed now.
 2021-07-28 23:26:39 -07:00
Neeraj Kashyap 67e3135386 Latest commit for moonstreamdb requirement 2021-07-28 21:37:12 -07:00
Neeraj Kashyap 03a9295681 Ethereum smart contract registry updater 2021-07-28 21:35:49 -07:00
Neeraj Kashyap 1a749fe736 Added most recent revision of moonstreamdb 2021-07-28 21:00:10 -07:00
Neeraj Kashyap c5422c920d More prep work 2021-07-28 20:45:19 -07:00
Neeraj Kashyap 0f2d50a475 Updated commit hash for moonstreamdb 2021-07-28 20:26:50 -07:00
Neeraj Kashyap f027f6bc6b Moved esd.py into the moonstreamcrawlers package 
Also removed requirements.txt - it had windows-specific pywin32 that
just failed to load on Linux.

`moonstreamcrawlers` environments should be set up using `setup.py`.
 2021-07-28 20:22:25 -07:00
Neeraj Kashyap 5c6dc77392 Made esd a submodule of the crawlers 2021-07-28 20:14:10 -07:00
kompotkot c1c554d759 Fixes for transactions writing to db 2021-07-29 01:49:31 +03:00
kompotkot 048ff0fe23 Break for unnecessary synchronization 2021-07-29 01:02:33 +03:00
kompotkot c6f50a6f61 Lazy crawl block during missing check 2021-07-29 00:57:30 +03:00
kompotkot 73da2e430a Sync latest blocks and optimized blocks list gen 2021-07-29 00:43:35 +03:00
Neeraj Kashyap b9c828fc7f Merge branch 'main' into smart-contract-crawlers 2021-07-28 10:34:55 -07:00
Neeraj Kashyap 1a99d6b778 Merge branch 'main' into smart-contract-crawlers 2021-07-28 10:02:53 -07:00
kompotkot 3420d6cf86 Missing block check for crawlers cli 2021-07-28 18:03:06 +03:00
kompotkot 728122aba7 Updated hash of moonstreamdb in requirements 2021-07-28 14:20:11 +03:00
kompotkot dbf9e7f15c Updated setup and requirements for db and crawlers 2021-07-28 13:17:39 +03:00
kompotkot 59c7ba68e0 Added moonstreamdb at crawlers module 2021-07-28 13:07:24 +03:00
kompotkot ca64ebee56 Environment variables for windows powershell 2021-07-28 00:47:01 +03:00
kompotkot d5ac1e9074 Fixed env vars for eth crawler 2021-07-28 00:28:45 +03:00
Neeraj Kashyap a0d642f4c6 Merge branch 'main' into smart-contract-crawlers 2021-07-27 09:56:26 -07:00
kompotkot 19b2ed19aa Small fixes in crawler 2021-07-27 17:46:13 +03:00
kompotkot bb16b28a60 Added check argument for crawler 2021-07-27 17:12:02 +03:00
yhtiyar 3c87f5269f fixed repository path 2021-07-27 16:21:19 +03:00
Neeraj Kashyap a59e6545e8 README for esd crawler 2021-07-27 00:58:30 -07:00
Neeraj Kashyap ef9c73c554 Python implementation of Ethereum Signature Database crawler 2021-07-27 00:52:14 -07:00
Neeraj Kashyap 11980328dc Documentation (init) 2021-07-26 23:07:57 -07:00
kompotkot a20e581b52 Working crawler to parse blocks and transactions at Ethereum blockchain 2021-07-27 01:15:50 +03:00
kompotkot 4b7d0cb35c Init of ethereum crawler folder 2021-07-26 22:16:59 +03:00
yhtiyar 9a2552d45f Added function/event crawler for 4byte.directory 2021-07-26 13:02:09 +03:00