Merge pull request #123 from moonstream-to/fix-tuple-array-parser

Fixing the tuple array parsing. Now type=eval is included

.github/workflows/lint.yml

@@ -17,8 +17,6 @@ jobs:
         run: pip install -U pip
       - name: Install dev dependencies
         run: pip install -e .[dev]
-      - name: Mypy type check
-        run: mypy moonworm/
       - name: Isort imports check
         run: isort --check moonworm/
       - name: Black syntax check

.github/workflows/release.yml

@@ -8,8 +8,8 @@ jobs:
   publish:
     runs-on: ubuntu-20.04
     steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
         with:
           python-version: '3.8'
       - name: Install dependencies
@@ -18,8 +18,8 @@ jobs:
           pip install -e .[distribute]
       - name: Build and publish
         env:
-          TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
-          TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+          TWINE_USERNAME: "__token__"
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
         run: |
           python setup.py sdist bdist_wheel
           twine upload dist/*

README.md

@@ -6,25 +6,26 @@ Moonworm is a set of tools that helps you develop/analyze blockchain dapps. Pump
 
 1. `moonworm watch` -  Tool to monitor and crawl(index) decoded smart contract data. It gets you historic/on going smart contract’s decoded `events` and `transactions`. No sweat, just provide `abi` and smart contract’s address and get stream of data. With this tool you can: analyze  incidents, set up alerting, build datasets, write sniping bots, etc.
 2. `moonworm generate-brownie` - Brownie on steroids. Generate python interface and cli for your smart contracts in “one click”, focus on smart contract development, `moonworm` will do the rest. In addition, you will have syntax highlights which will boost your speed on writing tests.
-
 ![moonworm](https://user-images.githubusercontent.com/19771534/164013435-74a9e816-74ef-4e05-a7e5-1f7f620896e7.jpg)
-
-
-1. `moonworm generate` - cli/ python interface generator for pure `web3` library. In case you prefer not to use `brownie`
+3. `moonworm generate` - cli/ python interface generator for pure `web3` library. In case you prefer not to use `brownie`
 
 ## Setup:
 
 ```bash
-pip install moonworm 
+pip install moonworm
 ```
 
+## Guides
+
+- [How to `watch`](./docs/how-to-watch.md) - How to use `moonworm watch` to get data about smart contract
+activity.
 
 ## Usage:
 
 ### `moonworm watch`:
 
 ```bash
-moonworm watch --abi <Path to abi file> --contract <Contract address> --web3 <Web3 provider url> --start <Start block> --end <End block>    
+moonworm watch --abi <Path to abi file> --contract <Contract address> --web3 <Web3 provider url> --start <Start block> --end <End block>
 ```
 
 Arguments:
@@ -44,7 +45,7 @@ Optional args:
 - `--only-events` Flag, if set: only watches events. Default=`False`
 - `--min-blocks-batch MIN_BLOCKS_BATCH` Minimum number of blocks to batch together. Default=100
 - `--max-blocks-batch MAX_BLOCKS_BATCH` Maximum number of blocks to batch together. Default=1000 **Note**: it is used only in `--only-events` mode
-- 
+-
 
 ### `moonworm generate-brownie`:
 
@@ -61,9 +62,9 @@ Arguments:
 **NOTE**: For better experience put generated files in sub directory of your brownie project. As an example:
 
 1. `cd myBrownieProject`
-2. `moonworm generate-brownie -p . -o generated/ -n MyContract` 
+2. `moonworm generate-brownie -p . -o generated/ -n MyContract`
 
-      3. Run the generated cli of the contract: `python3 generated/Mycontract.py -h` 
+      3. Run the generated cli of the contract: `python3 generated/Mycontract.py -h`
 
 ### `moonworm generate`:
 
@@ -79,7 +80,7 @@ Arguments:
 - `-name/-n NAME` Prefix name for generated files
 - `--cli` Flag to generate cli for given smart contract abi
 
- 
+
 
 ## FAQ:
 

docs/how-to-watch.md

@@ -0,0 +1,239 @@
+# How to `watch`
+
+This is a guide to the `moonworm watch` command, which you can use to gather data about smart contract
+usage on public blockchains.
+
+You do not have to click on any of the links in order to understand this guide. They are just there in
+case you want to dig deeper after you're done with this document.
+
+## Real use case
+
+This guide will show you how you could actually use `moonworm watch` to analyze the activity of the
+Crypto Unicorns NFTs on the Polygon blockchain.
+
+We use Polygon because it is easy to query it without setting up an account anywhere and giving over your
+credit card information.
+
+We use Crypto Unicorns because it has a long and varied history of on-chain activity which is interesting
+to analyze.
+
+## What you will need
+
+### Access to a node
+
+`moonworm watch` gets information about smart contract activity by connecting to blockchain nodes via
+their [JSON-RPC APIs](https://ethereum.org/en/developers/docs/apis/json-rpc/).
+
+All that `moonworm watch` needs in order to do its work is a URL to a JSON-RPC API for the blockchain
+you want to crawl data from.
+
+For Polygon, there is a free, public JSON-RPC URL available for anybody to use: https://polygon-rpc.com.
+
+If you want to query some other blockchain, you can set up an account at [QuickNode](https://www.quicknode.com/),
+[Infura](https://www.infura.io/), or [Alchemy](https://www.alchemy.com/).
+
+Infura and Alchemy are more generous with their free tiers if you're still in the experimental phases.
+QuickNode is the best of the three for data quality. [Read here for more information](https://blog.moonstream.to/2022/08/18/downsides-of-crawling-data-with-infura-and-alchemy/).
+
+### Smart contract address
+
+You will need to know the address of the smart contract that you want to gather data about.
+
+For example, the Crypto Unicorns NFT contract we will be gathering data for in this guide is available
+at [`0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f`](https://polygonscan.com/address/0xdc0479cc5bba033b3e7de9f178607150b3abce1f) on the Polygon network.
+
+You can usually find smart contract addresses on the explorer for their blockchains if their developers
+have published their source code. Try searching for the contract by name on [Etherscan](https://etherscan.io) or
+[Polygonscan](https://polygonscan.com) or [OP Mainnet Explorer](https://optimistic.etherscan.io/) or [BaseScan](https://basescan.org/) or \<insert blockchain explorer for your chain\>.
+
+If you can't find the contract addresses this way, you can find them by interacting with the application
+and recording the addresses that you submit transactions through.
+
+<!-- Deployment block: 21425773 -->
+
+### Smart contract ABI
+
+Smart contracts expose interfaces in the form of JSON objects called ABIs, short for "application binary
+interfaces".
+
+This repository contains some common ABIs in [`moonworm/fixture/abis/`](../moonworm/fixture/abis/).
+
+We will be using [`OwnableERC721.json`](../moonworm/fixture/abis/OwnableERC721.json) to analyze the NFT
+activity of the Crypto Unicorns contract. That particular contract has a lot more functionality than just
+the functionality for ERC721. This just shows that the ABI you use need not be exhaustive. You will only
+be able to decode activity corresponding to the ABI, though.
+
+### Do you need to set the `--poa` flag?
+
+Networks like Polygon have a slightly different block structure compared to Ethereum mainnet and other
+blockchains following its standard. If you call `moonworm watch` to crawl function calls on those chains,
+you may run into an error like this:
+
+```
+web3.exceptions.ExtraDataLengthError: The field extraData is 97 bytes, but should be 32. It is quite likely that you are connected to a POA chain. Refer to http://web3py.readthedocs.io/en/stable/middleware.html#geth-style-proof-of-authority for more details. The full extraData is: HexBytes('0xd682020983626f7288676f312e31372e32856c696e75780000000000000000008399cafaaf72cdb2dc5ff5a8c93b4af476d066c5cc57edfcc0c5e627b1e7893322634137e72db79744e99ba717031de8614eac7452f2e9b7dc2c3f21c30282f201')
+```
+
+If this is the case, you are likely on a chain that requires you to call `moonworm watch` with its `--poa` flag.
+
+This is an immutable property of the blockchain you are working with. For example, on Ethereum mainnet,
+you will never have to set `--poa`. And on Polygon, you will always have to set `--poa`.
+
+For more about this, [read the web3.py documentation](https://web3py.readthedocs.io/en/stable/middleware.html#why-is-geth-poa-middleware-necessary).
+
+In the example in this guide, since we are crawling from Polygon, we will be using `--poa`.
+
+### Do you need to set the `--only-events` flag?
+
+`moonworm watch` gives you the ability to crawl function calls to a contract and events emitted by that
+contract.
+
+Function calls and events are crawled in different ways, and function calls are much slower to crawl than
+event emissions.
+
+Most of the information you want is in the events, and so it is highly recommended to use the `--only-events`
+flag for speed of crawling.
+
+If you want to decode transactions sent to a contract in addition to the events emitted by that contract,
+leave off the `--only-events` flag.
+
+In the example in thise guide, we will be using `--only-events`.
+
+## Building the Crypto Unicorns NFT activity dataset
+
+First, you should set up your Python environment and install `moonworm` using:
+
+```bash
+pip install moonworm[moonstream]
+```
+
+Once installed, let us figure out how to invoke `moonworm watch` to construct our dataset:
+
+```bash
+moonworm watch -h
+```
+
+This should give you output that looks like this:
+
+```
+$ moonworm watch -h
+usage: moonworm watch [-h] -i ABI -c CONTRACT -w WEB3 [--db] [--network {}] [--start START] [--end END] [--poa] [--confirmations CONFIRMATIONS] [--min-blocks-batch MIN_BLOCKS_BATCH]
+                      [--max-blocks-batch MAX_BLOCKS_BATCH] [--batch-size-update-threshold BATCH_SIZE_UPDATE_THRESHOLD] [--only-events] [-o OUTFILE]
+
+options:
+  -h, --help            show this help message and exit
+  -i ABI, --abi ABI     ABI file path or 'erc20' or 'erc721' or cu
+  -c CONTRACT, --contract CONTRACT
+                        Contract address
+  -w WEB3, --web3 WEB3  Web3 provider
+  --db                  Use Moonstream database specified by 'MOONSTREAM_DB_URI' to get blocks/transactions. If set, need also provide --network
+  --network {}          Network name that represents models from db. If --db is set, required
+  --start START, -s START
+                        Block number to start watching from
+  --end END, -e END     Block number at which to end watching
+  --poa                 Pass this flag if u are using PoA network
+  --confirmations CONFIRMATIONS
+                        Number of confirmations to wait for. Default=15
+  --min-blocks-batch MIN_BLOCKS_BATCH
+                        Minimum number of blocks to batch together. Default=100
+  --max-blocks-batch MAX_BLOCKS_BATCH
+                        Maximum number of blocks to batch together. Default=1000
+  --batch-size-update-threshold BATCH_SIZE_UPDATE_THRESHOLD
+                        Number of minimum events before updating batch size (only for --only-events mode). Default=100
+  --only-events         Only watch events. Default=False
+  -o OUTFILE, --outfile OUTFILE
+                        Optional JSONL (JsON lines) file into which to write events and method calls
+```
+
+We already know that we will be using [`OwnableERC721.json`](../moonworm/fixture/abis/OwnableERC721.json)
+as our ABI, so copy that file over to your working directory. I'll assume you have retained the name `OwnableERC721.json`.
+
+We also know:
+- `--contract 0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f`
+- `--web3 "https://polygon-rpc.com"`
+- `--poa`
+- `--only-events`
+
+We can ignore `--db`, `--network`, `--confirmations`, `--min-blocks-batch`, `--max-blocks-batch`, `--batch-size-updated-threshold`.
+These events are useful if you want to run an ongoing crawl in production, but we are just building
+a small dataset as an example that we can play with in this guide.
+
+We need to determine `--start` and `--end`. These are the block number at which our crawl should begin
+and the block number at which our crawl should end. We don't need to specify `--end`. If we do not, the
+crawl will run forever.
+
+For this example, we will crawl the NFT activity in the first 300,000 blocks after the deployment of
+the Crypto Unicorns NFT contract. This roughly corresponds to its first week of activity.
+
+You can find the block at which Crypto Unicorns was deployed by [viewing the contract on Polygonscan](https://polygonscan.com/address/0xdc0479cc5bba033b3e7de9f178607150b3abce1f),
+but `moonworm` gives you a nice way to get teh deployment block directly from your command line:
+
+```bash
+moonworm find-deployment --web3 "https://polygon-rpc.com" --contract 0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f --interval 0.1
+```
+
+This produces the output:
+
+```
+$ moonworm find-deployment --web3 "https://polygon-rpc.com" --contract 0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f --interval 0.1
+21418707
+```
+
+That tells us that the Crypto Unicorns NFT contract was deployed in block 21,418,707. So we will use:
+- `--start 21418707`
+- `--end 21718707`
+
+We will save the dataset to a file called `cryptounicorns-21418707-21718707.json` and we will use the `--outfile cryptounicorns-21418707-21718707.json` argument
+to specify this.
+
+To sum things up, this will be our invocation (it is a multi-line command you can copy and paste into your terminal):
+
+```bash
+moonworm watch \
+    --abi OwnableERC721.json \
+    --contract 0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f \
+    --web3 "https://polygon-rpc.com" \
+    --poa \
+    --only-events \
+    --start 21418707 \
+    --end 21718707 \
+    --outfile cryptounicorns-21418707-21718707.json
+```
+
+On my machine, this took approximately 2 minutes and 55 seconds:
+
+```
+real    2m55.523s
+user    0m21.674s
+sys     0m2.795s
+```
+
+The resulting file contains one emitted event per line and contains 19,906 events:
+
+```bash
+$ wc -l cryptounicorns-21418707-21718707.json
+19906 cryptounicorns-21418707-21718707.json
+```
+
+Each line is a JSON object of the form:
+
+```
+$ tail -n1 cryptounicorns-21418707-21718707.json | jq .
+{
+  "event": "Transfer",
+  "args": {
+    "from": "0x8151EBBf408Af21B9c373199bf31fea61e6ED7F1",
+    "to": "0x1d1503479F7B0DC767CFF26b2E3d232f2BdDF483",
+    "tokenId": 2348
+  },
+  "address": "0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f",
+  "blockNumber": 21718698,
+  "transactionHash": "0x626a25697884c18905642421833e4969802bbae30d9bff5002b749a190aeb7ad",
+  "logIndex": 424
+}
+```
+
+Hopefully this has served to demystify the somewhat intimdating `moonworm watch` command-line tool.
+
+If you have any problems or confusion using `moonworm watch`, please do not hesitate to
+[create an issue](https://github.com/moonstream-to/moonworm/issues/new) or let us know how we can help on
+[Discord](https://discord.gg/K56VNUQGvA).

moonworm/abi.py

@@ -0,0 +1,78 @@
+"""
+ABI utilities, because web3 doesn't do selectors well.
+"""
+
+import glob
+import json
+import os
+from typing import Any, Dict, List, Optional
+
+from web3 import Web3
+
+
+def abi_input_signature(input_abi: Dict[str, Any]) -> str:
+    """
+    Stringifies a function ABI input object according to the ABI specification:
+    https://docs.soliditylang.org/en/v0.5.3/abi-spec.html
+    """
+    input_type = input_abi["type"]
+    if input_type.startswith("tuple"):
+        component_types = [
+            abi_input_signature(component) for component in input_abi["components"]
+        ]
+        input_type = f"({','.join(component_types)}){input_type[len('tuple'):]}"
+    return input_type
+
+
+def abi_function_signature(function_abi: Dict[str, Any]) -> str:
+    """
+    Stringifies a function ABI according to the ABI specification:
+    https://docs.soliditylang.org/en/v0.5.3/abi-spec.html
+    """
+    function_name = function_abi["name"]
+    function_arg_types = [
+        abi_input_signature(input_item) for input_item in function_abi["inputs"]
+    ]
+    function_signature = f"{function_name}({','.join(function_arg_types)})"
+    return function_signature
+
+
+def encode_function_signature(function_abi: Dict[str, Any]) -> Optional[str]:
+    """
+    Encodes the given function (from ABI) with arguments arg_1, ..., arg_n into its 4 byte signature
+    by calculating:
+    keccak256("<function_name>(<arg_1_type>,...,<arg_n_type>")
+
+    If function_abi is not actually a function ABI (detected by checking if function_abi["type"] == "function),
+    returns None.
+    """
+    if function_abi["type"] != "function":
+        return None
+    function_signature = abi_function_signature(function_abi)
+    encoded_signature = Web3.keccak(text=function_signature)[:4]
+    return encoded_signature.hex()
+
+
+def project_abis(project_dir: str) -> Dict[str, List[Dict[str, Any]]]:
+    """
+    Load all ABIs for project contracts and return then in a dictionary keyed by contract name.
+
+    Inputs:
+    - project_dir
+      Path to brownie project
+    """
+    build_dir = os.path.join(project_dir, "build", "contracts")
+    build_files = glob.glob(os.path.join(build_dir, "*.json"))
+
+    abis: Dict[str, List[Dict[str, Any]]] = {}
+
+    for filepath in build_files:
+        contract_name, _ = os.path.splitext(os.path.basename(filepath))
+        with open(filepath, "r") as ifp:
+            contract_artifact = json.load(ifp)
+
+        contract_abi = contract_artifact.get("abi", [])
+
+        abis[contract_name] = contract_abi
+
+    return abis

moonworm/cli.py

@@ -1,9 +1,10 @@
 import argparse
 import json
 import os
-from multiprocessing.sharedctypes import Value
 from pathlib import Path
 from shutil import copyfile
+from types import MappingProxyType
+from typing import Any
 
 from web3.main import Web3
 from web3.middleware import geth_poa_middleware
@@ -11,8 +12,7 @@ from web3.middleware import geth_poa_middleware
 from moonworm.crawler.ethereum_state_provider import Web3StateProvider
 from moonworm.watch import watch_contract
 
-from .contracts import CU, ERC20, ERC721, CULands
-from .crawler.utils import Network
+from .contracts import CU, ERC20, ERC721
 from .deployment import find_deployment_block
 from .generators.basic import (
     generate_contract_cli_content,
@@ -106,16 +106,34 @@ def handle_brownie_generate(args: argparse.Namespace):
 
     project_directory = args.project
     build_directory = os.path.join(project_directory, "build", "contracts")
+    intermediate_dirs: List[str] = []
+    if args.foundry:
+        build_directory = os.path.join(project_directory, "out")
 
     build_file_path = os.path.join(build_directory, f"{args.name}.json")
-    if not os.path.isfile(build_file_path):
-        raise IOError(
-            f"File does not exist: {build_file_path}. Maybe you have to compile the smart contracts?"
-        )
+    if args.foundry:
+        if args.sol_filename is not None:
+            build_file_path = os.path.join(
+                build_directory, args.sol_filename, f"{args.name}.json"
+            )
+            intermediate_dirs.append(args.sol_filename)
+        else:
+            build_file_path = os.path.join(
+                build_directory, f"{args.name}.sol", f"{args.name}.json"
+            )
+            intermediate_dirs.append(f"{args.name}.sol")
+    else:
+        if not os.path.isfile(build_file_path):
+            raise IOError(
+                f"File does not exist: {build_file_path}. Maybe you have to compile the smart contracts?"
+            )
 
     with open(build_file_path, "r") as ifp:
         build = json.load(ifp)
 
+    if args.foundry:
+        build["contractName"] = args.name
+
     relpath = os.path.relpath(project_directory, args.outdir)
     splitted_relpath = [
         f'"{item}"' for item in relpath.split(os.sep)
@@ -129,6 +147,8 @@ def handle_brownie_generate(args: argparse.Namespace):
         args.name,
         splitted_relpath_string,
         prod=args.prod,
+        foundry=args.foundry,
+        intermediate_dirs=intermediate_dirs,
     )
     write_file(interface, os.path.join(args.outdir, args.name + ".py"))
 
@@ -154,6 +174,9 @@ def handle_watch(args: argparse.Namespace) -> None:
     if args.db:
         if args.network is None:
             raise ValueError("Please specify --network")
+
+        from .crawler.networks import Network
+
         network = Network.__members__[args.network]
 
         from .crawler.moonstream_ethereum_state_provider import (
@@ -180,7 +203,6 @@ def handle_watch(args: argparse.Namespace) -> None:
                 state_provider.clear_db_session()
 
     else:
-
         watch_contract(
             web3=web3,
             state_provider=Web3StateProvider(web3),
@@ -215,6 +237,14 @@ def generate_argument_parser() -> argparse.ArgumentParser:
     """
     Generates the command-line argument parser for the "moonworm" command.
     """
+    networks: MappingProxyType[Any, Any] = MappingProxyType({})
+    try:
+        from .crawler.networks import Network
+
+        networks = Network.__members__
+    except Exception:
+        pass
+
     parser = argparse.ArgumentParser(description="Moonworm: Manage your smart contract")
     parser.add_argument(
         "-v",
@@ -256,7 +286,7 @@ def generate_argument_parser() -> argparse.ArgumentParser:
 
     watch_parser.add_argument(
         "--network",
-        choices=Network.__members__,
+        choices=networks,
         default=None,
         help="Network name that represents models from db. If --db is set, required",
     )
@@ -345,12 +375,22 @@ def generate_argument_parser() -> argparse.ArgumentParser:
         "-p",
         "--project",
         required=True,
-        help=f"Path to brownie project directory",
+        help=f"Path to brownie/foundry project directory",
+    )
+    generate_brownie_parser.add_argument(
+        "--foundry",
+        action="store_true",
+        help="Project is using Foundry (if not specified, the assumption is that the project uses brownie)",
+    )
+    generate_brownie_parser.add_argument(
+        "--sol-filename",
+        required=False,
+        help="Name of solidity file containing your contract; required if --foundry, moonworm will look for build artifacts in out/<this filename>, defaults to the contract name if not provided",
     )
     generate_brownie_parser.add_argument(
         "--prod",
         action="store_true",
-        help="Generate shippable python interface, in which abi and bytecode will be included inside the generated file",
+        help="Generate self-contained python interface, in which ABI and bytecode will be included inside the generated file",
     )
     generate_brownie_parser.set_defaults(func=handle_brownie_generate)
 

moonworm/crawler/example.py

@@ -43,7 +43,6 @@ if __name__ == "__main__":
     ]
 
     def run():
-
         if len(sys.argv) < 2:
             print("Usage: eventscanner.py http://your-node-url")
             sys.exit(1)

moonworm/crawler/log_scanner.py

@@ -303,7 +303,6 @@ class EventScanner:
         all_processed = []
 
         for event_type in self.events:
-
             # Callable that takes care of the underlying web3 call
             def _fetch_events(_start_block, _end_block):
                 return _fetch_events_chunk(
@@ -385,7 +384,6 @@ class EventScanner:
         all_processed = []
 
         while current_block <= end_block:
-
             self.state.start_chunk(current_block, chunk_size)
 
             # Print some diagnostics to logs to try to fiddle with real world JSON-RPC API performance

moonworm/crawler/moonstream_ethereum_state_provider.py

@@ -4,26 +4,15 @@ from typing import Any, Dict, List, Optional, Union
 from eth_typing.evm import ChecksumAddress
 from hexbytes.main import HexBytes
 from sqlalchemy.orm import Session
-from sqlalchemy.sql.base import NO_ARG
 from web3 import Web3
 
 from .ethereum_state_provider import EthereumStateProvider
-from .networks import (
-    MODELS,
-    EthereumLabel,
-    EthereumTransaction,
-    MumbaiLabel,
-    MumbaiTransaction,
-    PolygonLabel,
-    PolygonTransaction,
-    XDaiTransaction,
-    yield_db_session_ctx,
-)
-from .utils import Network
+from .networks import MODELS, Network, tx_raw_types
 
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
 
+
 # TODO(yhtiyar) When getting block from db, filter it by `to` address, it will be faster
 # also get blocks in bunch
 class MoonstreamEthereumStateProvider(EthereumStateProvider):
@@ -81,9 +70,7 @@ class MoonstreamEthereumStateProvider(EthereumStateProvider):
 
     @staticmethod
     def _transform_to_w3_tx(
-        tx_raw: Union[
-            EthereumTransaction, MumbaiTransaction, PolygonTransaction, XDaiTransaction
-        ],
+        tx_raw: tx_raw_types,
     ) -> Dict[str, Any]:
         tx = {
             "blockNumber": tx_raw.block_number,

moonworm/crawler/networks.py

@@ -2,50 +2,15 @@ from typing import Dict
 
 try:
     from moonstreamdb.db import yield_db_session_ctx
-    from moonstreamdb.models import (
-        Base,
+    from moonstreamdb.models import (  # state/moonstream_event_state dependency maybe removed in the future
         EthereumBlock,
         EthereumLabel,
-        EthereumTransaction,
-        MumbaiBlock,
-        MumbaiLabel,
-        MumbaiTransaction,
-        PolygonBlock,
-        PolygonLabel,
-        PolygonTransaction,
-        XDaiBlock,
-        XDaiLabel,
-        XDaiTransaction,
     )
+    from moonstreamdb.networks import MODELS, Network, tx_raw_types
+
 except ImportError:
     print("this feature requires moonstreamdb which is not installed")
     print("to enable, run: `pip install moonworm[moonstream]`")
     raise ImportError(
         "moonstreamdb not installed, to install, run: `pip install moonworm[moonstream]`"
     )
-
-
-from .utils import Network
-
-MODELS: Dict[Network, Dict[str, Base]] = {
-    Network.ethereum: {
-        "blocks": EthereumBlock,
-        "labels": EthereumLabel,
-        "transactions": EthereumTransaction,
-    },
-    Network.mumbai: {
-        "blocks": MumbaiBlock,
-        "labels": MumbaiLabel,
-        "transactions": MumbaiTransaction,
-    },
-    Network.polygon: {
-        "blocks": PolygonBlock,
-        "labels": PolygonLabel,
-        "transactions": PolygonTransaction,
-    },
-    Network.xdai: {
-        "blocks": XDaiBlock,
-        "labels": XDaiLabel,
-        "transactions": XDaiTransaction,
-    },
-}

moonworm/crawler/state/moonstream_event_state.py

@@ -1,7 +1,7 @@
-from moonstreamdb.models import EthereumBlock, EthereumLabel
 from sqlalchemy.orm import Query, Session
 from web3 import Web3
 
+from ..networks import EthereumBlock, EthereumLabel
 from .event_scanner_state import EventScannerState
 
 BLOCK_TIMESTAMP_CACHE = {}

moonworm/crawler/utils.py

@@ -1,8 +0,0 @@
-from enum import Enum
-
-
-class Network(Enum):
-    ethereum = "ethereum"
-    polygon = "polygon"
-    mumbai = "mumbai"
-    xdai = "xdai"

moonworm/deployment.py

@@ -3,6 +3,7 @@ Allows users to inspect the conditions under which a smart contract was deployed
 
 The entrypoint for this functionality is [`find_deployment_block`][moonworm.deployment.find_deployment_block].
 """
+
 import logging
 import os
 import time

moonworm/generators/basic.py

@@ -17,6 +17,7 @@ import black.mode
 import inflection
 import libcst as cst
 
+from ..abi import encode_function_signature
 from ..version import MOONWORM_VERSION
 
 CONTRACT_TEMPLATE_PATH = os.path.join(os.path.dirname(__file__), "contract.py.template")
@@ -94,11 +95,11 @@ def normalize_abi_name(name: str) -> str:
 
 
 def python_type(evm_type: str) -> List[str]:
-    if evm_type.endswith("]"):
+    if evm_type.endswith("]") and not evm_type.endswith("][]"):
         return ["List"]
-    if evm_type.startswith(("uint", "int")):
+    if evm_type.startswith(("uint", "int")) and "[" not in evm_type:
         return ["int"]
-    if evm_type.startswith(("int", "int")):
+    if evm_type.startswith(("int", "int")) and "[" not in evm_type:
         return ["int"]
     elif evm_type.startswith("bytes"):
         return ["bytes"]
@@ -186,7 +187,9 @@ PROTECTED_ARG_NAMES: Set[str] = {
 }
 
 
-def function_spec(function_abi: Dict[str, Any]) -> Dict[str, Dict[str, Any]]:
+def function_spec(
+    function_abi: Dict[str, Any], is_overloaded: bool = False
+) -> Dict[str, Dict[str, Any]]:
     """
     Accepts function interface definitions from smart contract ABIs. An example input:
     {
@@ -226,6 +229,11 @@ def function_spec(function_abi: Dict[str, Any]) -> Dict[str, Dict[str, Any]]:
         ],
         "transact": False,
     }
+
+    If the caller provides `is_overloaded`, the function signature is suffixed to the method and CLI
+    names with the appropriate dashes. No changes are made to the ABI name itself. For example, generated
+    brownie interfaces rely on brownie's internal deduplication logic (based on argument types) to delegate
+    behavior to the right version of the brownie contract method.
     """
     abi_name = function_abi.get("name")
     if abi_name is None:
@@ -235,6 +243,11 @@ def function_spec(function_abi: Dict[str, Any]) -> Dict[str, Dict[str, Any]]:
     function_name = normalize_abi_name(underscored_name)
     cli_name = inflection.dasherize(underscored_name)
 
+    if is_overloaded:
+        signature = encode_function_signature(function_abi)
+        function_name += f"_{signature}"
+        cli_name += f"-{signature}"
+
     default_input_name = "arg"
     default_counter = 1
 
@@ -275,7 +288,8 @@ def function_spec(function_abi: Dict[str, Any]) -> Dict[str, Dict[str, Any]]:
         inputs.append(input_spec)
 
     transact = True
-    if function_abi.get("stateMutability") == "view":
+    state_mutability = function_abi.get("stateMutability", "").lower()
+    if state_mutability == "view" or state_mutability == "pure":
         transact = False
 
     spec = {
@@ -327,7 +341,6 @@ def generate_contract_constructor_function(
 
 
 def generate_contract_function(func_object: Dict[str, Any]) -> cst.FunctionDef:
-
     default_param_name = "arg"
     default_counter = 1
     func_params = []

moonworm/generators/brownie.py

@@ -7,7 +7,7 @@ The entrypoint to code generation is [`generate_brownie_interface`][moonworm.gen
 import copy
 import logging
 import os
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Set
 
 import libcst as cst
 from libcst._nodes.statement import SimpleStatementLine
@@ -21,18 +21,38 @@ BROWNIE_INTERFACE_TEMPLATE_PATH = os.path.join(
 BROWNIE_INTERFACE_PROD_TEMPLATE_PATH = os.path.join(
     os.path.dirname(__file__), "brownie_contract_prod.py.template"
 )
+BROWNIE_INTERFACE_FOUNDRY_TEMPLATE_PATH = os.path.join(
+    os.path.dirname(__file__), "brownie_contract_foundry.py.template"
+)
 try:
     with open(BROWNIE_INTERFACE_TEMPLATE_PATH, "r") as ifp:
         BROWNIE_INTERFACE_TEMPLATE = ifp.read()
     with open(BROWNIE_INTERFACE_PROD_TEMPLATE_PATH, "r") as ifp:
         BROWNIE_INTERFACE_PROD_TEMPLATE = ifp.read()
+    with open(BROWNIE_INTERFACE_FOUNDRY_TEMPLATE_PATH, "r") as ifp:
+        BROWNIE_INTERFACE_FOUNDRY_TEMPLATE = ifp.read()
 except Exception as e:
     logging.warn(
-        f"WARNING: Could not load cli template from ({BROWNIE_INTERFACE_TEMPLATE_PATH})/({BROWNIE_INTERFACE_PROD_TEMPLATE_PATH}):"
+        f"WARNING: Could not load cli template from ({BROWNIE_INTERFACE_TEMPLATE_PATH})/({BROWNIE_INTERFACE_PROD_TEMPLATE_PATH})/({BROWNIE_INTERFACE_FOUNDRY_TEMPLATE_PATH}):"
     )
     logging.warn(e)
 
 
+def get_overloaded_functions(abi: List[Dict[str, Any]]) -> Set[str]:
+    """
+    Return a set containing the function names of all overloaded functions.
+    """
+    function_name_counters: Dict[str, int] = {}
+    for item in abi:
+        if item["type"] != "function":
+            continue
+        if item["name"] in function_name_counters:
+            function_name_counters[item["name"]] += 1
+        else:
+            function_name_counters[item["name"]] = 1
+    return {name for name, count in function_name_counters.items() if count > 1}
+
+
 def generate_brownie_contract_class(
     abi: List[Dict[str, Any]],
     contract_name: str,
@@ -78,6 +98,8 @@ def generate_brownie_contract_class(
     contract_constructor = get_constructor(abi)
     contract_constructor["name"] = "constructor"
 
+    overloaded_functions = get_overloaded_functions(abi)
+
     class_functions = (
         [class_constructor]
         + [
@@ -86,7 +108,9 @@ def generate_brownie_contract_class(
             generate_verify_contract(),
         ]
         + [
-            generate_brownie_contract_function(function)
+            generate_brownie_contract_function(
+                function, function.get("name", "") in overloaded_functions
+            )
             for function in abi
             if function["type"] == "function"
         ]
@@ -187,8 +211,10 @@ def generate_assert_contract_is_instantiated() -> cst.FunctionDef:
     return function_def
 
 
-def generate_brownie_contract_function(func_object: Dict[str, Any]) -> cst.FunctionDef:
-    spec = function_spec(func_object)
+def generate_brownie_contract_function(
+    func_object: Dict[str, Any], is_overloaded: bool = False
+) -> cst.FunctionDef:
+    spec = function_spec(func_object, is_overloaded)
     func_params = []
     func_params.append(cst.Param(name=cst.Name("self")))
 
@@ -214,7 +240,6 @@ def generate_brownie_contract_function(func_object: Dict[str, Any]) -> cst.Funct
             f"return self.contract.{func_raw_name}({','.join(param_names)})"
         )
     else:
-
         func_params.append(
             cst.Param(
                 name=cst.Name(value="block_number"),
@@ -516,7 +541,7 @@ def generate_verify_contract_handler(contract_name: str) -> Optional[cst.Functio
 
 
 def generate_cli_handler(
-    function_abi: Dict[str, Any], contract_name: str
+    function_abi: Dict[str, Any], contract_name: str, is_overloaded: bool = False
 ) -> Optional[cst.FunctionDef]:
     """
     Generates a handler which translates parsed command line arguments to method calls on the generated
@@ -525,7 +550,7 @@ def generate_cli_handler(
     Returns None if it is not appropriate for the given function to have a handler (e.g. fallback or
     receive). constructor is handled separately with a deploy handler.
     """
-    spec = function_spec(function_abi)
+    spec = function_spec(function_abi, is_overloaded)
     function_name = spec["method"]
 
     function_body_raw: List[cst.CSTNode] = []
@@ -540,7 +565,10 @@ def generate_cli_handler(
 
     # If a transaction is required, extract transaction parameters from CLI
     requires_transaction = True
-    if function_abi["stateMutability"] == "view":
+    if (
+        function_abi["stateMutability"] == "view"
+        or function_abi["stateMutability"] == "pure"
+    ):
         requires_transaction = False
 
     if requires_transaction:
@@ -734,8 +762,16 @@ def generate_cli_generator(
         "inputs": [],
     }
 
+    overloaded_functions = get_overloaded_functions(abi)
+
     specs: List[Dict[str, Any]] = [constructor_spec, verify_contract_spec]
-    specs.extend([function_spec(item) for item in abi if item["type"] == "function"])
+    specs.extend(
+        [
+            function_spec(item, item.get("name", "") in overloaded_functions)
+            for item in abi
+            if item["type"] == "function"
+        ]
+    )
 
     for spec in specs:
         subparser_statements: List[SimpleStatementLine] = [cst.Newline()]
@@ -803,6 +839,22 @@ def generate_cli_generator(
                         value=cst.parse_expression("eval"),
                     ),
                 )
+            elif param["type"] == "tuple[]":
+                call_args.append(
+                    cst.Arg(
+                        keyword=cst.Name(value="type"),
+                        value=cst.parse_expression("eval"),
+                    ),
+                )
+            elif param["type"] == "Any":
+                # In general case, we just use a Python `eval` to parse the input from the command line.
+                # This is similar to the way we handle `tuple` arguments.
+                call_args.append(
+                    cst.Arg(
+                        keyword=cst.Name(value="type"),
+                        value=cst.parse_expression("eval"),
+                    ),
+                )
 
             add_argument_call = cst.Call(
                 func=cst.Attribute(
@@ -893,9 +945,15 @@ def generate_brownie_cli(
         add_deploy_handler,
         add_verify_contract_handler,
     ]
+
+    overloaded_functions = get_overloaded_functions(abi)
     handlers.extend(
         [
-            generate_cli_handler(function_abi, contract_name)
+            generate_cli_handler(
+                function_abi,
+                contract_name,
+                function_abi.get("name") in overloaded_functions,
+            )
             for function_abi in abi
             if function_abi.get("type") == "function"
             and function_abi.get("name") is not None
@@ -916,6 +974,8 @@ def generate_brownie_interface(
     cli: bool = True,
     format: bool = True,
     prod: bool = False,
+    foundry: bool = True,
+    intermediate_dirs: Optional[List[str]] = None,
 ) -> str:
     """
     Generates Python code which allows you to interact with a smart contract with a given ABI, build data, and a given name.
@@ -942,6 +1002,11 @@ def generate_brownie_interface(
     7. `prod`: If True, creates a self-contained file. Generated code will not require reference to an
     existing brownie project at its runtime.
 
+    8. `foundry`: If True, assumes a Foundry project structure.
+
+    9. intermediate_dirs: Currently only used for Foundry projects. Path to build file via intermediate
+    build subdirectory which takes the name of the Solidity file that the contract is implemented in.
+
 
     ## Outputs
     The generated code as a string.
@@ -965,6 +1030,14 @@ def generate_brownie_interface(
             contract_body=contract_body,
             moonworm_version=MOONWORM_VERSION,
         )
+    elif foundry:
+        content = BROWNIE_INTERFACE_FOUNDRY_TEMPLATE.format(
+            contract_body=contract_body,
+            moonworm_version=MOONWORM_VERSION,
+            relative_path=relative_path,
+            build_subdir=intermediate_dirs[0],
+        )
+
     else:
         content = BROWNIE_INTERFACE_TEMPLATE.format(
             contract_body=contract_body,

moonworm/generators/brownie_contract.py.template

@@ -1,4 +1,4 @@
-# Code generated by moonworm : https://github.com/bugout-dev/moonworm
+# Code generated by moonworm : https://github.com/moonstream-to/moonworm
 # Moonworm version : {moonworm_version}
 
 import argparse

moonworm/generators/brownie_contract_foundry.py.template

@@ -0,0 +1,85 @@
+# Code generated by moonworm : https://github.com/moonstream-to/moonworm
+# Moonworm version : {moonworm_version}
+
+import argparse
+import json
+import os
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Union
+
+from brownie import Contract, network, project
+from brownie.network.contract import ContractContainer
+from eth_typing.evm import ChecksumAddress
+
+
+PROJECT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), {relative_path}))
+BUILD_DIRECTORY = os.path.join(PROJECT_DIRECTORY, "out", "{build_subdir}")
+
+def boolean_argument_type(raw_value: str) -> bool:
+    TRUE_VALUES = ["1", "t", "y", "true", "yes"]
+    FALSE_VALUES = ["0", "f", "n", "false", "no"]
+
+    if raw_value.lower() in TRUE_VALUES:
+        return True
+    elif raw_value.lower() in FALSE_VALUES:
+        return False
+
+    raise ValueError(
+        f"Invalid boolean argument: {{raw_value}}. Value must be one of: {{','.join(TRUE_VALUES + FALSE_VALUES)}}"
+    )
+
+def bytes_argument_type(raw_value: str) -> str:
+    return raw_value
+
+def get_abi_json(abi_name: str) -> List[Dict[str, Any]]:
+    abi_full_path = os.path.join(BUILD_DIRECTORY, f"{{abi_name}}.json")
+    if not os.path.isfile(abi_full_path):
+        raise IOError(
+            f"File does not exist: {{abi_full_path}}. Maybe you have to compile the smart contracts?"
+        )
+
+    with open(abi_full_path, "r") as ifp:
+        build = json.load(ifp)
+
+    abi_json = build.get("abi")
+    if abi_json is None:
+        raise ValueError(f"Could not find ABI definition in: {{abi_full_path}}")
+
+    return abi_json
+
+
+def contract_from_build(abi_name: str) -> ContractContainer:
+    # This is workaround because brownie currently doesn't support loading the same project multiple
+    # times. This causes problems when using multiple contracts from the same project in the same
+    # python project.
+    PROJECT = project.main.Project("moonworm", Path(PROJECT_DIRECTORY))
+
+    abi_full_path = os.path.join(BUILD_DIRECTORY, f"{{abi_name}}.json")
+    if not os.path.isfile(abi_full_path):
+        raise IOError(
+            f"File does not exist: {{abi_full_path}}. Maybe you have to compile the smart contracts?"
+        )
+
+    with open(abi_full_path, "r") as ifp:
+        foundry_build = json.load(ifp)
+
+    build = {{
+        "type": "contract",
+        "ast": foundry_build["ast"],
+        "abi": foundry_build["abi"],
+        "contractName": abi_name,
+        "compiler": {{
+            "version": foundry_build["metadata"]["compiler"]["version"],
+        }},
+        "language": foundry_build["metadata"]["language"],
+        "bytecode": foundry_build["bytecode"]["object"],
+        "sourceMap": foundry_build["bytecode"]["sourceMap"],
+        "deployedBytecode": foundry_build["deployedBytecode"]["object"],
+        "deployedSourceMap": foundry_build["deployedBytecode"]["sourceMap"],
+        "pcMap": {{}},
+    }}
+
+    return ContractContainer(PROJECT, build)
+
+
+{contract_body}

moonworm/generators/brownie_contract_prod.py.template

@@ -1,4 +1,4 @@
-# Code generated by moonworm : https://github.com/bugout-dev/moonworm
+# Code generated by moonworm : https://github.com/moonstream-to/moonworm
 # Moonworm version : {moonworm_version}
 
 import argparse

moonworm/generators/cli.py.template

@@ -1,4 +1,4 @@
-# Code generated by moonworm : https://github.com/bugout-dev/moonworm
+# Code generated by moonworm : https://github.com/moonstream-to/moonworm
 # Moonworm version : {moonworm_version}
 
 import argparse
@@ -23,7 +23,7 @@ for abi_item in CONTRACT_ABI:
         CONTRACT_FUNCTIONS[abi_item["name"]] = abi_item
     if abi_item["type"] == "constructor":
         CONTRACT_FUNCTIONS["constructor"] = abi_item
-if CONTRACT_FUNCTIONS["constructor"] is None:
+if CONTRACT_FUNCTIONS.get("constructor", "") == "":
     CONTRACT_FUNCTIONS["constructor"] = {{"inputs" : []}}
 
 def init_web3(ipc_path: str) -> Web3:

moonworm/generators/contract.py.template

@@ -1,4 +1,4 @@
-# Code generated by moonworm : https://github.com/bugout-dev/moonworm
+# Code generated by moonworm : https://github.com/moonstream-to/moonworm
 # Moonworm version : {moonworm_version}
 import json
 import os

moonworm/tests/test_tester_provider.py

@@ -52,7 +52,6 @@ class MoonwormEthTesterTestCase(unittest.TestCase):
         send_value,
         tx_receipt,
     ):
-
         assert receiver_current_balance == receiver_previous_balance + send_value
         assert (
             sender_current_balance
@@ -60,7 +59,6 @@ class MoonwormEthTesterTestCase(unittest.TestCase):
         )
 
     def test_submit_transaction(self) -> None:
-
         sender = Web3.toChecksumAddress(PK_ADDRESS)
         self.web3.eth.send_transaction
         receiver = Web3.toChecksumAddress(self.web3.eth.accounts[1])
@@ -95,7 +93,6 @@ class MoonwormEthTesterTestCase(unittest.TestCase):
         )
 
     def test_submit_signed_transaction(self) -> None:
-
         sender = Web3.toChecksumAddress(PK_ADDRESS)
         self.web3.eth.send_transaction
         receiver = Web3.toChecksumAddress(self.web3.eth.accounts[1])

moonworm/version.py

@@ -1 +1 @@
-MOONWORM_VERSION = "0.5.3"
+MOONWORM_VERSION = "0.9.0"

moonworm/watch.py

@@ -113,6 +113,8 @@ def watch_contract(
     None. Results are printed to stdout and, if an outfile has been provided, also to the file.
     """
 
+    contract_abi = [item for item in contract_abi if item.get("name") is not None]
+
     current_batch_size = min_blocks_batch
     state = MockState()
     crawler = FunctionCallCrawler(

moonworm/web3_util.py

@@ -53,7 +53,6 @@ def get_nonce(web3: Web3, address: ChecksumAddress) -> Nonce:
 def submit_transaction(
     web3: Web3, transaction: Union[TxParams, Any], signer_private_key: str
 ) -> HexBytes:
-
     """
     Signs and submits json transaction to blockchain from the name of signer
     """

setup.py

@@ -15,14 +15,14 @@ setup(
         "black",
         "inflection",
         "libcst",
-        "pysha3<2.0.0,>=1.0.0",
+        "pysha3<2.0.0,>=1.0.0; python_version < '3.6'",
         "tqdm",
-        "typing-extensions<4,>=3.7.4",
-        "web3 >=5.27.0",
+        "typing-extensions",
+        "web3>=5.27.0",
     ],
     extras_require={
-        "dev": ["isort", "mypy", "wheel", "web3[tester] >=5.27.0"],
-        "moonstream": ["moonstreamdb >= 0.3.2"],
+        "dev": ["isort", "mypy", "wheel", "web3>=5.27.0"],
+        "moonstream": ["moonstreamdb>=0.3.3"],
         "distribute": ["setuptools", "twine", "wheel"],
     },
     description="moonworm: Generate a command line interface to any Ethereum smart contract",
@@ -37,7 +37,7 @@ setup(
         "Topic :: Software Development :: Libraries",
     ],
     python_requires=">=3.6",
-    url="https://github.com/bugout-dev/moonworm/",
+    url="https://github.com/moonstream-to/moonworm/",
     entry_points={"console_scripts": ["moonworm=moonworm.cli:main"]},
     include_package_data=True,
 )