Massive Download [ETH] smart contracts

[chrusso]

I'm looking for the most valid path to download and keep downloading every single smart contract on the ethereum network.

As I've been programming for a few years, I've been thinking about using cURL to make this. But I'm aware this is probably very outdated as it should be possible to be done somehow using the blockchain...

I'm willing to store all the contracts in a traditional MySQL database, in order to run multiple security audits on them. Is there any other database engine compatible with PHP that you would recommend to store them?

How would you approach this?

Kind regards;
Chris

[starmyc]

I'm looking for the most valid path to download and keep downloading every single smart contract on the ethereum network.

As I've been programming for a few years, I've been thinking about using cURL to make this. But I'm aware this is probably very outdated as it should be possible to be done somehow using the blockchain...

I'm willing to store all the contracts in a traditional MySQL database, in order to run multiple security audits on them. Is there any other database engine compatible with PHP that you would recommend to store them?

How would you approach this?

You won't be able to use the Ethereum blockchain to retrieve contracts because they are stored in their compiled form in it, and you would need to disassemble them unfortunately.

A lot of contract creators upload their contracts on ethercan.io though. But etherscan.io doesn't provide an API to download them neither.

[chrusso]

Hi starmyc,

Thanks for helping to sort out this equation, which I find really interesting.

I have found that etherscan.io actually hold information about the solidity source code of each contract. However I don't really know if this information is the actual code before being compiled to bytecode, or if it's a simple reconstruction of the code using a decompiler, which isn't the same for my goal.

https://etherscan.io/address/0xa701122c1b67220a8b6883d03c8ad67896b12466#code

Seems like etherscan.io has an API as well, which is available here: https://etherscan.io/apis

However it's limited to 5 requests per second, and the options are quite limited, hence it will probably take an ethernity to clone all desired information.

I have found that it's possible to get closer to the goal, using geth, and then processing the data, moreover, I have been recommended to use QuickBlocks (https://quickblocks.io/) which is mostly based on C++ and seems to work quite fast.

Yet, the code is compiled and I'm not yet aware of any pre-coded function that allows to clone each smart contract into a db (even the compiled version).

I've found as well that it's possible to decompile the code, at least partially using comaeio/porosity (https://github.com/comaeio/porosity) which makes a quick replacement based on the detected expressions. As following:

• Hash: 0x0A19B14A (trade) (1 references)
• Hash: 0x0B927666 (order) (1 references)
• Hash: 0x19774D43 (orderFills) (1 references)
• Hash: 0x278B8C0E (cancelOrder) (1 references)
• Hash: 0x2E1A7D4D (withdraw) (1 references)
• Hash: 0x338B5DEA (depositToken) (1 references)
• Hash: 0x46BE96C3 (amountFilled) (1 references)
• Hash: 0x508493BC (tokens) (1 references)
• Hash: 0x54D03B5C (changeFeeMake) (1 references)
• Hash: 0x57786394 (feeMake) (1 references)
• Hash: 0x5E1D7AE4 (changeFeeRebate) (1 references)
• Hash: 0x65E17C9D (feeAccount) (1 references)
• Hash: 0x6C86888B (testTrade) (1 references)
• Hash: 0x71FFCB16 (changeFeeAccount) (1 references)
• Hash: 0x731C2F81 (feeRebate) (1 references)
• Hash: 0x8823A9C0 (changeFeeTake) (1 references)
• Hash: 0x8F283970 (changeAdmin) (1 references)
• Hash: 0x9E281A98 (withdrawToken) (1 references)
• Hash: 0xBB5F4629 (orders) (1 references)
• Hash: 0xC281309E (feeTake) (1 references)
• Hash: 0xD0E30DB0 (deposit) (1 references)
• Hash: 0xE8F6BC2E (changeAccountLevelsAddr) (1 references)
• Hash: 0xF3412942 (accountLevelsAddr) (1 references)
• Hash: 0xF7888AEC (balanceOf) (1 references)
• Hash: 0xF851A440 (admin) (1 references)
• Hash: 0xFB6E155F (availableVolume) (1 references)

I'm still wondering exactly which functions or which modifications might be required to make QuickBlocks store the desired information into BigQuery or MongoDB.

The idea seems possible, however not simple to achieve.

[chrusso]

I'll proceed to modify the source code or make a small extension, and try with a small portion of the blockchain, if it works, then it's just about scaling. I'll report here in case it works.

[starmyc]

I have found that etherscan.io actually hold information about the solidity source code of each contract. However I don't really know if this information is the actual code before being compiled to bytecode, or if it's a simple reconstruction of the code using a decompiler, which isn't the same for my goal.

Contract owners can upload their contract to etherscan.io, which compile it to check that the contract is the same in the blockchain, before showing it on their site. This is why you'll find contracts with source code and others without on it.

I never studied contract decompilation or disassembling though..

[chrusso]

@starmyc thanks for the feedback once again.

By the way, I'm still in the process of setting up a full node in order to store the data in a traditional database.

Do you have any idea how this can be achieved? I'm running geth, and it has been doing operations for about 2 hours already:

INFO [05-08|17:48:36] Imported new block receipts              count=1    elapsed=202.267ms number=3039741 hash=814a44…a35dcb size=4.00B    ignored=0
INFO [05-08|17:48:36] Imported new state entries               count=1034 elapsed=7.078ms   processed=4939460 pending=7314  retry=0   duplicate=436 unexpected=752
INFO [05-08|17:48:37] Imported new block receipts              count=209  elapsed=478.939ms number=3039950 hash=b41477…fa0ad7 size=890.56kB ignored=0
INFO [05-08|17:48:39] Imported new block receipts              count=209  elapsed=779.399ms number=3040159 hash=e2d5f6…13db5d size=2.15mB   ignored=0
INFO [05-08|17:48:40] Imported new block receipts              count=1090 elapsed=1.511s    number=3041249 hash=a17fe9…dc8461 size=4.93mB   ignored=0
INFO [05-08|17:48:41] Imported new block headers               count=2048 elapsed=5.037s    number=3043836 hash=2b1435…cbdc35 ignored=0
INFO [05-08|17:48:41] Imported new block receipts              count=539  elapsed=738.282ms number=3041788 hash=138734…888c50 size=2.24mB   ignored=0
INFO [05-08|17:48:41] Imported new state entries               count=384  elapsed=800.763µs processed=4939844 pending=11030 retry=2   duplicate=436 unexpected=752
INFO [05-08|17:48:44] Imported new block receipts              count=393  elapsed=970.336ms number=3042181 hash=218dfe…ca188e size=2.35mB   ignored=0

I'm not even sure what it is doing...

Assuming it's synchronising but I honestly have no idea what's going on. And if so, once the process is done... where is the data being stored?