Bitcoin Forum

Bitcoin => Project Development => Topic started by: Kefkius on September 01, 2015, 11:25:16 PM



Title: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 01, 2015, 11:25:16 PM
Hashmal
Transaction Script IDE (Github (https://github.com/mazaclub/hashmal) | IRC (https://kiwiirc.com/client/irc.freenode.net/?#hashmal))

Hashmal is an IDE for Bitcoin transaction scripts. Its purpose is to make it easier to write, evaluate, and learn about transaction scripts. There are very few tools that help people learn about how transaction scripts work; hopefully Hashmal can do this.

Demonstration of stack evaluation.

Hashmal is intended for cryptocurrency developers and power users. It uses python-bitcoinlib (https://github.com/petertodd/python-bitcoinlib) for most of its functionality.

Key Features

  • Write scripts in human-readable form.
  • Observe the stack as scripts are executed.
  • See explanations of script operations as they are evaluated.

Notice

Hashmal is a project in its infancy. Please be careful if you intend to use any scripts you make in Hashmal with an actual blockchain. This is alpha software.

You can download Hashmal from Github (https://github.com/mazaclub/hashmal).


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Uptrenda on September 01, 2015, 11:40:51 PM
First POST and OP congratulations because you just solved a problem that every Bitcoin developer of the past few years has grappled with. I'm definitely going to be checking this out.

Edit: seems like an easy improvement to this is add another field for raw transactions to check that the TX is valid and to inspect the script works as intended. I just noticed what you wrote about multi-sig but should be quite easy to code this (pybitcoin-lib already has all the code.)


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 03, 2015, 03:36:44 AM
First POST and OP congratulations because you just solved a problem that every Bitcoin developer of the past few years has grappled with. I'm definitely going to be checking this out.

Edit: seems like an easy improvement to this is add another field for raw transactions to check that the TX is valid and to inspect the script works as intended. I just noticed what you wrote about multi-sig but should be quite easy to code this (pybitcoin-lib already has all the code.)

Thanks! The thing about CHECKSIG is that the Stack View tool was getting pretty crowded so I didn't put in any way to include a transaction. But you're definitely right in that it wouldn't be very difficult to do. I think the best way to accomplish that is to create a tool called "Transaction Deserializer" or some such, and have it be able to interact with the Stack View tool. I'll give that a shot.


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Luke-Jr on September 03, 2015, 03:52:36 AM
Would be nice if it could use libbitcoinconsensus for evaluation just to avoid any potential bugs.
I recall one time when Valgrind messed up an x86 instruction, it totally confused my debugging efforts for hours. :(
... and that was without money at stake ... :|

Disclosure: Shameless try-to-build-demand for reopening Bitcoin Core pull request #6178, "Refactor EvalScript into a CScriptExecution class, so single-stepping can be done" (https://github.com/bitcoin/bitcoin/pull/6178)


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Uptrenda on September 03, 2015, 04:17:10 AM
First POST and OP congratulations because you just solved a problem that every Bitcoin developer of the past few years has grappled with. I'm definitely going to be checking this out.

Edit: seems like an easy improvement to this is add another field for raw transactions to check that the TX is valid and to inspect the script works as intended. I just noticed what you wrote about multi-sig but should be quite easy to code this (pybitcoin-lib already has all the code.)

Thanks! The thing about CHECKSIG is that the Stack View tool was getting pretty crowded so I didn't put in any way to include a transaction. But you're definitely right in that it wouldn't be very difficult to do. I think the best way to accomplish that is to create a tool called "Transaction Deserializer" or some such, and have it be able to interact with the Stack View tool. I'll give that a shot.

Oh man, where were you 2 years ago when I was getting started? I'm wishing good things for this project  :)


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 03, 2015, 07:15:39 AM
Would be nice if it could use libbitcoinconsensus for evaluation just to avoid any potential bugs.
I recall one time when Valgrind messed up an x86 instruction, it totally confused my debugging efforts for hours. :(
... and that was without money at stake ... :|

Disclosure: Shameless try-to-build-demand for reopening Bitcoin Core pull request #6178, "Refactor EvalScript into a CScriptExecution class, so single-stepping can be done" (https://github.com/bitcoin/bitcoin/pull/6178)

I hadn't thought of that. It's a good idea, albeit one that would complicate starting Hashmal (e.g. compiling the C++ extension). The current method of stepping through scripts - via re-implementing EvalScript() from python-bitcoinlib - is rather inelegant. I'm also relatively unfamiliar with C++ (as opposed to C) extensions in Python. Anyway, there probably should be a better implementation of script evaluation, so that's currently the only option on the table.

First POST and OP congratulations because you just solved a problem that every Bitcoin developer of the past few years has grappled with. I'm definitely going to be checking this out.

Edit: seems like an easy improvement to this is add another field for raw transactions to check that the TX is valid and to inspect the script works as intended. I just noticed what you wrote about multi-sig but should be quite easy to code this (pybitcoin-lib already has all the code.)

Thanks! The thing about CHECKSIG is that the Stack View tool was getting pretty crowded so I didn't put in any way to include a transaction. But you're definitely right in that it wouldn't be very difficult to do. I think the best way to accomplish that is to create a tool called "Transaction Deserializer" or some such, and have it be able to interact with the Stack View tool. I'll give that a shot.

Oh man, where were you 2 years ago when I was getting started? I'm wishing good things for this project  :)

Thanks for the good wishes. :)


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Luke-Jr on September 03, 2015, 07:33:38 AM
Would be nice if it could use libbitcoinconsensus for evaluation just to avoid any potential bugs.
I recall one time when Valgrind messed up an x86 instruction, it totally confused my debugging efforts for hours. :(
... and that was without money at stake ... :|

Disclosure: Shameless try-to-build-demand for reopening Bitcoin Core pull request #6178, "Refactor EvalScript into a CScriptExecution class, so single-stepping can be done" (https://github.com/bitcoin/bitcoin/pull/6178)

I hadn't thought of that. It's a good idea, albeit one that would complicate starting Hashmal (e.g. compiling the C++ extension). The current method of stepping through scripts - via re-implementing EvalScript() from python-bitcoinlib - is rather inelegant. I'm also relatively unfamiliar with C++ (as opposed to C) extensions in Python. Anyway, there probably should be a better implementation of script evaluation, so that's currently the only option on the table.
libbitcoinconsensus, while it does use C++, exports only a C API :)
Should be able to just use ctypes I would think (with stock libbitcoinconsensus lib).
(Of course, this only works with #6178 merged :()


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: sAt0sHiFanClub on September 03, 2015, 08:07:40 AM
Well done, this looks really good. I think you are going in the right direction with this.  ;)


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: walletrecoveryservices on September 03, 2015, 08:35:08 AM
Nice idea OP. Thanks for your contribution to the community.


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: anton000 on September 03, 2015, 08:47:00 AM
Wow. Pretty awesome contribution op.

If this were only available 2 years go... Damn :D


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: antanst on September 03, 2015, 09:06:16 AM
Very nice, thank you!

Should there be a list of the Python dependencies somewhere, like a requirements file?


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: SteamGamesBTC.com on September 03, 2015, 09:58:48 AM
Python isn't so bad at all, Electrum is also written in Python and works. ;-) However the Windows version of Electrum is nicely compiled to the one executable file. Do you plan the same?


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 03, 2015, 10:34:46 AM
Very nice, thank you!

Should there be a list of the Python dependencies somewhere, like a requirements file?


Yeah, there's a requirements.txt now.

Python isn't so bad at all, Electrum is also written in Python and works. ;-) However the Windows version of Electrum is nicely compiled to the one executable file. Do you plan the same?

When it's a little more mature, definitely.


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: CanaryInTheMine on September 03, 2015, 02:24:25 PM
Great tool!! It's very useful, wish it was made sooner :)


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: georgem on September 03, 2015, 03:40:26 PM
exactly the tool I need right now. Thanks!


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 03, 2015, 05:46:05 PM
Thanks for the feedback. Repo transferred to its new home (https://github.com/mazaclub/hashmal). We're now in #hashmal on freenode.


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: mustyoshi on September 03, 2015, 08:03:17 PM
Any chance of porting this to Javascript?


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 04, 2015, 12:05:35 AM
Any chance of porting this to Javascript?

Not really, no. The whole thing was designed with Python in mind.

Anyway, there are now some basic usage docs here (https://github.com/mazaclub/hashmal/blob/master/doc/usage.adoc). I've also put in a basic Transaction Deserializer tool which can be upgraded with the functionality talked about earlier in the thread at a later time.

The most significant update is that you can now specify a "spending transaction" in the Stack Evaluator, and select one of its inputs. That input's scriptSig will be pushed to the stack before your script starts. "OP_CHECKSIG" and "OP_CHECKSIGVERIFY" should work now.

Pretty soon I'll have to start incrementing the version number and providing changelogs.


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: chrisrico on September 04, 2015, 05:59:02 PM
I'm having difficulty getting it to run. Here's what I'm getting after installing from git on OSX Yosemite.

Code:
$ hashmal
Traceback (most recent call last):
  File "/usr/local/bin/hashmal", line 13, in <module>
    import hashmal_lib
  File "/usr/local/lib/python2.7/site-packages/hashmal_lib/__init__.py", line 5, in <module>
    from main_window import HashmalMain
  File "/usr/local/lib/python2.7/site-packages/hashmal_lib/main_window.py", line 8, in <module>
    from dock_handler import DockHandler
  File "/usr/local/lib/python2.7/site-packages/hashmal_lib/dock_handler.py", line 4, in <module>
    from docks.addr_encoder import AddrEncoder
  File "/usr/local/lib/python2.7/site-packages/hashmal_lib/docks/__init__.py", line 6, in <module>
    import tx_deserializer
ImportError: No module named tx_deserializer


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 04, 2015, 06:15:55 PM
I'm having difficulty getting it to run. Here's what I'm getting after installing from git on OSX Yosemite.

Code:
$ hashmal
Traceback (most recent call last):
  File "/usr/local/bin/hashmal", line 13, in <module>
    import hashmal_lib
  File "/usr/local/lib/python2.7/site-packages/hashmal_lib/__init__.py", line 5, in <module>
    from main_window import HashmalMain
  File "/usr/local/lib/python2.7/site-packages/hashmal_lib/main_window.py", line 8, in <module>
    from dock_handler import DockHandler
  File "/usr/local/lib/python2.7/site-packages/hashmal_lib/dock_handler.py", line 4, in <module>
    from docks.addr_encoder import AddrEncoder
  File "/usr/local/lib/python2.7/site-packages/hashmal_lib/docks/__init__.py", line 6, in <module>
    import tx_deserializer
ImportError: No module named tx_deserializer

My bad. I hadn't included that file in the list of modules in setup.py. Fixed now.


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 05, 2015, 06:11:48 PM
Getting closer to a real 0.2.0a release. I've written about and uploaded screenshots of tools on the Github Wiki (https://github.com/mazaclub/hashmal/wiki).

There won't be a changelog until after 0.2.0a since so many things are changing. Here are some key recent changes:

  • Variables can be substituted into a transaction (see Variables tool documentation on Github wiki).
  • Variables can be saved to the config file.
  • Some stack log messages are now better (e.g. grammatically improved).

If you'd like to try Hashmal but don't know where you would start, check out the Example Scripts (https://github.com/mazaclub/hashmal/wiki/Example-Scripts) page.


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 19, 2015, 01:02:14 PM
Alright, there's now a plugin system. I'm looking for ideas for plugins, so if you have any, please post them. Note that plugin basically refers to one or more tools in the form of dock widgets, which are detachable widgets. The purpose of plugins is to provide extra functionality when you have a certain requirement installed.

The proof-of-concept plugin that I'm working on is Pycoin, which provides a GUI for pycoin (https://github.com/richardkiss/pycoin).

Also, there's a "chainparams" system that allows you to specify the format of transactions. For example, in Peercoin and its forks, transactions have a timestamp field after their version field. By selecting the Peercoin chainparams preset, you can deserialize Peercoin transactions, and even set the timestamp for a transaction in the Transaction Builder tool.

Anyway, ideas for useful plugins are welcome! :)


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: benehsv on September 21, 2015, 11:22:52 PM
I am also getting an error on osx 10.10

Traceback (most recent call last):
  File "./hashmal", line 17, in <module>
    gui.main()
  File "/Users/bbuenz/Programmierung/CryptoWorkspace/hashmal/hashmal_lib/__init__.py", line 14, in main
    self.main_window = HashmalMain(self.app)
  File "/Users/bbuenz/Programmierung/CryptoWorkspace/hashmal/hashmal_lib/main_window.py", line 38, in __init__
    self.dock_handler.do_default_layout()
  File "/Users/bbuenz/Programmierung/CryptoWorkspace/hashmal/hashmal_lib/dock_handler.py", line 90, in do_default_layout
    self.dock_widgets['Variables'].setVisible(True)
KeyError: 'Variables'


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on September 23, 2015, 05:57:26 PM
I am also getting an error on osx 10.10

Traceback (most recent call last):
  File "./hashmal", line 17, in <module>
    gui.main()
  File "/Users/bbuenz/Programmierung/CryptoWorkspace/hashmal/hashmal_lib/__init__.py", line 14, in main
    self.main_window = HashmalMain(self.app)
  File "/Users/bbuenz/Programmierung/CryptoWorkspace/hashmal/hashmal_lib/main_window.py", line 38, in __init__
    self.dock_handler.do_default_layout()
  File "/Users/bbuenz/Programmierung/CryptoWorkspace/hashmal/hashmal_lib/dock_handler.py", line 90, in do_default_layout
    self.dock_widgets['Variables'].setVisible(True)
KeyError: 'Variables'

Did you run setup.py?


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on December 29, 2015, 04:10:22 PM
Alright, so I've done a lot on Hashmal since I last posted here. It really grew from a "Transaction Script IDE" to a full-fledged toolbox for cryptocurrency development. You can deserialize transactions/blocks, encode/decode addresses, etc. There's an entire plugin system. I've tried to keep the Github Wiki (https://github.com/mazaclub/hashmal/wiki) up-to-date with the all the plugins that Hashmal comes with.

Here's a short list of what the new plugins that Hashmal includes do:
  • Deserialize transactions/blocks
  • Encode/Decode addresses
  • Retrieve blockchain data from block explorers or via RPC
  • Create transactions from scratch

I'm currently working on a large refactor of the Stack Evaluator. Here is how it looks now (https://github.com/mazaclub/hashmal/tree/0cc5a04452190c5a92a40a7eddfe3d908a933859) (apologies for the big image):

https://i.imgur.com/kfRRN0O.png

The issue I'm currently dealing with is displaying the contents of an inner script (as in Pay-To-Script-Hash scriptSigs) in human-readable form. When I have issues like that solved, I'll merge it into the master branch.

--
edit: For your viewing pleasure, here are a couple of examples with the latest changes:

7 7 7:

https://i.imgur.com/cvUs3M4.png

string literal:

https://i.imgur.com/uDVUuIt.png


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: JackH on December 30, 2015, 10:47:23 AM
There is a similar tool written in Javascript for Forth here: http://skilldrick.github.io/easyforth/

What would be interesting you implement is a mock miner that can be configured. This way we can test the more exotic op_codes and understand how they would behave from a inclusion point of view in the blocks. Do you have support for BIP65 already?


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on December 31, 2015, 03:17:37 AM
There is a similar tool written in Javascript for Forth here: http://skilldrick.github.io/easyforth/

What would be interesting you implement is a mock miner that can be configured. This way we can test the more exotic op_codes and understand how they would behave from a inclusion point of view in the blocks. Do you have support for BIP65 already?

Seems pretty cool. About the mock miner: That could potentially be a useful plugin. I'm not sure exactly what you mean by an "inclusion point of view" though.

BIP65 support isn't implemented. I've been putting it off in hopes that somebody else would implement it in python-bitcoinlib. :P But the underlying script evaluation API in python-bitcoinlib will take some changes to support including block height and/or block time in the evaluation, so I guess other people have the same idea that I do. Ultimately I'll implement it regardless, since I need to put in a way to override opcode behavior based on what chainparams are active. For example, a blockchain like CLAMS uses a different CHECKLOCKTIMEVERIFY implementation than Bitcoin does.


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: JackH on December 31, 2015, 02:24:07 PM
By inclusion I just ment on how the entire path from op_code creation to block inclusion by a miner would look. But I guess there also needs to be a limit to how much you can keep expanding it, without ending up with a full copy of Bitcoin :)


Title: Re: Hashmal - Transaction Script IDE (alpha)
Post by: Kefkius on January 06, 2016, 04:22:25 PM
By inclusion I just ment on how the entire path from op_code creation to block inclusion by a miner would look. But I guess there also needs to be a limit to how much you can keep expanding it, without ending up with a full copy of Bitcoin :)

Yeah, I see what you mean.

OP_CHECKLOCKTIMEVERIFY is implemented for the Clams blockchain in branch develop (https://github.com/mazaclub/hashmal/tree/develop). It will take a little more effort to get it implemented for Bitcoin. The way I've implemented it is such that you can specify data about the block that a script is in via a new tab, "Block", of the Stack Evaluator. In the following screenshot, I've set "Block height" to 800000.

https://i.imgur.com/KJdSmzY.png