Bitcoin Forum

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Greetings,

I've been working on a Python front-end for bitcoind for the past few days.
The principal motivation was to give myself a reasonable way of monitoring my node without using CLI.

I thought it might be useful to others so I've made it available on GitHub: https://github.com/azeteki/bitcoind-ncurses

As it's been hacked together rapidly it is fairly limited on features and 'just about works' at present.
Best to treat it as a proof of concept for now, though it is perfectly usable for watching only.

As always, make sure you audit before running (I've tried to seperate out potentially vulnerable stuff like RPC calls as much as possible).

Regards,
Atelopus Zeteki
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCgAGBQJTrfn2AAoJEOK9FOwsfUWPCykQAK2yBLXlg3iJ6D1OSGjKeh1r
YrsXHwODQ9vx92ucIdQEQ085+E3qKqVBOTv0hQHmKC7BSo7XoDfSvuCxYjzQEtWX
8Kpn+scIzi2GQXL+wRCVKVo72lLdzjr3r/nPNm36n63LmDqBE14HTe1rgnfsw6oH
CRe8Czdx6q12dicBaARvBxs07s8lARr/rIxxvB2SPyGLTkdNbg5jzhf28FiB8XyI
MrxdCt0OHYLPlArHDswJlUVwgu6/Us8aL3e3nxYvlEJPqM/dX/4rlMembl3eNTmJ
3JiKkNFIITcKvy3p8Z0jHHo5l8YwOjN4H4M70lhL4JAuJJCTYb+oqKuUoxcWObDG
XiXrWfx8/TRfBYDFmqys3hWz7gGm3ssTsobb5cdvywC00DrSfTUKGnNvzLDWID5A
yTek73cs6scgDOrdRZQr7sGUI+rKd80B2v/AxA2zGfgq7WQ7WKJbAwkJVnp6HhOG
XbQk91geG07XE/nQbxa2hfEUvNMQL3wfPWyvEc4oBMnJqheeptb5tpw30mq8+tpQ
gz7vUgBkJLiGi0P/zyi0YLDWCJxIFaCBj4BoFFFmpbMF1tO+XTSTlMhX81pO4vrw
2agjnMd/bI1S2MXH3LNgu9K1blio47rrPTsvDJX/MFkeAky7da2rj/POWeCE4j81
y8JclVhdbEHfO/DdoGFv
=84Pj
-----END PGP SIGNATURE-----

Here are some screenshots of the various modes.

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-splash.png

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-monitor.png

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-block.png

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-tx.png

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-peers.png

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-wallet.png

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-net.png

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-forks.png

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-console.png

Very neat.  Perhaps to go with the bandwidth stats some mempool stats and a time since last received block?— for the latter we might need to adds something to the rpc, I don't think we expose what time we received a block.

Actually, the 'received ... ago' display is time since last received block (including verification time).
This is because at present it constantly (every 2 seconds or so) polls with getinfo.

If blockheight changes then it goes ahead and pushes a getblockhash/getblock combo to find timestamp and size.
My understanding is that there is no need to check getbestblockhash because the height should never reduce.

This fails for the first block on startup and just displays the time of first contact with bitcoind.

Looks very cool, I will look into this today.

Edit: good to see it needs Python 2.7

Thank you both.
virtualx, it may work with older Python versions, 2.7 is all I've tested.

Pushed a few updates since the original post.
A lot more is event driven now.
The transaction and block browsers have scrolling support (tested with 200KB transactions with hundreds of inputs/outputs) and should be _much_ smoother.
Only remaining obstacle is the main updating view (called every second or so).
I intend to set up and test over SSH at some point to see if there are issues with redrawing.

If anyone has feature requests I will be glad to look into them as time allows.
Next is to add support for querying an arbitrary block, prev/next block and so on.

A few updates:

Lots of speed improvements and eliminating needless queries.
Better error handling if commands fail or connection fails.
Various bug fixes.
The block viewer now supports browsing through older blocks and arbitrary block height or hash.
The genesis transaction would fail as bitcoind does not index it.
Formatting changes to monitor mode (age of transactions).
Difficulty, nethash and mempool transactions added to monitor mode.
A peer viewer has been added - currently listed is sync node status, incoming/outgoing status, node IP (if available), received and sent data.

I wanted to thank users again for making suggestions and testing so far.

Nice piece of software! Bug report:

- Go to block 309951.
- Go to transaction 2. Don't view it, just move the cursor down.
- Go backwards to block 309950. There's only one tx in this block.
- Oops, cursor thinks it's still in tx 2. If you hit space to view it, the program dies.

Good catch.
This should now be fixed in latest github master.

A wallet transaction view mode has been added.
Note that as of yet, bitcoind-ncurses does not implement unlocking the wallet, creating or signing transactions, or any other key-related functions.
This is simply for viewing historical/unconfirmed transactions that are relevant to your wallet (in listsinceblock).

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-wallet.png

It may break on odd forms of transactions.
If you want to try and break it and help me improve the checking, feel free to send me odd transactions on testnet.

mr19pWU5UbtFuNu4xxUEeNfeXLwjBggYzN

The real time stats are mesmerizing! Thanks you for this!

Indeed! :)

Added a feature to seek for block by timestamp.
It searches by estimating a 10 minute block interval and bouncing back and forth until it thinks it's close enough.
It gets within a few blocks in all the cases I've tested. Could be optimized more but it's good enough for now.

A few more updates:

Changed the timestamp format to ~ISO standard (YYYY-MM-DD hh:mm:ss)
Already queried blocks are cached to improve the speed of block view mode seeking
Monitor mode now displays estimated fee information for latest block (it assumes block subsidy and fees are fully claimed by the miner)
Monitor mode now displays average size of transactions in latest block

This is a pretty good project and certainly helps those of us that use BitcoinD and would like a better overview of things. Nice work, very nice work actually!

Added a 'verbose' mode for the transaction view.

It attempts to display corresponding previous outputs from the inputs.

It may be quite slow for transactions with many inputs. It's not enabled by default, though.

It is now possible to resize vertically most of the views and have them scale dynamically.
So on a 50 row terminal you can show more inputs/outputs at once in the transaction view mode, for example.

Minimum size is now 75x12, down from 75x20.

Textmode just wouldn't be textmode without...

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-splash.png

:D

Thanks for this.  Tested it out and it works well. 

I have disabled wallet on bitcoind.  Perhaps the app can detect that when going to the wallet screen?
The peer view could be more interesting with country of the peer?

What happens? Does it simply display nothing, or is there some sort of error/crash? I can't test right now.

This looks pretty cool, it reminds me of the days of the old BBS. Wil you add soundblaster 16 sounds by any chance?  :)

Not near it right now.  It doesn't crash, but I think it says transaction not found or something similar, and asks for a txid.

The country of peers based on geoip would be nice to have on the peers screen.

See this:

Strange. I have just tested it with --disablewallet and for me the 'W' hotkey just doesn't do anything.
I believe there is possibly a bug that will stop it from updating information though (it'll silently think it's in 'wallet mode').
When you get time, please check for me exactly what happens, it would be useful to know.


I don't have geoiplookup on my system and I'm not really interested in adding dependencies just for cosmetic stuff. Sorry.
If there's some way to do it with standard python libs or you find the will to code up a list for me in Python I would be happy to add it.

A debug console has been added (similar to the one in bitcoin-qt, or just ./bitcoin-cli command...) so that you can query directly yourself without leaving the program or having to type './bitcoin-cli -testnet ...' each time.

It's not complete and will likely fail on certain commands, so I would treat it as a toy (works fine for getblock, getblockhash, getwalletinfo, getinfo, help, etc) and not use it for producing transactions or anything that could result in loss of funds for now.

gmaxwell suggested that I add sounds from the blockchain. I think live mempool would be better. But unfortunately my musical abilities are extremely limited. :P

Updated to v0.0.14.

A few cosmetic tweaks to console view and a few bugfixes.
TX view verbose mode now shows whether transaction outputs are spent or not.
Monitor view now flashes when it detects an incoming block.
SPACE hotkey has been switched for enter at user request.
TX view coinbase no longer shows nonsense like negative fees.
TX view switches between outputs/inputs using TAB now.

Testnet displays units as 'TNC' to avoid confusion.
Previously you would rely on the coloring of the header only, which is not ideal for mono terminals or colorblind users.

Various cosmetic changes here and there.

More updates:

The tool now supports bitcoin.conf files natively, no need to edit the file.
Copy it in or run using the -c switch and point to it.

There is a now a footer 'menu system' that follows through all views so that you don't have to remember hotkeys as much.

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-monitor.png (https://github.com/azeteki/bitcoind-ncurses)

You can view transactions from your wallet by locating them with the cursor and pressing ENTER. Previously you'd need to copy paste which is hard without a tmux clone or X-terminal.

Verbose mode should now show whether transaction outputs unconfirmed spent or confirmed spent.

edit:

A number of drastic performance upgrades have been made.
The limiting factor for speed is generally now bitcoind-rpc rather than hardcoded time.sleep nonsense.
CPU usage on my machine is very low; though there are a few small tweaks I could make (customizable display update interval, for example).

v0.0.19.

Added a feature to view coinbase strings. You can read the messages miners leave there.
Verbose mode in the TX view should now be much faster for large transactions.
A basic bandwidth chart has been added.

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-net.png

Thanks looking good.  Just updated.  It's correctly identifying the fact that I'm running with wallet disabled.

Probably a matter of personal taste but I find the J/K bindings counterintuitive to move among blocks. I'd expect J to go to the next block instead of the previous one since J is usually used to move forward. Re: "HOME/END: quicker" I'd use PgDn/PgUp for that, and maybe relegate Home/End to even larger jumps.

Although if it were me, I'd use H/L to browse blocks and J/K to move up/down in the transaction list, both in the Block and Wallet windows. Specifically, not having J/K in the Wallet window bugs me somewhat :).

I just git pulled from a1989f3 to 6276b9f and noticed that now it's way faster to move among the transactions in the wallet, well done on this!

My plan is to move towards making hotkeys configurable via .conf or similar.

I actually don't really like J/K for block navigation either.

It used to be left/right but after adding the footer and allowing left/right to navigate views I changed it.

As a side note, this is trivial to change without even knowing Python, though unwieldy. Go into hotkey.py, look for keymap = {... and change the keys that map block_seek_forward_one and block_seek_back_one.


Originally I had a lazy time.sleep in both the RPC and interface threads, just to get up and running.

Now it's more sensible. The sleep function waits on data.

So if a message is passed from RPC to interface or vice versa, response should be instant, rather than waiting for the next iteration of the loop.

If you want some real fun, turn up your keyboard repeat rate to max. :D

After a bit of pain and debugging I successfully got bitcoind-ncurses working for both btcd and bitcoind simultaneously!

https://i.imgur.com/hmaVKfh.png

What needed modifying for btcd?

By looking at his screenshot I'm guessing that btcd doesn't support all the newest bitcoin rpc calls. From the screenshot I can see he had to remove chainwork display as I think it was only added via getblockchaininfo relatively recently to bitcoin core.

First issue was just getting it to run on Windows.
https://github.com/PRab/bitcoind-ncurses/commit/d13bfd3a59c86b097593d746c4f4b910eb355659

After that, I needed to figure out why btcd wasn't working. Nothing needed to change in btcd, but btcd requires SSL. I haven't been able to get SSL to work for bitcoind/bitcoin-qt so it was a bit tricky for me to test.
https://github.com/azeteki/bitcoind-ncurses/issues/5
https://github.com/jgarzik/python-bitcoinrpc/issues/35

No; that should fail over safely.

A lot of the items on the monitor view should just disappear if the required info can't be obtained.

'disablewallet' support for example just works de facto because the getbalance req fails, and so nothing is displayed.

Not everything is totally safe against this (I don't really sanity check the information from bitcoind; if getblock reports no hash it might die, for example).

Unfortunately I don't really have the prereqs to test things like btcd.
My system is old and rather slow.
bitcoind reindex takes me a few weeks for example.
Hopefully at some point I will be able to justify buying a new system.

Updated to v0.0.20.

A few small additions:

Errors field, for example 'This is a pre-release test build' for development releases
(note that this is only checked on boot; it does not refresh as of yet, this is planned)

Transaction view no longer quits the program if a tx is not found
Transaction verbose mode now colours outputs according to their spent/unspent status
A number of modularization updates that should ease code accessibility and speed up the program in areas
Monitor view now displays implicit hashrate as per diff (the hashrate that would be required to sustain average 10 minute block rate)
Peer view now displays node sync height and time connected
Monitor view now displays estimatefee data (on bitcoind development release only)
RPCSSL support (untested by me; thanks to PRab and mikegogulski on GitHub)

I wanted to thank all of the people who have used and tested bitcoind-ncurses so far and especially those who have donated.
It is a good feeling to know that your work is appreciated. So, thank you all very much.

Beautiful thank you

WRT peer height.

The syncheight is based on what the peer has advertised to us, so if a peer is not synced up yet its syncheight will be -1. You might want to fall back to displaying the starting height there if and only if it's value is more than one less than your current height and syncheight is -1.

My peers page has no heights

You're running a non-git version of bitcoind. Next major version will have them for you.

Gotcha

This application is awesome, thank you azeteki.

I've been considering using something like a Trezor for storing my BTC for quite some time... but I think that putting an old PC to work as a wallet is better (seeing as it's in a safe, is secured behind multiple firewalls and is only electronically accessible via SSH). At the moment I do everything on the CLI, but your app makes it a lot easier/visually appealing to work with bitcoind on a terminal.

Thanks again :) .

I am glad to hear that you found it useful. :)

As of late a number of life issues has meant that I've had to take a step back from Bitcoin for the time being.
With a little luck I may be able to continue development (in addition, I would like to branch outside of bitcoind-ncurses and pursue a number of other ideas).
The world will decide this in time.

I check for updates weekly :)

I'm sad to have not heard from you for a while...

Though I'd come by to point out: If you get some time and interest again; git master getchaintips rpc should be a pretty nice source of information for another screen in this tool.

I just discovered this.
This is awesome!
Thanks a lot for making this.
I love console guis.

BTW, anyone ever used this on a production system?

Adding transaction creation would be awesome, even though it's risky business. :)

gmaxwell,

I've missed being able to stay in touch with the field. Glad to see that bitcoin has not yet died. :)

getchaintips looks quite interesting. If I'm interpreting the source comments correctly (I can't compile and test right now) then it matches a feature I was intending to add to bitcoind-ncurses - monitoring forks and their eventual resolve.


Indeed.
There are a few different approaches to this whole business that I can think of.

The first is to rely on bitcoind to perform the wallet functionality. Tell bitcoind to send some coins, and bitcoind consults its' wallet.dat and so forth.
Least likely to go wrong, but limited in scope.

The second and more ambitious is to effectively produce a standalone wallet that uses bitcoind to connect to the p2p network only. In this case bitcoind-ncurses would have its' own 'wallet.dat' format, produce its own keys, do its' own coin control, and so on and so forth. Much more 'fun' for me, and also achieving the secondary aim of producing a viable split between the 'node' that is bitcoind and the 'wallet' that is bitcoind, but requiring that I really know what I'm doing.

What I would need to do to feel comfortable with either solution is to spend a lot of time studying how bitcoind achieves its' wallet functionality and also ensure that Python is even capable of producing a reasonably secure wallet (which would entail a fair amount of crypto-study).

To elaborate on the latter point - take a basic function like 'walletpassphrase'. This needs to be handled very carefully. You can end up in a situation where the passphrase sits around in RAM for the rest of all eternity after being used. Or perhaps the size of the program's RAM usage increases by some predefined way based on passphrase content. Maybe the query takes a different amount of time depending on the length of the password, etcetera. Some of these probably matter as side channel attacks and others may not. If I am to produce a wallet tool for general consumption then I have a moral obligation to do everything within my power to ensure that it is safe because people can and will use it to store lots of money.

It makes me very happy that people have found my work useful so far. Finding some solution to fund the roof over my head whilst also pursuing this sort of project remains the challenge we all struggle with.

For standalone wallet, doesn't armory have a python library you can use?

Delete bitcoin-config.h and rerun autogen.sh.  Also make clean before building. Some of the changes we made have upset the build system for dirty build directories. Sorry for that. Hopefully this will get you going. You may also need to install libgmp-devel if you don't have it currently.

There would be a middle ground here: using the new watch-only addresses of bitcoind, you could delegate all the tracking to bitcoind. Private keys and coin control could then be handled by bitcoind-ncurses. Would be a nice compromise as it would allow you to get all the potential while getting around having to implement your own rescanning, indexing, and tracking code.


I think bitpop's suggestion of taking a look at armory is a good one. It's also written in python and solved most of these problems already. An ncurses interface to some of armory's functions (while relying only on bitcoind) would be an awesome combination!

Feature wise, what I would love is to see a ncurses interface to coincontrol: constructing transactions by selecting UTXOs from the list of available ones and computing fees following different schemes (free, pre 0.8.2, current, manual).


Indeed. :) I can only say: keep up the good work! Hope you'll find some income in the space that allows you to continue this work. If this ever happens to myself, I would happily donate as well...

I agree on the armory points.
Also thanks for the tool.  Very nice.

v0.0.21 - basic addition of 'getchaintips' for users running master

Very small update that hopefully explains itself for users on git master. I imagine the feature will make it in to v0.10. Should fail safe on non-git though I've not tested.

https://raw.githubusercontent.com/azeteki/bitcoind-ncurses/master/screenshots/bitcoind-ncurses-forks.png

v0.0.22.

Added some basic logging functionality which should help for debugging purposes and identify areas where speedups can be made. (getrawmempool vs. getnetworkinfo is a big one at present).
For now the loglevel is hardcoded in rpc.py. Right at the top is a function with 'if loglevel > x'.

loglevel 0: (default) Nothing.
loglevel 1: Regular updates, new blocks, halt request.
loglevel 2: All RPC requests.
loglevel 3: Time taken for bitcoind to respond to each RPC request.

Example at log level 2:


Screenshots have been updated with new functionality, see the second post in the thread or github.

Updated

This is so awesome for managing bitcoin on webservers. Thanks for this gold.

Thank you vertoe.

If anyone has used this over SSH or similar some feedback would be useful. I have only tested locally. Others have suggested ideas on how to emulate latency that I'll be looking into later.

I plan to introduce more configuration options at some point to avoid the need for people to go in and change hardcoded stuff.

For example, at present the RPC backend updates every 2 seconds and the monitor view every second. Over a high latency link this might be too often. You can change this if you know where to look but that's hardly ideal.

I would also appreciate feedback from anyone who has tried this on an RPi, Cubox or similar machine (e.g. top/htop cpu usage stats, if it's usable, etc.) That would help me to get an idea of where to focus development.

I use it on digital ocean, works perfectly on ssh

Live mempool transaction view would be cool

Hi Amphibian,

I played around a bit and noticed ipv6 addresses exceed the cell width on the peers page.
https://github.com/azeteki/bitcoind-ncurses/pull/13

https://cloud.githubusercontent.com/assets/6561620/5533549/c3b7aff6-8a4a-11e4-8212-85529fd2e6d4.png

Really loving it.
--vertoe


/edit, I just released a basic version for darkcoin (https://darkcointalk.org/threads/darkcoind-ncurses-terminal-front-end-for-darkcoin-daemon.3315/).

This is really cool

It would be great if we could get following parameter :
Estimated  time to sync upto to latest block on network ..

Just one question. Does it allow for overview of accounts and wallets connected to accounts?

I can easily see this being used vs a GUI if it can track accounts. Basically the RPC command known as " getaccountaddress " and "  getaccount " and " getaddressesbyaccount " and most importantly " getbalance  [account] [minconf=1]  " .

It would be by far the best tracker to track multiple accounts and values and movement of coins for a wallet service, or at least offer a different perspective.

Hi JackH. Sorry for the delay in responding.

I have not added account tracking because my impression based on hanging around IRC is that accounts are a deprecated feature and may well end up being removed completely at some point.

Feel free to hack away and submit pull reqs if you are that way inclined though.

(The transaction/wallet side is really more of a curiosity than anything else to me personally at present.)

Well, it's been a while!

A few small tweaks have been made to the repo.
Andrew Chen helpfully posted a fix for Bitcoin Core v0.12 in my absence, which has now been merged.

I have changed my GitHub username - a PGP signed confirmation of this with the same key as my original post here will be forthcoming (currently battling with importing the key).

- esotericnonsense/azeteki

Glad you're back

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I, Atelopus Zeteki <azeteki@safe-mail.net> am the owner of the public key:

0x47DA40099E00994C Daniel Edgecumbe <daniele@esotericnonsense.com>

and the GitHub username:

https://github.com/esotericnonsense

and the domain name:

https://esotericnonsense.com

The Daily Telegraph 2016-05-02 'Families in dark as doctors let patients die'
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJXJ1KsAAoJEOK9FOwsfUWP+CoQAKKyh/8MUlZxsahH+KjJG5Kf
3z/jcMZVJyyOD/Pc7Gd6Iy+LBWVenmUdwtRKeuzIENX/aZ0vU7bnXqNBPG3NgcQE
tMbewxEyMbdGUi7LqKdTB1yFiX+BODXL3v5WpUAGfhjbxNe8kXCErPp1iUQhErlc
Z5zK9+cmiCTqPQScaUsu4kQu0QsFILWa7wjs23OTMJRjxSYF1aXiqircwgsMg7Q9
9f6Xd9zGLL2mCFVB2rFvi+ROlOcMVZ2/DfVwydW0mbBe9u67AEbZjQ65Pjy5tkbJ
hByn10B55+sYYdbypFUTzVgmyhW8N3Lzz5yEkKYZHhCiNdP+uJaqDW7QV8f0M6uB
MMCpz+hppFQWdmq63VCbTE/GqbhDczeRr8h4UpJ5LNw9cWbPqrBIlbFLdzY7RWI6
nOP4fjce4YJHxCympo6ZP1PgZHLj6Yhdc5TcaSM+/cSQXF8BtmkdYFJHcpeZCch9
YWrGjpyxtr7X8+QCkXGGxlYN4AOoDk05CnT9ZS+tIonPLvTzF2RpOXJRH3wxNR74
Pr08SVDK95r7S9JJEknVGflKV++CeE0dtWBqseUBryKIqBNQKqSQmEjIuigWaCL7
nrOKngfI/m3GEu6nVtuUNnQZ22BDYgCyUV6b7qXCGAUhhuGzojnizNgls/UJpDzw
atwbohdtFZc5/BonemHP
=sBpY
-----END PGP SIGNATURE-----

Welcome back.

I would like to point out that it would still be quite interesting if you can add support for accounts and wallets. Its a big part of what some companies need, due to the fact that we have the logic broken down on the accounts level directly from BitcoinD. Having a great overview in command line, vs. the output we have today would be a good features. Hope you can reconsider and have it added.

This is for nodes. Never keep bitcoin online.