sx command line utilities - Empower The Sysadmin With Bitcoin Tools

[gwlloyd]

I'm now on my fifth attempt using 2 different servers (one a pretty much clean Debian VM and one an established server).. every time I get nearly to the end of the blockchain sync it just seems to sit around not receiving new blocks and spewing reams of errors such as below. I'm assuming blockchain database corruption but it always seems to be somewhere in the 27#### block range (I've just noticed this). Is this normal? Is there maybe something about this range of blocks, was that when someone was spamming the blockchain? Getting a bit frustrated here sapping so much time and bandwidth and still haven't got a sync'd copy to play with. I don't want to download a torrent copy.


WARNING: Error storing memory pool transaction c0c958698d61b260a885115a7e2ccec1a25c8a121c0da9ec6165f0ce11920c9b: Spent input not found

Nope it looks fine. Those errors aren't a problem. Example: memory pool isn't full yet and there's txs depending on other txs in the mempool that aren't yours. It's normal. I'll make the wording sound less extreme.

Important is that you're able to run 'sx fetch-last-height' and the block height matches blockchain.info or somewhere else.

I need to document these tools better. Sorry about that. I'll be making efforts towards that goal as documentation is important.

The docs aren't that bad and the error does make sense - if it hasn't got the full blockchain then there are going to be transactions that don't make sense to it, so it can't store them.

It's sticking on block 278746 every time and once it gets to that block it is still using CPU but nowhere near the levels it normally uses when sync'ing. Looking up the blocks around this time it doesn't seem to be a gorilla block. Any ideas? Perhaps start fresh without my local bitcoind as one of the manual nodes.. is there any way my bitcoind could have a bad blockchain and not know about it?

[1713574730]

1713574730

[1713574730]

1713574730

[genjix]

have you configured it to only connect to your bitcoind? if so, how many blocks does your bitcoind have? use ./bitcoind getinfo to see. if bitcoind has the correct number of blocks then try restarting obelisk and/or restarting bitcoind. does it start syncing?

[gwlloyd]

bitcoind has been running for months, fully up-to-date and regularly used. I've tried pointing obelisk to it exclusively, not pointing to it at all and pointing to it with other public connections too. restarting obelisk makes no difference and like I say I have started a fresh obelisk on a different server and when it hits this block, the same happens.


Edit:  Debian 7 servers (one 64, one 32), open files limits increased, installed using your automatic install script and I'm very familiar with Linux & coding so feel free to get technical.

[genjix]

I'm using Debian Wheezy, what are you using? When did you install it? When did you start the daemon? Are you using an SSD? I would recommend an SSD because a lot of the code assumes you're using SSDs (0 seek times) as I believe they're the future of storage.

[gwlloyd]

One is a fresh Wheezy and one is an upgraded Debian 6 that has been in use for years. I installed Obelisk about 10 days ago but I've done a few attempts to sync in that time.. I'm not using SSD but they are VirtualBox VMs so it's sortof SSD - that wouldn't explain why it always gets to this specific block and then never any further. I guess I'll just try another sync from the start.

[genjix]

ok, if you do then run the latest script. I've made lots of changes.

[gwlloyd]

ok, if you do then run the latest script. I've made lots of changes.

FYI on the latest install script it fails to remove the package libzmq on debian because it doesn't exist, only libzmq-dev does in the standard repositories (or at least on mine install)..  removing that item from the script corrects the issue.

[genjix]

ok, if you do then run the latest script. I've made lots of changes.

FYI on the latest install script it fails to remove the package libzmq on debian because it doesn't exist, only libzmq-dev does in the standard repositories (or at least on mine install)..  removing that item from the script corrects the issue.

Weird, because I'm testing on Debian Wheezy virtual machines. I'll look further, thanks.

Edit: OK, I've changed it to a wildcard.

[genjix]

From now on use tcp://obelisk.unsystem.net:8081 as your server if you don't have one. No guarantees about uptime (at least for now) but I'll try to migrate it as I get more servers and maintain it as a public one.

[gwlloyd]

From now on use tcp://obelisk.unsystem.net:8081 as your server if you don't have one. No guarantees about uptime (at least for now) but I'll try to migrate it as I get more servers and maintain it as a public one.

If I donate a cheap VPS and point it at your balancer does that help?

[genjix]

From now on use tcp://obelisk.unsystem.net:8081 as your server if you don't have one. No guarantees about uptime (at least for now) but I'll try to migrate it as I get more servers and maintain it as a public one.

If I donate a cheap VPS and point it at your balancer does that help?

It's OK, we did the darkwallet campaign. It's more about manpower to develop the tools since we're testing and improving the software.

[instagibbs]

edit2: Getting an error with the libbitcoin step now. Help? I already checked, my libbitcoin dependencies all seem correct:

make[1]: Entering directory `/usr/local/src/libbitcoin-git/src'
  CXX    satoshi_serialize.lo
  CXX    getx_responder.lo
  CXX    threadpool.lo
  CXX    network/network.lo
network/network.cpp: In member function 'void libbitcoin::perform_connect_with_timeout::start(boost::asio::ip::basic_resolver<boost::asio::ip::tcp>::iterator, size_t, libbitcoin::network::connect_handler)':
network/network.cpp:77:9: error: 'async_connect' is not a member of 'boost::asio'

Getting this error. I think I've installed everything people are saying to install, and can't find any obvious packages via googling.

edit: It appears that software needs g++ newer than the version I'm running, according to its github page.

[gwlloyd]

Sorry but I have to conclude that Obelisk doesn't work on low memory (1.5 Gb), non-SSD virtual machines. Just for anyone out there trying. I've lost count how many times I've had to rm -R ./* and start again over the past 4 weeks or so due to some fatal error or another. Shame, because I really didn't want to use any more electricity with yet another physical server but I really want to have these sx tools available to me so I guess it has to be so.

[genjix]

edit2: Getting an error with the libbitcoin step now. Help? I already checked, my libbitcoin dependencies all seem correct:

make[1]: Entering directory `/usr/local/src/libbitcoin-git/src'
  CXX    satoshi_serialize.lo
  CXX    getx_responder.lo
  CXX    threadpool.lo
  CXX    network/network.lo
network/network.cpp: In member function 'void libbitcoin::perform_connect_with_timeout::start(boost::asio::ip::basic_resolver<boost::asio::ip::tcp>::iterator, size_t, libbitcoin::network::connect_handler)':
network/network.cpp:77:9: error: 'async_connect' is not a member of 'boost::asio'

Getting this error. I think I've installed everything people are saying to install, and can't find any obvious packages via googling.

edit: It appears that software needs g++ newer than the version I'm running, according to its github page.

yep although this error is caused by using an older version of boost

[genjix]

Sorry but I have to conclude that Obelisk doesn't work on low memory (1.5 Gb), non-SSD virtual machines. Just for anyone out there trying. I've lost count how many times I've had to rm -R ./* and start again over the past 4 weeks or so due to some fatal error or another. Shame, because I really didn't want to use any more electricity with yet another physical server but I really want to have these sx tools available to me so I guess it has to be so.

you can still use the tools and we're running a public server at obelisk.unsystem.net:8081

[gwlloyd]

Ugh, my server is currently down because ever since I've updated the sx stack, it errors out. Tried removing the blockchain, no luck.

Code:

Using config file: /etc/obelisk/worker.cfg
Press CTRL-C to shut down.
DEBUG [worker]: Listening: tcp://*:9091
INFO [worker]: worker ready
DEBUG [worker]: Heartbeat: tcp://*:9092
obworker: blockchain/database/stealth_database.cpp:17: libbitcoin::stealth_database::stealth_database(libbitcoin::mmfile&): Assertion `file_.data() != nullptr' failed.
Aborted (core dumped)

I'm no expert and I'm sure Amir will reply, until then.. did you "init blockchain" correctly (matching your configured folder) after removing the data and also have you tried a new location just in case of some kind of lock preventing access to the DB file.

[grue]

the multisig documentation is broken. sx sign-input ouputs debug messages so it breaks the shell substitution in the second last step. Even if you fix that, you will get:

Code:

Usage: set-input FILENAME N SIGNATURE_AND_PUBKEY_SCRIPT

[dexX7]

Hmm.. I'm not sure, what I'm doing wrong. I installed sx with the install script on several systems.

After running "sx fetch-last-height" and some other commands I receive the following error:

Code:

terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_S_construct null not valid

Any help is very appreciated!

[genjix]

Hmm.. I'm not sure, what I'm doing wrong. I installed sx with the install script on several systems.

After running "sx fetch-last-height" and some other commands I receive the following error:

Code:

terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_S_construct null not valid

Any help is very appreciated!

What system are you using? 32 bit or 64 bit? Thanks.

[dexX7]

I tried Ubuntu 13.10 x64, Ubuntu 12.04.3 x64, Debian 7.0 x64, Arch Linux 2013.05 x64 and Fedora 19 x64.

The installation process was close to https://github.com/mastercoin-MSC/omniwallet#setup.

I used droplets with 1 GB RAM.

To confirm I'm not missing something very, very essential:

After a basis setup with apt-get updates etc. running the install script should be sufficient to get results from sx without setting up an obelisk server, correct?

Edit: Error remains. This time with a 4 GB Ubuntu 13.10 x64 box and libboost1.54-dev.

Edit 2: Running an Obelisk worker fails at some point:

Code:

INFO [poller]: Block #495 00000000e47349de5a0193abc5a2fe0be81cb1d1987e45ab85f3289d54cddc4d
INFO [poller]: Block #496 00000000b0c5a240b2a61d2e75692224efd4cbecdf6eaf4cc2cf477ca7c270e7
INFO [poller]: Block #497 00000000c39ea29ad310c1f80409e24b3fbfd671c0b3599198b7cfdebf790bde
INFO [poller]: Block #498 00000000693a6d6b068cab3e207d570764f6bad293e3e98920246eeda81c496a
INFO [poller]: Block #499 00000000806df68baab17e49e567d4211177fef4849ffd8242d095c6a1169f45
INFO [poller]: Block #500 000000004ff664bfa7d217f6df64c1627089061429408e1da5ef903b8f3c77db
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)