Bitcoin Forum
December 11, 2017, 10:51:36 PM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Standalone Armory -- Struggling with python/OS issues  (Read 2275 times)
etotheipi
Legendary
*
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
January 30, 2013, 06:10:29 AM
 #1

I've decided it's finally time for Armory to manage its own bitcoind instance.  Instead of having the user start bitcoind/Bitcoin-Qt and wait for it to sync, then start Armory.   Armory will actually require bitcoind/-qt to not be running, so that it can start its own bitcoind instance (and setup the RPC interface if it's not setup yet).  It would then use the RPC interface to monitor bitcoind's progress sync'ing.

This will be a major usability upgrade -- the user will spend the same amount of time getting from week-old cold boot to having a synchronized bitcoind+Armory but they didn't have to do any of the work.  And it will also likely be a better experience than Bitcoin-Qt, since Armory likes to explain things to the user, so that they have some idea of what's going on and what to expect.  Armory will be able to estimate how long until bitcoind is synchronized, and tell the user how it's only a one-time synchronization and future loads will be much faster, etc.  And it will only connect "the old way" when it's fully sync'd (which resolves more usability problems, because users get impatient and open Armory prematurely and it chokes).

However, I'm having some serious problems with the implementation.  I can't reliably guarantee that bitcoind is shutdown, especially if Armory were to crash.  So here's what I've got right now:

  • I have added a SatoshiDaemonManager to Armory.  It will autodetect .bitcoin dir and find the bitcoind executable.
  • If bitcoin.conf does not exist, it creates it and sets the permissions to 600 (Linux only, still need to figure out windows)
  • Use the subprocess module to launch:  "sdm.bitcoind = subprocess.Popen([pathToBitcoind, arg1, arg2, ...])"
  • Include a call to sdm.bitcoind.terminate() in the shutdown procedure to do a clean shutdown of bitcoind before exiting cleanly.
Now's the hard part: dealing with unclean shutdowns.   subprocess.Popen() creates a fork that doesn't necessary go away when the parent process does.  I've been doing testing in Linux, and it seems to be inconsistent.  But it definitely can leave a ghost bitcoind running which then blocks the user's attempt to restart Bitcoin-Qt or Armory.    Embarrassed

The really challenging part of this is that I think I can make a solution work in Linux, but it is most definitely not cross-platform.  And Windows is even more complicated, well, because it's windows.

One thing I had considered was wrapping Armory itself with an outer script that will run Armory itself in a subprocess, wait for it to quit/die, then check whether the process it spawned is still open and kill it if necessary (will write out the PID when Popen is called, to a file in ARMORY_HOME_DIR, which the outer script can check for after the Armory subprocess exits).  The nice thing about this solution is that even if Armory segfaults, the outer script will finish running.

However, it's not so clear how easy this will be in windows (which has a totally different process management system), and it certainly will require an OS-dependent code branch -- and both may be kinda complicated.  I presume it will work, it just may be a lot of effort...

Another solution could be to just check for open Bitcoin-Qt/bitcoind when Armory is started, and ask the user if they are willing to restart it.  This is probably not going to work, because it would allow them to reopen Armory, but prevent them from opening Bitcoin-Qt after Armory dies (on the upside, it's good for creating competitive advantage for Armory Smiley)

An ideal solution would be to find a way to create a subprocess that is linked to the current process -- if the current process goes away for any reason, the subprocess should too.  So far, I don't see a clean way to do that.  It looked like process id groups were the solution in Linux, but may not have an equivalent in Windows.

By the way:  I will have an option to just use an existing bitcoind/Bitcoin-Qt instance, but most users, especially those who have never used Bitcoin, will benefit dramatically from having this as the default, all-in-one, standalone solution.  It will probably show up in the preferences to "Allow using existing already-open Bitcoin-Qt/bitcoind instances".  

Recommendations, welcome!




Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
1513032696
Hero Member
*
Offline Offline

Posts: 1513032696

View Profile Personal Message (Offline)

Ignore
1513032696
Reply with quote  #2

1513032696
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
picobit
Hero Member
*****
Offline Offline

Activity: 547


Decor in numeris


View Profile
January 30, 2013, 09:39:10 AM
 #2

One problem you are going to have to face is that there is no way to guarantee that a codes cleanup part runs.  You can e.g. try to catch any reasonable Unix signal and then clean up, but you cannot catch SIGKILL and SIGSTOP, the user can always do a "kill -9" on the Python process.

One way of doing it that I would consider is a separate "guardian process".  Armory starts bitcoind, gets its PID.  Armory then starts the guardian process, passing its own and bitcoind's PID.  Every second, the guardian process checks that Armory is still running, and kills bitcoind if not.  Every second Armory checks that the guardian is running, kills bitcoind and shuts down cleanly if not (*).  Every second Armory checks that bitcoind is running, and restarts it (and the guardian) if not.

(*) Restarting the guardian would be tempting, but it will only crash if something is seriously wrong.
etotheipi
Legendary
*
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
January 31, 2013, 12:36:27 AM
 #3

One problem you are going to have to face is that there is no way to guarantee that a codes cleanup part runs.  You can e.g. try to catch any reasonable Unix signal and then clean up, but you cannot catch SIGKILL and SIGSTOP, the user can always do a "kill -9" on the Python process.

One way of doing it that I would consider is a separate "guardian process".  Armory starts bitcoind, gets its PID.  Armory then starts the guardian process, passing its own and bitcoind's PID.  Every second, the guardian process checks that Armory is still running, and kills bitcoind if not.  Every second Armory checks that the guardian is running, kills bitcoind and shuts down cleanly if not (*).  Every second Armory checks that bitcoind is running, and restarts it (and the guardian) if not.

(*) Restarting the guardian would be tempting, but it will only crash if something is seriously wrong.


Excellent idea!  I like it.  Either Armory will shutdown both, or neither of them.  If it doesn't shut down either of them, the guardian will take care of it. 

This should be fairly straightforward in Linux.   But, I still have to figure out how to apply this in Windows.  I hate to create a new python project and convert it to an executable with py2exe, but anything is better than batch/DOS scripting...


Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
February 05, 2013, 02:35:40 PM
 #4

I'm not sure how vital this is.  Sure, it's kind of a pain to manually setup both bitcoind and armory, but isn't bitcoind going to go away eventually anyway?  Why bother with this when you could be working on removing bitcoind all together?

K1773R
Legendary
*
Online Online

Activity: 1694


/dev/null


View Profile
February 05, 2013, 09:45:40 PM
 #5

One problem you are going to have to face is that there is no way to guarantee that a codes cleanup part runs.  You can e.g. try to catch any reasonable Unix signal and then clean up, but you cannot catch SIGKILL and SIGSTOP, the user can always do a "kill -9" on the Python process.

One way of doing it that I would consider is a separate "guardian process".  Armory starts bitcoind, gets its PID.  Armory then starts the guardian process, passing its own and bitcoind's PID.  Every second, the guardian process checks that Armory is still running, and kills bitcoind if not.  Every second Armory checks that the guardian is running, kills bitcoind and shuts down cleanly if not (*).  Every second Armory checks that bitcoind is running, and restarts it (and the guardian) if not.

(*) Restarting the guardian would be tempting, but it will only crash if something is seriously wrong.

why should this prevent me to kill both at the same time?

[GPG Public Key]  [Devcoin Builds]  [BBQCoin Builds]  [Multichain Blockexplorer]  [Multichain Blockexplorer - PoS Coins]  [Ufasoft Miner Linux Builds]
BTC/DVC/TRC/FRC: 1K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM AK1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: NK1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: LKi773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: EK1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: bK1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
goatpig
Moderator
Legendary
*
Offline Offline

Activity: 1694

Armory Developer


View Profile
February 06, 2013, 07:55:52 AM
 #6

For windows,

CreateProcessWithLogonW is your go to WinAPI function to start a process from another one. I haven't read into it thouroughly, but I think you can set it so that the created thread will be terminated if the original thread dies. Granted, I'd have to dig into it.

Would have to then make a small dll with the windows specific code, and integrate all that for the windows environment part.

picobit
Hero Member
*****
Offline Offline

Activity: 547


Decor in numeris


View Profile
February 07, 2013, 09:49:19 AM
 #7

why should this prevent me to kill both at the same time?

It won't.  It is not a 100% solution, you can certainly shoot yourself in the foot.  But it will solve the problem of a process dying due to a bug or being killed by the user.


EDIT: In case you don't think the solution is elegant: it is because it ain't! Smiley
etotheipi
Legendary
*
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
February 20, 2013, 12:32:01 AM
 #8

I have one lingering, serious issue with this solution as I work to implement it...  creating bitcoin.conf in Windows with a username and password, and then setting read permisison on that file only for that user.  Permissions on Windows is pretty darned ugly.  On the other hand, I think Windows blocks off users' directories by default, so maybe this isn't so important -- but I need to be diligent in making sure this is done right.  I don't want to induce security holes in Bitcoin-Qt because of this!

So in both Windows and Linux:

(1) I check to see if the /home/username/.bitcoin directory is there (or equiv on Windows).  
(2) If not (because they've never run Bitcoin-Qt), I create it.
(3) Check whether .bitcoin/bitcoin.conf file exists
(4) If not (because they've never used RPC interface), then I create it with simple username and a random 128-bit password
(4a) If we just created the file, set the permissions on it so no other users can read it!  In Linux, it's trivial to use os.chmod(...).  Windows, not so simple!
(5) Start bitcoind as a separate process
(6) Start up "guardian" process (as recommended by picobit), pass it the PID of both bitcoind and the PID of Armory.
(7) Read bitcoin.conf file and extract username and password
(8) Connect to bitcoind RPC, and monitor synchronization status
(9) On clean shutdown, kill bitcoind, then kill guardian.
(9a) If Armory is killed, then the guardian will see the PID disappear (or change names), and it will kill the bitcoind instance and exit.

There's a lot of complexity with #8, because there is not a hard definition of "synchronized", but I think I have something sufficient figured out for that (estimating when bitcoind is sync'd, and only then starting the Armory BlockDataManager).  My main concern is making sure that 4a is done correctly.

In Linux, it seems like default behavior is for users to be able to read each other directories (including malicious system processes can, too), and thus it is very important to set the permissions on that file.  In Windows, it looks like default is for users to not even be able to open the home directories of other users.  I tried digging into the Windows permissions on the file, and it's not even clear to me how those permissions are supposed to be set.

Anyone here even use bitcoind.exe in Windows?  Do you do anything to secure your bitcoin.conf file?

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
ErebusBat
Hero Member
*****
Offline Offline

Activity: 560

I am the one who knocks


View Profile
February 20, 2013, 02:52:10 PM
 #9

Been awhile since I have lived in windows, but I used to use cacls from batch scripts to change the rights.

IIRC something like this would probably do what you need:
Code:
CACLS bitcoin.conf /G "Owner":F
That should removed any existing ACLs and then grant the Owner of the file full control.

See:
  http://ss64.com/nt/cacls.html
  http://stackoverflow.com/questions/2928738/how-to-grant-permission-to-users-for-a-directory-using-command-line-in-windows

░▒▓█ Coinroll.it - 1% House Edge Dice Game █▓▒░ • Coinroll Thread • *FREE* 100 BTC Raffle

Signup for CEX.io BitFury exchange and get GHS Instantly!  Don't wait for shipping, mine NOW!
etotheipi
Legendary
*
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
February 21, 2013, 01:13:35 AM
 #10

Been awhile since I have lived in windows, but I used to use cacls from batch scripts to change the rights.

IIRC something like this would probably do what you need:
Code:
CACLS bitcoin.conf /G "Owner":F
That should removed any existing ACLs and then grant the Owner of the file full control.

See:
  http://ss64.com/nt/cacls.html
  http://stackoverflow.com/questions/2928738/how-to-grant-permission-to-users-for-a-directory-using-command-line-in-windows

Thanks Erebus, I will play with that. I had seen something about CACLS, but didn't realize that's a ubiquitous, reliable cmd tool in Windows.  (it looks like icacls is the one to use, actually; cacls has been deprecated, apparently).



As I implement this solution further, it's feeling very hack-y.  I know it's not elegant.  And I'm likely to have problems with non-standard installations, etc.  For standard setups it looks like it will work fine, and in some cases I will be installing/copying with a standard configuration.  But I am wondering if there's a better way.  One thing that crossed my mind was that I could try to jam all the Satoshi code into a library, and change the main() function to a regular function that could be called in a separate thread from inside Armory.  Then I don't have to deal with any of this multi-process stuff...

But that comes with it's own inconveniences -- mainly that the user no longer gets the choice about which bitcoind/-qt version to use, and I have to a lot of work to integrate and compile new version of bitcoind directly into my codebase (this is probably pretty ugly in Windows).  I have to manually integrate and modify each new release of the satoshi client, and if, say, there was a critical vulnerability/update to the satoshi client, I would have to critically update and release a new version of Armory immediately.

I just don't feel like there's any clean solutions here.  But I feel like something must be done to improve accessibility -- having specific requirements/instructions for downloading and installing a separate program, and then not starting/initializing Armory until certain conditions are met... it's a lot to ask of marginally-interested users who don't really understand Bitcoin...


Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
rini17
Sr. Member
****
Offline Offline

Activity: 340


GO http://bitcointa.lk !!! My new nick: jurov


View Profile WWW
February 21, 2013, 12:13:51 PM
 #11

By the way:  I will have an option to just use an existing bitcoind/Bitcoin-Qt instance, but most users, especially those who have never used Bitcoin, will benefit dramatically from having this as the default, all-in-one, standalone solution.  It will probably show up in the preferences to "Allow using existing already-open Bitcoin-Qt/bitcoind instances".  

Recommendations, welcome!
How will users benefit running bitcoind only when armory is started? I call BS, bitcoind can (and is meant to) run all the time in background. With 0.8 disk trashing is no longer an issue. Running it as a service benefits from more open connections, always most recent data readily at hand and also user's transactions propagate quicker. Maybe if user has low-spec computer... but then he shouldn't use bitcoin-qt nor armory at all...

CoinBr.com: First online MPEx brokerage launched beta! Easy to use interface and reasonable fees. Charts for MPEx stocks: live.coinbr.com * My Blog *
etotheipi
Legendary
*
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
February 21, 2013, 12:57:58 PM
 #12

How will users benefit running bitcoind only when armory is started? I call BS, bitcoind can (and is meant to) run all the time in background. With 0.8 disk trashing is no longer an issue. Running it as a service benefits from more open connections, always most recent data readily at hand and also user's transactions propagate quicker. Maybe if user has low-spec computer... but then he shouldn't use bitcoin-qt nor armory at all...

Once I get Armory's resource usage down, then it too will run in the background, along with Bitcoin-Qt, bitcoind.  The user will just keep both programs up.  Running and keeping Armory open will be done in the same way that users would run and keep Bitcoin-Qt/bitcoind open all the time.

The problem with your logic is assuming that more than 10% of target users understand or care about these details.  They heard about Bitcoin from a friend and want to get some coins and buy stuff from Bitcoinstore.com because it has good prices.  I have instantly lost that user if they have to install multiple programs and follow directions to start it.  And it's unnecessary -- I can manage it myself, and they'd prefer me to manage it for them.  This can be win-win for the vast majority of users.   And its default behavior can be the same as bitcoind/-qt:  minimize to tray and run all the time.  Nothing's really different than the way they run bitcoind/-qt, except they have a much more powerful interface on top of bitcoin-qt.

For the 10% of users that do care/understand:  that's what the "Allow using existing already-open Bitcoin-Qt/bitcoind instances" option is for.

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
rini17
Sr. Member
****
Offline Offline

Activity: 340


GO http://bitcointa.lk !!! My new nick: jurov


View Profile WWW
February 22, 2013, 12:15:55 PM
 #13

How will users benefit running bitcoind only when armory is started? I call BS, bitcoind can (and is meant to) run all the time in background. With 0.8 disk trashing is no longer an issue. Running it as a service benefits from more open connections, always most recent data readily at hand and also user's transactions propagate quicker. Maybe if user has low-spec computer... but then he shouldn't use bitcoin-qt nor armory at all...

Once I get Armory's resource usage down, then it too will run in the background, along with Bitcoin-Qt, bitcoind.  The user will just keep both programs up.  Running and keeping Armory open will be done in the same way that users would run and keep Bitcoin-Qt/bitcoind open all the time.

The problem with your logic is assuming that more than 10% of target users understand or care about these details.  They heard about Bitcoin from a friend and want to get some coins and buy stuff from Bitcoinstore.com because it has good prices.  I have instantly lost that user if they have to install multiple programs and follow directions to start it.  And it's unnecessary -- I can manage it myself, and they'd prefer me to manage it for them.  This can be win-win for the vast majority of users.   And its default behavior can be the same as bitcoind/-qt:  minimize to tray and run all the time.  Nothing's really different than the way they run bitcoind/-qt, except they have a much more powerful interface on top of bitcoin-qt.

For the 10% of users that do care/understand:  that's what the "Allow using existing already-open Bitcoin-Qt/bitcoind instances" option is for.
We mix two issues here. One is transparent bundling/installation of bitcoind together with armory. Other,and completely indep. dent, is running bitcoind as service on startup by default. The first thing is clearly preferable, the second thing can be configurable and I recommend the default to be "yes". As opposed to all the other windows crapware that has no reservations against draining computer resources, running bitcoind server is useful by strengthening whole network. If users don't care anyway, where's the problem? Who doesn't care, he will lose bitcoins to malware soon anyway.

CoinBr.com: First online MPEx brokerage launched beta! Easy to use interface and reasonable fees. Charts for MPEx stocks: live.coinbr.com * My Blog *
picobit
Hero Member
*****
Offline Offline

Activity: 547


Decor in numeris


View Profile
February 22, 2013, 10:10:51 PM
 #14

We mix two issues here. One is transparent bundling/installation of bitcoind together with armory. Other,and completely indep. dent, is running bitcoind as service on startup by default. The first thing is clearly preferable, the second thing can be configurable and I recommend the default to be "yes". As opposed to all the other windows crapware that has no reservations against draining computer resources, running bitcoind server is useful by strengthening whole network. If users don't care anyway, where's the problem? Who doesn't care, he will lose bitcoins to malware soon anyway.

In principle you are probably right.  It is just not nice to start background processes on users machines without their explicit permission.  Of course many windows programs do just that - but that is no reason to do it as well.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!