Starting preliminary 0.94 testing - "Headless fullnode"

[Carlton Banks]

That's fine, as long as I perceive no risk to me or my affects, then I'm as happy doing that as I've ever been (you're publishing to github after all).

My understanding of the AGPL license is that if you were to modify Armory source and distribute binaries (or run a commercial activity) based on those changes without publishing the altered code publicly, you would be in infringement. I don't think this affects changes for personal and non commercial use. Regardless, if you were to fork Armory publicly (say on a public github repo), you could do as many changes as you like. That's just my interpretation however.

From my perspective there is no license infringement in doing those changes under these conditions, and you have my guarantee I won't be coming after you over this. Again, I'm no lawyer and these words only engage me. I don't have the credentials to speak for the business on this matter.

If anything the responsibility is mine since I asked you to do these changes.

I have next to zero expectation that ATI would do something like that, but stranger things have happened. I'm just not going to use my own valuable time in order to observe something I believe is wrong. So, testing: yes. Reading/acknowledging conditions in "freedom" licenses: no.

"I hereby state the conditions under which you can be free". It's absurd, and fell out of relevance as soon as the first digitally signed commits were available for repo software.

[picobit]

The GPL states that you can use and modify the code in any way you like, with the sole exception that if you distribute the original or modified code you must do so under the same license.  Other open source the licenses do not contain that restriction.  The AGPL place the extra restriction on your use of the code that if you make a publicly facing server using modified code, you must make those modifications publicly available.  In principle, that is a good idea, you then know what you get as a service.  But it does restrict using other peoples codes for your own business purposes.  Fortunately, Armory is not publicly facing, it has Bitcoin Core between itself and the world.  Otherwise, the license would indeed prevent you from trying out such a modification! 

While I agree that trying to protect freedom by enforcing restrictions to said freedom is a bit absurd, it sort of makes sense under the current copyright laws, if you do not want others to make a little change to your code and then claiming it as their own - possibly taking away your own freedom to use it.

Unfortunately, the open source license situation is by now completely out of control, with dozens of slightly incompatible licenses out there.  Trying to create something using parts from several projects is a nightmare.  It became obviously absurd once the GNU Public License stopped being compatible with itself! (version 2 and 3 are incompatible!).

</rant>

[Carlton Banks]

While I agree that trying to protect freedom by enforcing restrictions to said freedom is a bit absurd, it sort of makes sense under the current copyright laws, if you do not want others to make a little change to your code and then claiming it as their own - possibly taking away your own freedom to use it.

You can prove you wrote the code without licensing it, and that was possible pretty much as of the advent of PGP (which was less than 10 years after Stallman's original publication of GPL). Copyright as a basic concept makes no sense for openly distributed code, so there is literally no reason to license the code at all. If someone tries to copyright your code after the fact, then the truth will be plain for all to see.

Unfortunately, the open source license situation is by now completely out of control, with dozens of slightly incompatible licenses out there.  Trying to create something using parts from several projects is a nightmare.  It became obviously absurd once the GNU Public License stopped being compatible with itself! (version 2 and 3 are incompatible!).

</rant>

Glad to hear about how inconvenient this "licensed freedom" stuff is in reality (whoever would have guessed? lol). If I was a lawyer looking at that previous sentence, I would be slavering: "Delicious conflict! Don't stop, I'm almost there..."

[picobit]

Glad to hear about how inconvenient this "licensed freedom" stuff is in reality (whoever would have guessed? lol).

I used to be a strong supporter of open source licenses; and I still firmly support the principle.  But I must admit that they are showing signs of failure. 

BTW, copyright is not something you "claim".  Copyright is something you have automatically if you write something, so if you do *not* release your software under some sort of license then you can certainly sue anybody who uses it in their own code later.  Open source licenses are probably a necessary evil; the original idea being to "subvert" copyright law to do the opposite of what the law intends: to allow you to reuse the code instead of forbid it.  I think that GPL and friends to a large degree have succeeded in that, but their own success (and the resulting proliferation of licenses) hold the seed to failure.

I guess we are neither 100% agreeing, nor 100% disagreeing, but certainly approaching 100% off-topic... 

[Carlton Banks]

All I would add is that people re-using your code or stealing it wholesale doesn't matter.

Someone passing off others ideas as their own only behave that way because they're not sufficiently creative to do it themself. They'll always end up with a magpie's nest of other people's work, so it's all self correcting. If they want new ideas, they will have to go again to those people who actually produce original stuff, you can't fool anyone behaving like that. OTOH, keeping your work proprietary and close to your chest encourages any non-contributors who are privy to the code to manipulate the situation. Proving attribution of unpublished, unsigned work in that case would be far more difficult.

[btchris]

Fortunately, Armory is not publicly facing, it has Bitcoin Core between itself and the world.

Unless you choose to use, say, an armoryd instance to power your blockchain explorer or web store (these are not my words...).

[jl2012]

I'm building a supernode database with 0.93.99.1. Do I need to do it again when 0.94 is finalized and officially released?

[goatpig]

I'm building a supernode database with 0.93.99.1. Do I need to do it again when 0.94 is finalized and officially released?

Unless something goes stupid, I don't intent to modify the format of the DB ever again. I don't expect I'll have to change it to fix the current bug Carlton Banks and btchris are experiencing.

On the other hand I'd like some feedback on stability, since a lot of changes were to improve supernode stability.

[jl2012]

I'm building a supernode database with 0.93.99.1. Do I need to do it again when 0.94 is finalized and officially released?

Unless something goes stupid, I don't intent to modify the format of the DB ever again. I don't expect I'll have to change it to fix the current bug Carlton Banks and btchris are experiencing.

On the other hand I'd like some feedback on stability, since a lot of changes were to improve supernode stability.

On a VM with 4 cores and 29GB RAM, it took 8.4 hours to build to block 270000, about 1/3 of the total blockchain

[jl2012]

I'm building a supernode database with 0.93.99.1. Do I need to do it again when 0.94 is finalized and officially released?

Unless something goes stupid, I don't intent to modify the format of the DB ever again. I don't expect I'll have to change it to fix the current bug Carlton Banks and btchris are experiencing.

On the other hand I'd like some feedback on stability, since a lot of changes were to improve supernode stability.

On a VM with 4 cores and 29GB RAM, it took 8.4 hours to build to block 270000, about 1/3 of the total blockchain

It died around block 270000 with "segmentation fault". Nothing special in log

EDIT: I restarted and it shows "Scanning from 270396 to 357270"

[goatpig]

It died around block 270000 with "segmentation fault". Nothing special in log

EDIT: I restarted and it shows "Scanning from 270396 to 357270"

Well then I guess both fullnode and supernode suffer from the same symptoms. As for the resume height, that looks correct to me.

[Carlton Banks]

It died around block 270000 with "segmentation fault". Nothing special in log

EDIT: I restarted and it shows "Scanning from 270396 to 357270"

Well then I guess both fullnode and supernode suffer from the same symptoms. As for the resume height, that looks correct to me.

Would it help if I try building a Supernode Db? I'm retiring the 0.9.x bitcoin blockchain from my system disk, so I'll have the space and the patience to let it build.

[jl2012]

It died around block 270000 with "segmentation fault". Nothing special in log

EDIT: I restarted and it shows "Scanning from 270396 to 357270"

Well then I guess both fullnode and supernode suffer from the same symptoms. As for the resume height, that looks correct to me.

How do I know if it's running properly? Wait until it finishes 272500?

EDIT: It passed 272500. Can I assume my database is not corrupted?

[goatpig]

EDIT: It passed 272500. Can I assume my database is not corrupted?

Yes

Would it help if I try building a Supernode Db? I'm retiring the 0.9.x bitcoin blockchain from my system disk, so I'll have the space and the patience to let it build.

I got some changes coming in soon, I'd rather you wait on those

[jl2012]

It died around block 270000 with "segmentation fault". Nothing special in log

EDIT: I restarted and it shows "Scanning from 270396 to 357270"

Well then I guess both fullnode and supernode suffer from the same symptoms. As for the resume height, that looks correct to me.

How do I know if it's running properly? Wait until it finishes 272500?

EDIT: It passed 272500. Can I assume my database is not corrupted?

The armoryd suddenly runs very slowly and makes my VM very unresponsive

It took 2089s to complete 265000-267500
3848s for 267500-270000

everything was fine up to this point

it died with "segmentation fault" at 270396 and I restarted

It took 3291s to complete 270396-272500
11847s to complete 272500-275000

the VM becomes very unresponsive. I renice the armoryd to 19 but seems not very helpful. Few minutes after I renice it, it died showing "Killed" but I don't think I did anything to kill it manually. The VM becomes normal after armoryd is dead.

I know later blocks will take longer to process but I don't think it would suddenly take 3x longer.

EDIT: after restarting armoryd it took only 3717s to complete 272867-275000. So there must be some problem before

[jl2012]

It became very slow again and I restarted the VM. However, it is redoing ~4000 completed blocks


-WARN  - 1432179524: (BlockUtils.cpp:1071) Scanning from 272867 to 357353
-WARN  - 1432183241: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 275000
-WARN  - 1432190441: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 277500
-WARN  - 1432212681: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 280000

(I killed it here with Control-C)

Log file opened at 1432217549: /home/xxx/.armory/armorycpplog.txt
-INFO  - 1432217554: (BlockUtils.cpp:850) blkfile dir: /home/xxx/.bitcoin/blocks
-INFO  - 1432217554: (BlockUtils.cpp:851) lmdb dir: /home/xxx/.armory/databases
-INFO  - 1432217554: (lmdb_wrapper.cpp:439) Opening databases...
-INFO  - 1432217554: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad
-INFO  - 1432217554: (BlockUtils.cpp:1253) Total number of blk*.dat files: 272
-INFO  - 1432217554: (BlockUtils.cpp:1254) Total blockchain bytes: 36,422,961,275
-INFO  - 1432217554: (BlockUtils.cpp:1628) Reading headers from db
-INFO  - 1432217567: (BlockUtils.cpp:1654) Found 357398 headers in db
-DEBUG - 1432217567: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1432217574: (BlockUtils.cpp:1295) Left off at file 271, offset 65665089
-INFO  - 1432217574: (BlockUtils.cpp:1298) Reading headers and building chain...
-INFO  - 1432217574: (BlockUtils.cpp:1299) Starting at block file 271 offset 65665089
-INFO  - 1432217574: (BlockUtils.cpp:1301) Block height 357353
-DEBUG - 1432217574: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1432217575: (BlockUtils.cpp:1337) Looking for first unrecognized block
-INFO  - 1432217575: (BlockUtils.cpp:1489) Loading block data... file 271 offset 65665081
-ERROR - 1432217575: (BlockUtils.cpp:516) Next block header found at offset 65665089
-INFO  - 1432217888: (BlockUtils.cpp:544) Reading raw blocks finished at file 271 offset 87382506
-INFO  - 1432217888: (BlockUtils.cpp:1354) Wrote blocks to DB in 7.70157s
-INFO  - 1432217888: (BlockUtils.cpp:1371) Checking dupIDs from 276324 onward
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1556) Starting with:
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1557) 1 workers
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1558) 1 writers
-WARN  - 1432217900: (BlockUtils.cpp:1071) Scanning from 276325 to 357427

[goatpig]

It became very slow again and I restarted the VM. However, it is redoing ~4000 completed blocks


-WARN  - 1432179524: (BlockUtils.cpp:1071) Scanning from 272867 to 357353
-WARN  - 1432183241: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 275000
-WARN  - 1432190441: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 277500
-WARN  - 1432212681: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 280000

(I killed it here with Control-C)

Log file opened at 1432217549: /home/xxx/.armory/armorycpplog.txt
-INFO  - 1432217554: (BlockUtils.cpp:850) blkfile dir: /home/xxx/.bitcoin/blocks
-INFO  - 1432217554: (BlockUtils.cpp:851) lmdb dir: /home/xxx/.armory/databases
-INFO  - 1432217554: (lmdb_wrapper.cpp:439) Opening databases...
-INFO  - 1432217554: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad
-INFO  - 1432217554: (BlockUtils.cpp:1253) Total number of blk*.dat files: 272
-INFO  - 1432217554: (BlockUtils.cpp:1254) Total blockchain bytes: 36,422,961,275
-INFO  - 1432217554: (BlockUtils.cpp:1628) Reading headers from db
-INFO  - 1432217567: (BlockUtils.cpp:1654) Found 357398 headers in db
-DEBUG - 1432217567: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1432217574: (BlockUtils.cpp:1295) Left off at file 271, offset 65665089
-INFO  - 1432217574: (BlockUtils.cpp:1298) Reading headers and building chain...
-INFO  - 1432217574: (BlockUtils.cpp:1299) Starting at block file 271 offset 65665089
-INFO  - 1432217574: (BlockUtils.cpp:1301) Block height 357353
-DEBUG - 1432217574: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1432217575: (BlockUtils.cpp:1337) Looking for first unrecognized block
-INFO  - 1432217575: (BlockUtils.cpp:1489) Loading block data... file 271 offset 65665081
-ERROR - 1432217575: (BlockUtils.cpp:516) Next block header found at offset 65665089
-INFO  - 1432217888: (BlockUtils.cpp:544) Reading raw blocks finished at file 271 offset 87382506
-INFO  - 1432217888: (BlockUtils.cpp:1354) Wrote blocks to DB in 7.70157s
-INFO  - 1432217888: (BlockUtils.cpp:1371) Checking dupIDs from 276324 onward
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1556) Starting with:
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1557) 1 workers
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1558) 1 writers
-WARN  - 1432217900: (BlockUtils.cpp:1071) Scanning from 276325 to 357427

The wording is off. It signals the highest buffered block, not the highest processed and written block (cause I was lazy and didn't change this kinda stuff)

[jl2012]

It became very slow again and I restarted the VM. However, it is redoing ~4000 completed blocks


-WARN  - 1432179524: (BlockUtils.cpp:1071) Scanning from 272867 to 357353
-WARN  - 1432183241: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 275000
-WARN  - 1432190441: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 277500
-WARN  - 1432212681: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 280000

(I killed it here with Control-C)

Log file opened at 1432217549: /home/xxx/.armory/armorycpplog.txt
-INFO  - 1432217554: (BlockUtils.cpp:850) blkfile dir: /home/xxx/.bitcoin/blocks
-INFO  - 1432217554: (BlockUtils.cpp:851) lmdb dir: /home/xxx/.armory/databases
-INFO  - 1432217554: (lmdb_wrapper.cpp:439) Opening databases...
-INFO  - 1432217554: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad
-INFO  - 1432217554: (BlockUtils.cpp:1253) Total number of blk*.dat files: 272
-INFO  - 1432217554: (BlockUtils.cpp:1254) Total blockchain bytes: 36,422,961,275
-INFO  - 1432217554: (BlockUtils.cpp:1628) Reading headers from db
-INFO  - 1432217567: (BlockUtils.cpp:1654) Found 357398 headers in db
-DEBUG - 1432217567: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1432217574: (BlockUtils.cpp:1295) Left off at file 271, offset 65665089
-INFO  - 1432217574: (BlockUtils.cpp:1298) Reading headers and building chain...
-INFO  - 1432217574: (BlockUtils.cpp:1299) Starting at block file 271 offset 65665089
-INFO  - 1432217574: (BlockUtils.cpp:1301) Block height 357353
-DEBUG - 1432217574: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1432217575: (BlockUtils.cpp:1337) Looking for first unrecognized block
-INFO  - 1432217575: (BlockUtils.cpp:1489) Loading block data... file 271 offset 65665081
-ERROR - 1432217575: (BlockUtils.cpp:516) Next block header found at offset 65665089
-INFO  - 1432217888: (BlockUtils.cpp:544) Reading raw blocks finished at file 271 offset 87382506
-INFO  - 1432217888: (BlockUtils.cpp:1354) Wrote blocks to DB in 7.70157s
-INFO  - 1432217888: (BlockUtils.cpp:1371) Checking dupIDs from 276324 onward
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1556) Starting with:
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1557) 1 workers
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1558) 1 writers
-WARN  - 1432217900: (BlockUtils.cpp:1071) Scanning from 276325 to 357427

The wording is off. It signals the highest buffered block, not the highest processed and written block (cause I was lazy and didn't change this kinda stuff)

So it took 8 hours to buffer ~5000 blocks without processing? How could I know the actual progress?

[goatpig]

It became very slow again and I restarted the VM. However, it is redoing ~4000 completed blocks


-WARN  - 1432179524: (BlockUtils.cpp:1071) Scanning from 272867 to 357353
-WARN  - 1432183241: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 275000
-WARN  - 1432190441: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 277500
-WARN  - 1432212681: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 280000

(I killed it here with Control-C)

Log file opened at 1432217549: /home/xxx/.armory/armorycpplog.txt
-INFO  - 1432217554: (BlockUtils.cpp:850) blkfile dir: /home/xxx/.bitcoin/blocks
-INFO  - 1432217554: (BlockUtils.cpp:851) lmdb dir: /home/xxx/.armory/databases
-INFO  - 1432217554: (lmdb_wrapper.cpp:439) Opening databases...
-INFO  - 1432217554: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad
-INFO  - 1432217554: (BlockUtils.cpp:1253) Total number of blk*.dat files: 272
-INFO  - 1432217554: (BlockUtils.cpp:1254) Total blockchain bytes: 36,422,961,275
-INFO  - 1432217554: (BlockUtils.cpp:1628) Reading headers from db
-INFO  - 1432217567: (BlockUtils.cpp:1654) Found 357398 headers in db
-DEBUG - 1432217567: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1432217574: (BlockUtils.cpp:1295) Left off at file 271, offset 65665089
-INFO  - 1432217574: (BlockUtils.cpp:1298) Reading headers and building chain...
-INFO  - 1432217574: (BlockUtils.cpp:1299) Starting at block file 271 offset 65665089
-INFO  - 1432217574: (BlockUtils.cpp:1301) Block height 357353
-DEBUG - 1432217574: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1432217575: (BlockUtils.cpp:1337) Looking for first unrecognized block
-INFO  - 1432217575: (BlockUtils.cpp:1489) Loading block data... file 271 offset 65665081
-ERROR - 1432217575: (BlockUtils.cpp:516) Next block header found at offset 65665089
-INFO  - 1432217888: (BlockUtils.cpp:544) Reading raw blocks finished at file 271 offset 87382506
-INFO  - 1432217888: (BlockUtils.cpp:1354) Wrote blocks to DB in 7.70157s
-INFO  - 1432217888: (BlockUtils.cpp:1371) Checking dupIDs from 276324 onward
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1556) Starting with:
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1557) 1 workers
-WARN  - 1432217900: (BlockWriteBatcher.cpp:1558) 1 writers
-WARN  - 1432217900: (BlockUtils.cpp:1071) Scanning from 276325 to 357427

The wording is off. It signals the highest buffered block, not the highest processed and written block (cause I was lazy and didn't change this kinda stuff)

So it took 8 hours to buffer ~5000 blocks without processing? How could I know the actual progress?

Buffering, processing and writing data are all different tasks that take place in parallel. A batch of blocks is buffered, processed and written, then the batch is cleaned up. Only after a batch has been written will the DB resume past that point. The buffering threads have a bottleneck (only 3 buffers in the waiting queue at most). The processing threads have no bottleneck (and usually out pace the writing threads in supernode) which is why the scanning is getting really slow (RAM gets filled with too much data to allow the writers to run properly). As for why this happens, it's cause I forgot to turn the processing threads bottleneck back on when I was doing some tests o.o"!

As for progress, I'll improve the log message with the upcoming fixes.

[jl2012]

As for progress, I'll improve the log message with the upcoming fixes.

Thanks. I hope it could be logged by fixed time interval, not by block interval. It would be very helpful for slow machine like mine.