newb4now
|
|
September 14, 2015, 12:38:25 AM |
|
what makes the leveldb integration at boolberry different than what monero is doing? Or it is somewhat similar?
I know the mixin rules, pruning, difficulty algorithm and emission calendar are all different
|
|
|
|
smooth
Legendary
Offline
Activity: 2968
Merit: 1198
|
|
September 14, 2015, 01:26:48 AM Last edit: September 14, 2015, 09:01:58 AM by smooth |
|
what makes the leveldb integration at boolberry different than what monero is doing? Or it is somewhat similar?
The leveldb here is only being used to store list of blocks, not all the other in-memory stuff that makes up a node (transactions, block index, output indexes, etc.). Monero moves all that stuff to the DB. Advantages of BBR approach: 1. Simpler to implement and get right 2. Some operations still using the in-memory data may be more efficient Disadvantages of BBR approach: 3. Higher remaining memory usage (but with BBR's pruning and lower volume of usage, probably not too bad) 4. Some possibility of inconsistencies between the two data stores in-mem and database (clintar's fix addresses one instance of this) 5. Still requires the periodic "save blockchain" timer, which pauses many daemon functions during the saving process (though the process should be faster without needing to save the block list) #1 is pretty significant IMO. Given that these projects are all works in progress, building the ultimate solution on the first go is not aways the best approach. EDIT: added #5
|
|
|
|
clintar2
|
|
September 14, 2015, 03:44:38 AM |
|
what makes the leveldb integration at boolberry different than what monero is doing? Or it is somewhat similar?
The leveldb here is only being used to store list of blocks, not all the other in-memory stuff that makes up a node (transactions, block index, output indexes, etc.). Monero moves all that stuff to the DB. Advantages of BBR approach: 1. Simpler to implement and get right 2. Some operations still using the in-memory data may be more efficient Disadvantages of BBR approach: 3. Higher remaining memory usage (but with BBR's pruning and lower volume of usage, probably not too bad) 4. Some possibility of inconsistencies between the two data stores in-mem and database (clintar's fix addresses one instance of this) #1 is pretty significant IMO. Given that these projects are all works in progress, building the ultimate solution on the first go is not aways the best approach. Is the other instance you refer to if the saved blockchain is more recent that the lmdb database (which I'm not sure how that would happen in practice unless you copied a newer blockchain.bin where there's an older db), or are you talking about something else?
|
|
|
|
smooth
Legendary
Offline
Activity: 2968
Merit: 1198
|
|
September 14, 2015, 05:03:26 AM Last edit: September 14, 2015, 08:57:21 AM by smooth |
|
Is the other instance you refer to if the saved blockchain is more recent that the lmdb database (which I'm not sure how that would happen in practice unless you copied a newer blockchain.bin where there's an older db), or are you talking about something else?
Just in general that the process isn't transactional the way an ACID database commit normally would be so things can go wrong in various ways. Without carefully auditing the ordering of events and the behavior of various operating systems it is hard to be 100% sure what failure modes are possible. For example, another failure that occasionally happens on Windows (for all cryptonotes) is the blockchain.bin file gets corrupted during save since there isn't a really good way to replace a file atomically (on Linux you can rename-replace, which mostly works). This would leave you with a valid database but an invalid blockchain.bin. Of course this would probably be no worse than the previous (no db) case, but would be worse than the full db case. This is fixable, but somewhat annoying to fix, especially if all the data is going to end up in a database eventually anyway (I have no idea as to your plans). BTW, in case my comments were not clear I do think this is nice work by crypto_zoidberg and clintar.
|
|
|
|
clintar2
|
|
September 14, 2015, 08:50:00 AM |
|
Is the other instance you refer to if the saved blockchain is more recent that the lmdb database (which I'm not sure how that would happen in practice unless you copied a newer blockchain.bin where there's an older db), or are you talking about something else?
Just in general that the process isn't transnational the way an ACID database commit normally would be so things can go wrong in various ways. Without carefully auditing the ordering of events and the behavior of various operating systems it is hard to be 100% sure what failure modes are possible. For example, another failure that occasionally happens on Windows (for all cryptonotes) is the blockchain.bin file gets corrupted during save since there isn't a really good way to replace a file atomically (on Linux you can rename-replace, which mostly works). This would leave you with a valid database but an invalid blockchain.bin. Of course this would probably be no worse than the previous (no db) case, but would be worse than the full db case. This is fixable, but somewhat annoying to fix, especially if all the data is going to end up in a database eventually anyway (I have no idea as to your plans). BTW, in case my comments were not clear I do think this is nice work by crypto_zoidberg and clintar. Ok, thanks for the input. Just wondered if there was something I should be looking into I hadn't thought of. I don't really know the plans, either. Just was contributing what I came up with as a work-around. I agree having everything in the database does take care if these issues, too. Pretty slick how cz implemented the db, though, huh? At least I thought so. Anyone have any suggestions for things they want to see in the codebase?
|
|
|
|
languagehasmeaning
|
|
September 14, 2015, 11:53:36 PM |
|
Is the other instance you refer to if the saved blockchain is more recent that the lmdb database (which I'm not sure how that would happen in practice unless you copied a newer blockchain.bin where there's an older db), or are you talking about something else?
Just in general that the process isn't transnational the way an ACID database commit normally would be so things can go wrong in various ways. Without carefully auditing the ordering of events and the behavior of various operating systems it is hard to be 100% sure what failure modes are possible. For example, another failure that occasionally happens on Windows (for all cryptonotes) is the blockchain.bin file gets corrupted during save since there isn't a really good way to replace a file atomically (on Linux you can rename-replace, which mostly works). This would leave you with a valid database but an invalid blockchain.bin. Of course this would probably be no worse than the previous (no db) case, but would be worse than the full db case. This is fixable, but somewhat annoying to fix, especially if all the data is going to end up in a database eventually anyway (I have no idea as to your plans). BTW, in case my comments were not clear I do think this is nice work by crypto_zoidberg and clintar. Ok, thanks for the input. Just wondered if there was something I should be looking into I hadn't thought of. I don't really know the plans, either. Just was contributing what I came up with as a work-around. I agree having everything in the database does take care if these issues, too. Pretty slick how cz implemented the db, though, huh? At least I thought so. Anyone have any suggestions for things they want to see in the codebase? I would love to see support for multisig transactions! Is that a realistic goal?
|
|
|
|
|
newb4now
|
|
September 16, 2015, 04:59:39 PM |
|
what makes the leveldb integration at boolberry different than what monero is doing? Or it is somewhat similar?
The leveldb here is only being used to store list of blocks, not all the other in-memory stuff that makes up a node (transactions, block index, output indexes, etc.). Monero moves all that stuff to the DB. Advantages of BBR approach: 1. Simpler to implement and get right 2. Some operations still using the in-memory data may be more efficient Disadvantages of BBR approach: 3. Higher remaining memory usage (but with BBR's pruning and lower volume of usage, probably not too bad) 4. Some possibility of inconsistencies between the two data stores in-mem and database (clintar's fix addresses one instance of this) 5. Still requires the periodic "save blockchain" timer, which pauses many daemon functions during the saving process (though the process should be faster without needing to save the block list) #1 is pretty significant IMO. Given that these projects are all works in progress, building the ultimate solution on the first go is not aways the best approach. EDIT: added #5 Thanks for your detailed and impartial response! I found it very helpful
|
|
|
|
clintar2
|
|
September 16, 2015, 09:41:27 PM |
|
what makes the leveldb integration at boolberry different than what monero is doing? Or it is somewhat similar?
The leveldb here is only being used to store list of blocks, not all the other in-memory stuff that makes up a node (transactions, block index, output indexes, etc.). Monero moves all that stuff to the DB. Advantages of BBR approach: 1. Simpler to implement and get right 2. Some operations still using the in-memory data may be more efficient Disadvantages of BBR approach: 3. Higher remaining memory usage (but with BBR's pruning and lower volume of usage, probably not too bad) 4. Some possibility of inconsistencies between the two data stores in-mem and database (clintar's fix addresses one instance of this) 5. Still requires the periodic "save blockchain" timer, which pauses many daemon functions during the saving process (though the process should be faster without needing to save the block list) #1 is pretty significant IMO. Given that these projects are all works in progress, building the ultimate solution on the first go is not aways the best approach. EDIT: added #5 Yeah, #5 is probably the main reason I'd like to see everything changed to db. I might try to tackle that based on cz's main blocks changes.
|
|
|
|
crypto_zoidberg (OP)
|
|
September 18, 2015, 06:07:35 AM |
|
what makes the leveldb integration at boolberry different than what monero is doing? Or it is somewhat similar?
The leveldb here is only being used to store list of blocks, not all the other in-memory stuff that makes up a node (transactions, block index, output indexes, etc.). Monero moves all that stuff to the DB. Advantages of BBR approach: 1. Simpler to implement and get right 2. Some operations still using the in-memory data may be more efficient Disadvantages of BBR approach: 3. Higher remaining memory usage (but with BBR's pruning and lower volume of usage, probably not too bad) 4. Some possibility of inconsistencies between the two data stores in-mem and database (clintar's fix addresses one instance of this) Agree with this, actually this is the main reason why DB implementaton stil in branch. The way we decided to go here - is to step-by-step move parts of storage from boost serialization to leveldb - is way that let us easely reduce memory usage, but at other hand we need always have consistent state of both parts: serialized storage and leveldb. The problem is that leveldb write changes right after every operations (there are different options, but still the idea of level db is like this). Clintar made workaround for current implementation, but this won't work if i'll move other containers to leveldb(or the workaround become be very-very complicated). So now it's only betta. Leveldb have some kind of "snapshots", this could work but db can't be reverted to this snapshots, so it useless i guess.
The only way i see now is to move all stored data to leveldb and use batch writes to keep data consistent.
5. Still requires the periodic "save blockchain" timer, which pauses many daemon functions during the saving process (though the process should be faster without needing to save the block list)
#1 is pretty significant IMO. Given that these projects are all works in progress, building the ultimate solution on the first go is not aways the best approach.
EDIT: added #5
|
|
|
|
languagehasmeaning
|
|
September 19, 2015, 08:38:17 AM |
|
I love boolberry but I now have a new favorite thread: https://bitcointalk.org/index.php?topic=1148538.0(come join if you like chess) Back on topic..... I would love to hear feedback from those testing the new db code from github. Do we have any data about how many people have tested it and any trouble (if any) they have ran into? I thought the cointopay.com boolberry acceptance news was a good thing. Has anyone used the service yet?
|
|
|
|
newb4now
|
|
September 19, 2015, 08:08:49 PM |
|
what makes the leveldb integration at boolberry different than what monero is doing? Or it is somewhat similar?
The leveldb here is only being used to store list of blocks, not all the other in-memory stuff that makes up a node (transactions, block index, output indexes, etc.). Monero moves all that stuff to the DB. Advantages of BBR approach: 1. Simpler to implement and get right 2. Some operations still using the in-memory data may be more efficient Disadvantages of BBR approach: 3. Higher remaining memory usage (but with BBR's pruning and lower volume of usage, probably not too bad) 4. Some possibility of inconsistencies between the two data stores in-mem and database (clintar's fix addresses one instance of this) Agree with this, actually this is the main reason why DB implementaton stil in branch. The way we decided to go here - is to step-by-step move parts of storage from boost serialization to leveldb - is way that let us easely reduce memory usage, but at other hand we need always have consistent state of both parts: serialized storage and leveldb. The problem is that leveldb write changes right after every operations (there are different options, but still the idea of level db is like this). Clintar made workaround for current implementation, but this won't work if i'll move other containers to leveldb(or the workaround become be very-very complicated). So now it's only betta. Leveldb have some kind of "snapshots", this could work but db can't be reverted to this snapshots, so it useless i guess.
The only way i see now is to move all stored data to leveldb and use batch writes to keep data consistent.
5. Still requires the periodic "save blockchain" timer, which pauses many daemon functions during the saving process (though the process should be faster without needing to save the block list)
#1 is pretty significant IMO. Given that these projects are all works in progress, building the ultimate solution on the first go is not aways the best approach.
EDIT: added #5
I sounds like you have put a lot of thought into this. Thank you for your careful planning. I am really excited about the future of boolberry
|
|
|
|
bitcoinrocks
Legendary
Offline
Activity: 1372
Merit: 1000
|
|
September 20, 2015, 07:05:36 PM |
|
Is there a Slack channel for BBR?
|
|
|
|
boolberry
|
|
September 21, 2015, 02:57:03 AM |
|
Is there a Slack channel for BBR?
Not yet but that is a good idea. SuperNet has a slack channel set up and we are included but that is not really the same thing as having our own.
|
|
|
|
jd1959
|
|
September 21, 2015, 03:12:33 AM |
|
Is there a Slack channel for BBR?
I personally believe that slack is over rated. Mostly just people chatting about irrelevant crap. It looks like a good place for Dev's to chat about works in progress with each other, then make announcements here. I've been on the Supernet slack Chanel and it was a waste of time for someone who can't contribute to Code to solve issues. I'm just an end user and don't mean any offense to Dev's, I can't do what they can. Jon
|
dICO Disguised Instant Cash Out
|
|
|
boolberry
|
|
September 21, 2015, 03:24:21 AM |
|
I love boolberry but I now have a new favorite thread: https://bitcointalk.org/index.php?topic=1148538.0(come join if you like chess) Back on topic..... I would love to hear feedback from those testing the new db code from github. Do we have any data about how many people have tested it and any trouble (if any) they have ran into? I thought the cointopay.com boolberry acceptance news was a good thing. Has anyone used the service yet? I like chess so don't mind your post at all! Thanks for the reminder about cointopay. I will tweet about them again
|
|
|
|
yassin54
Legendary
Offline
Activity: 1540
Merit: 1000
|
|
September 21, 2015, 09:54:33 AM |
|
Is there a Slack channel for BBR?
there are channel BBR in slack supernet go invit supernet here: http://slackinvite.supernet.org/and add channel #BBR You are Welcome!
|
|
|
|
bitcoinrocks
Legendary
Offline
Activity: 1372
Merit: 1000
|
|
September 21, 2015, 01:29:47 PM |
|
Do you mean the #boolberry channel or is #BBR different?
|
|
|
|
Nanashi74
Newbie
Offline
Activity: 8
Merit: 0
|
|
September 21, 2015, 07:31:21 PM |
|
getting the following errors when compiling from git: Linking CXX static library libcommon.a Error running link command: No such file or directory make[3]: *** [src/libcommon.a] Error 2 make[3]: Leaving directory `/home/nanashi/.boolb/boolberry/build/release' make[2]: *** [src/CMakeFiles/common.dir/all] Error 2 Linking CXX static library libcrypto.a Error running link command: No such file or directory make[3]: *** [src/libcrypto.a] Error 2 make[3]: Leaving directory `/home/nanashi/.boolb/boolberry/build/release' make[2]: *** [src/CMakeFiles/crypto.dir/all] Error 2 make[2]: Leaving directory `/home/nanashi/.boolb/boolberry/build/release' make[1]: *** [all] Error 2 make[1]: Leaving directory `/home/nanashi/.boolb/boolberry/build/release' make: *** [build-release] Error 2 mint 17.2
|
|
|
|
clintar2
|
|
September 21, 2015, 07:49:49 PM |
|
getting the following errors when compiling from git: Linking CXX static library libcommon.a Error running link command: No such file or directory make[3]: *** [src/libcommon.a] Error 2 make[3]: Leaving directory `/home/nanashi/.boolb/boolberry/build/release' make[2]: *** [src/CMakeFiles/common.dir/all] Error 2 Linking CXX static library libcrypto.a Error running link command: No such file or directory make[3]: *** [src/libcrypto.a] Error 2 make[3]: Leaving directory `/home/nanashi/.boolb/boolberry/build/release' make[2]: *** [src/CMakeFiles/crypto.dir/all] Error 2 make[2]: Leaving directory `/home/nanashi/.boolb/boolberry/build/release' make[1]: *** [all] Error 2 make[1]: Leaving directory `/home/nanashi/.boolb/boolberry/build/release' make: *** [build-release] Error 2 mint 17.2 Did you install build-essential first?
|
|
|
|
|