[ANN][BURST] Burst | Efficient HDD Mining | New 1.2.3 Fork block 92000

[burstcoin]

...
What more do you want? BURST already has more built in features than any online banking out there.

this:

The Burst dev needs to convert Burst network into decentralized dropbox ASAP, when users would pay for distributed encrypted storing of their files with BURST coins. This is the only way for Burst to survive.

Have you looked to this : http://storj.io/  ??

what i hope the difference between burst data storage and storj.io is we are using pay by data usage instead of storj's pay for disk space, so storing small files would be relatively very cheap, this way the burst node network can evolve into CDN serving nodes other than file storage

the hardest thing to design that is that you have to rely that there are some data copies available whenever you want to access them.
the best would be to be able to book a certain amount of bandwidth plus a guaranteed period of time plus a specified number of copies whenever you put a storage object into the network. compared to dropbox this can be realized by folder attributes.
this concept is a service and no mining except if you add a asset which can only be earned if you put a specific nonce into the storage objects. maybe if you "host" 256mb encrypted data you get allowed to add one nonce for it. these nonces have to have another weight than simple mining nonces. each time you host the file as long as 10% of the booked time you are allowed to add another nonce or the weight of yours get adjusted (similar to the timekoin generator reward). if the period of booked time for the data package has exceeded it can either get renewed and if not the accounts hosted the data can keep their nonces for that package in reverse order (10 month hosted, 20 month additional reward).
technically it is like a huge private torrent network backed by the blockchain (as tracker) and miners.
for me it makes most sense at the moment to add filesharing like an asset.
if you want to store files you require burst to buy this asset. if you only want to host data you receive this asset and can trade it for burst.
this concept utilizes the mining infastructure people have (except the bandwidth usage) to store files but also requires people to pay similar like if the miners use their infastructure for pure mining.
if i say 1 gb local storage may mine 18 burst within a month and i want to have 500 copies of my data i would have to pay 9000 burst for 1 gb cloud storage at the current diff. if you reduce the number of copies to 50 1gb would cost 50 dollarcent at current prices.
if the price stays stable at the current level this calculation would be fine. if the price settles at 2000 satoshi or more anytime soon its a way too expensive to use.
one of the most tricky designs is to make it for the miners similar provitable to host files as to mine. tricky are the redundant copies you require. nobody finally really cares to not loose data blocks on system failures. to prevent and recover this i can think of a infastructure which maintains itself. lets say you have defined periods of time to resync your hosted files or someone else gets them assigned and payed for. also the bandwidth usage is huge. this means if someone mines today with 4 tb he could only upload on average 4gb a day and blocks his line completely. if you factor this in you come to a ratio of how many files can be stored. with todays 4500 miners 18 tb cloudstorage could be realized. if you say you keep 100 copies of the files you end up at about 200gb which every miner stores.
if you try to keep everything in correlation with the block rewards and market prices its hard to predict what will happen to such a service.
i think over time the redundant copies make it much more expensive to use than if you host your data on a dedicated  server.
i currently dont know how price structures for data hosting look like but backblaze offers a complete backup for exactly 5 dollar a month.
if i sum everything up this is what a dropbox like burst extension requires:
-you define the number of copies you want to have
-you define the guaranteed bandwidth you can access your files with
-you define how long the files shall be stored
-after the defined time the storage service renews itself based on current market prices for the same time if your balance allows it
-you define who can access it
-miners have to have similar profits wether they store files or use their diskspace for mining (correlation between block reward and diff)
-the network reassigns lost storage slots to other people if a wallet could not be accessed for a certain period of time
-storage may not cost more than regular cloud storage (depends on the number of copies)

Rather than trying ensure a certain number of copies, an alternate approach is to make it most profitable to host data pieces that have the fewest other mirrors. Here's what I'm thinking: Use payments for storage as a 'storage reward'. The storage reward for a file will come around seemingly randomly but selected deterministically by running network state(gensig, etc) through a hashing-based algorithm that also takes a probability of a block having a storage reward for that file. When the storage reward time for a file comes up, there will also be a deterministically selected portion of the file. Miners who have the file stored can calculate a proof-of-knowledge of that piece of the file based off a merkle tree root that was stored in the blockchain when the file was stored, and announce an intent to collect the reward containing a hash of something along the lines of sha256(userId concat sha256(knowledge proof)) so that the knowledge proof is not yet revealed. After a set amount of time, any node can announce the proof-of-knowlege. A selection method is then used to select one of the miners to receive the storage reward. Using this method one shouldn't have to attempt to ensure a certain number of copies, as the most effective method of mining is browse the dht seeking out files that have the fewest number of copies available, and host copies of them, since they offer the best chance of receiving a storage reward, as they will have the fewest amount of people competing for the reward.

[1714526128]

1714526128

[1714526128]

1714526128

[1714526128]

1714526128

[unamis76]

Since this has been around for a while, I think it's important to ask... Is there any wear and tear done by this kind of mining, any loss of performance? Specially on SSD's...

[Nevril]

Since this has been around for a while, I think it's important to ask... Is there any wear and tear done by this kind of mining, any loss of performance? Specially on SSD's...

SSD makes no sense in this kind of mining. I wouldn't put a plot on it because basically you have few space, and bandwidth doesn't matter that much on those sizes.
As far as I can tell (mining since 17 days) all my HDD are running fine, I have 4 on my gaming laptop, 2 internal (SSD and personal data) and 2 external (mining). No performance losses, even when gaming except for the CPU going to full load when a new block is found, but it lasts few seconds and if you don't reserve all the cores for mining you won't lose more than 2 or 3 fps for those seconds.

[majere]

Since this has been around for a while, I think it's important to ask... Is there any wear and tear done by this kind of mining, any loss of performance? Specially on SSD's...

Make sure your HDDs aren't parking often, this will kill them fast. If they do, add something like this to 'crontab -e':

Code:

*/2 * * * * echo `/bin/date` > /Volumes/hdd1/keepalive.txt
*/2 * * * * echo `/bin/date` > /Volumes/hdd2/keepalive.txt

Also, ensure plot files aren't fragmented and large enough stagger is used so they're read sequentially. This way tear should be minimal.

For SSDs the above doesn't matter much, reading during mining shouldn't put much stress on them.

p.s. Regarding storj: most ppl I talked to don't like the idea of storing files on their hdds (even encrypted ones). They're afraid of illegal content potentially being stored. Burst doesn't store any real data and many view this as an advantage.

[hero18688]

I am wondering if mining on a 5400 rpm HDD has the same performance as a 7200 rpm hdd.If their capacity is the same.

[crowetic]

Since this has been around for a while, I think it's important to ask... Is there any wear and tear done by this kind of mining, any loss of performance? Specially on SSD's...

Make sure your HDDs aren't parking often, this will kill them fast. If they do, add something like this to 'crontab -e':

Code:

*/2 * * * * echo `/bin/date` > /Volumes/hdd1/keepalive.txt
*/2 * * * * echo `/bin/date` > /Volumes/hdd2/keepalive.txt

Also, ensure plot files aren't fragmented and large enough stagger is used so they're read sequentially. This way tear should be minimal.

For SSDs the above doesn't matter much, reading during mining shouldn't put much stress on them.

p.s. Regarding storj: most ppl I talked to don't like the idea of storing files on their hdds (even encrypted ones). They're afraid of illegal content potentially being stored. Burst doesn't store any real data and many view this as an advantage.

I agree about the storage part,  I don't think burst needs to worry about becoming decentralized storage, the issue you mentioned as well as bandwidth come into play, also, not sure how they would work it with so many people having different connections and speeds. but yea... BURST doesn't need to be storage, it's the mining that's the innovation here.

it COULD however, be a decentralized marketplace, with escrow. That could already happen, and would be a great plan. IMO

[crowetic]

I am wondering if mining on a 5400 rpm HDD has the same performance as a 7200 rpm hdd.If their capacity is the same.

You're wondering loudly. lol.

I honestly don't know since I haven't run anything but 7200 RPMs, but I don't believe there should be that much of a difference. Maybe someone with a little more knowledge on the underside of how burst actulally mines could tell you.

But I would say it's definitely fine to mine on them, and it won't be that much of a difference.

[fivebells]

I am wondering if mining on a 5400 rpm HDD has the same performance as a 7200 rpm hdd.If their capacity is the same.

It depends on how long it takes to read a set of scoops.  If it takes more than 30s or so, you may sometimes miss a short deadline.

[majere]

I am wondering if mining on a 5400 rpm HDD has the same performance as a 7200 rpm hdd.If their capacity is the same.

It depends on how long it takes to read a set of scoops.  If it takes more than 30s or so, you may sometimes miss a short deadline.

Sequential read speed matters the most. My new 5400 rpm drive with 170 mb/s outperforms older 7200 rpm WDBlack which reads 120 mb/s. But they finish reading plots simultaneously because the latter has lower capacity.

Some 7200 rpm drives also tend to vibrate more and require extra measures to prevent resonance effect, so I'm not sure if they're better if you're building a rack.

This mostly matters for pools where you accumulate results before sending them to the pool as you can miss a block if one of HDDs is really slow.

[Irontiga]

Since this has been around for a while, I think it's important to ask... Is there any wear and tear done by this kind of mining, any loss of performance? Specially on SSD's...

Make sure your HDDs aren't parking often, this will kill them fast. If they do, add something like this to 'crontab -e':

Code:

*/2 * * * * echo `/bin/date` > /Volumes/hdd1/keepalive.txt
*/2 * * * * echo `/bin/date` > /Volumes/hdd2/keepalive.txt

Also, ensure plot files aren't fragmented and large enough stagger is used so they're read sequentially. This way tear should be minimal.

For SSDs the above doesn't matter much, reading during mining shouldn't put much stress on them.

p.s. Regarding storj: most ppl I talked to don't like the idea of storing files on their hdds (even encrypted ones). They're afraid of illegal content potentially being stored. Burst doesn't store any real data and many view this as an advantage.

I agree about the storage part,  I don't think burst needs to worry about becoming decentralized storage, the issue you mentioned as well as bandwidth come into play, also, not sure how they would work it with so many people having different connections and speeds. but yea... BURST doesn't need to be storage, it's the mining that's the innovation here.

it COULD however, be a decentralized marketplace, with escrow. That could already happen, and would be a great plan. IMO

Hold on, could we not enable "forging" again...but with a twist. Here's the idea.

You can upload photos/videos for the marketplace
.whenever an account is "forging" it is downloading photos, not all of them, but a few of them at least. The fee involved with a photo/video is calculated based on its size, and therefore hosting the photo is profitable, because part of the fee goes to you, the host/"forger". This way it is optional, but useful. Also, the photo only needsnto be in a couple of places, as the owner of the photo is responsible for making sure its always safe on his wallet.

[bathrobehero]

I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

[Irontiga]

I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

Use a vm with linux.

OR

Mine on dev pool with the java miner....doesn't use the ram up like that

[Irontiga]

I am wondering if mining on a 5400 rpm HDD has the same performance as a 7200 rpm hdd.If their capacity is the same.

You're wondering loudly. lol.

I honestly don't know since I haven't run anything but 7200 RPMs, but I don't believe there should be that much of a difference. Maybe someone with a little more knowledge on the underside of how burst actulally mines could tell you.

But I would say it's definitely fine to mine on them, and it won't be that much of a difference.

Both mine just fine...just the plots will be read a little faster on the 7200rpm drive. If you did the thing with splitting up scoops then there would be an advantage for faster drives, but as it is with only reading 1/4096 scoops per plot it doesn't matter. Even USB 2.0 drives work fine....at least pinballdude says it does...dunno if he's using like 4tb, cause that would cause problems i'm guessing.

[bathrobehero]

I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

Use a vm with linux.

OR

Mine on dev pool with the java miner....doesn't use the ram up like that

Just tried pocminer_pool_v1 and memory usage went from 8% to 99% plus some pagefile in less than minute.
There has to be some memory management config to prevent this via the registry because  it's not the actual miner using that amount of memory but some kind of read caching which uses the memory 10-20 minutes after the miner has been closed and even when the plots are relocated.

I'd rather not use linux - for now.

[burstcoin]

I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

I'm surprised that is a problem with 8GB. I have a machine on Windows 7 with 4GB of ram running 2 instances of the java miner, one for each of two 2TB drives all plotted with stagger 8191 by another machine. Disabling prefetch was vital, and the system is unusably slow, but it's been mining stable for weeks.

Disabling prefetch/superfetch is the registry tweak.

[Irontiga]

I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

Use a vm with linux.

OR

Mine on dev pool with the java miner....doesn't use the ram up like that

Just tried pocminer_pool_v1 and memory usage went from 8% to 99 plus some pagefile in less than minute.
There has to be some memory management config to prevent this via the registry beucase not the actual miner is using that amount of memory but some kind of read caching which uses the memory 10-20 minutes after the miner has been closed and even when the plots are relocated.

I'd rather not use linux - for now.

Windows vm in windows? You can limit resources like that...

[Irontiga]

I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

I'm surprised that is a problem with 8GB. I have a machine with 4GB of ram running 2 instances of the java miner, one for each of two 2TB drives all plotted with stagger 8191 by another machine. Disabling prefetch was vital, and the system is unusably slow, but it's been mining stable for weeks.

You may want to update the OP about the bmpool asset....

[DrTrouble]

Just throwing this out here, but has anyone tried storage with de-duplication to get more plots on disk?  Like this free up to 1TB DXiV1000 virtual appliance VM - http://www.quantum.com/products/disk-basedbackup/dxiv1000/index.aspx .  I turned one on at work as a test for our backup system and it de-duped about 1TB of disk (12VMs) down to 50GB (40:1) compression.  

I am guessing that the way the plots are created this is not possible but I would like to toss this out there...

[bathrobehero]

I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

I'm surprised that is a problem with 8GB. I have a machine on Windows 7 with 4GB of ram running 2 instances of the java miner, one for each of two 2TB drives all plotted with stagger 8191 by another machine. Disabling prefetch was vital, and the system is unusably slow, but it's been mining stable for weeks.

Disabling prefetch/superfetch is the registry tweak.

Interesting. I disabled the pagefile completely and now when it climbs up to 100% memory usage it trashes itself back to ~13% at which point the plot reading is at ~58%. When that happes it stops completely along with disk reading until the next block. So with insufficient memory it doesn't seem to read everything. This is the case with both C miners and the Java miner crashes with error reading file.

[Irontiga]

I offer 1k BURST to someone who can tell me how can I (pool)mine with 5.5 TB without running out of 8 GB memory in a few minutes. I used all kinds of stagger sizes from 1024 to 16k, plot sizes from 2 GB to 5.5 TB using bipben's gpu plotter and tried uray's burst-miner and blago & dcct's miner. Using win7 and tried disabling prefetch/superfetch and various windows services without success.

I'm surprised that is a problem with 8GB. I have a machine on Windows 7 with 4GB of ram running 2 instances of the java miner, one for each of two 2TB drives all plotted with stagger 8191 by another machine. Disabling prefetch was vital, and the system is unusably slow, but it's been mining stable for weeks.

Disabling prefetch/superfetch is the registry tweak.

Interesting. I disabled the pagefile completely and now when it climbs up to 100% memory usage it trashes itself back to ~13% at which point the plot reading is at ~58%. When that happes it stops completely along with disk reading until the next block. So with insufficient memory it doesn't seem to read everything. This is the case with both C miners and the Java miner crashes with error reading file.

Strange. It should read everything....what is your stagger atm?