Bitcoin Forum
March 29, 2024, 07:07:49 AM *
News: Latest Bitcoin Core release: 26.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: Bitcoin addon: Distributed block chain storage  (Read 3332 times)
lucif (OP)
Sr. Member
****
Offline Offline

Activity: 462
Merit: 250


Clown prophet


View Profile
May 07, 2013, 09:59:53 AM
 #1

Hi. I am brain center of DIANNA Project Smiley Actually my project had stuck on future scalability problem. DIANNA needs easily accessible Bitcoin block chain indexed by block hash and transaction hash.

This is very big problem for local computers - its a [giga|tera|peta]bytes of data being constantly read.

I know Bitcoin community also have worries about Bitcoin database size and its load in future.

I think we may solve this problem together once and forever.

What do I need is to design and implement distributed, redunant, DHT-like (HASH=[block/transaction body]) storage for Bitcoin. Each network participant will store some piece of block chain and respond for inbound queries.

As benefit for most network participants - there will be no need to store 100% of data locally and read 100% of data locally. Each will serve his own piece of chain. And as network will grow, storage size/read capacity will also grow.

I think it is okay to put blocks bodies to DHT-like storage a leave only block headers locally as trusted chain skeleton.

I have no expirence with C and DHT. Could someone participate with this? I can sponsor this task. And if community need this feature, other sponsors welcome.
1711696069
Hero Member
*
Offline Offline

Posts: 1711696069

View Profile Personal Message (Offline)

Ignore
1711696069
Reply with quote  #2

1711696069
Report to moderator
1711696069
Hero Member
*
Offline Offline

Posts: 1711696069

View Profile Personal Message (Offline)

Ignore
1711696069
Reply with quote  #2

1711696069
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1711696069
Hero Member
*
Offline Offline

Posts: 1711696069

View Profile Personal Message (Offline)

Ignore
1711696069
Reply with quote  #2

1711696069
Report to moderator
1711696069
Hero Member
*
Offline Offline

Posts: 1711696069

View Profile Personal Message (Offline)

Ignore
1711696069
Reply with quote  #2

1711696069
Report to moderator
1711696069
Hero Member
*
Offline Offline

Posts: 1711696069

View Profile Personal Message (Offline)

Ignore
1711696069
Reply with quote  #2

1711696069
Report to moderator
BenTuras
Hero Member
*****
Offline Offline

Activity: 826
Merit: 1001



View Profile
May 07, 2013, 10:16:00 AM
 #2

Sounds like a nice job for http://hadoop.apache.org/ ?

I am selling in stock OneStringMiner boards, based on the Bitfury chips. Have a look here: https://bitcointalk.org/index.php?topic=495536.0
lucif (OP)
Sr. Member
****
Offline Offline

Activity: 462
Merit: 250


Clown prophet


View Profile
May 07, 2013, 10:18:45 AM
 #3

No. This should be custom DHT implementation in untrusted network.

Each client should store some number of blocks from his local headers - not arbitrary data.
proff
Newbie
*
Offline Offline

Activity: 46
Merit: 0


View Profile
May 07, 2013, 10:26:45 AM
 #4

Why not adapt existing solutions for this? Plenty of distributed key-value storage software is freely available. Is there some special requirement in your case that is not satisfied?
lucif (OP)
Sr. Member
****
Offline Offline

Activity: 462
Merit: 250


Clown prophet


View Profile
May 07, 2013, 10:32:41 AM
 #5

Yes, I think it will need to adapt some existing DHT solution with custom modifications for redunancy and storage filter (it should store only some part of chain listed in local headers plus some small percent of arbitrary orphaned blocks/transactions)
Sukrim
Legendary
*
Offline Offline

Activity: 2618
Merit: 1006


View Profile
May 07, 2013, 01:13:53 PM
 #6

Isn't bloom filtering already enough for your project? If you need to trust third parties anyways, you could just do a spv client that accesses the chain with bloom filters if it needs specific transactions. This already is implemented in bitcoin-qt

https://www.coinlend.org <-- automated lending at various exchanges.
https://www.bitfinex.com <-- Trade BTC for other currencies and vice versa.
bluemeanie1
Sr. Member
****
Offline Offline

Activity: 280
Merit: 257


bluemeanie


View Profile WWW
June 11, 2013, 03:41:02 AM
 #7

Hi. I am brain center of DIANNA Project Smiley Actually my project had stuck on future scalability problem. DIANNA needs easily accessible Bitcoin block chain indexed by block hash and transaction hash.

This is very big problem for local computers - its a [giga|tera|peta]bytes of data being constantly read.

I know Bitcoin community also have worries about Bitcoin database size and its load in future.

I think we may solve this problem together once and forever.

What do I need is to design and implement distributed, redunant, DHT-like (HASH=[block/transaction body]) storage for Bitcoin. Each network participant will store some piece of block chain and respond for inbound queries.

As benefit for most network participants - there will be no need to store 100% of data locally and read 100% of data locally. Each will serve his own piece of chain. And as network will grow, storage size/read capacity will also grow.

I think it is okay to put blocks bodies to DHT-like storage a leave only block headers locally as trusted chain skeleton.

I have no expirence with C and DHT. Could someone participate with this? I can sponsor this task. And if community need this feature, other sponsors welcome.


It's probably possible to store blocks in a DHT with some kind of reasonable response time.  These structures are designed to store data of this capacity.

For many applications though, you need to keep a copy of the block in local storage so you can validate the chain, thus there is a good probability that the block is available at a given node.  From here you need to determine 1) how to locate the block efficiently 2) which node is appropriate to poll- for instance balancing the load on all nodes that have a given block is optimal.  

It's certainly possible to optimize the basic DHT algorithm, which is designed for general use.  You might also need to implement a broadcast function for the transactions.

Just who IS bluemeanie?    On NXTautoDAC and a Million Stolen NXT

feel like your voice isn't being heard? PM me.   |   stole 1M NXT?
lucif (OP)
Sr. Member
****
Offline Offline

Activity: 462
Merit: 250


Clown prophet


View Profile
June 11, 2013, 05:31:37 AM
 #8

For many applications though, you need to keep a copy of the block in local storage so you can validate the chain
For initial validation - yes, client have to download each block via DHT and build chain headers. But it really don't need to store each block body. Only headers as trusted chain. Block bodies theirselfs can be distributed in untrusted DHT storage, as each client has local chain headers and modified block can not be accepted (as its hash will change).

Can anyone start implementing this? I can be a first donator of this task.
bluemeanie1
Sr. Member
****
Offline Offline

Activity: 280
Merit: 257


bluemeanie


View Profile WWW
June 11, 2013, 03:41:57 PM
 #9

For many applications though, you need to keep a copy of the block in local storage so you can validate the chain
For initial validation - yes, client have to download each block via DHT and build chain headers. But it really don't need to store each block body. Only headers as trusted chain. Block bodies theirselfs can be distributed in untrusted DHT storage, as each client has local chain headers and modified block can not be accepted (as its hash will change).

Can anyone start implementing this? I can be a first donator of this task.

I've spoken to a few DHT experts on this, most of them are optimistic about this idea.  The problem is basically how to optimize the Distributed Hash Table algorithm for block storage.  You can certainly optimize the algorithm in this case.  BTW- is there any good documentation on how the current relay mechanism works for Bitcoin?  I had several people ask me for this and I couldn't find anything.

btw- I notice many on this board seem to think that 'developers' are some kind of cheap resource that you could just conjure up by throwing a few dollars around.  While there might be many people able and willing to write a few lines of code, these kind of problems are very complex and there are really very few people who are capable of solving them effectively.  I've already seen several projects releasing code that appears to perform a(much desired) function, but fails in very important ways.  Of course these problems wont show until long after it's released and people have invested real money in the system.

Just who IS bluemeanie?    On NXTautoDAC and a Million Stolen NXT

feel like your voice isn't being heard? PM me.   |   stole 1M NXT?
Sukrim
Legendary
*
Offline Offline

Activity: 2618
Merit: 1006


View Profile
June 11, 2013, 04:42:31 PM
 #10

I also don't really see the point in this...
If you're after a full block chain, you are probably still for very long much better off just downloading a torrent or so.
If you want to have a lite client instead, use ultraprune and/or the SPV+ mode that is in the works.
If you just want to have access to older transactions, use bloom filters.

Also it seems the OP does NOT want to distribute block storage, but rather individual transactions indexed by hash (even if they have been pruned away). To make this somehow secure I guess one needs: TX hash, remaining merkle branch + block hash. Then one can see that this transaction was actually in the block that it claims. Still I am not too sure if a local database or a trusted local key-value store (noSQL) would not be more suited to the task.

https://www.coinlend.org <-- automated lending at various exchanges.
https://www.bitfinex.com <-- Trade BTC for other currencies and vice versa.
bluemeanie1
Sr. Member
****
Offline Offline

Activity: 280
Merit: 257


bluemeanie


View Profile WWW
June 11, 2013, 05:33:53 PM
 #11

In the Confidence Chains system you have MANY block chains, and the users might download several of them for various purposes(if they want to trade an asset or participate in a market or auction).  Thus there must be a useful and robust way to retrieve this data over a P2P network.  Only the identities need to hear about transactions, and you can do this with most DHT implementations.  There is an important question for this system: how do you bootstrap a block chain over a p2p network.  Bitcoin solves this in a very specific way, I need a more generalized solution.

With Bitcoin, it's unclear as to how to best optimize the p2p network.  For me lack of docs on this subject might indicate there are some hidden exploits.

Just who IS bluemeanie?    On NXTautoDAC and a Million Stolen NXT

feel like your voice isn't being heard? PM me.   |   stole 1M NXT?
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1006



View Profile
June 11, 2013, 05:37:32 PM
 #12

The problem is basically how to optimize the Distributed Hash Table algorithm for block storage.
Already solved by the Freenet project:

https://freenetproject.org
bluemeanie1
Sr. Member
****
Offline Offline

Activity: 280
Merit: 257


bluemeanie


View Profile WWW
June 11, 2013, 05:40:22 PM
 #13

The problem is basically how to optimize the Distributed Hash Table algorithm for block storage.
Already solved by the Freenet project:

https://freenetproject.org

excuse me?

Just who IS bluemeanie?    On NXTautoDAC and a Million Stolen NXT

feel like your voice isn't being heard? PM me.   |   stole 1M NXT?
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1006



View Profile
June 11, 2013, 05:54:08 PM
 #14

excuse me?
Freenet's data store implements a distributed, redundant, content-addressed file system where information remains persistent while storages nodes randomly enter and leave the network, without requiring users to explicitly configure replication because the nodes automatically handle that.

Get rid of the unnecessary (and slow) anonymity layer and Freenet is perfect for storing large content-addressed data sets like the Bitcoin blockchain.
bluemeanie1
Sr. Member
****
Offline Offline

Activity: 280
Merit: 257


bluemeanie


View Profile WWW
June 11, 2013, 06:48:02 PM
 #15

While Freenet can do that, it's not necessarily PERFECT for the job.

Bitcoin is a special case, because the data is highly redundant(practically every node has every block, or most of the blocks).  You can optimize the DHT algo to take advantage of this.  Freenet is one of many technologies in this class.  Some DHTs concentrate on security, and in Bitcoin there is no need for transport security as data at that level is public(theres nothing in a Block anyone wants to hide).

Transaction relays are a different story I think.

Just who IS bluemeanie?    On NXTautoDAC and a Million Stolen NXT

feel like your voice isn't being heard? PM me.   |   stole 1M NXT?
lucif (OP)
Sr. Member
****
Offline Offline

Activity: 462
Merit: 250


Clown prophet


View Profile
June 11, 2013, 07:19:46 PM
 #16

I also don't really see the point in this...
If you're after a full block chain, you are probably still for very long much better off just downloading a torrent or so.
If you want to have a lite client instead, use ultraprune and/or the SPV+ mode that is in the works.
If you just want to have access to older transactions, use bloom filters.
If Bitcoin doesn't need this... Alternative chains, or even contracts (trading via chains) will really need indexed access to both bitcoin blocks and transactions.

So they will require somewhat of additional bitcoin client + database somewhere around. It is overkill.

May be it is possible to make an addon to bitcoin, which will optionally turn bitcoin client into DHT participant along with regular local 100% database.
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1006



View Profile
June 11, 2013, 07:40:58 PM
 #17

Bitcoin is a special case, because the data is highly redundant(practically every node has every block, or most of the blocks).  You can optimize the DHT algo to take advantage of this.
There's no point implementing a new distributed data store unless you're also going to eliminate the need for every node to keep a complete copy of the blockchain.
bluemeanie1
Sr. Member
****
Offline Offline

Activity: 280
Merit: 257


bluemeanie


View Profile WWW
June 11, 2013, 09:15:33 PM
 #18

Bitcoin is a special case, because the data is highly redundant(practically every node has every block, or most of the blocks).  You can optimize the DHT algo to take advantage of this.
There's no point implementing a new distributed data store unless you're also going to eliminate the need for every node to keep a complete copy of the blockchain.

the nodes still need the blocks to VALIDATE the transactions.  If we're talking about SPV, then they don't really need the blocks at all, just the headers.

Just who IS bluemeanie?    On NXTautoDAC and a Million Stolen NXT

feel like your voice isn't being heard? PM me.   |   stole 1M NXT?
Sukrim
Legendary
*
Offline Offline

Activity: 2618
Merit: 1006


View Profile
June 13, 2013, 09:32:54 AM
 #19

You don't need all transactions to validate new transactions, only the UTXO set, which will be the work of SPV+ clients (see the "ulimate blockchain compression" topic).

https://www.coinlend.org <-- automated lending at various exchanges.
https://www.bitfinex.com <-- Trade BTC for other currencies and vice versa.
RoadTrain
Legendary
*
Offline Offline

Activity: 1386
Merit: 1009


View Profile
June 16, 2013, 04:15:20 AM
 #20

As far as I understand DIANNA there's already a solution.
DIANNA client must include sort of lite Bitcoin client. It will store the block headers to verify the blockchain.
While Bloom filter can facilitate the process of verifying DIANNA transactions.

As the DIANNA transaction is linked to a specific Bitcoin transaction, we can simply request this transaction from Bitcoin network using Bloom filter.
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!