Bitcoin Forum
November 22, 2017, 01:47:09 PM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Alternative storage for the blockchain  (Read 2306 times)
Ikinoki
Full Member
***
Offline Offline

Activity: 128


View Profile
November 22, 2013, 11:59:50 PM
 #1

Hi everyone,

I know that leveldb is fine for embedded, but we need to think big. Soon or later the Nodes will also be a business.
So leveldb will start to slow down soon or later.

So is there a way to store blockchain in SQL (postgresql, mysql), No-SQL (mongo, orient), key-value storages (redis, memcached)?
It would be really nice to choose a medium, especially for big business.

Donations to 1LHTGFYHfMDgfgmDcYugW6RsKKfKBRfLVg
1511358429
Hero Member
*
Offline Offline

Posts: 1511358429

View Profile Personal Message (Offline)

Ignore
1511358429
Reply with quote  #2

1511358429
Report to moderator
1511358429
Hero Member
*
Offline Offline

Posts: 1511358429

View Profile Personal Message (Offline)

Ignore
1511358429
Reply with quote  #2

1511358429
Report to moderator
Join ICO Now Coinlancer is Disrupting the Freelance marketplace!
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1511358429
Hero Member
*
Offline Offline

Posts: 1511358429

View Profile Personal Message (Offline)

Ignore
1511358429
Reply with quote  #2

1511358429
Report to moderator
1511358429
Hero Member
*
Offline Offline

Posts: 1511358429

View Profile Personal Message (Offline)

Ignore
1511358429
Reply with quote  #2

1511358429
Report to moderator
1511358429
Hero Member
*
Offline Offline

Posts: 1511358429

View Profile Personal Message (Offline)

Ignore
1511358429
Reply with quote  #2

1511358429
Report to moderator
moderate
Member
**
Offline Offline

Activity: 70

nearly dead


View Profile
November 23, 2013, 12:41:48 AM
 #2


I know that leveldb ..

So is there a way to store blockchain in key-value storages ?


Have you tried finding out what leveldb is ?
Sukrim
Legendary
*
Offline Offline

Activity: 2198


View Profile
November 23, 2013, 01:36:21 AM
 #3

Do you know what levelDB is used for in bitcoind? Roll Eyes

Anyways, if you need a faster version of it, you can check facebook's pimped version: http://rocksdb.org/

https://bitfinex.com <-- leveraged trading of BTCUSD, LTCUSD and LTCBTC (long and short) - 10% discount on fees for the first 30 days with this refcode: x5K9YtL3Zb
Mail me at Bitmessage: BM-BbiHiVv5qh858ULsyRDtpRrG9WjXN3xf
gmaxwell
Moderator
Legendary
*
qt
Offline Offline

Activity: 2338



View Profile
November 23, 2013, 02:42:28 AM
 #4

Why don't you go do some testing and find out where it falls down?

Otherwise it's all just buzzwords: REDIS HAZ WEBSCALE!!!!

You might also try looking at the implementations which alreay have used postgresql...

Bitcoin will not be compromised
Ikinoki
Full Member
***
Offline Offline

Activity: 128


View Profile
November 23, 2013, 02:49:59 AM
 #5

If you want to create a service doing more than 100000 tps and addresses, leveldb WILL NOT WORK, it's written in the website of leveldb.

With redis one can achieve either a very fast distributed cache, or a total distributed ACID fast-enough solution.
With SQL you could keep the remainder with fast access for statistics and other operations.

You will want distributed for fault-tolerance as well.

What I want is to write a 100000 tps wallet web frontend with security (so leveldb is only local or through NFS but I'm not crazy in that way) separating DB from actual read and write nodes and different keys for all of them.
File-based key-value like leveldb won't help with this, though it's perfect for smaller deployments.

Donations to 1LHTGFYHfMDgfgmDcYugW6RsKKfKBRfLVg
Ikinoki
Full Member
***
Offline Offline

Activity: 128


View Profile
November 23, 2013, 02:52:11 AM
 #6

It's either this, or multiple bitcoind entities, which does not sound right...

Donations to 1LHTGFYHfMDgfgmDcYugW6RsKKfKBRfLVg
Sukrim
Legendary
*
Offline Offline

Activity: 2198


View Profile
November 23, 2013, 03:49:50 AM
 #7

Yeah, looks like MtGox tried the multiple bitcoind thingie - they started to get into race conditions and double spend each other from time to time.

https://bitfinex.com <-- leveraged trading of BTCUSD, LTCUSD and LTCBTC (long and short) - 10% discount on fees for the first 30 days with this refcode: x5K9YtL3Zb
Mail me at Bitmessage: BM-BbiHiVv5qh858ULsyRDtpRrG9WjXN3xf
Ikinoki
Full Member
***
Offline Offline

Activity: 128


View Profile
November 23, 2013, 04:17:59 AM
 #8

Sure, because you'll need to cross verify the txs, and it's not that easy with no direct leveldb access and only json capabilities of bitcoind, and I'm not sure it's possible to partly lock leveldb to make a transaction atomically.

Donations to 1LHTGFYHfMDgfgmDcYugW6RsKKfKBRfLVg
justusranvier
Legendary
*
Offline Offline

Activity: 1400



View Profile WWW
November 23, 2013, 04:46:29 AM
 #9

Both btcd and Bits of Proof are capable of swapping out block storage databases.
Ikinoki
Full Member
***
Offline Offline

Activity: 128


View Profile
November 26, 2013, 01:51:42 AM
 #10

That's fine, a shame though they are not in Python Sad

Donations to 1LHTGFYHfMDgfgmDcYugW6RsKKfKBRfLVg
wumpus
Hero Member
*****
qt
Offline Offline

Activity: 812

No Maps for These Territories


View Profile
November 26, 2013, 07:14:03 AM
 #11

It's the wallet implementation in bitcoind (which uses berkelydb) that has performance/scaling problems, not the block chain handling (which uses leveldb).

The 0.9 release will include "disable wallet" mode, and I expect in time that will be the recommended way of running your own full node. You can use any SPV-ish wallet client (or multiple!) on top.

Sure, because you'll need to cross verify the txs, and it's not that easy with no direct leveldb access and only json capabilities of bitcoind, and I'm not sure it's possible to partly lock leveldb to make a transaction atomically.
You may be able to do what you want through the raw transaction API (createrawtransaction/signrawtransaction/sendrawtransaction/getrawtransaction). There is no direct leveldb access but there are low-level queries in the JSON interface.

You can also lock specific outputs from being spent using lockunspent.

Bitcoin Core developer [PGP] Warning: For most, coin loss is a larger risk than coin theft. A disk can die any time. Regularly back up your wallet through FileBackup Wallet to an external storage or the (encrypted!) cloud. Use a separate offline wallet for storing larger amounts.
jdbtracker
Hero Member
*****
Offline Offline

Activity: 687


Minimum Effort/Maximum effect


View Profile
November 26, 2013, 08:47:44 AM
 #12

Hi everyone,

I know that leveldb is fine for embedded, but we need to think big. Soon or later the Nodes will also be a business.
So leveldb will start to slow down soon or later.

So is there a way to store blockchain in SQL (postgresql, mysql), No-SQL (mongo, orient), key-value storages (redis, memcached)?
It would be really nice to choose a medium, especially for big business.

I think SQLite is part of BerkleyDB, I went and looked at the performance between SQLite and LevelDB is pretty much the same... so what needs improvement?

If you think my efforts are worth something; I'll keep on keeping on.
I don't believe in IQ, only in Determination.
Ikinoki
Full Member
***
Offline Offline

Activity: 128


View Profile
November 26, 2013, 08:37:51 PM
 #13

SQLite is not part of berkeley db, they are even of different types (SQL obviously vs noSQL key-value store in file)

Is not get/set rawtransaction basically what sx can do? And newly announced protocoin? I might look into this. Sounds interesting.

Donations to 1LHTGFYHfMDgfgmDcYugW6RsKKfKBRfLVg
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!