Bitcoin Forum
November 18, 2024, 08:19:32 AM *
News: Latest Bitcoin Core release: 28.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 [2]  All
  Print  
Author Topic: New database for blockchain  (Read 442 times)
MixMAx123 (OP)
Full Member
***
Offline Offline

Activity: 161
Merit: 168


View Profile
November 07, 2021, 08:04:05 PM
 #21

My previous experience shows that the deserialization of transactions is the smallest problem and goes very fast.
Loading the transaction from the BLK files takes much longer, because min. 2 hard disk access is needed.
I'm trying to reduce this to a hard disk access.
PrimeNumber7
Copper Member
Legendary
*
Offline Offline

Activity: 1666
Merit: 1901

Amazon Prime Member #7


View Profile
November 08, 2021, 05:20:36 AM
 #22

You should deserialize each transaction (this will take a long initial time), and place transaction data in one NoSQL table (minus vin and vout arrays), harvest the vin and vout data to make a utxo table, and take the addresses and values out of each vout to construct an addresses table.

And of course you can fetch each block, deserialize the data and create a blocks table while you're at it.

SQL databases are horribly inefficient. NoSQLs like MongoDB are better.
You are describing a relational database structure (such as SQL) in a convoluted way.

A NoSQL collection of blocks might break down as follows:
record for block x --> transactions --> list of vIns, each of which references a previously confirmed transaction and index to that transaction and list of vOuts, each of which reference an address and amount.

If a particular vOut is spent, it would be very difficult to lookup that transaction in a NoSQL database.

If you are using a relational database (such as SQL), you can create an index on any column you are using a WHERE clause on (or are aggregating data on). So if you ever will filter by txid, you can create an index on the txid column in your transactions table. Querying a SQL database when filtering by data in a column that is indexed will be very fast, while querying when filtering by data in a column that is not indexed will be slow (or maybe very slow if your database has many records).

If you need to frequently access data, you can also create a temporary table. Bitcoin core already does this (or something very similar) by keeping the UTXO set in RAM.
Skybuck
Full Member
***
Offline Offline

Activity: 386
Merit: 111


View Profile
December 08, 2021, 03:46:21 AM
 #23

I want to create a new database for the blockchain.
I would like to create a file for each block.
eg: 000000000019D6689C085AE165831E934F763AE46A2A6C172B3F1B60A8CE26F.Block
That would be more than 700,000 files.

NTFS can be 4.294.967.295 files in a folder.
So theoretically it should go.
Question:
Has anyone ever tested such a thing?
Would the file system crashed or something similar?

My idea is through the file name of the blockhash is to find and open the blocks very quickly.
Ideas and suggestions are welcome :-)

I think you should give GIT a try, it's basically a key=value storage systems, does something similiar as to what you want, store content with a content hash.

Git does have some decent performance thanks to big linux kernel development approx 2 GB in size, ofcourse bitcoin much larger 150x larger.

I am not sure how git works internally, have not gotten that far yet with it's analysis... it does use a database stored in a .git folder.

GIT is open source I believe, so you could analyze the source code to see how it works.

GIT is also very similiar to BITCOIN... GIT also uses "blocks" which are called "commits" and it also calculates hashes, sha-1 I believe.

And these blocks/commits are also linked to each other in a similiar way.

Perhaps somebody involved with GIT was also involved in the creation of BITCOIN ! Wink
Pages: « 1 [2]  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!