MixMAx123 (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 29, 2021, 11:07:37 PM |
|
I want to create a new database for the blockchain. I would like to create a file for each block. eg: 000000000019D6689C085AE165831E934F763AE46A2A6C172B3F1B60A8CE26F.Block That would be more than 700,000 files.
NTFS can be 4.294.967.295 files in a folder. So theoretically it should go. Question: Has anyone ever tested such a thing? Would the file system crashed or something similar?
My idea is through the file name of the blockhash is to find and open the blocks very quickly. Ideas and suggestions are welcome :-)
|
|
|
|
odolvlobo
Legendary
Offline
Activity: 4438
Merit: 3341
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 03:12:16 AM |
|
Do you really think that your database is better? There is more to a database than how to organize the files.
|
Join an anti-signature campaign: Click ignore on the members of signature campaigns. PGP Fingerprint: 6B6BC26599EC24EF7E29A405EAF050539D0B2925 Signing address: 13GAVJo8YaAuenj6keiEykwxWUZ7jMoSLt
|
|
|
pooya87
Legendary
Offline
Activity: 3570
Merit: 10849
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 03:46:38 AM |
|
That's extremely inefficient. You will end up with a lot of files that are so tiny (176-300 bytes) and opening each file and reading it is still consuming the biggest percentage of the total time. My idea is through the file name of the blockhash is to find and open the blocks very quickly.
The current way of storing multiple blocks in a file and storing an index for their location is fast enough. Why do you need more speed?
|
|
|
|
HCP
Legendary
Offline
Activity: 2086
Merit: 4318
<insert witty quote here>
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 03:59:16 AM |
|
Or to ask it in a slightly different way... what do you believe you will gain from this setup as opposed to the current setup?
|
|
|
|
MixMAx123 (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 07:14:17 AM |
|
The current way of storing multiple blocks in a file and storing an index for their location is fast enough. Why do you need more speed?
I need a faster database for my block Explorer. All TX to get an address takes too long. This address, e.g. does not work anymore: 12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX See here: https://bitcoinexplorer.org/address/12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX Currently, it is so that there are minimal 2 hard disk access necessary to get a TX. The first for the TX index in the Leveldb and the second then in the BLK.DAT I am looking for a way with just a hard disk access. Maybe I can manage to load the indexing in the RAM or to work with bloom filter.
|
|
|
|
baro77
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 07:26:05 AM |
|
Hi, some unleashed thoughts Choosing a DB is always a threshold choice, depending on what you want to achieve in terms of, for example, storage occupation, query speed (which by itself depends on query type you have in mind), resilience to adverse events (e.g. power leakage causing DB shutting down abruptly in the the middle of a transaction (=set of operation to be executed or not atomically, aka as just a unique comprehensive one), and so on... Given that, it's uncommon to begin thinking about it from beneath file system, or chunks size. A common way of thinking is that a DB is good if in some way it intrinsically maps the "shape" of the data is going to hold, without needing brand new "structures" (whichever it could mean) between the way the DB operates and the contained data. The heuristic at the base of this reasoning is that the shape of the data determines which operations we could or could not do effectively and efficiently on it, so choosing a DB mirroring that shape is the best you can do without further constraints on the interesting operations. That said an interesting world to look at is the Graph Databases ones. As far as I know the reference here is NEO4J; it also has a free community edition: https://neo4j.com/download-center/#community and Greg Walker made a PoC for Bitcoin blockchain: https://neo4j.com/blog/import-bitcoin-blockchain-neo4j/ and https://learnmeabitcoin.com/neo4j/My 2 cents ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) baro
|
|
|
|
MixMAx123 (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 07:48:44 AM Last edit: October 30, 2021, 08:03:42 AM by MixMAx123 |
|
Choosing a DB is always a threshold choice, depending on what you want to achieve in terms of, for example, storage occupation, query speed (which by itself depends on query type you have in mind), resilience to adverse events (e.g. power leakage causing DB shutting down abruptly in the the middle of a transaction (=set of operation to be executed or not atomically, aka as just a unique comprehensive one), and so on... Given that, it's uncommon to begin thinking about it from beneath file system, or chunks size.
There are two requirements for the database that are very hard. 1. The blockchain is> 400GB 2. I have to read more than 2000 transactions from the blockchain for displaying an address in the block explorer. I am looking for an efficient approach to this problem. I do not know any database that focuses this specific problem. As they have already described correctly. Most databases are made for applications that are often occurring and not specialized. I want to program the database itself, which makes it possible, more or less. It's worth a try.
|
|
|
|
|
MixMAx123 (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 11:08:16 AM Last edit: October 30, 2021, 11:21:06 AM by MixMAx123 |
|
That address is working "fine" on other block explorers:
"Our monetary system works "fine" for other banks." Did you noticed that the balance on which you have linked block explorers is different? ... "working fine" ;-) Of course it works somehow, that was not the question here. I can, if I have the resources, also load the whole blockchain into the RAM, then it works great fast. .... --txid-limit it does not work.
|
|
|
|
ABCbits
Legendary
Offline
Activity: 2996
Merit: 7864
Crypto Swap Exchange
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 11:43:15 AM |
|
That's extremely inefficient. You will end up with a lot of files that are so tiny (176-300 bytes) and opening each file and reading it is still consuming the biggest percentage of the total time.
Don't forget the bloat due to filesystem block size (these days most filesystem have 4KB as default block size). I want to program the database itself, which makes it possible, more or less. It's worth a try.
Consider taking course on Database. Or just run Electrum server locally and disable the rate limit.
|
|
|
|
pooya87
Legendary
Offline
Activity: 3570
Merit: 10849
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 11:55:56 AM |
|
Did you noticed that the balance on which you have linked block explorers is different?
Just because some dumb block explorers have bugs doesn't change anything about what HCP said. The bug in this case is that because these explorers were too lazy to code "search by script" and there is no address associated by P2PK, they decided that it is fine to convert a P2PK script to an entirely different script called P2PKH and show that address. Which is why they are counting 50 BTC more for this address whereas the 50 BTC output is actually associated with the P2PK script not the P2PKH (the address you posted) script.
|
|
|
|
aliashraf
Legendary
Offline
Activity: 1456
Merit: 1174
Always remember the cause!
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 12:23:38 PM |
|
Using an alternative approach to persistency of blocks in a blockchain is not a bad idea per se, but OP's proposal, using FS for this purpose is the worst idea ever as most folks have mentioned correctly above, still it remains a good topic to discuss as long as we forget about FS thing and look for better alternatives, IMO.
I'd suggest a full-fledged RDBMS, obviously.
|
|
|
|
MixMAx123 (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 02:57:23 PM |
|
Ok, I think I will try to pack all RAW transactions directly in a Leveldb database. Has anyone tried that already?
|
|
|
|
Quickseller
Copper Member
Legendary
Offline
Activity: 2968
Merit: 2370
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 03:12:10 PM Last edit: October 30, 2021, 03:39:17 PM by Quickseller |
|
The current way of storing multiple blocks in a file and storing an index for their location is fast enough. Why do you need more speed?
I need a faster database for my block Explorer. All TX to get an address takes too long. This address, e.g. does not work anymore: 12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX See here: https://bitcoinexplorer.org/address/12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX Currently, it is so that there are minimal 2 hard disk access necessary to get a TX. The first for the TX index in the Leveldb and the second then in the BLK.DAT I am looking for a way with just a hard disk access. Maybe I can manage to load the indexing in the RAM or to work with bloom filter. Most likely you need to update your config file to reflect txindex=1 This is going to increase the amount of disk space required to run your node, and it will take longer to sync. When you are running any kind of database, you need to create an index on any column that you will potentially search by. Putting information in RAM, increasing CPU capacity, increasing RAM, changing the file structure etc will not fix your problem. Ok, I think I will try to pack all RAW transactions directly in a Leveldb database. Has anyone tried that already?
This will make your database inefficient. It will require multiple queries any time you want to look something up. Lebeldb is not a relational database, and IMO you need a relational database in order to run a block explorer.
|
|
|
|
DaveF
Legendary
Offline
Activity: 3598
Merit: 6573
Crypto Swap Exchange
|
Ok, I think I will try to pack all RAW transactions directly in a Leveldb database. Has anyone tried that already?
Not that leveldb is bad, but why not mariadb? There are major corporations running multi terabyte databases on it. A few hundred GB of blockchain data is not even going to make it stress. A 1 TB or 2 TB SSD and a server with 32 GB of ram and you should have no issues at all. You keep trying to do things the hard way when people have done them other better ways many times before. -Dave
|
|
|
|
aliashraf
Legendary
Offline
Activity: 1456
Merit: 1174
Always remember the cause!
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 30, 2021, 10:01:44 PM |
|
Ok, I think I will try to pack all RAW transactions directly in a Leveldb database. Has anyone tried that already?
Not that leveldb is bad, but why not mariadb? Leveldb is bad for the job, once it comes to explorer class of applications, because it lacks the querying power of an RDBMS such as MariaDB which you have already recommended thoughtfully, the latter is open source and guaranteed to remain open source inheriting the rich history and highly crafted source code of the legendary MySQL.
|
|
|
|
TheArchaeologist
|
Ok, I think I will try to pack all RAW transactions directly in a Leveldb database. Has anyone tried that already?
Hi, I have done this in a RDBMS (MariaDB). Although after a while I got rid of storing the RAW transactions myself, just made a couple of database tables that store the information I need (like block, transaction, address, etc). In the rare occasions I do need a raw transaction I just provide the txid to the rpc api from core (getrawtransaction). Just keep in mind that adding indexes will make it more efficient but it will also add to the storage space required. I got it setup on a 2TB sdd which is now approx. used for 50%. If I would have kept all the RAW transactions in the DB it would mean I would probably run into storage problems by now.
|
Sooner or later you're going to realize, just as I did, that there's a difference between knowing the path and walking the path
|
|
|
DaveF
Legendary
Offline
Activity: 3598
Merit: 6573
Crypto Swap Exchange
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
October 31, 2021, 11:18:43 AM |
|
Ok, I think I will try to pack all RAW transactions directly in a Leveldb database. Has anyone tried that already?
Hi, I have done this in a RDBMS (MariaDB). Although after a while I got rid of storing the RAW transactions myself, just made a couple of database tables that store the information I need (like block, transaction, address, etc). In the rare occasions I do need a raw transaction I just provide the txid to the rpc api from core (getrawtransaction). Just keep in mind that adding indexes will make it more efficient but it will also add to the storage space required. I got it setup on a 2TB sdd which is now approx. used for 50%. If I would have kept all the RAW transactions in the DB it would mean I would probably run into storage problems by now. You can bring in the raw transactions and just compress that one table: https://mariadb.com/kb/en/innodb-page-compression/I would *think* that it would still be much faster then using a rpc call to get the data. For single person using it, it probably would not matter. If it's exposed for the world to use then you need to have it in a db. -Dave
|
|
|
|
TheArchaeologist
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 02, 2021, 11:58:05 AM Merited by HCP (5), ABCbits (2) |
|
You can bring in the raw transactions and just compress that one table: https://mariadb.com/kb/en/innodb-page-compression/I would *think* that it would still be much faster then using a rpc call to get the data. For single person using it, it probably would not matter. If it's exposed for the world to use then you need to have it in a db. -Dave It all depends on the use-case you have. If for instance I would create a webservice and expose it to the world to return the raw transaction based on a txid in the request it would make very much sense to store all the transactions in a DB. But if I would like to be able to analyze/query the blockchain I wouldn't get very far only storing the rawtransactions in a DB. Raw transactions are optimized to contain as much data as needed in a minimum storage space. However good luck creating a query like "when was the last time adress X received over 0.1 BTC" based on raw transaction data only. There is a reason the UTXO's are stored in a DB by core itself. So what I meant (and what I did) I extracted all the info needed from the raw transactions and stored them in a several RDBMS tables. Added some indexes and this a perfect solution for my use-case. So in the end I had no need for the raw transactions themselves anymore. TLDR: Just storing raw transactions in a RDBMS table doesn't mean all the data within is easy accessible.
|
Sooner or later you're going to realize, just as I did, that there's a difference between knowing the path and walking the path
|
|
|
NotATether
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 07, 2021, 06:49:23 PM |
|
You should deserialize each transaction (this will take a long initial time), and place transaction data in one NoSQL table (minus vin and vout arrays), harvest the vin and vout data to make a utxo table, and take the addresses and values out of each vout to construct an addresses table.
And of course you can fetch each block, deserialize the data and create a blocks table while you're at it.
SQL databases are horribly inefficient. NoSQLs like MongoDB are better.
|
|
|
|
|