Blockchains Really Need a Better Database...So We Built One

[charleshoskinson]

Enjoy https://github.com/input-output-hk/iodb/blob/master/doc/auth_dict.md

[eternalgloom]

Why is this a thing? Care to explain, in layman's terms, for us non technical users?
Is it just that it is more efficient to do it this way? And what are the chances of this being used over the current database?

[charleshoskinson]

It's an experiment using new authenticated data structures to vastly improve the throughput and efficiency of IO for cryptocurrencies. As consensus algorithms become more efficient, there will eventually be a bottleneck from the database side of things. So let's make sure that's not the limiting factor.

[2112]

It's an experiment using new authenticated data structures to vastly improve the throughput and efficiency of IO for cryptocurrencies. As consensus algorithms become more efficient, there will eventually be a bottleneck from the database side of things. So let's make sure that's not the limiting factor.

How about a decent writeup for the technical and other knowledgeable people?

I took a quick look and it looks like some sort of confused mishmash:

1) authenticated data structures for local, private database? Why? The left hand doesn't trust the right hand? Multiple personality disorder? Just sensible error detection scheme should be enough to verify the database integrity.

2) it seems to have some optimizations for flash backing storage, but does it through a file-systems with unspecified limitations. A lot of flash-optimized software takes advantage of the fact that one way of writing (e.g. from zero to one) is very cheap whereas the opposite change (from one to zero) requires time-consuming block erase. Are you doing that? Explicitly naming the constraints would be very helpful.

3) Lost interest in further reading after the two above drawbacks.

There's an user here "jl777" (https://bitcointalk.org/index.php?action=profile;u=177323;sa=showPosts) and on other forums, that has a competing idea of database explicitly oriented for cryptocurrencies. He calls it "Iguana," but the problem is that he works alone, produces extremely hard to read code, and is in general hard to communicate with. Therefore he mostly gets ignored, both here and elsewhere, despite showing very promising results.

Are you similarly sponsoring financially a lone genius coder?

How about producing a couple of pages of introductory whitepaper, written at the level of B.Sc. or similar, that would explain why your mousetrap is better?

[khalized]

Enjoy https://github.com/input-output-hk/iodb/blob/master/doc/auth_dict.md

Basically you can try it out yourself by creating a crypto solely for its purpose.
For better implementation you can use side-chain technology and use bitcoins database as its main chain. So that during any updates it will not disrupt your own chain.

Let me know if you come out with something beneficial. I would like to get into it.

[jbreher]

A lot of flash-optimized software takes advantage of the fact that one way of writing (e.g. from zero to one) is very cheap whereas the opposite change (from one to zero) requires time-consuming block erase.

While this is true at a storage cell level, contemporary system architectures employ such NAND Flash not as raw devices, but within subsystems known as SSDs. SSDs maintain the same semantics at the interface as hard disk drives. These are block-access devices, where some larger construct - the logical block (typically 4096 or 512 bytes of user data) are read or written at one time (i.e., as an atomic unit). Such SSDs already manage all the writing internally, so as to make write behavior indistinguishable regardless of data pattern.

Some NAND and NOR Flash is sometimes included within a system, to be accessed directly via the pins of the chip package. These indeed can benefit from optimized flash file systems. However, these are typically small in size, and meant to save boot Firmware, BIOS settings and other non-user data items related to the management of the computer system itself. I am unaware of any typical commercially available system that employs such raw flash for general data storage suitable for the filesystem upon which one would store a blockchain database.

TL;DR, In today's environment, there is no benefit to application SW (e.g. a database system) managing writes to storage in a manner aware of bit polarity in relation to previous writes to same location. This, and other complications, are abstracted away by mechanisms within the underlying storage devices.

[NyeFe]

Awesome I love the extensive documentation. I'll definitely give it a good read.

[2112]

While this is true at a storage cell level, contemporary system architectures employ such NAND Flash not as raw devices, but within subsystems known as SSDs. SSDs maintain the same semantics at the interface as hard disk drives. These are block-access devices, where some larger construct - the logical block (typically 4096 or 512 bytes of user data) are read or written at one time (i.e., as an atomic unit). Such SSDs already manage all the writing internally, so as to make write behavior indistinguishable regardless of data pattern.

Some NAND and NOR Flash is sometimes included within a system, to be accessed directly via the pins of the chip package. These indeed can benefit from optimized flash file systems. However, these are typically small in size, and meant to save boot Firmware, BIOS settings and other non-user data items related to the management of the computer system itself. I am unaware of any typical commercially available system that employs such raw flash for general data storage suitable for the filesystem upon which one would store a blockchain database.

TL;DR, In today's environment, there is no benefit to application SW (e.g. a database system) managing writes to storage in a manner aware of bit polarity in relation to previous writes to same location. This, and other complications, are abstracted away by mechanisms within the underlying storage devices.

Uh, oh!

Apparently the SSD technology (Solid-State Drive) is still affected by the SSDD phenomenon (Same Shit, Different Day). I don't think you consciously bullshit in your post, you don't seem like a SSD salesman. You seem to be unaware of the fact that SSD devices are affected by almost exactly the same phenomenon: the write block size is smaller than the erase block size. I'm not going to get into details, just give the link to Wikipedia's writeup about https://en.wikipedia.org/wiki/Write_amplification .

I measured write amplification with some older release of Bitcoin on a range of storage devices. It was as high as 128. It doesn't take a rocket scientist to write a backend storage for Bitcoin with lower write amplification.

As far as commercial products the good way to start with open source is to research Linux's https://en.wikipedia.org/wiki/Memory_Technology_Device . On the closed source front many products that ostensibly support only FAT and exFAT filesystems have appropriate optimizations in the code path that handles preallocated files of constant size.

The reason I'm writing all this is to see if there's any intelligent life left in Charles' Hoskinson's organization. I clearly can see that you are intelligent, if somewhat not-up-to-date. The question is still open about his org: it seems that it is just a marketing organization with absolutely no technical background. Lets see if they can post any intelligent followup.

[jbreher]

TL;DR, In today's environment, there is no benefit to application SW (e.g. a database system) managing writes to storage in a manner aware of bit polarity in relation to previous writes to same location. This, and other complications, are abstracted away by mechanisms within the underlying storage devices.

You seem to be unaware of the fact that SSD devices are affected by almost exactly the same phenomenon: the write block size is smaller than the erase block size.

While it is true that current NAND Flash based SSDs do indeed have the same fundamental issue you describe*, that complexity is completely abstracted from the user interface. There are no user accessible 'knobs or switches' which allow any layer above the native SCSI, ATA, or NVMe interface to dictate the physical placement of any logical block of data***.

* (incidentally, the issue you initially implied - erased flash is in a '1' state, while it takes a write to flip a bit to '0', thereby one might ensure subsequent writes to a given block only flip '1s' to '0s', is completely orthogonal to the (substitute) issue that NAND is written in pages, and multiple pages make up an erase block)

It doesn't take a rocket scientist to write a backend storage for Bitcoin with lower write amplification.

Sure, you can coalesce writes. You can do large writes. You can do sequential writes. These are things that work on HDD too, and have been known for roughly a half-century. But you cannot reach into the drive through the interface and dictate physical placement. At least not with any volume-shipping SSD.

The reason I'm writing all this is to see if there's any intelligent life left in Charles' Hoskinson's organization. I clearly can see that you are intelligent, if somewhat not-up-to-date.

No, I assure you that I am completely up to date - both with respect to shipping product, and well into things on the horizon for a good portion of the storage industry.

When a logical block is written, there is no way to dictate that it be written to a previously-written page that would require only 1>0 changes, due to the 0-bitmask of the new data being a proper superset of the 0-bitmask of the old data. Other than devices that do internal data reduction**, each written logical block is written to a physical block (part of a page) that has never been written since the last time that erase block has been erased.

**As for the case of controllers that perform data reduction, such a write ends up being a nop, as the data in the logical block already exists in entirety (in logical form) elsewhere in storage. The device merely creates a pointer to the existing data. Nevertheless, such controllers do not expose any interface by which some layer above the device might affect this behavior.

***Command standards have recently added Logical Block Markup Descriptors -- hints that certain logical blocks 'belong together' -- but these are hints, not directives, and are not yet supported by IO stacks in any OS likely to be employed as a Bitcoin node.

While I am not an SSD salesman, nor part of any Charles Hoskinson organization, I am a senior technologist in the research division of the largest developer of storage devices on the planet. Will you be at Flash Memory Summit week after next? I'd like to buy you a beer or coffee for more in-depth discussion.

[2112]

While I am not an SSD salesman, nor part of any Charles Hoskinson organization, I am a senior technologist in the research division of the largest developer of storage devices on the planet. Will you be at Flash Memory Summit week after next? I'd like to buy you a beer or coffee for more in-depth discussion.

I want to sincerely thank you for the offer. Unfortunately for me I'm preoccupied with relocation and it is rather unlikely that I would be able to go to Santa Clara,CA,USA on the short notice.

It is too bad that as a senior technologist in storage system you are not allowed to discuss openly the write amplification issue that is affecting the modern file systems and DBMS a great deal. I do understand that those discussions could really be conducted under NDA or while maintaining an anonymity cover like Satoshi's Nakamoto's.

[Kprawn]

I should think this thread must be under this section, because it's still under development? --> https://bitcointalk.org/index.php?board=12.0 ... I do not get the whole idea about this, so please explain

it in non-technical terms. You wrote a more scalable and efficient database to be used with the Blockchain? Is this not what the banks are doing with their Blockchain-based permissioned ledger at the

moment? Good luck with the project, and hopefully we will see a open system, and not a permissioned version. 

[jbreher]

While I am not an SSD salesman, nor part of any Charles Hoskinson organization, I am a senior technologist in the research division of the largest developer of storage devices on the planet. Will you be at Flash Memory Summit week after next? I'd like to buy you a beer or coffee for more in-depth discussion.

I want to sincerely thank you for the offer. Unfortunately for me I'm preoccupied with relocation and it is rather unlikely that I would be able to go to Santa Clara,CA,USA on the short notice.

Understood. While it has been some time since I've been to a major Bitcoin symposium, who knows when our paths may cross. I'll leave the offer open.

It is too bad that as a senior technologist in storage system you are not allowed to discuss openly the write amplification issue that is affecting the modern file systems and DBMS a great deal.

Well, I can discuss write amp in generalities. But given that the command layer protocols abstract away the low-level physical mapping from upper layers, there is not much to be done by system software. As stated above, the techniques available above the device interface consist of the same things that have been known to benefit HDDs as well for quite some time.

Interestingly, HDDs are becoming more complex internally as well, with essentially the equivalent of SSD's FTL (Flash Translation Layer) handling fully indirected logical to physical block mapping. This is the case for several HDDs today, and this will be more prevalent as time goes on.

[BADecker]

Here comes the mandatory Bitcoin database

In the wake of multiple terror attacks, the European Commission is moving towards creating a mandatory, centralized database of Bitcoin ownership.

Of course, their official reason is that Bitcoin is being used to finance terrorism.

So for everyone's safety and security they need even more authority to spy on people's finances.

But the reality is that these governments have hated Bitcoin since the beginning.

They don't like the idea of a decentralized currency that they can't control.

In the conventional financial system, governments have the power to regulate the banks and seize any account they want.

Read more at https://www.sovereignman.com/trends/here-comes-the-mandatory-bitcoin-database-20049/.