Title: [Nxt] Implements Blockchain Pruning Post by: Daedelus on April 15, 2015, 09:50:13 AM Blockchain Pruning to be included with the next release of Nxt. This will drastically reduce blockchain bloat in future, compared to the blockchain so far ;D
Source: https://nxtforum.org/nrs-releases/nrs-v1-5-1e/ Release 1.5.1e This is a development release for testing only. Source code is not provided. Change log: This is an experimental release which adds the Prunable Messages feature. It will be enabled at the same block as the Voting and Phasing features. This is a required update for all testnet nodes, but is also possible to run on main net. New features: Prunable plain and encrypted messages. Plain and encrypted messages can now be created as prunable. For a prunable message, the message data itself is never a part of the transaction bytes. Instead, only a 32 byte sha256 hash of it is included in the bytes that are being signed, used to verify the signature, or to generate the full transaction hash or id. This makes it possible to completely remove the message data at a later time, keeping only that 32 byte hash, and still be able to verify the transaction signature and have the transaction hash and id unaffected by the pruning. Prunable messages have a predefined lifetime, after which their prunable parts are deleted from the blockchain. This lifetime is measured starting from the transaction timestamp of the message. When a node receives a transaction or a block from a peer, it is only required that the prunable parts of any prunable message are also included if their expiration time has not yet been reached. If it has, and a valid message hash is included instead, the block or transaction will still be accepted. Currently the minimum lifetime of all prunable data is set to 24 h for testnet, to allow easier testing of this feature. For main net, it is tentatively set to two weeks, but this is subject to change before the stable release. Note that pruning is performed at the same time as derived table trimming, which by default is every 1000 blocks, so the actual removal of the prunable data from the database will happen with some delay after their expiration time. A node can choose to keep prunable messages longer, by setting the nxt.maxPrunableLifetime property to a larger value. It cannot be set to prune them earlier. Changing this value only affect transactions received after the change. Pruning can be disabled completely by setting this property to -1. Prunable messages that have not yet been expired, but are past the minimum lifetime, are only retrievable using the getPrunableMessage(s) APIs, but are not included as part of their transaction JSON. If a transaction containing prunable attachments is phased, the prunable parts are nevertheless saved and available immediately, not at finish height. If their expiration deadline comes before the phasing finish height, they will be pruned and not available at finish height. Fees for prunable message attachments are set at 0.1 NXT per 1024 bytes, with the first 1024 bytes free (the regular transaction fee depending on its type still applies). This is again subject to change before the stable release. The full size of prunable message attachments is limited to 42 kbytes. Note that the full size is still included for the purpose of calculating the total block payload, even though only the 32 byte hash is in the transaction bytes. The size of regular plain and encrypted messages in this release has been restricted back to 1000 bytes, and will likely be reduced even further, before the stable release. This will be done in order to encourage users to switch to prunable instead of regular messages. Fees for regular message attachments will also be increased substantially. Creating prunable plain messages of less than 28 bytes is not allowed, as at such small sizes it is more efficient to store the full message instead of a 32 byte hash of it. There is no lower limit on prunable encrypted messages. It is not possible for a transaction to have both prunable plain and prunable encrypted message at the same time. It is also not possible to have both prunable and regular message of the same type (plain or encrypted). It is however possible to have a regular plain message and an encrypted prunable message, or a prunable plain message and a regular encrypted message. Prunable encrypt-to-self messages are not currently supported as there seems to be no good use case for them. Prunable encrypted messages can optionally be compressed before the encryption (default is to compress). The compression status is stored and does not need to be specified when decrypting. Regular encrypted messages are still compressed by default, but this compression can now optionally be disabled. For backwards compatibility, since their current bytes format does not store the compression status, this needs to be specified again when decrypting, else an error or unreadable data will be returned. New APIs: GetPrunableMessage - returns the prunable message for a given transaction id, optionally decrypting it if encrypted and if a secretPhrase is supplied. GetPrunableMessages - returns all prunable messages for a given account id, optionally limiting to only those with another account as recipient or sender, and decrypting them if a secretPhrase is supplied. Prunable messages that have already been pruned are not returned by the above APIs. The UI for those APIs will be implemented in a later release. TrimDerivedTables - a debug API to trigger a derived tables trim, and prunable tables pruning. Changed APIs: All APIs that create a new transaction now also accept optional messageIsPrunable or encryptedMessageIsPrunable boolean parameters (default false). If true, the message or encrypted message attachment, if any, is created as a prunable message. To control whether compression is performed before the encryption or not, the new compressMessageToEncrypt and compressMessageToEncryptToSelf parameters can be used (default true). Prunable messages, if not yet pruned, are also returned as part of the transaction json by the getAccountTransactions API, using the same fields as the corresponding regular messages, but adding a messageIsPrunable or encryptedMessageIsPrunable boolean flag. ReadMessage now also handles prunable message attachments, if any. It takes an optional uncompressDecryptedMessage and uncompressDecryptedMessageToSelf (default true) parameters, only used for non-prunable messages (not needed for prunable ones). DecryptFrom accepts an optional uncompressDecryptedMessage parameter, and encryptTo accepts an optional compressMessageToEncrypt parameter, default true. INCOMPATIBLE changes: BroadcastTransaction has been modified to optionally accept all parameters that are needed to create a prunable plain or encrypted message (client side encryption only). This is required because the data for such messages is never a part of the transaction bytes themselves, so trying to broadcast a transaction that has a prunable part by only submitting its bytes to broadcastTransaction WILL FAIL, unless the corresponding parameters necessary to add the prunable parts are also supplied. Note that they need to exactly match the original parameters with which the transaction bytes have been created and signed. For non-prunable messages, broadcastTransaction behaves as before, but users are encouraged to start switching to prunable messages in view of the planned size restrictions and fee increases. The EncryptedData java class no longer handles compression internally, this is left to the caller of the encrypt and decrypt methods to do. Other changes: GetPolls now supports an optional includeFinished parameter (default false). Multiple bugfixes and improvements in the Voting System and Phasing UI. The limit on transaction deadline of 1440 minutes has been removed. It is now possible to create a transaction with a deadline of up to 32767 minutes. This has been done because many use cases of phasing require that the transaction bytes are prepared much earlier than the actual broadcasting of the transaction to the blockchain. This release will perform a database upgrade with a full rescan on first start. On testnet, it will cause a rollback to block 256935. Title: Re: [Nxt] Implements Blockchain Pruning on testnet Post by: Hollowman338 on April 15, 2015, 11:04:38 AM Great work guys 8)
On a long enough timeline and given enough adoption, is there a way to prune older transactions themselves and still maintain a secure blockchain? Title: Re: [Nxt] Implements Blockchain Pruning on testnet Post by: Troonetpt on April 15, 2015, 11:24:42 AM sounds great, the blockchain of bitcoin is too large now, even the nxt block Chian is a little big, it is good to reduce the size of block chain.
Title: Re: [Nxt] Implements Blockchain Pruning on testnet Post by: Daedelus on April 15, 2015, 03:30:33 PM Great work guys 8) On a long enough timeline and given enough adoption, is there a way to prune older transactions themselves and still maintain a secure blockchain? There is a method planned for that too, this is just the first step. Title: Re: [Nxt] Implements Blockchain Pruning on testnet Post by: Daedelus on April 16, 2015, 08:28:05 AM New APIs added and tweaks made to pruning. Plain and encrypted messages now prunable in the client UI. ;D
Source: https://nxtforum.org/nrs-releases/nrs-v1-5-2e Release 1.5.2e *snip* This is a development release for testing only. Source code is not provided. Change log: This is an experimental release. It is a required update for all testnet nodes, but is also possible to run on main net. Added getAllPrunableMessages API, returning all currently available prunable messages in reverse block timestamp order. Added verifyPrunableMessage API, which can be used to verify that a prunable message obtained from another node (a service provider) matches the hash found in the blockchain, i.e. has not been tampered with. Show maxPrunableLifetime setting in getState and getBlockchainStatus. Show current number of prunable messages in getState. Changing the nxt.maxPrunableLifetime property now also affects existing prunable messages (next time pruning is run), as transaction timestamps instead of expiration timestamps are now used. Reverted DGS goods delivery maximum size to 10 kbytes, same as in 1.4. For goods of a larger size, a prunable encrypted message should be included in the delivery instead. Removed core restriction on 28 byte minimum prunable message length, this will be checked by the client UI only. Added nxt.includeExpiredPrunables property, to be able to force to still include the prunable parts in the returned transaction JSON, even if expired, as long as they are still available. Can be used to force an archival node to always return those, thus allowing another archival node that feeds from it to also get all of them. Limit the number of unconfirmed transactions that can be brought in memory to nxt.maxUnconfirmedTransactions, no limit by default. If set, transactions with lowest fee/size ratio are dropped from the unconfirmed pool first. The ordering is same as the one used when selecting which transactions to include in a new block. Added requeueUnconfirmedTransactions debug API. Added support for uncompressed non-prunable encrypted messages and encrypted to self messages, as a new appendix version, to avoid the need to specify the compression status as a request parameter when reading them. Added setLogging API, to allow changing log levels without having to restart the server. Added eventRegister and eventWait APIs, to be used for registering listeners and waiting for server events instead of polling. Changed the parameters accepted by broadcastTransaction. The prunable parts must be submitted either in the transactionJSON, or, if using transactionBytes, in a new prunableAttachmentJSON parameter, which has the same format as the transaction attachment json. The prunableAttachmentJSON is also returned by the signTransaction and getTransactionBytes APIs now. Added support for prunable plain and encrypted messages in the client UI. Added optional buyer parameter to getDGSGoodsPurchases API. Voting system UI improvements and bugfixes. Title: Re: [Nxt] Implements Blockchain Pruning on testnet Post by: jabo38 on April 16, 2015, 02:49:57 PM that is pretty cool ;D
Title: Re: [Nxt] Implements Blockchain Pruning on testnet Post by: DecentralizeEconomics on April 17, 2015, 07:38:50 AM NXT is the trailblazer of decentralized technology!
Title: Re: [Nxt] Implements Blockchain Pruning on testnet Post by: HCLivess on April 27, 2015, 11:48:06 AM No transaction pruning based on Merkele trees?
Title: Re: [Nxt] Implements Blockchain Pruning on testnet Post by: Daedelus on April 27, 2015, 12:02:32 PM No transaction pruning based on Merkele trees? I think the long term plan might be described as "rolling re-genesis". The states of all objects in Nxt are recorded at a given block and the clients then use this as the 'new genesis'. This is done every x blocks automatically. There are very few concrete details about that element so far so don't quote me ;D Ask Chuckone on Nxtforum. Most likely, as with message pruning above, there will be services set up to store everything (not just the hashes) and charge others to retrieve the pruned data if required. Everyone else will have the hashes of the data so they can confirm that the date provided matches the hash of the tx they have in their blockchain. Title: Re: [Nxt] Implements Blockchain Pruning on testnet Post by: Daedelus on May 16, 2015, 11:37:15 AM Anyone looking to trial Nxt unreleased features can do so now. There is a call for non technical users to test the testnet; your uncle, your mother your boyfriend.. everyone needs to give it a try.
https://nxtforum.org/general-discussion/%28core%29-release-or-not-release/ Using the testnet it easy (they designed it that way ;D ) What is testnet? It is a play area for new features to be trialled and broken before going on the mainnet that we all use otherwise. It looks the same, works the same as you are used to but you get to play with the newest features before they are released ;D Things beginners need to know: 1) Don't use your mainnet passwords on the testnet. 2) Do not use your mainnet passwords on the testnet. Create new testnet accounts you only use on testnet, do this and you will be completely safe. 3) Put the testnet client folder in a different place to you mainnet client folder. Cos you can't have two 'nxt' folders in the same place (stop laughing please) 4) You need to edit one file. No programming skills required, just basic typing skills. Go in to the folder nxt > conf > and open nxt.default_properties in notepad. Where it says nxt.isTestnet=false change it to nxt.isTestnet=true This tells the client to connect to testnet and not mainnet. Save the file and close. 5) This is the link to the testnet client, you can't use the mainnet one: http://localhost:6876/index.html (http://localhost:6876/index.html) 6) Once you have the testnet client running with a brand new testnet account set up, post you account ID here: https://nxtforum.org/testnet/some-testnxt-to-test-asset-exchange/ 7) By doing step 6), you will shortly be sent some FREE TestNXT! (that aren't worth anything). So you will then have the ammunition to...
Personally, I am going to create an MSCurrency and use it as tokens in a vote (yes, you can vote with stuff other than NXT ;D ). If you want to take part, please post your testnet account here and I will send you a few tokens. If I have convinced you to give it a whirl (what is the worst that could happen?), the latest testnet client is 1.5.8e and you can download it from: https://nxtforum.org/nrs-releases/nrs-v1-5-8e/ If this post is scary and frightening (understandable), then don't worry. Just ignore it and make sure you have upgraded the latest stable mainnet client (at time of writing NRS v1.4.18) Please post any questions and I will find someone who can answer them ;D Title: Re: [Nxt] Implements Blockchain Pruning - implemented Post by: Daedelus on October 26, 2015, 10:55:55 PM This thread was to help test and raise awareness of blockchain pruning on the Nxt testnet.
Nxt successfully implemented Prunable Data and it has been on the main net for many months now, contributing to reducing blockchain bloat of the Nxt blockchain. So thread is now locked. Thanks for everyone who helped out ;D |