[NXT] API 2 Brainstorming

[ferment]

We r planning to create other API. Old API will be called Legacy API and become obsolete. New API will be split to 2 parts - Low-level API and High-level API. If u r planning to write a client software u could start a thread to discuss what LL and HL API calls u need.

Before starting with APIs, here's what I understand to be the abstractions in NXT. I've tried to normalize the attributes as much as possible.

Abstractions

Block
- has many transactions
- has a generator account
- has a timestamp
- can be an orphan

Transaction
- has a type/subtype:
   Payment: subtypes: Ordinary Payment
   Messaging: subtypes: Arbitrary message, Alias assignment
   Colored coins: subtypes: Asset issuance, Asset transfer, Ask order placement, Bid order placement, Ask order cancel, Bid order cancel
- has a sender account
- has a recipient account
- has attachments
- has a timestamp
- has fee
- has amount
- has confirmations

Account
- has a secret phrase
- "created" in blockchain by receiving a transaction
- send transactions
- receive transactions
- generates blocks and receives fees (transparent forging)
- generates hallmarks/tokens
- balance defined by transactions and received block generation fees

Peer
- has an address
- has state: non-connected, connected, disconnected
- optionally hallmarked by an account

Alias
 -has a name and URI
 -assigned by a type of transaction using attachment
 -uri updated through a type of transaction using attachment

Asset
-assigned by a type of transaction using attachment?
[not released]

Arbitrary Message
-assigned by a type of transaction using attachment?
[not released]

Exchange
-has AskOrder
-has BidOrder
[not released]

So before brainstorming/designing an API, anything above incorrect?

Any more details on Asset, AM, Exchange, and TF attributes (deadline, etc)?

[1715299860]

1715299860

[1715299860]

1715299860

[1715299860]

1715299860

[1715299860]

1715299860

[1715299860]

1715299860

[Come-from-Beyond]

I'd like to add that everything can be done by using only one API - Low-level or High-level. Low-level requires a developer to understand internals of blockchain and transactions. High-level API is for casual programmers. They'll add a lot of overhead to server part, so the owner of a node may decide to switch High-level API support off.

[EmoneyRu]

Time from last block API request (add to getState)

Useful for stop-on-fork-chain monitoring

[Come-from-Beyond]

Low-level API won't have Accounts. This data is recovered via blockchain analysis.

[marcus03]

In the terminology of the v1 API:

• Unconfirmed transactions included in getAccountTransactionIds (or a second method for getting these)
• Include the next forging account as a return value of the getBlock and getState call
• Get the AliasID for an alias

That's all for now... :-)

[marcus03]

One more regarding orphan cleanup:

Right now my approach to clean up orphans on the client side would be like this:

• Make a list of all blocks ids
• Walk down the chain from the highest block and remove these blocks in the chain from my list
• The remaining blocks in the list would be the orphan blocks and would need to be deleted, together with their transactions

Is there a better approach? Like is there or could there been an API call that would return all block ids that NRS knows are orphans?

[ferment]

Low-level API won't have Accounts. This data is recovered via blockchain analysis.

Is that the same for AM, Alias, and Assets since they are just transaction types?

[Come-from-Beyond]

Low-level API won't have Accounts. This data is recovered via blockchain analysis.

Is that the same for AM, Alias, and Assets since they are just transaction types?

Yes. Low-level API will be just getBlock and getTransactions, maybe a few other requests.

[ferment]

Updated:

Block:
- Can be orphaned

Transaction types:
  Payment: subtypes: Ordinary Payment
  Messaging: subtypes: Arbitrary message, Alias assignment
  Colored coins: subtypes: Asset issuance, Asset transfer, Ask order placement, Bid order placement, Ask order cancel, Bid order cancel


Peer states: non-connected, connected, disconnected

[joefox]

reserved!

[ferment]

Low level API Ideas draft 1

Please note that I'm not super interested in exact names right now, but gathering functionality. I'll use existing names or similar.

My approach is to look at the 3 fundamental low level abstractions (block, transaction, and peer) and present APIs for handling CRUD type operations. For this pass, I'll leave out parameters and also specifics of return value (object vs id vs bytes).

NRS:
getVersion
getTime

Blocks:
getGenesisBlockDate
getLastBlock
getFirstBlock
getBlock
generateBlock
getOrphanBlocks
getNextGenerateTime/getNextGenerator (or something similar) - help?

Transactions:
createTransaction
getTransaction
getUnconfirmedTransactions
broadcastTransaction

Peers:
getPeers
getPeer
connectPeer
disconnectPeer
getLastFeedingPeer? (lastBlockchainFeeder)

With these calls, I think you can do everything essential.

What did I miss?

[Come-from-Beyond]

Ok, let's continue brainstorming.

[marcus03]

As you said - just brainstorming:

What takes most of the time is when my client updates from NRS is getting the updated confirmation counts for transactions with less than 720 confirmations, because there are lots of back and forth calls since you have to call getTransaction again and again over a lot of transactions.

Would be nice to have some kind of batch mode, so e.g. send me all transactions (or just the confirmations counts) that are in block x. Or block x to y. Or block x,y,z, etc. with just one request.

It's probably enough to just get the updated confirmations counts, not everything that is returned in getTransaction, since the rest stays the same.

[ferment]

High level API Ideas draft 1

Please note that I'm not super interested in exact names right now, but gathering functionality. I'll use existing names or similar.

NRS:
getState

Account:
getAccount
getAccountPublicKey
getBalance
encodeToken
decodeToken

Blocks:
getAccountGeneratedBlocks
getBlockTransactions
getNextBlock(current)
getPreviousBlock(current)
getBlocks(start, end)

Transactions:
getAccountTransactions - this is a mid-level call since it returns raw transactions.

Payments:
sendMoney (or sendPayment)
getPayment
getAccountPayments

Alias:
getAlias
getAliasByName
assignAlias
transferAlias
updateAlias (yes, I know its the same as assign, but hide that at high level)
getAccountAliases
searchAlias
searchAliasURI

Arbitrary Message:
The use of the term "message" is somewhat confusing in this context. Maybe just Document or Data?

getMessage
createMessage
updateMessage
getAccountMessages

Colored Coins:
*Need help here*

issueAsset
getAsset
getAssets
updateAsset
transferAsset
getAccountAssets

sendAsset
getAssetBalance
getAssetOwners

placeAskOrder
placeBidOrder
cancelAskOrder
cancelBidOrder
executeAskOrder (better verb?)
executeBidOrder (better verb?)

getAskOrder
getBidOrder
getAskOrders
getBidOrders
getAccountAskOrders
getAccountBidOrders
getAssetAskOrders
getAssetBidOrders
getLastAssetPrice

What did I miss?

[ferment]

On the low level side, it might be nice to have some kind of streaming api for blocks and transactions.

streamBlocks?
streamTransactions?

[ferment]

As you said - just brainstorming:

What takes most of the time is when my client updates from NRS is getting the updated confirmation counts for transactions with less than 720 confirmations, because there are lots of back and forth calls since you have to call getTransaction again and again over a lot of transactions.

Would be nice to have some kind of batch mode, so e.g. send me all transactions (or just the confirmations counts) that are in block x. Or block x to y. Or block x,y,z, etc. with just one request.

It's probably enough to just get the updated confirmations counts, not everything that is returned in getTransaction, since the rest stays the same.

Added getBlockTransactions to high level

[ferment]

One more regarding orphan cleanup:

Right now my approach to clean up orphans on the client side would be like this:

• Make a list of all blocks ids
• Walk down the chain from the highest block and remove these blocks in the chain from my list
• The remaining blocks in the list would be the orphan blocks and would need to be deleted, together with their transactions

Is there a better approach? Like is there or could there been an API call that would return all block ids that NRS knows are orphans?

Added getOrphanBlocks() to the low level.

[ferment]

While we're at high level stuff... how about some blockchain walking calls?

getNextBlock
getPreviousBlock

[Come-from-Beyond]

While we're at high level stuff... how about some blockchain walking calls?

getNextBlock
getPreviousBlock

How the server will know what the current block is?

[ferment]

While we're at high level stuff... how about some blockchain walking calls?

getNextBlock
getPreviousBlock

How the server will know what the current block is?

Pass it in. So you could walk with this pseudo code:

Code:

last = getLastBlock
while last
  last = getPreviousBlock(last)
end

next = getFirstBlock
while next
 next = getNextBlock(next)
end

[ferment]

While we're at high level stuff... how about some blockchain walking calls?

getNextBlock
getPreviousBlock

How the server will know what the current block is?

Another set of cool ones would be getting a range:

getPreviousBlocks(start, end)
getNextBlocks(start, end)

[ferment]

For the high level, batch payments?

sendMultiplePayments via POST with json.

Personally I think sendMoney and anything involving creating transactions should be POST.

[ferment]

Another thing that would be nice is REST urls for the api instead "nxt?requestType". This helps simplify movement to a new api while supporting the old for a while.

Examples:

GET /api/block/2680262203532249785
GET /api/block/list?start=2680262203532249785&end=3705873229391760257

GET /api/account/id?secret=SECRET
GET /api/account/balance
GET /api/account/aliases

GET /api/hallmark/encode?secret=SECRET&weight=100&date=2014-1-9&host=nxt.awesome.is

POST /api/payment/send
POST /api/alias/assign

This style would also make the list of API calls much smaller by condensing them.

[marcus03]

Would a public/private key encryption for the SECRET make sense? Since we'll never have valid SSL certificates?

-Client requests public key from server
-Server send public key to client
-Client enrcrypts SECRET with servers public key. SECRET -> ESECRET
-Client can send commands with parameter SECRET or ESECRET
-Server decrypts ESECRET to SECRET and does the sendMoney, etc.

[Come-from-Beyond]

getPreviousBlocks(start, end)
getNextBlocks(start, end)

Can't it be replaced with just

getBlocks(start, end)

?

[Come-from-Beyond]

Would a public/private key encryption for the SECRET make sense? Since we'll never have valid SSL certificates?

-Client requests public key from server
-Server send public key to client
-Client enrcrypts SECRET with servers public key. SECRET -> ESECRET
-Client can send commands with parameter SECRET or ESECRET
-Server decrypts ESECRET to SECRET and does the sendMoney, etc.

Clients must sign transactions by themselves.

[ferment]

getPreviousBlocks(start, end)
getNextBlocks(start, end)

Can't it be replaced with just

getBlocks(start, end)

?

oh yeah, it can get the timestamp of each block to determine direction. smart...

[ferment]

getPreviousBlocks(start, end)
getNextBlocks(start, end)

Can't it be replaced with just

getBlocks(start, end)

?

We could do something similar with:
getNextBlock(current)
getPreviousBlock(current)

getBlock(offset) - where offset is plus/minus.

Although for APIs I like the literal stuff like getNextBlock(current).

[Come-from-Beyond]

oh yeah, it can get the timestamp of each block to determine direction. smart...

Not smart, just lazy.

[marcus03]

Would a public/private key encryption for the SECRET make sense? Since we'll never have valid SSL certificates?

-Client requests public key from server
-Server send public key to client
-Client enrcrypts SECRET with servers public key. SECRET -> ESECRET
-Client can send commands with parameter SECRET or ESECRET
-Server decrypts ESECRET to SECRET and does the sendMoney, etc.

Clients must sign transactions by themselves.

I was looking for a way to get the plain text secret off the communcation channel between client and server.

If you currently use sendMoney, assignAlias, getAccountId, etc. the SECRET is in plain text and my idea was to prevent this. Won't these calls be available in API v2?

When the client signs the transaction, there is no need to send the SECRET, right?

[Come-from-Beyond]

Right

[ferment]

When the client signs the transaction, there is no need to send the SECRET, right?

Right

So, all of the APIs that currently have a secretPhrase need a variation that assumes the client does their own crypto.

It seems like a bounty project in itself to implement NXT client libraries that do the crypto in different languages.

[Come-from-Beyond]

So, all of the APIs that currently have a secretPhrase need a variation that assumes the client does their own crypto.

It seems like a bounty project in itself to implement NXT client libraries that do the crypto in different languages.

Seems so. Anyway, Legacy API will be supported too. For the time being.

[marcus03]

So, all of the APIs that currently have a secretPhrase need a variation that assumes the client does their own crypto.

It seems like a bounty project in itself to implement NXT client libraries that do the crypto in different languages.

Seems so. Anyway, Legacy API will be supported too. For the time being.

What is the advantage of moving the crypto from server to client?
On the contrary, wouldn't it be better to even have the secret input dialog in the core, so that clients don't even get access to it?

[Come-from-Beyond]

What is the advantage of moving the crypto from server to client?
On the contrary, wouldn't it be better to even have the secret input dialog in the core, so that clients don't even get access to it?

Service Providers will earn money by working as public nodes. And casual users won't need to install Java and NRS server part.

[^[GS]^]

I think it would need to have an API to see if an account is or not forging.

isForging(ACCOUNTID) = Boolean(True/False)

[wesleyh]

Api call to see number of peers that accepted a transaction. This allows us to set a good value for pushTreshold.

[ferment]

I think it would need to have an API to see if an account is or not forging.

isForging(ACCOUNTID) = Boolean(True/False)

Forging is an interesting issue as I think it's technically in what I refer to as the Client supporting servlet (lock and unlock) and not the api. So, I'm not sure how forging will be handled.

I put generateBlock and a placeholder for a call to determine when an account could try to generate a block.

C-f-B?

[ferment]

Api call to see number of peers that accepted a transaction. This allows us to set a good value for pushTreshold.

This would be new functionality since right now Peer.sendToAllPeers (called by broadcastTransaction) doesn't keep any responses.

[Come-from-Beyond]

I put generateBlock and a placeholder for a call to determine when an account could try to generate a block.

C-f-B?

Shouldn't we leave block forging for clients? Coz it's the same as signing transactions, u have to use ur secret key.

[ferment]

I put generateBlock and a placeholder for a call to determine when an account could try to generate a block.

C-f-B?

Shouldn't we leave block forging for clients? Coz it's the same as signing transactions, u have to use ur secret key.

Yes, that's the purpose of generateBlock, but we also need an API for determining when a client can generate blocks like the response that is used to update the timer in the current client. Basically, an API that encapsulates the TF logic to suggest when a client can attempt to generate a block.

[hiksush2]

For High-level:

getAllAccounts() - possibly with filtering parameters and sort/limit

I'm not sure what the Account object provides currently, but these would be useful:

getBalance()
getTotalTransferAmounts() - returns total amount of NXT transferred.  filter by in/out
getTransactions() - filter by activity type (in/out/alias creation/etc.)
getTimeFirstActivity() - filter by activity type (in/out/alias creation/etc.)
getTimeLastActivity() - filter by activity type (in/out/alias creation/etc.)
getBlocksGenerated() - correct term?
getFeeEarned() - correct term?
getAliases()

[wesleyh]

Is it possible to have an api that searches for incoming account transactions with a specific description?  (payment with attached description)

This would be good to verify payments.. Sort of like the system with bank transfers can have a message attached to it.

[wesleyh]

API call to get peers that have public bot access enabled.

[CIYAM]

An API call to get the AM size limit (currently 1000) would be helpful (in order to chop up data and send it in max. size chunks efficient).

[wesleyh]

Enable jsonp so that javascript can call peer via bot access like so: (to get around same origin policy)

?...&callback=myCallback

Can also implement CORS: http://enable-cors.org/server.html

[wesleyh]

add server date to getState as well as lastBlockDate.

[oliviern]

Hello,

I have two main api addons to ask, because I'm faced with a few limitations in my mobile app :

- forged blocks statistics for a given account :
I've been said I have to crawl blocks to know who has computed each, and sum the coins for each account number (I may be wrong...). If it sounds possible, a api call giving forging stats for a nxt account would be relevant

- tokens to replace the passphrase into api calls :
To send money, one has to send his own passphrase through http. This is truly not secured, because communications can be listened, and modified 'reliable' nodes can store passphrases using a api proxy for example... I'd rather have a solution to use api without any passphrase. To do so, one may imagine the following services :

     getAppToken(application_name, application_secretkey) : returns a token (token_app), used to identify application provider. this token can be given publicly to the users who want to give the app access to their accounts

     allowApp(token_app,account_passphrase, array of allowed functions) : returns a token (token_account_app), which certifies that the account owner allows the app to access to a list of api calls (send money, send message, ...)

The token_account_app is sent by the user to the app owner, who can use it to sign secured API calls, with his own application_secretkey. The called node has to verify the matching between application_secretkey, token_account_app, and allowed api services. The application_secretkey allows to certify it is called from the application owner. (this is close to app_ID+app_secretkey used into google,fb,twitter apis for example).

Doing so, the user does not have anymore to send his passphrase to send money, for example. He can also call a disallowApp(token_app,account_passphrase), which returns the same token_account_app, but remove all allowed functions. Doing so, the application can not call anymore any API functions. The allowed functions list could be stored into the blockchain, or even locally on the application dedicated node (ie the user allows a specified node to do API calls).

I've tried to keep it simple, and this is certainly not perfect. This is a very first proposal, in order to talk about with nxt teams. Maybe we could also use aliases to identify applications, or even hack the messages process to validate transactions... Whatever solution is used, I do think it is relevant not to send user's passphrase into the api calls made from external services.

Feedbacks welcome. Thanks,

Olivier

PS : another subject... Wouldn't it be possible to imagine an API call made to check a nxt node code has not been modified (in order to hack accounts for example) ? I know... this function could be modified by the node owner, to return the good checksum... so, this should be made externally, by a external peer. For example, the API service returns the server local main files, calling an external peer hashing service (hashing algorithm being secret, and peer dependent). If it matches, one can say the node uses not modified files. To hack this, one should have to modify hashing algorithm on all peers, which sounds quite difficult. The tested node could also send fake files, but maybe there is a solution to hash running code (and not local files) ?... don't know... this is just a thought.

[ferment]

- forged blocks statistics for a given account :
I've been said I have to crawl blocks to know who has computed each, and sum the coins for each account number (I may be wrong...). If it sounds possible, a api call giving forging stats for a nxt account would be relevant

Try getAccountBlockIds in current API.

[oliviern]

Can't wait either...

[ferment]

I'd rather have a solution to use api without any passphrase.

The best way to do this is to have the client do all of the signing and transmit transaction data to the server. No secret phrases would ever leave the client.

[wesleyh]

Ability to set timestampend so that it returns transactions BEFORE the timestamp. Good for pagination.

As well as a way to set a limit.

[wesleyh]

Ability to get transactions where sender = x and recipient = Y. Especially useful for Arbitrary messaging instead of having to loop through potentially thousands of transactions.

[l8orre]

Ability to get all assets owned by an account. CFB already mentioned including a call for this.

[l8orre]

Ability to get transactions where sender = x and recipient = Y. Especially useful for Arbitrary messaging instead of having to loop through potentially thousands of transactions.

Do I understand correctly that atm it is neccessary to query ALL transactions and then filter the list in the client? yup. looks like it. I am actually catching up here...

[l8orre]

heyhey, I made some experiments - my raspi needs TWO SECONDS to answer a query - while the testnet has a replytime of ~150ms

My concern is the following: when you have to resolve a chain of queries, like when going from block ->transactions -> transaction detail  -- 2 seconds per query does become a factor - you get to resolve 10 or 20 queries, get stuck for 20 to 40 seconds ?!?!?! Think about fast messaging ?!

[BaiMangal]

Ability to get all assets owned by an account. CFB already mentioned including a call for this.

yeah this one is important!

[wesleyh]

Account statistics; number of messages / aliases / assets owned, number of transactions done, number of forged blocks, forged block fees, creation date of account, etc..