Isis ATP [Automated Trading Platform] - Discussion

[Raoul Duke]

I'm getting a lot of these

Code:

INFO: Attempting to sell 0.136541308148148140568582942222519704955630004405975341796875 of 0.92165383 BTC available
Nov 09, 2012 3:30:08 PM org.open.payment.alliance.isis.atp.TradingAgent marketOrder
INFO: Market Order return value: f432d137-feda-4bc0-8c6e-721e8ff3dcf9
Nov 09, 2012 3:30:08 PM org.open.payment.alliance.isis.atp.TradingAgent marketOrder
SEVERE: Failed to sell 0.136541308148148140568582942222519704955630004405975341796875 at current market price.
Please investigate

The thing is, it did sell that value. I suspect it's only trading one way. As it doesn't recognize the sale happened it never executes the other side of the trade, thus, there s no arbitrage profit.

[1714038635]

1714038635

[Aido]

Very strange.

The logic used in the trading agent is to first attempt a sell order. If the return value is null the sell order failed so no need to attempt the buy side of the arbitrage.

But, I can clearly see from the output that the return value is not null so the sell order was successful. Yet, the trading agent thought it was not successful so did not attempt the buy order.

I will investigate this shortly when I have time.


UPDATE:

OK, I found it.

In TradingAgent.java:

Code:

if(marketOrderReturnValue != null){
    success = false;
}

!= should be ==.

I will investigate update this shortly when I have time.

For the time being please disable arbitrage by using the --use-arbitrage=false switch. Ah, an actual use for this new feature.

Normal non-arbitrage trades will still be executed even though the log will tell you different. This may also mess up the P/L calculations a bit until I fix it.

[Aido]

A few more commits, please test:

commit 1b3477509ea5aa8365a0f90d2dba649b82ab1f4a

    Deprecating --debug-live in favour of --simulation-mode

I think --simulation-mode is more descriptive and intuitive than --debug-live. For now --debug-live=true is same as --simulation-mode=false and vice versa, but eventually debug-live will be removed.



commit 27717c11e3a51f38fcac070bc9223f9306b58232

    Fixed false positives on failed market order test in trading agent

This hopefully fixes this issue:

Code:

INFO: Attempting to sell 0.136541308148148140568582942222519704955630004405975341796875 of 0.92165383 BTC available
Nov 09, 2012 3:30:08 PM org.open.payment.alliance.isis.atp.TradingAgent marketOrder
INFO: Market Order return value: f432d137-feda-4bc0-8c6e-721e8ff3dcf9
Nov 09, 2012 3:30:08 PM org.open.payment.alliance.isis.atp.TradingAgent marketOrder
SEVERE: Failed to sell 0.136541308148148140568582942222519704955630004405975341796875 at current market price.
Please investigate

commit 62f40b79892f52ef349af47c1c31430413d5d7d6

    Bound slf4j to the logback logging framework and tidied up log messages.

The logback framework is extremely flexible and opens up the possibility to easily add log file rotation, remote logging, e-mail of logs etc. by simple editing the logback.xml file.

[Aido]

A few more commits:

commit 734f36f7324faebf2bd1e7950bf5a256fb83a802

    Changed logback config to log brief messages to screen and verbose messages to a rolling log file

Log messages sent to screen are now more brief so most stay on single line. More detailed messages are sent to the aido.log log file. This file rolls over when it reaches 32MB and keeps a history of eight gzipped log files.



commit 2a1431b9716aee722baeab7e1a8ba9e84908593a

    Fixed NullPointerException in Trading Agent

Self explanatory.


commit f51f5223fbcc65dca6ebd70a29b86fb060132f58

    Changed trading agent to use BigMoney for currency amounts instead of BigDecimal

Ditto!

[Raoul Duke]

Can we disable the trend engine and leave only the arbitrage one working?
Or passing it --simulation-mode=true will put the arbitrage engine also in simulation mode?

[Raoul Duke]

I've been getting this:

Code:

Isis started successfully.
Exception in thread "Thread-19" java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:894)
	at java.util.HashMap$KeyIterator.next(HashMap.java:928)
	at org.open.payment.alliance.isis.atp.ArbitrageEngine.getHighestBid(ArbitrageEngine.java:174)
	at org.open.payment.alliance.isis.atp.ArbitrageEngine.run(ArbitrageEngine.java:50)
	at java.lang.Thread.run(Thread.java:722)

and

Code:

2012-11-10 17:50:38,043 : Weight is 0.14666666666666667
Exception in thread "Thread-49" java.lang.NullPointerException
	at org.open.payment.alliance.isis.atp.TradingAgent.evalBid(TradingAgent.java:227)
	at org.open.payment.alliance.isis.atp.TradingAgent.run(TradingAgent.java:115)
	at java.lang.Thread.run(Thread.java:722)

Code:

2012-11-10 17:49:36,990 : Weight is 0.14473684210526316
Exception in thread "Thread-44" java.lang.NullPointerException
	at org.open.payment.alliance.isis.atp.TradingAgent.evalBid(TradingAgent.java:227)
	at org.open.payment.alliance.isis.atp.TradingAgent.run(TradingAgent.java:115)
	at java.lang.Thread.run(Thread.java:722)

I think these 2 last ones happen when it's supposed to trade.

And I also get the following

Code:

2012-11-10 17:53:45,695 : Arbitrage Engine has detected an after fee profit opportunity of 56,48830811% on currency pair CAD/GBP
2012-11-10 17:53:45,695 : ***Conversion Factors***
2012-11-10 17:53:45,696 : Highest Bid: (GBP, 1.5729537882080256)
2012-11-10 17:53:45,696 : Lowest Ask: (CAD, 0.997070707070707)
2012-11-10 17:53:46,413 : Market Buy Order return value: null
2012-11-10 17:53:46,414 : Buy failed. Arbitrage could not trade CAD 10.66823 with GBP 0.00000000

Any idea why the trade fails? Because I have balance on CAD, GBP and BTC.

[daybyter]

I had similar bugs in my code, so maybe I can help a bit...

You get the first error, when you create an iterator, then iterate over a list (or hash map in your case) and then modify the list, while you loop over it.

Potential solution is too avoid an iterator and just check for the end of the list after each loop iteration.

[scrybe]

Can we disable the trend engine and leave only the arbitrage one working?
Or passing it --simulation-mode=true will put the arbitrage engine also in simulation mode?

Recommendation/request:

Instead of having parameters that are set at command line for the trading, arbitrage, and simulation settings, could we handle them in the registry so we only have 1 place to manage settings? It might also be nice to be able to turn different modes on/off during a longer run by updating the registry without relaunching the program.

Thoughts?

[Aido]

The NullPointerException problem was introduced when I changed the trading agent to use BigMoney instead of BigDecimal for currency amounts.

I am currently debugging it and hope to have a fix soon. I've added some debug messages and it seems that some values are null because the trading agent isn't parsing the config file values into BigMoney correctly.

daybyter is correct about the ConcurrentModificationException problem and it's also on my todo list. This isn't as frequent as the NullPointerException which is stopping trades.

The past few commits have introduced a bump in the road but I'll get over it with the next commit. I think I may have been introducing too many changes lately without testing them as thoroughly as I should have.

Can we disable the trend engine and leave only the arbitrage one working?
Or passing it --simulation-mode=true will put the arbitrage engine also in simulation mode?

--simulation-mode and --debug-live don's affect arbitration trades. In fact, if the arbitration engine decides to perform an arbitration trade it temporarily disables the other trades.

I have been thinking about sorting out these switches but for now the way you suggest should work.

Instead of having parameters that are set at command line for the trading, arbitrage, and simulation settings, could we handle them in the registry so we only have 1 place to manage settings? It might also be nice to be able to turn different modes on/off during a longer run by updating the registry without relaunching the program.

I don't think my Linux machine has a registry .  At the moment it's convenient for me to have the switches on the command line while I test them. But I may reconsider in the future when I tidy up how they work. Also, if not supplied in command line, both --simulation-mode and --use-arbitration default to true.

[Raoul Duke]

Your machine doesn't have a registry but it does have ~/.java/.userPrefs/org/open/payment/alliance/isis/atp/prefs.xml

[scrybe]

Your machine doesn't have a registry but it does have ~/.java/.userPrefs/org/open/payment/alliance/isis/atp/prefs.xml

Hey, that looks suspiciously like the key in the registry...
[HKEY_CURRENT_USER\Software\JavaSoft\Prefs\org\open\payment\alliance\isis\atp]

Thanks, I didn't know how that translated.

[Aido]

A few more commits to play with and test (https://github.com/aido/IsisATP/commits/master):

commit 81057758348b0f35c8c1803086c13aec676c1b04

    Tweak highFactor and lowFactor values in arbitrage engine

commit 6e95342753e71029215d773ab42c9f5a40232456

    Moved build_IsisATP.sh to make.sh and updated

commit 42018488c33d7c00ca6c5e3f6755b2fc8e2e4b99

    Attempted fix for ConcurrentModificationException error in arbitrage engine

commit bc90d5cf77d884aee452a3e85f6834145048eec3

    Fixed BigDecimal.ONE.divide problem in arbitrage engine which was stopping market orders

commit d2474d5d8dbd1be153aa8acf74517d3b543991cc

    Fixed BigMoney parsing in trading agent which was causing NullPointerException


I think I fixed the ConcurrentModificationException error. It will need more testing to confirm.
I left the latest commit run for a few hours and didn't receive any errors. Please test and report back if it re-occurs.

I have now hopefully ironed out most of the bugs introduced with the previous bunch of commits.

[Aido]

I see the following rare exception occur in the ticker manager:

Code:

Exception in thread "Thread-8" java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:371)
at java.util.ArrayList.get(ArrayList.java:384)
at org.open.payment.alliance.isis.atp.TickerManager.getLastTick(TickerManager.java:160)
at org.open.payment.alliance.isis.atp.AccountManager.getLastTick(AccountManager.java:155)
at org.open.payment.alliance.isis.atp.ArbitrageEngine.executeTrade(ArbitrageEngine.java:135)
at org.open.payment.alliance.isis.atp.ArbitrageEngine.run(ArbitrageEngine.java:86)
at java.lang.Thread.run(Thread.java:722)

The problematic code is as follows:

Code:

ATPTicker tick;
synchronized(tickerCache) {
	tick = tickerCache.get(tickerCache.size()-1);
}

I don't understand why it is throwing an ArrayIndexOutOfBoundsException when the code checks the size of the array with tickerCache.size()-1

any ideas?

[daybyter]

Well, your exception already tells you the problem...your index is -1 , so the size() seems to be 0. The array is empty...

btw: this thread gave me the great idea to use a special class for amounts and prices. Thank you!

[Aido]

Well, your exception already tells you the problem...your index is -1 , so the size() seems to be 0. The array is empty...

Nice one!

thanks daybyter.

btw: this thread gave me the great idea to use a special class for amounts and prices. Thank you!

Care to share?

[Aido]

A couple more commits:

commit 6d95bf71f75a7fb251102fd2083804d397bab4ee

    Improved exception handling for disconnection from exchange

When a com.xeiam.xchange.HttpException is thrown we now create a test socket to the exchange. If this test socket fails to connect we wait one minute before trying again. At the moment the test socket is as follows:

    testSock = new Socket("www.mtgox.com",80);

mtgox.com port 80 is hardcoded. I couldn't figure out how to dynamically get the host and port from ExchangeSpecification. Any suggestions on how to do this?


commit 46ea53d031019356d0815069d82a533d22fc7463

    Fix for ArrayIndexOutOfBoundsException in getLastTick method of Ticker Manager

This fix is a bit of a hack. I'm sure it will result if some NullPointerException's further down the line.


Also, when testing it may be a good idea to redirect STDERR to a file so error messages can be analysed later:

    java -jar aido.jar --simulation-mode=false 2>> aido.error


I think this thread may have gone off topic a bit and become more of a crash course in java. Apologies.

[daybyter]

I use log4j in my app. It should fill your requirements.

[scrybe]

good grief, how much precision do we need?

Nov 13, 2012 11:58:46 AM org.open.payment.alliance.isis.atp.TradingAgent evalAsk
INFO: 0.0000009982608695652174034262937851735841832123696804046630859375 was less than the configured limit of 0.01
There just isn't enough momentum to trade at this time.

[daybyter]

That's good and necessary....

I use factor 100000000 for the price and the amount. However, if you start calculating with those amount and do (amount * price) / 100000000 you have a problem with longs. I use an ugly hack and cast temporarily to double, but I guess I'll use some BigDecimal-like format in the future. Just a lot of places, where I have to modify my code...

[Aido]

New commit on github:

commit 790351ae9d849c9fee4f11dff25d602efbade343

    A possibly better fix for ArrayIndexOutOfBoundsException in getLastTick method of TickerManager that will not cause NullPointerException's later in code.

A few other cosmetic changes in there also.

Linux/UNIX/Mac OS users now get fancy coloured log output, INFO, WARN and ERROR etc.  Windows users........sigh!

good grief, how much precision do we need?

The latest version keeps the precison in calculations but no longer prints out the unnecessarily long string of digits to the log, rounding output to eight decimal places instead.

I use log4j in my app. It should fill your requirements.

I changed IsisATP to use logback (http://logback.qos.ch/). According to the logback website:

Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.

XChange also uses logback so the class files were already included in the original IsisATP jar file.

I'm impressed by the power of logback. If a user doesn't like the format or how IsisATP logs its output they can use their own logback config file to decide how and where log output is presented. Want your logs mailed to you? Easy....just add an e-mail appender to your logback config. And, all without changing a single line of code.



Some of the calculations used in the arbitrage engine need to be looked at. For example, I just don't believe the following output:

    Arbitrage Engine has detected an after fee profit opportunity of 58.43022025% on currency pair USD/GBP

Anyone care to try and get their head around the calculations and advise?