Gekko - a javascript trading bot and backtesting platform

[lx]

I might be wrong but mailer seems broken (just pulled the repo, was fine before)
TypeError: Cannot read property 'what' of undefined
    at module.exports.send (C:\Users\lixnet\Downloads\gekko-master\mailer.js:65:
19)
    at EventEmitter.emit (events.js:117:20)
    at bound [as emit] (C:\Users\lixnet\Downloads\gekko-master\node_modules\loda
sh\dist\lodash.js:729:21)
    at TradingMethod.advice (C:\Users\lixnet\Downloads\gekko-master\methods\expo
nential-moving-averages.js:170:12)
    at bound [as advice] (C:\Users\lixnet\Downloads\gekko-master\node_modules\lo
dash\dist\lodash.js:729:21)
    at TradingMethod.calculateEMAs (C:\Users\lixnet\Downloads\gekko-master\metho
ds\exponential-moving-averages.js:98:10)
    at bound [as calculateEMAs] (C:\Users\lixnet\Downloads\gekko-master\node_mod
ules\lodash\dist\lodash.js:729:21)
    at EventEmitter.emit (events.js:92:17)
    at bound [as emit] (C:\Users\lixnet\Downloads\gekko-master\node_modules\loda
sh\dist\lodash.js:729:21)
    at CandleCalculator.calculateCandle (C:\Users\lixnet\Downloads\gekko-master\
methods\realtime-candle-fetcher.js:177:

[whydifficult]

I might be wrong but mailer seems broken (just pulled the repo, was fine before)
TypeError: Cannot read property 'what' of undefined

Nice catch, should be fixed now.

[lx]

Tried to go into real trading @ BTCE and had a message 'BTCe returned ad error, retrying' looping
Also tried to go into real trading @MtGox and had this @ launch :
2013-12-22 16:51:20 (INFO):     Using normal settings to monitor the live market
2013-12-22 16:51:20 (INFO):     Profit reporter active on simulated balance
2013-12-22 16:51:20 (DEBUG):    Setup email adviser.
2013-12-22 16:51:20 (DEBUG):    getting balance & fee from Mt. Gox

TypeError: Cannot read property 'message' of null
    at Trader.checkUnauthorized (/home/ubuntu/gekko/exchanges/mtgox.js:88:9)
    at bound [as checkUnauthorized] (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at calculate (/home/ubuntu/gekko/exchanges/mtgox.js:102:10)
    at bound (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Request._callback (/home/ubuntu/gekko/node_modules/mtgox-apiv2/mtgox.js:48:11)
    at Request.self.callback (/home/ubuntu/gekko/node_modules/mtgox-apiv2/node_modules/request/index.js:148:22)
    at Request.EventEmitter.emit (events.js:98:17)
    at Request.<anonymous> (/home/ubuntu/gekko/node_modules/mtgox-apiv2/node_modules/request/index.js:876:14)
    at Request.EventEmitter.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (/home/ubuntu/gekko/node_modules/mtgox-apiv2/node_modules/request/index.js:827:12)

Putting the 'secret' key seems to trigger this issue ... any idea ?
(API keys have the correct rights setup, I can use them from within my own software)

[whydifficult]

Darn, I am sorry for that. Fixed now.

[whydifficult]

Tried to go into real trading @ BTCE and had a message 'BTCe returned ad error, retrying' looping
Also tried to go into real trading @MtGox and had this @ launch :
2013-12-22 16:51:20 (INFO):     Using normal settings to monitor the live market
2013-12-22 16:51:20 (INFO):     Profit reporter active on simulated balance
2013-12-22 16:51:20 (DEBUG):    Setup email adviser.
2013-12-22 16:51:20 (DEBUG):    getting balance & fee from Mt. Gox

TypeError: Cannot read property 'message' of null
    at Trader.checkUnauthorized (/home/ubuntu/gekko/exchanges/mtgox.js:88:9)
    at bound [as checkUnauthorized] (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at calculate (/home/ubuntu/gekko/exchanges/mtgox.js:102:10)
    at bound (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Request._callback (/home/ubuntu/gekko/node_modules/mtgox-apiv2/mtgox.js:48:11)
    at Request.self.callback (/home/ubuntu/gekko/node_modules/mtgox-apiv2/node_modules/request/index.js:148:22)
    at Request.EventEmitter.emit (events.js:98:17)
    at Request.<anonymous> (/home/ubuntu/gekko/node_modules/mtgox-apiv2/node_modules/request/index.js:876:14)
    at Request.EventEmitter.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (/home/ubuntu/gekko/node_modules/mtgox-apiv2/node_modules/request/index.js:827:12)

Putting the 'secret' key seems to trigger this issue ... any idea ?
(API keys have the correct rights setup, I can use them from within my own software)

The error was a bug that has since been fixed.

The looping of 'BTCe returned ad error, retrying' has probably something do to with bitcoincharts (make sure the interval is above 15min). I am currently working on a version that doesn't use bitcoincharts anymore but the limited BTC-e [,CEX.io & Bitstamp] API. We have a lot of problems with the bitcoincharts API and Gekko is trusting an unknown third party for data it uses to give you trading advice.

[patriotek]

Ouch...

OK, so again, I have the issue that it seems to be trading backwards.

This, now, is on BTC-e, all defaults, trading for real. Before, it was happening on CEX.io in sim-mode.

Here's the log of my trades (all times UTC):

Date     Amount        Type       Price         Total               Manual or Gekko

03:57   B0.0170397   bought    $630.00     $10.73501101   Manual
04:18   B0.0170057   sold        $635.50     $10.80712235   Manual
04:32   B0.0170656   bought    $632.00     $10.78545921   Manual
(So far so good, right?)
13:55   B0.0170315   sold        $617.10     $10.51013866   Gekko (Are you freakin' kidding me?)
Then Gekko crashed.  It was not running when I got home, so I re-ran it.
12:43   B0.0159465   bought    $657.77     $10.48912931   Gekko (Umm... Really?)

I'm not sure what caused Gekko to crash.  If I had to take a guess just now, I'd say it was likely a flakey moment in my InterWebs connection (quite frequent, unfortunately).  What I CAN say is that:

1.  Gekko darn well should be able to stand up to a loss of Internet, and recover from it when returned
2.  Gekko should re-read logs from last run to determine if the Buy order it's creating is going to lose us money
and
3.  When the price is low, it's time to BUY, NOT SELL...

[whydifficult]

I'm not sure what caused Gekko to crash.  If I had to take a guess just now, I'd say it was likely a flakey moment in my InterWebs connection (quite frequent, unfortunately).  What I CAN say is that:

1.  Gekko darn well should be able to stand up to a loss of Internet, and recover from it when returned
2.  Gekko should re-read logs from last run to determine if the Buy order it's creating is going to lose us money
and
3.  When the price is low, it's time to BUY, NOT SELL...

1. Gekko is an open source project: it might contain bugs that will crash it. That said, it is able to recover after connection interrupts.
2 I am working on a new version that does store history, I am working on this on my own in my spare time and I need people to help me test it, read more here.
3. You are so right, I am now going to change Gekko so that it will never sell when the price is low again and never buy when the price is high again...

If you download Gekko (open source software), turn it on at defaults settings, load real money on the account and expect to pull money out of the air I can't really help you at all.

Gekko is a (free and open source) trading bot that watches trends according to how you configure it yourself. If you feel the default trendwatcher is flawed:

1. Don't use it for real money.
2. Complain about it to the guy who popularized it.
3. Come up with better settings that will lead to profit (using Gekko's backtester & paper traders or external tools).
4. Configure Gekko to use your better settings.

-----

I have some awesome news guys, but I do need your help!

I am now working on a big rewrite that will store the trade history persistently on disk, this means that it will store all data and doesn't rely on API's to always be able to fetch newest stuff anymore. This means:

- Able to watch CEX.io & Bitstamp again
- Able to watch BTC-e (including all altcoin markets)
- Ability to add basically all other exchanges without requiring them to handover older historical data.

It isn't done yet and at this moment it will only save historical trades (so no advice / mail / trading yet) but I need people who want to help me test to get this solid. More info here!

[northcape]

I’m currently running the localDB branch on the side and it seems to record the data just fine from what I can tell.

The db file produced seems to be similar to a CSV, but I noticed that it isn’t ordered by the first fieldname ("s").

Right now my file has entries with "s" values from 388 to 482 in a random order, then a line that says

Code:

{"$$indexCreated":{"fieldName":"s","unique":true,"sparse":false}}

and then entries 483 to 522 neatly ordered by their incremental "s" value.

Could you maybe give me a very short explanation of what the field names stand for?
I just wanted to make sure that this is intended behaviour.

[whydifficult]

I’m currently running the localDB branch on the side and it seems to record the data just fine from what I can tell.

The db file produced seems to be similar to a CSV, but I noticed that it isn’t ordered by the first fieldname ("s").

Right now my file has entries with "s" values from 388 to 482 in a random order, then a line that says

Code:

{"$$indexCreated":{"fieldName":"s","unique":true,"sparse":false}}

and then entries 483 to 522 neatly ordered by their incremental "s" value.

Could you maybe give me a very short explanation of what the field names stand for?
I just wanted to make sure that this is intended behaviour.

What data is it storing?

I had three options to store historical trade data in a way that would let me access the data again:

- Store all trades
- Store candles at the size that is in the current config
- Store 1 minute candles.

The first option creates the biggest databases. I need to store the data in a way that works cross platform and I want to limit the memory footprint. On markets which lots of volumes this can mean a couple of MB in a busy hour. This is not really an acceptable size for me.
The second option means that if you want to change from 60 min to 30 min candles you cannot use previous history.

So I went for the first option: I know exactly how much data per day is required and I am able to calculate every candle (> 1 minute) on the fly based on those.

How is it stored?

I didn't write all the code for storing the local database myself, instead I used neDB. Databases are hard to do correctly so I'm happy I don't have to reinvent the wheel here.

The first line tells the database that there is an index on the `s` field, which means we can query specific `s` recods faster. The other lines are part of the neDB filesystem (stored in JSON). Each record is a 1 minute candle with the following properties:

- s: start minute since UTC midnight (we store each day in a different database, check it in the filename)
- o: open price of this minute
- l: lowest price of this minute
- h: highest price of this minute
- c: close price of this minute
- v: total trade volume of this minute

Can I re use other data?

I am going to write tools that can convert this database into a popular CSV format that I see a lot in the forums, also the other way around (but if the CSV candles are > 1m this means you loose accuracy in the data).

[northcape]

Great explanation, thank you.

I take this means that

Code:

{"$$indexCreated":{"fieldName":"s","unique":true,"sparse":false}}

does not always need to be the very first entry of the database file and the lines don’t need to be ordered by incrementally the "s"-key value?

Other than that the localDB branch works fine for me. Looking forward to it because I just ran into this issue again:

https://github.com/askmike/gekko/issues/51

Oddly enough it loads the first entries just fine, but gekko fails on the last (most recent) candles, retries a few times and then shuts down.I assume there are no connectivity or DNS issues, otherwise it wouldn’t be able to download the first 99 candles.

[plethora]

- Store all trades
- Store candles at the size that is in the current config
- Store 1 minute candles.

The first option creates the biggest databases.
...
So I went for the first option: I know exactly how much data per day is required and I am able to calculate every candle (> 1 minute) on the fly based on those.

Do you mean that you went for the third option?

[whydifficult]

Great explanation, thank you.

I take this means that

Code:

{"$$indexCreated":{"fieldName":"s","unique":true,"sparse":false}}

does not always need to be the very first entry of the database file and the lines don’t need to be ordered by incrementally the "s"-key value?

Other than that the localDB branch works fine for me. Looking forward to it because I just ran into this issue again:

https://github.com/askmike/gekko/issues/51

Oddly enough it loads the first entries just fine, but gekko fails on the last (most recent) candles, retries a few times and then shuts down.I assume there are no connectivity or DNS issues, otherwise it wouldn’t be able to download the first 99 candles.

I also really don't like their API, hopefully I'll get the new method working quickly.

- Store all trades
- Store candles at the size that is in the current config
- Store 1 minute candles.

The first option creates the biggest databases.
...
So I went for the first option: I know exactly how much data per day is required and I am able to calculate every candle (> 1 minute) on the fly based on those.

Do you mean that you went for the third option?

oops, yes that's what I meant.

[plethora]

Code:

  short: 10,
  long: 21,
  // amount of candles to remember and base initial EMAs on
  candles: 100,

How come the default number of candles are 100 when the long EMA is only using 21? Is this only usable for the backtesting function? Is is safe to set candles: 21 if I'm not using backtesting?

[whydifficult]

Code:

  short: 10,
  long: 21,
  // amount of candles to remember and base initial EMAs on
  candles: 100,

How come the default number of candles are 100 when the long EMA is only using 21? Is this only usable for the backtesting function? Is is safe to set candles: 21 if I'm not using backtesting?

Nope, EMA works by calculating an average out of the current price + the last average price. So just like the blockchain it calculates new EMAs on top of all the old ones. The older the candle the (exponentially) less value it has in the current EMA. So:

- 100 candles means that it needs to calculate at least 100 EMAs on top of each other.
- 21 long means that the the old EMAs are more important than the new one. The result is a line that reacts slower to trends and lags behind a little (because new candles have little impact).
- 10 short means that the old EMAs are less important than the new one. The result is a line that reacts faster to trends and lags less behind than the long one.

If you lower the 100 it will result in the first EMAs calculated not really being EMA values.

[vual]

Any hints with this?
Is mtgox-apiv2 not up to date with latest node?
Very new to this "node" technology.

First:

Code:

sudo npm install
npm ERR! Error: failed to fetch from registry: mtgox-apiv2

So i went to the url, https://registry.npmjs.org/mtgox-apiv2 yes its there, nice.

Then I did,

Code:

sudo npm install mtgox-apiv2 --registry http://registry.npmjs.org/
npm http GET http://registry.npmjs.org/mtgox-apiv2
npm http 304 http://registry.npmjs.org/mtgox-apiv2
npm ERR! Error: No compatible version found: mtgox-apiv2@'>=1.0.0- <1.1.0-'
npm ERR! No valid targets found.

node -v v0.6.19
npm -v 1.1.4

Does this not support my node version? I just grabbed node from git.
thanks for any help.

EDIT:
Sorry turns out maybe i must of grabbed the node source from the wrong github, getting new source for node now https://github.com/joyent/node

To anyone else having this issue make sure you "apt-get remove nodejs" turns out ubuntu comes prepacked with a shitty old version.

[whydifficult]

I'm developing a new version of Gekko that is able to deal with limited APIs such as: Bitstamp, CEX.io & BTC-e (so all altcoin markets as well). There is a discussion taking place about how Gekko should store historical data. Here is the context and here is the specific issue.

If anyone is interested in running Gekko, could you weigh in on the discussion with:

- On what machine do you want to run Gekko (desktop, laptop, raspberry PI or VPS/cloud instance)
- How big can the history be that Gekko stores on disk (in MB).
- How much memory can Gekko consume per market you want to watch (1-10MB, 10-100MB or more?)

If you want to input on the implementation side that's also fine. Either do it here, on github or on IRC: #gekkobot (freenode).

[whydifficult]

I've put out my first bounty, whoho!

If anybody wants to make some money by contributing to Gekko:

I don't really understand why we are looking at trends for CEX.io, as the value of 1 GHS depends not only on the value for which you can sell it, but also on how much BTC it will mine for you. I propose a new method that doesn't look at trends but uses an online mining profit calculator to determine how much BTC 1 GHS is going to bring you, if the price is < profit, buy all the GHS and if the profit is < price sell all the GHS.

I will write this when my todo list is a bit smaller, so I'm putting a bounty out if someone can do it before me: If someone can implement it (in the localDB branch) I will pay that person 0.01 BTC (Remember I am poor because I opensourced Gekko)!

Expectations:

- Same coding style as the EMAC / core.
- Uses an online mining calculator to figure out how much profit 1 GHS will bring you. The settings need to be configurable in `config.js` (settings are stuff like `block halving time` / `diff increase time` / `diff increase %`).
- If it will profit more than the current price, emit `long` advice. If it will profit less than current price, emit `short` advice.
- Needs to be plugged into the tradeAdvisor just like the EMAC is.

'm obv. only paying out the bounty once, and to the first pull request that satisfies all expectations.

Here is the context.

[patriotek]

@ whydifficult

Forgive me please, if you will, for the tone of my last... Had a rough day (one of many), and this issue was just the bitter icing on a dirt cake.  Ya' know what they say about excuses and back-sides...lol

I do understand that Gekko is a WIP, and may possibly crash.  I also understand that default settings may not be the best for a given set of circumstances.

I guess what I do not yet understand is why, in 2 separate situations on 2 different exchanges (and I believe on different systems), it consistently lost money rather than making any gains whatsoever, and crashed:

1.  CEX.io in sim-mode with settings changed according to recommendations made here in this thread

2.  BTC-e in trading mode with defaults, but with no indication that I've yet seen that the defaults are not appropriate for it.

Might I ask the following, as it sounds like you're re-writing much of it already:

How difficult might it be to set it up such that depending upon which exchange / currency pair you set it to run with, it sets safe or at least lower risk defaults accordingly?  As well, it might be valuable to set up the startup scripts so that if the service fails, it can be stopped and restarted.  I haven't had any time (nor do I have much supporting knowledge) to check into how this might be done, but it was an idea that I had...

If I weren't so behind on modern software development (last I programmed was Basic on C-64 and crApple 2c's...hehe), I'd dig in and help.  For that matter, I'm still hoping to, to some minimal extent.  If I weren't so dead broke, I'd donate now.  I will donate once Gekko's helping me dig outta' this pit I'm in!!

Again, my apologies for previous attitude, and appreciation for your work on this!



edit:  Shoulda' gone to bed, but...  This got my attention in the config's:
  sellTreshold: -0.25,
  buyTreshold: 0.25

Forgive my financial ignorance (I'm quickly learning), but might these be reversed?  In my mind, you'd want to sell at EMA+N.nn rather than -N.nn, and buy at -N.nn, but I can't be sure that I'm interpreting things correctly, or even in the same ball-park...

[gatuso5]

how to set a maximum transaction volume in BTC/USD??
Example: maxUSD: 50

So program will not use all acc money.

[whydifficult]

@ whydifficult

Forgive me please, if you will, for the tone of my last... Had a rough day (one of many), and this issue was just the bitter icing on a dirt cake.  Ya' know what they say about excuses and back-sides...lol

I do understand that Gekko is a WIP, and may possibly crash.  I also understand that default settings may not be the best for a given set of circumstances.

I guess what I do not yet understand is why, in 2 separate situations on 2 different exchanges (and I believe on different systems), it consistently lost money rather than making any gains whatsoever, and crashed:

1.  CEX.io in sim-mode with settings changed according to recommendations made here in this thread

2.  BTC-e in trading mode with defaults, but with no indication that I've yet seen that the defaults are not appropriate for it.

Might I ask the following, as it sounds like you're re-writing much of it already:

How difficult might it be to set it up such that depending upon which exchange / currency pair you set it to run with, it sets safe or at least lower risk defaults accordingly?  As well, it might be valuable to set up the startup scripts so that if the service fails, it can be stopped and restarted.  I haven't had any time (nor do I have much supporting knowledge) to check into how this might be done, but it was an idea that I had...

If I weren't so behind on modern software development (last I programmed was Basic on C-64 and crApple 2c's...hehe), I'd dig in and help.  For that matter, I'm still hoping to, to some minimal extent.  If I weren't so dead broke, I'd donate now.  I will donate once Gekko's helping me dig outta' this pit I'm in!!

Again, my apologies for previous attitude, and appreciation for your work on this!

(I'm sorry if I sound a bit grumpy, this is not to you personally but I'm going to link to this general answer every time this happens.)

Gekko is not a money making machine. Gekko is a tool for traders to automate buying and selling according to simple mathmetical methods.

Gekko works as long as it is able to run the implemented algorithmes correctly over either realtime market data or historical market data. Configuring the algorithmes in a way they are profitable is not up to Gekko and also not up to me (I wrote the default config).

The default config was based on this thread by a trader. It appears during the last few months the proposed settings are not as profitable as they once were. So if you are losing money there might be two cases:

- You configured an algorithm in a way that it won't generate profit. -> blame yourself, read that thread and first backtest potential new configurations before letting Gekko play with your money again - also use the paper trading (Gekko will simulate profits without your money at risk).
- There is a bug in Gekko so it implements the algos wrongly or interpets the market data wrong, etc. -> blame me & tell me about what is wrong.

--

I'm getting tired of people leeching Gekko when the default settings are profitable and not taking part in any discussion and start complaining when it isn't anymore. Everybody is free to come up with more profitable settings and I'll put them in the default config.

--

edit:  Shoulda' gone to bed, but...  This got my attention in the config's:
  sellTreshold: -0.25,
  buyTreshold: 0.25

Forgive my financial ignorance (I'm quickly learning), but might these be reversed?  In my mind, you'd want to sell at EMA+N.nn rather than -N.nn, and buy at -N.nn, but I can't be sure that I'm interpreting things correctly, or even in the same ball-park...

So those thresholds are part of the EMA method, a trendwatching algo. It is trying to detect trends: if it thinks the price is going up it will advice to buy (so you can sell later when the price is higher and have profit) and if the price is going down it will advice to sell.

sellThreshold determines how fast the trend has to be going down for it to sell (hence negative).
buyThreshold determines how fast the trend has to be going up for it to buy (hence positve).

Note you might not find these parameters in Goomboo's thread, that's mainly because in a flat market Gekko will detect any minor price movement as a new trend, advice to buy/sell and the exchange will eat all your money away in fees.

how to set a maximum transaction volume in BTC/USD??
Example: maxUSD: 50

So program will not use all acc money.

Not possible in this version, coming up in the next. The only thing you can do for now is get all your other funds out of the exchange.