Bitcoind taking > 10GB memory

[zvs]

I noticed bitcoind was taking 10GB of memory, so I restarted it & it quickly started climbing again.

I noticed this peer:

        "addr" : "xxx",
        "services" : "00000001",
        "lastsend" : 1371730754,
        "lastrecv" : 1371730752,
        "bytessent" : 270796,
        "bytesrecv" : 73563150,
        "conntime" : 1371730642,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.2.1/",
        "inbound" : true,
        "startingheight" : 242438,
        "banscore" : 0

-------

This was after being connected to me for about 30 seconds.

It was occurring on both of my nodes, both from same IP.

I firewalled & started it w/ 5 max outbound but still with listen on... so far after about 30m, the memory behavior has been normal, it's using 400MB right now.

But it's not like I can keep the person from using a different IP, even if I block an entire subnet?

[zvs]

This was about 30 seconds after a restart & the very top of my list, so he had me on addnode also, but:

        "addr" : "xxx:55443",
        "services" : "00000001",
        "lastsend" : 1371730754,
        "lastrecv" : 1371730765,
        "bytessent" : 268610,
        "bytesrecv" : 93228,
        "conntime" : 1371730635,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : true,
        "startingheight" : 242438,
        "banscore" : 0,
        "syncnode" : true
    },
    {
        "addr" : "xxx:56947",
        "services" : "00000001",
        "lastsend" : 1371730758,
        "lastrecv" : 1371730764,
        "bytessent" : 267688,
        "bytesrecv" : 5629,
        "conntime" : 1371730635,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.0/",
        "inbound" : true,
        "startingheight" : 242438,
        "banscore" : 0
    },
    {
        "addr" : "xxx:1217",
        "services" : "00000001",
        "lastsend" : 1371730754,
        "lastrecv" : 1371730765,
        "bytessent" : 268438,
        "bytesrecv" : 3439,
        "conntime" : 1371730635,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.2/",
        "inbound" : true,
        "startingheight" : 242438,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371730754,
        "lastrecv" : 1371730758,
        "bytessent" : 6273,
        "bytesrecv" : 81803,
        "conntime" : 1371730635,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : false,
        "startingheight" : 242438,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371730763,
        "lastrecv" : 1371730763,
        "bytessent" : 15118,
        "bytesrecv" : 2092,
        "conntime" : 1371730635,
        "version" : 70001,
        "subver" : "/Mini-Grokking Satoshi:0.8.1.99/",
        "inbound" : false,
        "startingheight" : 242438,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371730754,
        "lastrecv" : 1371730740,
        "bytessent" : 5044,
        "bytesrecv" : 78106,
        "conntime" : 1371730635,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1.99/Eligius:2/",
        "inbound" : false,
        "startingheight" : 242438,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371730754,
        "lastrecv" : 1371730758,
        "bytessent" : 6572,
        "bytesrecv" : 77576,
        "conntime" : 1371730635,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.2.2/",
        "inbound" : false,
        "startingheight" : 242438,
        "banscore" : 0
    },
    {
        "addr" : "xxx:3505",
        "services" : "00000001",
        "lastsend" : 1371730767,
        "lastrecv" : 1371730760,
        "bytessent" : 13978337,
        "bytesrecv" : 1379,
        "conntime" : 1371730637,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.2.2/",
        "inbound" : true,
        "startingheight" : 114994,
        "banscore" : 0
    },
    {
        "addr" : "xxx:41332",
        "services" : "00000001",
        "lastsend" : 1371730754,
        "lastrecv" : 1371730752,
        "bytessent" : 270796,
        "bytesrecv" : 73563150,
        "conntime" : 1371730642,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.2.1/",
        "inbound" : true,
        "startingheight" : 242438,
        "banscore" : 0
    },
    {
        "addr" : "xxx:33972",
        "services" : "00000001",
        "lastsend" : 1371730762,
        "lastrecv" : 1371730762,
        "bytessent" : 274862,
        "bytesrecv" : 227,
        "conntime" : 1371730760,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : true,
        "startingheight" : 242438,
        "banscore" : 0
    }

----

It looks like it was causing me to send out a good amount of BS data too... hmm, well, I guess you can't really notice it here.  The guy above was just requesting blocks.  But in a prior log, I had sent out some godawful amt of bytes to a bunch of peers (a bitcoind instance that had been running for maybe 5 minutes)

[r3wt]

yeah, someone was downloading the blockchain. it was probably me. i noticed when i was dling yesterday i kept dropping in number of connections and would have to restart. you have to realize this is p2p software. the blockchain is downloaded from the first block source the client finds when it launches for the first time on a new computer.

[BenTuras]

Looks to me your "friend" at "addr" : "xxx:3505" is missing about a 130,000 blocks ! (242438 - 114994). So I wouldn't call the 14Mb of data that you send to him BS. You're sending him the blockchain data from last year(s) ;-)

*grins* the perpetrator just revealed himself ;-)

[r3wt]

Looks to me your "friend" at "addr" : "xxx:3505" is missing about a 130,000 blocks ! (242438 - 114994). So I wouldn't call the 14Mb of data that you send to him BS. You're sending him the blockchain data from last year(s) ;-)

*grins* the perpetrator just revealed himself ;-)

yeah, it might have been me actually. that sounds about right as far as the blockheight when i started.

[zvs]

        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371735350,
        "lastrecv" : 1371735347,
        "bytessent" : 34484995,
        "bytesrecv" : 1582697,
        "conntime" : 1371735153,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.2.2/",
        "inbound" : false,
        "startingheight" : 242445,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371735350,
        "lastrecv" : 1371735342,
        "bytessent" : 13728018,
        "bytesrecv" : 1584120,
        "conntime" : 1371735153,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : false,
        "startingheight" : 242445,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371735350,
        "lastrecv" : 1371735349,
        "bytessent" : 7512448,
        "bytesrecv" : 1580997,
        "conntime" : 1371735154,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : false,
        "startingheight" : 242445,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371735349,
        "lastrecv" : 1371735349,
        "bytessent" : 88559980,
        "bytesrecv" : 81707,
        "conntime" : 1371735154,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : false,
        "startingheight" : 242445,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371735349,
        "lastrecv" : 1371735342,
        "bytessent" : 23535,
        "bytesrecv" : 1203573,
        "conntime" : 1371735154,
        "version" : 60001,
        "subver" : "/Satoshi:0.6.3/",
        "inbound" : false,
        "startingheight" : 242445,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371735349,
        "lastrecv" : 1371735349,
        "bytessent" : 52893,
        "bytesrecv" : 82593833,
        "conntime" : 1371735154,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : false,
        "startingheight" : 242445,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371735349,
        "lastrecv" : 1371735349,
        "bytessent" : 91552895,
        "bytesrecv" : 81040,
        "conntime" : 1371735154,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.2.2/",
        "inbound" : false,
        "startingheight" : 242445,
        "banscore" : 0
    },
    {
        "addr" : "xxx:8333",
        "services" : "00000001",
        "lastsend" : 1371735350,
        "lastrecv" : 1371735349,
        "bytessent" : 84017011,
        "bytesrecv" : 6589163,
        "conntime" : 1371735154,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : false,
        "startingheight" : 242445,
        "banscore" : 0
    },

closed off bitcoind at 3GB memory this time, about *more like 180 seconds in

so, any solutions?  besides making light of some horrible bug?

[BenTuras]

Are you running on Windows or Linux or Mac OSX ?
For Windows, it would help if you could show the information of the Task Manager for the running bitcoind program.
For Linux, the information of (the program) top for the running bitcoind.
For Mac OSX, the information Activity Monitor is showing.

This could be a start to try to explain what's going on.

A develop will need to run bitcoind with a profiler active to see if it is caused by a memory leak when the memory usage goes above 1Gb.

[zvs]

nogleg.com and nogleg.net are both running on ubuntu

here it is on windows 7 ultimate edition, 64 bit

it'd probably be more, except it was saturating my upstream.

and, yes, my blockchain is up to date

[zvs]

If I use my usual 750'ish connections, it goes above 1GB memory in about 10 seconds.  Reaches 10GB in a few minutes.

I'm not accepting all these new horse battery staple transactions, either

Windows client is standard client with 8 outgoing connections.  Started with entire blockchain

[zvs]

changed this in net.h:

inline unsigned int ReceiveFloodSize() { return 1000*GetArg("-maxreceivebuffer", 1*50); }
inline unsigned int SendBufferSize() { return 1000*GetArg("-maxsendbuffer", 1*100); }

for it to take longer to use up all my memory, also this:

int64 CTransaction::nMinRelayTxFee = 1000000

yet,

        "addr" : "xxxxx:60331",
        "services" : "00000001",
        "lastsend" : 1371737909,
        "lastrecv" : 1371737918,
        "bytessent" : 525043,
        "bytesrecv" : 40556302,
        "conntime" : 1371737637,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.2.2/",
        "inbound" : true,
        "startingheight" : 242446,
        "banscore" : 0
  
how is that even possible?

on the plus side, i think it was the stored transactions that fall within the 0.0001 new minrelayfee that were causing the memory usage to get to >10GB within a few minutes, since i'm holding steady at 500MB now with 500 connections

[zvs]

I guess most of the clowns up above never upgraded past the 0.0005 versions.

this is lol:

debug.log:2013-06-20 14:42:31 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:31 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:31 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:32 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:33 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:35 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:36 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:36 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:37 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:38 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:40 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:40 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:43 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:43 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:43 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:44 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:44 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:44 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:45 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:45 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:45 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:45 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:46 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:46 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:47 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:47 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:48 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:48 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:48 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:49 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:50 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:51 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:51 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:52 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:53 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:53 socket no message in first 30 seconds, 0 1
debug.log:2013-06-20 14:42:54 socket no message in first 30 seconds, 0 1

it took me about 10x longer than normal to get connected to 500 nodes

        "addr" : "xxxx:8333",
        "services" : "00000001",
        "lastsend" : 1371739889,
        "lastrecv" : 1371739889,
        "bytessent" : 10997,
        "bytesrecv" : 12191999,
        "conntime" : 1371739017,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : false,
        "startingheight" : 242447,
        "banscore" : 0

(this one is mr fancy pants)

        "addr" : "xxxx:8333",
        "services" : "00000001",
        "lastsend" : 1371739882,
        "lastrecv" : 1371739884,
        "bytessent" : 13150,
        "bytesrecv" : 115633854,
        "conntime" : 1371739018,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.1/",
        "inbound" : false,
        "startingheight" : 242447,
        "banscore" : 0

        "addr" : "xxxx:8333",
        "services" : "00000001",
        "lastsend" : 1371739888,
        "lastrecv" : 1371739889,
        "bytessent" : 6929,
        "bytesrecv" : 42675157,
        "conntime" : 1371739012,
        "version" : 70001,
        "subver" : "/Satoshi:0.8.0/",
        "inbound" : false,
        "startingheight" : 242447,
        "banscore" : 0

moral of the story: downgrade versions, or modify source yourself

doh, fixed an IP

[zvs]

so, is this fixed in the latest version?

it wasn't very nice to get 10GB used up in 5 minutes, some modifications would be good to remove

that  wasnt a 'small memory leak',   it was some huge 'memory storage' exploit

[zvs]

I had to change the nonstandard transaction to a 1 DoS amount, as someone was sending me thousands of nonstandard transactions, raising system load to >5