John Tobey (OP)
|
|
July 01, 2013, 05:48:44 PM |
|
I'll probably just pick one of those chains. I didn't suspect trying to get 30 chains loaded in Abe would require you to fix a bunch of stuff Your reports are helping me improve the code. Thank you!!
|
|
|
|
Jouke
|
|
July 02, 2013, 07:50:46 AM Last edit: July 02, 2013, 01:23:07 PM by Jouke |
|
Run this SQL and see if it is fixed: UPDATE datadir SET chain_id = 1, loader = 'rpc' WHERE datadir_id = 1;
In the datadir table I have the following columns: datadir_id dirname blkfile_number blkfile_number blkfile_offset chain_id datadir_loader Whoops, I guess I meant: UPDATE datadir SET chain_id = 1, datadir_loader = 'rpc' WHERE datadir_id = 1;How does Abe keep track of blockfiles and blockfile offsets when it is updating from RPC?
Putting chain_id and datadir_loader in the table was bad design. I think they get their values from the "chain" and "loader" (or "default-loader") config elements when the datadir is first used, and thereafter, the config values are ignored. I plan to make it use the config values only, but meanwhile, I think you can copy the correct values into the table row, and it will work. Thanks. I see it is adding mempool transactions now. But I did understand correctly that it also updates blocks right? It does. Ah, I can look up mempool transactions, but it does not add the addresses in that transaction to the database? I was hoping I could do a /unspent/ on mempool transactions as well.
|
Koop en verkoop snel en veilig bitcoins via iDeal op Bitonic.nl
|
|
|
John Tobey (OP)
|
|
July 02, 2013, 03:14:22 PM |
|
Ah, I can look up mempool transactions, but it does not add the addresses in that transaction to the database? I was hoping I could do a /unspent/ on mempool transactions as well.
This is a little tricky to do. Abe does not delete transactions that can never get into the main chain because they contain inputs that are spent elsewhere. /unspent/ excludes these from its output by filtering on block_tx and chain_candidate.in_longest = 1. This also excludes mempool transactions. Can you see them on the address page?
|
|
|
|
|
Jouke
|
|
July 02, 2013, 09:20:18 PM |
|
Ah, I can look up mempool transactions, but it does not add the addresses in that transaction to the database? I was hoping I could do a /unspent/ on mempool transactions as well.
This is a little tricky to do. Abe does not delete transactions that can never get into the main chain because they contain inputs that are spent elsewhere. /unspent/ excludes these from its output by filtering on block_tx and chain_candidate.in_longest = 1. This also excludes mempool transactions. Can you see them on the address page? I'll find some time to play with the code.
|
Koop en verkoop snel en veilig bitcoins via iDeal op Bitonic.nl
|
|
|
robanswe
Newbie
Offline
Activity: 24
Merit: 0
|
|
July 02, 2013, 09:27:56 PM |
|
Hello, I really appreciate the work you are putting into this. I currently only got one "problem" with it. It takes forever to do the "initial data load". I'm trying to load the bitcoin blockchain into a mysql db and I'm currently at block 176100 and it takes around 5seconds/block is this normal or I'm I doing something wrong?
|
|
|
|
crendore
|
|
July 02, 2013, 10:38:23 PM |
|
Hello, I really appreciate the work you are putting into this. I currently only got one "problem" with it. It takes forever to do the "initial data load". I'm trying to load the bitcoin blockchain into a mysql db and I'm currently at block 176100 and it takes around 5seconds/block is this normal or I'm I doing something wrong?
It takes a while, depending on your system, but that sounds about right. i think it took me almost 2 days to get it running the first time. What might make sense though, is having the database it is creating hosted seperately, like on mega.co.nz or something, so that people could just download the database directly instead of creating it everytime. Obviously any database dump would be a little bit outdated, but then you could run abe and update from that point.
|
|
|
|
K1773R
Legendary
Offline
Activity: 1792
Merit: 1008
/dev/null
|
|
July 02, 2013, 11:04:26 PM |
|
Hello, I really appreciate the work you are putting into this. I currently only got one "problem" with it. It takes forever to do the "initial data load". I'm trying to load the bitcoin blockchain into a mysql db and I'm currently at block 176100 and it takes around 5seconds/block is this normal or I'm I doing something wrong?
db @ tmpfs and its extreme fast
|
[GPG Public Key]BTC/DVC/TRC/FRC: 1 K1773RbXRZVRQSSXe9N6N2MUFERvrdu6y ANC/XPM A K1773RTmRKtvbKBCrUu95UQg5iegrqyeA NMC: N K1773Rzv8b4ugmCgX789PbjewA9fL9Dy1 LTC: L Ki773RBuPepQH8E6Zb1ponoCvgbU7hHmd EMC: E K1773RxUes1HX1YAGMZ1xVYBBRUCqfDoF BQC: b K1773R1APJz4yTgRkmdKQhjhiMyQpJgfN
|
|
|
John Tobey (OP)
|
|
July 03, 2013, 01:57:32 AM |
|
Thanks for your awesome contribution to the open source community. I really appreciate the work you've done!
Happy to hear it. Actually, this ought to be very simple. I'd do it right now, but I am not at my secure workstation. Where q_getreceivedbyaddress calls abe.store.get_received, it could pass the optional block_height parameter to ignore blocks after a certain number. That number would be something like: abe.store.get_block_number(chain['id']) + 1 - N
|
|
|
|
bbxx
|
|
July 05, 2013, 08:53:54 AM |
|
i have some problems with bitcoin blockchain database size is huge! i am at block 228k and already it took 31.6 GB! 142,140,449 records inserted so far. my cpu i5 3570, 4gb ram, 50gb assigned for vdi. ubuntu server, mysql, 1gb left on ssd i need only one information, balance on given address. can i clear some of tables to reduce mysql database size ? should i use postgresql or maybe there is a fork with reduced db size needed ? what i can do to reduce resource usage ?
|
|
|
|
Jouke
|
|
July 05, 2013, 09:05:04 AM |
|
i have some problems with bitcoin blockchain database size is huge! i am at block 228k and already it took 31.6 GB! 142,140,449 records inserted so far. my cpu i5 3570, 4gb ram, 50gb assigned for vdi. ubuntu server, mysql, 1gb left on ssd i need only one information, balance on given address. can i clear some of tables to reduce mysql database size ? should i use postgresql or maybe there is a fork with reduced db size needed ? what i can do to reduce resource usage ? https://bitcointalk.org/index.php?topic=22785.msg1554489;topicseen#msg1554489
|
Koop en verkoop snel en veilig bitcoins via iDeal op Bitonic.nl
|
|
|
mapleshadow
Member
Offline
Activity: 79
Merit: 10
|
|
July 08, 2013, 05:17:00 PM |
|
my conf is wdc wdc.conf: dbtype sqlite3 connect-args abe.sqlite datadir = [{ "dirname": "/home/mapleshadow/.worldcoin", "chain": "Worldcoin", "code3": "WDC", "address_version": "\u0049" }] int-type str upgrade port 2750 host 12.16.2.24 DataStore.py: {"chain":"Worldcoin", "code3":"WDC", "address_version":"\x49", "magic":""}, ERROR-1: failed to load /home/maple/.worldcoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/maple/.worldcoin/bitcoin.conf' catch_up_rpc: abort Opened /home/mapleshadow/.worldcoin/blocks/blk00000.dat Abe initialized. Listening on http://12.16.2.24:2750WHY?
|
|
|
|
John Tobey (OP)
|
|
July 08, 2013, 06:02:30 PM |
|
my conf is wdc wdc.conf: dbtype sqlite3 connect-args abe.sqlite datadir = [{ "dirname": "/home/mapleshadow/.worldcoin", "chain": "Worldcoin", "code3": "WDC", "address_version": "\u0049" }] int-type str upgrade port 2750 host 12.16.2.24 DataStore.py: {"chain":"Worldcoin", "code3":"WDC", "address_version":"\x49", "magic":""}, ERROR-1: failed to load /home/maple/.worldcoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/maple/.worldcoin/bitcoin.conf' catch_up_rpc: abort Opened /home/mapleshadow/.worldcoin/blocks/blk00000.dat Abe initialized. Listening on http://12.16.2.24:2750WHY? Abe tries to load data over RPC, and when that fails, it reads block files. (See default-loader in abe.conf to choose one or the other.) Abe does not know about Worldcoin, so it looks for RPC credentials in bitcoin.conf. There is no bitcoin.conf, so it logs an error and tries to load blocks from files, starting with blocks/blk00000.dat. It did not find any blocks in that file, so perhaps the file is empty or the block format is not the same as Bitcoin's. I am surprised that the configured dirname is in /home/mapleshadow but Abe looked for bitcoin.conf in /home/maple.
|
|
|
|
mapleshadow
Member
Offline
Activity: 79
Merit: 10
|
|
July 09, 2013, 02:04:50 AM |
|
Sorry, I was wrong Paths are maple
|
|
|
|
mapleshadow
Member
Offline
Activity: 79
Merit: 10
|
|
July 09, 2013, 02:09:42 AM |
|
That I can build wdc of abe services? my conf is wdc wdc.conf: dbtype sqlite3 connect-args abe.sqlite datadir = [{ "dirname": "/home/maple/.worldcoin", "chain": "Worldcoin", "code3": "WDC", "address_version": "\u0049" }] int-type str upgrade port 2750 host 12.16.2.24 DataStore.py: {"chain":"Worldcoin", "code3":"WDC", "address_version":"\x49", "magic":""}, ERROR-1: failed to load /home/maple/.worldcoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/maple/.worldcoin/bitcoin.conf' catch_up_rpc: abort Opened /home/maple/.worldcoin/blocks/blk00000.dat Abe initialized. Listening on http://12.16.2.24:2750WHY? Abe tries to load data over RPC, and when that fails, it reads block files. (See default-loader in abe.conf to choose one or the other.) Abe does not know about Worldcoin, so it looks for RPC credentials in bitcoin.conf. There is no bitcoin.conf, so it logs an error and tries to load blocks from files, starting with blocks/blk00000.dat. It did not find any blocks in that file, so perhaps the file is empty or the block format is not the same as Bitcoin's. I am surprised that the configured dirname is in /home/mapleshadow but Abe looked for bitcoin.conf in /home/maple.
|
|
|
|
John Tobey (OP)
|
|
July 09, 2013, 04:10:23 PM |
|
That I can build wdc of abe services?
I don't know. Can you get Abe working with BTC or NMC or LTC? Those are known to work. I need a little more information to understand your problem. What is your python command line? How big is /home/mapleshadow/.worldcoin/blocks/blk00000.dat ? What exact version of Abe do you have? (Post the output of "git rev-parse HEAD" and "git --no-pager diff")
|
|
|
|
mapleshadow
Member
Offline
Activity: 79
Merit: 10
|
|
July 09, 2013, 05:26:06 PM |
|
Yes Operating in BTC state, there is no problem I do not know LTC, Not tested home/mapleshadow/.worldcoin/blocks/blk00000.dat have 500M Size ABE version is GITHUB the MASTER -------- My configuration as described above, for the WDC, then it is an error
|
|
|
|
mapleshadow
Member
Offline
Activity: 79
Merit: 10
|
|
July 09, 2013, 05:29:19 PM |
|
Also blk00000.dat file, for WDC, it is not present, the default is blk0001.dat and blkindex.dat files. Currently reading blk00000.dat file that I copied blk0001.dat file, rename came.
|
|
|
|
madmax_ger
|
|
July 13, 2013, 04:21:56 PM |
|
hi guys, I have some trouble with installing ABE on my macbook. everything went fine so far, I.. * installed mysql 32bit, it's running, path configured * installed python 2.7, it's running * installed python-mysql in 32bit arch (i386) mode, seems to be running * installed ABE, no problem so far then when I want to run ABE... $ ARCHFLAGS="-arch i386" python -m Abe.abe --config /usr/local/mysql/abe-my.cnf --commit-bytes 100000 --no-serve this comes up: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/Library/Python/2.7/site-packages/Abe/abe.py", line 2026, in <module> sys.exit(main(sys.argv[1:])) File "/Library/Python/2.7/site-packages/Abe/abe.py", line 2020, in main store = make_store(args) File "/Library/Python/2.7/site-packages/Abe/abe.py", line 115, in make_store store = DataStore.new(args) File "/Library/Python/2.7/site-packages/Abe/DataStore.py", line 2843, in new return DataStore(args) File "/Library/Python/2.7/site-packages/Abe/DataStore.py", line 139, in __init__ store.module = __import__(args.dbtype) File "build/bdist.macosx-10.7-intel/egg/MySQLdb/__init__.py", line 19, in <module> File "build/bdist.macosx-10.7-intel/egg/_mysql.py", line 7, in <module> File "build/bdist.macosx-10.7-intel/egg/_mysql.py", line 6, in __bootstrap__ ImportError: dlopen(/Users/thisuser/.python-eggs/MySQL_python-1.2.4-py2.7-macosx-10.7-intel.egg-tmp/_mysql.so, 2): no suitable image found. Did find: /Users/thisuser/.python-eggs/MySQL_python-1.2.4-py2.7-macosx-10.7-intel.egg-tmp/_mysql.so: mach-o, but wrong architecture I ran "file <filename>" for mysql (bin file) and the given .so file above, it's both i386. Can anyone help me out?
|
|
|
|
John Tobey (OP)
|
|
July 14, 2013, 12:08:18 PM |
|
ImportError: dlopen(/Users/thisuser/.python-eggs/MySQL_python-1.2.4-py2.7-macosx-10.7-intel.egg-tmp/_mysql.so, 2): no suitable image found. Did find: /Users/thisuser/.python-eggs/MySQL_python-1.2.4-py2.7-macosx-10.7-intel.egg-tmp/_mysql.so: mach-o, but wrong architecture Sorry, this is an issue with Python modules on Mac, and I have no relevant experience. Put this in a file and run it with "python FILE.py". It will have to complete without error before you can use Abe:
|
|
|
|
|