Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: izitors on May 31, 2019, 05:13:35 PM



Title: Obtaining all transactions until 2013
Post by: izitors on May 31, 2019, 05:13:35 PM
Tell me where I can download the Blockchain Transaction ID until 2013? That is, I want to take only part of the information about the blockchain from 2009 to 2013?


Title: Re: Obtaining all transactions until 2013
Post by: achow101 on May 31, 2019, 07:42:42 PM
Transaction IDs are not stored in the blockchain. There are probably APIs that you can use. Otherwise you will need to download the blockchain and hash every transaction to get its ID. The blockchain between 2009 and 2013 is pretty small so it won't take a lot of time or disk space to download that part.


Title: Re: Obtaining all transactions until 2013
Post by: Thirdspace on May 31, 2019, 11:27:24 PM
try contacting TheArchaeologist (https://bitcointalk.org/index.php?action=profile;u=2307759), he may have saved a copy of all txids while conducting his experiments (https://bitcointalk.org/index.php?topic=2488493.msg44199873#msg44199873)
he was working with blockhashes, merkle roots and also doing sha256 on public addresses and all other stuff
can you tell us what you are going to do with all those transaction ids? similar to what TheArchaeologist has done?


Title: Re: Obtaining all transactions until 2013
Post by: DaCryptoRaccoon on May 31, 2019, 11:40:26 PM
If you start to download the blockchain you can use something like bitcoin abe to parse the chain into a database.

I have started on a small guide on how to set this up.

Code:
				    SETUP

 :Required:

Ubuntu, MySQL Python, MySQL Client, MySQL Server, Python, Python2, Blockchain,

Downloading the bitcoin blockchain two options.  

1. Download & Install bitcoin core wallet and sync with network


################################################################################

Install Python MySQL


$ sudo apt-get install python-mysqldb

   Install MySQL Client & Server

$ sudo apt-get install mysql-client mysql-server

To configure the MySQL instance with InnoDB engine support.  
If you installed with Debian/Ubuntu then InnoDB is enabled by default.  
To check for InnoDB support, issue "SHOW ENGINES" and look in the output
for "InnoDB" with "YES" next to it.  If "skip-innodb" appears in the server
configuration (my.cnf or my.ini) then remove it and restart the server.

################################################################################

SETUP MYSQL

Log into MySQL as root (i.e: mysql -u root) and give commands.
Don't forget to change the PASSWORD

    create database abe;
    CREATE USER 'abe'@'localhost' IDENTIFIED BY 'YOUR_PASSWORD_HERE';
    grant all on abe.* to abe;

The above will

1.  Create a database called abe
2.  Create a user called abe for localhost with the password you enter above
3.  Will grant all permisions to abe for abe user


################################################################################

DOWNLOAD AND INSTALL BITCOIN-ABE

This will allow you to check getrecievedbyaddress calls quickly without being rate limited by online API checkers.

$ git clone https://github.com/bitcoin-abe/bitcoin-abe.git
cd bitcoin-abe
python setup.py install

once installed open file abe.conf
(Included is a copy of the file with section uncommented all you need to do is edit the password and DB info if you changed it)
If You use the original config file from abe you will need to follow steps below and uncomment and modify lines.

################################################################################

abe.conf GUIDE

uncomment lines 33 & 34 and update line 34 with your password / user and db info for MySQL

# MySQL example; see also README-MYSQL.txt:
dbtype MySQLdb
connect-args {"user":"abe","db":"abe","passwd":"YOUR_PASSWORD_HERE"}

Uncomment lines 57 & 58

# Specify port and/or host to serve HTTP instead of FastCGI:
port 2750
host localhost

This will open port 2750 on localhost to allow the wallet checker to connect to the API

Next scroll down to the section where datadir is listed, you must specifiy the path to the blockchain directory this path should contain the bitcoin.conf & blocks & chainstate folders.

Uncomment Lines 136 / 137 / 139 / 140
**Do not uncomment line 138 "loader"**  as we are not caling via RPC rather via the API and http request to gettecievedbyaddress + addy

datadir += [{
        "dirname": "/path/to/blockchain",
#        "loader": "rpc",    # See the comments for default-loader below.
        "chain": "Bitcoin"
      }]

################################################################################

START THE IMPORT

You can now start to load the data to abe from the blockchain run the following command from the bitcoin-abe-master dir

$ python -m Abe.abe --config abe.conf --commit-bytes 100000 --no-serve --datadir /path/to/blockchain

You should see output like:

block_tx 1 1
     block_tx 2 2
block_tx 3 3
     block_tx 4 4
block_tx 5 5
     block_tx 6 6
block_tx 7 7
     block_tx 8 8

( THIS PROCESS WILL TAKE A VERY LONG TIME POSSIBLY 2-6 DAYS DEPENDING ON YOUR SYSTEM SPEC )

You can stop the process at any time and re-start the load will continue from the last block loaded after checking the chain.
The data that is loaded can also be used to search while data in importing but only up to the latest block number processed.

Next step can be done now if you want to scan small search space or wait for full import of data before continuing.


################################################################################

CHECK MYSQL

Go back to MySQL and log in and type

mysql> use abe;

mysql> show tables;

Output will show.

+-----------------+
| Tables_in_abe   |
+-----------------+
| abe_lock        |
| block           |
| block_next      |
| block_seq       |
| block_tx        |
| block_txin      |
| chain           |
| chain_candidate |
| chain_seq       |
| chain_summary   |
| configvar       |
| datadir         |
| datadir_seq     |
| multisig_pubkey |
| orphan_block    |
| pubkey          |
| pubkey_seq      |
| tx              |
| tx_seq          |
| txin            |
| txin_detail     |
| txin_seq        |
| txout           |
| txout_approx    |
| txout_detail    |
| txout_seq       |
| unlinked_tx     |
| unlinked_txin   |
+-----------------+



################################################################################


LAUNCH BITCOIN ABE (LOCALHOST)

If you have this working you can now launch ABE from the bitcoin-abe-master dir to view the api in localhost.

$ python -m Abe.abe --config abe.conf

Open browser and navagate to: localhost:2750  

You should now see the ABE block explorer running.


################################################################################

Quicker ways to do this but if you wanted some kind of database of transaction this would be one way to host it all locally.


Title: Re: Obtaining all transactions until 2013
Post by: pooya87 on June 01, 2019, 04:13:28 AM
you can use Blockchair.com explorer to export almost everything as long as the number of cells is less than 1 million, otherwise it may cost you.
visit: https://blockchair.com/bitcoin/transactions?s=time(asc)&q=time(..2013-12-31)# i've already set the date for you but the rest of the options won't register in the url so at the bottom of the page deselect every other option you don't want such as total output, output count,... and only have hash and  time options selected. now you should see 60432140 cells. either reduce the time to get 1 million and export manually or contact them and see if the payment works for you and get the data that way.