New Indicator: Number of sites accepting bitcoin

[enmaku]

Seems like you could just yank the HTML of the wiki page and count <li> tags, keep historical records and whip up some charts. I have zero time, but any old PHP wizard should be capable of that, right?

[notme]

Seems like you could just yank the HTML of the wiki page and count <li> tags, keep historical records and whip up some charts. I have zero time, but any old PHP wizard should be capable of that, right?

That will get you the table of contents as well as a few other things you don't want.

Counting <a> tags with class "external" should get you pretty close.  So, yeah, in many languages this could be done in 5 lines.  You'd miss out on some of the restaurants that don't have a link, but it wouldn't be too hard to special-case those.

[enmaku]

Seems like you could just yank the HTML of the wiki page and count <li> tags, keep historical records and whip up some charts. I have zero time, but any old PHP wizard should be capable of that, right?

That will get you the table of contents as well as a few other things you don't want.

Counting <a> tags with class "external" should get you pretty close.  So, yeah, in many languages this could be done in 5 lines.  You'd miss out on some of the restaurants that don't have a link, but it wouldn't be too hard to special-case those.

Those things don't really change though, so it should be easy to count those and subtract them from the total. In any case, it should be easy enough to implement.

[Spekulatius]

would it be possible to group the resulting numbers in categories, like physical goods, professional services, restaurants, etc.?

[Spekulatius]

If you call a coding offer "trolling for money" and not having any takers after a few days "unsuccessful" then sure...

When it is in response to a call for volunteers, absolutely.

I did not read this

Oh, and the best thing would of course be, if someone could actually write a prog that counts the entries automatically and keeps them up to date, making me redundant

As a call for volunteers. I apologize.

I dont think you know how to do it

I don't think you're really Bill Cosby

[scribe]

This one's been going round my head a few days, but one evening and a few glasses of red wine, and I think I'm on the right track. Here's a dump from my script in its current take:

http://www.exmosis.net/btctt/stats.json

(Best viewed with some kind of JSON-prettifier in your browser.)

It runs through a set number of revisions in the Trade History, and grabs the last version for each month that it finds, then does a bunch of parsing to count the items in each category. It's pretty basic at the moment - it's not handling any special exceptions or tables in the page, for instance.

I figure some kind of CSV output might be useful too, and I'll probably get it running once a night to keep the latest timepoint up to date as well, at some point. Sticking the data into a database would make it easier to do the counting as well.

Feel free to suggest anything else, or point out inaccuracies. My spare coding time is fairly limited these days, so don't expect miracles to happen too quickly though. And if it's useful, small tips go a long way...

[vokain]

would it be possible to group the resulting numbers in categories, like physical goods, professional services, restaurants, etc.?

Can we keep a Google Docs or Wiki up on https://en.bitcoin.it/wiki/Main_Page?
That way we can edit as need be and also keep track easily

[Fiyasko]

Can we please get a list of all those sites?
Lol seriously, It's too hard to google bitcoin accepting merchants.

Im noticing alot of chatter related to people reading some form of list of the sites...
All i have is en.bitcoin.it/wiki/trade
How/where can i get a list of the 800+ sites that accept bitcoin.

Also, Fwiw, Screw 4shared, "no account? No download"

[Spekulatius]

This one's been going round my head a few days, but one evening and a few glasses of red wine, and I think I'm on the right track. Here's a dump from my script in its current take:

http://www.exmosis.net/btctt/stats.json

(Best viewed with some kind of JSON-prettifier in your browser.)

It runs through a set number of revisions in the Trade History, and grabs the last version for each month that it finds, then does a bunch of parsing to count the items in each category. It's pretty basic at the moment - it's not handling any special exceptions or tables in the page, for instance.

I figure some kind of CSV output might be useful too, and I'll probably get it running once a night to keep the latest timepoint up to date as well, at some point. Sticking the data into a database would make it easier to do the counting as well.

Feel free to suggest anything else, or point out inaccuracies. My spare coding time is fairly limited these days, so don't expect miracles to happen too quickly though. And if it's useful, small tips go a long way...

Thanks for your efforts mate!
It seems to do a pretty good job at counting already. Although the tables need to be included in a future version.
I used http://jsonviewer.stack.hu/ to look at your script from above. Is there a more appropriate one to use? Maybe some output in tables would be handy.

Can we keep a Google Docs or Wiki up on https://en.bitcoin.it/wiki/Main_Page?
That way we can edit as need be and also keep track easily Smiley

Also, Fwiw, Screw 4shared, "no account? No download"

I will consider a new hoster for my next update. It would need to facilitate .png, .ai, .ods -file hosting and downloading.
Will look into google docs as well.
Any other suggestions?

[scribe]

This one's been going round my head a few days, but one evening and a few glasses of red wine, and I think I'm on the right track. Here's a dump from my script in its current take:

http://www.exmosis.net/btctt/stats.json

(Best viewed with some kind of JSON-prettifier in your browser.)

Thanks for your efforts mate!
It seems to do a pretty good job at counting already. Although the tables need to be included in a future version.
I used http://jsonviewer.stack.hu/ to look at your script from above. Is there a more appropriate one to use? Maybe some output in tables would be handy.

Done and done - have included tabled items, and also added in CSV output for exporting into spreadsheets.

Here's data taken from the last 3000 edits, which goes back to February 2011:

CSV: http://www.exmosis.net/btctt/stats.csv
JSON: http://www.exmosis.net/btctt/stats.json

The CSV version means you can load it straight into a Google Doc using =importData(), like this:

https://docs.google.com/spreadsheet/ccc?key=0AnMuKQGBY6LhdFZsbzU4WlFSS3J6TDIyUHItdXFtWlE

(Second sheet contains a quick chart. I'll leave charting the breakdowns up to you...)

And thanks for the jsonviewer link. I usually use browser plugins, but might come in handy when I can't do that.

[Spekulatius]

Thank you very much!

Ill look into it tomorrow and will give you my feedback. The best thing would be, that I would only have to insert the values your application gives me into my graphical chart on the front page and thats it!

We'll see how we get there.

Cheers,
Spekulatius

[Spekulatius]

UPDATE! 

@ scribe:

Thank you for your nifty tool, today I compared its results with mine and they match pretty well (some differences due to differing ways to count or different due days). I would like to update my graph solely based on your tool, but in order to do that I would like to know what dates exactly it uses to set the numbers for its output, like what calendar day (and time) does it set for March 2012 for example?

Also, I like the google docs table you have provided. Unfortunately the graph looks disgusting from an aethetic point of view. Is there any way to beautify it a bit? Unfortunately, I cannot alter anything on there to make it look different, maybe you can? Would it in those regards even be possible to have an ever updating graph on the front post?

Thanks, you are my man,
Spekulatius

[tgmarks]

This a pretty tight and informative chart.  I appreciate it being put together.

[scribe]

Thanks for the update - late reply due to being on holiday here...

@ scribe:

Thank you for your nifty tool, today I compared its results with mine and they match pretty well (some differences due to differing ways to count or different due days). I would like to update my graph solely based on your tool, but in order to do that I would like to know what dates exactly it uses to set the numbers for its output, like what calendar day (and time) does it set for March 2012 for example?

Good question - because it's based on wiki edits, I used the latest edit available for each month, i.e. state of the wiki page at the end of each month. I could change this to use the first edit of the month fairly easily, if that's better - or possibly an average of the two. (I'm not keen on averaging all edits, just to avoid hammering the wiki...)

Some more info/ideas from my blogpost write-up here:

http://bitcoinlife.wordpress.com/2012/07/12/counting-bitcoin-merchants/

Also, I like the google docs table you have provided. Unfortunately the graph looks disgusting from an aethetic point of view. Is there any way to beautify it a bit? Unfortunately, I cannot alter anything on there to make it look different, maybe you can? Would it in those regards even be possible to have an ever updating graph on the front post?

Yeah, Google Docs is, uh, "basic" - to be kind. Colours are basically flat, Google-style. Highcharts is much nicer in terms of output - I'll see if I can change the JSON output to feed out as Highcharts format, but not sure how easy it is to then save that as an image somewhere.

Would also be good to save stats into a database rather than pulling from the wiki history each time (for historical data) - and maybe do a version for the German list...

Edit: Highcharts output wasn't too tricky. Still need to play with the look, but a first draft version exists here, with "export as image" button included in top-right corner:

http://www.exmosis.net/btctt/highchart.php

Not sure why older categories don't show up, so there's a bit of a mismatch before the start of 2012. Also managed to get it to read the script's previous JSON output, so I don't have to scrape the whole history each time the script runs now. (Easier than handling a database...) I'll set it up at some point to run daily, at which point the highcharts page above should at least update the latest figure whenever changes to the wiki page are made.

[Spekulatius]

Update  

This time some nifty merchant reviews included!!


@ Scribe:

I tried to do it with your charts this time, but unfortunately they dont seem to update and dont show the August stats atm. Would that change automatically at the end of the month?

Do you think you could get this highcharts implementation done?

Cheers,
Spekulatius

[scribe]

@ Scribe:

I tried to do it with your charts this time, but unfortunately they dont seem to update and dont show the August stats atm. Would that change automatically at the end of the month?

Do you think you could get this highcharts implementation done?

Finally got round to setting this up to run daily - http://www.exmosis.net/btctt/stats.csv should update the latest month's figure when any changes are made.

Not got much time to look at highcharts for a few weeks. The current page has an "export image" button top-right, but can't think of a way to do this automatically. Also for some reason the chart isn't displaying properly in my browser until you turn some categories off and on.

Keep up the analysis

[sethsethseth]

With BitPay recently accepting its 1000th merchant, how can this chart show less than that for the entire community?

[inbox]

.

[kjlimo]

With BitPay recently accepting its 1000th merchant, how can this chart show less than that for the entire community?

Not every merchant is listed on the wiki databases these charts are based off of.

The fact that they are close seems reasonable given how prominent and convenient Bitpay services are (were?).

However, I doubt Bitpay has the entire bitcoin community processing payments...

I'm guessing that puts the total number of businesses somewhere around 1,500, but monitoring these other databases and lists are still helpful from a relative month to month point of view to show the rate of growth.

So far, it's been slow & steady with similar number of businesses getting listed each month. 

We'll see when it switches to exponential (2013?)!  Make sure to buy your bitcoins before that happens!

[notme]

With BitPay recently accepting its 1000th merchant, how can this chart show less than that for the entire community?

Not every merchant is listed on the wiki databases these charts are based off of.

The fact that they are close seems reasonable given how prominent and convenient Bitpay services are (were?).

However, I doubt Bitpay has the entire bitcoin community processing payments...

I'm guessing that puts the total number of businesses somewhere around 1,500, but monitoring these other databases and lists are still helpful from a relative month to month point of view to show the rate of growth.

So far, it's been slow & steady with similar number of businesses getting listed each month. 

We'll see when it switches to exponential (2013?)!  Make sure to buy your bitcoins before that happens!

All that, plus I'm a bit pay merchant.  Click the button here to buy me a dollar : http://www.yrral.net/pay_me.html

*note bit-pay.com seems to be down at the moment, so if you really want to buy me dollars you'll have to wait .