Bitcoin Forum
October 31, 2024, 11:15:27 PM *
News: Latest Bitcoin Core release: 28.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: [Closed] 0.04 BTC to get all google results  (Read 767 times)
graphicalx (OP)
Member
**
Offline Offline

Activity: 87
Merit: 10

Need logo design, banner design? PM me!


View Profile WWW
June 10, 2016, 04:55:00 PM
Last edit: June 15, 2016, 09:49:28 AM by graphicalx
 #1

Hi, for a particular query, Google says it has 8000 results, but I'm unable to get more than 1000 results through a scraper.
Its basically a list of all pages on a subdomain, for example xyz.abc.com/result1, /result2 and so on. I'm using Google in this way:
Quote
site:xyz.abc.com

You're welcome to use any other way to get all the pages on xyz.abc.com

0.03 0.04 BTC if you manage to get 5000 results or more, and a bonus if you can get 7000 or more

PM me if you think you can do this
Go to this link to get the google query: link removed
First person to get me the 5000+ unique links gets the 0.04, you can first send me the 5000+ to reserve your prize and work further on 7000+ for the bonus.

I just need the urls, i.e. xyz.abc.com/result1, xyz.abc.com/result2, and so on

Thanks, let me know if you have any questions

NOTE: Using a regular scraper will get you around 500 links, using some tricks will get you 800~, but i need 5000 or more to award the 0.04

fund my procrastination: 1Hf1XG5cBnuFTxBVXUfUnK5hWLSBENqwnM [QR]
jambola2
Legendary
*
Offline Offline

Activity: 1120
Merit: 1038


View Profile
June 10, 2016, 05:17:11 PM
 #2

You can do it legally/legitimately here: https://developers.google.com/custom-search/json-api/v1/overview#prerequisites

It costs 5$ per 1k, so 25$ (0.043 BTC) for 5000 of them.

Any scraper will need to change IP addresses frequently as google detects and blocks this activity.

No longer active on bitcointalk, however, you can still reach me via PMs if needed.
tounsi7orr
Newbie
*
Offline Offline

Activity: 46
Merit: 0


View Profile
June 10, 2016, 05:34:08 PM
 #3

I can write a smart script that scrap all search results and put them in a text or excel file. I sent you PM.
graphicalx (OP)
Member
**
Offline Offline

Activity: 87
Merit: 10

Need logo design, banner design? PM me!


View Profile WWW
June 10, 2016, 09:40:47 PM
 #4

Thanks, already tried the API way, they just give 50 results this way.

Replied to PM.

fund my procrastination: 1Hf1XG5cBnuFTxBVXUfUnK5hWLSBENqwnM [QR]
graphicalx (OP)
Member
**
Offline Offline

Activity: 87
Merit: 10

Need logo design, banner design? PM me!


View Profile WWW
June 11, 2016, 06:18:46 AM
 #5

Increased prize to 0.04, and gave instructions in original post including the search query so you can try without PMing me.

First person to send me 5000+ links (pref in CSV form but any will work) will get the prize

fund my procrastination: 1Hf1XG5cBnuFTxBVXUfUnK5hWLSBENqwnM [QR]
bitcoinblog
Sr. Member
****
Offline Offline

Activity: 501
Merit: 252


View Profile
June 11, 2016, 09:59:03 AM
Last edit: June 11, 2016, 10:49:31 AM by bitcoinblog
 #6

You want like this ?

Links removed on request from OP.

** FREE SPACE **
Angell
Hero Member
*****
Offline Offline

Activity: 565
Merit: 503



View Profile
June 11, 2016, 11:25:22 AM
 #7

The only way to do this is hack the server that holds all links.

graphicalx (OP)
Member
**
Offline Offline

Activity: 87
Merit: 10

Need logo design, banner design? PM me!


View Profile WWW
June 11, 2016, 11:30:57 AM
 #8

I haven't lost hope yet Smiley i got almost double the results by going each year separately. I'll try some other ideas later today

fund my procrastination: 1Hf1XG5cBnuFTxBVXUfUnK5hWLSBENqwnM [QR]
Angell
Hero Member
*****
Offline Offline

Activity: 565
Merit: 503



View Profile
June 11, 2016, 11:49:37 AM
 #9

I haven't lost hope yet Smiley i got almost double the results by going each year separately. I'll try some other ideas later today

I have a hacker friend. I'll meet with him today so maybe he has a better idea. I'll contact you back when I get his opinion.

graphicalx (OP)
Member
**
Offline Offline

Activity: 87
Merit: 10

Need logo design, banner design? PM me!


View Profile WWW
June 11, 2016, 12:07:55 PM
 #10

I'm all for tricks etc, but no hacking please!

fund my procrastination: 1Hf1XG5cBnuFTxBVXUfUnK5hWLSBENqwnM [QR]
graphicalx (OP)
Member
**
Offline Offline

Activity: 87
Merit: 10

Need logo design, banner design? PM me!


View Profile WWW
June 15, 2016, 09:51:13 AM
 #11

Thanks everyone. hermesesus was able to get closest, able to get 1327 unique links.

Thanks again!

fund my procrastination: 1Hf1XG5cBnuFTxBVXUfUnK5hWLSBENqwnM [QR]
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!