|
Title: [Closed] 0.04 BTC to get all google results Post by: graphicalx on June 10, 2016, 04:55:00 PM Hi, for a particular query, Google says it has 8000 results, but I'm unable to get more than 1000 results through a scraper.
Its basically a list of all pages on a subdomain, for example xyz.abc.com/result1, /result2 and so on. I'm using Google in this way: Quote site:xyz.abc.com You're welcome to use any other way to get all the pages on xyz.abc.com Go to this link to get the google query: First person to get me the 5000+ unique links gets the 0.04, you can first send me the 5000+ to reserve your prize and work further on 7000+ for the bonus. I just need the urls, i.e. xyz.abc.com/result1, xyz.abc.com/result2, and so on Thanks, let me know if you have any questions NOTE: Using a regular scraper will get you around 500 links, using some tricks will get you 800~, but i need 5000 or more to award the 0.04 Title: Re: 0.03 to get all google results Post by: jambola2 on June 10, 2016, 05:17:11 PM You can do it legally/legitimately here: https://developers.google.com/custom-search/json-api/v1/overview#prerequisites
It costs 5$ per 1k, so 25$ (0.043 BTC) for 5000 of them. Any scraper will need to change IP addresses frequently as google detects and blocks this activity. Title: Re: 0.03 to get all google results Post by: tounsi7orr on June 10, 2016, 05:34:08 PM I can write a smart script that scrap all search results and put them in a text or excel file. I sent you PM.
Title: Re: 0.03 to get all google results Post by: graphicalx on June 10, 2016, 09:40:47 PM Thanks, already tried the API way, they just give 50 results this way.
Replied to PM. Title: Re: 0.04 BTC to get all google results Post by: graphicalx on June 11, 2016, 06:18:46 AM Increased prize to 0.04, and gave instructions in original post including the search query so you can try without PMing me.
First person to send me 5000+ links (pref in CSV form but any will work) will get the prize Title: Re: 0.04 BTC to get all google results Post by: bitcoinblog on June 11, 2016, 09:59:03 AM You want like this ?
Links removed on request from OP. Title: Re: 0.04 BTC to get all google results Post by: Angell on June 11, 2016, 11:25:22 AM The only way to do this is hack the server that holds all links.
Title: Re: 0.04 BTC to get all google results Post by: graphicalx on June 11, 2016, 11:30:57 AM I haven't lost hope yet :) i got almost double the results by going each year separately. I'll try some other ideas later today
Title: Re: 0.04 BTC to get all google results Post by: Angell on June 11, 2016, 11:49:37 AM I haven't lost hope yet :) i got almost double the results by going each year separately. I'll try some other ideas later today I have a hacker friend. I'll meet with him today so maybe he has a better idea. I'll contact you back when I get his opinion. Title: Re: 0.04 BTC to get all google results Post by: graphicalx on June 11, 2016, 12:07:55 PM I'm all for tricks etc, but no hacking please!
Title: Re: [Closed] 0.04 BTC to get all google results Post by: graphicalx on June 15, 2016, 09:51:13 AM Thanks everyone. hermesesus was able to get closest, able to get 1327 unique links.
Thanks again! |