Google partial blackholing of Bitcointalk?

[gmaxwell]

I've noticed a large number of my old posts are no longer discoverable via google, maybe on the order of half that I search for (really informally).

Here is an example:

https://bitcointalk.org/index.php?topic=1427885.msg14601127#msg14601127

You should be able to find it by googling for substrings:

Example.

Yet you can't.  In fact, AFAICT, no substring will make google return anything on the first page of that thread.

If you google the thread title you get the second page of the thread and only the second page of the thread.

Duck-duck-go, however, returns it just fine.

I find it pretty demoralizing to have my posts vanish from search indexes, especially because last year Reddit changed so that negative-voted posts don't get search indexed and as a result thousands of posts of mine in rbtc effectively vanished from the internet.

[1714921725]

1714921725

[Foxpup]

It's not just Bitcointalk. I've noticed that Google no longer indexes (or at least no longer displays in search results) pages that haven't been updated in over 5 years, effectively causing huge portions of the web to vanish. It's one of the reasons I now use DuckDuckGo almost exclusively.

[Insanerman]

I think this is because of their recent February update with their search engine. Google recently updated their broad-core algorithm in its SEO to be improved with Passage Ranking. I've seen this article that explains it very well -- https://www.searchenginejournal.com/google-passage-ranking-martin-splitt/388206/#close

With that update, it is not only your thread that is affected, but the general results from any users who searched with exact keyword on Google. Based on my understanding with Passage Ranking, its algorithm ranks the results based on the overall content of the page independently -- which leads to your 2nd thread page be shown first than the main OP.

[Jet Cash]

It's not just Bitcointalk. I've noticed that Google no longer indexes (or at least no longer displays in search results) pages that haven't been updated in over 5 years, effectively causing huge portions of the web to vanish. It's one of the reasons I now use DuckDuckGo almost exclusively.

That's a bummer!  I thought the idea with Google was to provide an index to archives, as well as current stuff.

[NeuroticFish]

There's a discussion that started last year on Google starting to act different.
Back then they were still returning for me relevant pages. Lately I started getting basically crap. Now I know why.


I've done exact search for some areas in that old post and.. woah, even older/weaker search engines (yahoo, lycos, excite) don't return bitcointalk. WTF?!
Until know it's DuckDuckGo and Yandex returning it, and I'm not really comfortable with the thought of getting to use Yandex search.

[LTU_btc]

OP, at least I got your thread on my search results and it's shown on the top:

It's not just Bitcointalk. I've noticed that Google no longer indexes (or at least no longer displays in search results) pages that haven't been updated in over 5 years, effectively causing huge portions of the web to vanish. It's one of the reasons I now use DuckDuckGo almost exclusively.

So, if I'm going to search for post made in 2014, it's likely that I'm not going to find it?
At least I have different experience. I don't have problems to get old content in search results. It applies to Bitcointalk and other websites.
I have tried to use DuckDuckGo, but I wasn't happy with search results.

[Quickseller]

I think this is because of their recent February update with their search engine. Google recently updated their broad-core algorithm in its SEO to be improved with Passage Ranking. I've seen this article that explains it very well -- https://www.searchenginejournal.com/google-passage-ranking-martin-splitt/388206/#close

With that update, it is not only your thread that is affected, but the general results from any users who searched with exact keyword on Google. Based on my understanding with Passage Ranking, its algorithm ranks the results based on the overall content of the page independently -- which leads to your 2nd thread page be shown first than the main OP.

If I understand that blog post correctly, google has updated their algo to return results that are relevant to the query being asked, rather than key words in a query. So searching for cars with the best safety rating might return a result that talks about a safety metric and ranks cars accordingly, but might not have the term "best" nor "safety" in the content of the page. 

In the past, when searching for something, you might encounter a webpage that has many random keywords unrelated to eachother, and no actual useful content. This change seems to be making it less likely these types of pages will be returned in search results.

[LoyceV]

I've noticed that Google no longer indexes (or at least no longer displays in search results) pages that haven't been updated in over 5 years

I can imagine this will be an improvement in some cases: I often get completely outdated information when I search something^*. But if Google can't distinguish between what is and what isn't actually outdated, I'd rather get all relevant search results instead of only the most recent ones.

^*Example: Covid travel restrictions. Any search results older than a few weeks are outdated, but a 5 year limit isn't going to improve that.

[NeuroticFish]

I can imagine this will be an improvement in some cases: I often get completely outdated information when I search something^*. But if Google can't distinguish between what is and what isn't actually outdated, I'd rather get all relevant search results instead of only the most recent ones.

^*Example: Covid travel restrictions. Any search results older than a few weeks are outdated, but a 5 year limit isn't going to improve that.

Google always had the option to restrict the results to the ones from a certain period. And it was great, and it should do for your use case.
But if no proper results are found whatsoever, there's nothing to restrict...

[LoyceV]

Google always had the option to restrict the results to the ones from a certain period.

I've never clicked "Tools" before. Mind blown!

[Quickseller]

But if Google can't distinguish between what is and what isn't actually outdated, I'd rather get all relevant search results instead of only the most recent ones.

I think they can, it's just matter whether it's cheap or profitable for Google to do it.

Giving search results is not going to generate any profits for google. The ads next to the search results is what generates revenue for google. This means that relevant search results will generate more revenue for google as people use google more often to query information. Google wants to maximize the search results for people that answer the question they are asking.

Making a change to determine what is outdated and what isn’t is not expensive, but it is probably difficult.

[Pmalek]

You should be able to find it by googling for substrings:
Yet you can't.  In fact, AFAICT, no substring will make google return anything on the first page of that thread.

DuckDuckGo doesn't return any results either when searching for substrings.
When the below text is placed in quotation marks on DuckDuckGo, it doesn't find anything.

"All of these radical improvements in scalablity, privacy, and flexibility show up when you realize that "turing complete" is the wrong tool, that what our systems do is verification, not computation."

A search without the quotation marks shows no results from Bitcointalk.

If you put that same text in quotation marks in Google, one of the results will take you to https://bitcointalk.org/index.php?topic=1361602.840 where one user quoted your text. Still, the correct thread doesn't show up in the search results.     

What is weird is that even when you search with "site:bitcointalk.org" and entering a substring like the one below, Google still suggests the other thread that I linked to above and nothing else. DuckDuckGo does much better here as well.

Deciding if an arbitrarily complex condition was met doesn't require a turing complete language or what not-- the verification of a is in P not NP

Search result:

[dkbit98]

What google is doing borderline evil and I guess that is why they removed clause ''don't be evil'' from their code few years ago.
I stopped using google for search and for emails, only thing I need it is for my youtube account, and I use all other alternative search engines that are privacy oriented  like DuckDuckGo, Qwant and Startpage.
Problem is that google holds nearly %90 of market shares for search engines and most people still use only google for searching anything on internet.
I tried searching several of my bitcointalk topics and I also couldn't find any results on google.

Duck-duck-go, however, returns it just fine.

Yandex search is also showing similar results like DuckDuckGo, but it's obvious that google changed their algo code and I don't think it's only crypto or bitcointalk related but it is probably part of some bigger censorship plans.
Maybe this problem with google became much bigger in February of 2021 with their new manual censorship policy:
https://www.breitbart.com/tech/2021/02/09/google-quietly-escalates-manual-search-censorship/

[UserU]

What google is doing borderline evil and I guess that is why they removed clause ''don't be evil'' from their code few years ago.
I stopped using google for search and for emails, only thing I need it is for my youtube account, and I use all other alternative search engines that are privacy oriented  like DuckDuckGo, Qwant and Startpage.
Problem is that google holds nearly %90 of market shares for search engines and most people still use only google for searching anything on internet.

Imagine being the Big Brother of the Internet and having that "Don't Be Evil" motto, is kinda like an oxymoron.

But isn't it quite inconvenient for you given that you lose access to personalized services? Having to manually find contents on YouTube rather than getting served the recommendations from accessing the home page everytime?

Also ngl, Google search is second to none when you are searching for niche terms.

[The Sceptical Chymist]

It's not just Bitcointalk. I've noticed that Google no longer indexes (or at least no longer displays in search results) pages that haven't been updated in over 5 years, effectively causing huge portions of the web to vanish.

I've noticed that without actually recognizing the implications of what Google's doing--if you search for pretty much anything, all you get are relatively recent results.  Not only that, the results are usually from mainstream websites that are probably employing a ton of SEO.  That's why I stopped using Google as a search engine some time ago, though I can't say DDG is ideal.

I haven't posted anything on bitcointalk so critical that I'd want it archived for decades, so this doesn't bother me so much--but there are tons of very helpful posts (especially in the technical sections) that really ought to be searchable on Google, and it sucks that they're basically being sucked down a memory hole.  That's Google for you, though.

Imagine being the Big Brother of the Internet and having that "Don't Be Evil" motto, is kinda like an oxymoron.

That motto is waaaay outdated and they know it, too.

[NotATether]

I'm working on reposting some of the more informative threads on my website, unedited except perhaps for garbage posts in between removed. Hopefully that will solve the site rankings problem.

Going through the tech board it still shocks me how much spam it had during 2017 and 2018.

[Welsh]

The great news is, privacy focused search engines such as Duck Duck Go are becoming much more usable than they were say, two years ago. Duck Duck Go is by far not perfect, but I've been using it as my primary search engine for well over 3 years. I've only ever had to resort to Google a few times, when a site specifically sends you through Google, or Duck Duck Go isn't yielding the search I need.

Although, I'm not quite sure why Google are not indexing older content correctly. I don't really understand the motives behind it. Newer information, doesn't mean its better than the old content. I'm guessing, as Google tries to serve the masses it will try, and provide relevant up to date information, as that's what users of their platform look for. This is probably especially important when concerning statistics. Although, I'd prefer to have them serve the entire history if possible, and just priorities newer content over old. 

[gmaxwell]

I've noticed that Google no longer indexes (or at least no longer displays in search results) pages that haven't been updated in over 5 years, effectively causing huge portions of the web to vanish.

They are also severely down weighing anything that isn't HTTPS... but the effect of this is to severely bury tons of great technical/academic/hobby info at the expense of promoting garbage filler sites and social media.

But that decision wouldn't impact bitcointalk.

I have been using duck duck for most of my searches, though it too sucks compared to google of old. -- just differently.  My concern here isn't just me though-- it's about the kind of distortions that hiding BCT threads has on the wider world.

[mk4]

If you tried doing SEO(search engine optimization) work for a certain blog or website, like damn you'd immediately realize how much of a headache it is to work with Google. After months and months of work you finally get your site to have decent traffic from Google search results, and the first thing you know a few months later, boom. Articles getting deranked. Google is notorious for changing things up with the algorithm A LOT, that could simply either effect your site in a good way, or sometimes even almost kill it in terms of traffic.

[NeuroticFish]

Google is notorious for changing things up with the algorithm A LOT, that could simply either effect your site in a good way, or sometimes even almost kill it in terms of traffic.

Still, searching for terms in the page should yield that page, even if the last in results, not ... just vanish.
I find the last changes of Google a total #fail. Actually since yesterday I also switched to DDG, although Google served me wonderfully since its early beginning.