Bitcoin Forum

I've noticed a large number of my old posts are no longer discoverable via google, maybe on the order of half that I search for (really informally).

Here is an example:

https://bitcointalk.org/index.php?topic=1427885.msg14601127#msg14601127

You should be able to find it by googling for substrings:

Example (https://www.google.com/search?q="The+program+is+big,+and+there+are+fixed+number+of+parties+to+the+contract.+They'll+likely+cooperate+so+long+as+the+threat+of+the+program+execution+exists.").

Yet you can't.  In fact, AFAICT, no substring will make google return anything on the first page of that thread.

If you google the thread title (https://www.google.com/search?q="Turing+completeness+and+state+for+smart+contract") you get the second page of the thread and only the second page of the thread.

Duck-duck-go, however, returns it just fine.

I find it pretty demoralizing to have my posts vanish from search indexes, especially because last year Reddit changed so that negative-voted posts don't get search indexed and as a result thousands of posts of mine in rbtc effectively vanished from the internet.

It's not just Bitcointalk. I've noticed that Google no longer indexes (or at least no longer displays in search results) pages that haven't been updated in over 5 years, effectively causing huge portions of the web to vanish. It's one of the reasons I now use DuckDuckGo almost exclusively. :(

I think this is because of their recent February update with their search engine. Google recently updated their broad-core algorithm in its SEO to be improved with Passage Ranking. I've seen this article that explains it very well -- https://www.searchenginejournal.com/google-passage-ranking-martin-splitt/388206/#close

With that update, it is not only your thread that is affected, but the general results from any users who searched with exact keyword on Google. Based on my understanding with Passage Ranking, its algorithm ranks the results based on the overall content of the page independently -- which leads to your 2nd thread page be shown first than the main OP.

That's a bummer!  I thought the idea with Google was to provide an index to archives, as well as current stuff.

There's a discussion that started last year on Google starting to act different (https://bitcointalk.org/index.php?topic=5278070.0).
Back then they were still returning for me relevant pages. Lately I started getting basically crap. Now I know why.

---

I've done exact search for some areas in that old post and.. woah, even older/weaker search engines (yahoo, lycos, excite) don't return bitcointalk. WTF?!
Until know it's DuckDuckGo and Yandex returning it, and I'm not really comfortable with the thought of getting to use Yandex search.

OP, at least I got your thread on my search results and it's shown on the top:
https://i.ibb.co/NSHVSXK/Screenshot-2021-04-29-01-30-40-045-com-android-chrome.jpg (https://ibb.co/NSHVSXK)
So, if I'm going to search for post made in 2014, it's likely that I'm not going to find it?
At least I have different experience. I don't have problems to get old content in search results. It applies to Bitcointalk and other websites.
I have tried to use DuckDuckGo, but I wasn't happy with search results.

If I understand that blog post correctly, google has updated their algo to return results that are relevant to the query being asked, rather than key words in a query. So searching for cars with the best safety rating might return a result that talks about a safety metric and ranks cars accordingly, but might not have the term "best" nor "safety" in the content of the page. 

In the past, when searching for something, you might encounter a webpage that has many random keywords unrelated to eachother, and no actual useful content. This change seems to be making it less likely these types of pages will be returned in search results.

I can imagine this will be an improvement in some cases: I often get completely outdated information when I search something^*. But if Google can't distinguish between what is and what isn't actually outdated, I'd rather get all relevant search results instead of only the most recent ones.

^*Example: Covid travel restrictions. Any search results older than a few weeks are outdated, but a 5 year limit isn't going to improve that.

Google always had the option to restrict the results to the ones from a certain period. And it was great, and it should do for your use case.
But if no proper results are found whatsoever, there's nothing to restrict...

I've never clicked "Tools" before. Mind blown!

Giving search results is not going to generate any profits for google. The ads next to the search results is what generates revenue for google. This means that relevant search results will generate more revenue for google as people use google more often to query information. Google wants to maximize the search results for people that answer the question they are asking.

Making a change to determine what is outdated and what isn’t is not expensive, but it is probably difficult.

DuckDuckGo doesn't return any results either when searching for substrings.
When the below text is placed in quotation marks on DuckDuckGo, it doesn't find anything.
A search without the quotation marks shows no results from Bitcointalk.

If you put that same text in quotation marks in Google, one of the results will take you to https://bitcointalk.org/index.php?topic=1361602.840 where one user quoted your text. Still, the correct thread doesn't show up in the search results.     

What is weird is that even when you search with "site:bitcointalk.org" and entering a substring like the one below, Google still suggests the other thread that I linked to above and nothing else. DuckDuckGo does much better here as well.

Search result:

https://talkimg.com/images/2023/11/12/z2hg3.jpeg

What google is doing borderline evil and I guess that is why they removed clause ''don't be evil'' from their code few years ago.
I stopped using google for search and for emails, only thing I need it is for my youtube account, and I use all other alternative search engines that are privacy oriented  like DuckDuckGo, Qwant and Startpage.
Problem is that google holds nearly %90 of market shares for search engines and most people still use only google for searching anything on internet.
I tried searching several of my bitcointalk topics and I also couldn't find any results on google.


Yandex search is also showing similar results like DuckDuckGo, but it's obvious that google changed their algo code and I don't think it's only crypto or bitcointalk related but it is probably part of some bigger censorship plans.
Maybe this problem with google became much bigger in February of 2021 with their new manual censorship policy:
https://www.breitbart.com/tech/2021/02/09/google-quietly-escalates-manual-search-censorship/

Imagine being the Big Brother of the Internet and having that "Don't Be Evil" motto, is kinda like an oxymoron.

But isn't it quite inconvenient for you given that you lose access to personalized services? Having to manually find contents on YouTube rather than getting served the recommendations from accessing the home page everytime?

Also ngl, Google search is second to none when you are searching for niche terms.

I've noticed that without actually recognizing the implications of what Google's doing--if you search for pretty much anything, all you get are relatively recent results.  Not only that, the results are usually from mainstream websites that are probably employing a ton of SEO.  That's why I stopped using Google as a search engine some time ago, though I can't say DDG is ideal.

I haven't posted anything on bitcointalk so critical that I'd want it archived for decades, so this doesn't bother me so much--but there are tons of very helpful posts (especially in the technical sections) that really ought to be searchable on Google, and it sucks that they're basically being sucked down a memory hole.  That's Google for you, though.

That motto is waaaay outdated and they know it, too.

I'm working on reposting some of the more informative threads on my website, unedited except perhaps for garbage posts in between removed. Hopefully that will solve the site rankings problem.

Going through the tech board it still shocks me how much spam it had during 2017 and 2018.

The great news is, privacy focused search engines such as Duck Duck Go are becoming much more usable than they were say, two years ago. Duck Duck Go is by far not perfect, but I've been using it as my primary search engine for well over 3 years. I've only ever had to resort to Google a few times, when a site specifically sends you through Google, or Duck Duck Go isn't yielding the search I need.

Although, I'm not quite sure why Google are not indexing older content correctly. I don't really understand the motives behind it. Newer information, doesn't mean its better than the old content. I'm guessing, as Google tries to serve the masses it will try, and provide relevant up to date information, as that's what users of their platform look for. This is probably especially important when concerning statistics. Although, I'd prefer to have them serve the entire history if possible, and just priorities newer content over old. 

They are also severely down weighing anything that isn't HTTPS... but the effect of this is to severely bury tons of great technical/academic/hobby info at the expense of promoting garbage filler sites and social media. :(

But that decision wouldn't impact bitcointalk.

I have been using duck duck for most of my searches, though it too sucks compared to google of old. -- just differently.  My concern here isn't just me though-- it's about the kind of distortions that hiding BCT threads has on the wider world.

If you tried doing SEO(search engine optimization) work for a certain blog or website, like damn you'd immediately realize how much of a headache it is to work with Google. After months and months of work you finally get your site to have decent traffic from Google search results, and the first thing you know a few months later, boom. Articles getting deranked. Google is notorious for changing things up with the algorithm A LOT, that could simply either effect your site in a good way, or sometimes even almost kill it in terms of traffic.

Still, searching for terms in the page should yield that page, even if the last in results, not ... just vanish.
I find the last changes of Google a total #fail. Actually since yesterday I also switched to DDG, although Google served me wonderfully since its early beginning.

I can argue on this.

https://i.imgur.com/Cc1skla.png https://i.imgur.com/NqRmm4C.png

Yes, everyone is talking about the page experience that says your page should be engaging and built for a user and stuff but that doesn't mean old pages are automatically removed now. There are certain topics that require freshness and old pages might be not displayed when searched by particular keywords anymore but google cannot just vanish all pages that are not updated in over 5 years or even 10 years. That's millions of terabytes of data.

Yes. That is why we buy Yoast's SEO plugin, because the free version is basically an SEO tweaker on crutches, and both of them won't work if you're using plain old html and not Wordpress/Drupal/Magento which makes the problem of SEO sort of stillborn for those kind of sites (like Bitcointalk).

Surprisingly this Bitcointalk ranks quite well even without SEO, but that's one hell rare of a case without any reliance on such.

Does it rank well? Really? By what big search query in Blockchain, Crypto, Bitcoin niches it ranks well? "cryptocurrency", "what is bitcoin", "buy bitcoin", "what is crypto", "how does crypto mining work", "what is blockchain", etc. the forum isn't close to the top 10. Almost half of the traffic is direct visits (I assume) and it ranks only by brand names or keywords with "forum" in it.

Of course we can't cover all aspects, but the keywords "community" and "forum" keeps it on the first page. And being ranked higher than the top 10,000 websites, not too shabby.

True. Unfortunately because of this case, it seems like tinkering with the algorithm could end up with some fuck-ups as well. or maybe like with Foxpup response, they're just kinda incentivizing writers to post and update their content more(which has it's own pros and cons).

That would be rather stupid, but I would not be surprised.
However, from my tests, old 2010, 2012, 2013 StackOverflow answers are still returned by Google if the terms fit.

Maybe Greg is right and the problem has to be researched mostly focused on Bitcointalk.

You can thank the ancient 2009 domain and the thousands of backlinks from other sites for that  ;)


The bitcoin keywords are HEAVILY competitive and there's swathes of web pages optimizing for all those above keywords which is why you don't see BTCT anywhere close to there.

Fun fact: all the bitcoin keywords have a volume of like <5000, except for "bitcoin price" which has a whooping 250,000 volume which shows just how many more people search for that than anything else (including investors, nocoiners, journalists and such). I'd show you a pic of the LSIgraph of the keywords but they want me to pay $1,000 a year for the access.

SEO is, as I found out, one hell of an expensive job.


If that annoys you then wait until you hear that neilpatel.com site (a well-known SEO agency) told me that Google makes around 8 algorithm changes a day!

Yes, it does. But as I said, I've fixed the problem: switched to DDG.
Since I search for information and not own websites, Google changes in algo usually didn't affect me. As I said, as long as the result is there, I don't care that much if it's the first or 75th. If it's 75th I may refine the search and still find it.

But what Greg has shown was that no matter what you do, you will no longer get that page as result. And that's imho a radical change.

that's why I suggested analyzing these keywords and creating relevant pages with valuable content. The forum has everything to compete for these "heavy" keywords: age, authority, traffic, backlinks, etc.

that particular keyword (bitcoin price) doesn't make any sense to have a rank on, I guess. People search it to find out the price, nothing more, don't they?

that's right. But another fun fact is in most SEO communities people get that guy not seriously, he talks general stuff, not really technical/factual information. Seroundtable.com is a go-to source when it comes to G updates.

My guess: Either Greg is right and it's a Bitcointalk problem, or Google is AB testing. I kinda doubt they would do a carpet update totally killing search traffic for old content, but hey, who knows.

Not exactly true. Pretty much every article you read about Bitcoin's price or stock performance is technically a "bitcoin price" keyword.

Yes, but when people search exactly "bitcoin price" they do not have other intent but to find out the latest price. There might be other articles that cover various topics but people who search for bitcoin prices have no interest in them. This is what I meant.

That appeared on the top because you searched using the main title "Turing completeness and state for smart contract". What OP highlighted is specific post within that page. This is the issue of SEO and how this forum SMF is designed. It's not related to old or new results. I have a suggestion for theymos to solve this....

Well here is a small suggestion what i think can be done in terms of SEO.. not sure how much work need to done by theymos to do this. But i believe this will be very effective and might solve this current issue where google not showing the results. I have faced similar issue and i was able to solve it. I will try to explain it here..

Since the main topic have unique URL with topic id and each post have their own unique URL with msg number. This is good, but it's not good enough for search engines

Main topic:
https://bitcointalk.org/index.php?topic=1427885
https://i.imgur.com/vD4S7m1.png

Specific post in topic:
https://bitcointalk.org/index.php?topic=1427885.msg14601127
https://i.imgur.com/HjszN71.png

If you notice in both cases page title and content remains the same (at least for the first page 20 posts), so there's no way google going to keep track of the full page (#msg14601127 part of url will be resolved on client side to scroll to specific post)
 
However these days its a normal practice to include the relevant page title for each post or reply. This can be seen in twitter. Twitter changes the title of the page to specific tweet even if its a reply.

Main tweet and notice the title
https://twitter.com/DocumentingBTC/status/1388231761096847364
Title of the page

Subsequent reply and see how the page title changes
https://twitter.com/LastCoinStandng/status/1388233347520897026
Title of the page

Twitter adding main tweet's handle + reply handle + reply tweet in Page Title.

Similar thing can be seen happening on many other website quora is another example.


Suggestion to show different page title for specific posts:

If the page is accessed using specific ".msg14601127" in the URL
then bitcointalk should show the different "Page Title" and add some text from the post itself in this case google and search engines will keep separate index of it.

Yes there can be other solutions too but this is the easiest one.. another solution could be starting the page content from the specific post mentioned in the URL. But this will be complex specially in the current SMF.

So when user is accessing the page using this URL https://bitcointalk.org/index.php?topic=1427885.msg14601127#msg14601127

Page title should also change like below, google shows 60 characters on the result page but there is no specific limit to characters to be included in the title.

https://i.imgur.com/vJ2MS4k.png

Neil Patel has some good tools and advice, just that the consultation fees are a premium to begin with. But for someone to invest so much effort in building his brand, that's justifiable.

On a side note, I've seen alot of hate for that fella on another discussion forum. It seems like that poor fella keeps getting flak over his looks ;D

Most search engines offer an advantage in ranking results to those pages that are optimized for mobile devices. And Google is the first to make such changes to its algorithm. Accordingly, since Bitcointalk does not have a mobile version of the site, all search results will be displayed after those pages that are optimized for mobile devices.

Well, it does.. and least for some definition of mobile (https://bitcointalk.org/index.php?wap). :P
Not the point, the issue I'm raising is that chunks of the site aren't indexed at all, not just poorly ranked.

One possibility is that the site is being targeted by parties trying to keep information out of google.  For example, after forum user cy ph er doc (spaces to avoid the same fate for this thread) participated in a scam that ripped off a lot of people he ended up in court over it which resulted in his bitcoin address being posted Bitcointalk.  Although his bitcoin address is still on bitcointalk you can't find any of those pages in google.  So maybe he paid some service to vanish the information from the internet and they're doing something shady to get bct pages removed from google indexes? -- and stuff like that.  Maybe the cumulative effect of various scammers trying to keep pages on bct that expose them out of google might be hurting the sites visibility?

Links that are posted on bitcointalk tend to rank well on google search results. This is why there have been so many spambots posting links on threads, and even creating accounts that never get used with links on their profiles.

Maybe theymos can provide some insight as to if google's crawlers are visiting old threads at rates less frequent than usual.

Website admin here. Structured data really only helps your search results if you also have a high pagerank (I'm sure you already know this).

Structured data is only an addon as far as Googlebot is concerned and doesn't influence your search results much. A site's not going anywhere without a ton of backlinks.

Edit: After checking your post history it's apparent that you are another alt of humanrightsfoundation so I'm not surprised at your dislike of BTCT.

Some update on the situation.

Since this thread was made, I've switched from Google (which I've used basically since its inception) to DuckDuckGo, which was sometimes better, sometimes worse.
Today I've been searching (with DuckDuckGo) for something I knew it's on bitcointalk, still DuckDuckGo didn't find it for me.
I thought "what if?" and I tried with google this time and it worked. Surprise-surprise.

So I came back to this topic and clicked on Greg's example in the starting post. First page didn't return the correct page, but after clicking the
"If you like, you can repeat the search with the omitted results included." link it has returned the "missing topic" (as the second result!).

I find it an improvement - not the best, but still closer to what should be - hence I thought it worth telling.

It has been discussed on other forums that part of the issue is forums like this themselves.

If I search for how to do something with my car, instead of winding up on a forum, I get links to other sites. Blogs, youtube, medium posts. Almost like G is penalizing forums.
And lets face it, that might be a people problem, they are thinking don't TELL me how to reset the oil warning light, give me a 1 minute video.

Since G does give links that have been clicked more times more weight in a search (or at least they did) it's kind of a self fulfilling prophecy. Nobody clicks on the 10 step how to, so it gets buried, then you can't even find it. Just the videos on how to do it.

If more people search for something about BTC and wind up clicking on github, or medium, or wherever then links here become harder to find.

-Dave

Starting today on August 3rd google is again allowing crypto and bitcoin advertisement so my speculation is that this may also have a positive effect on bitcointalk forum.
This only applies for US customers for now, and works only for products like crypto exchanges and wallets that are registered with FinCEN or a federal/state chartered bank.
I am not sure if this applies on forums like bitcointalk but we can monitor situation and see if something changes in next few weeks.
Source: https://support.google.com/adspolicy/answer/10688110?hl=en&ref_topic=29265

Note that I totally stopped using google search and I am using alternative search engines like DuckDuckGo, Qwant and Brave search that I started testing recently.

Generally, DuckDuckGo isn't as good at indexing, and finding what your searching for, and sometimes you have to revert to using Google or another search engine. However, if you can avoid that, and find it manually its a much better improvement in terms of privacy. Also, I've used DuckDuckGo for a long time, it used to be absolutely awful a few years ago, and it was hard to keep using it despite the obvious benefits to privacy, but nowadays its much better, and returns the expected results about 90% of the time. Google in terms of indexing, and technical searching is better, but I expect DuckDuckGo to keep getting better, and be much more closer to Google given enough time.

I haven't read too much into it because it doesn't really apply to me much, but I assume by the way you phrased it that this is offering an incentive to those that are following the Know Your Customer (KYC) guidelines. So, might not have any effect on the forum.

While I agree completely that Google is not good for privacy, I'd say that my statement was not about how good, bad or evil google (and its search) is, it was more to bring the news that Bitcointalk gets indexed a bit better again. And since most users do search with Google, this should be a good news.

Definitely. The majority of the users registering, and landing on this forum would have been directed here via word of mouth or through Google. I can't deny that. My angle of my reply was that alternatives to Google are becoming better, and therefore we should see more, and more users using different search engines in then next couple of years, so targeting each search engine for ranking is important. Although, I'm not going to deny that Google will likely have the monopoly on searches for quite some time.

I doubt this forum would be affected since it's not offering any services that's under Google's scrutiny such as licensed exchanges.