Google partial blackholing of Bitcointalk?

[mediaBuzz]

It's not just Bitcointalk. I've noticed that Google no longer indexes (or at least no longer displays in search results) pages that haven't been updated in over 5 years, effectively causing huge portions of the web to vanish. It's one of the reasons I now use DuckDuckGo almost exclusively.

I can argue on this.



Yes, everyone is talking about the page experience that says your page should be engaging and built for a user and stuff but that doesn't mean old pages are automatically removed now. There are certain topics that require freshness and old pages might be not displayed when searched by particular keywords anymore but google cannot just vanish all pages that are not updated in over 5 years or even 10 years. That's millions of terabytes of data.

[1714912888]

1714912888

[1714912888]

1714912888

[1714912888]

1714912888

[NotATether]

If you tried doing SEO(search engine optimization) work for a certain blog or website, like damn you'd immediately realize how much of a headache it is to work with Google. After months and months of work you finally get your site to have decent traffic from Google search results, and the first thing you know a few months later, boom. Articles getting deranked. Google is notorious for changing things up with the algorithm A LOT, that could simply either effect your site in a good way, or sometimes even almost kill it in terms of traffic.

Yes. That is why we buy Yoast's SEO plugin, because the free version is basically an SEO tweaker on crutches, and both of them won't work if you're using plain old html and not Wordpress/Drupal/Magento which makes the problem of SEO sort of stillborn for those kind of sites (like Bitcointalk).

[UserU]

Yes. That is why we buy Yoast's SEO plugin, because the free version is basically an SEO tweaker on crutches, and both of them won't work if you're using plain old html and not Wordpress/Drupal/Magento which makes the problem of SEO sort of stillborn for those kind of sites (like Bitcointalk).

Surprisingly this Bitcointalk ranks quite well even without SEO, but that's one hell rare of a case without any reliance on such.

[mediaBuzz]

Yes. That is why we buy Yoast's SEO plugin, because the free version is basically an SEO tweaker on crutches, and both of them won't work if you're using plain old html and not Wordpress/Drupal/Magento which makes the problem of SEO sort of stillborn for those kind of sites (like Bitcointalk).

Surprisingly this Bitcointalk ranks quite well even without SEO, but that's one hell rare of a case without any reliance on such.

Does it rank well? Really? By what big search query in Blockchain, Crypto, Bitcoin niches it ranks well? "cryptocurrency", "what is bitcoin", "buy bitcoin", "what is crypto", "how does crypto mining work", "what is blockchain", etc. the forum isn't close to the top 10. Almost half of the traffic is direct visits (I assume) and it ranks only by brand names or keywords with "forum" in it.

[UserU]

Does it rank well? Really? By what big search query in Blockchain, Crypto, Bitcoin niches it ranks well? "cryptocurrency", "what is bitcoin", "buy bitcoin", "what is crypto", "how does crypto mining work", "what is blockchain", etc. the forum isn't close to the top 10. Almost half of the traffic is direct visits (I assume) and it ranks only by brand names or keywords with "forum" in it.

Of course we can't cover all aspects, but the keywords "community" and "forum" keeps it on the first page. And being ranked higher than the top 10,000 websites, not too shabby.

[mk4]

Still, searching for terms in the page should yield that page, even if the last in results, not ... just vanish.
I find the last changes of Google a total #fail. Actually since yesterday I also switched to DDG, although Google served me wonderfully since its early beginning.

True. Unfortunately because of this case, it seems like tinkering with the algorithm could end up with some fuck-ups as well. or maybe like with Foxpup response, they're just kinda incentivizing writers to post and update their content more(which has it's own pros and cons).

[NeuroticFish]

they're just kinda incentivizing writers to post and update their content more(which has it's own pros and cons).

That would be rather stupid, but I would not be surprised.
However, from my tests, old 2010, 2012, 2013 StackOverflow answers are still returned by Google if the terms fit.

Maybe Greg is right and the problem has to be researched mostly focused on Bitcointalk.

[NotATether]

Surprisingly this Bitcointalk ranks quite well even without SEO, but that's one hell rare of a case without any reliance on such.

You can thank the ancient 2009 domain and the thousands of backlinks from other sites for that  

Does it rank well? Really? By what big search query in Blockchain, Crypto, Bitcoin niches it ranks well? "cryptocurrency", "what is bitcoin", "buy bitcoin", "what is crypto", "how does crypto mining work", "what is blockchain", etc. the forum isn't close to the top 10. Almost half of the traffic is direct visits (I assume) and it ranks only by brand names or keywords with "forum" in it.

The bitcoin keywords are HEAVILY competitive and there's swathes of web pages optimizing for all those above keywords which is why you don't see BTCT anywhere close to there.

Fun fact: all the bitcoin keywords have a volume of like <5000, except for "bitcoin price" which has a whooping 250,000 volume which shows just how many more people search for that than anything else (including investors, nocoiners, journalists and such). I'd show you a pic of the LSIgraph of the keywords but they want me to pay $1,000 a year for the access.

SEO is, as I found out, one hell of an expensive job.

Still, searching for terms in the page should yield that page, even if the last in results, not ... just vanish.
I find the last changes of Google a total #fail. Actually since yesterday I also switched to DDG, although Google served me wonderfully since its early beginning.

If that annoys you then wait until you hear that neilpatel.com site (a well-known SEO agency) told me that Google makes around 8 algorithm changes a day!

[NeuroticFish]

If that annoys you then wait until you hear that neilpatel.com site (a well-known SEO agency) told me that Google makes around 8 algorithm changes a day!

Yes, it does. But as I said, I've fixed the problem: switched to DDG.
Since I search for information and not own websites, Google changes in algo usually didn't affect me. As I said, as long as the result is there, I don't care that much if it's the first or 75th. If it's 75th I may refine the search and still find it.

But what Greg has shown was that no matter what you do, you will no longer get that page as result. And that's imho a radical change.

[mediaBuzz]

The bitcoin keywords are HEAVILY competitive and there's swathes of web pages optimizing for all those above keywords which is why you don't see BTCT anywhere close to there.

that's why I suggested analyzing these keywords and creating relevant pages with valuable content. The forum has everything to compete for these "heavy" keywords: age, authority, traffic, backlinks, etc.

Fun fact: all the bitcoin keywords have a volume of like <5000, except for "bitcoin price" which has a whopping 250,000 volume which shows just how many more people search for that than anything else (including investors, nocoiners, journalists and such). I'd show you a pic of the LSIgraph of the keywords but they want me to pay $1,000 a year for the access.

that particular keyword (bitcoin price) doesn't make any sense to have a rank on, I guess. People search it to find out the price, nothing more, don't they?

If that annoys you then wait until you hear that neilpatel.com site (a well-known SEO agency) told me that Google makes around 8 algorithm changes a day!

that's right. But another fun fact is in most SEO communities people get that guy not seriously, he talks general stuff, not really technical/factual information. Seroundtable.com is a go-to source when it comes to G updates.

[mk4]

That would be rather stupid, but I would not be surprised.
However, from my tests, old 2010, 2012, 2013 StackOverflow answers are still returned by Google if the terms fit.

Maybe Greg is right and the problem has to be researched mostly focused on Bitcointalk.

My guess: Either Greg is right and it's a Bitcointalk problem, or Google is AB testing. I kinda doubt they would do a carpet update totally killing search traffic for old content, but hey, who knows.

[NotATether]

that particular keyword (bitcoin price) doesn't make any sense to have a rank on, I guess. People search it to find out the price, nothing more, don't they?

Not exactly true. Pretty much every article you read about Bitcoin's price or stock performance is technically a "bitcoin price" keyword.

[mediaBuzz]

that particular keyword (bitcoin price) doesn't make any sense to have a rank on, I guess. People search it to find out the price, nothing more, don't they?

Not exactly true. Pretty much every article you read about Bitcoin's price or stock performance is technically a "bitcoin price" keyword.

Yes, but when people search exactly "bitcoin price" they do not have other intent but to find out the latest price. There might be other articles that cover various topics but people who search for bitcoin prices have no interest in them. This is what I meant.

[shahzadafzal]

OP, at least I got your thread on my search results and it's shown on the top:

That appeared on the top because you searched using the main title "Turing completeness and state for smart contract". What OP highlighted is specific post within that page. This is the issue of SEO and how this forum SMF is designed. It's not related to old or new results. I have a suggestion for theymos to solve this....

Well here is a small suggestion what i think can be done in terms of SEO.. not sure how much work need to done by theymos to do this. But i believe this will be very effective and might solve this current issue where google not showing the results. I have faced similar issue and i was able to solve it. I will try to explain it here..

Since the main topic have unique URL with topic id and each post have their own unique URL with msg number. This is good, but it's not good enough for search engines

Main topic:
https://bitcointalk.org/index.php?topic=1427885


Specific post in topic:
https://bitcointalk.org/index.php?topic=1427885.msg14601127


If you notice in both cases page title and content remains the same (at least for the first page 20 posts), so there's no way google going to keep track of the full page (#msg14601127 part of url will be resolved on client side to scroll to specific post)
 
However these days its a normal practice to include the relevant page title for each post or reply. This can be seen in twitter. Twitter changes the title of the page to specific tweet even if its a reply.

Main tweet and notice the title
https://twitter.com/DocumentingBTC/status/1388231761096847364
Title of the page

Code:

Documenting Bitcoin 📄 on Twitter_ _The Mayor of Jackson, Tennessee, @MayorConger, is talking with state legislatures about allowing the city to mine #bitcoin and adding it to their balance sheet

Subsequent reply and see how the page title changes
https://twitter.com/LastCoinStandng/status/1388233347520897026
Title of the page

Code:

lastcoinstanding 🔴 on Twitter_ _@DocumentingBTC @MayorConger It's worth keeping in mind that this is another example of how innovative government policy involves not so much creating new incentives but rather just dismantl

Twitter adding main tweet's handle + reply handle + reply tweet in Page Title.

Similar thing can be seen happening on many other website quora is another example.


Suggestion to show different page title for specific posts:

If the page is accessed using specific ".msg14601127" in the URL
then bitcointalk should show the different "Page Title" and add some text from the post itself in this case google and search engines will keep separate index of it.

Yes there can be other solutions too but this is the easiest one.. another solution could be starting the page content from the specific post mentioned in the URL. But this will be complex specially in the current SMF.

So when user is accessing the page using this URL https://bitcointalk.org/index.php?topic=1427885.msg14601127#msg14601127

Page title should also change like below, google shows 60 characters on the result page but there is no specific limit to characters to be included in the title.

[UserU]

If that annoys you then wait until you hear that neilpatel.com site (a well-known SEO agency) told me that Google makes around 8 algorithm changes a day!

Neil Patel has some good tools and advice, just that the consultation fees are a premium to begin with. But for someone to invest so much effort in building his brand, that's justifiable.

On a side note, I've seen alot of hate for that fella on another discussion forum. It seems like that poor fella keeps getting flak over his looks

[tvplus006]

Most search engines offer an advantage in ranking results to those pages that are optimized for mobile devices. And Google is the first to make such changes to its algorithm. Accordingly, since Bitcointalk does not have a mobile version of the site, all search results will be displayed after those pages that are optimized for mobile devices.

[gmaxwell]

since Bitcointalk does not have a mobile version of the site,

Well, it does.. and least for some definition of mobile.

all search results will be displayed after those pages that are optimized for mobile devices.

Not the point, the issue I'm raising is that chunks of the site aren't indexed at all, not just poorly ranked.

One possibility is that the site is being targeted by parties trying to keep information out of google.  For example, after forum user cy ph er doc (spaces to avoid the same fate for this thread) participated in a scam that ripped off a lot of people he ended up in court over it which resulted in his bitcoin address being posted Bitcointalk.  Although his bitcoin address is still on bitcointalk you can't find any of those pages in google.  So maybe he paid some service to vanish the information from the internet and they're doing something shady to get bct pages removed from google indexes? -- and stuff like that.  Maybe the cumulative effect of various scammers trying to keep pages on bct that expose them out of google might be hurting the sites visibility?

[Quickseller]

Yes. That is why we buy Yoast's SEO plugin, because the free version is basically an SEO tweaker on crutches, and both of them won't work if you're using plain old html and not Wordpress/Drupal/Magento which makes the problem of SEO sort of stillborn for those kind of sites (like Bitcointalk).

Surprisingly this Bitcointalk ranks quite well even without SEO, but that's one hell rare of a case without any reliance on such.

Does it rank well? Really? By what big search query in Blockchain, Crypto, Bitcoin niches it ranks well? "cryptocurrency", "what is bitcoin", "buy bitcoin", "what is crypto", "how does crypto mining work", "what is blockchain", etc. the forum isn't close to the top 10. Almost half of the traffic is direct visits (I assume) and it ranks only by brand names or keywords with "forum" in it.

Links that are posted on bitcointalk tend to rank well on google search results. This is why there have been so many spambots posting links on threads, and even creating accounts that never get used with links on their profiles.

Maybe theymos can provide some insight as to if google's crawlers are visiting old threads at rates less frequent than usual.

[NotATether]

If you need to optimize things you can and should do with structured data.

Website admin here. Structured data really only helps your search results if you also have a high pagerank (I'm sure you already know this).

Structured data is only an addon as far as Googlebot is concerned and doesn't influence your search results much. A site's not going anywhere without a ton of backlinks.

Edit: After checking your post history it's apparent that you are another alt of humanrightsfoundation so I'm not surprised at your dislike of BTCT.

[NeuroticFish]

Some update on the situation.

Since this thread was made, I've switched from Google (which I've used basically since its inception) to DuckDuckGo, which was sometimes better, sometimes worse.
Today I've been searching (with DuckDuckGo) for something I knew it's on bitcointalk, still DuckDuckGo didn't find it for me.
I thought "what if?" and I tried with google this time and it worked. Surprise-surprise.

So I came back to this topic and clicked on Greg's example in the starting post. First page didn't return the correct page, but after clicking the
"If you like, you can repeat the search with the omitted results included." link it has returned the "missing topic" (as the second result!).

I find it an improvement - not the best, but still closer to what should be - hence I thought it worth telling.