There's a new "Google Jail" for independent wikis

226 pointsposted 10 hours ago
by pizzaiolo
(weirdgloop.org)

78 Comments

SyneRyder

an hour ago

Some of this might just be problems with the websites themselves. With the Path Of Exile 2 Wiki example, I'm looking at the sitemaps. Once you get through the first sitemap of nested sitemaps (ugh, not a fan of those), the very first actual sitemap returns:

  XML Parsing Error: no root element found
  Location: https://www.poe2wiki.net/sitemap/sitemap-poe_wiki-poe2_wiki--NS_0-0.xml
  Line Number 18040, Column 1
(EDIT: Of course, after I post this, it's now working again...)

On to the Hytale Wiki example. I don't know if what I'm about to say applies to Google, but I'm approaching it from the perspective of my tiny dumb indieweb indexer for my personal search engine. It is much easier for me to index from a sitemap rather than try to crawl a website, so I basically look exclusively at sitemaps.

Looking at Hytale Wiki, my process:

* Site has a robots.txt file - good!

* Robots.txt mentions a sitemap - excellent!

* The sitemap is stored at /images/sitemaps/index.xml ... oh. I would usually not index anything from a /images/ folder, because I want to index pages only, not images. This would likely trip my exclusion filters. Let's ignore that and continue.

* The sitemap is a list of an additional 26 nested Gzipped sitemaps. Oh.

This is the point where my tiny dumb indexer would stop. Gzipped sitemaps are part of the standard, but they're relatively rare on the web for small sites. They typically only get used if a sitemap file exceeds the 50,000 pages-per-sitemap limit. In this case, 26 * 50,000 makes the HyTale Wiki look like a 1.3 Million page site. Do I really want to index 1.3 Million pages, an estimated 13GB of indexed text data, about a video game I'll probably never play?

My search index is storage constrained, and my indexer is very time constrained. The time I spend indexing your site is time not spent indexing another, possibly higher quality website. So at this point, I'd just grab the front page and disappear... like Google apparently does too.

Of course, the HyTale site isn't 1.3 Million pages, it's only 4,277 articles. That would all fit in the root sitemap file, and that might be the better approach for getting indexed.

iamacyborg

2 minutes ago

The poe2wiki issue is a little different to what is documented, albeit only for the last few months, previously we certainly had visibility issues.

The recent issues relate to site errors due to aggressive crawling of uncached, server-intensive pages (diffs, etc) hidden behind residential proxies. We were being hit sufficiently hard that the server had stability issues and we were penalised by Google.

SyneRyder

an hour ago

Me again! I couldn't figure out why I was getting transient errors on that Poe2wiki sitemap, so I tried looking at other sitemaps on the site. I'm getting transient errors on all of them.

Then I realized - the sitemap files are being overwritten in real time. Every edit on the Wiki is causing the sitemap file to be edited in real time. That's why the sitemaps sometimes stop right in the middle of a filename when I access it - the sitemap file is in the process of being rewritten.

That's a behavior unique to a Wiki, and might explain the entire phenomenon.

masklinn

42 minutes ago

Makes sense. Whatever generates those site maps should use the classic Unix trick of writing a separate file then renaming. Also maybe don’t update lastmods on each individual update, it’s a waste of resources since the granularity of lastmod is usually the day (though you can make it the second if you want).

KingMob

34 minutes ago

This is all valid critique, but I have great difficulty believing Google's crawler behaves anything like yours.

I would expect Google to handle broken or missing sitemaps, honestly.

unglaublich

3 hours ago

> Subdomains of existing domains are totally fine

Damn, another push to centralizing and walling off user content in 'fandoms', 'reddits' and other closed communities?

Have the internet cards been handed out now, and are we in the end-game?

Will old domains resell for premiums like low-background-radiation steel?

captainbland

2 hours ago

It wouldn't surprise me. It seems like a mirror of the social media move from social graphs to recommendation engines. You are either producer or consumer and never shall consumer be promoted organically.

anal_reactor

2 hours ago

The answer is simple. Limit your time on social media the way you'd be limiting a child and focus on IRL interactions.

captainbland

12 minutes ago

I also find IRL social interactions also want pacing to some degree. But I don't know, I find my brain is sort of wired for these sorts of novel, pseudonymous interactions from a young age.

KingMob

31 minutes ago

Words of wisdom to heed. Thank you, user anal_reactor :‑D

Ygg2

an hour ago

That or Google is getting so unusable even Discord is a better search engine.

Perz1val

2 hours ago

Not the end, just google bleeding itself out as a search engine while LLMs take over. Chatgpt often links me random, actually decent sites that I wouldn't have gotten from a simple google search on first page, because it would've been exclusively seo slop

bityard

an hour ago

ChapGPT (and all other hosted L To me) still use a search engine behind the scenes whenever you ask a question. Often, it uses those seo slop sites as primary sources when formulating an answer. Which is why, even though hallucinations are now somewhat rare these days, you _still_ should independently verify their output for anything important.

arjie

39 minutes ago

I host my blog etc. on Mediawiki. It has a bunch of defaults that make it not SEO friendly. I was absent from all search engines for a year and told a friend how it was funny that I guess Google no longer ranks personal sites and he told me he’d look and see.

It turned out that default Mediawiki stuff doesn’t do a lot of standard SEO stuff. I just did all of it. From the jsonld tags, to the meta description, to making sure the sitemaps were correct. As soon as I did, my site started showing up on Google.

I cannot speak to these wikis’ problems but the first port of call for me would be checking that everything “standard” for this is now done. It’s just a reality of the web these days. I do get random traffic from Google now. Much less than I used to under the old regime in the 2000s but it feels more like a secular change than the flat zero listings I previously did.

I wish I had kept track of everything so that one day I could say what I did but I just followed literally everything, including removing the index.php thing that Mediawiki uses by default. So I cannot even say which actions worked and which didn’t. I can’t even recall which changes to base I did do. Link rel canonical. A robots.txt that disables access to pages that accidentally duplicate content (e.g. permalink pages). Hard to tell. But the difference was stark 0 to hundreds of pages. Literally zero dude. And it happened weeks after the changes and very suddenly.

remedan

5 hours ago

I enjoy the this extension that redirects search results from Fandom and other such terrible hosts to their higher quality indie alternatives.

https://github.com/IndieWikiBuddy/indie-wiki-buddy

iamacyborg

4 hours ago

This is great for users, but it’s terrible for the SEO of those independent wikis as you’re telling Google you want to click on Fandom and Fextra links.

/edit

It appears this is no longer the case and the extension does things a lot better than it used to!

maroider

4 hours ago

That might be true for the default config in the extension, but you can also configure the extension to de-emphasize fandom wikis or outright hide them in search results. I don't quite remember the exact default config though.

skhameneh

6 hours ago

For a while, Fandom was having annoying ads and content that would get past my ad blocker. But a recent update has caused them to block users with ad blockers.

I've been looking for explicit alternatives to Fandom for some content and it's annoyingly sparse.

Just the other day I saw some meme about how if we saw as many ads taking over sites like we do now, that was a sure sign of having your machine getting infected with a virus.

Generally I've found a good portion of websites that block users with ad blockers tend to be the worst offenders with aggressive ads...

dspillett

2 hours ago

> Generally I've found a good portion of websites that block users with ad blockers tend to be the worst offenders with aggressive ads...

It is a common cycle: as they get more aggressive with trying to make the ad based business model work, more and more people are pushed to installing blockers. At this point there is a choice: find a less crappy business model (not an easy task, many have failed) or double down and get more iffy with the adverts and who you partner with to serve them. Eventually it gets to a point where such a high proportion or the viewers are using blockers and the next step in ads/stalking is a step too far even for them, and then the blocker blocking attempts start.

nicce

4 hours ago

> Just the other day I saw some meme about how if we saw as many ads taking over sites like we do now, that was a sure sign of having your machine getting infected with a virus.

That used to be the reality around decade ago for those who don’t know. Some applications changed the default front page of the browser, hijacked other websites to inject popups and so on

AlAjem

3 hours ago

>around decade ago

Two decades now actually. By 2016 we were already seeing the mass transition to browsing on phones instead of PCs, Windows 10 was already out and for reasons I am not quite sure of the old homepage and toolbar hijackers were on the decline and adblockers were on the rise although I am sure there were still people using malware ridden Internet Explorers for years after.

https://en.wikipedia.org/wiki/Browser_hijacking

nicce

3 hours ago

Ouch, already 11 years since the Windows 10 release.

kotaKat

3 hours ago

> But a recent update has caused them to block users with ad blockers.

I'm noticing that Ad-Shield is becoming a continual fight these days with pageloads that complete then they fly in with a "ha ha here's a fake error report turn off your adblocker aye?".

Every time their live support chat is open (why would the anti-adblock firm even offer one), I send them Goatse.

dspillett

2 hours ago

> Every time their live support chat is open […], I send them Goatse.

Unfortunately there is little chance that a human is anywhere near the other end of that chat these days, so your evil is somewhat wasted.

> (why would the anti-adblock firm even offer one)

They want to appear to care. And appear to be a company of real people, who you might like if you met in real life, who will be out of pocket if you don't look at their ads.

IanCal

an hour ago

If there is a person on the other end it’s not someone who has any link to the company, they’ll be an outsourced service, so all you’d be doing is sending graphic porn to a poorly paid worker.

It’s like screaming at a cashier because the supermarket head office made annoying changes to the company.

jimnotgym

5 hours ago

I'm going to agree.

I launched a website that is niche a few weeks ago, but totally unique. After a couple of weeks search console wakes up to tell me that they have indexed one page in 80. And that the top search term that is finding my site says exactly the name of one of my pages, yet Google is showing my homepage in search!

Way to help your users

mrweasel

4 hours ago

Yet you can be absolutely sure that Google is training their AI products on the same pages they do not want to show to their search users.

SyneRyder

2 hours ago

Have you added a robots.txt and a sitemap.xml, even an RSS file?

While I don't know if this will help with Google, I have my own tiny dumb search indexer, and a sitemap is by far the easiest way for me to index an entire website. I'll discover the sitemap from reading your robots.txt file. RSS helps too, but my indexer uses that mostly to find fresh pages without going through your entire sitemap again. I know Kagi's tiny Teclis indexer also uses RSS files for discovery for their indie web index.

einpoklum

4 hours ago

> Way to help your users

Dear Jim,

At Google, our intent is to help our stockholders, not our users.

Sincerely,

Alphabet inc.

phyzix5761

4 hours ago

That's every company

sph

4 hours ago

It’s not every company having the effective monopoly on what can be found (i.e. what exists) on the web.

phyzix5761

an hour ago

Every company is in it for the profits and benefits of its shareholders. Its idiotic to think that companies have benevolent intent toward their customers. If that wasn't the case they would go out of business.

latexr

37 minutes ago

Not every company is publicly traded, so they don’t have shares nor shareholders. Also, not every company is a massive behemoth who only cares about profit above all. Plenty of people start companies with the goal of providing something useful and good at fair prices and value their customer relationships.

We should really stop with this cynical view that every company is run by greedy bastards who have active contempt for their customers and using it to excuse the behaviour of the worst offenders.

accountrequired

3 hours ago

Whatever that github wiki mirror is doing, try that. You know the one, always at the top of search results...

miladyincontrol

7 hours ago

I dont know about this, I've been heading a wiki for a rather specific niche that was made about 1.5 years ago and it easily got most top results for it's focus in just a few months.

It sounds like the issue isnt for "independent wikis" but for wikis competing with an existing domain already hosting one on the topic, in this case fandom.com's.

cookmeplox

5 hours ago

What's the wiki? I'm always interested to get more data points on this, because it has definitely applied to wikis for brand new topics that aren't competing with anything on Fandom (it's not the same as the normal "duplicate content penalty" that people talk about)

rao-v

6 hours ago

As a bandaid solution we probably just need a good base domain for game wikis attached to a somewhat trustworthy foundation (that won’t sellout to Fandom in a week). Agree that wierdgloop.org is not … great

copper-float

5 hours ago

What's wrong with weird gloop? The RuneScape wikis are very high quality, same with the Minecraft wiki. Cook and others running are transparent and seem to have good intentions.

cookmeplox

5 hours ago

I think you can love what we're doing with the wikis, but also be candid that it's not an optimal reader-facing domain name. It was never really intended to be that - it's literally just named after a random item in RuneScape I discovered

But it ended up being the only reasonable place we owned where we could put Fortnite, Overwatch and Valheim wikis that wouldn't get crushed by this new Google jail situation. Of course the alternative is what the parent comment suggests ("probably just need a good base domain for game wikis") - but you have to get THAT domain out of Google jail first. Chicken and egg.

yreg

3 hours ago

Thank you so much for what you do.

> Of course the alternative is what the parent comment suggests ("probably just need a good base domain for game wikis") - but you have to get THAT domain out of Google jail first.

So why not get such domain, wait out until it is let out of Google Jail and then put all the wikis on subdomains?

I imagine you would quickly gain trust and people would immediately recognize the links and know that this is the place to go. As opposed to the current state where the wikis have separate domain names and for each game we need to learn which one is the "good one".

⸻

Second question: Any thoughts on liberating non-gaming wikis from fandom, such as TV show wikis?

Shank

3 hours ago

The domain is suboptimal. The group is great!

TylerE

3 hours ago

It's two english words both of which have negative conotations?

deaddodo

an hour ago

"gloop" is a neutral/funny word. The adjective "weird" is what makes it somewhat negative.

fakwandi_priv

6 hours ago

What’s wrong with wiki.gg?

dreadpiratewiki

5 hours ago

run by the same people that sold Gamepedia to Fandom, is owned by private equity, and has been rapidly increasing ads/monetization over the last couple years

josephjrobison

6 hours ago

The bar has been raised on all web content for Google to index, esp new domains like you say but even some subdomains.

That answer is quality content (higher than AI or human medians), links and pr and social activity and mentions, and then user experience.

microtherion

5 hours ago

And yet, I have to dig my way through pages upon pages of AI trash sites for many Google queries. Somehow THOSE don’t seem to have a problem getting indexed (Maybe they run a sufficient number of ads?).

willtemperley

an hour ago

Given none of the wikis pointed to in the article have ads, I'd say you're right.

charlieyu1

3 hours ago

Have they fixed the scam problem though?

watwut

5 hours ago

Google search results dont care about content quality. And did not for years already. It killed good quality blogs and sites long before the AI slop came in.

bariscan

2 hours ago

this is not only wiki, unfortunately google tries something at least for 6 months and "shadowing" is real thing. no manual action, no information, indexes here and valid; but no imp & clicks.

make some social noise. signals has always good impact.

KingMob

6 hours ago

This is a shame because Fandom is frequently terrible, but also consumes all the oxygen.

rob74

4 hours ago

One hand washes the other: they buy ads from Google (I think), so Google prefers them in the indexing process, so they get more traffic, so Google prefers them even more.

nephihaha

6 hours ago

It depends on which part. It is certainly heading down the same route as Wikipedia with all the problems that entails. You're usually better off on a smaller Fandom wiki as it won't have beeb taken over yet.

The James Bond wiki is particularly bad, but Memory Alpha seems to be better run.

pndy

an hour ago

> Memory Alpha seems to be better run.

Some few months ago I couldn't find actors credits for particular roles they had in Star Trek franchise. For whatever reasons these bits were removed but still were present within Wayback Machine.

Then again, in the past I've seen hostile take-overs of fan made wikis into wikia/fandom "infrastructure". In one case wiki was copied, in time articles were slightly edited and new content was added containing... unsolicited fan theories.

The other case: wiki was copied 1:1 due to some inside disagreements, then incorporated into fandom and in approx. 2 years abandoned. The original died as well but someone managed to restore it from Wayback - this time in read-only mode

rob74

4 hours ago

Well at least Wikipedia doesn't pester you with tons of ads that sometimes make it hard to even find the content (except for the occasional donation banner, but that's a minor annoyance compared to what Fandom does).

thaumasiotes

4 hours ago

> except for the occasional donation banner, but that's a minor annoyance compared to what Fandom does

Well, I guess that's true.

It's worth noting, though, that wikipedia's donation demands cover more than the entire screen on a phone.

account42

4 hours ago

It's also worth noting that both Wikipedia and Fandom were co-founded by Jimmy Wales.

thaumasiotes

4 hours ago

Well... why? Is he involved with either of them now? Is there a shared persisting influence of some kind?

einpoklum

4 hours ago

> It is certainly heading down the same route as Wikipedia with all the problems that entails.

Can you elaborate on what that means? i.e. what aspect of Wikipedia, and in what way Fandom is heading there?

sylware

28 minutes ago

It is much much worse than that: the real jail is the "web javascript" of the web engines of the whatwg cartel, well roughly speaking.

nephihaha

6 hours ago

Google has been removing blogs from its searches for years now. You have to search for them specifically. All part of the de-democratisation of the web.

LoganDark

5 hours ago

Anyone remember when "Fandom" was called Wikia? It really started going down the gutter when it started TikTokifying itself by trying to act as an entertainment hub. It was supposed to be a neutral resource for people to build community-driven knowledge bases but it turned into some real gamified garbage. I've never really been comfortable with the concept of addressing a "fandom" as one whole anyway, even in general, so the new act is just complete crap to me.

ButlerianJihad

5 hours ago

Fandom probably hosts more copyrighted images and media than the Internet Archive does.

Entertainment studios can often turn a blind eye to use of their properties when it's by fans, for fans, and promotes those franchises. They are not sending C&D to people making avatars, or cosplaying, or memeing their screengrabs into notoriety.

It is weird to think that the WMF and Wikia/Fandom had a common founder and origin.

fergie

5 hours ago

I am slightly on the fence about this. If the goal is simply to provide a good platform for people to create and share information with each other, then it shouldn't matter if the wiki is indexed in Google or not, in fact "invisibility" might even increase the quality of contributions.

OTOH if the goal is to monetize other people's contributions then yes, I totally get why the "Google jail" would be bad for that, but I'm just not sure that its a cause worth fighting for- that road basically leads to a new Fandom.

tuetuopay

4 hours ago

That's ignoring how video game wikis are used. People will google "<game> <place where I'm stuck>" and completely miss the wiki. Or "<game> <mechanic>". Etc. So reachability and proper indexing by search engines is a must.

I've hit this on a few wikis where, for some (at the time inexplicable) reason, I'd get only the crappy Fandom wiki with above queries, and a good wiki when searching only for "<game> wiki" after seeing a link to it on reddit.

When the reason is to move away from Fandom, get people what they are looking for, and perhaps attract contributors, this is an issue. Your average gamer won't route through the Wiki's main page, rather rely on the top Google search result for their issue.

billyp-rva

37 minutes ago

> That's ignoring how video game wikis are used. People will google "<game> <place where I'm stuck>" and completely miss the wiki. Or "<game> <mechanic>". Etc. So reachability and proper indexing by search engines is a must.

Well, it's a "must" because over the years people have been trained to do this, instead of going directly to a quality website they trust. However, this has also trained website owners to neglect their brand and focus only on SEO, leading to the ad-infested wiki sites we have today.

Google went through all of this many (many) years ago with the "content farm" problem, so this is nothing new, really.

cookmeplox

5 hours ago

I think this logic falls apart if you realize that wikis only "work" when you're able to get some of the readers to turn into long-term contributors, and losing 90% of your traffic by not showing up on Google is a great way to kill that funnel.

There's a ton of historical evidence of wiki migrations (1) starting out with all of the editors and none of the readers, (2) not doing anything to get those readers to the new site, and (3) ultimately losing the whole war because the reader->editor flow was still happening on the Fandom wiki. "Invisibility" is absolutely not desirable here, even if your only goal is maximizing the amount of good contributions.

fergie

an hour ago

Yes, but the point is that maybe in 2026 we should be looking at ways to pay contributors/creators, rather than steal from them (slightly hyperbolic, but you get the gist)

jdranczewski

4 hours ago

I think it's fair to say that if the wiki's intent is to share information with people, you want people (and not just the inner circle of those in the know) to be able to find it. And the way most people find things is on Google.

sersi

4 hours ago

Personally I liked it 10 years ago when I searched for something and Google actually delivered useful results that were related to what I was searching. Nowadays, when I search for anything, I mostly get blogspam, ai slop and useless articles or in the very best scenario, good content on a page infested with ads (luckily there's ublock origin).

So yeah, when my goal is to search for content that would be in a wiki, I'm ecstatic when I get an independent wiki over fandom. The Google Jail is the opposite of what I'd want.

Nowadays, I use Kagi which is better than Google but it's still not to the level of Google in 2010.