[SOLVED] robots.txt-alike broken for Arch bbs signatures?

Awebb · 2013-06-03 21:27:36

In a complicated seizure of self-glorification, I asked Google about ARDCAD (I wanted to see, if the respective thread can be found). Instead, I found this thread: https://bbs.archlinux.org/viewtopic.php?id=161877. The only ARDCAD in there is in litemotiv's signature. Now, normally, those signatures are not being scanned by crawlers like Google and we are very happy about that, because of all those people who insist on having "their specs" in the signature. I'd like to find out, whether this is an isolated incident or if there is anything to bug-report to the bug-tracker of the relatively bug-free fluxbb. Maybe ✪ is an attractor for crawlers?

Last edited by Awebb (2015-03-29 11:58:00)

karol · 2013-06-03 21:33:45

'site:bbs.archlinux.org ARDCAD' returns dozens of threads with litemotiv's signature.

litemotiv · 2013-06-03 21:38:04

I get about 27000 results for:

https://www.google.com/#q=site:bbs.archlinux.org+ardcad

Pretty old stuff too with ARDCAD signatures:

[SOLVED]ugly appearance on root applications (Page 1 ...
bbs.archlinux.org › ... › [SOLVED]ugly appearance on root applications
11 mrt. 2012 - 7 berichten - 2 auteurs
ARDCAD 2013 participant ✪. Offline. #3 2012-03-11 19:55:02 ... /home/<yourname>/.gtkrc-2.0. ✪ ARDCAD 2013 participant ✪. Offline ...
[SOLVED]ugly appearance on root applications (Page 1 ...
bbs.archlinux.org › ... › [SOLVED]ugly appearance on root applications
11 mrt. 2012 - 7 berichten - 2 auteurs
ARDCAD 2013 participant ✪. Offline. #3 2012-03-11 19:55:02. lgeek: Member ... ARDCAD 2013 participant ✪. Offline. #5 2012-03-11 20:05:48 ...
Looking for simple touch-to-command utility (Page 1 ...
bbs.archlinux.org › ... › Looking for simple touch-to-command utility
14 jan. 2012 - 7 berichten - 2 auteurs
Does anyone know of such a utility? ✪ ARDCAD 2013 participant ✪. Offline ... ARDCAD 2013 participant ✪. Offline. #4 2012-01-15 13:27:24 ...
the little things that make you go 'ooh, nice' (Page 61) / GNU ...
bbs.archlinux.org › ... › the little things that make you go 'ooh, nice'
7 dagen geleden - 6 berichten - 4 auteurs
Reinstalling Arch is not normal. On --force it is. --force, not even once. ✪ ARDCAD 2013 participant ✪. Offline ...

karol · 2013-06-03 21:41:57

litemotiv wrote:

Pretty old stuff too with ARDCAD signatures

I think that's because Google constantly reindexes the internet. If you want old pages, use the cached version or go to waybackmachine.

Awebb · 2013-06-04 07:47:12

Let's probe some more signatures and see, if it's just me and litemotiv.

I will more or less randomly choose signatures from the Arch bbs and run them through google.

This user - This Query - Arch bbs query: CHECK, google finds the sig.
This user - This query - Arch bbs query: CHECK, google finds the sig.

We now have three and a half examples of failed filtering of the signature. Now I'd like to know, whether this is a setting unset in the Arch bbs or something to request over at the fluxbb bugtracker.

EDIT: The only bug at the fluxbb bugtracker concerning signatures is related to images.

Last edited by Awebb (2013-06-04 07:58:57)

progandy · 2013-06-04 08:10:07

We now have three and a half examples of failed filtering of the signature. Now I'd like to know, whether this is a setting unset in the Arch bbs or something to request over at the fluxbb bugtracker.

There are two ways to hide the signature: for all guests (search engines as well as unregistered users, already possible) or only search engines (this is AFAIK not possible yet)

IsSuE · 2013-06-04 10:46:08

What's ARDCAD 2013 anyway?

fsckd · 2013-06-04 14:00:16

progandy wrote:

There are two ways to hide the signature: for all guests (search engines as well as unregistered users, already possible) or only search engines (this is AFAIK not possible yet)

Googlebot and other crawlers have specific user agents which identify them. I think it's worth a shot asking FluxBB folks to block sigs from appearing to search engines.

Awebb · 2013-06-04 17:22:39

IsSuE wrote:

What's ARDCAD 2013 anyway?

Heh. It's the Arch Registration Date Coincidence Appreciation Day.

Blocking the robots might be too much work. I once learned in an Ubuntu board, that one can block crawlers from indexing certain parts on a website by adding certain meta-tags (maybe in scheme.org). Signatures should just be outfitted with such tags. But I am no web developer. Could anyone have a look how this works, so we can assemble a patch and file a feature request over at FluxBB HQ?

karol · 2013-06-04 17:40:03

IsSuE wrote:

What's ARDCAD 2013 anyway?

https://bbs.archlinux.org/viewtopic.php … 6#p1249386

Awebb · 2013-06-04 17:45:15

Awareness, yes :-D

karol · 2013-06-04 18:00:50

Awebb wrote:

Awareness, yes :-D

I've spend a couple minutes looking for that 'Awareness' post, I even stared thinking that either
* you didn't post it and I remembered it wrong
* it has since been removed / edited

Due to our Arch Registration Date Coincidence, you're obviously forgiven ;P
I enjoyed reading that thread a second time :-)

I think that 'Appreciation' sounds even cooler.

SidK · 2013-06-04 18:12:10

Awebb wrote:

Blocking the robots might be too much work. I once learned in an Ubuntu board, that one can block crawlers from indexing certain parts on a website by adding certain meta-tags (maybe in scheme.org). Signatures should just be outfitted with such tags. But I am no web developer. Could anyone have a look how this works, so we can assemble a patch and file a feature request over at FluxBB HQ?

Sadly this cannot be done in a friendly way (http://stackoverflow.com/questions/1497 … -of-a-page). User agent detection is possible, but generally not thought of as a clean solution. The googleon/googleoff solution I think you found on the ubuntu forums is also debunked in that link. Google themselves say how entire pages can be tagged as non-indexable (https://support.google.com/webmasters/b … swer=93710), but not parts of pages.

http://webmasters.stackexchange.com/que … -of-a-page provides working solutions, but all of them are hacks and nothing I'd want to see in the fluxbb codebase.

Perhaps a cleaner solution is to not show signatures to any logged out users (which would include search engines).

Edit: reading the fluxbb code they have a global option to enable/disable signatures. But not whether one to flag if the signatures should only be shown the logged in users or for everyone.

Edit2: Solved it; FluxBB has a Guest user to which it assigns various settings (timezones, whether it sees signatures, language, themes etc). In the DB this is the user with ID=1, username=Guest.
If you set show_sig to 0 for the Guest user, then logged out users, including search engines, don't see any signatures.

UPDATE users SET show_sig=0 WHERE id=1;

Ideally this toggle would be created as an admin panel option, manually fiddling DB values is a slippery slope.

Last edited by SidK (2013-06-04 18:38:08)

Awebb · 2013-06-04 19:29:18

Thanks, SidK.

Can anybody confirm, whether google bots are used to index FluxBB, instead of a simple scan of the entire page?

SidK · 2013-06-05 12:38:24

I don't know what you mean by "a simple scan of the entire page". Google does all its web crawling using the google bots.

Awebb · 2015-03-29 11:57:47

litemotiv wrote:

I get about 27000 results for:
https://www.google.com/#q=site:bbs.archlinux.org+ardcad
Pretty old stuff too with ARDCAD signatures:

Old Thread cleanup day. This search now only shows this thread and the grrr thread. Either google has fixed itself or FluxBB has. Anyway, marked as solved.

Arch Linux

#1 2013-06-03 21:27:36

[SOLVED] robots.txt-alike broken for Arch bbs signatures?

#2 2013-06-03 21:33:45

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#3 2013-06-03 21:38:04

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#4 2013-06-03 21:41:57

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#5 2013-06-04 07:47:12

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#6 2013-06-04 08:10:07

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#7 2013-06-04 10:46:08

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#8 2013-06-04 14:00:16

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#9 2013-06-04 17:22:39

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#10 2013-06-04 17:40:03

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#11 2013-06-04 17:45:15

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#12 2013-06-04 18:00:50

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#13 2013-06-04 18:12:10

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#14 2013-06-04 19:29:18

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#15 2013-06-05 12:38:24

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

#16 2015-03-29 11:57:47

Re: [SOLVED] robots.txt-alike broken for Arch bbs signatures?

Board footer