PDA

View Full Version : Visits By Strange Bots


Dave
05-02-2008, 07:46
Hi,

I have been looking at my webstats and seem to be getting regular visits from two different IP addresses being
208.99.195.54
38.98.120.71

For example today 208.99.195.54 has already visited my site 8 times and looked at over 300 pages. 38.98.120.71 has visted 3 times today and viewed over 500 pages!

From searching the web 208.99.195.54 seems to be a site everyone is banning using the robots.txt file. 38.98.120.71 is an american price comparison site called shopwikki.com

Is anybody else getting regular visits from these ip addresses?

I would appreciate it if Richard could do a bit of investigation and consider adding these IP's to robots.txt to ban them? I am pretty sure they are not any of the main search engines and are concerned about any potential bandwidth problems they may cause.

Many thanks,

Dave
Andrew
05-02-2008, 10:58
Dave,

Yes, we have been visited by both of those sites as well, although nowhere near the amount of page visits. Probably because you have much more content than we currently do??

Andy

Needundies.com
Dave
05-02-2008, 11:11
Dave,

Yes, we have been visited by both of those sites as well, although nowhere near the amount of page visits. Probably because you have much more content than we currently do??

Andy

Needundies.com

Thanks Andy,

We currently have something like 1,000 products loaded and are only about 1/3 way through so have some way to go yet!
Hopefully Richard will be able to just block these IP addresses

Dave
petad
05-02-2008, 17:42
Yes, we've been getting visits too. Often they place anywhere between 20 and 30 items in the basket too, which must be screwing up our abandoned sales stats, amongst other things. Most recently (last two or three days) it has been shopwiki.com, but on a different IP to the one Dave has reported - 8.11.2.98

Pete
Dave
06-02-2008, 07:47
Yes, we've been getting visits too. Often they place anywhere between 20 and 30 items in the basket too, which must be screwing up our abandoned sales stats, amongst other things. Most recently (last two or three days) it has been shopwiki.com, but on a different IP to the one Dave has reported - 8.11.2.98

Pete

Pete,

Interesting one as can not find much on the web for 8.11.2.98. Perhaps they have found so many people blocking the other IP they are now coming in using a different one.

We have not getting items added to the basket but large number of page views and visits.

Dave
Rich
06-02-2008, 11:39
These robots are an absolute pain, no doubt, but there's very little we can do to discourage them from visiting. They're very unlikely to be following what's written in robots.txt as this is an advisory file - it doesn't have any enforcement qualities. These bots are quite aggressive and are unlikely to follow Internet protocol conventions.

On the subject of banning IP addresses - we've considered this on many occasions but have so far decided against it. The problem is, you generally don't know which IPs affect large numbers of users. Not every user has a unique IP address.

For example - ban an AOL IP and you can inadvertently ban half of Europe (as they use "proxy" addresses for many users). In addition, banning IPs is reasonably ineffective - dynamic IPs change regularly, and delegations of bots can have any number of IP addresses between them.

What you often find is that bot activity comes and goes. The bots that are very active right now might well reduce or cease activity before too long. Let's hope so!
Dave
06-02-2008, 12:46
Thanks Richard, Will monitor it and hopefully it will die a natural death:)

Regards,

Dave
petad
07-02-2008, 11:56
What you often find is that bot activity comes and goes. The bots that are very active right now might well reduce or cease activity before too long. Let's hope so!
I agree that banning IP addresses is a non-starter.

I was just wonding why these sites/bots are doing this? Do they gather usability stats of sites or something? I can't see the point...

Pete
paul
07-02-2008, 13:07
Other than bandwidth, if bots are not malicious, arent they a good thing? eg shopwiki (referred to above) appears to be a site which may list and link your product pages. And how often are they malicious and how do we know?
Rich
16-09-2008, 00:07
On the subject of banning IP addresses - we've considered this on many occasions but have so far decided against it. The problem is, you generally don't know which IPs affect large numbers of users. Not every user has a unique IP address.

For example - ban an AOL IP and you can inadvertently ban half of Europe (as they use "proxy" addresses for many users). In addition, banning IPs is reasonably ineffective - dynamic IPs change regularly, and delegations of bots can have any number of IP addresses between them.

Despite my earlier postings on this subject ... it is now possible to ban IP addresses (either full or partial) via the Admin Console. The field can be found towards the bottom of Site -> Configuration: Site.

I would, however, ask that you heed the words above - and treat the feature with due care! :eek:
Dave
24-09-2008, 07:38
Rich,

On one of our sites we are getting visits from fatlens.com which looks like an american search engine/shopping comparison site.

They currently have 12 sessions (if this is the right word) visiting our site and each session has over 10 items in the shopping basket

Looking at the ip address it is only the last two digits which change and was wondering about doing a partial block on the ip address :eek:

The easiest solution would be to block the bot fatbot in the robots.txt file (assuming they will honour this) and was wondering if you would consider doing this?
Mike
24-09-2008, 15:53
Dave

I think Rich has set up the IP address blocker to work with either full or partial IP addresses. I've tried this putting in only the first three sets of numbers and it seems to work OK.
Dave
24-09-2008, 16:11
Dave

I think Rich has set up the IP address blocker to work with either full or partial IP addresses. I've tried this putting in only the first three sets of numbers and it seems to work OK.

Thanks Mike, will give this a try and see how we get on. Fatbot has been on the site nearly all day adding products to the basket and it does make a mess of your stats!
petad
24-09-2008, 17:53
Thanks Mike, will give this a try and see how we get on. Fatbot has been on the site nearly all day adding products to the basket and it does make a mess of your stats!
Hi Dave

I've never seen them before, and today we have loads of visits from fatlens.com, which I assume is the same thing (last two characters of the IP are different, as you said).

I'm in two mind about blocking them, as it depends what they are doing. If they are indexing our site and products, then that would be a good thing, I think...?

Pete

Edit: it looks like th ebot is on behalf of thefind.com, which seems to have indexed products from half the planet.
little-linguist
24-09-2008, 21:30
Hi Dave

and today we have loads of visits from fatlens.com.

We have too. The fatlens bot must be knackered after all that crawling all day!
petad
24-09-2008, 21:48
We have too. The fatlens bot must be knackered after all that crawling all day!
LOL l-l

...mind you, he's not that tired, as he's still at it on our site, hours later. Wish I had the same stamina...
Dave
25-09-2008, 08:31
Funny how fatlens has been going since 2005 and its only now they are trawling through our sites! They are an american price comparison site and seem to be just for the american market because you can narrow your search to number of miles from state/zip code

We did a partial block on the IP address yesterday and it seems to have done the trick and they have gone away and our stats look back to normal this morning

We don't export so there would be little point in having our products listed on their site
pinkypie
30-11-2008, 21:53
JUst to give you the heads up that IP 208.99.195.54 is trawling again! I think I am going to block them, they have had more page views than anyone else today and they are currently all of my 3 'visitors'.
Gillf
05-10-2009, 11:05
Hi,

Further info on Fatbot for everyone. Although it is an American comparison site, I believe that they are also behind the site called "The Find".... there is a UK version of the The Find which might be why they have only just become active in BluePark sites this year. (if the UK version of the find is new - I have not checked that out.) I know they are a pain in the neck and I actually contacted them to complain. They said their crawling was discreet! I'd hate to see them when they are not being discreet. All that said we have had orders that came in from The Find, so we are putting up with them for the moment.

All the best
Gill
Lola Luna - The Best French G-Strings (http://www.ever-so-sexy.com/lola-luna-micro-g-strings.html)