No choice but to block all traffic coming from China

winstep · **Joined:** Thu Feb 26, 2004 8:30 pm **Posts:** 12258

Most of them are probably "search engine spiders"

nexter · **Posted:** Fri Sep 27, 2024 8:41 pm

winstep wrote:

Most of them are probably "search engine spiders"

Hmmm... UCOs? (Unidentified Crawling Objects)

Wouldn't have thought there are that many around...

nexter · **Posted:** Tue Oct 01, 2024 4:01 am

>300 guests really does seem excessive and downright spooky. Unless perhaps it's LLM bots?

winstep · **Joined:** Thu Feb 26, 2004 8:30 pm **Posts:** 12258

Troubles coming from China seem never ending.

The problem now seems to be the sheer number of AI bots crawling forums (this forum and everyone else's, this is a GLOBAL issue). The other day I had to block the whole Alibaba Cloud as well as some other bot networks, as things were so bad that at one point we had over 1,500 bots crawling the forums (now reduced to about 40 or so).

Today things got so bad that despite no more than 160 or so bot "guests" online, when trying to access the forums the server was throwing the occasional "too many connections" SQL error. The Winstep's server Xeon CPU usage was pegged at 100% too.

A quick look allowed me to determine that the problematic traffic was all coming from the TenCent network. Ended up having to block the whole Tencent IP address space at the firewall level to fix the problem.

This literal invasion of AI crawling bots is so serious that in some cases it renders sites/forums completely inaccessible (the equivalent of a DDOS attack as was happening here today). They use fake user agents, obey no rules, make statistics unusable (e.g. how many people read a particular post), waste other people's money in bandwidth, etc, etc...

Sometimes the only way to solve the problem is to ban whole countries, that is how bad this is becoming.

For anyone interested, check the first 4 minutes of the following video:

nexter · **Posted:** Tue May 20, 2025 8:56 pm

winstep wrote:

Troubles coming from China seem never ending.

The problem now seems to be the sheer number of AI bots crawling forums (this forum and everyone else's, this is a GLOBAL issue). The other day I had to block the whole Alibaba Cloud as well as some other bot networks, as things were so bad that at one point we had over 1,500 bots crawling the forums (now reduced to about 40 or so).

Today things got so bad that despite no more than 160 or so bot "guests" online, when trying to access the forums the server was throwing the occasional "too many connections" SQL error. The Winstep's server Xeon CPU usage was pegged at 100% too.

A quick look allowed me to determine that the problematic traffic was all coming from the TenCent network. Ended up having to block the whole Tencent IP address space at the firewall level to fix the problem.
[
This literal invasion of AI crawling bots is so serious that in some cases it renders sites/forums completely inaccessible (the equivalent of a DDOS attack as was happening here today). They use fake user agents, obey no rules, make statistics unusable (e.g. how many people read a particular post), waste other people's money in bandwidth, etc, etc...

Sometimes the only way to solve the problem is to ban whole countries, that is how bad this is becoming.

For anyone interested, check the first 4 minutes of the following video:

As if a million reasons to hate (so-called) AI weren't enough! :twisted:

What a mess!

Winstep

Software Technologies

Winstep Forums

No choice but to block all traffic coming from China