Most of them are probably "search engine spiders"

Hmmm... UCOs? (Unidentified Crawling Objects) Wouldn't have thought there are that many around...

>300 guests really does seem excessive and downright spooky. Unless perhaps it's LLM bots?

Troubles coming from China seem never ending.

The problem now seems to be the sheer number of AI bots crawling forums (this forum and everyone else's, this is a GLOBAL issue). The other day I had to block the whole Alibaba Cloud as well as some other bot networks, as things were so bad that at one point we had over 1,500 bots crawling the forums (now reduced to about 40 or so).

Today things got so bad that despite no more than 160 or so bot "guests" online, when trying to access the forums the server was throwing the occasional "too many connections" SQL error. The Winstep's server Xeon CPU usage was pegged at 100% too.

A quick look allowed me to determine that the problematic traffic was all coming from the TenCent network. Ended up having to block the whole Tencent IP address space at the firewall level to fix the problem.

This literal invasion of AI crawling bots is so serious that in some cases it renders sites/forums completely inaccessible (the equivalent of a DDOS attack as was happening here today). They use fake user agents, obey no rules, make statistics unusable (e.g. how many people read a particular post), waste other people's money in bandwidth, etc, etc...

Sometimes the only way to solve the problem is to ban whole countries, that is how bad this is becoming.

For anyone interested, check the first 4 minutes of the following video:

As if a million reasons to hate (so-called) AI weren't enough!

What a mess!

Ok, since this discussion began, BMT Micro has added Alipay as a payment method, and I’ve also established a partnership with a Chinese reseller.

Given all that, I decided today to unblock China at the firewall level.

The moment I ran the script, the server’s bandwidth was instantly maxed out. For context, a brand‑new global release usually peaks at around 12% of total bandwidth per second. This time it hit 100% immediately. Everything slowed to a crawl — even my remote session became barely responsive. For a moment I genuinely thought I had broken something.

CPU usage looked normal, so it wasn’t obvious at first. Only when I checked bandwidth graphs did it click: total saturation.

Here’s my conclusion: the free version of Nexus has become so widespread in China that, even with the Winstep Update Manager staggering update checks across a full 24‑hour window (that's 86,400 seconds), the number of installations that happened to check for updates and started downloading it in the very first second was enough to completely overwhelm the server.

So I had no choice but block China again. I feel like I am sitting on top of a gold mine with a single spoon as my best tool to get to it lol

Looks like I am looking at CDN territory here.

Ok, so I forgot that if the Winstep Update Manager does not get a connection for some reason it will try again in 10 minutes and not in 24 hours. In practice this means ALL the installations of Nexus in China that are currently online will try to download the update within the same 10 minute window.

So, not as big of a gold mine as I initially imagined, but still not something to look down upon

EDIT: Looking at other numbers, about 200,000 installations of Nexus tried to download the update in the <10 minute window China was unblocked. So China has AT LEAST 200,000 unique installations of Nexus, and this is a low ball number as it does not take into account offline machines, etc

I unblocked China past 1:30 AM China time, so I opened the gate on China in the middle of the night for them. At that time most home PCs are off or sleeping. So, if 75% of home PCs were offline when I unblocked China, imagine what the actual install base might be.

Here talking to myself, but it's ok, some people might eventually read this and find it interesting.

Here is how I am on my way to try and solve this problem:

I've created and account at a CDN and I'm storing the setup files there. Then I am going to update the Winstep Update Manager so it

a) re-directs any downloads from China to the CDN instead of the Winstep server.

b) randomizes retry delays from the currently fixed 10 minute delay interval (which concentrated all the population of China using Winstep software into the same 10 minute window) and stagger that number randomly based on number of retries.

Now, since traffic from the CDN is paid per GB, at least for the moment I am only going to be redirecting traffic from China to them. Global traffic will still go to the Winstep server.

I assume all the 200,000 Chinese users who hit the small current version configuration file in the 10 minutes China was unblocked were all trying to upgrade Nexus to the latest v26.4 version. Because China has been blocked for over a year and therefore missed quite a few updates, the Winstep Update Manager was trying to download the full 40 MB setup file rather than the smaller 16 MB update file. This completely trashed the server.

Anyway, for this to work, Chinese users must first update the Winstep Update Manager, as this is required in order to implement the CDN re-direction and staggered retries (useful in case I have to block China again or the Winstep server goes offline for a while for whatever reason).

Fortunately updates of the Winstep Update Manager ALWAYS take precedence over updates of the main application (i.e. if the Update Manager sees an update to itself, it will download that first, before checking for any other available updates).

Now, here is the problem: even though the Update Manager update is a much smaller 2 MB file, we will still have at least 200,000 users trying to get to it in the space of 10 minutes. And this is the classic bootstrap problem.

That's 400 GB in 10 minutes, or 5.3 Gbit/s over a 1Gbps port. The server still cannot handle that, at a constant 1Gpbs usage it will take a *minimum* of 53 minutes to serve all the 200,000 users. The server will be almost unusable during that time frame but that is the price to pay to finally get this under control.

Holy crap! (Pardon my french. ) What a tsunami of connections. And the actual user base there of the free Nexus could well be ~800K! Nice potential market though.

Sorry, didn't see this till it was too late. Sheesh, what a bummer! Well, needs must, really, you got to grin and bear it I guess.

Ok, started gradually unblocking China at 7:30 PM my time yesterday (2:30 AM in China) and finished unblocking the whole of China today at around 3 PM my time (10 PM in China).

Because it was gradual, I managed to maintain bandwidth utilization levels on the Winstep server under control.

For now the numbers are like this:

About 400,000 unique Chinese users checked for updates. From those, only about 1 in 4 actually downloaded the updates (users need to manually confirm the download), so at this time I have over 92,500 fully updated Chinese users of the free version of Nexus (that corresponds to 3.46 TB of data provided by the CDN). This number will keep increasing for the next few days.

Also got a handful of "buy me a beer" gifts from China. Always better than nothing.

Maxmind, the GeoIP provider, already billed me for another "batch" of queries. Let's hope I do not end up regretting unblocking China.

Yeah, I have been getting server restricted due to number of connections messages so had assumed that you had been opening the China floodgates. Perhaps you may wish to prioritise traffic from non-China locations?

Yes, the restriction is intended to apply only to guest access. However, when your session expires, your next request may briefly be treated as unauthenticated until phpBB re-establishes your login from the persistent cookie. During that moment, the request can effectively be handled as a guest - I’ve seen this happen myself as well.

If you run into this, try refreshing the page a few times until it goes through. Once it does, it shouldn’t occur again for the remainder of that session.

Recently I had to lower the guest connection limit significantly, as bots have been consuming considerably more server CPU than before.



That’s not something I can easily do in a clean or reliable way, and the issue isn’t really the Chinese traffic itself but the ever-present bots - the recent increase in traffic just pushed things over the edge a bit.

At the moment there’s also a wave of upgrades coming from China (currently over 170,000 ), so traffic is temporarily higher than usual. Once that settles down, things should return to normal.