Winstep

Software Technologies


 Winstep Forums


Print view
Board index : Winstep Forums : General Discussion  [ 199 posts ] Go to page Previous  1 ... 6, 7, 8, 9, 10, 11, 12 ... 14  Next
Author Message
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Thu Sep 03, 2026 5:41 pm 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
winstep wrote:
What is clear is that the code isn’t at fault, exactly as I suspected. I could optimize every instruction until I turned blue, and this underlying issue would still show up.

Preliminary analysis of normal and high-CPU traces shows that the problem is not one badly written or runaway section of WorkShelf code. During the abnormal state, **the same ordinary routines** suddenly consume roughly ***four times*** as much CPU across the board.


Anyway, I need to go back to focus on this. It only happens when running under Windows 11, not under previous Windows versions, and at least the preliminary tests show that this is NOT the fault of the Winstep software itself but something external to it.

If it was an issue with the Winstep application itself, only ONE of the code paths would suddenly be using four times as much CPU as before and we would have found our smoking gun - what we are witnessing here is suddenly *everything* taking four times more CPU than before.

Same code, same actions, but taking much longer - or executing four times as often. I refuse to believe this latest theory but it really is as if the timers were firing 4 times more often that they should - more likely something in Windows is blocking execution making the same work take four times longer than it normally would, for reasons outside the control of the Winstep application.

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies


Back to top
 Profile WWW 
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Fri Sep 04, 2026 12:58 pm 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
I am determined to get to the bottom of this, even though it is proving extremely difficult to diagnose - even an AI analyzing both the low-CPU and high-CPU etl traces was unable to determine exactly what is going on, only that the same code was suddenly either taking 4 times longer to execute or being executed 4 times as often.

I added some temporary instrumentation to WorkShelf to measure the actual frequency and duration of the main timer procedures.

In the current LOW CPU state, the 10 ms timers are actually firing on average every 15.625 ms, with occasional intervals below 10 ms and maxima around 27-28 ms. The 1-second timers also show normal scheduling jitter of roughly ±10 ms.

Importantly, there was no timer reentrancy at all during the test.

So this gives us a good baseline. The real test will be to capture the same numbers when WorkShelf enters the abnormal high-CPU state and compare the two.

Stay tuned.

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies


Back to top
 Profile WWW 
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Fri Sep 04, 2026 10:25 pm 
Offline
User avatar

Joined: Thu Sep 08, 2016 10:10 pm
Posts: 861
winstep wrote:
I am determined to get to the bottom of this,

Stay tuned.

This might be useless to you but there is a possibility their updates which are notorious for stopping things working or adding problems that were not there before could be part of the issue. I know it is a pain in the rear, but possibly looking at a Windows list of update history might present something.
Anyway you know what you are doing far better than I could hope to, but I know their updates sometimes are not tested well enough before release and it wouldn't surprise me if this was at least part of the problem.

_________________
"They teach you there's a boundary line to music. But, man, there's no boundary line to art."
-Charlie Parker

“We can't expect a blind man to appreciate beautiful patterns or a deaf man to listen to bells and drums.”
― Zhuangzi


Back to top
 Profile  
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Sat Sep 05, 2026 5:00 am 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
It's not because of an update. There is something in Windows 11 that causes this and has been doing so since pretty much the beginning, because it does not happen under Windows 10.

Anyway, I'm still waiting for WorkShelf or the system to enter the "high CPU usage" phase. Naturally, now that I have all the timer profilers in place and I'm actually ready to catch it, it's taking its own sweet time. :D

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies


Back to top
 Profile WWW 
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Sat Sep 05, 2026 4:38 pm 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
What’s Really Going On With WorkShelf’s CPU Usage

WorkShelf isn't experiencing a traditional CPU spike where it gets stuck doing heavy math or processing a massive loop. Instead, it is suffering from "death by a thousand context switches" caused by a traffic jam inside Windows' core UI subsystem.

Here is a plain-English breakdown of what the ETL analysis is actually telling us.

1. The True Cause of the CPU Usage (The Traffic Jam)

WorkShelf's main UI thread is waking up over 1,000 times per second. 93% of the time it is paused, the wait reason is WrResource. In the context of the Win32k stacks seen in the trace, WrResource almost exclusively means one thing: The Global USER Critical Section.

Windows has a global lock for certain core GUI operations (like moving windows, checking system metrics, or querying window handles). Only one thread in the entire operating system can hold this lock at a time.

WorkShelf is constantly trying to do something UI-related - likely checking a state, polling a handle, or querying the environment. Every time it does, it slams into this global lock because another app (like iCUE, Photoshop, or Explorer) is currently using it. WorkShelf is forced to wait, goes to sleep, wakes up when the other app finishes, executes for a minuscule 85 microseconds, and then immediately hits the lock again.

All that CPU usage we see (1-1.7% on a 24-core beast) is just the overhead of the Windows kernel constantly parking and waking the WorkShelf thread 1,000 times a second, combined with those tiny micro-executions.

2. Why Other Apps Show Up in the Trace

The analysis notes that apps like iCUE, ArgusMonitor, and Photoshop are the ones "waking" WorkShelf. This does not mean they are communicating with the software.

Imagine a single-stall public restroom (the Windows USER lock). WorkShelf is waiting in line. iCUE is inside. When iCUE opens the door (releases the lock), WorkShelf is notified that it can now enter. Because apps like iCUE and ArgusMonitor heavily poll the system for hardware stats or draw custom overlays, they are constantly grabbing this lock, forcing WorkShelf into a start-and-stop stutter.

3. Why Restarting WorkShelf Doesn't Fix It

If restarting the application doesn't drop the CPU usage, the problem isn't a memory leak or a stuck variable inside the code. The problem is environmental. WorkShelf is reacting to a persistent state in the operating system.

4. The Timer Mystery is Solved

I had also previously noticed that the timers (which should fire at ~15.6ms intervals) were sometimes firing at <10ms intervals, making it look like timers were running "too fast."

The ETL confirms this is an illusion caused by the lock delays.

Expected: Timer fires at 0ms, 15ms, 30ms.

Reality: The 15ms timer gets blocked by the WrResource lock for 7 milliseconds. It finally executes at 22ms. The next timer still fires on schedule at 30ms.

Result: I measured the gap between the delayed execution (22ms) and the normal one (30ms) and saw an impossible 8ms gap. In reality the timers aren't fast; the thread scheduling is just lagging.

5. WorkShelf is trapped in a 1,000Hz loop

The ETL showed WorkShelf waking up and hitting the Win32k USER lock over 1,000 times per second. That is not your standard 15ms VB timer running its normal idle checks. A 15ms timer would max out at ~66 wakeups per second.

To hit 1,000 wakeups a second, WorkShelf's main thread is either stuck in a tight iterative loop (like a DoEvents loop or a highly aggressive polling fallback - but that be eliminated as a stuck DoEvents loop would not survive an application restart) constantly querying a Win32 API, or its message pump is being absolutely flooded with messages it is forced to process.

6. TL,DR;

The only sure thing so far is that this is not WorkShelf's fault, but something external in the Windows environment that is either causing it to constantly block and/or "wake up" 1,000 times per second.

Since this high CPU usage state is experienced by other users (so not something specific to my system + 3rd party applications I normally use here) and only happens under Windows 11 but not Windows 10, it must be related to something the OS is doing. Even if I do find out what it is, there is no guarantee I can fix it since it might not be "mine to fix"

Analysis continues.

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies


Back to top
 Profile WWW 
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Sat Sep 05, 2026 6:23 pm 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
I've finally been able to compare very detailed Windows performance traces taken while WorkShelf was in the normal CPU state and while Windows was reporting roughly 3-4 times higher CPU usage.

The surprising result is that WorkShelf does not actually appear to be doing 3-4 times more work in the high-CPU state.

The traces record exactly when the WorkShelf thread is physically running on the CPU and what code it is executing while doing so. In both states the thread runs for approximately the same amount of time, performs essentially the same GDI+/graphics work, makes approximately the same number of Windows GUI calls, and spends its time in almost exactly the same WorkShelf routines.

In other words, there is no runaway timer, infinite loop, message flood, graphics software-rendering fallback, or rogue background thread that suddenly appears after sleep.

Nevertheless, Windows Task Manager, Process Explorer and even WorkShelf's own CPU meter can report a much higher percentage after certain sleep/resume cycles. Since all of those ultimately depend on CPU accounting information supplied by Windows, the evidence suggests that something in Windows' CPU/power-management accounting changes after resume, causing the same WorkShelf workload to be charged as substantially more CPU usage.

This also explains one of the strangest symptoms: restarting WorkShelf does not fix the problem. A completely new WorkShelf process immediately shows the same elevated CPU usage because the abnormal state belongs to Windows, not to WorkShelf. Another sleep/resume cycle can then make the reported CPU usage return to normal again.

So, somewhat counter-intuitively, what appears to be WorkShelf suddenly consuming much more CPU seems to be primarily a Windows reporting/accounting issue rather than WorkShelf actually executing several times more code.

I'm still investigating exactly which Windows power-management mechanism changes after resume, but the performance traces now make it very difficult to blame the increase on WorkShelf's own code.

TL,DR; *Apparently* WorkShellf is NOT REALLY using more CPU, it only APPEARS to be doing so. There is one final test that can confirm or deny this.

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies


Back to top
 Profile WWW 
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Mon Sep 07, 2026 6:06 am 
Offline
User avatar

Joined: Thu Sep 08, 2016 10:10 pm
Posts: 861
winstep wrote:
Apparently* WorkShellf is NOT REALLY using more CPU, it only APPEARS to be doing so. There is one final test that can confirm or deny this.


Looking at everything you have tested so far and the incredible detail you went to in order to test this thoroughly, as an outsider, it is looking like the issue lies with Windows O/S Processes reading and/or Task manager. Like a phantom pregnancy.

_________________
"They teach you there's a boundary line to music. But, man, there's no boundary line to art."
-Charlie Parker

“We can't expect a blind man to appreciate beautiful patterns or a deaf man to listen to bells and drums.”
― Zhuangzi


Back to top
 Profile  
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Mon Sep 07, 2026 10:13 am 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
I’ve been doing some more profiling on the occasional high CPU usage issue after sleep/resume.

Again, the normal WorkShelf code itself does not appear to be the problem. I measured the main timers directly in both the normal low-CPU state and the abnormal >1% state, and they are running at essentially the same frequency and taking roughly the same amount of time to execute.

I also measured the actual Windows message traffic handled by WorkShelf. Again, the amount of work is basically the same in both states - in fact, the normal low-CPU state was processing slightly more timer messages per second.

Despite that, the main WorkShelf thread was using almost 6 times more CPU in the bad state.

Even more importantly, the PC went to sleep again and, after resuming, WorkShelf immediately returned to its normal <0.3% CPU usage without restarting the program.

So at this point it is becoming increasingly clear that this is related to some Windows state being changed/reset during sleep/resume, rather than WorkShelf suddenly doing more work.

That said, unlike what I assumed in my previous post, the increased CPU usage does appear to be real from Windows' point of view - the thread is being charged significantly more CPU execution time in the bad state.

The paradox - and what makes this investigation so frustrating - is why. The timer and message profiles show that WorkShelf is doing essentially the same amount of work in both states: virtually identical timer frequencies and roughly the same number of Windows timer messages being processed (~579 vs ~597 per second).

So there are currently two main possibilities:

1. Windows or the runtime is introducing additional overhead somewhere in the thread’s execution path after sleep/resume, making the same workload genuinely more expensive to process.

2. Less likely, but still possible, some processor power-management or timing/accounting state after resume is causing Windows to account for the same physical workload differently than it normally does.

There is one interesting clue in that direction: the CPU cycle measurements are radically different between the two states, which suggests processor frequency/power-state behavior may also be involved.

Either way, the application workload itself remains essentially unchanged. What changes is how expensive Windows makes that workload appear - or actually be - after certain sleep/resume cycles.

The next step is to work out which of those two explanations is the real one.

TL;DR: Returning from sleep seems to be toggling some Windows/CPU state that can make WorkShelf use - or at least be charged for - much more CPU while doing essentially the same amount of work.

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies


Back to top
 Profile WWW 
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Mon Sep 07, 2026 1:08 pm 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
The P-Core vs. E-Core theory disproven

Just to add: initially I had a theory that, since the intermittent higher CPU usage only seemed to happen under Windows 11 - and Windows 11 generally runs on newer CPUs such as Intel's 12th Gen hybrid processors - perhaps Windows was sometimes scheduling WorkShelf on one of the slower E-cores.

I tested this by forcing WorkShelf to run on a single E-core. CPU usage increased indeed, but only slightly, from roughly <=0.1% to around 0.3%.

The profiler code confirmed the same thing: CPU time rose from about 50 ms/sec to 81 ms/sec - noticeable, but nowhere near the ~288 ms/sec seen in the abnormal state.

So simply running WorkShelf on an E-core cannot explain the huge post-resume CPU increase.

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies


Back to top
 Profile WWW 
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Mon Sep 07, 2026 10:59 pm 
Offline
User avatar

Joined: Thu Sep 08, 2016 10:10 pm
Posts: 861
Unfortunately, knowing your app is not experiencing a problem, and also proving it isn't a problem, and also telling and showing people clearly, does not stop some people believing that Windows is right.

_________________
"They teach you there's a boundary line to music. But, man, there's no boundary line to art."
-Charlie Parker

“We can't expect a blind man to appreciate beautiful patterns or a deaf man to listen to bells and drums.”
― Zhuangzi


Back to top
 Profile  
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Tue Sep 08, 2026 10:24 am 
Offline
User avatar

Joined: Wed May 15, 2019 1:47 pm
Posts: 43
This seems to be the issue i posted about when 24H2 was released. Or is this different that effects all Windows 11 builds.


Back to top
 Profile  
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Tue Sep 08, 2026 10:54 am 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
Mystery solved: the intermittent high CPU usage is a Windows reporting bug

After quite a bit of profiling, tracing and comparing the normal and abnormal states, I think we can finally put this one to bed.

WorkShelf is not actually using all that extra CPU.

The application is entirely innocent. What appears to be happening is that, after some sleep/resume cycles, Windows starts incorrectly accounting for the amount of CPU time used by the WorkShelf thread. Another sleep/resume cycle can just as easily put Windows back into the correct state again.

This explains one of the strangest aspects of this problem: WorkShelf could be sitting at its normal <0.3% CPU usage, the PC would go to sleep, and after waking up WorkShelf could suddenly show >1%. Conversely, another sleep/resume cycle could make the problem disappear again - without WorkShelf ever being restarted.

Restarting WorkShelf while Windows was in the "bad" state did not help either - the new instance would immediately show the same high CPU usage.

Over the last few days I instrumented pretty much everything that could reasonably be responsible. I measured how often WorkShelf's timers were firing, how long the timer routines took to execute, how many Windows messages were being processed, animation frequency, etc.

The workload was essentially identical in both states.

The final proof came from taking a low-level Windows ETW trace while WorkShelf was displaying >1% CPU usage.

At the same time, the normal Windows CPU accounting functions - the same type of counters used to calculate the CPU percentage shown by applications - claimed that the main WorkShelf thread was consuming about 316 ms of CPU time per second.

However, the Windows scheduler trace, which records when the thread is physically placed on and removed from a processor, showed only about 116 ms per second of actual processor execution.

In other words, Windows was charging the thread roughly three times more CPU time than it was actually receiving from the processor.

That also explains why all the previous profiling looked so contradictory: Windows said CPU usage had shot through the roof, while the application itself was still doing almost exactly the same amount of work.

We've actually seen this Windows bug before

This immediately rang a very large bell.

Windows 11 22H2 introduced a CPU accounting bug which caused incorrect per-process CPU usage values. It did not only affect Winstep applications - it affected third-party applications in general and even Windows' own Task Manager when displaying per-process CPU information.

The issue was investigated independently back in 2023 and was eventually traced to Windows' timer/CPU accounting behaviour. Changing the Windows SerializeTimerExpiration setting corrected the reported CPU usage on affected systems.

Winstep actually added a system-wide workaround for this in version 23.11. The CPU Meter settings included a "Serialize Timer Expiration" option which could correct CPU reporting not only in Winstep applications but in other applications and Task Manager as well.

Later versions of Windows 11 appeared to have fixed the underlying problem, so this workaround stopped being necessary.

Apparently, however, some variation of that same ugly Windows bug has raised its head again.

The difference this time is that the broken accounting state seems to be intermittent. A sleep/resume cycle appears capable of flipping Windows between correct and incorrect CPU accounting states.

So the >1% value you sometimes see after resuming from sleep is essentially an illusion generated by Windows' CPU accounting machinery. WorkShelf itself has not suddenly started doing three or four times more work.

Whether this is literally the exact same SerializeTimerExpiration bug or a newer variation of it is almost academic at this point. The tracing has already established the important part beyond reasonable doubt:

the additional CPU usage is not coming from WorkShelf. It is Windows incorrectly reporting how much CPU time the application is consuming.

Which also means I could optimize WorkShelf until I turned blue in the face and it would make no difference to this particular issue. :-)

TL;DR: WorkShelf is not actually using the extra CPU. After certain sleep/resume cycles, Windows 11 appears to fall back into the same kind of CPU accounting bug previously associated with SerializeTimerExpiration, causing Task Manager and other applications to report roughly 3× more CPU time than the thread is actually receiving. Another sleep/resume cycle can flip Windows back to reporting the correct values.

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies


Back to top
 Profile WWW 
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Tue Sep 08, 2026 11:09 am 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
PumaUK wrote:
This seems to be the issue i posted about when 24H2 was released.


It is... that is how long I have been chasing this.

I've finally confirmed that it is a bug in Windows (see above), not an issue with the application itself. Unfortunately, that is not how users experiencing the apparent high CPU usage are going to see it.

Some good came out of all this, though: while chasing the problem I tightened and optimized pretty much every routine I could think of, so WorkShelf's actual CPU usage - the REAL one, not the one Windows is lying about - is now lower than it has ever been.

Case closed.

Sick of Microsoft and their lack of proper QA testing and their complete disregard for the users who (still) use Windows.

Being a single developer I at least have an excuse, but a multi-billion dollar company firing their entire QA department some years ago, turning their users into guinea pigs and then failing to listen to them when they report issues?!

Every empire eventually falls, and Microsoft will be no exception. Given the current hardware market, Microsoft’s continued insistence on requiring newer hardware for Windows 11, its increasing push to lock users more tightly into the OS and subscription-based services, and the growing number of people moving to Linux, I’d say that process may already have started.

Even though my business depends on Windows, I certainly won’t shed a tear when it happens.

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies


Back to top
 Profile WWW 
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Tue Sep 08, 2026 1:59 pm 
Offline
User avatar

Joined: Thu Sep 08, 2016 10:10 pm
Posts: 861
winstep wrote:

Every empire eventually falls, and Microsoft will be no exception. Given the current hardware market, Microsoft’s continued insistence on requiring newer hardware for Windows 11, its increasing push to lock users more tightly into the OS and subscription-based services, and the growing number of people moving to Linux, I’d say that process may already have started.


Until the very large corporate companies with Microsoft Select / Gold partnerships etcetera move to Linux and not just their servers, but every workstation, and it is compatible with every application their company needs which currently with MS comprise of more sales than General Pleb public, this will not occur, probably not in our lifetimes. While Virtual environments in speed and stability are definitely much better now, they still are not quite as good as they need to be in workstations which Linux would need to do to cover all the x64 and x86 apps companies use.

_________________
"They teach you there's a boundary line to music. But, man, there's no boundary line to art."
-Charlie Parker

“We can't expect a blind man to appreciate beautiful patterns or a deaf man to listen to bells and drums.”
― Zhuangzi


Back to top
 Profile  
 
 Post subject: Re: So, what's next after v26.6 ?
PostPosted: Tue Sep 08, 2026 3:10 pm 
Offline
Site Admin
Site Admin
User avatar

Joined: Thu Feb 26, 2004 8:30 pm
Posts: 13470
Rome did not fall in a day, but it still fell.

Anyway:

Windows 11 CPU Usage Reporting Fix

Following the investigation into the intermittent high CPU usage reported after sleep/resume, I have added the old Windows 11 CPU reporting workaround back to WorkShelf, this time under Troubleshooting Options.

The new Fix Windows 11 CPU Usage Reporting button gives access to the Serialize Timer Expiration setting that Winstep previously exposed when Windows 11 22H2 introduced incorrect per-process CPU usage reporting.

Profiling has confirmed that the current >1% WorkShelf readings are again caused by Windows misreporting CPU usage rather than WorkShelf actually doing more work, and the bad reporting state can appear or disappear after sleep/resume.

Setting Serialize Timer Expiration to Off fixed the earlier Windows 11 CPU reporting bug and is therefore expected to fix this one as well, although that has not yet been confirmed for this new intermittent variant.

The workaround is only enabled on systems that do not support Modern Standby, as before.


Attachments:
Screenshot 2026-09-08 160652.png
Screenshot 2026-09-08 160652.png [ 102.5 KiB | Viewed 3543 times ]

_________________
Jorge Coelho
Winstep Xtreme - Xtreme Power!
https://www.winstep.net - Winstep Software Technologies
Back to top
 Profile WWW 
 
Post new topic Reply to topic Board index : Winstep Forums : General Discussion  [ 199 posts ] Go to page Previous  1 ... 6, 7, 8, 9, 10, 11, 12 ... 14  Next
Display posts from previous:  Sort by  

Who is online

Users browsing this forum: Amazon [Bot], Majestic-12 [Bot] and 243 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: