Find out CPU (used) Time of processes.

[Jemboy]

I have a Terminal Server running a company app. Every now and then one of the instances of the app crashes but keeps running.
When this happens the roque program will hog the CPU and other users will start to complain about the server being slow (especially when doing big queries).

In the past I have tried several ways to "kill" the roque program, like taskmanager, using taskkill, using PsKill, using other 3rd party program killers, but to no avail.
The only way to get rid of the roque program is to restart the server, however because the server is use almost 24/7, I do not want to restart it every day.

At the moment, when we determine a roque program, a tasksheduler is turned on to reboot the server after the nightshift, but about 30 minutes before the morning shift starts.

One way to determine a program has gone roque, is to check out the "CPU Time" on the WIndows Task Manager.
This is the total time a program/process has used the CPU.
For most programs this only a few minutes, but definetely less than 1 hour during 24 hours.

I would like to find out the CPU (used) Time of all running programs (processes), to determine if there are programs which have used more then 1 hour CPU Time.
Whenever I could programatically determine a roque program in progress, I want to shedule a morning restart of the server.

My main problem is, I have not find a way to determine the "CPU Time" (see screenshot).
I have search the forum, tries several scripts and snippets to no avail.

So any help to determine this "CPU Time" for the running processes would be much appreciated!

[argumentum]

tasklist /v /fo csv >tasklist2.csv

Edited May 20, 2023 by argumentum
solved

[mistersquirrle]

Well I thought that it wasn't too hard, however it appears that on my system the normal function of _WinAPI_GetProcessTimes doesn't cut it. I needed to open the process myself (likely with debug enabled) to reliably get CPU time information. Check out this script:

 

expandcollapse popup

;~ #RequireAdmin
;~ #AutoIt3Wrapper_UseX64=y
#include <WinAPI.au3>
#include <ProcessConstants.au3>

Global $iNanoSecondsToMilliSeconds = 10000

Global $sProcessName = 'explorer.exe'
Global $iProcessId = ProcessExists($sProcessName)
If $iProcessId = 0 Or @error Then
    ConsoleWrite('Process does not exist, or error: ' & @error & @CRLF)
    Exit
EndIf


;~ Global $aProcessTimes = _WinAPI_GetProcessTimes()
Global $ghProcess = _WinAPI_OpenProcess($PROCESS_QUERY_LIMITED_INFORMATION, 0, $iProcessId, True)
Global $aProcessTimes = __WinAPI_GetProcessTimes($ghProcess)
If @error Then
    ConsoleWrite('Unable to get process times: ' & @error & ', ' & _WinAPI_GetLastError() & ', Msg: ' & _WinAPI_GetLastErrorMessage() & @CRLF)
    Exit
EndIf


;~ For $i = 0 To UBound($aProcessTimes) - 1
;~  ConsoleWrite('Index: ' & $i & ', type: ' & VarGetType($aProcessTimes[$i]) & ', string data: ' & String($aProcessTimes[$i]) & ', int data: ' & Int($aProcessTimes[$i]) & @CRLF)
;~ Next


ConsoleWrite('Process ID: ' & $iProcessId & ', name: ' & $sProcessName & ' times: ' & @CRLF)
ConsoleWrite(@TAB & 'Kernel mode: ' & __MillisecondToTimestamp(($aProcessTimes[1] / $iNanoSecondsToMilliSeconds)) & ' (' & $aProcessTimes[1] & ')' & @CRLF)
ConsoleWrite(@TAB & '  User mode: ' & __MillisecondToTimestamp(($aProcessTimes[2] / $iNanoSecondsToMilliSeconds)) & ' (' & $aProcessTimes[2] & ')' & @CRLF)
ConsoleWrite(@TAB & '      Total: ' & __MillisecondToTimestamp((($aProcessTimes[1] + $aProcessTimes[2]) / $iNanoSecondsToMilliSeconds)) & @CRLF)

Exit


Func __MillisecondToTimestamp($iMs, $bSeconds = False, $bColonSeparator = True)
    Local $sTimestamp

    If $bSeconds Then $iMs *= 1000
    $iMs = Int($iMs, 0) ; Auto Int

    ; Convert the milliseconds to days, hours, minutes, and seconds
    $iDays = (($iMs >= 86400000) ? Int($iMs / 86400000) : 0)
    $iHours = Int(Mod($iMs, 86400000) / 3600000)
    $iMinutes = Int(Mod($iMs, 3600000) / 60000)
    $iSeconds = Int(Mod($iMs, 60000) / 1000)

    ; Format the timestamp as [DD ]hh:mm:ss.zzz
    ; Not using StringFormat here because it's considerably slower when you do it 10K+ times, so probably doesn't matter at all for this
    $sTimestamp = _
            ($iDays > 0 ? String($iDays) & (($bColonSeparator) ? ' ' : 'd') : '') & _
            StringRight('0' & String($iHours), 2) & (($bColonSeparator) ? ':' : 'h') & _
            StringRight('0' & String($iMinutes), 2) & (($bColonSeparator) ? ':' : 'm') & _
            StringRight('0' & String($iSeconds), 2) & _
            ((Not $bSeconds) ? _
            (($bColonSeparator) ? '.' : 's') & StringRight('00' & String($iMs), 3) & (($bColonSeparator) ? '' : 'ms') : _
            (($bColonSeparator) ? '' : 's'))

    Return $sTimestamp
EndFunc   ;==>__MillisecondToTimestamp


Func __WinAPI_GetProcessTimes($hProcess)
;~  If Not $iPID Then $iPID = @AutoItPID

;~  Local $hProcess = DllCall('kernel32.dll', 'handle', 'OpenProcess', 'dword', ((_WinAPI_GetVersion() < 6.0) ? 0x00000400 : 0x00001000), _
;~          'bool', 0, 'dword', $iPID)
;~  If @error Or Not $hProcess[0] Then Return SetError(@error + 20, @extended, 0)

    Local $tFILETIME = DllStructCreate($tagFILETIME)
    Local $aCall = DllCall('kernel32.dll', 'bool', 'GetProcessTimes', 'handle', $hProcess, 'struct*', $tFILETIME, 'uint64*', 0, _
            'uint64*', 0, 'uint64*', 0)
    If __CheckErrorCloseHandle($aCall, $hProcess) Then Return SetError(@error, @extended, 0)

    Local $aRet[3]
    $aRet[0] = $tFILETIME
    $aRet[1] = $aCall[4]
    $aRet[2] = $aCall[5]
    Return $aRet
EndFunc   ;==>__WinAPI_GetProcessTimes

It did require a bit of tweaking, as since I mentioned using the default UDF function wasn't working on my system, with or without running as admin.

Here's the output for explorer.exe:

Process ID: 29536, name: explorer.exe times: 
    Kernel mode: 01:10:35.937 (42359375000)
      User mode: 00:17:03.328 (10233281250)
          Total: 01:27:39.265

 

Sidenote, I am planning on releasing a UDF that makes getting some process information like CPU time very easy and fast. I'm not sure when I'm going to post it, but keep an eye out if interested.

Edited May 20, 2023 by mistersquirrle

[AndyG]

Hi!

Nice example of the XY-Problem!

What the Problem is:

5 hours ago, Jemboy said:

Every now and then one of the instances of the app crashes

What the User think the Problem is:

5 hours ago, Jemboy said:

My main problem is, I have not find a way to determine the "CPU Time" (see screenshot).

No, you are lying to yourself! The MAIN problem is that the app is crashing....you need to fix THIS! You don't have to solve problems, you have to eliminate problems!

[argumentum]

6 hours ago, AndyG said:

The MAIN problem is that the app is crashing...

hmm, I have a user that has chrome browser hang in the background. Another has a WinWord hang in the background. These slowdown the PCs and the only way around is to close the process. Even tho you're right, at times something like this solves these annoying problems to save us from having to do it manually after a user's unhappy phone call.
I tell users that "I did not code that application" but until the developer fixes it, meh, it's a workaround. 

[TheXman]

13 hours ago, AndyG said:

The MAIN problem is that the app is crashing....you need to fix THIS!

For what it's worth, I tend to agree with @AndyG.  Identifying and fixing the root cause of an issue is almost always better than time spent patching or working around the issue.  If you fix (or at least mitigate) the issue, then you hopefully don't have to waste time dealing with it again and again or risk it causing additional problems.

19 hours ago, Jemboy said:

I would like to find out the CPU (used) Time of all running programs (processes), to determine if there are programs which have used more then 1 hour CPU Time.

13 hours ago, mistersquirrle said:

Well I thought that it wasn't too hard, however it appears that on my system the normal function of _WinAPI_GetProcessTimes doesn't cut it.

I have found WMI much more reliable in getting process info than some of the WinAPI UDF's that use Win32 API's.  I haven't dug into why, but when using the WinAPI UDF's, WMI returns some information that the WinAPI UDF's do not.

Here's a quick script that displays selected process info (including CPU time) for all running processes:

expandcollapse popup

#AutoIt3Wrapper_AU3Check_Parameters=-w 3 -w 4 -w 5 -w 6 -d

#include <Constants.au3>
#include <WinAPILocale.au3>
#include <Array.au3>


list_process_info()

;==========================================================================
Func list_process_info()
;==========================================================================
    Enum  $PROCESS_PID, $PROCESS_NAME, $PROCESS_EXE_PATH, $PROCESS_CPU_TIME, $PROCESS_COLS
    Local $aProcesses[0][$PROCESS_COLS]

    Local $oComError  = ObjEvent("AutoIt.Error", "com_error_handler"), _
          $oProcesses = ""

    Local $sTotalCpuTime  = ""


    With ObjGet("winmgmts:")
        ;Get list of currently running processes
        $oProcesses = .ExecQuery("SELECT Handle, Name, ExecutablePath, KernelModeTime, UserModeTime FROM Win32_Process")

        ;If no processes found, then return with message
        If $oProcesses.Count = 0 Then Return MsgBox($MB_ICONWARNING, "Warning", "No processes found.")
        If @error Then Return MsgBox($MB_ICONERROR, "WMI ExecQuery Error", $oComError.Description)

        ;For each process
        For $oProcess in $oProcesses
            With $oProcess
                ;Calculate total cpu time duration and convert to hh:mm:ss
                $sTotalCpuTime = _WinAPI_GetDurationFormat($LOCALE_USER_DEFAULT, .KernelModeTime + .UserModeTime, "hh:mm:ss")

                ;Add process info to the array
                _ArrayAdd($aProcesses, .Handle & "|" & .Name & "|" & .ExecutablePath & "|" & $sTotalCpuTime)
            EndWith
        Next
    EndWith

    ;Display process info
    _ArraySort($aProcesses, 1, 0, 0, $PROCESS_CPU_TIME) ;Sort by cpu time in descending order
    _ArrayDisplay($aProcesses, "List of processes", "", 0, Default, "PID|NAME|PATH|CPU TIME")
EndFunc

;==========================================================================
Func com_error_handler($oError)
;==========================================================================
    #forceref $oError
    Return ; Return so @error can be trapped by the calling function
EndFunc

Resulting array:

 

Edited May 20, 2023 by TheXman

[Jemboy]

I am overwhelmed by the many answers with solutions for my problem,🙃

The server I am looking to implemtent this is archaich (200x) so the solution from @mistersquirrle did not work, because it required admin rights while I was already administrator.
The solution from @TheXman did not work 100% because the "CPU Time" column stayed empty.

Both solutions however worked when I tried them on a Windows 10, Windows 11 desktop and a Windows 2016 and Windows 2019 server, so this might be attributed to the old Windows version.
I plan to upgrade the server to MS Server 2022 end of this year, so these sloution will probably work then.

 

For now I am going with the solution from @argumentum this worked from commandline without a problem, so I gonna process the CSV or the output for my purpose.

Thanks you all for your help.

 

[Jemboy]

On 5/20/2023 at 8:46 AM, AndyG said:

Hi!

Nice example of the XY-Problem!

What the Problem is:

What the User think the Problem is:

No, you are lying to yourself! The MAIN problem is that the app is crashing....you need to fix THIS! You don't have to solve problems, you have to eliminate problems!

@AndyG your totally right. I am aware of this "XY-problem", however in my real world I am not able to fix this root cause.

My software supplier needed over 3 years to implement a way to e-mailing invoices to our customers and it still has issues.
When a user input a wrong barcode, I asked them to warn the user with a *BEEP*.
However they didn't understand, so I made an AutoIt executable to show them.
In the end, they used my Autoit executable instead of programming the beep them self.
These are only 2 examples.

I have told them several times to look at the reasons for the crashes, but if simple things will take months or even years, I do not expect them to solve this in the next 10 years 😒
 

If it were up to me, I would switch to another software package tomorrow.
However the boss is very happy with the software package, because his issues are solved almost immediately.

 

 

Edited May 22, 2023 by Jemboy