Performance Counters in Windows - Measure Process, CPU, Network, Disk Usage

[alfa]

Hello,

I have been using this almost for a year now and it worked perfect since i had to upgrade to IE8.

I was using ie7 before.

I made a funtion so i could sleep the macro when IE process is more than 20, it was working very very nice.

But now with IE8 it just continues.

Did something change in IE 8 ?

Thanks in advance

expandcollapse popup

#include <_PDH_ProcessCounters.au3>
#include <clipboard.au3>
#Include <Excel.au3>
#Include <Array.au3>
#Include <File.au3>
;   --------------------    HOTKEY FUNCTION & VARIABLE --------------------
Global $bHotKeyPressed=False
Func _EscPressed()
    $bHotKeyPressed=True
EndFunc
;   --------------------    MAIN PROGRAM CODE   --------------------
HotKeySet("{Esc}", "_EscPressed")
_PDH_Init()
Local $poCounter,$hSplash,$sProcess,$iProcessID
Local $iCounterValue,$sSplashText,$iTemp
ShellExecute ("iexplore.exe")
sleep (7000)
$sProcess="iexplore.exe"     ; change to "autoit3_x64.exe" for 64-bit process :)
$PID = ProcessExists($sProcess)
$iProcessID=$PID
$poCounter=_PDH_ProcessObjectCreate($sProcess,$iProcessID)
_PDH_ProcessObjectAddCounters($poCounter,6) ; "% Processor Time"
;~ ; successful? Then enter loop
Func _sleepUntilXX()
If @error=0 And IsArray($poCounter) Then
    $iCounterValue=Round(_PDH_ProcessObjectUpdateCounters($poCounter,0)/$_PDH_iCPUCount)
    Do
        Sleep(200)
        $iCounterValue=_PDH_ProcessObjectUpdateCounters($poCounter,0)
        If @error=32 Then ExitLoop    ; Process can no longer be found \\istanbul-svr-01\production\BrioRpt\asp\5519_capacity_report_short.bqy
        $iCounterValue=Round($iCounterValue/$_PDH_iCPUCount)
   If $iCounterValue > 20 Then
    Sleep (1000)
   EndIf
    Until $iCounterValue = 0
EndIf
EndFunc
If FileExists("C:\Documents and Settings\aozben\Desktop\Cap Rep\*.xls") Then
FileDelete("C:\Documents and Settings\aozben\Desktop\Cap Rep\*.xls")
EndIf
_ClipBoard_SetData ("\\istanbul-svr-01\production\BrioRpt\asp\5519_capacity_report_short.bqy")
Send ("{Tab}")
send ("^v")
send ("{enter}")
WinWaitActive ("\\istanbul-svr-01\production\BrioRpt\asp\5519_capacity_report_short.bqy - Windows Internet Explorer")
ControlClick("\\istanbul-svr-01\production\BrioRpt\asp\5519_capacity_report_short.bqy - Windows Internet Explorer", "", "List12")
Send ("{down 10}")
Sleep (1000)
ControlClick("\\istanbul-svr-01\production\BrioRpt\asp\5519_capacity_report_short.bqy - Windows Internet Explorer", "", "List4")
Send ("{down}")
sleep (1000)
Send ("^a")
sleep (1000)
ControlClick("\\istanbul-svr-01\production\BrioRpt\asp\5519_capacity_report_short.bqy - Windows Internet Explorer", "", "StdBtn25")
Sleep(2000)
_sleepUntilXX()
Sleep(5000)

[KaFu]

As of IE8 the browser might start more than one process (take a look at the task-manager). You have to adjust you script to monitor for all process related to the IE instance you want to monitor or alternativly turn of the new behavior in IE itself by tweaking a reg-key (untested, and thinking about it also unrecommended).

[zpanderson]

I must be very giving

Here's something that monitors multiple instances, and also monitors for new and killed ones:

Example #2: % CPU Usage for Multiple Instances

expandcollapse popup

#include <Array.au3>
#include <_PDH_ProcessCounters.au3>

; --------------------  HOTKEY FUNCTION & VARIABLE --------------------

Global $bHotKeyPressed=False

Func _EscPressed()
    $bHotKeyPressed=True
EndFunc

; --------------------  WRAPPER FUNCTION --------------------

Func _PDH_ProcessObjectGetPID($poCounter)
    If Not IsArray($poCounter) Then Return SetError(1,0,0)
    Return $poCounter[0][3]
EndFunc

; --------------------  MAIN PROGRAM CODE --------------------

HotKeySet("{Esc}", "_EscPressed")

_PDH_Init()
Local $hSplash,$sSplashText,$sProcess,$iBadCounters=0
Local $aProcesses,$aCounters,$iTemp,$iProcIndex,$iTotal,$iLastPID=0,$iCounterValue

$sProcess="chrome.exe"
$aProcesses=ProcessList($sProcess)
$iTotal=$aProcesses[0][0]
If @error Or $iTotal=0 Then Exit _PDH_UnInit()

MsgBox(0, "Process count",$iTotal,1)

Dim $aCounters[$iTotal] ; Hold the counters in the array

; Create all the process counters
For $i = 0 to $iTotal-1
    $aCounters[$i]=_PDH_ProcessObjectCreate($sProcess,$aProcesses[$i+1][1])
    If @error Then $iBadCounters+=1
    _PDH_ProcessObjectAddCounters($aCounters[$i],6) ; "% Processor Time"
Next
$iLastPID=$aProcesses[$i][1]

; at least some good counters? Then enter loop
If $iBadCounters<>$iTotal Then
    $hSplash=SplashTextOn("'"&$sProcess&"' Processes Info","",360,160,Default,Default,16)
    Do
        ; Check for new processes
        $aProcesses=ProcessList($sProcess)
        If @error Or $aProcesses[0][0]=0 Then ExitLoop
        ; If more processess than we currently have, or last PID doesn't match, we better add new ones
        ;   (note: terminated ones are caught elsewhere
        If $aProcesses[0][0]>$iTotal Or $aProcesses[$aProcesses[0][0]][1]<>$iLastPID Then
            $iTemp=$aProcesses[0][0]
            ; Move backwards to find a PID that matches up
            For $iTemp=$aProcesses[0][0] To 1 Step -1
                If $iLastPID=$aProcesses[$iTemp][1] Then
                    $iProcIndex=$iTemp+1    ; set location in ProcessList where to start adding new counters from (1 past match)
                    ; Found a match, now to calculate how many new Counters to add
                    $iTemp=$aProcesses[0][0]-$iTemp
                    If $iTemp<=0 Then ExitLoop
                    ; Resize the array to accomomodate the new counters
                    ReDim $aCounters[$iTotal+$iTemp]
                    ; Add in the new counters
                    For $i=$iTotal To $iTotal+$iTemp-1
                        $aCounters[$i]=_PDH_ProcessObjectCreate($sProcess,$aProcesses[$iProcIndex][1])
                        _PDH_ProcessObjectAddCounters($aCounters[$i],6) ; "% Processor Time"
                        $iProcIndex+=1
                    Next
                    $iTotal+=$iTemp
                    ConsoleWrite("New Counters added, new Total:"&$iTotal&@CRLF)
                    ExitLoop
                EndIf
            Next
        EndIf
        $sSplashText=""
        ; Update all Process counters
        $i=0
        Do
            $iCounterValue=_PDH_ProcessObjectUpdateCounters($aCounters[$i],0)
            If @error=1 Or @error=32 Then
                ConsoleWrite("Process died. removing. New total:"&$iTotal-1&@CRLF)
;~              _PDH_ProcessObjectDestroy($aCounters[$i])   ; it's dead, and already destroyed if @error=1 or =32
                _ArrayDelete($aCounters,$i) ; Shift the array contents around
                $iTotal-=1  ; 1 less!
                $i-=1       ; to compensate for the upcoming increment
                If $iTotal=0 Then ExitLoop 2    ; did they all die!?
            Else
                If Not @error Then $iCounterValue=Round($iCounterValue/$_PDH_iCPUCount)
                $sSplashText&="PID #"&_PDH_ProcessObjectGetPID($aCounters[$i])&" = "&$iCounterValue&"% CPU Usage"&@CRLF
            EndIf
            $i+=1
        Until $i=$iTotal
        $iLastPID=_PDH_ProcessObjectGetPID($aCounters[$iTotal-1])
        $sSplashText&="[Esc] exits"
    ControlSetText($hSplash,"","[CLASS:Static; INSTANCE:1]",$sSplashText)
    #cs
        If 1 Then
            $aProcesses=ProcessList($sProcess)
            Dim $aTemp[$aProcesses[0][0]+1][2]
            $iTemp=0
            $aTemp[0][0]="ProcessList"
            $aTemp[0][1]="CounterList"
            For $i=1 To $aProcesses[0][0]
                $aTemp[$i][0]=$aProcesses[$i][1]
                If $iTemp<$iTotal Then $aTemp[$i][1]=_PDH_ProcessObjectGetPID($aCounters[$iTemp])
                $iTemp+=1
            Next
            _ArrayDisplay($aTemp,"List compare")
        EndIf
#ce
        Sleep(100)
    Until $bHotKeyPressed Or $iTotal=0
    WinClose($hSplash)
EndIf

; Destroy all the Process counters
For $i = 0 to $iTotal-1
    $aCounters[$i]=_PDH_ProcessObjectDestroy($aCounters[$i])
Next

_PDH_UnInit()

*edit 2nd time: wow, major logic errors. I made adjustments where necessary, and left a chunk of 'debug' code in just for the sake of it (I needed something to pause the loop and let me add/remove processes while making sure PID's synched up)

Could someone give some guidance on how to do this on a remote server?

Thanks!

[zpanderson]

Bump. Does anyone have an idea of how to do the above on a remote server?

[BrewManNH]

I don't think it can be done, ProcessList (which that example script uses extensively) only works on the local machine. If you can figure out how to get the process list from a remote machine, and save it to an array, it might just be possible.

[skreien]

I'm trying to use this on a Win2008R2 server with a total of 80 cores. It's coming back with idle of anything from 130% to 390%. Any ideas?

[Ascend4nt]

I'm trying to use this on a Win2008R2 server with a total of 80 cores.

Oh sure, make all us other 4-core PC users feel inferior! You CPU stud!

*frowns looking down at my small 4-core CPU*

ehe.. anyway, it appears to be that some calculation is off in the code. It should automatically adjust for the # of CPU's on your system. However, lets check that the code is actually reporting the right value. Here, try this little code snippet and tell me what it reports:

Local $stSystemInfo=DllStructCreate("ushort;short;dword;ptr;ptr;ulong_ptr;dword;dword;dword;short;short")
; If we are running in 32-bit mode on a 64-bit OS, we need to call a different API function
If Not @AutoItX64 And @OSArch<>"X86" Then
DllCall("kernel32.dll","none","GetNativeSystemInfo","ptr",DllStructGetPtr($stSystemInfo))
Else
DllCall("kernel32.dll","none","GetSystemInfo","ptr",DllStructGetPtr($stSystemInfo))
EndIf
If @error Then Exit -1

ConsoleWrite("# of Processors per GetSystemInfo = " & DllStructGetData($stSystemInfo, 7) & @CRLF)
ConsoleWrite("# of Processors in Environment Var = " & EnvGet("NUMBER_OF_PROCESSORS") & @CRLF)

[skreien]

It's definitely screwy.... I ran it twice back to back and got different answers. The first time I got 20, second time I got 60. This is on Win2008R2 and compiled as 64 bit exe.

[Ascend4nt]

It's definitely screwy.... I ran it twice back to back and got different answers. The first time I got 20, second time I got 60. This is on Win2008R2 and compiled as 64 bit exe.

Huh.. At first I thought maybe you were stretching the truth about an 80 processor system. But what you say seems pretty consistent with the way the system would handle a system with more than 64 logical processors. It has only become a possibility with Windows 7 and 2008R2 (and up) 64-bit O/S's

So, with more than 64 processors, Windows splits these processors up into groups of 64 or less, and then allows processes to run on only one group at a time. The original limit of 64 processors, and 64-bit processor affinity would become problematic, among other things, if Windows didn't do otherwise. So unless you execute the process in a certain way, you won't have control of which group it runs on. And of course the traditional method of getting # of processors only works in relation to the current GROUP of processors that the process has been assigned to.

Luckily, for processes that absolutely NEED to know about ALL the processors (and groups) on the system, Windows 7 & 2008R2 introduced a new set of API calls, one of which should do exactly what we are looking for - get the total # of logical processors for ALL groups. The following should work correctly for you:

; ALL_PROCESSOR_GROUPS = 0xFFFF
Local $aRet = DllCall("kernel32.dll", "dword", "GetActiveProcessorCount", "word", 0xFFFF)
ConsoleWrite("Number of Active processors = " & $aRet[0] & @CRLF)

I've already modified my PDH_Init function to attempt that call first, which should fix your problem. In fact, I just noticed that I hadn't uploaded the latest version of my PDH Performance Counters (from 6/2011), so I should see about repackaging and updating the whole thing. But first I'd like to hear back from you confirming that the above AutoIt code actually gives you the total # of processors that are on your system.

And for reference, here are some good resources (MSDN of course):

Processor Groups (Windows)

What's New in Processes and Threads (Windows)

Thanks!

Edited October 12, 2012 by Ascend4nt

[skreien]

Yep, I wasn't kidding about the 80 procs. I've got some big toys to play with.

That looks much better.

# of Processors per GetSystemInfo = 60

# of Processors in Environment Var = 60

Number of Active processors = 80

Edited October 16, 2012 by skreien

[skreien]

Can you post back when you upload the new version? I'd like to give it a try.

[Ascend4nt]

Sorry for the late update, I migrated to a SSD drive (wee!).

Anyway, this update addresses the 64+ CPU problem, as well as updates from June of last year! (oops). The previous updates addressed major Microsoft PDH.dll issues:

10/12/2012:

- Fix: For systems with >64 processors, wrong results were returned for the # of CPU's. A new test is run using 'GetActiveProcessorCount' (Win7+/2008R2+) to get the total number of logical processors in ALL processor groups

6/2/2011:

- MS Bugs found: _PDH_BrowseCounters() on Win7 causes around 40 DLL's to STAY loaded after the API call. Plus, reference count to PDH.DLL is increased by 2.

Workaround: Don't use this function unless ABSOLUTELY necessary!

However, using _PDH_ReloadDLL() can cut down on the # of DLL's that are loaded (to around 10 on my system) but note that subsequent calls to _PDH_BrowseCounter() will be slower!

- Added: _PDH_ReloadDLL(). I found a bug with these API calls (Win2000->Win7):

* PdhExpandWildCardPath [_PDH_GetCounterList()]

* PdhValidatePathW [_PDH_ValidatePath()]

Neither of the above recognize new object instances unless PDH.DLL is unloaded! Hence, _PDH_ReloadDLL() is there to fix this issue.

*NOTE: Use of this will cause subsequent _PDH_BrowseCounters() call's to take a long time to load, unless previous calls weren't made.

- Added: Internal function __PDH_ForceFreeDLL() -> to support _PDH_ReloadDLL(), and maybe fix 'Browse Counters Dialog' problems in the future

- Changed: TestPDH_PerformanceCounters() now calls _PDH_ReloadDLL() to allow it to 'see' new instances when Manual Entry is used.

- Changed: TestPDH_PerformanceCounters() now recognizes all-counters-dead scenarios.

- Fix: _PDH_GetCounterInfo() on Win2000 was broken.

- Fix: _WinTime_Format* functions handling of Milliseconds - was padding to 4 character instead of 3 characters long (Thx Joakim)

[skreien]

I compiled the testpdh_cpuusage.au3 after downloading the newest version and tried it on my big server... it still reports the wrong cpu count. It's coming back with 32 and 20 CPU's on back to back runs.

[Ascend4nt]

skreien, that's quite odd that you are getting 32 and 20 now.. you said you had 80 processors, which would have led me to believe you'd get some balance between instantiations that equals out to 80. Perhaps Windows is now splitting up the processor groups into more than 2 for your machine? That'd be quite interesting.. hmm

Okay, well, I've checked out the module in question (testpdh_cpuusage.au3), and I see that it uses a wildcard to collect information on all processors. The fact that the API call is returning less than 80 processors indicates to me that this is a limitation of the PDH module. This can't really be 'fixed' in my code, not without jumping through hoops and creating new processes that intentionally run on other processor groups. (The issue that I *did* fix with the last update was the per-process CPU usage calculation, which should reflect what Task Manager reports.)

As far as getting the overall CPU usage per processor, I believe the best method is to collect the kernel, user, and idle times for each processor, and then do calculations on every polling interval to determine what the current usage levels are. The problem is figuring out which of the new API functions to call to do this easily.. I was just poking around MSDN now and wound up with a bit of a headache! haha

Anyway, the general calculation can be seen in other example scripts on the forum. Here's a couple:
>Process CPU Usage - by Beege
>Function _ProcessGetCPU() - by Novatek

Also check out this CodeProject article (and the linked user reply):
Get CPU Usage with GetSystemTimes, 'The Way it's supposed to be...'

I hope that helps in some way.. my initial intention with code update was to address the 'Idle' process problem you had mentioned.. I hope that is at least fixed?

Thanks!

*update: See my >CPU Multi-Processor Usage w/o Performance Counters topic, which may or may not report for >64 processors (with Nt* based functions, its iffy if it follows Windows processor-groups logic or not)

Edited June 18, 2013 by Ascend4nt

[JackDinn]

Hi,

Iv been using your work on performance counters for a long while now and have posted earlier on this thread with a problem i had but now i have more of a general question.

What im looking to do is to be able to get the network throughput (download and/or upload) but just for a specific process and possibly any children of that process.

I have been using Inetget() and Inetgetinfo() and a timer to calculate the download speed of test files in 0.5 sec intervals but since iv been using multiple threaded download tests iv started getting problems on rare occasions.

I do have a user option to use the network performance monitors instead of my internal measurements but there are drawbacks to both methods.

So im just looking for another method , if i could get a performance counter to monitor a single process's throughput rather than a network adapter it might be ideal. Is this possible ?

Cheers,

JD.

[Ascend4nt]

I see IO Read/Write counters for individual processes, but they are a total of all I/O operations (file, network and device I/Os). I don't think you can get any more specific with network or I/O operations from Windows' Performance Counters, at least from what I can tell.

[JackDinn]

hi, did not get any email notification for some reason , sorry bout that :-/

Yea iv spent ages trying to find a "magic bullet" for this one. Iv already got 2 options for the user because neither one is perfect.

The default is very basically :-

For $x = 1 To $mult_list[0]
                $locx = StringSplit($mult_list[$x], ",")
                $dling[$x] = InetGet($locx[1], @TempDir & "\test_" & $x & ".tst", 1, 1)
            Next

This starts the multiple downloads (multi-threaded)

Then this gets the throughput every half sec :-

For $x = 1 To $mult_list[0]
            $downloaded_so_far = InetGetInfo($dling[$x], 0)
            $downloaded += $downloaded_so_far
        Next

However this does not work 100% ,

Then theres the option to use your code for the performance counters which does work better but for some reason i get reports of "doubled" results i.e. the return from the adapter throughput measurement is twice what it should be both in and out , although this is only rarely reported but has been enough different people for me to leave the first measument option in for the users.

Iv recently tried using netstats -e but that is weird , i suppose i just dont understand it properly but i tested it on one of my win7 comps and it reported a download throughput four times higher than it really is ?

how i tested it :-

I enter netstat -e and get the Bytes recived

then i download a test file of 5MB

then i get netstat again and minus one from the other

on win7 Im getting 20MB but on xp i get 5MB

Iv tried 3 different methods there and had quick looks at some others but in 2 years iv not found one single method that approx 5% of people dont have a problem with.

o yea and Iv looked at winpcap , jess a bit to much for me xD

[Ascend4nt]

I don't see why there would be doubled returns from Performance Counters, but I don't know which ones are being used and if there's wildcards being used etc. I suppose the reliability of the results given from performance providers could be questioned..

[Tjalve]

This i great stuff. I will probably be using this alot! I have two quick questions.

1. Can I use this to track the CPU Frequency of the CPU? I want to create a simple program that tracks the CPU frequiency as it chnages trough diffrent workloads. However im having issues finding a simple way to track the current CPU frequency. ive tried WMI and using external programs. So far ive aloways been able to get the max frequncy. I want the frequency thats beeing displayed in perfmon.exe (PPM_processor_0 - frequency). Is it doable?

2. Id like to do more or less teh same think like the above, but with IOPS to tyhe physical HDD. Im doing this today with perfmon and and im recordning a blg file and then convert it to CSV. But i would like to automate the process. Any idees? Any help would be greatly appritiated. Thank you.

[Tjalve]

I uesed this code from earlier in the thread and modified it to read CPU metrics:

#include <_PDH_PerformanceCounters.au3>
_PDH_Init(False, False)
$hPDH = _PDH_GetNewQueryHandle()
$hPDHCounter = _PDH_AddCounter($hPDH, "\ProcessorPerformance(PPM_Processor_0)\frequency")
_PDH_CollectQueryData($hPDH)
While Sleep(1000)
ToolTip("Processor Frequency: " & _PDH_UpdateCounter($hPDH, $hPDHCounter, "", False))
WEnd
_Exit()
Func _Exit()
_PDH_FreeQueryHandle($hPDH)
_PDH_UnInit()
Exit
EndFunc

The strange thing is. When i use: "ProcessorPerformance(PPM_Processor_0)frequency" i get an error:

PdhAddCounterW error [path:'ProcessorPerformance(PPM_Processor_0)percentage'], return:C0000BB8

PdhCollectQueryData error, return:800007D5

But if u use: "Processor(_Total)% Idle Time" it works great.

But i need the processor frequency

Anyone have any idees?