Batch processing help

ParoXsitiC · July 31, 2006

I have a script which will create X amount of browsers needed simtaneously and get the job done. However, after 7 browsers, performance gets worse PER browser. Therefore I want to try to make the function FindPrices() be able to process in groups of 5. If 23 items were selected then it would process items 1-5, then 6-10, then 11- 15, then 15-20 then 21 - 23.

I also know this code is not in the best shape so any advice and/or improving of code is welcome.

I have marked things I need help with or ideas with TODO stamps in the comments

Any ideas on the progress bar would be a big help too!

;Name:          FindShopPrices.au3
;Programmer:    Brian Hare
;Description:   This script is designed to allow a person with a neopet account to visit a shop. 
;               Press the ` button and a GUI will appears with the cheapest price, profit, and average prices
;               of each item in the shop. It will help in determine what items are a deal and what can be sold for how much profit. 
;               User also has ability to do Up-to-date queries on previously found items and find new items not previously found. 
;               Pressing ` again will hide the GUI and the user is free to visit the next shop and repeat
Opt("WinWaitDelay", 0)
Opt("WinTitleMatchMode", 2)
Opt("MouseCoordMode", 0)
Opt("PixelCoordMode", 0)
Opt("SendKeyDelay", 1)

#include <Array.au3>
#include <IE.au3>
#include <GuiConstants.au3>
#include <GuiListView.au3>

HotKeySet("`", "Extract_n_Load")

;Attempt to attach to current IE window. This will allow for instant ability to get HTML
Global $o_IE = _IECreate ("http://www.neopets.com/browseshop.phtml?owner=beaniepush", 1, 1, 0)
Global $ItemArray, $PriceArray
Global $HideForm = False
Global $Begin, $TotalTime
;These Const variables are used to help keep track of the multi dimensional array
Global Const $Index = 0, $Name = 1, $Price = 2, $CheapPrice = 3, $AvgPrice = 4


$Form = GUICreate("", 625, 500)

$KnownList = GUICtrlCreateListView("Item|Price|Cheapest|Profit|Avg", 50, 50, 500, 200, $LVS_SHOWSELALWAYS)
GUICtrlSendMsg($KnownList, $LVM_SETEXTENDEDLISTVIEWSTYLE, $LVS_EX_GRIDLINES, $LVS_EX_GRIDLINES)
GUICtrlSendMsg($KnownList, $LVM_SETEXTENDEDLISTVIEWSTYLE, $LVS_EX_FULLROWSELECT, $LVS_EX_FULLROWSELECT)


_GUICtrlListViewSetColumnWidth($KnownList, 0, 200)
_GUICtrlListViewSetColumnWidth($KnownList, 1, 65)
_GUICtrlListViewSetColumnWidth($KnownList, 2, 65)
_GUICtrlListViewSetColumnWidth($KnownList, 3, 65)
_GUICtrlListViewSetColumnWidth($KnownList, 4, 65)

$FindButton = GUICtrlCreateButton("Lookup Prices", 50, 430, 100, 25)
$ProgressBar = GUICtrlCreateProgress(50, 270, 400, 20, $PBS_SMOOTH)
$FindButton2 = GUICtrlCreateButton("Lookup Prices", 50, 20, 100, 25)

$UnknownList = GUICtrlCreateListView("Item|Price|Cheapest|Profit|Avg", 50, 300, 500, 125, $LVS_SHOWSELALWAYS)
GUICtrlSendMsg($UnknownList, $LVM_SETEXTENDEDLISTVIEWSTYLE, $LVS_EX_GRIDLINES, $LVS_EX_GRIDLINES)
GUICtrlSendMsg($UnknownList, $LVM_SETEXTENDEDLISTVIEWSTYLE, $LVS_EX_FULLROWSELECT, $LVS_EX_FULLROWSELECT)

_GUICtrlListViewSetColumnWidth($UnknownList, 0, 200)
_GUICtrlListViewSetColumnWidth($UnknownList, 1, 65)
_GUICtrlListViewSetColumnWidth($UnknownList, 2, 65)
_GUICtrlListViewSetColumnWidth($UnknownList, 3, 65)
_GUICtrlListViewSetColumnWidth($UnknownList, 4, 65)


While 1
    Sleep(100)
WEnd


Func Extract_n_Load()
    HotKeySet("`", "HideForm")
    ;Create arrays for sorting
    Local $Known_DESCENDING[_GUICtrlListViewGetSubItemsCount($KnownList) ]
    Local $Unknown_DESCENDING[_GUICtrlListViewGetSubItemsCount($UnknownList) ]
    Local $HTML, $Item, $Price, $CheapestPrice, $AvgPrice
    $HTML = _IEBodyReadHTML ($o_IE)
    ;Extract the Items and Prices from HTML. Depending on the type of shop the space before ? is optional
    $ItemArray = StringRegExp($HTML, "border=1></A> ?<BR><B>(.*?)</B><BR>", 3)
    $PriceArray = StringRegExp($HTML, "in stock<BR>Cost ?: (.*?) NP<BR><BR></TD>", 3)
    
    ;Cycle through each item on the page
    For $i = 0 To UBound($ItemArray) - 1
        ;Use the local varibles to help readability
        $Item = $ItemArray[$i]
        $Price = $PriceArray[$i]
        $CheapestPrice = IniRead("Prices.ini", $Item, "CheapestPrice", "?")
        $AvgPrice = IniRead("Prices.ini", $Item, "AvgPrice", "?")
        
        If $CheapestPrice = "?" Then ;If there is no record of this item then set it to the unknown list
            $String = $Item & "|" & $Price & "|" & $CheapestPrice & "|?|" & $AvgPrice
            GUICtrlCreateListViewItem($String, $UnknownList)
        Else ;If there is a record of this item then set it to the known list and retrieve
            $String = $Item & "|" & $Price & "|" & $CheapestPrice & "|" & CalcProfit($Price, $CheapestPrice) & "|" & $AvgPrice
            GUICtrlCreateListViewItem($String, $KnownList)
        EndIf
    Next
    
    ;Now that all the processing has taken place, show the form
    GUISetState(@SW_SHOW, $Form)

    While 1
        $MSG = GUIGetMsg()
        Select
            Case $MSG = $FindButton
                FindPrices($UnknownList, 1, 3) ;Process 1 search on selected items in UnknownList. Use 3 cheapest prices for AvgPrice
            Case $MSG = $FindButton2
                FindPrices($KnownList, 5, 3) ;Process 5 searches on selected items in KnownList. Use 3 cheapest prices for AvgPrice
            Case $MSG = $KnownList
                _GUICtrlListViewSort($KnownList, $Known_DESCENDING, GUICtrlGetState($KnownList))
            Case $MSG = $UnknownList
                _GUICtrlListViewSort($UnknownList, $Unknown_DESCENDING, GUICtrlGetState($UnknownList))
            Case $HideForm = True ;If the ` button was pressed while form was up, Hide it and reset everything
                GUISetState(@SW_HIDE, $Form)
                _GUICtrlListViewDeleteAllItems($KnownList)
                _GUICtrlListViewDeleteAllItems($UnknownList)
                $HideForm = False
                HotKeySet("`", "Extract_n_Load")
                ExitLoop
            Case $MSG = $GUI_EVENT_CLOSE
                Exit
        EndSelect
    WEnd
EndFunc

Func HideForm()
    $HideForm = True
EndFunc

Func FindPrices($List, $NumSearches, $AvgRank) ;AvgRank is how many of the cheapest prices to use in AvgPrice
    $Begin = TimerInit() ;Start time for Progress bar
    GUICtrlSetData($ProgressBar, 0)
    AdlibEnable("UpdateProgress", 100) ;Update Progress bar every 100 ms
    $SelectedItems = _GUICtrlListViewGetSelectedIndices($List, 1)
    Local $NumSelectedItems = $SelectedItems[0]
    Local $Item[$NumSelectedItems + 1][5] ;Used for referring to Item properties such as Index, Name, Price, Cheapest, Avg
    Local $Browser[$NumSelectedItems + 1][$NumSearches + 1] ;Used for referring to Item's Browser(s). Each search gets its own browser
    Local $URL = "http://www.neopets.com/market.phtml?type=process_wizard" ;Used in referer and navigation
    
    ;Each Item gets their own browser, and each search for that item's price gets their own browser
    ;Therefore, If 7 items are selected and each item will be searched 3 times, 21 browsers will open
    $NumBrowsers = $NumSelectedItems * $NumSearches
    
    ;700ms is very abstract because it is based on personal CPU power and internet bandwidth
    ;700ms is only true if $NumBrowsers is 3-7. 1, 2, 9, 10, 11 ..etc each have their own seperate delays
;TODO: A simple fix to this would be only allowing no more or less than 3-7 browsers run simtaneous

    $TotalTime = $NumBrowsers * 700
    
    ;Run through this first, so setting the text isn't delayed
    For $i = 1 To $NumSelectedItems
        $Item[$i][$Index] = $SelectedItems[$i]
        $Item[$i][$Name] = _GUICtrlListViewGetItemText($List, $Item[$i][$Index], 0)
        $Item[$i][$Price] = _GUICtrlListViewGetItemText($List, $Item[$i][$Index], 1)
        $Item[$i][$CheapPrice] = _GUICtrlListViewGetItemText($List, $Item[$i][$Index], 3) ;If UnknownList, this will be "?"
        $Item[$i][$AvgPrice] = _GUICtrlListViewGetItemText($List, $Item[$i][$Index], 4) ;If UnknownList, this will be "?"
        ;Cycle through each selected item and set CheapPrice, Profit, and AvgPrice Columns to Searching
        SetListViewText($List, $Item[$i][$Index], "Starting")
    Next 
    For $i = 1 To $NumSelectedItems 
        For $b = 1 To $NumSearches ;For each item, create a browser for the number of searches
            ;Use _IECreateQuick instead of _IECreate since IECreate will force navigate to about:blank 
            ;about:blank takes millisconds longer than no page at all. When making 20+ browsers it makes a difference
            $Browser[$i][$b] = _IECreateQuick(0) ;1 = visible 0 = invisible
            ;Use the navigate method instead of _IENavigate because it needs to send headers
            ;Refer to http://msdn.microsoft.com/library/default.asp?url=/workshop/browser/webbrowser/reference/methods/navigate.asp
            $Browser[$i][$b].navigate ($URL & "&shopwizard=" & $Item[$i][$Name] & "&criteria=exact", 64, 0, 0, "Referer: " & $URL)
            SetListViewText($List, $Item[$i][$Index], "Searching")
        Next
    Next
    
    ;Wait for each and every browser to be loaded.
    For $i = 1 To $NumSelectedItems
        For $b = 1 To $NumSearches
;TODO: Allow an item to discontinue itself from everything if it doesnt load within X seconds. 
        ;This will keep it from waiting 1 min for 1 browser and continue with the rest
            _IELoadWait ($Browser[$i][$b])
            SetListViewText($List, $Item[$i][$Index], "Found")
        Next
    Next
    
    For $i = 1 To $NumSelectedItems
        Local $TotalAvgPrice ;Declare varible out of the browser FOR loop because it will keep a running total for items, not browsers
        For $b = 1 To $NumSearches
            Local $TotalCheapestPrices = 0, $AvgMarketPrice = 0, $CheapestPrice = 0
            Local $HTML = _IEBodyReadHTML ($Browser[$i][$b])
            Local $MarketPrices = StringRegExp($HTML, "buy_cost_neopoints=(.*?)"">", 3)
            If @extended Then ;If found any prices
                ;If there isn't enough prices to do AvgRank, use every price on the page
                If UBound($MarketPrices) < $AvgRank Then 
                    For $Price in $MarketPrices
                        $TotalCheapestPrices = $TotalCheapestPrices + $Price
                    Next
                    $AvgMarketPrice = $TotalCheapestPrices / UBound($MarketPrices)
                Else
                    For $p = 0 To $AvgRank - 1
                        ;Get the top AvgRank cheapest prices, where AvgRank is 3 by default.
                        $TotalCheapestPrices = $TotalCheapestPrices + $MarketPrices[$p] 
                    Next
                    $AvgMarketPrice = $TotalCheapestPrices / $AvgRank
                EndIf

                ;The cheapest price is always the first one
                $CheapestPrice = $MarketPrices[0]
                
                ;If this is items first time don't bother comparing because Number("?") = 0
                If $Item[$i][$CheapPrice] = "?" Then
                    $Item[$i][$CheapPrice] = $CheapestPrice
                Else
                    ;Make sure both are numbers before comparing them
                    If Number($CheapestPrice) < Number($Item[$i][$CheapPrice]) Then $Item[$i][$CheapPrice] = $CheapestPrice
                EndIf
                $TotalAvgPrice = $TotalAvgPrice + $AvgMarketPrice
                ;Close the browser
                WinClose(_IEPropertyGet ($Browser[$i][$b], "hwnd"))
            Else ;No prices were found on the page. Decrease $b by 1 so it revisits the page and retries FOR loop
;TODO: Make sure this doesnt end up in an infinite loop (neopets sometimes disables search which can cause this)
                $b = $b - 1
                $Browser[$i][$b].navigate ($URL & "&shopwizard=" & $Item[$i][$Name] & "&criteria=exact", 64, 0, 0, "Referer: " & $URL)
                _IELoadWait ($Browser[$i][$b])
            EndIf
        Next
        $Item[$i][$AvgPrice] = Round($TotalAvgPrice / $NumSearches, 2)
        SetListViewText($List, $Item[$i][$Index], "Extracted")
    Next

    For $i = 1 To $NumSelectedItems
        ;Save the results to file for later use
        IniWrite("Prices.ini", $Item[$i][$Name], "CheapestPrice", $Item[$i][$CheapPrice])
        IniWrite("Prices.ini", $Item[$i][$Name], "AvgPrice", $Item[$i][$AvgPrice])
        IniWrite("Prices.ini", $Item[$i][$Name], "AvgRank", $AvgRank)
        IniWrite("Prices.ini", $Item[$i][$Name], "NumSearches", $NumSearches)
        
        If $List = $UnknownList Then 
            ;Move the item to KnownList with its new properties, then delete the old one
            GUICtrlCreateListViewItem($Item[$i][$Name] & "|" & $Item[$i][$Price] & "|" & $Item[$i][$CheapPrice] & "|" & CalcProfit($Item[$i][$Price], $Item[$i][$CheapPrice]) & "|" & $Item[$i][$AvgPrice], $KnownList)
        Else
            ;Update the item with its new properties
            _GUICtrlListViewSetItemText($List, $Item[$i][$Index], 0, $Item[$i][$Name])
            _GUICtrlListViewSetItemText($List, $Item[$i][$Index], 1, $Item[$i][$Price])
            _GUICtrlListViewSetItemText($List, $Item[$i][$Index], 2, $Item[$i][$CheapPrice])
            _GUICtrlListViewSetItemText($List, $Item[$i][$Index], 3, CalcProfit($Item[$i][$Price], $Item[$i][$CheapPrice]))
            _GUICtrlListViewSetItemText($List, $Item[$i][$Index], 4, $Item[$i][$AvgPrice])
        EndIf
    Next
    ;Delete all items that are selected. Can't delete one by one sinced indexes get mixed
    If $List = $UnknownList Then _GUICtrlListViewDeleteAllItems($UnknownList)
    AdlibDisable() ;Stop updating progress bar
    GUICtrlSetData($ProgressBar, 100)
EndFunc

Func SetListViewText($List, $Index, $Text)
        _GUICtrlListViewSetItemText($List, $Index, 2, $Text)
        _GUICtrlListViewSetItemText($List, $Index, 3, $Text)
        _GUICtrlListViewSetItemText($List, $Index, 4, $Text)
EndFunc
Func UpdateProgress()
    $Percent = Int((TimerDiff($Begin) / $TotalTime) * 100)
    GUICtrlSetData($ProgressBar, $Percent)
EndFunc

Func CalcProfit(ByRef $StringPrice, ByRef $StringCheapestPrice)
    Local $Price = CommaString2Number($StringPrice)
    Local $SellingPrice = CommaString2Number($StringCheapestPrice)
    
    $Profit = $SellingPrice - $Price
    Return $Profit
EndFunc

Func CommaString2Number(ByRef $String)
    ;Takes a string such as 423,000 and turns it to the number 423000
    Local $Number = Number(StringReplace($String, ",", ""))
    Return $Number
EndFunc

Func _IECreateQuick($f_visible = 1)
    Local $result, $f_mustUnlock = 0
    
    If Not $f_visible Then
        $result = __IELockSetForegroundWindow($LSFW_LOCK)
        If $result Then $f_mustUnlock = 1
    EndIf
    
    Local $o_object = ObjCreate("InternetExplorer.Application")
    If Not IsObj($o_object) Then
        __IEErrorNotify("Error", "_IECreate", "", "Browser Object Creation Failed")
        SetError($_IEStatus_GeneralError)
        Return 0
    EndIf
    
    $o_object.visible = $f_visible
    
    If $f_mustUnlock Then
        $result = __IELockSetForegroundWindow($LSFW_UNLOCK)
        If Not $result Then __IEErrorNotify("Warning", "_IECreate", "", "Foreground Window Unlock Failed!")
    EndIf
    Return $o_object
EndFunc

ParoXsitiC · July 31, 2006

Oh and for those who wish to test it out and don't want to make a NeoPet account:

Username: k24zc1uen36k

Password: autoit3

Excalibur · July 31, 2006

I havent studied your code, and this suggestion may require a complete reqire of your program (yeah, I know, I would cuss at someone suggesting that too :-P) But AutoIt does have functions and utilities that allows you to download the html pages directly and handle them directly in the code. Its ALOT ALOT ALOT!!!!!! more efficent, and you could do things MUCH FASTER! Something you may wanna concider. It would allow you ALOT more flexability in what you wanna do and scan aswell. Hope that helps. Refer to the AutoIt Functions section of the help document. :-)

Edit: I'm sorry, I looked at the help file, I dont know what I was thinking... Unless its in one of the beta versions, (I only have standard installed on this computer) but I dont see the functions I was looking for. What you could do is look up the proper syntax for making a connection and a request to an http server. I will check the beta function list in a bit. But it may be easier to do what your doing in a way, because of authentication and the way neopets is set up. You would have to create a system for cookies for authentication and all that mess.

About the solution you seek: what you could do is create an IE process, and get the PID (Process ID) of the process and put it in an array of 5. When your done with a window, remove it from the array, or set it to 0. Create a loop that checks for 0 places in the array, then when there is one, create another IE process of what you were doing.

Edit: On third thought and now that I'm reading your code. Ignore me. :-P I'm reading your code deeper now to see if I can find the right solution.

Edited July 31, 2006 by Excalibur

ParoXsitiC · July 31, 2006

I believe you were referring to _InetGetSource.

If this is indeed faster, I will use this instead. I thought to myself that it doe's not send headers those so it would'nt work but then I came across this forum post:

http://www.autoitscript.com/forum/lofivers...php?t16287.html

I checked the _InetGetSource function and he added header functionality to it but it was never documented to my knowledge. I tested this solution and it does work correctly.

Using this will be much faster than using IE I assume?

ParoXsitiC · July 31, 2006

I did a little test to determine which would be faster IE or Inet.

http://paroxsitic.50webs.com/Results.html

The left most column is the winner/loser.

B stands for Number of browsers, and Time/B is time per browser.

The left table was when I was doing a search in neopets. This was inaccurate because it depended on the speed of neopet's servers at the time, and the results of the test.

I repeated the test with a more reliable server (google) and a much smaller download to minimize errors. This test is shown to the right.

INet is much faster.

This works out because Its less coding, its much less error prone, and there is no loss of efficiency between 5 browsers or 20. Therefore I don't even need to use a batch process.

Is there a way to optimize Inet even more for my purpose? The function is pretty cryptic to me

Here is the code I used to run the 2nd test:

#include <IE.au3>
#include <Inet.au3>

$Headers =  "Referer: http://www.neopets.com/market.phtml?type=process_wizard"
$URL = "http://www.google.com/images/firefox/google.gif"
For $test=1 to 20
        $Begin = TimerInit()
        For $x =1 to $test
            $HTML = _INetGetSource($URL,$Headers)
            QuickOutput(@ScriptDir & "\TEST\INET"&$test&"-"&$x&".html",$HTML)
        Next 
        QuickOutput(@ScriptDir & "\TEST\Results.txt","INET("&$test&"):  "& TimerDiff($Begin), 1)

        $Begin = TimerInit()
        Dim $o_IE[$test+1]
        For $x =1 to $test
            $o_IE[$x] = _IECreateQuick(0)
            $o_IE[$x].navigate($URL, 64,0,0, $Headers)
        Next 

        For $x =1 to $test
            _IELoadWait ($o_IE[$x])
            $HTML = _IEBodyReadHTML ($o_IE[$x])
            QuickOutput(@ScriptDir & "\TEST\IE"&$test&"-"&$x&".html",$HTML)
            WinClose(_IEPropertyGet ($o_IE[$x], "hwnd"))
        Next 
        QuickOutput(@ScriptDir & "\TEST\Results.txt","IE("&$test&"):    "& TimerDiff($Begin), 1)
Next


    
    
Func _IECreateQuick($f_visible = 1)
    Local $result, $f_mustUnlock = 0
    
    If Not $f_visible Then
        $result = __IELockSetForegroundWindow($LSFW_LOCK)
        If $result Then $f_mustUnlock = 1
    EndIf
    
    Local $o_object = ObjCreate("InternetExplorer.Application")
    If Not IsObj($o_object) Then
        __IEErrorNotify("Error", "_IECreate", "", "Browser Object Creation Failed")
        SetError($_IEStatus_GeneralError)
        Return 0
    EndIf
    
    $o_object.visible = $f_visible
    
    If $f_mustUnlock Then
        $result = __IELockSetForegroundWindow($LSFW_UNLOCK)
        If Not $result Then __IEErrorNotify("Warning", "_IECreate", "", "Foreground Window Unlock Failed!")
    EndIf
    Return $o_object
EndFunc 






Func QuickOutput($Filename, $Output, $Mode = 2)
    Local $File = FileOpen($Filename,$Mode)
    FileWriteLine($File, $Output)
    FileClose($File)
EndFunc

BabyBeast · July 31, 2006

something you should try with IE.au3

1. disable graphic loading

2. create browser with the url (instead of create then navigate, save loading time)

3. grab the source

This way should be a lot faster when using ie.au3

ParoXsitiC · July 31, 2006

something you should try with IE.au3
1. disable graphic loading
2. create browser with the url (instead of create then navigate, save loading time)
3. grab the source
This way should be a lot faster when using ie.au3

1. I did that externally with an application called ad muncher. I disabled all media on neopets.com and it did help alot.

2. The _IECreate function uses the navigate code in it. So its the same thing just out of the function

Sign In

Batch processing help

Recommended Posts

ParoXsitiC

Link to comment

Share on other sites

ParoXsitiC

Link to comment

Share on other sites

Excalibur

Link to comment

Share on other sites

ParoXsitiC

Link to comment

Share on other sites

ParoXsitiC

Link to comment

Share on other sites

BabyBeast

Link to comment

Share on other sites

ParoXsitiC

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta