Andlin Posted November 15, 2011 Share Posted November 15, 2011 I have a spam filter that checks a greylist file for allowed IP address' the issue that i have is with Gmail and MSN Mail (for these purposes we will anly look at GMail) In normal operation when a mail server attempts a connection to the spam server it looks for the server's IP in a grey list file, if it is not there it regects the connection and requests a retry after x seconds (300). Then after 300 seconds if it detects the server attempting connection again it assumes that it is serious and adds it to the allow list with a date/time stamp and accepts the connection and then goes through the normal spam interogation. This is very handy as 90% of spammers do not retry. Problem: Gmail have a mail farm, so when the first server gets rejected the next attempt after the x seconds may come from a different IP address and as such it will then go into the reject and greylist wait, and so on........ Gmail in their SPF list their valid IP address' as following 216.239.32.0/19 64.233.160.0/19 66.249.80.0/20 72.14.192.0/18 209.85.128.0/17 66.102.0.0/20 74.125.0.0/16 64.18.0.0/20 207.126.144.0/20 173.194.0.0/16 Thats about 200,000 IP's (I dont believe they have that many mail servers, but who am i to say) Solution: Create a script that converts the CIDR to a list of IP addresses and append it to the current allow list, While checking for duplicates to ensure that the same IP doesnt appear twice. Problem with my solution: by checking a file (currently without GMail's IP's its about 19,580 Entries long) for an entry that allready exists it slows the script to a crawl. I have attached a copy of the au3 script along with the greylist.txt file that contains the allow list (I have prepopulated it with some entries), also i have attached cidr.txt which is a file that stores the list of cidr's that need to be added. (this is ppulated with some samples but you can add them from the list above) in the script at line 22/23 you can toggle whether or not to check for duplicates. It is important to check for duplicates as i plan to have this as a set and forget script that runs daily/weekly and as needed only maintain the cidr.txt file to add ranges to the allow list. Without checking for duplicates it would mean that the list would grow by over 200,000 entries every time it was run. I cannot see how i can speed up the duplicate process, and am hopeing that someone with a lot more talent than me will have some idea's Cheers. AndrewCIDR.au3GreyList.txtcidr.txt Link to comment Share on other sites More sharing options...
Andlin Posted November 15, 2011 Author Share Posted November 15, 2011 bump* Link to comment Share on other sites More sharing options...
Andlin Posted November 17, 2011 Author Share Posted November 17, 2011 expandcollapse popup#Include<Array.au3> #include<Date.au3> #include<File.au3> dim $Oct[4] dim $Parts[2] dim $Nmask Dim $StrIP Dim $Bin Dim $Bin1 Dim $BinIP Dim $Net Dim $Nodes Dim $DecIP[6] Dim $NoOfIPs Dim $Written Dim $Cidr Dim $Begin Dim $TDiff Dim $NewContent Dim $aData[8000000][2] Dim $aDataUnique[8000000][2] Dim $aLine[2] Dim $Line Dim $File Dim $xCounter $Begin = TimerInit() $Written = 0 $NewContent = "" ConsoleWrite("Read cidr.txt"& @CR) if Not _FileReadToArray("cidr.txt", $Cidr) Then MsgBox(4096,"Error", " Error reading CIDR to Array error:" & @error) Exit EndIf For $xCounter = 1 to $Cidr[0] $StrIP = $Cidr[$xCounter] $Parts = StringSplit($StrIP,"/") $Nmask = $Parts[2] $Oct = StringSplit($Parts[1],".") Call ("_ToBinary2", $Oct[1]) while StringLen($Bin) < 8 $Bin = "0" & $Bin WEnd $BinIP = $Bin Call ("_ToBinary2", $Oct[2]) while StringLen($Bin) < 8 $Bin = "0" & $Bin WEnd $BinIP = $BinIP & $Bin Call ("_ToBinary2", $Oct[3]) while StringLen($Bin) < 8 $Bin = "0" & $Bin WEnd $BinIP = $BinIP & $Bin Call ("_ToBinary2", $Oct[4]) while StringLen($Bin) < 8 $Bin = "0" & $Bin WEnd $BinIP = $BinIP & $Bin $Net = StringLeft($BinIP, $Nmask) $Nodes = StringRight($BinIP, 32-$Nmask) $x = StringLen($Nodes) $NoOfIPs = 1 for $i = 1 to $x $NoOfIPs = $NoOfIPs * 2 Next Call ("_todecimal", $BinIP) ;Get current contents of file $CurrContent = FileRead("GreyList.txt") $DateSet = _DateDiff( 'D',"1900/01/01 00:00:00", _NowCalc()) $DateSet = $DateSet + 365 ;Add 365 Days to the date ; Send IP Range to text file if FileExists($xCounter & ".txt") Then FileDelete($xCounter & ".txt") ConsoleWrite("createing file " & $xCounter & ".txt" & @CR) $File =FileOpen($xCounter & ".txt", 1) ; Check if file opened for writing OK If $File = -1 Then MsgBox(0, "Error", "Unable to open file.") Exit EndIf ProgressOn("Writing to file " & $xCounter & " of " & $Cidr[0], "Writing " & $NoOfIPs & " IP Addresses", "", -1, -1, 16+2) For $LineCounter = 0 to $NoOfIPs - 1 Call ("_ToBinary2", $LineCounter) while StringLen($Bin) < StringLen($Nodes) $Bin = "0" & $Bin WEnd $BinIP = $Net & $Bin Call ("_todecimal", $BinIP) FileWriteLine($File, $DecIP[5] & "~" & $DateSet) $Written = $Written + 1 $PC = $LineCounter / $NoOfIPs * 100 ProgressSet( $PC, $LineCounter) Next FileClose($File) ProgressSet(100 , "Complete", $StrIP & " - " & $xCounter & " of " & $Cidr[0]) sleep(500) ProgressOff() Next ;combine the text files into the allow list ConsoleWrite("Combine Text Files"& @CR) For $xCounter = 1 to $Cidr[0] $FileNew = FileRead($xCounter & ".txt") $NewContent = $NewContent & $FileNew FileDelete($xCounter & ".txt") Next $File = FileOpen("GreyList.txt",1) If $File = -1 Then MsgBox(0, "Error", "Unable to open file.") Exit EndIf ConsoleWrite("$File " & $File & @CR) ;ConsoleWrite("Delete old allow list"& @CR) ;FileDelete($File) ; to Ensure data not appended ConsoleWrite("Create New allow list"& @CR) FileWrite($File, $NewContent) FileClose($File) sleep(1000) ConsoleWrite("Create Array of new data"& @CR) $File = FileOpen("GreyList.txt",0) If $File = -1 Then MsgBox(0, "Error", "Unable to open file.") Exit EndIf ProgressOn("Createing Array", "", -1, -1, 16+2) $xCounter = 0 $nCounter = _FileCountLines("GreyList.txt") ConsoleWrite("$nCounter " & $nCounter & @CR) While 1 ;~ ConsoleWrite("$xcounter " & $xCounter & @CR) ;~ ConsoleWrite("$File " & $File & @CR) $Line = FileReadLine($File) If @error = -1 Then ExitLoop ;~ ConsoleWrite("Survived readline" & @CR) $aLine = StringSplit($Line, "~") $aData[$xCounter][0] = $aLine[1] $aData[$xCounter][1] = $aLine[2] $xCounter = $xCounter + 1 ProgressSet( $xCounter/$nCounter*100, $xCounter & " of " & $nCounter) ; Sleep(100) WEnd ProgressSet(100 , "Complete", $xCounter) sleep(500) ProgressOff() ConsoleWrite("ReDim array with new length of " & $xCounter & @CR) ReDim $aData[$xCounter][2] ; Sort Array ConsoleWrite("Sort Array"& @CR) ProgressOn("Sorting Array", "", -1, -1, 16+2) ProgressSet( 50, "Be Patient!") if Not _ArraySort($aData, 0, 0, 0, 0) Then MsgBox(4096,"Error", " Error sorting Array error:" & @error) Exit EndIf ProgressSet(100 , "Complete", $xCounter) Sleep(1000) ProgressOff() ;~ ;remove duplicates ;~ ConsoleWrite("Remove Duplicates"& @CR) ;~ ProgressOn("Removing Duplicates", "", -1, -1, 16+2) ;~ $xCounter = 0 ;~ $Removed = 0 ;~ $aDataCount = UBound($aData)-1 ;~ for $icounter = 0 to UBound($aData)-1 ;~ $xCounter = $xCounter + 1 ;~ if $icounter > 0 Then ;~ if $aData[$icounter][0] = $aData[$icounter-1][0] Then ;~ _ArrayDelete($aData, $icounter) ;~ $icounter = $iCounter -1 ;~ $Removed = $Removed + 1 ;~ EndIf ;~ EndIf ;~ ProgressSet($icounter/UBound($aData)*100 , $icounter) ;~ if $xCounter = $aDataCount+1 then ExitLoop ;~ Next ;~ ProgressSet(100 , "Complete") ;~ Sleep(1000) ;~ ProgressOff() ;remove duplicates ConsoleWrite("Remove Duplicates"& @CR) ProgressOn("Removing Duplicates", "", -1, -1, 16+2) $xCounter = 0 $Removed = 0 $nCounter = 0 $aDataCount = UBound($aData)-1 for $icounter = 0 to UBound($aData)-1 $xCounter = $xCounter + 1 if $icounter > 0 Then if $aData[$icounter][0] = $aData[$icounter-1][0] Then ; _ArrayDelete($aData, $icounter) ; $icounter = $iCounter -1 $Removed = $Removed + 1 Else $aDataUnique[$nCounter][0] = $aData[$icounter][0] $aDataUnique[$nCounter][1] = $aData[$icounter][1] $nCounter = $nCounter +1 EndIf Else $aDataUnique[$nCounter][0] = $aData[$icounter][0] $aDataUnique[$nCounter][1] = $aData[$icounter][1] $nCounter = $nCounter +1 EndIf ProgressSet($icounter/UBound($aData)*100 , $icounter) if $xCounter = $aDataCount+1 then ExitLoop Next ReDim $aDataUnique[$nCounter][2] ProgressSet(100 , "Complete") Sleep(1000) ProgressOff() ; Write array to file ConsoleWrite("Write new array with unique entries"& @CR) ProgressOn("Writeing Array", "", -1, -1, 16+2) while FileExists("GreyListSorted.txt") = 1 FileDelete("GreyListSorted.txt") sleep(10) ConsoleWrite("Deleteing GreyListSorted.txt" & @CR) WEnd $File = FileOpen("GreyListSorted.txt", 1) for $iCounter = 0 to UBound($aDataUnique)-1 ; ConsoleWrite("$icounter = " & $iCounter & @CR) FileWriteLine($File, $aDataUnique[$iCounter][0] & "~" & $aDataUnique[$iCounter][1]) ProgressSet( $iCounter/UBound($aDataUnique)*100, $iCounter) Next FileClose($File) ProgressSet(100 , "Complete", $xCounter) Sleep(5000) ProgressOff() ;FileDelete("GreyList.txt") $TDiff = TimerDiff($Begin) MsgBox(1,"Complete Report", $Written & " Written" & @CR & $Removed & " Skipped" & @CR & "Time Taken " & $TDiff) Exit Func _ToBinary2($DecNum) ; Calculate number of Binary positions $count = 0 $UNum = 0 $Bin1 = "" while $UNum <= $DecNum $count = $count + 1 $UNum = $UNum * 2 if $UNum = 0 then $UNum = 1 WEnd $count = $count - 1 if $count = 0 then $count = 1 While $count > 0 $x = 1 for $i = 2 to $count $x = $x * 2 Next if $DecNum >= $x Then $Bin1 = $Bin1 & "1" $DecNum = $DecNum - $x Else $Bin1 = $Bin1 & "0" EndIf $count = $count - 1 WEnd $Bin = $Bin1 Return $Bin EndFunc Func _ToDecimal($BinNum) ; split the binary to 4 groups of 8 $oct[1] = StringLeft($BinNum,8) $oct[2] = StringMid($BinNum,9,8) $oct[3] = StringMid($BinNum,17,8) $oct[4] = StringRight($BinNum,8) for $x = 1 to 4 $count = 1 $Dec = 0 for $i = 8 to 1 step -1 $Dec = $Dec + (StringMid($Oct[$x], $i, 1) * $count) $count = $count * 2 Next $DecIP[$x] = $Dec Next $DecIP[5] = $DecIP[1] & "." & $DecIP[2] & "." & $DecIP[3] & "." & $DecIP[4] EndFunc OK i have taken this from a script that took 17+ Houirs to run, down to first run taking less than 4 Minutes. Second run takes 39 Minutes as now the original data is 240,000 lines long and all the new entries will be rejected as duplicates. But this is far more acceptable. Although i can see that if the data gets too much i will run into issues with the 8,000,000 limit of a 2d array. But will cross that bridge when it happens. For any others reading this i will give an explanation of what i did. first write all the data to the file, allowing duplicates. then sort the data (2d Array) by the column required. Step through the data and compare each line ($aArray[$counter][0]) to the previous lines data. if it is a duplicate then ignore, if it is different then write the data to a new 2d array. save the new 2d array to file. The issue is of course that because you are writing the data to the array with duplicates, if you have allready run the app once and the Cidr.txt files contains entries that are allready posted then you have doubled the size of the array. I may have to revisit this as the time blows out, but if any others have suggestions they will be welcomed. Just wish AutoIT forum had a means to notify me via email when this post is changed. Cheers. Link to comment Share on other sites More sharing options...
BrewManNH Posted November 17, 2011 Share Posted November 17, 2011 Try using _ArrayUnique and see if that might speed up the process. What it does is remove any duplicates from a 1 or 2D array. Don't go by the help file description of what it does, it does work on 2D arrays. If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag GudeHow to ask questions the smart way! I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from. Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays. - ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script. - Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label. - _FileGetProperty - Retrieve the properties of a file - SciTE Toolbar - A toolbar demo for use with the SciTE editor - GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI. - Latin Square password generator Link to comment Share on other sites More sharing options...
Andlin Posted November 18, 2011 Author Share Posted November 18, 2011 Thanks for the reply BrewManNH. I did see the command _ArrayUnique and thought that all my prayers had been answered but unfortunatly i found that it only "Returns a 1-dimensional array containing only the unique elements of that Dimension" This was cut from the help file on it, and in my testing this is what i found. unless i got something wrong. Unfortunatley i have deleted the section of code where i was using the _arrayUnique (Damn i usually just comment sections out till i am complete and then want to tidyup). are you saying that i can populate a 2D array, then use _ArrayUnique to get rid of duplicates in column 1 and give a result that is still a 2D array? if so a short sample script would be brilliant. Cheers. Link to comment Share on other sites More sharing options...
BrewManNH Posted November 18, 2011 Share Posted November 18, 2011 No, _ArrayUnique will return a 1D array from one of the dimensions ("columns") of a 1 or 2D array. You can use that array to populate another array with its contents. You could use it to regenerate the 2D array with the current contents of that 2D array, and the contents of the 1D array returned from _ArrayUnique. If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag GudeHow to ask questions the smart way! I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from. Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays. - ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script. - Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label. - _FileGetProperty - Retrieve the properties of a file - SciTE Toolbar - A toolbar demo for use with the SciTE editor - GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI. - Latin Square password generator Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now