Add cidr to greylist

[Andlin]

I have a spam filter that checks a greylist file for allowed IP address' the issue that i have is with Gmail and MSN Mail (for these purposes we will anly look at GMail)

In normal operation when a mail server attempts a connection to the spam server it looks for the server's IP in a grey list file, if it is not there it regects the connection and requests a retry after x seconds (300). Then after 300 seconds if it detects the server attempting connection again it assumes that it is serious and adds it to the allow list with a date/time stamp and accepts the connection and then goes through the normal spam interogation. This is very handy as 90% of spammers do not retry.

Problem:

Gmail have a mail farm, so when the first server gets rejected the next attempt after the x seconds may come from a different IP address and as such it will then go into the reject and greylist wait, and so on........

Gmail in their SPF list their valid IP address' as following

216.239.32.0/19

64.233.160.0/19

66.249.80.0/20

72.14.192.0/18

209.85.128.0/17

66.102.0.0/20

74.125.0.0/16

64.18.0.0/20

207.126.144.0/20

173.194.0.0/16

Thats about 200,000 IP's (I dont believe they have that many mail servers, but who am i to say)

Solution:

Create a script that converts the CIDR to a list of IP addresses and append it to the current allow list, While checking for duplicates to ensure that the same IP doesnt appear twice.

Problem with my solution:

by checking a file (currently without GMail's IP's its about 19,580 Entries long) for an entry that allready exists it slows the script to a crawl.

I have attached a copy of the au3 script along with the greylist.txt file that contains the allow list (I have prepopulated it with some entries), also i have attached cidr.txt which is a file that stores the list of cidr's that need to be added. (this is ppulated with some samples but you can add them from the list above)

in the script at line 22/23 you can toggle whether or not to check for duplicates.

It is important to check for duplicates as i plan to have this as a set and forget script that runs daily/weekly and as needed only maintain the cidr.txt file to add ranges to the allow list. Without checking for duplicates it would mean that the list would grow by over 200,000 entries every time it was run.

I cannot see how i can speed up the duplicate process, and am hopeing that someone with a lot more talent than me will have some idea's

Cheers.

Andrew

CIDR.au3

GreyList.txt

cidr.txt

[Andlin]

bump*

[Andlin]

expandcollapse popup

#Include<Array.au3>
#include<Date.au3>
#include<File.au3>
dim $Oct[4]
dim $Parts[2]
dim $Nmask
Dim $StrIP
Dim $Bin
Dim $Bin1
Dim $BinIP
Dim $Net
Dim $Nodes
Dim $DecIP[6]
Dim $NoOfIPs
Dim $Written
Dim $Cidr
Dim $Begin
Dim $TDiff
Dim $NewContent
Dim $aData[8000000][2]
Dim $aDataUnique[8000000][2]
Dim $aLine[2]
Dim $Line
Dim $File
Dim $xCounter
$Begin = TimerInit()
$Written = 0
$NewContent = ""
ConsoleWrite("Read cidr.txt"& @CR)
if Not _FileReadToArray("cidr.txt", $Cidr) Then
MsgBox(4096,"Error", " Error reading CIDR to Array   error:" & @error)
Exit
EndIf
For $xCounter = 1 to $Cidr[0]
$StrIP = $Cidr[$xCounter]
$Parts = StringSplit($StrIP,"/")
$Nmask = $Parts[2]
$Oct = StringSplit($Parts[1],".")
Call ("_ToBinary2", $Oct[1])
while StringLen($Bin) < 8
  $Bin = "0" & $Bin
WEnd
$BinIP = $Bin
Call ("_ToBinary2", $Oct[2])
while StringLen($Bin) < 8
  $Bin = "0" & $Bin
WEnd
$BinIP = $BinIP & $Bin
Call ("_ToBinary2", $Oct[3])
while StringLen($Bin) < 8
  $Bin = "0" & $Bin
WEnd
$BinIP = $BinIP & $Bin
Call ("_ToBinary2", $Oct[4])
while StringLen($Bin) < 8
  $Bin = "0" & $Bin
WEnd
$BinIP = $BinIP & $Bin
$Net = StringLeft($BinIP, $Nmask)
$Nodes = StringRight($BinIP, 32-$Nmask)
$x = StringLen($Nodes)
$NoOfIPs = 1
for $i = 1 to $x
  $NoOfIPs = $NoOfIPs * 2
Next
Call ("_todecimal", $BinIP)
;Get current contents of file
$CurrContent = FileRead("GreyList.txt")
$DateSet = _DateDiff( 'D',"1900/01/01 00:00:00", _NowCalc())
$DateSet = $DateSet + 365 ;Add 365 Days to the date
; Send IP Range to text file
if FileExists($xCounter & ".txt") Then FileDelete($xCounter & ".txt")
ConsoleWrite("createing file " & $xCounter & ".txt" & @CR)
$File =FileOpen($xCounter & ".txt", 1)
; Check if file opened for writing OK
If $File = -1 Then
  MsgBox(0, "Error", "Unable to open file.")
  Exit
EndIf
ProgressOn("Writing to file " & $xCounter & " of " & $Cidr[0], "Writing " & $NoOfIPs & " IP Addresses", "", -1, -1, 16+2)
For $LineCounter = 0 to $NoOfIPs - 1
  Call ("_ToBinary2", $LineCounter)
  while StringLen($Bin) < StringLen($Nodes)
   $Bin = "0" & $Bin
  WEnd
  $BinIP = $Net & $Bin
  Call ("_todecimal", $BinIP)
  FileWriteLine($File, $DecIP[5] & "~" & $DateSet)
  $Written = $Written + 1
  $PC = $LineCounter / $NoOfIPs * 100
  ProgressSet( $PC, $LineCounter)
Next
FileClose($File)
ProgressSet(100 , "Complete", $StrIP & " - " & $xCounter & " of " & $Cidr[0])
sleep(500)
ProgressOff()
Next
;combine the text files into the allow list
ConsoleWrite("Combine Text Files"& @CR)
For $xCounter = 1 to $Cidr[0]
$FileNew = FileRead($xCounter & ".txt")
$NewContent = $NewContent & $FileNew
FileDelete($xCounter & ".txt")
Next
$File = FileOpen("GreyList.txt",1)
If $File = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
ConsoleWrite("$File " & $File & @CR)
;ConsoleWrite("Delete old allow list"& @CR)
;FileDelete($File) ; to Ensure data not appended
ConsoleWrite("Create New allow list"& @CR)
FileWrite($File, $NewContent)
FileClose($File)
sleep(1000)
ConsoleWrite("Create Array of new data"& @CR)
$File = FileOpen("GreyList.txt",0)
If $File = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
ProgressOn("Createing Array", "", -1, -1, 16+2)
$xCounter = 0
$nCounter = _FileCountLines("GreyList.txt")
ConsoleWrite("$nCounter " & $nCounter & @CR)
While 1
;~  ConsoleWrite("$xcounter " & $xCounter & @CR)
;~  ConsoleWrite("$File " & $File & @CR)
$Line = FileReadLine($File)
If @error = -1 Then ExitLoop
;~  ConsoleWrite("Survived readline" & @CR)
$aLine = StringSplit($Line, "~")
$aData[$xCounter][0] = $aLine[1]
$aData[$xCounter][1] = $aLine[2]
$xCounter = $xCounter + 1
ProgressSet( $xCounter/$nCounter*100, $xCounter & " of " & $nCounter)
; Sleep(100)
WEnd
ProgressSet(100 , "Complete", $xCounter)
sleep(500)
ProgressOff()
ConsoleWrite("ReDim array with new length of " & $xCounter & @CR)
ReDim $aData[$xCounter][2]
; Sort Array
ConsoleWrite("Sort Array"& @CR)
ProgressOn("Sorting Array", "", -1, -1, 16+2)
ProgressSet( 50, "Be Patient!")
if Not _ArraySort($aData, 0, 0, 0, 0) Then
MsgBox(4096,"Error", " Error sorting Array   error:" & @error)
Exit
EndIf
ProgressSet(100 , "Complete", $xCounter)
Sleep(1000)
ProgressOff()
;~ ;remove duplicates
;~ ConsoleWrite("Remove Duplicates"& @CR)
;~ ProgressOn("Removing Duplicates", "", -1, -1, 16+2)
;~ $xCounter = 0
;~ $Removed = 0
;~ $aDataCount = UBound($aData)-1
;~ for $icounter = 0 to UBound($aData)-1
;~  $xCounter = $xCounter + 1
;~  if $icounter > 0 Then
;~   if $aData[$icounter][0] = $aData[$icounter-1][0] Then
;~    _ArrayDelete($aData, $icounter)
;~    $icounter = $iCounter -1
;~    $Removed = $Removed + 1
;~   EndIf
;~  EndIf
;~  ProgressSet($icounter/UBound($aData)*100 , $icounter)
;~  if $xCounter = $aDataCount+1 then ExitLoop
;~ Next
;~ ProgressSet(100 , "Complete")
;~ Sleep(1000)
;~ ProgressOff()
;remove duplicates
ConsoleWrite("Remove Duplicates"& @CR)
ProgressOn("Removing Duplicates", "", -1, -1, 16+2)
$xCounter = 0
$Removed = 0
$nCounter = 0
$aDataCount = UBound($aData)-1
for $icounter = 0 to UBound($aData)-1
$xCounter = $xCounter + 1
if $icounter > 0 Then
  if $aData[$icounter][0] = $aData[$icounter-1][0] Then
;   _ArrayDelete($aData, $icounter)
;   $icounter = $iCounter -1
   $Removed = $Removed + 1
  Else
   $aDataUnique[$nCounter][0] = $aData[$icounter][0]
   $aDataUnique[$nCounter][1] = $aData[$icounter][1]
   $nCounter = $nCounter +1
  EndIf
Else
  $aDataUnique[$nCounter][0] = $aData[$icounter][0]
  $aDataUnique[$nCounter][1] = $aData[$icounter][1]
  $nCounter = $nCounter +1
EndIf
ProgressSet($icounter/UBound($aData)*100 , $icounter)
if $xCounter = $aDataCount+1 then ExitLoop
Next
ReDim $aDataUnique[$nCounter][2]
ProgressSet(100 , "Complete")
Sleep(1000)
ProgressOff()

; Write array to file
ConsoleWrite("Write new array with unique entries"& @CR)
ProgressOn("Writeing Array", "", -1, -1, 16+2)
while FileExists("GreyListSorted.txt") = 1
FileDelete("GreyListSorted.txt")
sleep(10)
ConsoleWrite("Deleteing GreyListSorted.txt" & @CR)
WEnd
$File = FileOpen("GreyListSorted.txt", 1)
for $iCounter = 0 to UBound($aDataUnique)-1
; ConsoleWrite("$icounter = " & $iCounter & @CR)
FileWriteLine($File, $aDataUnique[$iCounter][0] & "~" & $aDataUnique[$iCounter][1])
ProgressSet( $iCounter/UBound($aDataUnique)*100, $iCounter)
Next
FileClose($File)
ProgressSet(100 , "Complete", $xCounter)
Sleep(5000)
ProgressOff()
;FileDelete("GreyList.txt")
$TDiff = TimerDiff($Begin)
MsgBox(1,"Complete Report", $Written & " Written" & @CR & $Removed & " Skipped" & @CR & "Time Taken " & $TDiff)
Exit
Func _ToBinary2($DecNum)
; Calculate number of Binary positions
$count = 0
$UNum = 0
$Bin1 = ""
while $UNum <= $DecNum
  $count = $count + 1
  $UNum = $UNum * 2
  if $UNum = 0 then $UNum = 1
WEnd
$count = $count - 1
if $count = 0 then $count = 1

While $count > 0
  $x = 1
  for $i = 2 to $count
   $x = $x * 2
  Next
  if $DecNum >= $x Then
   $Bin1 = $Bin1 & "1"
   $DecNum = $DecNum - $x
  Else
   $Bin1 = $Bin1 & "0"
  EndIf
  $count = $count - 1
WEnd
  
$Bin = $Bin1
Return $Bin
EndFunc

Func _ToDecimal($BinNum)
; split the binary to 4 groups of 8
$oct[1] = StringLeft($BinNum,8)
$oct[2] = StringMid($BinNum,9,8)
$oct[3] = StringMid($BinNum,17,8)
$oct[4] = StringRight($BinNum,8)
for $x = 1 to 4
  $count = 1
  $Dec = 0
  for $i = 8 to 1 step -1
    $Dec = $Dec + (StringMid($Oct[$x], $i, 1) * $count)
   $count = $count * 2
  Next
  $DecIP[$x] = $Dec
Next
$DecIP[5] = $DecIP[1] & "." & $DecIP[2] & "." & $DecIP[3] & "." & $DecIP[4]
EndFunc

OK i have taken this from a script that took 17+ Houirs to run, down to first run taking less than 4 Minutes. Second run takes 39 Minutes as now the original data is 240,000 lines long and all the new entries will be rejected as duplicates. But this is far more acceptable. Although i can see that if the data gets too much i will run into issues with the 8,000,000 limit of a 2d array. But will cross that bridge when it happens.

For any others reading this i will give an explanation of what i did.

first write all the data to the file, allowing duplicates. then sort the data (2d Array) by the column required.

Step through the data and compare each line ($aArray[$counter][0]) to the previous lines data. if it is a duplicate then ignore, if it is different then write the data to a new 2d array.

save the new 2d array to file.

The issue is of course that because you are writing the data to the array with duplicates, if you have allready run the app once and the Cidr.txt files contains entries that are allready posted then you have doubled the size of the array.

I may have to revisit this as the time blows out, but if any others have suggestions they will be welcomed. Just wish AutoIT forum had a means to notify me via email when this post is changed.

Cheers.

[BrewManNH]

Try using _ArrayUnique and see if that might speed up the process. What it does is remove any duplicates from a 1 or 2D array. Don't go by the help file description of what it does, it does work on 2D arrays.

[Andlin]

Thanks for the reply BrewManNH.

I did see the command _ArrayUnique and thought that all my prayers had been answered but unfortunatly i found that it only "Returns a 1-dimensional array containing only the unique elements of that Dimension"

This was cut from the help file on it, and in my testing this is what i found.

unless i got something wrong.

Unfortunatley i have deleted the section of code where i was using the _arrayUnique (Damn i usually just comment sections out till i am complete and then want to tidyup).

are you saying that i can populate a 2D array, then use _ArrayUnique to get rid of duplicates in column 1 and give a result that is still a 2D array?

if so a short sample script would be brilliant.

Cheers.

[BrewManNH]

No, _ArrayUnique will return a 1D array from one of the dimensions ("columns") of a 1 or 2D array. You can use that array to populate another array with its contents. You could use it to regenerate the 2D array with the current contents of that 2D array, and the contents of the 1D array returned from _ArrayUnique.