Jump to content

IPRange Extractor


Recommended Posts

Hi again guys!, i had COVID-19 for twice and i couldn't check the forum since 3 or 4 months ago till now! i hope you will get better if you're fighting for beat COVID-19 :D

I have two question, first is about extracting all of the IP Address from an IP Ranges, for e.g: 192.168.1.1-192.255.255.255 (Start and End are variable and will be defined by the user) and for second one, i have a friend that he is Python programmer, he made a IP Parser that it can support large txt files (1TB) and it can parse all of them under 10min and it also supports low-end PCs that have 1 GB RAM!

The list that his program parses are:

#1765497        192.168.1.1     8082
#1765496        192.168.1.1     8084
#1965493        192.168.1.1     8089
#9565495        192.168.1.1     8086
#2565492        192.168.1.1     8081

and it converts very very fast to this:

192.168.1.1:8082
192.168.1.1:8084
192.168.1.1:8089
192.168.1.1:8086
192.168.1.1:8081

I wonder how to do this via AutoIt, if you can help me in this way, i will be happy✌❤

Thanks for your helps.

FIRST_QUESTION_TEST.au3

Edited by Colduction
Link to comment
Share on other sites

  • Moderators

To both of your questions, what have you tried on your own? I see no code.

More specifically on the first question "extract range of IPs" from what source? Are you saying just give it a start and end, and it spits out every address in between?

Edited by JLogan3o13

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites

  • Moderators

Still not seeing any code, show what you have tried even if it is not working as you would like.

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites

For others, here is the contents of Test.au3;

 

Global $g_sRegexPattern_T = "#.+?\t\t(.+)\t\t(.+)"
Global $g_asFR_T = StringSplit(FileRead(@ScriptDir & "\1.txt"), @CRLF, 3)

For $i In $g_asFR_T
    ConsoleWrite(StringRegExpReplace($i, $g_sRegexPattern_T, "$1:$2") & @CRLF)
Next

 

Edited by pseakins

Phil Seakins

Link to comment
Share on other sites

Hey,

For your first question, If you are unfamiliar with the concept, maybe incorporate Network IDs into that with subnet mask?

All IPs in Network 192.168.0.0/16 are 192.168.0.1 - 192.168.255.254 (16 bit subnet mask - 255.255.0.0)

All IPs in Network 192.168.0.0/24 are 192.168.0.1 - 192.168.0.254 (24 bit subnet mask - 255.255.255.0)

Computation would be like this:

4 octets

11111111.11111111.00000000.00000000 = (16 bits) 255.255.0.0

For each octet: (2 ^ 7) * bit + (2 ^ 6) * bit + ..... + (2 ^ 0) * bit 

 

Can also be like (subnet classing, iirc)

 11111111.11111111.11111000.00000000 = (21 bits) 255.255.248.0

This would leave 000.00000000 for the devices to get an IP.

IPs in this network would be: 192.168.248.1 - 192.168.255.254

First IP address of the network (all 0's after the 1 bits of Network ID) is part of the network ID and

Final address of the network (all 1's after the 1 bits of Network ID) is the broadcast IP, no device will get those.

Only mentioned this because your example range ends with 255. :) And that is the broadcast IP.

Link to comment
Share on other sites

Well... you can start by not splitting on every newline. RegEx will apply to each line:

Global $g_sRegexPattern_T = "#.+?\t\t(.+)\t\t(.+)"
ConsoleWrite(StringRegExpReplace(FileRead(@ScriptDir & "\1.txt"), $g_sRegexPattern_T, "$1:$2") & @CRLF)

All my code provided is Public Domain... but it may not work. ;) Use it, change it, break it, whatever you want.

Spoiler

My Humble Contributions:
Personal Function Documentation - A personal HelpFile for your functions
Acro.au3 UDF - Automating Acrobat Pro
ToDo Finder - Find #ToDo: lines in your scripts
UI-SimpleWrappers UDF - Use UI Automation more Simply-er
KeePass UDF - Automate KeePass, a password manager
InputBoxes - Simple Input boxes for various variable types

Link to comment
Share on other sites

18 hours ago, seadoggie01 said:

Well... you can start by not splitting on every newline. RegEx will apply to each line:

Global $g_sRegexPattern_T = "#.+?\t\t(.+)\t\t(.+)"
ConsoleWrite(StringRegExpReplace(FileRead(@ScriptDir & "\1.txt"), $g_sRegexPattern_T, "$1:$2") & @CRLF)

Hi @seadoggie01, it's not very important too, but thanks anyway. i need to solve numbered questions

Link to comment
Share on other sites

35 minutes ago, Colduction said:

i need to solve numbered questions

I don't understand what you mean here. Can you explain further? Is there more that you need to do that depends on each individual line? Even if you do, it would seem to be a lot better to apply the RegEx once then loop through lines instead of looping through lines and applying RegEx per line

All my code provided is Public Domain... but it may not work. ;) Use it, change it, break it, whatever you want.

Spoiler

My Humble Contributions:
Personal Function Documentation - A personal HelpFile for your functions
Acro.au3 UDF - Automating Acrobat Pro
ToDo Finder - Find #ToDo: lines in your scripts
UI-SimpleWrappers UDF - Use UI Automation more Simply-er
KeePass UDF - Automate KeePass, a password manager
InputBoxes - Simple Input boxes for various variable types

Link to comment
Share on other sites

Assuming your input file is TAB delimited, this is one way I would do it. The drawback here is that the entire file is read into memory which is not really necessary for this kind of simple conversion. I will probably make a second post where the file is processed line by line. which may or may not be quicker.

#include <File.au3>
#include <FileConstants.au3>
#include <Array.au3>

Dim $aFileData
$hFileOut = FileOpen(@ScriptDir & "\1out.txt", $FO_OVERWRITE)
_FileReadToArray(@ScriptDir & "\1.txt", $aFileData, $FRTA_NOCOUNT, @TAB)

;~  _ArrayDisplay($aFileData)

For $i = 0 To UBound($aFileData) - 1
  FileWriteLine($hFileOut, $aFileData[$i][1] & ":" & $aFileData[$i][2])
Next

FileClose($hFileOut)

 

The thing is, with file handling under Windows, when reading\writing a file, the file is accessed one (or more) sectors (512 bytes) at a time and the the required part of the IO buffer is then swapped to the application. So, two subsequent reads do not necessarily go and hit the disk because this is all handled by the operating system. The OS makes the decisions about what to do with stuff in the buffers. You can get all fancy with your code only to find that the OS is thinking for you. The FileWriteLine() statements in the above code will only be written to the disk when the buffer has sufficient bytes to warrant a disk access.

Phil Seakins

Link to comment
Share on other sites

This second example avoids loading files into memory. Both my examples dispense with RegEx, which surely would introduce its own overheads given that a regular expression would need to be interpreted by the regex handler. This example can handle any size file.

#include <FileConstants.au3>

$hFilein = FileOpen(@ScriptDir & "\1.txt")
$hFileOut = FileOpen(@ScriptDir & "\1out.txt", $FO_OVERWRITE)
$sLine = FileReadLine($hFilein)
$iEOF = @error
While Not $iEOF
  $sLine = StringSplit($sLine, @TAB)
  FileWriteLine($hFileOut, $sLine[2] & ":" & $sLine[3])
  $sLine = FileReadLine($hFilein)
  $iEOF = @error
WEnd
FileClose($hFilein)
FileClose($hFileOut)

 

Edited by pseakins
added FileClose()

Phil Seakins

Link to comment
Share on other sites

2 hours ago, Colduction said:

i think using Regex without loop can increase CPU loading

I'd tend to think the opposite. Usually (iirc) RegEx is much faster than attempting to parse the string in AutoIt.

Something else to consider... do you have Excel? You might could beat the python time utilizing Excel... I have no way to test that without a massive file though... hmm...

All my code provided is Public Domain... but it may not work. ;) Use it, change it, break it, whatever you want.

Spoiler

My Humble Contributions:
Personal Function Documentation - A personal HelpFile for your functions
Acro.au3 UDF - Automating Acrobat Pro
ToDo Finder - Find #ToDo: lines in your scripts
UI-SimpleWrappers UDF - Use UI Automation more Simply-er
KeePass UDF - Automate KeePass, a password manager
InputBoxes - Simple Input boxes for various variable types

Link to comment
Share on other sites

On 11/6/2020 at 3:27 AM, seadoggie01 said:

Usually (iirc) RegEx is much faster than attempting to parse the string in AutoIt.

I saw a benchmark on the internet that Original String functions are stronger than Regex

On 11/6/2020 at 3:27 AM, seadoggie01 said:

do you have Excel?

No, i have not, i'd prefer to use just a script (of mine) instead of using other 3rd party programs

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...