Sign in to follow this  
Followers 0
sam2332

code optamization help

7 posts in this topic

#1 ·  Posted (edited)

Ok im having some lag issues.

I am splitting a proxy list that is immensely huge (like 2.5 gb)and, I need some optimizations if you could share some :)

here is my code

#include <array.au3>
$file = FileOpen(@ScriptDir & "\proxys.txt", 0)
$list = FileRead($file)
FileClose($file)
$splita = StringSplit($list, @CRLF)
$file = FileOpen(@ScriptDir &"\fixed proxys.txt",1)
For $i = 1 To $splita[0] - 1
    $dataa = StringSplit($splita[$i], "      ")
    $dataa = StringSplit($dataa[1],":")
    for $b = 0 to ubound($dataa)-1
    Next
    if $dataa[0] = 2 then
        ConsoleWrite($dataa[1] & ":" & $dataa[2] & @CRLF)
    FileWrite($file,$dataa[1] & ":" & $dataa[2] & @crlf)
    EndIf
Next
FileClose($file)

the proxys are in this format

85.119.217.110:1080 server support: 4 Belgium (Meise)

85.119.217.111:1080 server support: 4 Belgium (Meise)

85.119.217.112:1080 server support: 4 Belgium (Meise)

85.119.217.113:1080 server support: 4 Belgium (Meise)

194.7.83.94:1080 server support: 4 Belgium (Tessenderlo)

200.87.196.131:1080 server support: 4 Bolivia (La Paz)

any help would be greatly appreciated

edit: hahahha i didnt even notice the empty for next loop lol it was for debug purposes xD

Edited by sam2332

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Try this.

But I'm not RegExp expert so check if RegExp format string is correct first!

#include <array.au3>

$file = FileOpen(@ScriptDir & "\proxys.txt", 0)
$list = FileRead($file)
FileClose($file)

$splita = StringRegExp($list, '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,4}', 3) ; 85.119.217.110:1080
$file = FileOpen(@ScriptDir &"\fixed proxys.txt",1)
For $i = 1 To UBound($splita) - 1
 ConsoleWrite($splita[$i] & @CRLF)
    FileWrite($file,$splita[$i] & @crlf)
Next
FileClose($file)

EDIT: In your code you have empty inner FOR/NEXT loop!!

Edited by Zedna

Share this post


Link to post
Share on other sites

lol, so you read 2.5 GB at once... only this could take a fare amount of time and then working on that giant data is also fare amount of time. Do a test, read first like 32 MB chunks and go upward 64, 128, .. until you find something better for data reading and manipulation. If you'll read at the middle of the last line of the chunk, leave it at a buffer for the next read. I did so and it was pretty fast and perfectly correct.

Share this post


Link to post
Share on other sites

thank you zedna that worked perfectly

and im also changing it to use filereadline so i dont have have a huge buffer lol

Share this post


Link to post
Share on other sites

Don't use FileReadLine!!

It's the worst thing you can do. It's not speed optimal in any way.

O.o ok lol

Share this post


Link to post
Share on other sites

Don't use FileReadLine!!

It's the worst thing you can do. It's not speed optimal in any way.

Use _WinAPI_SetFilePointer(), _WinAPI_ReadFile() instead.

http://www.autoitscript.com/forum/index.php?showtopic=76829

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0