Jump to content
Sign in to follow this  
sam2332

code optamization help

Recommended Posts

sam2332

Ok im having some lag issues.

I am splitting a proxy list that is immensely huge (like 2.5 gb)and, I need some optimizations if you could share some :)

here is my code

#include <array.au3>
$file = FileOpen(@ScriptDir & "\proxys.txt", 0)
$list = FileRead($file)
FileClose($file)
$splita = StringSplit($list, @CRLF)
$file = FileOpen(@ScriptDir &"\fixed proxys.txt",1)
For $i = 1 To $splita[0] - 1
    $dataa = StringSplit($splita[$i], "      ")
    $dataa = StringSplit($dataa[1],":")
    for $b = 0 to ubound($dataa)-1
    Next
    if $dataa[0] = 2 then
        ConsoleWrite($dataa[1] & ":" & $dataa[2] & @CRLF)
    FileWrite($file,$dataa[1] & ":" & $dataa[2] & @crlf)
    EndIf
Next
FileClose($file)

the proxys are in this format

85.119.217.110:1080 server support: 4 Belgium (Meise)

85.119.217.111:1080 server support: 4 Belgium (Meise)

85.119.217.112:1080 server support: 4 Belgium (Meise)

85.119.217.113:1080 server support: 4 Belgium (Meise)

194.7.83.94:1080 server support: 4 Belgium (Tessenderlo)

200.87.196.131:1080 server support: 4 Bolivia (La Paz)

any help would be greatly appreciated

edit: hahahha i didnt even notice the empty for next loop lol it was for debug purposes xD

Edited by sam2332

Share this post


Link to post
Share on other sites
Zedna

Try this.

But I'm not RegExp expert so check if RegExp format string is correct first!

#include <array.au3>

$file = FileOpen(@ScriptDir & "\proxys.txt", 0)
$list = FileRead($file)
FileClose($file)

$splita = StringRegExp($list, '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,4}', 3) ; 85.119.217.110:1080
$file = FileOpen(@ScriptDir &"\fixed proxys.txt",1)
For $i = 1 To UBound($splita) - 1
 ConsoleWrite($splita[$i] & @CRLF)
    FileWrite($file,$splita[$i] & @crlf)
Next
FileClose($file)

EDIT: In your code you have empty inner FOR/NEXT loop!!

Edited by Zedna

Share this post


Link to post
Share on other sites
Authenticity

lol, so you read 2.5 GB at once... only this could take a fare amount of time and then working on that giant data is also fare amount of time. Do a test, read first like 32 MB chunks and go upward 64, 128, .. until you find something better for data reading and manipulation. If you'll read at the middle of the last line of the chunk, leave it at a buffer for the next read. I did so and it was pretty fast and perfectly correct.

Share this post


Link to post
Share on other sites
sam2332

thank you zedna that worked perfectly

and im also changing it to use filereadline so i dont have have a huge buffer lol

Share this post


Link to post
Share on other sites
sam2332

Don't use FileReadLine!!

It's the worst thing you can do. It's not speed optimal in any way.

O.o ok lol

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.