sam2332 Posted October 13, 2009 Share Posted October 13, 2009 (edited) Ok im having some lag issues. I am splitting a proxy list that is immensely huge (like 2.5 gb)and, I need some optimizations if you could share some here is my code #include <array.au3> $file = FileOpen(@ScriptDir & "\proxys.txt", 0) $list = FileRead($file) FileClose($file) $splita = StringSplit($list, @CRLF) $file = FileOpen(@ScriptDir &"\fixed proxys.txt",1) For $i = 1 To $splita[0] - 1 $dataa = StringSplit($splita[$i], " ") $dataa = StringSplit($dataa[1],":") for $b = 0 to ubound($dataa)-1 Next if $dataa[0] = 2 then ConsoleWrite($dataa[1] & ":" & $dataa[2] & @CRLF) FileWrite($file,$dataa[1] & ":" & $dataa[2] & @crlf) EndIf Next FileClose($file) the proxys are in this format 85.119.217.110:1080 server support: 4 Belgium (Meise) 85.119.217.111:1080 server support: 4 Belgium (Meise) 85.119.217.112:1080 server support: 4 Belgium (Meise) 85.119.217.113:1080 server support: 4 Belgium (Meise) 194.7.83.94:1080 server support: 4 Belgium (Tessenderlo) 200.87.196.131:1080 server support: 4 Bolivia (La Paz) any help would be greatly appreciated edit: hahahha i didnt even notice the empty for next loop lol it was for debug purposes xD Edited October 13, 2009 by sam2332 Link to comment Share on other sites More sharing options...
Zedna Posted October 13, 2009 Share Posted October 13, 2009 (edited) Try this. But I'm not RegExp expert so check if RegExp format string is correct first! #include <array.au3> $file = FileOpen(@ScriptDir & "\proxys.txt", 0) $list = FileRead($file) FileClose($file) $splita = StringRegExp($list, '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,4}', 3) ; 85.119.217.110:1080 $file = FileOpen(@ScriptDir &"\fixed proxys.txt",1) For $i = 1 To UBound($splita) - 1 ConsoleWrite($splita[$i] & @CRLF) FileWrite($file,$splita[$i] & @crlf) Next FileClose($file) EDIT: In your code you have empty inner FOR/NEXT loop!! Edited October 13, 2009 by Zedna Resources UDF ResourcesEx UDF AutoIt Forum Search Link to comment Share on other sites More sharing options...
Authenticity Posted October 13, 2009 Share Posted October 13, 2009 lol, so you read 2.5 GB at once... only this could take a fare amount of time and then working on that giant data is also fare amount of time. Do a test, read first like 32 MB chunks and go upward 64, 128, .. until you find something better for data reading and manipulation. If you'll read at the middle of the last line of the chunk, leave it at a buffer for the next read. I did so and it was pretty fast and perfectly correct. Link to comment Share on other sites More sharing options...
sam2332 Posted October 13, 2009 Author Share Posted October 13, 2009 thank you zedna that worked perfectly and im also changing it to use filereadline so i dont have have a huge buffer lol Link to comment Share on other sites More sharing options...
Zedna Posted October 13, 2009 Share Posted October 13, 2009 and im also changing it to use filereadline so i dont have have a huge buffer lolDon't use FileReadLine!! It's the worst thing you can do. It's not speed optimal in any way. Resources UDF ResourcesEx UDF AutoIt Forum Search Link to comment Share on other sites More sharing options...
sam2332 Posted October 14, 2009 Author Share Posted October 14, 2009 Don't use FileReadLine!! It's the worst thing you can do. It's not speed optimal in any way.O.o ok lol Link to comment Share on other sites More sharing options...
Zedna Posted October 14, 2009 Share Posted October 14, 2009 Don't use FileReadLine!! It's the worst thing you can do. It's not speed optimal in any way.Use _WinAPI_SetFilePointer(), _WinAPI_ReadFile() instead.http://www.autoitscript.com/forum/index.php?showtopic=76829 Resources UDF ResourcesEx UDF AutoIt Forum Search Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now