81 replies to this topic

[blindwig]

Only 2 global variables - the other 2 take .001 seconds to make
New decoder - a lot faster

You mentioned that you have an AMD CPU. I have an Intel Pentium. They must be optimized differently, because every time you take out the global variables and say you see a speed increase, I see a decrease. For example, the last code you gave me (with no globals) versus the code I just posted versus the code you just posted:
Using a test file, about 50K, on my P4M 2.2GHz:
Encode: 1.8 seconds -> 1.2 seconds -> 1.6 seconds
Decode: 2.4 seconds -> 1.6 seconds -> 2.1 seconds

So on my machine, your new code is almost as slow as your old code. Obviously our machines process differently. I think that's why you keep moving the tables to local variables and I keep putting them back in globals.

Edited by blindwig, 09 March 2006 - 06:15 PM.

[mikeytown2]

Obviously our machines process differently. I think that's why you keep moving the tables to local variables and I keep putting them back in globals.

Knowing this solves a lot of problems. But one thing i did was shift the build time of the tables from when the process starts to when you need it. So you should compare the 2nd time around when doing an encode and a decode. I think that is the proper way of making a UDF. so that when you include it the script, it shouldn't take as long for the script to start. so in your test file call encode and decode to build the tables, then time them the 2nd time around.

Thanks for the explanation!

EDIT
I forgot to say that there should be no change in speed between your decoder and the one i made once the tables are built. so in my above post where i said "New decoder - a lot faster" i was referring to your decoder not the one i tweaked. I wrote this a little late last night, sorry for the misunderstanding.

Edited by mikeytown2, 09 March 2006 - 06:39 PM.

[blindwig]

Just for fun I did a comparison against the code you first posted here.
Encode: 108 -> 1.4 seconds
Decode: 107 -> 1.8 seconds
That's about 7600% increase for the encoder and over 5800% increase for the decoder!

[SmOke_N]

Just for fun I did a comparison against the code you first posted here.
Encode: 108 -> 1.4 seconds
Decode: 107 -> 1.8 seconds
That's about 7600% increase for the encoder and over 5800% increase for the decoder!

Where the hell where you when I was writing EnCodeIt?

Good job you guys!

[mikeytown2]

Just for fun I did a comparison against the code you first posted here.
Encode: 108 -> 1.4 seconds
Decode: 107 -> 1.8 seconds
That's about 7600% increase for the encoder and over 5800% increase for the decoder!

I know it got quicker, but i wasn't expecting this!

blindwig do you think this is ready to be added as a UDF?


I updated my _INetSmtpMail and _INetSmtpMail_Example files. they require _Base64.

_INetSmtpMail: Syntax Changed to look more like the current UDF one in INet.au3
_INetSmtpMail_Example: Changed to reflect the new syntax. I also added an option to send the email directly to the MX server, but it doesn't seem to work too well. make sure to read the msgbox's carefully when running the example code.
Grab new code from my second post

Can someone point me in the right direction so i can get sending mail directly, via the mx server working? or are the spam bots getting my email?

Also can someone tell me how i can send attachments via email using this awesome tool blindwig and me built (base64). What are the smtp commands to do that? A very quick google didn't give me anything.

Many thanks!

[blindwig]

Another big discovery!
I've been going through the functions looking for anything that can be done differently, and seeing if it would be faster to do it differently.
I just found out that Asc() is about 3 times slower than Number(BinaryString()).
Here's the proof:

$I_NumTrials = 1000

$a = 'A'

$i_Start = TimerInit()
For $i = 0 to $I_NumTrials
    $b = asc($a)
Next
$i_Time1 = TimerDiff($i_Start)

$i_Start = TimerInit()
For $i = 0 to $I_NumTrials
    $c = number(BinaryString($a))
Next
$i_Time2 = TimerDiff($i_Start)

MsgBox(0,@ScriptName,'Trial 1: ' & Round($i_Time1/1000,3) & @CRLF & 'Trial 2: ' & Round($i_Time2/1000,3) & @CRLF & 'Success: ' & ($b = $c))

On my machine:
Trial 1 = 0.1 seconds
Trial 2 = 0.035 seconds

So I'll replace all the Asc() function calls with Number(BinaryString()) and see how much more speed we can get out of this thing...

Also I'd like to try to figure out a way to re-write those 14-bit tables so that I don't have to do all that shifting to get the bits right. I'm thinking about a 2d table - instead of a 1d table of 12-bits, I would have a 2d table of 8 and 4 bits.

[mikeytown2]

Sounds good! are you going to do both or do you want me to take care of the Asc() to Number(BinaryString()) conversion?

EDIT
looks like your doing the conversion, cool

I did your test but with 100,000 trials this is what i got

Trial 1: 1.559
Trial 2: 1.966
Success: True

With 1000 Trials

Trial 1: 0.014
Trial 2: 0.019
Success: True

AMD vs Intell... crazy

Is there anyway to figure out what processor someone is using?

ps add in a sleep(3000) before you start each one

Edited by mikeytown2, 10 March 2006 - 06:35 PM.

[blindwig]

OK I posted about my strange findings with BinaryString() and Number() in support but it looks like the thread got lost in the sea of noobs, so I just posted a new thread to Bug Reports.

OK I put in the sleep commands like you said and my new results are these:
Trial 1: 0.065
Trial 2: 0.087
WTF? I really don't understand all the descrepencies....
But even if it is faster, the statement Number(BinaryString()) chokes on values of 46 or 128-255, so it's no good to me after all.
Poop.

Edited by blindwig, 11 March 2006 - 12:37 AM.

[blindwig]

Also I'd like to try to figure out a way to re-write those 14-bit tables so that I don't have to do all that shifting to get the bits right. I'm thinking about a 2d table - instead of a 1d table of 12-bits, I would have a 2d table of 8 and 4 bits.

WOOT!
I DID IT!
Decoder speed now up by 40%!!!
This means that the DECODER is now FASTER than the ENCODER by about 5%.

I was going to build a 2d lookup table with 2 columns to pre-split the 12-bits into 4 and 8 bit groups. But then I realized that one the first lookup I need the bits split 8+4, and on the second I need them 4+8. So then I thought about making 3 columns with 4 bits in each. But that wouldn't gain any speed, because all the bit shifts need to put the bits back together would kill the speed gained in the lookup. So then I thought of making 4 columns - 4,8,8,4 bits (all from the original 12 bits), but then it hit me - I only need 2 columns, 1 for 8+4 and 1 for 4+8, and I get all the bits shifted before putting them in the table so that the decoder doesn't have to do any bit shifting at all. So the look-up tables take a little longer to build, but the final decoder is much faster!
Here's the total function count for each 32-bits decoded:
Input:
4 x Asc() - to get the 32 bits
Look-Up:
2 x BitShift()
3 x Add
Output:
1 x StringLeft() - trim the 4th byte
1 x BinaryString() - convert 24-bits into 3 8-bit characters

Attached Files

•  20060312_MWR_Base64.zip   3.83K   212 downloads

[mikeytown2]

WOOT!
I DID IT!
Decoder speed now up by 40%!!!
This means that the DECODER is now FASTER than the ENCODER by about 5%.

So the look-up tables take a little longer to build, but the final decoder is much faster!

This is awesome!

Do you agree with me when i put the build tables in the encode/decode functions; or do you think i should keep the building of the tables outside of the encode/decode functions? Let me know.

Also i can't really explain why your timing was off for the asc() function, but i first noticed it when i tested 100,000. I think its because autoit is still loading stuff when it first starts to execute code.

EDIT
Opps i'm going to look into this but i get this error
_Base64.au3 (247) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$i_Temp = BitShift($gcai_Base64Table14[BitShift(Asc($as_CypherText[$i_Count + 0]),-7) + Asc($as_CypherText[$i_Count + 1])], -12) + $gcai_Base64Table14[BitShift(Asc($as_CypherText[$i_Count + 2]),-7) + Asc($as_CypherText[$i_Count + 3])]
$i_Temp = BitShift(^ ERROR

I get the error when the input is larger then 3 char's in lenght. I dont get it when i use files. I'll look into it more.

EDIT
Found the problem! You forgot to copy the new code to the section of code that didnt have the progress bar for the decoder. I'll post an update to the _base64 file once i get your opinion to the question above.

EDIT
I got sending emails directly to the MX server working now. that means no smtp servers needed to send email! I'll post the new code the same time i update _base54

Edited by mikeytown2, 13 March 2006 - 11:49 AM.

[blindwig]

Do you agree with me when i put the build tables in the encode/decode functions; or do you think i should keep the building of the tables outside of the encode/decode functions? Let me know.

No, I don't agree to that. None of the other UDFs do this except for VISA, which has to because it doesn't know ahead of time if you have VISA or not, so it has to be detected. Our look-up tables never change, so there's no reason not to know them ahead of time, so that doesn't really apply. Also, most programs I've worked with (example Photoshop, Adobe Acrobat, etc) load all of their plug-ins at the beginning during the splash screen. I think it's annoying to have a program building it's load-up stuff while you're working in it. You could argue "But the user shouldn't have to load the look-ups if they don't want to use them" and I would say "If they didn't want to use them, they wouldn't include your UDF in their project."
Those are my feeling, but it's your UDF (well, most if not all of the code is mine, but I'll give that to you ) and so it's your decision. Maybe you should ask some of the more senior members here, and/or put it to a vote. Just a suggestion...

Also i can't really explain why your timing was off for the asc() function, but i first noticed it when i tested 100,000. I think its because autoit is still loading stuff when it first starts to execute code.

Yeah, that's what I was thinking


I've got some more big speed increases.. I'm cleaning them up now and will post soon...

[mikeytown2]

No, I don't agree to that. None of the other UDFs do this except for VISA, which has to because it doesn't know ahead of time if you have VISA or not, so it has to be detected. Our look-up tables never change, so there's no reason not to know them ahead of time, so that doesn't really apply.

I've got some more big speed increases.. I'm cleaning them up now and will post soon...

Cool! I see your point for building base64 tables before we use them. The only reason why i would want to do it at build time is because none of the other UDF's in the beta folder have any operations globally. so thats the reason why i would do it when you need it. I totally see your point, but I'm just trying to stick to the standards. I totally agree with you, its just i never see it done like we did it in any of the UDF's included. Thats why I'm always changing it.

Maybe you should ask some of the more senior members here

Good idea. I'll wait for your update then I'll ask the question.

[blindwig]

I was going to build a 2d lookup table with 2 columns to pre-split the 12-bits into 4 and 8 bit groups. But then I realized that one the first lookup I need the bits split 8+4, and on the second I need them 4+8. So then I thought about making 3 columns with 4 bits in each. But that wouldn't gain any speed, because all the bit shifts need to put the bits back together would kill the speed gained in the lookup. So then I thought of making 4 columns - 4,8,8,4 bits (all from the original 12 bits), but then it hit me - I only need 2 columns, 1 for 8+4 and 1 for 4+8, and I get all the bits shifted before putting them in the table so that the decoder doesn't have to do any bit shifting at all. So the look-up tables take a little longer to build, but the final decoder is much faster!

Let me see if I can explain this a little better:
So we decode 32 bits and get 24bits. These 24 bits can form 3 8-bit characters, let's say "ABC". BinaryString() wants them as "CBA" so I have to shift the bits around. But doing it every 32-bits takes a lot of time, so I wanted to have the bits shifted ahead of time, in the table. So I break them up into 4-bit chunks, and shift them before they go into the table.
So I have this:
ABCDEF (each character represents 4 bits)
and BinaryString() wants this:
EFCDAB
So I make 2 values that each hold 12 bits, that add up to 24 bits:
ABC000
000DEF
======
ABCDEF
But I want the bits to be shifted so that they fit into the BinaryString(), so I do this:
00C0AB
EF0D00
======
EFCDAB

Viola!

[blindwig]

OK, 1 more big speed increase, mostly on the encoder:

I got rid of the _Base64Conv() function, and just dumped it directly into the _Base64Encode() function. So now the encoder looks even more ugly, but it's about 20% faster.
Also, I did some testing and was very suprised to find that negative bit-shifting (shifting to the left) is actually slower than plain multiplication. This comes as a suprise to me, because in most languages bit shifting is 10-20 times faster than integer math. The only explaination I can think of is that when AutoIt detects integer multiplication, it internally optimizes it by automatically switching over to bitshifting.

I went through the code and changed every negative bitshift operation into a multiplication.
I couldn't test for a positive bitshift, because AutoIt doesn't have an integer division operator (ie the "other half" of the mod() function). All I could do was int($n/$d), which tested to the exact same speed as a positive bitshift, and sometimes slightly slower. So I left all the positive bitshift operations in place.

The encoder increased another 10-15% in speed, for a total of about 30-35%.
The decoder was less affected, since I previously eliminated as many BitShift() calls as I could, so it only gained about 2-5% increase.

Attached Files

•  20060313_MWR_Base64.zip   3.84K   216 downloads

[mikeytown2]

OK, 1 more big speed increase, mostly on the encoder:

I got rid of the _Base64Conv() function, and just dumped it directly into the _Base64Encode() function. So now the encoder looks even more ugly, but it's about 20% faster.
Also, I did some testing and was very suprised to find that negative bit-shifting (shifting to the left) is actually slower than plain multiplication. This comes as a suprise to me, because in most languages bit shifting is 10-20 times faster than integer math. The only explaination I can think of is that when AutoIt detects integer multiplication, it internally optimizes it by automatically switching over to bitshifting.

I went through the code and changed every negative bitshift operation into a multiplication.
I couldn't test for a positive bitshift, because AutoIt doesn't have an integer division operator (ie the "other half" of the mod() function). All I could do was int($n/$d), which tested to the exact same speed as a positive bitshift, and sometimes slightly slower. So I left all the positive bitshift operations in place.

The encoder increased another 10-15% in speed, for a total of about 30-35%.
The decoder was less affected, since I previously eliminated as many BitShift() calls as I could, so it only gained about 2-5% increase.

the decoder is failing and i cant figure out why. use my example file with the input box test. You will see that its erasing the <- at the end of the output, or its throwing an array error. I couldn't find the problem, i hope u can.

Nice work on the encoder btw

[blindwig]

the decoder is failing and i cant figure out why. use my example file with the input box test. You will see that its erasing the <- at the end of the output, or its throwing an array error. I couldn't find the problem, i hope u can.

Nice work on the encoder btw

OK, looks like a typo in the last part of the decoder, where it deals with the last <4 bytes. Try this instead:

Select
            Case $ai_Bytes[0] = -1;File ended on a perfect octet
            Case $ai_Bytes[1] = -1;This should never happen
                SetError(-1)
            Case $ai_Bytes[2] = -1;Only the first 2 bytes to be considered
                $s_Out &= Chr($ai_Bytes[0] * 4 + BitShift($ai_Bytes[1], 4))
            Case $ai_Bytes[3] = -1;Only the first 3 bytes to be considered
                $s_Out &= Chr($ai_Bytes[0] * 4 + BitShift($ai_Bytes[1], 4)) _
                        & Chr(BitAND($ai_Bytes[1] * 16 + BitShift($ai_Bytes[2], 2), 255))
            Case Else;All 4 bytes to be considered
                $s_Out &= Chr($ai_Bytes[0] * 4 + BitShift($ai_Bytes[1], 4)) _
                        & Chr(BitAND($ai_Bytes[1] * 16 + BitShift($ai_Bytes[2], 2), 255)) _
                        & Chr(BitAND($ai_Bytes[2] * 64 + $ai_Bytes[3], 255))
        EndSelect

[mikeytown2]

Several Updates the the files in this thread...
_base64: Encoder and Decoder are faster thanks to blindwig. I kept this UDF file pretty much the same as what he put up here, so it builds the tables before you use any of the functions in _base64. It also has 4 Globals.

We are asking for some advice on how we should build the tables. Before you call any base64 functions or right in the middle?

_INetSmtpMail: Changed Syntax, Added support to send emails directly from your computer with no smtp server needed.
_INetSmtpMail_Example: Updated to deal with the new functionality of the UDF.

Grab from my second post

Please test out sending emails without a smtp server. so far it works with my usc email, gmail and hotmail.

EDIT
Aol doesn't like email coming from a dynamic IP address, so you need a smtp server to send email to an aol email box

Edited by mikeytown2, 15 March 2006 - 05:08 AM.

[nfwu]

Any tips on how to speed up code?

OK... let's say that my code is a whole pile of functions calling each other (More specifically, my 3d graphics library)

Version 0.4 (not released) takes 2 minutes to draw 1 frame!!!

Any tips?
???
#)

[blindwig]

Any tips on how to speed up code?

OK... let's say that my code is a whole pile of functions calling each other (More specifically, my 3d graphics library)

Version 0.4 (not released) takes 2 minutes to draw 1 frame!!!

Any tips?
???
#)

Some things I've learned in optimizing this base64 stuff:
Integer multiplication is actually faster than negative bitshifting (as in shifting to the left). This was a huge shock to me as I've always used left-shifting as a way to optimize multiplication. If you're doing a lot of math, then I'm assuming you did the same.
Calling functions (even if you call all the parameters ByRef) is very slow in AutoIt. With a quick look at your code, I see that you have functions that call functions that call functions. This is easy to maintain from a code stand-point, but it's very slow for execution. Make more specific functions instead of trying to have catch-all functions.
Eliminate conditionals whenever possible. This may mean having 2 copies of a function that are almost identical except for 1 little thing. Again, this is a pain to maintain, but much faster to execute.
Pre-calculate whatever you can. Looking up in an array is so much faster than having to calculate on the fly.
When there is more than 1 way to do something, write routines to do it several different ways and test them to see which way is the fastest.

Take a look at the different revisions of the code in this thread, and you can see the evolution of effeciency that we have discovered.

[drefty]

which version of autoit does this work with?

what is necessary to make this code compatible with the latest non-beta
release of autoit.