UnitTesting

This page is still a work in progress.

Article in progress

NOTE: Article in progress so constructive input about content is welcome. If you have language comments, want to flame me for spelling or have fine-picking needs please wait until this section is removed. .....

Initial notes

This is just a initial page. I have been allowed by Lexholm AS (www.lexholm.no) to improve and release a UnitTesting tool we have made. In the process I have also been allowed to do some documentation to ease adoption of unit testing in the AutoIt community. This page will be about UnitTesting in general and how you can implement your own framework. If you want to use the tool by Lexholm when it is released (it is not documented at it's current state) that is fine. But our main goal with this article is to get you intrested in and caring so much for your code that you start using an automatic approach to quality assurance of your own code. And thereby AutoIt in general.

AutoIt has moved from a limited special purpose scripting language to a real contender as an application language for small to medium sized projects. The developers of AutoIt have proven willingness to aggressively improve and extend the language and a core function library. Betas are released frequently and with an astonishing quality. It will surprise me and my colleges at Lexholm a lot if there is no automatic quality assurance philosophy and implementation used by the developers.

With the methods we describe here you can improve the quality of your own code and help the developer team assure that each beta release is as good as or better than the previous.

What is Unit Testing

Unit Testing is about caring for the code you write. It is about breaking the code down to testable units. When code is easy to test it is also easy to maintain and improve. Unit Testing is about quality assurance and confidence in the code you write. It is about investing a littel time upfront to gain on your investment when the project grows or has been put on ice for a while. Unoit Testing is basically putting together the simplest "best practise" rules you find about coding and make those rules second nature to you while you code.

What Unit Testing is not

It's not the silver bullet the "inventors" like it to be. Unit testing GUI applications can be a real challenge and sometimes impossible. But as your project evolves and you find hard to automate

Why should I adopt Unit Testing

Everytime you make a test you assure that some piece of your code works as intended. By following some simple rules you can make a simple automatic testing engine to make sure your code will work as you intend it to. If it does not you have to do improvements.

With current and later release of AutoIt. If it does not. You either have to decide to keep a version of AutoIt where all your tests do pass. Or you will have an incredible easy time locating the areas of your code where you have to do updates.

The simplest framework posible

Lets create a unittest.au3 file and save it in a include directory. A good thing to know is that you can have your own include directory

The UTAssert function

Now open your unittest.au3 file and add the simplest possible assert function.

Func UTAssert($bool, $msg="Assert Failure", $erl=@ScriptLineNumber)
     If NOT $bool Then 
         ConsoleWrite("(" & $erl & ") := " & $msg & @LF)
     EndIf
     Return $bool
EndFunc

A sample project using UTAssert

To make good use of our unit testing framework we should make sure our code is nicely divided into well defined function. If you are a hard core unit tester you should even know that the philosophy is to write the test code first and then fill in the code to make the tests work.

So, let's see. We need a nice little project showing us what to do. Let's start with an really easy text book sample. We invent the function Add. Add can take two arguments. But if the first one is an array the second one is not needed. If two values are provided they are added and the sum is returned. If an array is provided all elements in the array is added together. So a test setup would look something like this.

The first tests

UTAssert(Add(1) = 1)
UTAssert(Add(1,1) = 2)
UTAssert(Add(-1,1) = 0)
UTAssert(Add(-1,-1) = -2)

That was the easy part. We should add as many tests as we need to make sure the code we write will be working under all expected curcumstances. Now we have to cover the array requirement for the code

Local $arg1[9] = [1, 2, 3, 4, 5, 6, 7, 8, 9]
UTAssert(Add($arg1) = 45)

At this point none of the tests will pass because we have not written the function yet.

==== The function we are testing ====
Func Add($arg1, $arg2=0)
    Local $ret
    If IsArray($arg1) OR IsArray($arg2) Then 
    Else
        $ret = $arg1 + $arg2
    EndIf
    Return $ret
EndFunc

Running our first tests

Ok, So at this point we fire up the script and looks what it gives us

>Running:(3.2.2.0):E:\scite\..\autoit-v3.2.2.0\autoit3.exe "E:\CodeX\autoit\au3\au3UnitTest.au3"    
(23) := Assert Failure
+>AutoIT3.exe ended.rc:0

Obviously not so strange that the array test failed as we have not added any array code yet. So, let's continue with our quest.

Adding some more code to fix failed tests

Func Add($arg1, $arg2=0)
    Local $ret, $i
    If IsArray($arg1) OR IsArray($arg2) Then 
	  If IsArray($arg1) Then 
		 For $i = 0 to UBound($arg1) - 1
			$ret += Number($arg1[$i])
		 Next
	  Else
		 $ret += Number($arg1)
	  EndIf
    Else
        $ret = $arg1 + $arg2
    EndIf
    Return $ret
EndFunc

Another run with the tests we have created and.....Hey, it turnd out to be good. No red lines to jump to. That is really nice.

Reconsider our scenarios

Now it's time to ask. Did I write a test case for every possible use case I can think of? No, actually I did not. I'm missing several scenarios.

Adding more tests

<

>
Local $arg1[9] = [1, 2, 3, 4, 5, 6, 7, 8, 9]
UTAssert(Add($arg1, 10) = 55)
UTAssert(Add($arg1, $arg1) = 90) ;NOTE*

• Now this was not part of the original specification. Should I use it like this or should I return an error? If I return an error how will UTAssert react to it?

Running with the added tests

Anyhow, running this will show us that our function needs more work.

>Running:(3.2.2.0):E:\scite\..\autoit-v3.2.2.0\autoit3.exe "E:\CodeX\autoit\au3\au3UnitTest.au3"    
(31) := Assert Failure
(32) := Assert Failure
+>AutoIT3.exe ended.rc:0

==== Adding a bit of code again ====

So, we take a look at our code. After considering our code we find that both failures are easy to fix. All we have to do is to duplicate the code

If IsArray($arg1) Then 
   For $i = 0 to UBound($arg1) - 1
	  $ret += Number($arg1[$i])
   Next
Else
   $ret += Number($arg1)
EndIf

And change the variable name $arg1 to $arg2.

Identifying duplicate code blocks. ArgSum

But hey, any part of the code that is a duplicate like that should be refactored to it's own function. And that function should be tested separately with unit tests. So

Func ArgSum($arg)
   Local $ret, $i
   If IsArray($arg) Then 
	  For $i = 0 to UBound($arg) - 1
		 $ret += Number($arg[$i])
	  Next
   Else
	  $ret += Number($arg)
   EndIf
   Return $ret
EndFunc

So Putting this all together

We have the following code:

;#include-once<unittest.au3>
Local $arg1[9] = [1, 2, 3, 4, 5, 6, 7, 8, 9]
UTAssert(Add(39, 40) = 79)
UTAssert(Add($arg1) = 45)
UTAssert(Add($arg1, 10) = 55)
UTAssert(Add($arg1, $arg1) = 90) ;NOTE* Look above in the WIkI for the note about this test
;
;
;
;==== The function we are testing ====
Func Add($arg1, $arg2 = 0)
	Local $ret
	$ret = ArgSum($arg1) + ArgSum($arg2)
	Return $ret
EndFunc   ;==>Add
;
Func ArgSum($arg1)
	Local $ret, $x
	If IsArray($arg1) Then
		For $x = 0 To UBound($arg1) - 1
			$ret += Number($arg1[$x])
		Next
	Else
		$ret = Number($arg1)
	EndIf
	Return $ret
EndFunc   ;==>ArgSum
;
Func UTAssert($bool, $msg = "Assert Failure", $erl = @ScriptLineNumber)
	If Not $bool Then
		ConsoleWrite("(" & $erl & ") := " & $msg & @LF)
	EndIf
	Return $bool
EndFunc   ;==>UTAssert

How about some automation

AutoIt is all about automation. So a Unit Test tool for AutoIt should be able to help us out. The big question is actually. Can it be simpler than it already is? Probably not much while you write the code. But when you want to try your code against a new release of AutoIt then there are lots of things we can automate.

Automatically running the function we are working in

Now, that is a possibility. Say, all you have to do to run the function you are working in is to hit some key combination. To achieve this we would need:

• A method of identifying the function we are working in.
• A small program to write some code including the module we are working in and call the appropriate function.

As it is the SciTE4Autoit distribution has a way of identifying the function your caret is located in. You can observe this in the status bare on the left side. This information is also available through a variable.

A simple runner application

At this point we need a simple runner application. It has to:

• Accept function to run and script location.
• Wrap up the code to run the test function associated with the function.

;#include <unittest.au3>
;
Func UTAssert($bool, $msg="Assert Failure", $erl=@ScriptLineNumber, $error=@error, $extended=@extended)
     If NOT $bool Then 
         ConsoleWrite("(" & $erl & ") := " & $msg & @LF)
     EndIf
	 If $error <> 0 Then SetError($error, $extended, $error)
     Return $bool
EndFunc
;
Func ParseCmdLine(ByRef $arr, ByRef $CmdLine)
#cs;$arr will be formated according to this
   ;$arr[0] = Array items Containing data. On top of that is maintenance items
   ;$arr[1 ... n] = File names
   ;$arr[n .... $arr[0]] = Function names
   ;$arr[$arr[0] + 1 = File names count
   ;This is given by [0] and previous $arr[$arr[0] + 2 = Function names count
#ce   
   Local $paths = 1
   ReDim $arr[$CmdLine[0] + 2]
   For $i = 0 to $CmdLine[0]
	  $arr[$i] = $CmdLine[$i]
   Next
   $arr[$arr[0] + 1] = $paths
EndFunc
;
Func err($msg, $nr, $terminate=0, $erl=@ScriptLineNumber)
   dbg($msg, $nr, 0, $erl)
   IF $terminate Then Exit
EndFunc
;
Func dbg($msg, $error = @error, $extended = @extended, $erl = @ScriptLineNumber)
	ConsoleWrite("(" & $erl & ") : = (" & $error & ")(" & $extended & ") " & $msg & @LF)
	If $error <> 0 Then SetError($error, $extended, $error)
	Return $error
EndFunc   ;==>dbg
;
Func testParseCmdLine()
   Local $cmds[4] = [3, "testFunc1", "c:\test", "testFunc2"]
   Local $arr[4]
   ParseCmdLine($arr, $cmds)
   UTAssert(IsArray($arr))
   UTAssert($arr[0] = 3)
   UTAssert($arr[$arr[0] +1] = 1)
   UTAssert($arr[0] - $arr[$arr[0] +1] = 2)
   UTAssert($arr[1] = "c:\test")
   ;TODO: Should we antisipate sequence of functions to call?
   UTAssert($arr[1] = "testFunc1")
   UTAssert($arr[1] = "testFunc2")
EndFunc
; =====================================================================
; Selftesting part
; =====================================================================
If StringInStr(@ScriptName, "au3UTRunner.au3") Then 
   testParseCmdLine()
EndIf

Creating a UnitTestRunner

Now here is an interesting topic. What is a test runner? The simplest approach again is just to use SciTE and watch for lines starting with parenthesis wrapped around a number. If done like this wee need a script calling all of our test functions. Actually not a bad solutions. It does not give you anny statistics. And we just love to know how many tests we pass each time.

But, as we work on our UDF's we can do something like this.

#include-once
#inlcude <myFuncLib.au3>
#include <UnitTest.au3>
If StringInStr(@ScriptName, "filename") Then 
    testMyFunc1()
EndFunc
Func testMyFunc1()
    UTAssert(1 = 1, "Now this is a solution")
EndFunc

In Scite all we have to do now is hit F5 and look for those pesky error lines. Maybe not the cleanst possible but it is simple, and it works.

Easing the code test code cycle

The UTRunner is now ready to be integrated with SciTe4AutoIt.