PowerShell for Testers

Testing Applications with Powershell Download - PSExpect Edition

Mon, 02 Jun 2008 21:40:10 GMT

Like many of you, I noticed the download that was made available on the MSDN Tester Centre the other day - Testing Applications with Powershell - Sample Code. This link, along with downloads and article such as the UI Automation Specification and associated Community Promise gives me some hope that it might be possible to run the same test script both with and without the UI.

That's my hope for testing tools. Oh yeah, and that we can do that using business-oriented grammar in the scripts. And that we can define, scope, and reuse elements of that grammar in exploratory testing, semi-automated style.

Anyway, baby steps ... baby steps.

The post was great in that it raised the awareness of how Powershell might be used for testing. All of my examples in PSExpect are against libraries or other scripts. This example automates the UI. And that's cool if not entirely necessary in all contexts.

I replaced the verdicts within the posted sample code with calls to PSExpect Assert* functions. There were two of them, depicted below with the original code hashed (commented) out:

$result = get-listBox $lb "222"
#    if ($result -ge 0) {
#      write-host "Found '222' in ListBox!"
#    }
#    else {
#      write-host "Did NOT find '222' in ListBox"
#      $pass = $false
#    }
AssertEqual -Expected 0 -Actual $result -Label "TC-1" -Intent $Intention.ShouldPass
...

$text = get-textBox $tb
#    if ($text -eq "ad") {
#      write-host "Found 'ad' in TextBox!"
#    }
#    else {
#      write-host "Did NOT find 'ad' in TextBox"
#      $pass = $false
#    }
AssertEqual -Expected "ad" -Actual $text -Label "TC-2" -Intent $Intention.ShouldPass
...

and then there is the way that the test script reports back to the tester about the status of the test run:

#    if ($pass) {
#      write-host "`nTest scenario result = Pass" -foregroundcolor green
#    }
#    else {
#      write-host "`nTest scenario result = * FAIL *" -foregroundcolor red
#    }
RaiseAssertions

Running the revised test script and running it against the sample application to be tested generates the following two lines of output:

02/06/2008 12:39:26 PM,SHOULDPASS,PASSED,TC-1
02/06/2008 12:39:28 PM,SHOULDPASS,PASSED,TC-2

Since the tests pass, they show up as green. You can find the entire script at(http://www3.telus.net/~amgeras/samples/Test-TheApp.ps1). You will find that the script requires the PSExpect library in a specific location - just change that location to get the script to run (one line to edit).

We did drop a number of lines of script completing this refactoring. We also gained functionality since when you run the script, you are also generating a log file in behind the scenes that could be posted in the test results repository. That's all good. But there are other refactorings that we might consider that I will tackle when I get a moment.

First, from a testing perspective, the functions get-listBox is counter-intuitive since it returns an integer when given an item that it is supposed to contain. I understand why, but the consequences to the test script are obvious when you compare the two AssertEqual statements above.

In the first AssertEqual, the expected result is '0' yet we are really looking for the value '222'. In the second AssertEqual, we are looking for 'ad' and that is exactly what we truly are looking for. That's more intuitive. So in the next refactoring I would remove the comparator in the function and leave that comparison for the tester.

Second, and this only reflects my bias towards domain-driven design thinking, is that the sample application mimics some function that doesn't come through in the test script unless you look really close. It gets the point across for people like me to talk around, so it was fit for purpose.

But ...

If the screen functionality could be represented using a verb-noun naming convention, then that would provide the means for separating the test runner from the test actions from the set of verdicts and that would make it even more meaningful, and the test script more readable.

For example,

start-productSearch -Category $null -Item $null
test-productSearchResults -Expected 'Error'

start-productSearch -Category 'Product Name' -Item 'ad'
test-productSearchParam -Expected 'ad'
test-productSearchResults -Expected '222'

Now, the test runner is still the Powershell script (as it was before we did any refactoring). The screen action is start-productSearch. And the verdicts are test-productSearchParam and test-productSearchResults, both implemented in the two AssertEqual statements above but whose meanings were hidden in the interpretation of the script.

And this test script is now much more reflective of what the screen does and how we're testing it. It communicates better to the testers, and we have this function start-productSearch that we could reuse in other , longer-running scenarios of the larger application.

Right now, these last two refactorings are musings, but I'll post them and my findings along the way as soon as I get a chance.

Hyper-Testability with Powershell

Mon, 31 Mar 2008 02:04:50 GMT

Testability is a software quality attribute that is a measure of how easy (or difficult) it is to create test cases for a test target and, in combination, how easy (or difficult) it is to run those tests. High testability is a good thing since it means that the total cost of making changes to the software is correspondingly lower. It typically also means that you can have more than one individual contributing to the software since it is easy for them to determine whether or not their contribution has been a net positive or net negative contribution, simply by running the tests again. Hence, we like test automation.

It turns out that when we value testability, we tend to write software that enables us to automate more types of tests, and we set up this fantastic feedback loop where defining a test becomes a means of production - the tests specify the desired behaviour, guide the development, and then verify the as-built software for as long as that behaviour remains desired. Pretty nice, when we get to that level.

Tools like xUnit were revolutionary in the sense that they gave developers the means for valuing testability and a mechanism for expressing the low-level behaviour in the same language that our target software was (or would be) written in. Developers liked it, and now teams in general are much more inclined to automate unit tests than ever before.

But is it enough? That's a question that Brian Marick has been pondering publicly in his blog (it's one of those blogs that actually expands your mind when you read it) and in an on-line forum discussing automated functional testing. I was struck by the phrase "Perhaps it would be better to invest heavily in unprecedented amounts of built-in support for manual exploratory testing." the most. An alternative, he indicates, to automated functional testing with tools like FIT/FITNesse (read the blog entry to find out why this isn't agile heresy).

You can see where I'm going. If you have a set of Powershell cmdlets that exposes an application's functionality - and if the cmdlets expose ALL of the functionality - then 'exploring' means working with those cmdlets (or an API) in a Powershell session and using whatever testing heuristics you might have in your tester's toolkit to test the application. You may not be using the same UI as the end-users, however, there is great value to first confirming the functionality is good enough, and then second moving on to confirm that the UI is good enough.

The Powershell features that make exploratory testing an excellent experience are (but not limited to):

Objects on the pipeline, and in fact, having objects to work with at all.
The get-help cmdlet that can access custom help files for custom cmdlets.
The get-member (gm) cmdlet.
The get-history (h) cmdlet.
The naming convention for cmdlets.

You don't even really have to know the cmdlets ahead of time because of those features since you can retrieve their calling syntax at run-time. The get-history cmdlet can then be used to save useful cmdlets and sequences of calls, in order to support future semi-automated exploratory testing (where some scripts are run, but only for the sake of returning to a known state) or ultimately fully automated testing for something like detecting regression. Seems that the ability of a script for passing on knowledge isn't limited to administrators ...

I can envision, however, a mismatch between the cmdlets that an application designer foresees might be a useful way of exposing the functionality and the cmdlets that a tester might find useful. They will each bring, I believe, a different perspective.

So it might be wise to undertake this 'unprecedented support for exploratory testing' or hyper-testability using a test-driven approach where the functionality of those cmdlets is influenced, or possibly specified, by the examples that a tester might deem appropriate. Supporting manual exploratory testing by writing automated test scripts ahead of time ... seems contradictory. But there is a distinction between 'understanding-clarifying tests' (Brian Marick again) and 'verification and validation' tests. So using a test-driven approach for understanding and clarifying requirements, and ultimately the design of the cmdlets, isn't the same as writing tests for verifying and validating the resulting application.

The bottom line? Exposing application functionality in Powershell makes your application hyper-testable. Use test scripts that you/your testers write ahead of time as a means of designing the cmdlets and separate the concern of testing the functionality of the application from the concern of testing the user interface.

Envisioning Next-Generation Functional Testing Tools (Continued)

Tue, 19 Feb 2008 02:35:01 GMT

I know we're not supposed to come up with solutions before fully understanding the problem, but I've got just enough experience and wide-eyed insensibility to believe that I've seen something that just might work: testing mashups.

Microsoft Popfly is a set of tools for creating web pages and the like from pre-defined blocks. The one aspect that I am particularly intrigued by is called a mashup. You use a tool called the Mashup Creator to build a web page comprised of blocks that come from disparate sources. For example, build a page that lists your Facebook friends' statuses and your Twitter friends' most recent updates. I'm assuming it's called a 'mashup' after the term that's been coined for mixing songs into single tracks (some of my favourites have sampled from 5 or more songs).

Now imagine if the "blocks" used to build a mashup were Powershell cmdlets instead of web applications and that those cmdlets were designed to expose the functionality of an application under test. Each "mashup" you create is therefore another application scenario. Imagine that some of the blocks you use on a mashup are data sources so that you could make the scenario data-driven.

So, why do I think this might be a good idea?

First, it may help non-programmers to develop automated test scenarios that are executable. Second, the economics of building cmdlets for the application just changed: not only might the cmdlets be useful for system administration, but now they will also be useful in writing automated tests for the application.

Why cmdlets? Well they exhibit all the behaviour that I would want from blocks for building tests from - they have a naming convention, there are behavioural standards that they adhere to, and there is a mechanism already in place for stringing them together and running them as one scenario, so they're not just commands that can be invoked. If the 'noun' part of the cmdlet always refers to something from the business domain, then we're working at the same level of abstraction as (hopefully) the requirements are written - so as testers we're grounded, too. There is a tie-in to the whole world of domain-specific language and domain-driven development that just can't be ignored.

Now these aren't the only tests that would need to be considered for any system under development. A test strategy for any software product is going to include extensive developer (unit and low-level integration) testing and manual functional and acceptance testing. I get that. At the same time, automating more test scenarios means they are shareable and immediately executable by anyone on the team, enhancing communications and reducing communication overhead. That's a good thing, and worth striving for.

Behaviour Driven Development with Powershell

Fri, 30 Nov 2007 04:21:04 GMT

or, A Testament to the Flexibility of Powershell

The greatest impact that the Functional Testing Tools Workshop in Portland (October, 2007) had on me was to raise my awareness of behaviour-driven development, in particular, the 'given-when-then' pattern for structuring examples.

I will defer to other blogs on describing given-when-then in detail, you'll see what it means soon enough in the examples below. For now, think of given-when-then as a means of translating the meaning of a user story into a related set of acceptance criteria using examples. Something that has always been a bit awkward for business users and developers alike.

Given you adopt the story-telling convention popularized by Mike Cohn (I think),

As a <blank>
I want to <blank>
So that <blank>

when you want to translate the story into a set of related acceptance criteria using examples,
then use the following convention popularized by BDD proponents such as Dan North:

Given <a known set of conditions or state>
When <something happens>
Then <check for an expected set of conditions or state>

There are tools that help you do this and to make statements worded like this executable - in that case, they almost become executable requirements. They are certainly executable examples. The most-often-used example is for a simple account manager class (I've seen this example on a number of blogs but didn't pay attention to where it originated, but I think it was Dan North's again). I'm not the only person looking at this - there is a NBehave project on CodePlex that provides a mechanism for writing tests in this manner. And there are many ruby examples in both rspec and rbehave.

# Story: Transfer to cash account from savings account
# As a savings account holder
# I want to transfer money from my savings account
# So that I can get cash easily from an ATM

# Scenario 1: Savings account is in credit
given "My savings account balance is 100 and my cash account balance is 10"
when "I transfer 20 to my cash account from savings"
then "my cash account balance should be 30 and my savings account balance should be 80"

given "My savings account balance is 400 and my cash account balance is 100"
when "I transfer 100 to my cash account"
then "My cash account balance should be 200 and my savings account balance should be 300"

My opinion is that this is a clear way of expressing an example, albeit all that I have really seen are simplistic ones. I'm going to enjoy working through more complex examples on an upcoming project.

Now on to Powershell. It was always my intention to introduce some sort of behaviour-driven development helper functions in PSExpect, but I was never certain of what that could be. Until I went to the Functional Testing Tools conference in Portland. I love it when I can take some time off to think, and be inspired by a group of brilliant people.

Turns out we can make those examples above executable by writing a function and adding a set of aliases to our Powershell environment. The function accepts a string parameter and a script block parameter. All that it does is write out the string to a log file and then run the script block. I've called it Invoke-Block for lack of imagination, and it uses the PSExpect logging mechanism so that the same log file shows the results of assertions.

function global:Invoke-Block([string]$Label, [ScriptBlock]$Script)
{

    # build the log entry and write it out
    $entry = BuildLogEntry $Intention.ShouldPass $ResultPrefix.Passed "PSpec" $Label
    $added = $Assertions.Add($entry)
    if ($LogFileName -ne $null) {
        $logEntry = BuildMessage $entry
        $logEntry | out-file $LogFileName -append -width $LogFileLineLength
    }

    # now run the script block
    if ($Script)
    {
        &$Script
    }
}

set-alias given invoke-block
set-alias when invoke-block
set-alias then invoke-block

The aliases enable using the given-when-then triad in a Powershell script without generating any errors. This means that the executable version of the example is the same as above, with the script blocks added in. The example relies on a class library from the system under test (SUT) that the script loads using reflection (not shown) but the calling syntax to that library is hidden using a set of Powershell functions that utilize a verb-noun naming convention: create-account, transfer-money, set-accountbalance, and test-accountbalance (these are elements in the grammar of the domain-specific language, or ubiquitous language for this particular system under development).

# Story: Transfer to cash account from savings account
# As a savings account holder
# I want to transfer money from my savings account
# So that I can get cash easily from an ATM

# Scenario 1: Savings account is in credit
given "My savings account balance is 100 and my cash account balance is 10" {
set-variable -name SvAccount -value (create-account "Savings" 100) -scope script
set-variable -name csAccount -value (create-account "Cash" 10) -scope script
}

when "I transfer 20 to my cash account from savings" {
transfer-money -From $SvAccount -To $CsAccount -Amount 20
}

then "my cash account balance should be 30 and my savings account balance should be 80" {
test-accountbalance -Account $SvAccount -Expected 80 -Label "Savings.Balance.After"
test-accountbalance -Account $CsAccount -Expected 30 -Label "Cash.Balance.After"
}

given "My savings account balance is 400 and my cash account balance is 100" {
set-accountbalance -Account $SvAccount -Amount 400
set-accountbalance -Account $CsAccount -Amount 100
}

when "I transfer 100 to my cash account" {
transfer-money -From $SvAccount -To $CsAccount -Amount 100
}

then "My cash account balance should be 200 and my savings account balance should be 300" {
test-accountbalance -Account $SvAccount -Expected 300 -Label "Savings.Balance.After"
test-accountbalance -Account $CsAccount -Expected 200 -Label "Cash.Balance.After"
}

RaiseAssertions

The Output

29/11/2007 1:56:13 PM,SHOULDPASS,PASSED,PSpec,My savings account balance is 100 and my cash account balance is 10
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,I transfer 20 to my cash account from savings
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,my cash account balance should be 30 and my savings account balance should be 80
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Savings.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Cash.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,My savings account balance is 400 and my cash account balance is 100
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,I transfer 100 to my cash account
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,My cash account balance should be 200 and my savings account balance should be 300
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Savings.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Cash.Balance.After

The output above is from the log file that PSExpect generates on every run. That whole SHOULDPASS concept is there to support a programming style where you first create some tests that you expect to fail - you mark them SHOULDFAIL and then PSExpect prints those assertion failures in YELLOW instead of RED like it would if an assertion labelled with SHOULDPASS fails. I've come to rely on log files to use as evidence of passing tests, and to support debugging (since I tend to avoid using a step-through debugging tool on my own code; it's an anti-pattern for my personal coding style since using a tool like that means I'm confused and I don't have enough tests to show me the way).

A Way of Explanation

Yes, this really executes and the output demonstrates that each of the checks pass - this example is available in the most recent upload of PSExpect on the CodePlex site in the 'samples' directory. A couple of things worth pointing out ...

First, the starting curly brace of the script block must be on the same line as the call to invoke-block, otherwise Powershell parses the function call without the script block and errors abound. A minor inconvenience if that isn't already your scripting convention.

Second, the unfortunate use of set-variable in the first 'given' script block. That's really just in order to set the scope of variables (hence the use of the -scope parameter). Alternatively, you could declare the variable and instantiate it before any of the 'given' clauses. I don't think it matters, but this was the way that it was done in the NBehave example on the CodePlex site so I've copied that approach in my example.

Third, those wrapper functions. I've mentioned in other blog entries that Powershell uses the verb-noun naming convention for all the built-in cmdlets and that you are forced into this same convention when writing your own custom cmdlets. Well you should follow that form when writing Powershell functions too and write your functions like they are prototypes of cmdlets. Much better all-around style, and you don't lose any ground making changes to scripts if you choose to promote a function to a cmdlet. I prefer to use the wrapper functions in test scripts since they enhance the readability of the script for non-developers. I've rarely specified the wrapper functions ahead of time. Instead, I write the test scripts first and determine what wrapper functions I need based on the tests that I want to run. Yup, test-driven test helper functions.

If you like behaviour-driven development, and you're working with a .NET class library, then using the aliases for given-when-then as specified above seems a lot easier than writing compiled code versions of the test scripts. Further, and I shouldn't even say this, you don't even need PSExpect to do this - you can write you own version of invoke-block and set up your own aliases to match your own style. Meta-programming is just one of the advantages of a dynamic language.

Next Steps: Hyper-Testability

But there's more that can be done if you are using Powershell.

Visualize if the test authoring helper functions such as test-accountbalance, set-accountbalance, transfer-money, create-account were Powershell cmdlets instead of Powershell functions. It's nice to have those tester helper functions initially in Powershell because they can be quickly modified to match the needs of the test authors, however, they will eventually stabilize. That's when it's time to make them cmdlets and release them along with the rest of the code. Release the testing support tools as part of the working software you're delivering (at least when you're releasing into a test environment).

Having the cmdlets for supporting the automated testing (behaviour-driven or otherwise) makes any number of scenarios possible - and certainly enhances the testability of the system. And I'm willing to bet that there is massive overlap between the cmdlets used for system administration and the cmdlets used for testing. In other words, if you're planning to use cmdlets (and an MMC snap-in) to support the administration of your product, then taking it just-a-little further and creating cmdlets for testing purposes isn't all that much more work. And the result is something that I'm starting to label "hyper-testability" - where an empowered, educated tester can write any number of scenarios using building blocks provided by the system under test.

Three cheers for empowered testers, testable systems, and oh yes - three cheers for enlightened discourse among a group of really smart people that were kind enough to let me listen in.

Envisioning Next-Generation Functional Testing Tools

Thu, 11 Oct 2007 15:47:36 GMT

I'm at a workshop in Portland this week, participating in a workshop for envisioning the next-generation functional testing tools. My interest is in how dynamic languages like Powershell fit with functional testing.

My experience on my last project gave me a tremendous amount of hope that Powershell is effective for executing tests and I can easily envision that a cmdlet-adorned application reduces the cost of testing, especially if _all_ the features of the application are exposed through cmdlets. You just need the scripting skills and the domain knowledge to map the scripts to meaningful user scenarios.

The spreadsheet front-end no doubt that I slapped onto PSExpect helps the business users to get involved, but we need more than their involvement, we need them to be driving it. We need to support the communication and the craft - two things that are hard to balance.

I'm hoping this week to see what other people are doing, what they are saying they need, and what they have done in their practice to make automated functional testing more mainstream and more accessible to the people with the domain knowledge ...

The Developer-Tester Double-Triple

Wed, 09 May 2007 15:00:33 GMT

I've reached a milestone on my current project today – a Developer-Tester double-triple. Triple-digit automated unit tests and triple-digit automated acceptance tests. Unit tests are automated with NUnit and run interactively using TestDriven.net and NCover/NCoverExplorer that ships with TestDriven.net. The automated acceptance tests are all written in PowerShell using PSExpect. The unit tests require about 5-7 seconds to run and the acceptance tests take about 1-2 minutes.

This won't be all of the testing though, the automated unit tests and the automated acceptance tests are only part of the testing strategy on the project. There are GUI tests to come, but we've not yet decided on a tool for automating them, if any. Since we have a focus group of business testers, it might make sense for us to create test checklists for them to use and run those tests manually. We'll see. We can afford to be adaptable since we're adapting an existing site and the number of affected pages is less than 5.

There are also additional unit tests to come as the final integration point gets wrapped in. We've isolated that point for now with a mock provider but we'll have to swap out that mock and use the real thing soon –after it passes all its unit tests.

PowerShell Script Logging and Tracing

Wed, 18 Apr 2007 15:01:07 GMT

From Redmond, WA. I'm at the Platform Adoption Center (PAC) (Building 20) for Apr 18-19 at the PowerShell & MMC ISV Workshop.

In some scripts, especially longer ones, you might find it handy to use logging and tracing that you can turn up or down, depending on the context and depending on yours needs at the time you are running the script.

Using a logging framework would let you replace your 'write-host' statements with logging statements that will have the following characteristics:

You can turn them on or off with a single line edit (at the beginning of your script)
You can re-direct them to various destinations - console, file, event log - again with a single line edit
You can adjust the detail of the log entries - again with a single line edit

The need is so common in application development that there are multiple solutions. The three that stand out the most (probably) are the PowerShell cmdlets write-* (including write-debug, write-warning, write-error, etc), log4net, an open source solution available from Apache, and the Enterprise Library logging application block from Microsoft's patterns and practices group.

During script development, it is going to be handy to have verbose output of these logging statements turned up so that you know what is going on in the code, not just by looking at the results but also by looking at a record of has been run. During production script runs, you might need to have a log of everything your script does. Then again in some situations, you only want to see the logging statements that are associated with errors or other exceptional conditions.

Adding logging might be overkill for exploratory scripting - a scripting technique that PowerShell supports so well with the 'get-member' and 'get-history' cmdlets. But I can imagine that if you 'productionalize' those scripts developed in an exploratory scripting session, then replacing 'write-host' with a logging statement might be a valuable refactoring.

I'll let you choose which of the above options is best for you as an administrative scripter, but I'm inclined to think that you might go with the flow - whatever the application developers are using will probably already work for you. If there are no application developers to follow (or lead), then the PowerShell write-* cmdlets might serve you well enough.

Script Logging with Write-* Cmdlets

PowerShell has built-in cmdlets for logging that use the verb "write". Running the command "get-command write*" will list those commands for you (and a couple of other ones). Each one is controlled by a shell variable that ends with "Preference". For example, to turn the warning messages up, then set the variable $WarningPreference to "Continue". Other options are Stop, Inquire, and SilentlyContinue. To turn down logging, set each of the $*Preference session variables to "SilentlyContinue". To turn up logging, then set them all to "Continue".

Note - if you're building cmdlets, then there are base class methods that work with the same $*Preference shell variables. Your cmdlets will be better received if you use them.

Script Logging with Log4net

Log4net has the advantage of having a very small footprint and, appropriate for scripters, it doesn't require a configuration file. Once you downloaded and unzipped the log4net binaries in a known location (hereinafter referred to as $Log4NetHome) you can use it something like this:

$logpattern = `    "%date [%thread] %-5level [%x] - %message%newline"

# load the log4net library
[void][Reflection.Assembly]::LoadFile($Log4NetHome + "\log4net.dll")

# configure logging
[log4net.LogManager]::ResetConfiguration()
$Appender = new-object log4net.Appender.ConsoleAppender(`
    new-object log4net.Layout.PatternLayout($logpattern))[log4net.Config.BasicConfigurator]::Configure($Appender)

# determines the log statements that show up
$Appender.Threshold = [log4net.Core.Level]::Info$Log = [log4net.LogManager]::GetLogger($Host.GetType())

$Log.Info("Started the main script...")
trap
{

    $Log.Error("Failed...")
   return/continue/break
}

#... 
# run your script here, including other $Log.Info statements 
#...  
$Log.Info("Finished the main script.")

In this example, the value of $logpattern is what determines the output of each log statement - the items preceded by the % character are log4net macros that correspond to attributes of the log statement, including what you have specified, %message. The call to ResetConfiguration() is necessary if you tend to re-run the scripts in the same PowerShell session, otherwise you will have to take care to declare the logger variables as having script scope instead of the global scope (the default).

To configure the logging, you really only have to identify the layout of the log entries and an appender to use. As presented above, the ConsoleAppender is used so that the log entries appear in the console. There are also FileAppenders available as part of the framework. I've used the PatternLayout here so that the pattern string $logpattern can dictate what shows up for each log entry. There are other Layouts available, including a SimpleLayout that just lists the category and message for each log entry.

The output of the log entry in this case is something like the following:

2007-04-18 08:30:49,650 [Pipeline Execution Thread] INFO 
  [null] - Started testing parsing.
2007-04-18 08:30:49,400 [Pipeline Execution Thread] ERROR 
  RfcParseExceptionHandler [RFCWorker] - PRS-012:Unrecognized dish: Irregular
2007-04-18 08:30:49,650 [Pipeline Execution Thread] INFO
  [null] - Finished testing parsing.

Changing the log level from "Info" to "Error" would result in only that second log entry being logged - the other ones would be ignored and not show up in the output.

You can match the pattern layout string above to see what each macro generates in the log file. What I have also found handy was using the NDC stack in log4net to identify where the script was running at the time of the log entry. The contents of the NDC stack is presented in the pattern by %x, so you can see that in the three log entries that are in the sample output, the value is only expressed in the second one. Just so happens that's the one that was generated by the C# class library that the PowerShell script was testing. To use the NDC stack in your script, just push a string onto it in an appropriate line preceding what you want to log, and then pop it off the stack afterwards:

 
[log4net.NDC]::Push("test-getrfcaction-account")
//
//... do your thing, including log statements
//
[log4net.NDC]::Pop()

In this sample, that would put the phrase 'test-getrfcaction-account' will appear instead of the 'null' text that appears there now. If you push multiple values onto the stack, then as you might expect, the appended stack message will appear in those log statements, consisting of all values that have been pushed but not yet popped. Hey, it's a stack. You can appreciate that if you have several script file or several functions running, you might want to identify where the log entry is sourced from.

For my purposes, the developers were already using log4net in their application so using log4net in my test scripts seemed like a reasonable decision. I do find the single edits handy - I can exclude the informational log entries by changing the Appender.Threshold property to "Error". I change the log level alot because I'm exploring a bit with the PowerShell test scripts, so until I know for certain how the code I'm testing behaves, I keep it verbose. Last thing I do before adding the test scripts to the regression suite is change the log level to "Error".

So the happy conclusion is -

add log statements throughout your script
show your work by changing the log level threshold to "Info"
hide your work by changing the log level threshold to "Error"
change what shows up in each log entry by editing a pattern string
provide a basic version of your custom script call stack using the built-in logging stack (called the NDC)

Hope this helps!

Mission-Targets-Tests

Sun, 04 Mar 2007 21:05:22 GMT

Lately testing people have been talking about testing heuristics - tips, techniques, tactics, strategies for testing something. I have one that I call MTT.

M - Mission
T - Targets
T - Tests

But I remember it as "mean time to testing". Exploratory testing excels in part because the mean time to testing is small compared to (caution: wiki-speak ahead), say, BigUpFrontTestManagement. As an example of minimizing the mean time to testing, the example below uses PowerShell to explore the configuration and functionality of SQL Server Reporting Services 2005.

Mission

Encourage the team to build consensus on why testing will be conducted by paying attention to the project context - who tests and what their background is, what is the system under test, who supports the test environment, what are the communication patterns between testers and developers, ... post the essence of this as a 'test mission' in the team's work area and let them mark it up as the project evolves. The more people that are involved in the test effort, the more useful a 'testing working agreement' becomes. Testing faster requires that all the testers understand why they are testing so that they get to the point.

Targets

If the mission indicates why we are testing, then the targets drill deeper and identify what will be tested. I see a lot of testing without clearly-defined test targets. Some heuristics are test targets, and that helps a lot. Exploratory testing, even when automated, fits really well with test targets because listing them gives you a testing backlog, making citing progress and indicating expected completion dates simpler using something like testing burndown charts.

Consider automated exploratory testing with PowerShell - let's use the Microsoft Reporting Services installation test as an example. To use MTT heuristic and the mission stated above, the second step would be to create a testing backlog of the targets of the tests - in other words, what exactly would you want to test? As with all things agile, this list doesn't have to be complete because you have every right to revise this backlog and prioritize it as necessary, within the context of the test mission. It is important to avoid listing how these targets might be tested - that's left to exploration.

Microsoft Reporting Services Installation Test - Targets (Testing Backlog)

ReportServer - Service Control
ReportServer - Service Properties
Database - Reporting Services Db - Topology
Database - Reporting Services Db - Physical Configuration
Database - Reporting Services Db - Authentication
Database - Reporting Services Db - Authorization
Operations - Performance Monitors - Availability
Operations - Microsoft Operations Manager - Access
Operations - Microsoft Operations Manager - Alerts
Web Service - Availability
Web Service - Authentication
Web Service - Authorization
Web Service - Functional - Run Report
Report Manager - Availability
Report Manager - Authentication
Report Manager - Authorization
Report Manager - Functional - Reports
Report Manager - Functional - Data Sources
Report Manager - Capacity - Concurrency
Report Manager - Capacity - Reponse Time

Given this list of test targets, an exploratory test effort would start by estimating each one of the targets to create a testing burndown chart, and then diving in according to the priorities assigned in the backlog. PowerShell is incredibly useful for this given the 'get-member' cmdlet.

Tests

Continuing the Reporting Services example, the first thing that a tiny bit of investigating reveals is that SQL Server Reporting Services has a WMI class that exposes the configuration. Let's get that class and see what we can do with it.

$targetcomputer = "localhost"
$rs = get-wmiobject -computer $targetcomputer -namespace `
"root\Microsoft\SqlServer\ReportServer\v9\Admin" MSReportServer_ConfigurationSetting
$rs | get-member


Name	MemberType
----	----------
BackupEncryptionKey	Method
ConfigureSharePointExclusion	Method
CreateApplicationPool	Method
CreateVirtualDirectory	Method
DeleteEncryptedInformation	Method
DeleteEncryptionKey	Method
GenerateDatabaseCreationScript	Method
GenerateDatabaseRightsScript	Method
GenerateDatabaseUpgradeScript	Method
InitializeReportServer	Method
ListReportServersInDatabase	Method
ReencryptSecureInformation	Method
RemoveUnattendedExecutionAccount	Method
ResetVirtualDirectoryMappings	Method
RestoreEncryptionKey	Method
SetDatabaseConnection	Method
SetDatabaseLogonTimeout	Method
SetDatabaseQueryTimeout	Method
SetEmailConfiguration	Method
SetPortNumber	Method
SetSecureConnectionLevel	Method
SetServiceState	Method
SetUnattendedExecutionAccount	Method
SetWebServiceIdentity	Method
SetWindowsServiceIdentity	Method
ApplicationPoolActual	Property
ApplicationPoolConfigured	Property
ConnectionPoolSize	Property
DatabaseLogonAccount	Property
DatabaseLogonTimeout	Property
DatabaseLogonType	Property
DatabaseName	Property
DatabaseQueryTimeout	Property
DatabaseServerName	Property
InstallationID	Property
InstanceName	Property
IsInitialized	Property
IsSharePointExclusionConfigured	Property
IsSharePointInstalled	Property
IsWebServiceEnabled	Property
IsWindowsServiceEnabled	Property
PathName	Property
SecureConnectionLevel	Property
SenderEmailAddress	Property
SendUsingSMTPServer	Property
ServerPort	Property
ServiceName	Property
SMTPServer	Property
UnattendedExecutionAccount	Property
VirtualDirectory	Property
VirtualDirectoryHasSSLCertificate	Property
WebServiceIdentityActual	Property
WebServiceIdentityConfigured	Property
WebSite	Property
WindowsServiceIdentityActual	Property
WindowsServiceIdentityConfigured	Property
__CLASS	Property
__DERIVATION	Property
__DYNASTY	Property
__GENUS	Property
__NAMESPACE	Property
__PATH	Property
__PROPERTY_COUNT	Property
__RELPATH	Property
__SERVER	Property
__SUPERCLASS	Property
ConvertFromDateTime	ScriptMethod
ConvertToDateTime	ScriptMethod
Delete	ScriptMethod
GetType	ScriptMethod
Put	ScriptMethod

From an exploratory tester's perspective, this list is super useful. Now you can explore the configuration settings interactively using the $rs variable (they are all listed as Property in the above listing), or create a test script and use data-driven testing to check all the Reporting Services configuration parameters against expected values. The 'get-history' cmdlet is also useful to retrieve and save the commands that you have experimented with in a format that you can use later for building a script.

To do the data-driven testing in Excel, create a table with three columns and all the properties that you want to test for. Name the range TestRsConfig and save the worksheet.

TestCase	Property	Value
Security - Service Identity - As Configured	WindowsServiceIdentityConfigured	NT Authority\NetworkService
Security - Service Identity - Actual	WindowsServiceIdentityActual	NT Authority\NetworkService
Service - Instance	InstanceName	SQLEXPRESS
Service - ServiceName	ServiceName	ReportServer$SQLEXPRESS
ServicesDependedOn - Database - Server	DatabaseServerName	(LOCAL)\SQLEXPRESS
IIS - WebSite	WebSite	1
Security - IIS - WebServiceIdentity - Configured	WebServiceIdentityConfigured	ALAPTOP\ASPNET

Next create a script that includes a function called TestRsConfig (the same name as the named range above) as follows (this script uses the PSExpect testing library for PowerShell):

set-psdebug -strict -trace 0
$targetcomputer = "localhost"
$rs = get-wmiobject -computer $targetcomputer `
    -namespace "root\Microsoft\SqlServer\ReportServer\v9\Admin" `
    MSReportServer_ConfigurationSetting

# Exercises the target of the test - the 
# and responds with a pre-defined set of responses that were
# on the worksheet as the expected results
function TestRsConfig()
{
    param([string]$TestCase, [string]$RsConfigProperty, [string]$ExpectedValue)

    $ActualValue = $rs.PSBase.GetPropertyValue($RsConfigProperty).ToString()
    if ([string]::IsNullOrEmpty($ActualValue)) {$ActualValue = $null}

    if ($ExpectedValue.Trim() -eq "N/C")
    {
        AssertNull $ActualValue -Label $TestCase -Intention $Intention.ShouldPass    
    }
    else 
    {
        AssertEqual $ExpectedValue $ActualValue -Label $TestCase -Intention $Intention.ShouldPass
    }

    RaiseAssertions
}

# run the function library that contains the PowerShell Testing Functions
# the functions are defined as global so you don't need to use dot sourcing
if (!(Test-Path variable:_XLLIB)) { ..\src\DataLib.ps1 }
if (!(Test-Path variable:_TESTLIB)) { ..\src\TestLib.ps1 }

# Configuration and installation testing ...
# Confirm the Reporting Services configuration matches expectations documented in 
# the Excel workbook 'TestRsConfig.xslx'
write-host "Configuration testing ..."
$FieldNames = ("TestCase", "RsConfigProperty", "ExpectedValue")
start-test ((get-location).ToString() + "\TestRsConfig.xlsx") "Sheet1" `
    "TestRsConfig" $FieldNames

The line of note contains '$rs.PSBase.GetPropertyValue' - a PowerShell base method that is proving to be really useful in writing test scripts without hard-coding the property names. In my script, I've used 'N/C' to identify Reporting Services properties that are not configured at all on my laptop. If I had left them out and therefore avoided the if-then-else code section, the script would only have been two lines. (The start-test function from PSExpect runs the test and matches up the named range with the test function). Not bad!

But that's only part of the picture - the configuration of Reporting Services. The other part of the picture is basic functional testing. Reporting Services exposes its functionality as the Report Manager, a web application but also as Report Server, a web service. So - what if we used PowerShell to explore and test the Reporting Services web service? Using the same technique as was used with the Reporting Services WMI class, we can use get-member to help with the exploration. This doesn't mean that we don't have to test the Report Manager, but it does mean that we can get a quick check of the functionality without a manual test.

This requires a small bit of .NET working knowledge regarding web services. The trick is to use the .NET SDK 'wsdl.exe' utility to generate a proxy class for the web service, and then wrap that proxy class into an assembly that PowerShell can access:

wsdl http://localhost/ReportServer/ReportService2005.asmx?wsdl 
csc /target:library ReportService2005.cs

Now load this library into PowerShell and use get-member again to investigate the methods that are available:

[void][Reflection.Assembly]::LoadFile("C:\...\ReportingService2005.dll")

# Instantiate the proxy object for the web service
$repsvc = new-object ReportingService2005
$repsvc.Credentials = [System.Net.CredentialCache]::DefaultCredentials
$rs | gm

(I won't list all the methods here because there are many).

...
CancelAsync
CancelBatch
CancelBatchAsync
CancelJob
CancelJobAsync
CreateBatch
CreateBatchAsync
CreateDataDrivenSubscription
CreateDataDrivenSubscriptionAsync
CreateDataSource
CreateDataSourceAsync
CreateFolder
CreateFolderAsync
CreateLinkedReport
CreateLinkedReportAsync
CreateModel
CreateModelAsync
CreateObjRef
CreateReport
CreateReportAsync
CreateReportHistorySnapshot
CreateReportHistorySnapshotAsync
CreateResource
CreateResourceAsync
CreateRole
CreateRoleAsync
CreateSchedule
...

The (almost complete) example included in the PSExpect download uses the ListChildren, CreateDataSource, and CreateReport methods of the web service to check the functionality of Reporting Services. The test script uses functions that wrap the calls to the web service in the name of making the test easier to read for testers not familiar with the Reporting Services web service interface. This is not the complete test - ideally there would also be a 'render-report' and 'test-report' set of functions provided for the testers so that actually generating a report is also part of the test (work in progress).

write-host "Functional testing ..."
$DsBefore = get-datasourcecount
new-datasource -Name "FromPsScript"
test-datasourcecount -Expected ($DsBefore + 1) -Label "Datasources.Count.AfterNew"
remove-datasource -Name "FromPsScript"
test-datasourcecount -Expected ($DsBefore) -Label "Datasources.Count.AfterRemove"

# Verify the report create/delete functionality
$RsBefore = get-reportcount
new-report -Name "ReportFromPsScript"
test-reportcount -Expected ($RsBefore + 1) -Label "Reports.Count.AfterNew"
remove-report -Name "ReportFromPsScript"
test-reportcount -Expected $RsBefore -Label "Reports.Count.AfterRemove"

The example is available for download in the latest source code version of PSExpect. All together, this wasn't very much code. The 'mean-time-to-testing' is low considering the automation - and I have an artifact that other people can use for running the tests at any time.

RsLib.ps1 - Contains the wrapper functions new-datasource, test-datasourcecount, remove-datasource, etc.
Test-RsConfig.ps1 - the test script for both configuration and functional tests described above
Test-RsConfig.xslx - Excel spreadsheet containing the named range for properties and expected values used in configuration testing.

Summary

The 'get-member' cmdlet is extremely useful for exploratory testing, especially when combined with the easy access to WMI classes and to web services in PowewrShell. In the example, I've used the SQL Server Reporting Services WMI class to check the configuration and installation against expected values, and then used a proxy class around the Reporting Services web service to check basic functionality. What 'get-member' and PowerShell do for us in these cases is give us the tools and the means for exploring the targets of our tests. Oh yeah - everything is in a script that we can share and evolve as our understanding of the test target increases.

Lowering the Cost of Test Automation

Tue, 13 Feb 2007 23:37:09 GMT

It's been pretty well-documented that test automation, especially at the higher test levels such as system or acceptance testing, is something you should undertake carefully. Like any other development project, the economic argument has to be there to support the effort - and that economic model can be screwed up pretty easily by analysis-paralysis (planning tests without ever writing or running one) or by creating unmaintainable, brittle test scripts.

There are ways of lowering the cost of automating tests. There is a PowerShell hook, honest.

Increase the 'Test Cases to Test Script' Ratio

One of the reasons that FIT is spectacular is that it standardizes the test script and permits the tester to focus on providing examples in the form of combinations of test input and expected output. It does this through the use of test fixtures like the ColumnFixture. To use ColumnFixture, you design your test by creating a table containing all the test data (inputs and expected outputs) and then write a test fixture that translates a row in the table into a call to the system under test (SUT). Each row in the table represents a test case - and the running of each test case is all done implicitly by FIT and your test fixture.

It's pure-and-simple data-driven testing. You achieve a high 'test case to test script' ratio by virtue of never having to write the script beyond writing the test fixture. The framework implicitly provides the script - and runs it.

You can replicate the use of ColumnFixture in PowerShell/PSExpect by storing the data in Excel (and learning what a named range in Excel is). See \samples\test-getweatherperf.ps1 for an example. There are several advantages of using Excel for the test data:

You can change the scope of a test run by changing the definition of the named range on the worksheet - eliminating the need to comment out parts of an input file or worse yet, storing multiple versions of it.
You can use formulas to help specify the test data - so some of the test cases can use randomly-generated data instead of the same data every time.
You can ask business users or other domain experts to create or revise the worksheet - Excel is ubiquitous in corporations. Little or no training required.

But I've written about PowerShell/PSExpect and Excel before. On to other interesting methods of minimizing test automation costs.

Eliminate Waste - Part 1 - Use Test Targets Wisely (Lean Test Management)

'Waste' in testing comes in two forms: writing test scripts that target the wrong thing, and writing scripts in a form that makes them difficult to evolve as the system under test evolves, or as your understanding of the system under test evolves.

You can avoid testing the wrong thing by first building a list of 'test targets'. A test target is basically anything that needs testing. Application functionality, security, performance, etc. might be types of tests but they are not test targets. Well at least they aren't specific enough to be used wisely.

Break out the types of tests to get specific test targets. List the application functions (use cases, user stories, whatever) - those are functional test targets. List the application interfaces - those are integration test targets. Security test targets include authentication, authorization, and then any specific vulnerability you want to test for (sql command injection, os command injection, etc.). Performance test targets will be objectives-based like throughput, resource consumption limits, or user concurrency.

You need this list so that you can be wise about setting priorities and so that you can plan to write scripts for only those test targets that really matter. Host a workshop to decide these priorities so that the whole team is on the same page.

The list of test targets is - dare I say it - a testing backlog. If you're familiar with Scrum and the use of a product backlog, then take those same concepts and apply them to testing and the testing backlog. This process works well - I've used testing backlogs and Scrum concepts now on a large enterprise project (an SAP upgrade), an infrastructure project (computing infrastructure migration to Windows Server 2003), and a business intelligence project. All with favourable results.

Using a testing backlog is only part of the picture though - the scripts themselves can become waste if they are not written wisely.

Eliminate Waste - Part 2 - Write Lean Scripts (Or None At All)

Test scripts are unmaintainable when they hard-wire the test logic, the test data, and the calls to the system under test. Basically, if any of those three factors changes, then you have to revise the script. As a test author, you want a test automation strategy that enables you to avoid the hard-wiring trifecta.

For manual tests, avoid writing scripts if you can - instead, rely on your testers to make the leap from the testing backlog to the test results. they can journal their way through the test targets and still provide excellent test results documentation. If you can't avoid writing manual test scripts, then choose to write them using only language from the domain - avoid referring to specific user interface elements, avoid binding the script to a particular processing style, etc. For example, say 'approve invoice' not 'click the Approve button'. Then if that handy little 'Approve' button disappears in a later revision of the system, the test script survives without changes.

Eliminate Waste - Part 3 - Dependency Injection for Testers (Keyword-Driven Testing)

Automate tests but minimize the test script's dependency on the automation API of the system under test. This is analogous to writing manual test scripts that minimize the script's dependency on the user interface elements such as buttons or menus.

Consider the test script whose automation depends directly on the system under test. Any changes whatsoever, and the test script may have to be revised. To do this in PowerShell, for example, your PowerShell test script would make calls directly against the assemblies that make up the system under test.

Now consider injecting a dependency that shields the test script from the system under test. It's true that the test fixture may have to change, but your test scripts - ideally - would not be subject to the same degree of change as the test fixture might.

Unlike the FIT test fixtures that allow you to run several test cases, these injected test fixtures translate test steps into calls to the system under test. That's key. A better name for them is actually 'test-step fixtures', given their role. Give these test-step fixtures names that originate with the business domain, and you have given life to 'keyword-driven testing' in the domain of the system under test. Otherwise known as the ubiquitous language or domain-driven design.

I'm advocating using PowerShell to write the test scripts and PowerShell cmdlets to implement the test-step fixtures (the bridge). Until their usefulness is proven in context, the cmdlets can be prototyped as PowerShell functions.

Two roles emerge: the business test scripter that knows the business domain, and the developer scripter that knows the automation API of the system under test. The test scripter uses the test-step fixtures to assemble scenario tests. The developer scripter builds the test-step fixtures, either proactively or in response to the needs of the test scripter.

So does this lower the cost of test automation? We've decreased the coupling between the tests and the system under test by injecting a dependency that we control - so we've made the test less susceptible to technical changes, if not entirely to requirements changes. But even then - the scenario tests are built from modular components (test-step fixtures) - and modularity is a good thing - because it increases the likelihood of high maintainability and extensibility. The test-step fixtures may change, but ideally the test scripts themselves would only be adapted as the complexity of the system under test evolves.

On to the PowerShell hook. Write the test-step fixtures in the form of cmdlets or PowerShell functions, following the same verb-noun naming convention that we might use for building any cmdlet or function. EXCEPT, for testing purposes, we would prefer to take the 'nouns' from the domain (maybe even the verbs but then we could no longer build cmdlets unless the verb was also on the standard list of PowerShell verbs).

For example:

$MyAccount = get-account 'name'
set-contract $MyAccount

Once you've established a number of the domain-specific verb-noun combinations, then you can build test scenarios in PowerShell using them whose complexity matches the complexity of the system under test - they can evolve independently but at the same pace. Those test-steps listed above in the example expose nothing of the automation API for the system under test - a good thing if we want to retain the test scripts usefulness over time.

In summary, here are the ways that PowerShell excels for keyword-driven testing:

The verb-noun style is PowerShell vernacular - we're not introducing anything new by asking the test scripters to use cmdlets or functions that follow a verb-noun naming convention.
PowerShell provides access to both System and custom assemblies and namespaces. You might use either or both to implement the test-step fixtures.
The system under test may require an administration module that might also utilize PowerShell and cmdlets - there would be overlap in administration-type cmdlets and the test-step fixtures so that building them for one purpose would help satisfy the other requirement as well.
PowerShell provides exception handling so that test scripters can control the impact of any failed test-steps.
PowerShell provides all the control structures so that test scripters can control the flow and execution of test scripts - including looping. Many other test automation styles only provide rudimentary control, if any.
The test script development cycle is short - you write a script, you run it, edit, run it again, etc. very quickly. You can even prototype script steps in the console.
Test-step fixtures can vary in their complexity - you can extend their domain-specific nature by giving them domain-oriented parameters. For example, new-customer 'GOOD CREDIT', might generate a new customer with a good credit rating to use in the test script.

For a slightly more detailed example, there is one posted for PSExpect that uses this style of test scripting - samples\Test-Vcs.ps1. The sample domain is a volunteer coordination system (vcs).

PSExpect 0.3 Alpha

Sun, 04 Feb 2007 19:09:34 GMT

Release 0.3 (Alpha) of PSExpect has been uploaded - hopefully it is clear from the samples and the tests that are included that PowerShell is well-suited for automating tests using both data-driven and keyword-driven techniques.

Release notes for 0.3 Alpha are as follows:

Addition of a 'start-test' function in DataLib.ps1 that uses an Excel named range and a function you build to test a target script or application library. See \samples\Test-GetWeatherPerf.ps1 for an example.
Addition of a 'select-row' function in DataLib.ps1 for those data-driven testers that want more control of their test script and its use of Excel
The Excel functions now use reflection to call the Excel COM API - so these should work in cultures other than 'en-US'
Addition of a keyword-driven testing example - using verb-noun pairs from the business domain to write maintainable test scripts.
AssertBlock is renamed to Assert-Block and is now pipeline-aware
AssertFaster is now pipeline-aware
AssertThrows is now pipeline-aware

Next up: a better explanation of keyword-driven testing using PowerShell. I've included an example in 0.3 Alpha but only in-line comments to explain what it going on and why.

A Compatibility Story

Thu, 01 Feb 2007 23:36:35 GMT

Kudos to the PowerShell Team and the Office Team. I have a number of test cases for and samples of PSExpect that rely on Excel - part of my kick to support data-driven testing using PowerShell.

Yesterday, Microsoft released PowerShell 1.0 for Vista, and my Vista test machine also has Office 2007. The PSExpect tests pass on Vista/Office 2007, as they do on Windows XP/Office 2003. Including the ones that use Excel.

That's a compatibility story, and I'm impressed.