Like many of you, I noticed the download that was made available on the MSDN Tester Centre the other day - Testing Applications with Powershell - Sample Code.  This link, along with downloads and article such as the UI Automation Specification and associated Community Promise gives me some hope that it might be possible to run the same test script both with and without the UI.

That's my hope for testing tools.  Oh yeah, and that we can do that using business-oriented grammar in the scripts.  And that we can define, scope, and reuse elements of that grammar in exploratory testing, semi-automated style.

Anyway, baby steps ... baby steps.

The post was great in that it raised the awareness of how Powershell might be used for testing.  All of my examples in PSExpect are against libraries or other scripts.  This example automates the UI.  And that's cool if not entirely necessary in all contexts.

I replaced the verdicts within the posted sample code with calls to PSExpect Assert* functions. There were two of them, depicted below with the original code hashed (commented) out:

$result = get-listBox $lb "222"
#    if ($result -ge 0) {
#      write-host "Found '222' in ListBox!"
#    }
#    else {
#      write-host "Did NOT find '222' in ListBox"
#      $pass = $false
#    }
AssertEqual -Expected 0 -Actual $result -Label "TC-1" -Intent $Intention.ShouldPass
...

$text = get-textBox $tb
#    if ($text -eq "ad") {
#      write-host "Found 'ad' in TextBox!"
#    }
#    else {
#      write-host "Did NOT find 'ad' in TextBox"
#      $pass = $false
#    }
AssertEqual -Expected "ad" -Actual $text -Label "TC-2" -Intent $Intention.ShouldPass
...

and then there is the way that the test script reports back to the tester about the status of the test run:

#    if ($pass) {
#      write-host "`nTest scenario result = Pass" -foregroundcolor green
#    }
#    else {
#      write-host "`nTest scenario result = * FAIL *" -foregroundcolor red
#    }
RaiseAssertions

Running the revised test script and running it against the sample application to be tested generates the following two lines of output:

02/06/2008 12:39:26 PM,SHOULDPASS,PASSED,TC-1
02/06/2008 12:39:28 PM,SHOULDPASS,PASSED,TC-2

Since the tests pass, they show up as green.  You can find the entire script at(http://www3.telus.net/~amgeras/samples/Test-TheApp.ps1).  You will find that the script requires the PSExpect library in a specific location - just change that location to get the script to run (one line to edit).

We did drop a number of lines of script completing this refactoring. We also gained functionality since when you run the script, you are also generating a log file in behind the scenes that could be posted in the test results repository.  That's all good.  But there are other refactorings that we might consider that I will tackle when I get a moment.

First, from a testing perspective, the functions get-listBox is counter-intuitive since it returns an integer when given an item that it is supposed to contain.  I understand why, but the consequences to the test script are obvious when you compare the two AssertEqual statements above.

In the first AssertEqual, the expected result is '0' yet we are really looking for the value '222'.  In the second AssertEqual, we are looking for 'ad' and that is exactly what we truly are looking for.  That's more intuitive.  So in the next refactoring I would remove the comparator in the function and leave that comparison for the tester.

Second, and this only reflects my bias towards domain-driven design thinking, is that the sample application mimics some function that doesn't come through in the test script unless you look really close.   It gets the point across for people like me to talk around, so it was fit for purpose. 

But ...

If the screen functionality could be represented using a verb-noun naming convention, then that would provide the means for separating the test runner from the test actions from the set of verdicts and that would make it even more meaningful, and the test script more readable.

For example,

start-productSearch -Category $null -Item $null
test-productSearchResults -Expected 'Error'

start-productSearch -Category 'Product Name' -Item 'ad'
test-productSearchParam -Expected 'ad'
test-productSearchResults -Expected '222'

Now, the test runner is still the Powershell script (as it was before we did any refactoring).  The screen action is start-productSearch.  And the verdicts are test-productSearchParam and test-productSearchResults, both implemented in the two AssertEqual statements above but whose meanings were hidden in the interpretation of the script.

And this test script is now much more reflective of what the screen does and how we're testing it.  It communicates better to the testers, and we have this function start-productSearch that we could reuse in other , longer-running scenarios of the larger application.

Right now, these last two refactorings are musings, but I'll post them and my findings along the way as soon as I get a chance.

Testability is a software quality attribute that is a measure of how easy (or difficult) it is to create test cases for a test target and, in combination, how easy (or difficult) it is to run those tests.  High testability is a good thing since it means that the total cost of making changes to the software is correspondingly lower.  It typically also means that you can have more than one individual contributing to the software since it is easy for them to determine whether or not their contribution has been a net positive or net negative contribution, simply by running the tests again.  Hence, we like test automation.

It turns out that when we value testability, we tend to write software that enables us to automate more types of tests, and we set up this fantastic feedback loop where defining a test becomes a means of production - the tests specify the desired behaviour, guide the development, and then verify the as-built software for as long as that behaviour remains desired.  Pretty nice, when we get to that level.

Tools like xUnit were revolutionary in the sense that they gave developers the means for valuing testability and a mechanism for expressing the low-level behaviour in the same language that our target software was (or would be) written in.  Developers liked it, and now teams in general are much more inclined to automate unit tests than ever before.

But is it enough?  That's a question that Brian Marick has been pondering publicly in his blog (it's one of those blogs that actually expands your mind when you read it) and in an on-line forum discussing automated functional testing.  I was struck by the phrase "Perhaps it would be better to invest heavily in unprecedented amounts of built-in support for manual exploratory testing." the most.  An alternative, he indicates, to automated functional testing with tools like FIT/FITNesse (read the blog entry to find out why this isn't agile heresy).

You can see where I'm going. If you have a set of Powershell cmdlets that exposes an application's functionality - and if the cmdlets expose ALL of the functionality - then 'exploring' means working with those cmdlets (or an API) in a Powershell session and using whatever testing heuristics you might have in your tester's toolkit to test the application.  You may not be using the same UI as the end-users, however, there is great value to first confirming the functionality is good enough, and then second moving on to confirm that the UI is good enough.

The Powershell features that make exploratory testing an excellent experience are (but not limited to):

• Objects on the pipeline, and in fact, having objects to work with at all.
• The get-help cmdlet that can access custom help files for custom cmdlets.
• The get-member (gm) cmdlet.
• The get-history (h) cmdlet.
• The naming convention for cmdlets.

You don't even really have to know the cmdlets ahead of time because of those features since you can retrieve their calling syntax at run-time.  The get-history cmdlet can then be used to save useful cmdlets and sequences of calls, in order to support future semi-automated exploratory testing (where some scripts are run, but only for the sake of returning to a known state) or ultimately fully automated testing for something like detecting regression.  Seems that the ability of a script for passing on knowledge isn't limited to administrators ...

I can envision, however, a mismatch between the cmdlets that an application designer foresees might be a useful way of exposing the functionality and the cmdlets that a tester might find useful.  They will each bring, I believe, a different perspective.

So it might be wise to undertake this 'unprecedented support for exploratory testing' or hyper-testability using a test-driven approach where the functionality of those cmdlets is influenced, or possibly specified, by the examples that a tester might deem appropriate.  Supporting manual exploratory testing by writing automated test scripts ahead of time ... seems contradictory. But there is a distinction between 'understanding-clarifying tests' (Brian Marick again) and 'verification and validation' tests.  So using a test-driven approach for understanding and clarifying requirements, and ultimately the design of the cmdlets, isn't the same as writing tests for verifying and validating the resulting application.

The bottom line?  Exposing application functionality in Powershell makes your application hyper-testable.  Use test scripts that you/your testers write ahead of time as a means of designing the cmdlets and separate the concern of testing the functionality of the application from the concern of testing the user interface.

I know we're not supposed to come up with solutions before fully understanding the problem, but I've got just enough experience and wide-eyed insensibility to believe that I've seen something that just might work: testing mashups.

Microsoft Popfly is a set of tools for creating web pages and the like from pre-defined blocks.  The one aspect that I am particularly intrigued by is called a mashup.  You use a tool called the Mashup Creator to build a web page comprised of blocks that come from disparate sources.  For example, build a page that lists your Facebook friends' statuses and your Twitter friends' most recent updates.  I'm assuming it's called a 'mashup' after the term that's been coined for mixing songs into single tracks (some of my favourites have sampled from 5 or more songs).

Now imagine if the "blocks" used to build a mashup were Powershell cmdlets instead of web applications and that those cmdlets were designed to expose the functionality of an application under test.  Each "mashup" you create is therefore another application scenario.  Imagine that some of the blocks you use on a mashup are data sources so that you could make the scenario data-driven.

So, why do I think this might be a good idea?

First, it may help non-programmers to develop automated test scenarios that are executable.  Second, the economics of building cmdlets for the application just changed: not only might the cmdlets be useful for system administration, but now they will also be useful in writing automated tests for the application.

Why cmdlets? Well they exhibit all the behaviour that I would want from blocks for building tests from - they have a naming convention, there are behavioural standards that they adhere to, and there is a mechanism already in place for stringing them together and running them as one scenario, so they're not just commands that can be invoked.  If the 'noun' part of the cmdlet always refers to something from the business domain, then we're working at the same level of abstraction as (hopefully) the requirements are written - so as testers we're grounded, too.  There is a tie-in to the whole world of domain-specific language and domain-driven development that just can't be ignored.

Now these aren't the only tests that would need to be considered for any system under development.  A test strategy for any software product is going to include extensive developer (unit and low-level integration) testing and manual functional and acceptance testing. I get that.   At the same time, automating more test scenarios means they are shareable and immediately executable by anyone on the team, enhancing communications and reducing communication overhead.  That's a good thing, and worth striving for.

or, A Testament to the Flexibility of Powershell

The greatest impact that the Functional Testing Tools Workshop in Portland (October, 2007) had on me was to raise my awareness of behaviour-driven development, in particular, the 'given-when-then' pattern for structuring examples.

I will defer to other blogs on describing given-when-then in detail, you'll see what it means soon enough in the examples below.  For now, think of given-when-then as a means of translating the meaning of a user story into a related set of acceptance criteria using examples.  Something that has always been a bit awkward for business users and developers alike.

Given you adopt the story-telling convention popularized by Mike Cohn (I think),

As a <blank>
I want to <blank>
So that <blank>

when you want to translate the story into a set of related acceptance criteria using examples,
then use the following convention popularized by BDD proponents such as Dan North:

Given <a known set of conditions or state>
When <something happens>
Then <check for an expected set of conditions or state>

There are tools that help you do this and to make statements worded like this executable - in that case, they almost become executable requirements.  They are certainly executable examples.  The most-often-used example is for a simple account manager class (I've seen this example on a number of blogs but didn't pay attention to where it originated, but I think it was Dan North's again).  I'm not the only person looking at this - there is a NBehave project on CodePlex that provides a mechanism for writing tests in this manner.  And there are many ruby examples in both rspec and rbehave.

# Story: Transfer to cash account from savings account
# As a savings account holder
# I want to transfer money from my savings account
# So that I can get cash easily from an ATM

# Scenario 1: Savings account is in credit
given "My savings account balance is 100 and my cash account balance is 10"
when "I transfer 20 to my cash account from savings"
then "my cash account balance should be 30 and my savings account balance should be 80"

given "My savings account balance is 400 and my cash account balance is 100"
when "I transfer 100 to my cash account"
then "My cash account balance should be 200 and my savings account balance should be 300"

My opinion is that this is a clear way of expressing an example, albeit all that I have really seen are simplistic ones.  I'm going to enjoy working through more complex examples on an upcoming project.

Now on to Powershell.  It was always my intention to introduce some sort of behaviour-driven development helper functions in PSExpect, but I was never certain of what that could be.  Until I went to the Functional Testing Tools conference in Portland.  I love it when I can take some time off to think, and be inspired by a group of brilliant people.

Turns out we can make those examples above executable by writing a function and adding a set of aliases to our Powershell environment.  The function accepts a string parameter and a script block parameter.  All that it does is write out the string to a log file and then run the script block.  I've called it Invoke-Block for lack of imagination, and it uses the PSExpect logging mechanism so that the same log file shows the results of assertions.

function global:Invoke-Block([string]$Label, [ScriptBlock]$Script)
{

    # build the log entry and write it out
    $entry = BuildLogEntry $Intention.ShouldPass $ResultPrefix.Passed "PSpec" $Label
    $added = $Assertions.Add($entry)
    if ($LogFileName -ne $null) {
        $logEntry = BuildMessage $entry
        $logEntry | out-file $LogFileName -append -width $LogFileLineLength
    }
   
    # now run the script block
    if ($Script)
    {
        &$Script
    }
}

set-alias given invoke-block
set-alias when invoke-block
set-alias then invoke-block

The aliases enable using the given-when-then triad in a Powershell script without generating any errors.  This means that the executable version of the example is the same as above, with the script blocks added in.  The example relies on a class library from the system under test (SUT) that the script loads using reflection (not shown) but the calling syntax to that library is hidden using a set of Powershell functions that utilize a verb-noun naming convention: create-account, transfer-money, set-accountbalance, and test-accountbalance (these are elements in the grammar of the domain-specific language, or ubiquitous language for this particular system under development).

# Story: Transfer to cash account from savings account
# As a savings account holder
# I want to transfer money from my savings account
# So that I can get cash easily from an ATM

# Scenario 1: Savings account is in credit
given "My savings account balance is 100 and my cash account balance is 10" {
    set-variable -name SvAccount -value (create-account "Savings" 100) -scope script
    set-variable -name csAccount -value (create-account "Cash" 10) -scope script
}

when "I transfer 20 to my cash account from savings" {
    transfer-money -From $SvAccount -To $CsAccount -Amount 20
}

then "my cash account balance should be 30 and my savings account balance should be 80" {
    test-accountbalance -Account $SvAccount -Expected 80 -Label "Savings.Balance.After"
    test-accountbalance -Account $CsAccount -Expected 30 -Label "Cash.Balance.After"
}

given "My savings account balance is 400 and my cash account balance is 100" {
    set-accountbalance -Account $SvAccount -Amount 400
    set-accountbalance -Account $CsAccount -Amount 100
}

when "I transfer 100 to my cash account" {
    transfer-money -From $SvAccount -To $CsAccount -Amount 100
}

then "My cash account balance should be 200 and my savings account balance should be 300" {
    test-accountbalance -Account $SvAccount -Expected 300 -Label "Savings.Balance.After"
    test-accountbalance -Account $CsAccount -Expected 200 -Label "Cash.Balance.After"
}

RaiseAssertions

The Output

29/11/2007 1:56:13 PM,SHOULDPASS,PASSED,PSpec,My savings account balance is 100 and my cash account balance is 10
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,I transfer 20 to my cash account from savings
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,my cash account balance should be 30 and my savings account balance should be 80
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Savings.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Cash.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,My savings account balance is 400 and my cash account balance is 100
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,I transfer 100 to my cash account
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,My cash account balance should be 200 and my savings account balance should be 300
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Savings.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Cash.Balance.After

The output above is from the log file that PSExpect generates on every run.  That whole SHOULDPASS concept is there to support a programming style where you first create some tests that you expect to fail - you mark them SHOULDFAIL and then PSExpect prints those assertion failures in YELLOW instead of RED like it would if an assertion labelled with SHOULDPASS fails.  I've come to rely on log files to use as evidence of passing tests, and to support debugging (since I tend to avoid using a step-through debugging tool on my own code; it's an anti-pattern for my personal coding style since using a tool like that means I'm confused and I don't have enough tests to show me the way).

A Way of Explanation

Yes, this really executes and the output demonstrates that each of the checks pass - this example is available in the most recent upload of PSExpect on the CodePlex site in the 'samples' directory.  A couple of things worth pointing out ...

First, the starting curly brace of the script block must be on the same line as the call to invoke-block, otherwise Powershell parses the function call without the script block and errors abound.  A minor inconvenience if that isn't already your scripting convention.

Second, the unfortunate use of set-variable in the first 'given' script block.  That's really just in order to set the scope of variables (hence the use of the -scope parameter).  Alternatively, you could declare the variable and instantiate it before any of the 'given' clauses.  I don't think it matters, but this was the way that it was done in the NBehave example on the CodePlex site so I've copied that approach in my example.

Third, those wrapper functions.  I've mentioned in other blog entries that Powershell uses the verb-noun naming convention for all the built-in cmdlets and that you are forced into this same convention when writing your own custom cmdlets.  Well you should follow that form when writing Powershell functions too and write your functions like they are prototypes of cmdlets.  Much better all-around style, and you don't lose any ground making changes to scripts if you choose to promote a function to a cmdlet.  I prefer to use the wrapper functions in test scripts since they enhance the readability of the script for non-developers.  I've rarely specified the wrapper functions ahead of time.  Instead, I write the test scripts first and determine what wrapper functions I need based on the tests that I want to run.  Yup, test-driven test helper functions.

If you like behaviour-driven development, and you're working with a .NET class library, then using the aliases for given-when-then as specified above seems a lot easier than writing compiled code versions of the test scripts.  Further, and I shouldn't even say this, you don't even need PSExpect to do this - you can write you own version of invoke-block and set up your own aliases to match your own style.  Meta-programming is just one of the advantages of a dynamic language.

Next Steps: Hyper-Testability

But there's more that can be done if you are using Powershell.

Visualize if the test authoring helper functions such as test-accountbalance, set-accountbalance, transfer-money, create-account were Powershell cmdlets instead of Powershell functions.  It's nice to have those tester helper functions initially in Powershell because they can be quickly modified to match the needs of the test authors, however, they will eventually stabilize.  That's when it's time to make them cmdlets and release them along with the rest of the code.  Release the testing support tools as part of the working software you're delivering (at least when you're releasing into a test environment).

Having the cmdlets for supporting the automated testing (behaviour-driven or otherwise) makes any number of scenarios possible - and certainly enhances the testability of the system.  And I'm willing to bet that there is massive overlap between the cmdlets used for system administration and the cmdlets used for testing.  In other words, if you're planning to use cmdlets (and an MMC snap-in) to support the administration of your product, then taking it just-a-little further and creating cmdlets for testing purposes isn't all that much more work.  And the result is something that I'm starting to label "hyper-testability" - where an empowered, educated tester can write any number of scenarios using building blocks provided by the system under test.

Three cheers for empowered testers, testable systems, and oh yes - three cheers for enlightened discourse among a group of really smart people that were kind enough to let me listen in.

I'm at a workshop in Portland this week, participating in a workshop for envisioning the next-generation functional testing tools.  My interest is in how dynamic languages like Powershell fit with functional testing.

My experience on my last project gave me a tremendous amount of hope that Powershell is effective for executing tests and I can easily envision that a cmdlet-adorned application reduces the cost of testing, especially if _all_ the features of the application are exposed through cmdlets.  You just need the scripting skills and the domain knowledge to map the scripts to meaningful user scenarios.

The spreadsheet front-end no doubt that I slapped onto PSExpect helps the business users to get involved, but we need more than their involvement, we need them to be driving it.  We need to support the communication and the craft - two things that are hard to balance.

I'm hoping this week to see what other people are doing, what they are saying they need, and what they have done in their practice to make automated functional testing more mainstream and more accessible to the people with the domain knowledge ...

A.

I've reached a milestone on my current project today – a Developer-Tester double-triple. Triple-digit automated unit tests and triple-digit automated acceptance tests. Unit tests are automated with NUnit and run interactively using TestDriven.net and NCover/NCoverExplorer that ships with TestDriven.net. The automated acceptance tests are all written in PowerShell using PSExpect. The unit tests require about 5-7 seconds to run and the acceptance tests take about 1-2 minutes.

This won't be all of the testing though, the automated unit tests and the automated acceptance tests are only part of the testing strategy on the project. There are GUI tests to come, but we've not yet decided on a tool for automating them, if any. Since we have a focus group of business testers, it might make sense for us to create test checklists for them to use and run those tests manually. We'll see. We can afford to be adaptable since we're adapting an existing site and the number of affected pages is less than 5.

There are also additional unit tests to come as the final integration point gets wrapped in. We've isolated that point for now with a mock provider but we'll have to swap out that mock and use the real thing soon –after it passes all its unit tests.

From Redmond, WA.  I'm at the Platform Adoption Center (PAC) (Building 20) for Apr 18-19 at the PowerShell & MMC ISV Workshop.

In some scripts, especially longer ones, you might find it handy to use logging and tracing that you can turn up or down, depending on the context and depending on yours needs at the time you are running the script.

Using a logging framework would let you replace your 'write-host' statements with logging statements that will have the following characteristics:

• You can turn them on or off with a single line edit (at the beginning of your script)
• You can re-direct them to various destinations - console, file, event log - again with a single line edit
• You can adjust the detail of the log entries - again with a single line edit

The need is so common in application development that there are multiple solutions.  The three that stand out the most (probably) are the PowerShell cmdlets write-* (including write-debug, write-warning, write-error, etc), log4net, an open source solution available from Apache, and the Enterprise Library logging application block from Microsoft's patterns and practices group.

During script development, it is going to be handy to have verbose output of these logging statements turned up so that you know what is going on in the code, not just by looking at the results but also by looking at a record of has been run.  During production script runs, you might need to have a log of everything your script does.  Then again in some situations, you only want to see the logging statements that are associated with errors or other exceptional conditions.

Adding logging might be overkill for exploratory scripting - a scripting technique that PowerShell supports so well with the 'get-member' and 'get-history' cmdlets.  But I can imagine that if you 'productionalize' those scripts developed in an exploratory scripting session, then replacing 'write-host' with a logging statement might be a valuable refactoring.

I'll let you choose which of the above options is best for you as an administrative scripter, but I'm inclined to think that you might go with the flow - whatever the application developers are using will probably already work for you.  If there are no application developers to follow (or lead), then the PowerShell write-* cmdlets might serve you well enough.

Script Logging with Write-* Cmdlets

PowerShell has built-in cmdlets for logging that use the verb "write".  Running the command "get-command write*" will list those commands for you (and a couple of other ones).  Each one is controlled by a shell variable that ends with "Preference". For example, to turn the warning messages up, then set the variable $WarningPreference to "Continue". Other options are Stop, Inquire, and SilentlyContinue. To turn down logging, set each of the $*Preference session variables to "SilentlyContinue". To turn up logging, then set them all to "Continue".

Note - if you're building cmdlets, then there are base class methods that work with the same $*Preference shell variables.  Your cmdlets will be better received if you use them.

Script Logging with Log4net

Log4net has the advantage of having a very small footprint and, appropriate for scripters, it doesn't require a configuration file.  Once you downloaded and unzipped the log4net binaries in a known location (hereinafter referred to as $Log4NetHome) you can use it something like this:

$logpattern = `    "%date [%thread] %-5level [%x] - %message%newline"

# load the log4net library
[void][Reflection.Assembly]::LoadFile($Log4NetHome + "\log4net.dll")

# configure logging
[log4net.LogManager]::ResetConfiguration()
$Appender = new-object log4net.Appender.ConsoleAppender(`
    new-object log4net.Layout.PatternLayout($logpattern))[log4net.Config.BasicConfigurator]::Configure($Appender)

# determines the log statements that show up
$Appender.Threshold = [log4net.Core.Level]::Info$Log = [log4net.LogManager]::GetLogger($Host.GetType())

$Log.Info("Started the main script...")
trap
{

    $Log.Error("Failed...")
   return/continue/break
}

#... 
# run your script here, including other $Log.Info statements 
#...  
$Log.Info("Finished the main script.")

In this example, the value of $logpattern is what determines the output of each log statement - the items preceded by the % character are log4net macros that correspond to attributes of the log statement, including what you have specified, %message.  The call to ResetConfiguration() is necessary if you tend to re-run the scripts in the same PowerShell session, otherwise you will have to take care to declare the logger variables as having script scope instead of the global scope (the default).

To configure the logging, you really only have to identify the layout of the log entries and an appender to use.  As presented above, the ConsoleAppender is used so that the log entries appear in the console.  There are also FileAppenders available as part of the framework.  I've used the PatternLayout here so that the pattern string $logpattern can dictate what shows up for each log entry.  There are other Layouts available, including a SimpleLayout that just lists the category and message for each log entry.

The output of the log entry in this case is something like the following:

2007-04-18 08:30:49,650 [Pipeline Execution Thread] INFO 
  [null] - Started testing parsing.
2007-04-18 08:30:49,400 [Pipeline Execution Thread] ERROR 
  RfcParseExceptionHandler [RFCWorker] - PRS-012:Unrecognized dish: Irregular
2007-04-18 08:30:49,650 [Pipeline Execution Thread] INFO
  [null] - Finished testing parsing.

Changing the log level from "Info" to "Error" would result in only that second log entry being logged - the other ones would be ignored and not show up in the output.

You can match the pattern layout string above to see what each macro generates in the log file.  What I have also found handy was using the NDC stack in log4net to identify where the script was running at the time of the log entry.  The contents of the NDC stack is presented in the pattern by %x, so you can see that in the three log entries that are in the sample output, the value is only expressed in the second one.  Just so happens that's the one that was generated by the C# class library that the PowerShell script was testing.  To use the NDC stack in your script, just push a string onto it in an appropriate line preceding what you want to log, and then pop it off the stack afterwards:

 
[log4net.NDC]::Push("test-getrfcaction-account")
//
//... do your thing, including log statements
//
[log4net.NDC]::Pop()

In this sample, that would put the phrase 'test-getrfcaction-account' will appear instead of the 'null' text that appears there now.  If you push multiple values onto the stack, then as you might expect, the appended stack message will appear in those log statements, consisting of all values that have been pushed but not yet popped.  Hey, it's a stack.  You can appreciate that if you have several script file or several functions running, you might want to identify where the log entry is sourced from.

For my purposes, the developers were already using log4net in their application so using log4net in my test scripts seemed like a reasonable decision.  I do find the single edits handy - I can exclude the informational log entries by changing the Appender.Threshold property to "Error".  I change the log level alot because I'm exploring a bit with the PowerShell test scripts, so until I know for certain how the code I'm testing behaves, I keep it verbose. Last thing I do before adding the test scripts to the regression suite is change the log level to "Error".

So the happy conclusion is -

• add log statements throughout your script
• show your work by changing the log level threshold to "Info"
• hide your work by changing the log level threshold to "Error"
• change what shows up in each log entry by editing a pattern string
• provide a basic version of your custom script call stack using the built-in logging stack (called the NDC)

Hope this helps!

Lately testing people have been talking about testing heuristics - tips, techniques, tactics, strategies for testing something.  I have one that I call MTT.

M - Mission
T - Targets
T - Tests

But I remember it as "mean time to testing".  Exploratory testing excels in part because the mean time to testing is small compared to (caution: wiki-speak ahead), say, BigUpFrontTestManagement.  As an example of minimizing the mean time to testing, the example below uses PowerShell to explore the configuration and functionality of SQL Server Reporting Services 2005.

Mission

Encourage the team to build consensus on why testing will be conducted by paying attention to the project context - who tests and what their background is, what is the system under test, who supports the test environment, what are the communication patterns between testers and developers, ... post the essence of this as a 'test mission' in the team's work area and let them mark it up as the project evolves.  The more people that are involved in the test effort, the more useful a 'testing working agreement' becomes.  Testing faster requires that all the testers understand why they are testing so that they get to the point.

Targets

If the mission indicates why we are testing, then the targets drill deeper and identify what will be tested. I see a lot of testing without clearly-defined test targets.  Some heuristics are test targets, and that helps a lot.  Exploratory testing, even when automated, fits really well with test targets because listing them gives you a testing backlog, making citing progress and indicating expected completion dates simpler using something like testing burndown charts.

Consider automated exploratory testing with PowerShell - let's use the Microsoft Reporting Services installation test as an example.  To use MTT heuristic and the mission stated above, the second step would be to create a testing backlog of the targets of the tests - in other words, what exactly would you want to test?  As with all things agile, this list doesn't have to be complete because you have every right to revise this backlog and prioritize it as necessary, within the context of the test mission.  It is important to avoid listing how these targets might be tested - that's left to exploration.

Microsoft Reporting Services Installation Test - Targets (Testing Backlog)

ReportServer - Service Control
ReportServer - Service Properties
Database - Reporting Services Db - Topology
Database - Reporting Services Db - Physical Configuration
Database - Reporting Services Db - Authentication
Database - Reporting Services Db - Authorization
Operations - Performance Monitors - Availability
Operations - Microsoft Operations Manager - Access
Operations - Microsoft Operations Manager - Alerts
Web Service - Availability
Web Service - Authentication
Web Service - Authorization
Web Service - Functional - Run Report
Report Manager - Availability
Report Manager - Authentication
Report Manager - Authorization
Report Manager - Functional - Reports
Report Manager - Functional - Data Sources
Report Manager - Capacity - Concurrency
Report Manager - Capacity - Reponse Time

Given this list of test targets, an exploratory test effort would start by estimating each one of the targets to create a testing burndown chart, and then diving in according to the priorities assigned in the backlog.  PowerShell is incredibly useful for this given the 'get-member' cmdlet.

Tests

Continuing the Reporting Services example, the first thing that a tiny bit of investigating reveals is that SQL Server Reporting Services has a WMI class that exposes the configuration.  Let's get that class and see what we can do with it.

$targetcomputer = "localhost"
$rs = get-wmiobject -computer $targetcomputer -namespace `
    "root\Microsoft\SqlServer\ReportServer\v9\Admin" MSReportServer_ConfigurationSetting
$rs | get-member

From an exploratory tester's perspective, this list is super useful.  Now you can explore the configuration settings interactively using the $rs variable (they are all listed as Property in the above listing), or create a test script and use data-driven testing to check all the Reporting Services configuration parameters against expected values.  The 'get-history' cmdlet is also useful to retrieve and save the commands that you have experimented with in a format that you can use later for building a script.

To do the data-driven testing in Excel, create a table with three columns and all the properties that you want to test for.  Name the range TestRsConfig and save the worksheet.

Next create a script that includes a function called TestRsConfig (the same name as the named range above) as follows (this script uses the PSExpect testing library for PowerShell):

set-psdebug -strict -trace 0
$targetcomputer = "localhost"
$rs = get-wmiobject -computer $targetcomputer `
    -namespace "root\Microsoft\SqlServer\ReportServer\v9\Admin" `
    MSReportServer_ConfigurationSetting

# Exercises the target of the test - the 
# and responds with a pre-defined set of responses that were
# on the worksheet as the expected results
function TestRsConfig()
{
    param([string]$TestCase, [string]$RsConfigProperty, [string]$ExpectedValue)

    $ActualValue = $rs.PSBase.GetPropertyValue($RsConfigProperty).ToString()
    if ([string]::IsNullOrEmpty($ActualValue)) {$ActualValue = $null}

    if ($ExpectedValue.Trim() -eq "N/C")
    {
        AssertNull $ActualValue -Label $TestCase -Intention $Intention.ShouldPass    
    }
    else 
    {
        AssertEqual $ExpectedValue $ActualValue -Label $TestCase -Intention $Intention.ShouldPass
    }

    RaiseAssertions
}

# run the function library that contains the PowerShell Testing Functions
# the functions are defined as global so you don't need to use dot sourcing
if (!(Test-Path variable:_XLLIB)) { ..\src\DataLib.ps1 }
if (!(Test-Path variable:_TESTLIB)) { ..\src\TestLib.ps1 }

# Configuration and installation testing ...
# Confirm the Reporting Services configuration matches expectations documented in 
# the Excel workbook 'TestRsConfig.xslx'
write-host "Configuration testing ..."
$FieldNames = ("TestCase", "RsConfigProperty", "ExpectedValue")
start-test ((get-location).ToString() + "\TestRsConfig.xlsx") "Sheet1" `
    "TestRsConfig" $FieldNames

The line of note contains '$rs.PSBase.GetPropertyValue' - a PowerShell base method that is proving to be really useful in writing test scripts without hard-coding the property names. In my script, I've used 'N/C' to identify Reporting Services properties that are not configured at all on my laptop.  If I had left them out and therefore avoided the if-then-else code section, the script would only have been two lines.  (The start-test function from PSExpect runs the test and matches up the named range with the test function). Not bad!

But that's only part of the picture - the configuration of Reporting Services.  The other part of the picture is basic functional testing.  Reporting Services exposes its functionality as the Report Manager, a web application but also as Report Server, a web service.  So - what if we used PowerShell to explore and test the Reporting Services web service?  Using the same technique as was used with the Reporting Services WMI class, we can use get-member to help with the exploration.  This doesn't mean that we don't have to test the Report Manager, but it does mean that we can get a quick check of the functionality without a manual test.

This requires a small bit of .NET working knowledge regarding web services.  The trick is to use the .NET SDK 'wsdl.exe' utility to generate a proxy class for the web service, and then wrap that proxy class into an assembly that PowerShell can access:

wsdl http://localhost/ReportServer/ReportService2005.asmx?wsdl 
csc /target:library ReportService2005.cs

Now load this library into PowerShell and use get-member again to investigate the methods that are available:

[void][Reflection.Assembly]::LoadFile("C:\...\ReportingService2005.dll")

# Instantiate the proxy object for the web service
$repsvc = new-object ReportingService2005
$repsvc.Credentials = [System.Net.CredentialCache]::DefaultCredentials
$rs | gm

(I won't list all the methods here because there are many). 

...
CancelAsync
CancelBatch
CancelBatchAsync
CancelJob
CancelJobAsync
CreateBatch
CreateBatchAsync
CreateDataDrivenSubscription
CreateDataDrivenSubscriptionAsync
CreateDataSource
CreateDataSourceAsync
CreateFolder
CreateFolderAsync
CreateLinkedReport
CreateLinkedReportAsync
CreateModel
CreateModelAsync
CreateObjRef
CreateReport
CreateReportAsync
CreateReportHistorySnapshot
CreateReportHistorySnapshotAsync
CreateResource
CreateResourceAsync
CreateRole
CreateRoleAsync
CreateSchedule
...

The (almost complete) example included in the PSExpect download uses the ListChildren, CreateDataSource, and CreateReport methods of the web service to check the functionality of Reporting Services.  The test script uses functions that wrap the calls to the web service in the name of making the test easier to read for testers not familiar with the Reporting Services web service interface.  This is not the complete test - ideally there would also be a 'render-report' and 'test-report' set of functions provided for the testers so that actually generating a report is also part of the test (work in progress).

write-host "Functional testing ..."
$DsBefore = get-datasourcecount
new-datasource -Name "FromPsScript"
test-datasourcecount -Expected ($DsBefore + 1) -Label "Datasources.Count.AfterNew"
remove-datasource -Name "FromPsScript"
test-datasourcecount -Expected ($DsBefore) -Label "Datasources.Count.AfterRemove"

# Verify the report create/delete functionality
$RsBefore = get-reportcount
new-report -Name "ReportFromPsScript"
test-reportcount -Expected ($RsBefore + 1) -Label "Reports.Count.AfterNew"
remove-report -Name "ReportFromPsScript"
test-reportcount -Expected $RsBefore -Label "Reports.Count.AfterRemove"

The example is available for download in the latest source code version of PSExpect.  All together, this wasn't very much code.  The 'mean-time-to-testing' is low considering the automation - and I have an artifact that other people can use for running the tests at any time.

RsLib.ps1 - Contains the wrapper functions new-datasource, test-datasourcecount, remove-datasource, etc.
Test-RsConfig.ps1 - the test script for both configuration and functional tests described above
Test-RsConfig.xslx - Excel spreadsheet containing the named range for properties and expected values used in configuration testing.

Summary

The 'get-member' cmdlet is extremely useful for exploratory testing, especially when combined with the easy access to WMI classes and to web services in PowewrShell.  In the example, I've used the SQL Server Reporting Services WMI class to check the configuration and installation against expected values, and then used a proxy class around the Reporting Services web service to check basic functionality.  What 'get-member' and PowerShell do for us in these cases is give us the tools and the means for exploring the targets of our tests.  Oh yeah - everything is in a script that we can share and evolve as our understanding of the test target increases.

It's been pretty well-documented that test automation, especially at the higher test levels such as system or acceptance testing, is something you should undertake carefully.  Like any other development project, the economic argument has to be there to support the effort - and that economic model can be screwed up pretty easily by analysis-paralysis (planning tests without ever writing or running one) or by creating unmaintainable, brittle test scripts.

There are ways of lowering the cost of automating tests.  There is a PowerShell hook, honest.

Increase the 'Test Cases to Test Script' Ratio

One of the reasons that FIT is spectacular is that it standardizes the test script and permits the tester to focus on providing examples in the form of combinations of test input and expected output.  It does this through the use of test fixtures like the ColumnFixture.  To use ColumnFixture, you design your test by creating a table containing all the test data (inputs and expected outputs) and then write a test fixture that translates a row in the table into a call to the system under test (SUT).  Each row in the table represents a test case - and the running of each test case is all done implicitly by FIT and your test fixture.

It's pure-and-simple data-driven testing.  You achieve a high 'test case to test script' ratio by virtue of never having to write the script beyond writing the test fixture.  The framework implicitly provides the script - and runs it.

You can replicate the use of ColumnFixture in PowerShell/PSExpect by storing the data in Excel (and learning what a named range in Excel is).  See \samples\test-getweatherperf.ps1 for an example.  There are several advantages of using Excel for the test data:

1. You can change the scope of a test run by changing the definition of the named range on the worksheet - eliminating the need to comment out parts of an input file or worse yet, storing multiple versions of it.  
2. You can use formulas to help specify the test data - so some of the test cases can use randomly-generated data instead of the same data every time.  
3. You can ask business users or other domain experts to create or revise the worksheet - Excel is ubiquitous in corporations.  Little or no training required.

But I've written about PowerShell/PSExpect and Excel before.  On to other interesting methods of minimizing test automation costs.

Eliminate Waste - Part 1 - Use Test Targets Wisely (Lean Test Management)

'Waste' in testing comes in two forms: writing test scripts that target the wrong thing, and writing scripts in a form that makes them difficult to evolve as the system under test evolves, or as your understanding of the system under test evolves.

You can avoid testing the wrong thing by first building a list of 'test targets'.  A test target is basically anything that needs testing.  Application functionality, security, performance, etc. might be types of tests but they are not test targets.  Well at least they aren't specific enough to be used wisely.

Break out the types of tests to get specific test targets.  List the application functions (use cases, user stories, whatever) - those are functional test targets.  List the application interfaces - those are integration test targets.  Security test targets include authentication, authorization, and then any specific vulnerability you want to test for (sql command injection, os command injection, etc.).  Performance test targets will be objectives-based like throughput, resource consumption limits, or user concurrency.

You need this list so that you can be wise about setting priorities and so that you can plan to write scripts for only those test targets that really matter.  Host a workshop to decide these priorities so that the whole team is on the same page.

The list of test targets is - dare I say it - a testing backlog.  If you're familiar with Scrum and the use of a product backlog, then take those same concepts and apply them to testing and the testing backlog.  This process works well - I've used testing backlogs and Scrum concepts now on a large enterprise project (an SAP upgrade),