I use a fictional Volunteer Coordination System (VCS) in much of my work as an example.  It’s a small application based on the need to coordinate volunteers at conferences.  I was a volunteer coordinator for some large conferences, and while (I think) I was successful, it was a lot of manual work.

Hence the idea to provide some assistance with an application.  The goal of the application is to help volunteer coordinators to ensure adequate coverage of all the various requirements the conference managers have, and to help volunteers optimize their conference experience, between balancing their work requirements and their desire to attend sessions. 

The Short Version

Building a graphical user interface is more complex than building a PowerShell cmdlet.  If you build the System Administrator and User Administrator tasks as cmdlets, you save the effort required to build and test graphical user interfaces for those tasks.  Your PowerShell-friendly system administrators will love you for considering their perspective, and you’ve delivered faster because of the reduced effort on the project.

How do I know cmdlets are less expensive than graphical user interfaces?  You save time by avoiding most (but not all) user interaction design effort, and by reducing the test effort.  If you write your cmdlets using a test-driven development style, then you can promote all those automated tests that helped you work out the cmdlet interface as regression tests with minimal effort.  To do the same thing with a graphical user interface, you will need at least another tool to help with the automation, the tests will be harder to maintain, and on top of that, you will probably still need the expertise of an exploratory tester to give you feedback on the user experience.

How many functions could I divert to cmdlets? This will depend on the application target domain.  In the example below, 9 of 28 user tasks might be candidates for building as cmdlets.  This could mean that you’ve saved the effort required to build 9 screens, or more, depending on the desired user interaction. Seems worthy from my perspective.

What if I built cmdlets for all the functionality and then layered the graphical user interface on top of them as needed?  By Jove, I think you get the point.  That’s a great idea.  You’ve saved testing time since you can use simple PowerShell scripts to test all the domain functionality without the GUI (bypass testing).  This reduces the scope of the testing that occurs outside the GUI and helps you to respond to changes later on in the project.  And, it leaves you room to adapt to the user’s needs, just in case they decide later they need a GUI for one or more of their tasks.  In the meantime, you’ve deferred the development of that GUI to as late as is responsible (ideally).

The Long Version

Consider the functionality of this application based on a breakdown by user type (or persona).  Some of these have some business rules behind them, some of them are aspects of CRUD (create, read, update, delete).  Some of the functionality is required before the conference, some of it is required after the conference.

Conference Manager, focused more on the conference program than on logistics

• Add Conference
• Remove Conference
• Add Conference Session
• Remove Conference Session
• Set Volunteer Work Requirement

Volunteer Coordinator, focused more on conference logistics than on the program

• Add Volunteer Role
• Remove Volunteer Role
• Screen Applicant
• Hire Volunteer
• Drop Volunteer
• Add Work Shift
• Remove Work Shift
• Check Coverage
• Assess Volunteers Against Volunteer Work Requirement
• Approve Work Shift Trade
• Notify Volunteer
• Broadcast Notify Volunteers

Applicant, will work for conference fee waiver

• Apply for Volunteer Position
• Accept Volunteer Position Offer

Volunteer, wants to maximize conference experience

• View Volunteer Role Description
• Sign Up for Work Shift
• Block off Time to Attend Conference Session
• Initiate Work Shift Trade
• Accept Work Shift Trade
• Decline Work Shift After Having Signed Up
• View Personal Conference Schedule
• Assess Self Against Volunteer Work Requirement
• Notify Volunteer Coordinator

There might be more.  As a scope definition, that’s plenty of detail.  Certainly more than the goal description, so it’s a start.  Strong verbs, strong nouns, makes for a reasonable model of the desired functionality.  There’s some ambiguity since the business rules are not modeled, however, for the most part, it seems clear what the intended functionality is.

Solution Concept #1: All functionality is exposed via the web browser.

To build this solution, you first do a couple of architectural spikes to set the high-level design, and then, armed with the feedback you got from those spikes, you plan a number of releases and iterations that ultimately deliver the desired functionality.  For each of the above, the tasks would be something like, ordered in any which way you feel comfortable, but in any case, refactoring as you go:

• Elaborate requirements by identifying acceptance tests
• Create the user interface storyboard (say using balsamiq)
• Verify and validate the user interface storyboard against usability, branding objectives
• Automate acceptance tests (excluding GUI)
• Automate selected acceptance tests for regression suite (including GUI)
• Build the presentation layer (user interface objects and tests)
• Build the business services layer (process and control objects, entities, and tests)
• Build the data services layer (database and tests)
• Verify and validate the functionality (exploratory testing using the GUI)

Of course, these are arbitrary.  Maybe your list is longer.  I’m also excluding the logistics around source code management and version control.

Solution Concept #2: User administration functionality is exposed via PowerShell cmdlets, remainder exposed via web browser

To build this solution, you would first make a decision whether or not the user task would be considered “administration”.  To make this decision, the greatest factors are the user persona, the frequency and timing of the usage of the functionality, and how well the functionality maps to a graphical user interface.

I posit that all the Conference Manager tasks are administrative.  They won’t be run very often, do not require a lot of information to be typed in, and are performed by a role that aligns pretty well with my definition of “administrator”.  That saves 5 web forms.

I posit that some of the Volunteer Coordinator tasks are administrative, and I would break it down based on when the task is most likely to be needed.  Everything that is typically run before the conference is administrative, everything typically run during the conference  is not.  That saves 4 web forms.

For the tasks that are exposed by graphical user interface, the development tasks from solution concept #1 would apply.  For those based on cmdlets, the development tasks are as follows:

• Elaborate the requirements by identifying acceptance tests
• Automate the acceptance tests by writing PowerShell scripts
• Build the presentation layer (the command interface and snap-in)
• Build the business services layer (process and control objects, entities, and tests)
• Build the data services layer (database and tests)
• Verify and validate the functionality (exploratory testing using the PowerShell console)

This feels like a simpler list of development tasks.  Depending on the target architecture, this could also mean one less programming language and one less third party library (for example, no javascript and no jQuery) for these functions.

“The way you explore complex ecosystems is you just try lots and lots and lots of things, and you hope that everybody who fails fails informatively so that you can at least find a skull on a pikestaff near where you're going.”
– Clay Shirky, http://www.shirky.com/herecomeseverybody/2008/04/looking-for-the-mouse.html

I know that @cshirky wasn’t talking about agile development in that post.  My belief, however, is that software development does qualify as a complex ecosystem and that there is tremendous value in “failing generously” when you do fail.

“Failing generously” has two aspects – the informative aspect highlighted by @cshirky being the public aspect, the one that you might hear in discussions at conferences or read in publications.

“Failing generously” is also a willingness, an openness, an agape of sorts, to avoid blaming the tool, the process, co-workers, the customer, or yourself.  Truth be known, the real reason for the failure is probably a myriad of those things, slap-dashed together with just the right timings put in place for … the failure. The etymology of the failure might be as complex as the ecosystem itself.

Fail generously. Study the context, plant pikestaffs for yourself and for others. Learn from it and understand what you could do better next time. If you simply must blame, then at least be willing to forgive, yourself included.

Writing Powershell scripts is generally a lot of fun, but once in a while you might come across a need to become test-infected, that is, become a little obsessive about how well your script is tested before you distribute it.  There might be several reasons for getting infected: you’ve been burned, you have to distribute your scripts to people that don’t have quick and easy access to you, or the intent of the script has rather serious overtones and/or consequences should something go wrong.

Not to say there is anything wrong with _always_ being test-infected.  But sometimes you just want to have fun with the script and jam it out there for other Powershell hacks like me to have a look at.  There is real value in that.

Caveats aside, let’s look at test-infecting an existing script.  Steve Murawski gleefully (via twitter) offered his script Show-ADObject.ps1 as lab rat for my intended infection.  Here’s what happened.

My stated starting point was the existing Show-ADObject script.  To infect the script, the first step I undertook was to create a test script that describes the intended behaviour.  The second step was to refactor the target script until such a time that a) all test cases are automated and b) all test cases pass.

After running Steve’s script, I came up with seven behaviour-defining test cases.  I chose to describe each one of those test cases using the given-when-then phrasing from behaviour-driven development (BDD) since the appropriate aliases are built into the PSExpect testing framework.

Steve’s code had some dependencies that I have inoculated against infection through isolation.  In other words, I made them out of scope of my tests.  In particular, the graphing routine and the graphing class library that was used.  Consequently none of the following defining behaviours refer to the actual visual graph itself – only the items that are being fed into that graphing routine.

Written this way, the tests should read as naturally as plain text.  When you say each test, don’t skip to the text in quotes – read them with the words given-when-then included.

    # first defining test case - collect the nodes for a valid AD class name
    #
    #
    given "an empty list of objects to map and a valid class name 
        that I know has entries"
    when "I request to view the node map for that class"
    then "there should be more than zero nodes to view"

    # second defining test case - collect the nodes for a valid set of AD class names
    #
    #
    given "an empty list of objects to map and a list of valid class names 
        that I know each have entries"
    when "I request to view the node map for those classes"
    then "there should be more nodes than were discovered with 
        just a single class name"

    # third defining test case - display a list of AD class names
    #
    #
    given "an empty list of AD class names"
    when "I request the list of valid AD class names"
    then "there should be more than zero valid AD class names returned"
    then "known class names like group, organizationalunit should be in the list"

    # fourth defining test case - adding color to the nodes
    #
    #
    given "an empty list of objects to map and a list of valid class names 
        that I know each have entries"
    when "I request to view a colorized node map for those classes" 
    then "there should be more than zero nodes to view and filtering only nodes 
        without color should not yield any objects"

    # fifth defining test case - displaying help and usage examples
    #
    #
    when "I request to view the help and usage examples"
    then "the answer should be text and contain key help contents"

    # sixth defining test case - performing the check for required files
    # when all files are in place
    #
    given "a list of files that must be available for the script to perform 
        as expected and that all the files are in place"
    when "I request to check to see if those files exist"
    then "the answer from the check should be true and the missing files report 
        should be empty" 

    # seventh defining test case - performing the check for required files 
    # when all files are _NOT_ in place
    #
    given "a list of files that must be available for the script to perform 
        as expected and that _not_ all the files are in place"
    when "I request to check to see if those files exist"
    then "the answer from the check should be false and the missing files report 
        should contain the number of missing files"

These seven defined behaviours form the backlog of work that I have to do in order to make the infection complete.  The next step is to provide the script blocks for each of the given-when-then phrases, in effect, automating the test case.  It’s best to implement one test case at a time so that you can get to the first passing test quicker – you’ll enjoy the positive feedback you get when you work this way.

Writing these code blocks is a translation exercise – you translate the natural language in text into Powershell script, calling the target script in order to exercise the true target of your test.  In this case, the target of the test is the function Get-ADObjectsToMap.  I’ve refactored its signature based on what the test needed.  Sure, there is a design element to the translation and you might come up with a different signature, but the point is – the function signature needs to support the test.  So the function Get-ADObjectsToMap isn’t allowed to perform any input or output – it must be written to accept input from the test script and then it must return something that the test script can inspect.  This enables the test oracle – the indicator of the pass/fail of the test – to be automated.

The ‘then’ clause has one test condition in it, confirming that at least something came back from the dip into Active Directory. The test condition is the line that starts with ‘Assert’, that is, a call to one of the functions in PSExpect.  I thought about adding another test condition to confirm that a specific item was being returned, but rejected the idea since I thought that was more a test of the Get-QADObject cmdlet than it was of my script.  Borderline.  At least this way, I’ve not injected any test condition that is specific to my Active Directory.

Running the script the first time failed miserably since I hadn’t created the target function with the right signature yet.  So parsing errors everywhere.  Next step: fix the parsing errors so that the script actually runs cleanly, albeit still fails the test.  This is an important step in test-driven development – getting to the first clean fail.  Next step after that, perhaps obviously, is to write the script so that the test passes.

Querying for group
03/01/2009 3:20:29 PM,SHOULDPASS,PASSED,PSpec,an empty list of objects to map and a valid class name that I know has entries
03/01/2009 3:20:29 PM,SHOULDPASS,PASSED,PSpec,I request to view the node map for that class
03/01/2009 3:20:42 PM,SHOULDPASS,PASSED,PSpec,there should be more than zero nodes to view
03/01/2009 3:20:42 PM,SHOULDPASS,PASSED,ObjectsToMap.Single.Count:237

Now you can choose to move on to the next test case, or you can choose to refactor the code written so far in order to improve its quality.  I recommend you refactor immediately so that it doesn’t feel like work later on.  Keep your focus tight.  Refactor to eliminate side effects, improve modularity, improve readability, etc.  Do this as you go instead of waiting until the end.

The process really doesn’t change for the remaining test cases with the exception that, every once in a while, you may have to go back to a previously-passing test and revise it based on what you have learned from getting other test cases to pass.  This was certainly required in this case once it got time to handle the coloring of the nodes that were intended to be graphed.

Download both the test script and the (refactored) target script here:

http://www3.telus.net/~amgeras/samples/Test-ShowADObjectR.ps1

http://www3.telus.net/~amgeras/samples/Show-ADObjectR.ps1

Summary

Steps for test-infecting a script:

1. Write a test script that describes the intended behaviour.  The script should not run since you haven’t written the target script yet.
2. Get the test script running by filling in _just enough_ of the target script so that there are no syntax or parsing errors – the test script should run smoothly, but still fail the test cases.  The PSExpect testing framework was designed to be able to highlight test case failures yet still have the test script run smoothly.
3. Choose a single defining test case – ideally the most significant or intended-behaviour-defining one – and write the target script so that the test case passes.  Avoid over-engineering the target script to handle the other test cases – instead, focus on the one test case that you’ve chosen and get it passing, quickest way possible.
4. Refactor your target script to meet your personal or enterprise standards.  If in Step 3 you’ve got the test case passing in the simplest possible way, Step 4 is to keep the test passing but to improve its quality.
5. Choose the next-best defining test case and repeat Steps 3 and 4.  This time, you will need to ensure that the first test case stays passing.  This might mean refactoring your target script some more in order to handle the new test case.  It may also mean refactoring your first test case.  By the end of this step, though, all the tests you’ve tried to get passing, should be.
6. Rinse. Lather. Repeat.  That is, repeat Step 5 until you’ve run out of behaviour-defining test cases.
7. Refactor.  This time, add some test cases that are only there to confirm the stated quality.  These are tests like how to deal with null inputs, how to deal with exceptions, etc.   These are supporting test cases since they don’t describe core intended behaviour, nonetheless may be necessary to confirm the intended design or quality level.

I can’t remember who got me interested in wordle.net but it struck me that I would love to see all of my Twitter posts as a “Beautiful Word Cloud” (from their site). The Java applet on that site transforms a bunch of text (or a site) into a word cloud with the size of the word being representative of its frequency in the text.  I saw a the post (and source code) from Adam Franco for retrieving Twitter content using PHP but was more interested in a Powershell version, and I didn’t want the posts in XML – that wouldn’t be useful input to Wordle.

Starting from the end … what I wanted was to use either of the following lines in Powershell:

"testfirst" | Export-Tweets | out-file adam.txt

or

Export-Tweets -User "testfirst" | out-file adam.txt

The first of these uses the pipeline to receive the Twitter username, useful if you want to put it into a loop to say, extract the contents of more than one user account.  The second one is a more literal request for the tweets of a single Twitterer.  In both cases, the output is to be saved in a file name of my choice using the handy out-file cmdlet.  In other words, I don’t want a script that hard-codes the output file name.

Recognize that first you have to get the Export-Tweets function into memory, so perhaps a more complete scenario that depicts my intended usage is the following, typed into a Powershell console:

.\Export-Tweets.ps1
"testfirst" | Export-Tweets | out-file adam.txt

The following script accomplishes these modest goals.  There are two Powershell tricks to point out.  The first trick is enabling pipeline input for a function.  You’ll see that in the definition of the function Export-Tweets since it has the begin{}, process{}, and end{} blocks.  The pipeline input is enabled in the process{} block while the explicit syntax with the User parameter is enabled in the end{} block.  The trick is to define the function that does the work in the begin{} block.

The second trick is substituting % for foreach.  This really cuts down on the script length, however, use that substitution with caution if you’re writing scripts that other people have to read.  It’s a useful and learnable enough reduction that I wanted to use it in this case.

Here’s the script:

function global:AppendTweets([xml] $tweetsPage)
{
    $tweetsPage.selectnodes("/statuses/status") | % { $_.selectSingleNode("text").get_InnerXml() | write-output }
}

function global:CountTweets([xml] $tweetsPage)
{
    return $tweetsPage.selectnodes("/statuses/status").Count
}

function global:GetTweetsPage([string] $user, [string] $page)
{
    
    [string]$urlbase="http://twitter.com/statuses/user_timeline/"
    [string]$url=$urlbase + $user + ".xml?page=" + $page
    
    write-host "Connecting to URL " $url

    $webclient= New-Object "System.Net.WebClient"
    [xml] $tweetsPage=$webclient.DownloadString($url)
    return $tweetsPage
}

function global:Export-Tweets([string] $User)
{
    begin
    {
        function GetUserTweets ([string] $User)
        {
            $pageIndex = 1
            $numTweets = 1
            while ($numTweets -ge 1)
            { 
                $tweetsPage = GetTweetsPage $user $pageIndex
                $numTweets = CountTweets $tweetsPage
                if ($numTweets -ge 1)
                {
                    AppendTweets $tweetsPage
                }
                $pageIndex += 1
            }
        }

    }

    process
    {
        if ($_)
        {
            GetUserTweets $_
        }
    }

    end
    {
        if ($User)
        {
            GetUserTweets $User
        }
    }

}

Otherwise known as the functional testing tools workshop, this workshop will be held in advance of the Agile 2008 conference in Toronto this coming August.

My goals/intentions for a next-generation functional testing tool are simple:

• test-first support
• test-last support
• exploratory testing support
• one environment for all three of the above

In other words, I want it all.  Because I believe that testers need to have all styles/techniques in their personal portfolio, and I don't want them to have to switch between tools when they choose the style that best fits their current test mission.

So what does that mean?

Test-first support means elaborating requirements and designs using tests.  That may or may not be a specification, that's a semantic I'm choosing not to care about.  I just want to elaborate the requirements using examples and I want to run those examples and fulfill that happy test-driven development-with-customer-tests cycle.  Ditto for designs.  An environment that supports this would allow me to craft these examples in the absence of the system and it needs to do that because I'm not describing a system (yet).  I'm describing a domain.  I envision doing things like registering verbs, registering nouns and then constructing examples using verb-noun combinations of those registered verbs and nouns in specific sequences.  The nouns have state-like attributes so that I can describe things like 'good customer' and 'bad customer' without a ton of syntax.  Domain-specific, yes.  Language? Script? Don't know.  I know it's not XML because 'flow' doesn't just appear out of an XML document, and it's important that sequence and control does come easily when reading whatever artifact I'm describing.

Test-last support means verifying and validating a system under test, again using examples.  I may choose to run these as manual or automated tests.  Same thing - register verbs, nouns, build tests from verb-noun combinations in sequence.  With states.  You record-playback-refactor within the context of a single verb-noun combination; building larger scripts later, and only once those verb-noun combinations are registered.  In the age of Supervising Controller or Model-View-Presenter or Model-View-Controller, I believe these verb-noun combinations should work inside or outside the presentation layer; so back-end business process testing can be done at a different time than the front-end human-computer interaction testing.  But the scenarios we use at the back-end can in fact be re-run to include the front-end when the time is right.

Exploratory testing support means interacting with the system to explore its functionality, maintaining state as I go, providing me with logs and the ability to adjust that system state as I go.  I can adapt those logs for auditors to see later.  I can share the logs with other testers, and they are more productive because of that.  In other words, I can share what I have learned about the domain and the system with others on the team.  I would expect that again, the common verbs, nouns, and states that have been registered form the backdrop for this.

And yes, I want to switch between any of test-first, test-last, and exploratory testing, even within a single project.  Because all requirements are not equal, and the micro-context for any given requirement might just implicate one testing style over the other.

Maybe this is too abstract, but we have to start somewhere.  I'd like to think that we start by confirming support for all schools of testing, and to quote Jerry Weinberg at CAST 2008 - "Business is too complicated for any one school of testing.  Learn from them all.  Taste everything, only swallow what you want/need."

Like many of you, I noticed the download that was made available on the MSDN Tester Centre the other day - Testing Applications with Powershell - Sample Code.  This link, along with downloads and article such as the UI Automation Specification and associated Community Promise gives me some hope that it might be possible to run the same test script both with and without the UI.

That's my hope for testing tools.  Oh yeah, and that we can do that using business-oriented grammar in the scripts.  And that we can define, scope, and reuse elements of that grammar in exploratory testing, semi-automated style.

Anyway, baby steps ... baby steps.

The post was great in that it raised the awareness of how Powershell might be used for testing.  All of my examples in PSExpect are against libraries or other scripts.  This example automates the UI.  And that's cool if not entirely necessary in all contexts.

I replaced the verdicts within the posted sample code with calls to PSExpect Assert* functions. There were two of them, depicted below with the original code hashed (commented) out:

$result = get-listBox $lb "222"
#    if ($result -ge 0) {
#      write-host "Found '222' in ListBox!"
#    }
#    else {
#      write-host "Did NOT find '222' in ListBox"
#      $pass = $false
#    }
AssertEqual -Expected 0 -Actual $result -Label "TC-1" -Intent $Intention.ShouldPass
...

$text = get-textBox $tb
#    if ($text -eq "ad") {
#      write-host "Found 'ad' in TextBox!"
#    }
#    else {
#      write-host "Did NOT find 'ad' in TextBox"
#      $pass = $false
#    }
AssertEqual -Expected "ad" -Actual $text -Label "TC-2" -Intent $Intention.ShouldPass
...

and then there is the way that the test script reports back to the tester about the status of the test run:

#    if ($pass) {
#      write-host "`nTest scenario result = Pass" -foregroundcolor green
#    }
#    else {
#      write-host "`nTest scenario result = * FAIL *" -foregroundcolor red
#    }
RaiseAssertions

Running the revised test script and running it against the sample application to be tested generates the following two lines of output:

02/06/2008 12:39:26 PM,SHOULDPASS,PASSED,TC-1
02/06/2008 12:39:28 PM,SHOULDPASS,PASSED,TC-2

Since the tests pass, they show up as green.  You can find the entire script at(http://www3.telus.net/~amgeras/samples/Test-TheApp.ps1).  You will find that the script requires the PSExpect library in a specific location - just change that location to get the script to run (one line to edit).

We did drop a number of lines of script completing this refactoring. We also gained functionality since when you run the script, you are also generating a log file in behind the scenes that could be posted in the test results repository.  That's all good.  But there are other refactorings that we might consider that I will tackle when I get a moment.

First, from a testing perspective, the functions get-listBox is counter-intuitive since it returns an integer when given an item that it is supposed to contain.  I understand why, but the consequences to the test script are obvious when you compare the two AssertEqual statements above.

In the first AssertEqual, the expected result is '0' yet we are really looking for the value '222'.  In the second AssertEqual, we are looking for 'ad' and that is exactly what we truly are looking for.  That's more intuitive.  So in the next refactoring I would remove the comparator in the function and leave that comparison for the tester.

Second, and this only reflects my bias towards domain-driven design thinking, is that the sample application mimics some function that doesn't come through in the test script unless you look really close.   It gets the point across for people like me to talk around, so it was fit for purpose. 

But ...

If the screen functionality could be represented using a verb-noun naming convention, then that would provide the means for separating the test runner from the test actions from the set of verdicts and that would make it even more meaningful, and the test script more readable.

For example,

start-productSearch -Category $null -Item $null
test-productSearchResults -Expected 'Error'

start-productSearch -Category 'Product Name' -Item 'ad'
test-productSearchParam -Expected 'ad'
test-productSearchResults -Expected '222'

Now, the test runner is still the Powershell script (as it was before we did any refactoring).  The screen action is start-productSearch.  And the verdicts are test-productSearchParam and test-productSearchResults, both implemented in the two AssertEqual statements above but whose meanings were hidden in the interpretation of the script.

And this test script is now much more reflective of what the screen does and how we're testing it.  It communicates better to the testers, and we have this function start-productSearch that we could reuse in other , longer-running scenarios of the larger application.

Right now, these last two refactorings are musings, but I'll post them and my findings along the way as soon as I get a chance.

Testability is a software quality attribute that is a measure of how easy (or difficult) it is to create test cases for a test target and, in combination, how easy (or difficult) it is to run those tests.  High testability is a good thing since it means that the total cost of making changes to the software is correspondingly lower.  It typically also means that you can have more than one individual contributing to the software since it is easy for them to determine whether or not their contribution has been a net positive or net negative contribution, simply by running the tests again.  Hence, we like test automation.

It turns out that when we value testability, we tend to write software that enables us to automate more types of tests, and we set up this fantastic feedback loop where defining a test becomes a means of production - the tests specify the desired behaviour, guide the development, and then verify the as-built software for as long as that behaviour remains desired.  Pretty nice, when we get to that level.

Tools like xUnit were revolutionary in the sense that they gave developers the means for valuing testability and a mechanism for expressing the low-level behaviour in the same language that our target software was (or would be) written in.  Developers liked it, and now teams in general are much more inclined to automate unit tests than ever before.

But is it enough?  That's a question that Brian Marick has been pondering publicly in his blog (it's one of those blogs that actually expands your mind when you read it) and in an on-line forum discussing automated functional testing.  I was struck by the phrase "Perhaps it would be better to invest heavily in unprecedented amounts of built-in support for manual exploratory testing." the most.  An alternative, he indicates, to automated functional testing with tools like FIT/FITNesse (read the blog entry to find out why this isn't agile heresy).

You can see where I'm going. If you have a set of Powershell cmdlets that exposes an application's functionality - and if the cmdlets expose ALL of the functionality - then 'exploring' means working with those cmdlets (or an API) in a Powershell session and using whatever testing heuristics you might have in your tester's toolkit to test the application.  You may not be using the same UI as the end-users, however, there is great value to first confirming the functionality is good enough, and then second moving on to confirm that the UI is good enough.

The Powershell features that make exploratory testing an excellent experience are (but not limited to):

• Objects on the pipeline, and in fact, having objects to work with at all.
• The get-help cmdlet that can access custom help files for custom cmdlets.
• The get-member (gm) cmdlet.
• The get-history (h) cmdlet.
• The naming convention for cmdlets.

You don't even really have to know the cmdlets ahead of time because of those features since you can retrieve their calling syntax at run-time.  The get-history cmdlet can then be used to save useful cmdlets and sequences of calls, in order to support future semi-automated exploratory testing (where some scripts are run, but only for the sake of returning to a known state) or ultimately fully automated testing for something like detecting regression.  Seems that the ability of a script for passing on knowledge isn't limited to administrators ...

I can envision, however, a mismatch between the cmdlets that an application designer foresees might be a useful way of exposing the functionality and the cmdlets that a tester might find useful.  They will each bring, I believe, a different perspective.

So it might be wise to undertake this 'unprecedented support for exploratory testing' or hyper-testability using a test-driven approach where the functionality of those cmdlets is influenced, or possibly specified, by the examples that a tester might deem appropriate.  Supporting manual exploratory testing by writing automated test scripts ahead of time ... seems contradictory. But there is a distinction between 'understanding-clarifying tests' (Brian Marick again) and 'verification and validation' tests.  So using a test-driven approach for understanding and clarifying requirements, and ultimately the design of the cmdlets, isn't the same as writing tests for verifying and validating the resulting application.

The bottom line?  Exposing application functionality in Powershell makes your application hyper-testable.  Use test scripts that you/your testers write ahead of time as a means of designing the cmdlets and separate the concern of testing the functionality of the application from the concern of testing the user interface.

I know we're not supposed to come up with solutions before fully understanding the problem, but I've got just enough experience and wide-eyed insensibility to believe that I've seen something that just might work: testing mashups.

Microsoft Popfly is a set of tools for creating web pages and the like from pre-defined blocks.  The one aspect that I am particularly intrigued by is called a mashup.  You use a tool called the Mashup Creator to build a web page comprised of blocks that come from disparate sources.  For example, build a page that lists your Facebook friends' statuses and your Twitter friends' most recent updates.  I'm assuming it's called a 'mashup' after the term that's been coined for mixing songs into single tracks (some of my favourites have sampled from 5 or more songs).

Now imagine if the "blocks" used to build a mashup were Powershell cmdlets instead of web applications and that those cmdlets were designed to expose the functionality of an application under test.  Each "mashup" you create is therefore another application scenario.  Imagine that some of the blocks you use on a mashup are data sources so that you could make the scenario data-driven.

So, why do I think this might be a good idea?

First, it may help non-programmers to develop automated test scenarios that are executable.  Second, the economics of building cmdlets for the application just changed: not only might the cmdlets be useful for system administration, but now they will also be useful in writing automated tests for the application.

Why cmdlets? Well they exhibit all the behaviour that I would want from blocks for building tests from - they have a naming convention, there are behavioural standards that they adhere to, and there is a mechanism already in place for stringing them together and running them as one scenario, so they're not just commands that can be invoked.  If the 'noun' part of the cmdlet always refers to something from the business domain, then we're working at the same level of abstraction as (hopefully) the requirements are written - so as testers we're grounded, too.  There is a tie-in to the whole world of domain-specific language and domain-driven development that just can't be ignored.

Now these aren't the only tests that would need to be considered for any system under development.  A test strategy for any software product is going to include extensive developer (unit and low-level integration) testing and manual functional and acceptance testing. I get that.   At the same time, automating more test scenarios means they are shareable and immediately executable by anyone on the team, enhancing communications and reducing communication overhead.  That's a good thing, and worth striving for.

or, A Testament to the Flexibility of Powershell

The greatest impact that the Functional Testing Tools Workshop in Portland (October, 2007) had on me was to raise my awareness of behaviour-driven development, in particular, the 'given-when-then' pattern for structuring examples.

I will defer to other blogs on describing given-when-then in detail, you'll see what it means soon enough in the examples below.  For now, think of given-when-then as a means of translating the meaning of a user story into a related set of acceptance criteria using examples.  Something that has always been a bit awkward for business users and developers alike.

Given you adopt the story-telling convention popularized by Mike Cohn (I think),

As a <blank>
I want to <blank>
So that <blank>

when you want to translate the story into a set of related acceptance criteria using examples,
then use the following convention popularized by BDD proponents such as Dan North:

Given <a known set of conditions or state>
When <something happens>
Then <check for an expected set of conditions or state>

There are tools that help you do this and to make statements worded like this executable - in that case, they almost become executable requirements.  They are certainly executable examples.  The most-often-used example is for a simple account manager class (I've seen this example on a number of blogs but didn't pay attention to where it originated, but I think it was Dan North's again).  I'm not the only person looking at this - there is a NBehave project on CodePlex that provides a mechanism for writing tests in this manner.  And there are many ruby examples in both rspec and rbehave.

# Story: Transfer to cash account from savings account
# As a savings account holder
# I want to transfer money from my savings account
# So that I can get cash easily from an ATM

# Scenario 1: Savings account is in credit
given "My savings account balance is 100 and my cash account balance is 10"
when "I transfer 20 to my cash account from savings"
then "my cash account balance should be 30 and my savings account balance should be 80"

given "My savings account balance is 400 and my cash account balance is 100"
when "I transfer 100 to my cash account"
then "My cash account balance should be 200 and my savings account balance should be 300"

My opinion is that this is a clear way of expressing an example, albeit all that I have really seen are simplistic ones.  I'm going to enjoy working through more complex examples on an upcoming project.

Now on to Powershell.  It was always my intention to introduce some sort of behaviour-driven development helper functions in PSExpect, but I was never certain of what that could be.  Until I went to the Functional Testing Tools conference in Portland.  I love it when I can take some time off to think, and be inspired by a group of brilliant people.

Turns out we can make those examples above executable by writing a function and adding a set of aliases to our Powershell environment.  The function accepts a string parameter and a script block parameter.  All that it does is write out the string to a log file and then run the script block.  I've called it Invoke-Block for lack of imagination, and it uses the PSExpect logging mechanism so that the same log file shows the results of assertions.

function global:Invoke-Block([string]$Label, [ScriptBlock]$Script)
{

    # build the log entry and write it out
    $entry = BuildLogEntry $Intention.ShouldPass $ResultPrefix.Passed "PSpec" $Label
    $added = $Assertions.Add($entry)
    if ($LogFileName -ne $null) {
        $logEntry = BuildMessage $entry
        $logEntry | out-file $LogFileName -append -width $LogFileLineLength
    }
   
    # now run the script block
    if ($Script)
    {
        &$Script
    }
}

set-alias given invoke-block
set-alias when invoke-block
set-alias then invoke-block

The aliases enable using the given-when-then triad in a Powershell script without generating any errors.  This means that the executable version of the example is the same as above, with the script blocks added in.  The example relies on a class library from the system under test (SUT) that the script loads using reflection (not shown) but the calling syntax to that library is hidden using a set of Powershell functions that utilize a verb-noun naming convention: create-account, transfer-money, set-accountbalance, and test-accountbalance (these are elements in the grammar of the domain-specific language, or ubiquitous language for this particular system under development).

# Story: Transfer to cash account from savings account
# As a savings account holder
# I want to transfer money from my savings account
# So that I can get cash easily from an ATM

# Scenario 1: Savings account is in credit
given "My savings account balance is 100 and my cash account balance is 10" {
    set-variable -name SvAccount -value (create-account "Savings" 100) -scope script
    set-variable -name csAccount -value (create-account "Cash" 10) -scope script
}

when "I transfer 20 to my cash account from savings" {
    transfer-money -From $SvAccount -To $CsAccount -Amount 20
}

then "my cash account balance should be 30 and my savings account balance should be 80" {
    test-accountbalance -Account $SvAccount -Expected 80 -Label "Savings.Balance.After"
    test-accountbalance -Account $CsAccount -Expected 30 -Label "Cash.Balance.After"
}

given "My savings account balance is 400 and my cash account balance is 100" {
    set-accountbalance -Account $SvAccount -Amount 400
    set-accountbalance -Account $CsAccount -Amount 100
}

when "I transfer 100 to my cash account" {
    transfer-money -From $SvAccount -To $CsAccount -Amount 100
}

then "My cash account balance should be 200 and my savings account balance should be 300" {
    test-accountbalance -Account $SvAccount -Expected 300 -Label "Savings.Balance.After"
    test-accountbalance -Account $CsAccount -Expected 200 -Label "Cash.Balance.After"
}

RaiseAssertions

The Output

29/11/2007 1:56:13 PM,SHOULDPASS,PASSED,PSpec,My savings account balance is 100 and my cash account balance is 10
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,I transfer 20 to my cash account from savings
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,my cash account balance should be 30 and my savings account balance should be 80
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Savings.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Cash.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,My savings account balance is 400 and my cash account balance is 100
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,I transfer 100 to my cash account
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,My cash account balance should be 200 and my savings account balance should be 300
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Savings.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Cash.Balance.After

The output above is from the log file that PSExpect generates on every run.  That whole SHOULDPASS concept is there to support a programming style where you first create some tests that you expect to fail - you mark them SHOULDFAIL and then PSExpect prints those assertion failures in YELLOW instead of RED like it would if an assertion labelled with SHOULDPASS fails.  I've come to rely on log files to use as evidence of passing tests, and to support debugging (since I tend to avoid using a step-through debugging tool on my own code; it's an anti-pattern for my personal coding style since using a tool like that means I'm confused and I don't have enough tests to show me the way).

A Way of Explanation

Yes, this really executes and the output demonstrates that each of the checks pass - this example is available in the most recent upload of PSExpect on the CodePlex site in the 'samples' directory.  A couple of things worth pointing out ...

First, the starting curly brace of the script block must be on the same line as the call to invoke-block, otherwise Powershell parses the function call without the script block and errors abound.  A minor inconvenience if that isn't already your scripting convention.

Second, the unfortunate use of set-variable in the first 'given' script block.  That's really just in order to set the scope of variables (hence the use of the -scope parameter).  Alternatively, you could declare the variable and instantiate it before any of the 'given' clauses.  I don't think it matters, but this was the way that it was done in the NBehave example on the CodePlex site so I've copied that approach in my example.

Third, those wrapper functions.  I've mentioned in other blog entries that Powershell uses the verb-noun naming convention for all the built-in cmdlets and that you are forced into this same convention when writing your own custom cmdlets.  Well you should follow that form when writing Powershell functions too and write your functions like they are prototypes of cmdlets.  Much better all-around style, and you don't lose any ground making changes to scripts if you choose to promote a function to a cmdlet.  I prefer to use the wrapper functions in test scripts since they enhance the readability of the script for non-developers.  I've rarely specified the wrapper functions ahead of time.  Instead, I write the test scripts first and determine what wrapper functions I need based on the tests that I want to run.  Yup, test-driven test helper functions.

If you like behaviour-driven development, and you're working with a .NET class library, then using the aliases for given-when-then as specified above seems a lot easier than writing compiled code versions of the test scripts.  Further, and I shouldn't even say this, you don't even need PSExpect to do this - you can write you own version of invoke-block and set up your own aliases to match your own style.  Meta-programming is just one of the advantages of a dynamic language.

Next Steps: Hyper-Testability

But there's more that can be done if you are using Powershell.

Visualize if the test authoring helper functions such as test-accountbalance, set-accountbalance, transfer-money, create-account were Powershell cmdlets instead of Powershell functions.  It's nice to have those tester helper functions initially in Powershell because they can be quickly modified to match the needs of the test authors, however, they will eventually stabilize.  That's when it's time to make them cmdlets and release them along with the rest of the code.  Release the testing support tools as part of the working software you're delivering (at least when you're releasing into a test environment).

Having the cmdlets for supporting the automated testing (behaviour-driven or otherwise) makes any number of scenarios possible - and certainly enhances the testability of the system.  And I'm willing to bet that there is massive overlap between the cmdlets used for system administration and the cmdlets used for testing.  In other words, if you're planning to use cmdlets (and an MMC snap-in) to support the administration of your product, then taking it just-a-little further and creating cmdlets for testing purposes isn't all that much more work.  And the result is something that I'm starting to label "hyper-testability" - where an empowered, educated tester can write any number of scenarios using building blocks provided by the system under test.

Three cheers for empowered testers, testable systems, and oh yes - three cheers for enlightened discourse among a group of really smart people that were kind enough to let me listen in.

I'm at a workshop in Portland this week, participating in a workshop for envisioning the next-generation functional testing tools.  My interest is in how dynamic languages like Powershell fit with functional testing.

My experience on my last project gave me a tremendous amount of hope that Powershell is effective for executing tests and I can easily envision that a cmdlet-adorned application reduces the cost of testing, especially if _all_ the features of the application are exposed through cmdlets.  You just need the scripting skills and the domain knowledge to map the scripts to meaningful user scenarios.

The spreadsheet front-end no doubt that I slapped onto PSExpect helps the business users to get involved, but we need more than their involvement, we need them to be driving it.  We need to support the communication and the craft - two things that are hard to balance.

I'm hoping this week to see what other people are doing, what they are saying they need, and what they have done in their practice to make automated functional testing more mainstream and more accessible to the people with the domain knowledge ...

A.

I've reached a milestone on my current project today – a Developer-Tester double-triple. Triple-digit automated unit tests and triple-digit automated acceptance tests. Unit tests are automated with NUnit and run interactively using TestDriven.net and NCover/NCoverExplorer that ships with TestDriven.net. The automated acceptance tests are all written in PowerShell using PSExpect. The unit tests require about 5-7 seconds to run and the acceptance tests take about 1-2 minutes.

This won't be all of the testing though, the automated unit tests and the automated acceptance tests are only part of the testing strategy on the project. There are GUI tests to come, but we've not yet decided on a tool for automating them, if any. Since we have a focus group of business testers, it might make sense for us to create test checklists for them to use and run those tests manually. We'll see. We can afford to be adaptable since we're adapting an existing site and the number of affected pages is less than 5.

There are also additional unit tests to come as the final integration point gets wrapped in. We've isolated that point for now with a mock provider but we'll have to swap out that mock and use the real thing soon –after it passes all its unit tests.

From Redmond, WA.  I'm at the Platform Adoption Center (PAC) (Building 20) for Apr 18-19 at the PowerShell & MMC ISV Workshop.

In some scripts, especially longer ones, you might find it handy to use logging and tracing that you can turn up or down, depending on the context and depending on yours needs at the time you are running the script.

Using a logging framework would let you replace your 'write-host' statements with logging statements that will have the following characteristics:

• You can turn them on or off with a single line edit (at the beginning of your script)
• You can re-direct them to various destinations - console, file, event log - again with a single line edit
• You can adjust the detail of the log entries - again with a single line edit

The need is so common in application development that there are multiple solutions.  The three that stand out the most (probably) are the PowerShell cmdlets write-* (including write-debug, write-warning, write-error, etc), log4net, an open source solution available from Apache, and the Enterprise Library logging application block from Microsoft's patterns and practices group.

During script development, it is going to be handy to have verbose output of these logging statements turned up so that you know what is going on in the code, not just by looking at the results but also by looking at a record of has been run.  During production script runs, you might need to have a log of everything your script does.  Then again in some situations, you only want to see the logging statements that are associated with errors or other exceptional conditions.

Adding logging might be overkill for exploratory scripting - a scripting technique that PowerShell supports so well with the 'get-member' and 'get-history' cmdlets.  But I can imagine that if you 'productionalize' those scripts developed in an exploratory scripting session, then replacing 'write-host' with a logging statement might be a valuable refactoring.

I'll let you choose which of the above options is best for you as an administrative scripter, but I'm inclined to think that you might go with the flow - whatever the application developers are using will probably already work for you.  If there are no application developers to follow (or lead), then the PowerShell write-* cmdlets might serve you well enough.

Script Logging with Write-* Cmdlets

PowerShell has built-in cmdlets for logging that use the verb "write".  Running the command "get-command write*" will list those commands for you (and a couple of other ones).  Each one is controlled by a shell variable that ends with "Preference". For example, to turn the warning messages up, then set the variable $WarningPreference to "Continue". Other options are Stop, Inquire, and SilentlyContinue. To turn down logging, set each of the $*Preference session variables to "SilentlyContinue". To turn up logging, then set them all to "Continue".

Note - if you're building cmdlets, then there are base class methods that work with the same $*Preference shell variables.  Your cmdlets will be better received if you use them.

Script Logging with Log4net

Log4net has the advantage of having a very small footprint and, appropriate for scripters, it doesn't require a configuration file.  Once you downloaded and unzipped the log4net binaries in a known location (hereinafter referred to as $Log4NetHome) you can use it something like this:

$logpattern = `    "%date [%thread] %-5level [%x] - %message%newline"

# load the log4net library
[void][Reflection.Assembly]::LoadFile($Log4NetHome + "\log4net.dll")

# configure logging
[log4net.LogManager]::ResetConfiguration()
$Appender = new-object log4net.Appender.ConsoleAppender(`
    new-object log4net.Layout.PatternLayout($logpattern))[log4net.Config.BasicConfigurator]::Configure($Appender)

# determines the log statements that show up
$Appender.Threshold = [log4net.Core.Level]::Info$Log = [log4net.LogManager]::GetLogger($Host.GetType())

$Log.Info("Started the main script...")
trap
{

    $Log.Error("Failed...")
   return/continue/break
}

#... 
# run your script here, including other $Log.Info statements 
#...  
$Log.Info("Finished the main script.")

In this example, the value of $logpattern is what determines the output of each log statement - the items preceded by the % character are log4net macros that correspond to attributes of the log statement, including what you have specified, %message.  The call to ResetConfiguration() is necessary if you tend to re-run the scripts in the same PowerShell session, otherwise you will have to take care to declare the logger variables as having script scope instead of the global scope (the default).

To configure the logging, you really only have to identify the layout of the log entries and an appender to use.  As presented above, the ConsoleAppender is used so that the log entries appear in the console.  There are also FileAppenders available as part of the framework.  I've used the PatternLayout here so that the pattern string $logpattern can dictate what shows up for each log entry.  There are other Layouts available, including a SimpleLayout that just lists the category and message for each log entry.

The output of the log entry in this case is something like the following:

2007-04-18 08:30:49,650 [Pipeline Execution Thread] INFO 
  [null] - Started testing parsing.
2007-04-18 08:30:49,400 [Pipeline Execution Thread] ERROR 
  RfcParseExceptionHandler [RFCWorker] - PRS-012:Unrecognized dish: Irregular
2007-04-18 08:30:49,650 [Pipeline Execution Thread] INFO
  [null] - Finished testing parsing.

Changing the log level from "Info" to "Error" would result in only that second log entry being logged - the other ones would be ignored and not show up in the output.

You can match the pattern layout string above to see what each macro generates in the log file.  What I have also found handy was using the NDC stack in log4net to identify where the script was running at the time of the log entry.  The contents of the NDC stack is presented in the pattern by %x, so you can see that in the three log entries that are in the sample output, the value is only expressed in the second one.  Just so happens that's the one that was generated by the C# class library that the PowerShell script was testing.  To use the NDC stack in your script, just push a string onto it in an appropriate line preceding what you want to log, and then pop it off the stack afterwards:

 
[log4net.NDC]::Push("test-getrfcaction-account")
//
//... do your thing, including log statements
//
[log4net.NDC]::Pop()

In this sample, that would put the phrase 'test-getrfcaction-account' will appear instead of the 'null' text that appears there now.  If you push multiple values onto the stack, then as you might expect, the appended stack message will appear in those log statements, consisting of all values that have been pushed but not yet popped.  Hey, it's a stack.  You can appreciate that if you have several script file or several functions running, you might want to identify where the log entry is sourced from.

For my purposes, the developers were already using log4net in their application so using log4net in my test scripts seemed like a reasonable decision.  I do find the single edits handy - I can exclude the informational log entries by changing the Appender.Threshold property to "Error".  I change the log level alot because I'm exploring a bit with the PowerShell test scripts, so until I know for certain how the code I'm testing behaves, I keep it verbose. Last thing I do before adding the test scripts to the regression suite is change the log level to "Error".

So the happy conclusion is -

• add log statements throughout your script
• show your work by changing the log level threshold to "Info"
• hide your work by changing the log level threshold to "Error"
• change what shows up in each log entry by editing a pattern string
• provide a basic version of your custom script call stack using the built-in logging stack (called the NDC)

Hope this helps!

Lately testing people have been talking about testing heuristics - tips, techniques, tactics, strategies for testing something.  I have one that I call MTT.

M - Mission
T - Targets
T - Tests

But I remember it as "mean time to testing".  Exploratory testing excels in part because the mean time to testing is small compared to (caution: wiki-speak ahead), say, BigUpFrontTestManagement.  As an example of minimizing the mean time to testing, the example below uses PowerShell to explore the configuration and functionality of SQL Server Reporting Services 2005.

Mission

Encourage the team to build consensus on why testing will be conducted by paying attention to the project context - who tests and what their background is, what is the system under test, who supports the test environment, what are the communication patterns between testers and developers, ... post the essence of this as a 'test mission' in the team's work area and let them mark it up as the project evolves.  The more people that are involved in the test effort, the more useful a 'testing working agreement' becomes.  Testing faster requires that all the testers understand why they are testing so that they get to the point.

Targets

If the mission indicates why we are testing, then the targets drill deeper and identify what will be tested. I see a lot of testing without clearly-defined test targets.  Some heuristics are test targets, and that helps a lot.  Exploratory testing, even when automated, fits really well with test targets because listing them gives you a testing backlog, making citing progress and indicating expected completion dates simpler using something like testing burndown charts.

Consider automated exploratory testing with PowerShell - let's use the Microsoft Reporting Services installation test as an example.  To use MTT heuristic and the mission stated above, the second step would be to create a testing backlog of the targets of the tests - in other words, what exactly would you want to test?  As with all things agile, this list doesn't have to be complete because you have every right to revise this backlog and prioritize it as necessary, within the context of the test mission.  It is important to avoid listing how these targets might be tested - that's left to exploration.

Microsoft Reporting Services Installation Test - Targets (Testing Backlog)

ReportServer - Service Control
ReportServer - Service Properties
Database - Reporting Services Db - Topology
Database - Reporting Services Db - Physical Configuration
Database - Reporting Services Db - Authentication
Database - Reporting Services Db - Authorization
Operations - Performance Monitors - Availability
Operations - Microsoft Operations Manager - Access
Operations - Microsoft Operations Manager - Alerts
Web Service - Availability
Web Service - Authentication
Web Service - Authorization
Web Service - Functional - Run Report
Report Manager - Availability
Report Manager - Authentication
Report Manager - Authorization
Report Manager - Functional - Reports
Report Manager - Functional - Data Sources
Report Manager - Capacity - Concurrency
Report Manager - Capacity - Reponse Time

Given this list of test targets, an exploratory test effort would start by estimating each one of the targets to create a testing burndown chart, and then diving in according to the priorities assigned in the backlog.  PowerShell is incredibly useful for this given the 'get-member' cmdlet.

Tests

Continuing the Reporting Services example, the first thing that a tiny bit of investigating reveals is that SQL Server Reporting Services has a WMI class that exposes the configuration.  Let's get that class and see what we can do with it.

$targetcomputer = "localhost"
$rs = get-wmiobject -computer $targetcomputer -namespace `
    "root\Microsoft\SqlServer\ReportServer\v9\Admin" MSReportServer_ConfigurationSetting
$rs | get-member

From an exploratory tester's perspective, this list is super useful.  Now you can explore the configuration settings interactively using the $rs variable (they are all listed as Property in the above listing), or create a test script and use data-driven testing to check all the Reporting Services configuration parameters against expected values.  The 'get-history' cmdlet is also useful to retrieve and save the commands that you have experimented with in a format that you can use later for building a script.

To do the data-driven testing in Excel, create a table with three columns and all the properties that you want to test for.  Name the range TestRsConfig and save the worksheet.

Next create a script that includes a function called TestRsConfig (the same name as the named range above) as follows (this script uses the PSExpect testing library for PowerShell):

set-psdebug -strict -trace 0
$targetcomputer = "localhost"
$rs = get-wmiobject -computer $targetcomputer `
    -namespace "root\Microsoft\SqlServer\ReportServer\v9\Admin" `
    MSReportServer_ConfigurationSetting

# Exercises the target of the test - the 
# and responds with a pre-defined set of responses that were
# on the worksheet as the expected results
function TestRsConfig()
{
    param([string]$TestCase, [string]$RsConfigProperty, [string]$ExpectedValue)

    $ActualValue = $rs.PSBase.GetPropertyValue($RsConfigProperty).ToString()
    if ([string]::IsNullOrEmpty($ActualValue)) {$ActualValue = $null}

    if ($ExpectedValue.Trim() -eq "N/C")
    {
        AssertNull $ActualValue -Label $TestCase -Intention $Intention.ShouldPass    
    }
    else 
    {
        AssertEqual $ExpectedValue $ActualValue -Label $TestCase -Intention $Intention.ShouldPass
    }

    RaiseAssertions
}

# run the function library that contains the PowerShell Testing Functions
# the functions are defined as global so you don't need to use dot sourcing
if (!(Test-Path variable:_XLLIB)) { ..\src\DataLib.ps1 }
if (!(Test-Path variable:_TESTLIB)) { ..\src\TestLib.ps1 }

# Configuration and installation testing ...
# Confirm the Reporting Services configuration matches expectations documented in 
# the Excel workbook 'TestRsConfig.xslx'
write-host "Configuration testing ..."
$FieldNames = ("TestCase", "RsConfigProperty", "ExpectedValue")
start-test ((get-location).ToString() + "\TestRsConfig.xlsx") "Sheet1" `
    "TestRsConfig" $FieldNames

The line of note contains '$rs.PSBase.GetPropertyValue' - a PowerShell base method that is proving to be really useful in writing test scripts without hard-coding the property names. In my script, I've used 'N/C' to identify Reporting Services properties that are not configured at all on my laptop.  If I had left them out and therefore avoided the if-then-else code section, the script would only have been two lines.  (The start-test function from PSExpect runs the test and matches up the named range with the test function). Not bad!

But that's only part of the picture - the configuration of Reporting Services.  The other part of the picture is basic functional testing.  Reporting Services exposes its functionality as the Report Manager, a web application but also as Report Server, a web service.  So - what if we used PowerShell to explore and test the Reporting Services web service?  Using the same technique as was used with the Reporting Services WMI class, we can use get-member to help with the exploration.  This doesn't mean that we don't have to test the Report Manager, but it does mean that we can get a quick check of the functionality without a manual test.

This requires a small bit of .NET working knowledge regarding web services.  The trick is to use the .NET SDK 'wsdl.exe' utility to generate a proxy class for the web service, and then wrap that proxy class into an assembly that PowerShell can access:

wsdl http://localhost/ReportServer/ReportService2005.asmx?wsdl 
csc /target:library ReportService2005.cs

Now load this library into PowerShell and use get-member again to investigate the methods that are available:

[void][Reflection.Assembly]::LoadFile("C:\...\ReportingService2005.dll")

# Instantiate the proxy object for the web service
$repsvc = new-object ReportingService2005
$repsvc.Credentials = [System.Net.CredentialCache]::DefaultCredentials
$rs | gm

(I won't list all the methods here because there are many). 

...
CancelAsync
CancelBatch
CancelBatchAsync
CancelJob
CancelJobAsync
CreateBatch
CreateBatchAsync
CreateDataDrivenSubscription
CreateDataDrivenSubscriptionAsync
CreateDataSource
CreateDataSourceAsync
CreateFolder
CreateFolderAsync
CreateLinkedReport
CreateLinkedReportAsync
CreateModel
CreateModelAsync
CreateObjRef
CreateReport
CreateReportAsync
CreateReportHistorySnapshot
CreateReportHistorySnapshotAsync
CreateResource
CreateResourceAsync
CreateRole
CreateRoleAsync
CreateSchedule
...

The (almost complete) example included in the PSExpect download uses the ListChildren, CreateDataSource, and CreateReport methods of the web service to check the functionality of Reporting Services.  The test script uses functions that wrap the calls to the web service in the name of making the test easier to read for testers not familiar with the Reporting Services web service interface.  This is not the complete test - ideally there would also be a 'render-report' and 'test-report' set of functions provided for the testers so that actually generating a report is also part of the test (work in progress).

write-host "Functional testing ..."
$DsBefore = get-datasourcecount
new-datasource -Name "FromPsScript"
test-datasourcecount -Expected ($DsBefore + 1) -Label "Datasources.Count.AfterNew"
remove-datasource -Name "FromPsScript"
test-datasourcecount -Expected ($DsBefore) -Label "Datasources.Count.AfterRemove"

# Verify the report create/delete functionality
$RsBefore = get-reportcount
new-report -Name "ReportFromPsScript"
test-reportcount -Expected ($RsBefore + 1) -Label "Reports.Count.AfterNew"
remove-report -Name "ReportFromPsScript"
test-reportcount -Expected $RsBefore -Label "Reports.Count.AfterRemove"

The example is available for download in the latest source code version of PSExpect.  All together, this wasn't very much code.  The 'mean-time-to-testing' is low considering the automation - and I have an artifact that other people can use for running the tests at any time.

RsLib.ps1 - Contains the wrapper functions new-datasource, test-datasourcecount, remove-datasource, etc.
Test-RsConfig.ps1 - the test script for both configuration and functional tests described above
Test-RsConfig.xslx - Excel spreadsheet containing the named range for properties and expected values used in configuration testing.

Summary

The 'get-member' cmdlet is extremely useful for exploratory testing, especially when combined with the easy access to WMI classes and to web services in PowewrShell.  In the example, I've used the SQL Server Reporting Services WMI class to check the configuration and installation against expected values, and then used a proxy class around the Reporting Services web service to check basic functionality.  What 'get-member' and PowerShell do for us in these cases is give us the tools and the means for exploring the targets of our tests.  Oh yeah - everything is in a script that we can share and evolve as our understanding of the test target increases.

It's been pretty well-documented that test automation, especially at the higher test levels such as system or acceptance testing, is something you should undertake carefully.  Like any other development project, the economic argument has to be there to support the effort - and that economic model can be screwed up pretty easily by analysis-paralysis (planning tests without ever writing or running one) or by creating unmaintainable, brittle test scripts.

There are ways of lowering the cost of automating tests.  There is a PowerShell hook, honest.

Increase the 'Test Cases to Test Script' Ratio

One of the reasons that FIT is spectacular is that it standardizes the test script and permits the tester to focus on providing examples in the form of combinations of test input and expected output.  It does this through the use of test fixtures like the ColumnFixture.  To use ColumnFixture, you design your test by creating a table containing all the test data (inputs and expected outputs) and then write a test fixture that translates a row in the table into a call to the system under test (SUT).  Each row in the table represents a test case - and the running of each test case is all done implicitly by FIT and your test fixture.

It's pure-and-simple data-driven testing.  You achieve a high 'test case to test script' ratio by virtue of never having to write the script beyond writing the test fixture.  The framework implicitly provides the script - and runs it.

You can replicate the use of ColumnFixture in PowerShell/PSExpect by storing the data in Excel (and learning what a named range in Excel is).  See \samples\test-getweatherperf.ps1 for an example.  There are several advantages of using Excel for the test data:

1. You can change the scope of a test run by changing the definition of the named range on the worksheet - eliminating the need to comment out parts of an input file or worse yet, storing multiple versions of it.  
2. You can use formulas to help specify the test data - so some of the test cases can use randomly-generated data instead of the same data every time.  
3. You can ask business users or other domain experts to create or revise the worksheet - Excel is ubiquitous in corporations.  Little or no training required.

But I've written about PowerShell/PSExpect and Excel before.  On to other interesting methods of minimizing test automation costs.

Eliminate Waste - Part 1 - Use Test Targets Wisely (Lean Test Management)

'Waste' in testing comes in two forms: writing test scripts that target the wrong thing, and writing scripts in a form that makes them difficult to evolve as the system under test evolves, or as your understanding of the system under test evolves.

You can avoid testing the wrong thing by first building a list of 'test targets'.  A test target is basically anything that needs testing.  Application functionality, security, performance, etc. might be types of tests but they are not test targets.  Well at least they aren't specific enough to be used wisely.

Break out the types of tests to get specific test targets.  List the application functions (use cases, user stories, whatever) - those are functional test targets.  List the application interfaces - those are integration test targets.  Security test targets include authentication, authorization, and then any specific vulnerability you want to test for (sql command injection, os command injection, etc.).  Performance test targets will be objectives-based like throughput, resource consumption limits, or user concurrency.

You need this list so that you can be wise about setting priorities and so that you can plan to write scripts for only those test targets that really matter.  Host a workshop to decide these priorities so that the whole team is on the same page.

The list of test targets is - dare I say it - a testing backlog.  If you're familiar with Scrum and the use of a product backlog, then take those same concepts and apply them to testing and the testing backlog.  This process works well - I've used testing backlogs and Scrum concepts now on a large enterprise project (an SAP upgrade), an infrastructure project (computing infrastructure migration to Windows Server 2003), and a business intelligence project.  All with favourable results.

Using a testing backlog is only part of the picture though - the scripts themselves can become waste if they are not written wisely.

Eliminate Waste - Part 2 - Write Lean Scripts (Or None At All)

Test scripts are unmaintainable when they hard-wire the test logic, the test data, and the calls to the system under test.  Basically, if any of those three factors changes, then you have to revise the script.  As a test author, you want a test automation strategy that enables you to avoid the hard-wiring trifecta.

For manual tests, avoid writing scripts if you can - instead, rely on your testers to make the leap from the testing backlog to the test results.  they can journal their way through the test targets and still provide excellent test results documentation.  If you can't avoid writing manual test scripts, then choose to write them using only language from the domain - avoid referring to specific user interface elements, avoid binding the script to a particular processing style, etc.  For example, say 'approve invoice' not 'click the Approve button'.  Then if that handy little 'Approve' button disappears in a later revision of the system, the test script survives without changes.

Eliminate Waste - Part 3 - Dependency Injection for Testers (Keyword-Driven Testing)

Automate tests but minimize the test script's dependency on the automation API of the system under test.  This is analogous to writing manual test scripts that minimize the script's dependency on the user interface elements such as buttons or menus.

Consider the test script whose automation depends directly on the system under test.  Any changes whatsoever, and the test script may have to be revised.  To do this in PowerShell, for example, your PowerShell test script would make calls directly against the assemblies that make up the system under test.

Now consider injecting a dependency that shields the test script from the system under test.  It's true that the test fixture may have to change, but your test scripts - ideally - would not be subject to the same degree of change as the test fixture might.   

Unlike the FIT test fixtures that allow you to run several test cases, these injected test fixtures translate test steps into calls to the system under test.  That's key.  A better name for them is actually 'test-step fixtures', given their role.  Give these test-step fixtures names that originate with the business domain, and you have given life to 'keyword-driven testing' in the domain of the system under test.  Otherwise known as the ubiquitous language or domain-driven design.

I'm advocating using PowerShell to write the test scripts and PowerShell cmdlets to implement the test-step fixtures (the bridge).  Until their usefulness is proven in context, the cmdlets can be prototyped as PowerShell functions.

Two roles emerge: the business test scripter that knows the business domain, and the developer scripter that knows the automation API of the system under test.  The test scripter uses the test-step fixtures to assemble scenario tests.  The developer scripter builds the test-step fixtures, either proactively or in response to the needs of the test scripter.

So does this lower the cost of test automation?  We've decreased the coupling between the tests and the system under test by injecting a dependency that we control - so we've made the test less susceptible to technical changes, if not entirely to requirements changes.  But even then - the scenario tests are built from modular components (test-step fixtures) - and modularity is a good thing - because it increases the likelihood of high maintainability and extensibility.  The test-step fixtures may change, but ideally the test scripts themselves would only be adapted as the complexity of the system under test evolves.

On to the PowerShell hook.  Write the test-step fixtures in the form of cmdlets or PowerShell functions, following the same verb-noun naming convention that we might use for building any cmdlet or function.  EXCEPT, for testing purposes, we would prefer to take the 'nouns' from the domain (maybe even the verbs but then we could no longer build cmdlets unless the verb was also on the standard list of PowerShell verbs).

For example:

$MyAccount = get-account 'name'
set-contract $MyAccount

Once you've established a number of the domain-specific verb-noun combinations, then you can build test scenarios in PowerShell using them whose complexity matches the complexity of the system under test - they can evolve independently but at the same pace.  Those test-steps listed above in the example expose nothing of the automation API for the system under test - a good thing if we want to retain the test scripts usefulness over time.

In summary, here are the ways that PowerShell excels for keyword-driven testing:

1. The verb-noun style is PowerShell vernacular - we're not introducing anything new by asking the test scripters to use cmdlets or functions that follow a verb-noun naming convention.
2. PowerShell provides access to both System and custom assemblies and namespaces.  You might use either or both to implement the test-step fixtures.
3. The system under test may require an administration module that might also utilize PowerShell and cmdlets - there would be overlap in administration-type cmdlets and the test-step fixtures so that building them for one purpose would help satisfy the other requirement as well.
4. PowerShell provides exception handling so that test scripters can control the impact of any failed test-steps.
5. PowerShell provides all the control structures so that test scripters can control the flow and execution of test scripts - including looping.  Many other test automation styles only provide rudimentary control, if any.
6. The test script development cycle is short - you write a script, you run it, edit, run it again, etc. very quickly.  You can even prototype script steps in the console.
7. Test-step fixtures can vary in their complexity - you can extend their domain-specific nature by giving them domain-oriented parameters.  For example, new-customer 'GOOD CREDIT', might generate a new customer with a good credit rating to use in the test script.

For a slightly more detailed example, there is one posted for PSExpect that uses this style of test scripting  - samples\Test-Vcs.ps1.  The sample domain is a volunteer coordination system (vcs).

Release 0.3 (Alpha) of PSExpect has been uploaded - hopefully it is clear from the samples and the tests that are included that PowerShell is well-suited for automating tests using both data-driven and keyword-driven techniques.

Release notes for 0.3 Alpha are as follows:

• Addition of a 'start-test' function in DataLib.ps1 that uses an Excel named range and a function you build to test a target script or application library. See \samples\Test-GetWeatherPerf.ps1 for an example.
• Addition of a 'select-row' function in DataLib.ps1 for those data-driven testers that want more control of their test script and its use of Excel
• The Excel functions now use reflection to call the Excel COM API - so these should work in cultures other than 'en-US'
• Addition of a keyword-driven testing example - using verb-noun pairs from the business domain to write maintainable test scripts.
• AssertBlock  is renamed to Assert-Block and is now pipeline-aware
• AssertFaster is now pipeline-aware
• AssertThrows is now pipeline-aware

Next up: a better explanation of keyword-driven testing using PowerShell.  I've included an example in 0.3 Alpha but only in-line comments to explain what it going on and why.

Kudos to the PowerShell Team and the Office Team.  I have a number of test cases for and samples of PSExpect that rely on Excel - part of my kick to support data-driven testing using PowerShell.

Yesterday, Microsoft released PowerShell 1.0 for Vista, and my Vista test machine also has Office 2007.  The PSExpect tests pass on Vista/Office 2007, as they do on Windows XP/Office 2003.  Including the ones that use Excel.

That's a compatibility story, and I'm impressed.

The technique for using Excel as the source of the data to support data-driven testing continues to evolve, but this time I think I've come to the place where I want to be for Release 0.3 of PSExpect (http://www.codeplex.com/psexpect).

In 0.3, there is a new function in DataLib.ps1 called 'start-test' that takes a workbook name, a worksheet name, a range name, and a set of field names as input, and runs the tests that are described in that range.  It does this by calling a function that has the same name as the range - much like FIT calls classes/methods based on the contents of the input table.

So here's the process to follow to use start-test.

1. Describe your tests in Excel.  You should include both inputs and expected outputs in this table.  There is no need to differentiate the inputs from expected outputs like in FIT (in FIT, you specify expected outputs by including parentheses since they represent method calls).  The column titles are meaningless in PSExpect, so you can use titles that make the worksheet easier to read and understand as needed.
2. Define a named range (Insert->Name->Define...) that EXCLUDES the column titles but includes all the rows.  This is why the column titles are meaningless. Unfortunately, a hyphen is not a valid character to use in a range name in Excel so your test fixture can't have a name that looks like most PoSh functions and cmdlets.
3. Write a test fixture (function) in PoSh that has exactly the same name as the name you gave the range in Step 2.  The parameters should match the columns in the table by position (because we build a string in PoSh for execution using invoke-expression).  This function must include an appropriate Assert* function call that determines whether or not the test case passes.
4. Call start-test and provide the workbook name, the worksheet name, the range name, and an array of fieldnames as parameters.
5. Run the script - your results will be on the console and in the log file.

Using 'start-test' is demonstrated in a performance test for the weather web service I've used in previous samples, \samples\Test-GetWeatherPerf.ps1.   The Excel table and the test script are listed below.

# Exercises the target of the test - the get-weather web service
# and then verifies the results against the expected values that are also on the 
# worksheet (in this simple case, the $title parameter)
function TestGetWeather()
{
    param(  [string]$TestCase, `
            [string]$Remark, `
            [string]$Zip, `
            [string]$Units, `
            [string]$Title)

    if ($Units -eq "$null") { $Units = $null }
    if ($Zip -eq "$null") { $Zip = $null }
    
    [string]$urlbase="http://xml.weather.yahoo.com/forecastrss"
    [string]$url=$urlbase + "?p="+$Zip+"&u="+$Units
    
    write-host Connecting to $url
    
    # create .NET Webclient object and call the web service
    $webclient = new-object "System.Net.WebClient"
    $targetBlock = {[xml]$weather=$webclient.DownloadString($url)}

    # measure the results and verify
    $targetBlock | AssertFaster -MaximumTime 300 -Label $TestCase
}

# run the function library that contains the PowerShell Testing Functions
# the functions are defined as global so you don't need to use dot sourcing
if (!(Test-Path variable:_XLLIB)) { ..\src\DataLib.ps1 }
if (!(Test-Path variable:_TESTLIB)) { ..\src\TestLib.ps1 }

# run the test by calling DataLib\start-test function
$FieldNames = ("TestCase","Remark","Zip","Units","Title")
start-test ((get-location).ToString() + "\TestGetWeather.xls") "Sheet1" `
    "TestGetWeather" $FieldNames

The only thing that might change in future releases is a way of eliminating the need to pass in an array of field names to start-test.  This is a left-over from the way that I've implemented the functions start-test depends on (such as select-row).  In those underlying functions, I wanted to ensure that the tester retained control over how the fields were named in order to avoid them having to ALT-TAB between the test data and the routine in order to get the field names right.  This has the additional benefit of NOT restricting column titles on the worksheet, so you can keep them readable and understandable - they aren't property or method names on a test fixture or class.

Notes for those familiar with FIT

• PSExpect doesn't use the column titles in the worksheet, FIT uses them as property names and method calls.
• PSExpect relies on the order of parameters, FIT does not.
• PSExpect uses the name you give the range in Excel as the name of the test fixture to call, FIT uses a table title cell.
• PSExpect requires the test fixture function include the pass/fail decision with a call to an Assert* function, FIT does textual comparisons.
• PSExpect generates colourized output to the console and plain text in a log file, FIT returns the input table marked up to show pass/fail.

I've almost got 0.3 out the door - all the tests of the functions are still passing but I want to include an example of keyword-driven testing, and that's taking some time to get to.

Note: You will need the Jan 22 version of the PSExpect source code to use the functionality described in this entry.

Continuing along the lines of the previous blog entry on the revised set of functions for accessing Excel from PowerShell, there is another new function that I'd like to introduce called 'select-row'.  Good thing I've only made beta releases of the library so far (http://www.codeplex.com/psexpect) because this is evolving as I learn more about PowerShell and how we might want to use it to write test scripts.

The 'select-row' function consolidates all the Excel functionality into a single function so that the tester writing a test script relying on this data doesn't have to know anything about calling COM objects from PowerShell.  Nor does the tester have to know anything about the Excel COM object model.

To use select-row, you have to know that Excel uses workbooks, workbooks contain worksheets, and that on a worksheet, a range of cells can be named.  But you don't have to know the Excel API or that you have to release the COM objects when you're done with them.

The following is the re-written script for testing the 'get-process' cmdlet using Excel to store the test data.

# specify the range location including workbook, worksheet, range
$theFolder = get-location
$WorkbookName = $theFolder.ToString() + "\TestGetProcessFromExcel.xls"

# if this worksheet name doesn't exist in the workbook, the error is ugly
# and Excel is left orphaned and running in the background
$WorksheetName = "Sheet1"

$RangeName = "TestGetProcessFixture"

# specify how to reference the row contents
# this isn't read from the worksheet so that the tester has control over how 
# the columns are referenced in the rest of the test script
$FieldNames = "TestCase","Remark","ProcessName","ShouldExist"

# retrieve the contents of the range
$rangeAsList =  `
 select-row $WorkbookName $WorksheetName $RangeName $FieldNames

# make sure it's not null
AssertNotNull $rangeAsList "GPSXL-1"

# for each row,
foreach ($item in $rangeAsList) {

    # exercise the target of the test (function doesn't exist in this code listing)
    $proc = TestGetProcessFixture $item.ProcessName

 
    # check the actual results against the expected results on the worksheet
    # and use the label from the worksheet's first column 
    AssertEqual $proc $item.ShouldExist $item.TestCase

}
RaiseAssertions

Much simpler than before.  I've used the verb 'select' in the sense that the function returns an array of objects (in this case, hashtables) and I've used the 'row' noun since that is what it being represented by each hashtable.  All of the COM object-release-stuff from the last blog entry is inside the select-row function so that you don't have to worry about it.

I've left the other functions in place because this function just calls them in order to work, but also for those testers that are using many workbooks and don't want to be starting/stopping Excel many times in their test script.

In PSExpect\DataLib.ps1, we've created a small number of Excel-related functions that hopefully make it easier to use Excel for storing test data. What this does for you is increase the "test script to test case" ratio, ideally lowering the cost of testing (as would any other data-driven testing technique). We chose Excel in part because of its ubiquity but also because any end-users just might be able to help fill it out, especially for functional tests of custom-developed applications.

The primary goal in providing this library is to write tests that are shielded from the Excel COM API (object model). In other words, when Excel 2007 comes out and the COM API is different or there is a better .NET-oriented way of accessing Excel, then we can re-write DataLib.ps1 and all of the test scripts we've built that rely on Excel will still function. It also minimizes how much of Excel the test author really needs to know.

The secondary goal was to provide a set of functions that could be used with any culture setting.  There were problems running the Excel COM API from .NET if the culture was not 'en-US'.  This  has been resolved in the library using reflection since you can pass in the culture as a parameter to the method invocation.  This had a ripple effect in that the Range property behaved differently than it did calling it directly.  All the tests pass on the two machines that I have access to (one with 'en-US' the other with 'en-CA').

With the current release of the source code, the library provides net new functions 'open-workbook', 'get-worksheet', and 'get-range'.  The 'collect' function has been replaced by 'collect-range'.  The usage of these functions is demonstrated in src\TestCollectFromExcel.ps1, samples\TestGetWeather.ps1, and samples\TestGetProcessFromExcel.ps1.  Here is a brief example from one of those scripts:

    # get the data
    $workbook = open-workbook $excel $WorkbookFullPath
    $worksheet = get-worksheet $workbook $WorksheetName
    $range = get-range $worksheet $RangeName
    $rangeAsList = collect-range $range $FieldNames

The only messy part is in cleaning up the references to the COM Runtime Callable Wrappers (RCW). All the scripts above contain the code for releasing the COM objects that are used, but because the Range COM API call returns a System.Object[] of COM objects, we have also provided a function called 'release-range' that loops through this array and releases each of the COM objects.

    $excel.Quit()
    $a = release-range $range
    $range = $null
    $a = [System.Runtime.InteropServices.Marshal]::ReleaseComObject($worksheet)
    $a = [System.Runtime.InteropServices.Marshal]::ReleaseComObject($workbook)
    $a = [System.Runtime.InteropServices.Marshal]::ReleaseComObject($excel)

Feedback is welcome on this version of the source, especially if you are not a 'en-US' user of PowerShell and Excel.

* Updated - the Jan 21/2007 source code version of the PSExpect library now works with cultures other than 'en-US'.

In a previous blog entry, I demonstrated how to obtain the test data from Excel using PowerShell.  I'm using the January 21/2007 version of the PSExpect library.  This version adds functions for working with Excel and should work with culture settings other than 'en-US'.

In this blog entry, I will demonstrate testing a web service using PowerShell.  The test data will be in Excel so that the tests are data-driven instead of run directly from script (at least as many as I can).  The web service that will be the target of the test was the subject of a blog entry by Jeffrey Hicks and highlighted by Jeffrey Snover in the PowerShell Team Blog.

Start by first designing the data-driven part.  We should design a table that contains the required inputs to the we service and our expected output.  The following table lists some of the test cases for the web service based on its required inputs and its output. 

The key concept here is that you are designing this table. You choose the inputs based on the specifications of the web service, and you won't have much control over that, true enough.  The rest, you choose.  In other words, it's a design task, and the output is the design of the test fixture that you will code up in PowerShell and call from the test script.

In testing terminology, you are specifying the test oracle, that is, the way that you choose to automate the pass/fail decision.  In the case of the above table, the test oracle is simple and consists of only the Expected Title in the results of the call to the web service.  We will test the web service in the live production environment, so we can't cook up the web-service to return a predictable temperature or wind speed.  So we can't use that data as part of the test oracle.  As an aside, if we were really doing this in a test environment, then you might in fact consider working with a known data set so that you could use more fields than just the title as the test oracle.

In test automation terminology, by creating this table, you have designed the text fixture for the web service.  The next step is to code this test fixture in PowerShell.  The test fixture needs to accept the inputs required to call the web service and return a single output value, the ExpectedTitle.  The 'TestCase' and 'Remarks' fields are used for test documentation only and won't show up as inputs - they will show up in the log of the tests as they run.

# Exercises the target of the test - the get-process cmdlet
# and responds with a pre-defined set of responses that were
# on the worksheet as the expected results
function TestGetWeatherFixture([string] $zip, [string] $units)
{
    write-host $zip $units
    if ($units -eq "$null") { $units = $null }
    if ($zip -eq "$null") { $zip = $null }
    
    [string]$urlbase="http://xml.weather.yahoo.com/forecastrss"
    [string]$url=$urlbase + "?p="+$zip+"&u="+$units
    
    write-host Connecting to $url
    
    # create .NET Webclient object and call the web service
    $webclient=New-Object "System.Net.WebClient"
    [xml]$weather=$webclient.DownloadString($url)

    # translate the results to match the expected output
    return $weather.rss.channel.item.Title.ToString()
}

The function first translates the units since it only receives text input values - a blank cell would also have worked but isn't as easy to work with.  I like testers to be deliberate, so I don't mind forcing them to enter '$null' instead of leaving the cells blank.  Then the function builds the URL required to call the web service, and calls it.  The result is cast as an XML document for each reference, that is, you can use the dot notation to access the various items within the document that is returned from the web service.  Accessing XML content this way was one of the nicest surprises I noticed in PowerShell.  That will become extremely useful to me in many projects, I know it.

The next step is to create the test script that first retrieves the contents of the worksheet from Excel and then loops through each row, calling the test fixture each time.

function TestGetWeather
{

    # specify the workbook full path
    $theFolder = get-location
    $WorkbookFullPath = $theFolder.ToString() + "\TestGetWeather.xls"

    # identify the worksheet that the test data is on
    $WorksheetName = "Sheet1"

    # identify a range - it's a good idea to use Excel's named ranges
    # so that you don't have to change your script if you add test cases.
    # You just have to re-define the named range.
    $RangeName = "TestCases"

    # specify the names of each column - doesn't have to agree with what is
    # on the spreadsheet
    $FieldNames = "TestCase","Remark","Zip","Units","Title"

    # retrieve a list of the contents of the range
    $fileExists = test-path $WorkbookFullPath
    if (!$fileExists) { return $null }

    # start Excel
    $excel = New-Object -comobject Excel.Application

    # get the data
    $workbook = open-workbook $excel $WorkbookFullPath
    $worksheet = get-worksheet $workbook $WorksheetName
    $range = get-range $worksheet $RangeName
    $rangeAsList = collect-range $range $FieldNames

    # make sure it's not null
    AssertNotNull $rangeAsList "GWXL-1"

    # continue 
    if ($rangeAsList -ne $null) {
        write-host "There are " $rangeAsList.Count " test cases."
    
    
        # for each row in the test cases range,
        foreach ($item in $rangeAsList) {
    
            # exercise the target of the test
            $proc = TestGetWeatherFixture $item.Zip $item.Units
    
            # check the actual results against the expected results on the worksheet
            # and use the label from the worksheet's first column 
            $actual = $proc.PadRight(30," ").Substring(0,29).Trim()
            $expected = $item.Title
            AssertEqual $expected $actual $item.TestCase
    
        }
        RaiseAssertions

    } else {
        write-host "Couldn't find the workbook."
    }

    # this is how you ensure that no references are left on your COM object so
    # that it does actually quit
    $excel.Quit()
    $a = release-range $range
    $a = [System.Runtime.InteropServices.Marshal]::ReleaseComObject($worksheet)
    $a = [System.Runtime.InteropServices.Marshal]::ReleaseComObject($workbook)
    $a = [System.Runtime.InteropServices.Marshal]::ReleaseComObject($excel)

}

This function uses the DataLib.ps1 functions from PSExpect in order to retrieve the data from Excel.  Another difference from FIT to highlight is the use of the $FieldNames array to specify how each of the fields will be referred to later in the test script.  I did think of retreiving this automatically from the Excel worksheet, but on thinking about it and trying this out for a bit, I realized that it was easier specifying the field names in the test script since that is where they will be used.  So it's a matter of scrolling up to see the field names instead of ALT-TAB to Excel to see them.  As long as the order is the same, the field names in the test script do not have to match what is in Excel.  Notice in the script the single-valued test oracle - the line that begins with 'AssertEqual ...'.

Here's the output you get when you run this script:

----------------------------

There are 6 test cases.
35801 C
Connecting to http://xml.weather.yahoo.com/forecastrss?p=35801&u=C
35801 F
Connecting to http://xml.weather.yahoo.com/forecastrss?p=35801&u=F
35801 Z
Connecting to http://xml.weather.yahoo.com/forecastrss?p=35801&u=Z
35801 $null
Connecting to http://xml.weather.yahoo.com/forecastrss?p=35801&u=$null
$null F
Connecting to http://xml.weather.yahoo.com/forecastrss?p=$null&u=F
2 F
Connecting to http://xml.weather.yahoo.com/forecastrss?p=2&u=F
1/12/2007 3:47:51 PM,SHOULDPASS,PASSED,GWXL-1
1/12/2007 3:47:54 PM,SHOULDPASS,PASSED,TC-1
1/12/2007 3:47:54 PM,SHOULDPASS,PASSED,TC-2
1/12/2007 3:47:55 PM,SHOULDPASS,PASSED,TC-3
1/12/2007 3:47:56 PM,SHOULDPASS,PASSED,TC-4
1/12/2007 3:47:57 PM,SHOULDPASS,PASSED,TC-5
1/12/2007 3:48:02 PM,SHOULDPASS,PASSED,TC-6

--------------------------

As an aside, implementing these tests before the web service exists would lead to a data-driven, test-first development style for the web service.  You would run the tests and watch them all fail (you might have to code the test fixture to make it handle the non-existent web service better) and then use the failing tests as a "to do" list.  Code only enough to get the first test case to pass.  Then proceed to the next one, refactoring (and unit testing) the code as you go.  This is called story-driven or acceptance-test development since the tests specified here mirror the requirements, not necessarily the structure of the code that implements the web service.  In the above example, the web service is trivial enough that the functional tests we've provided look a lot like unit tests - but that isn't always the case.  Nor are these functional tests covering the full range of the requirements of this particular web service.  But if you did create a script and Excel worksheet that did cover all the requirements - that would be excellent documentation for the web service!

In summary, functional testing of a web service using PowerShell isn't much different from functional testing of a cmdlet.  Your first step is to design the text fixture by creating a table with the necessary inputs and the outputs that will help you to make the pass/fail decision.  The next step is to build the test fixture in PowerShell, and then build a test script that retrieves the data from Excel and runs it through the test fixture.  Using PSExpect, the functions to retrieve the Excel data and to automate the pass/fail decision are provided - you have to supply the test fixture and the test script that loops through the data.  Future versions of DataLib component within PSExpect may reduce the amount of test code you have to write (making it more like FIT) but in the meantime, it does offer a way to get data-driven testing happening sooner.

Happy testing!