PowerShell for Testers

Testability is a software quality attribute that is a measure of how easy (or difficult) it is to create test cases for a test target and, in combination, how easy (or difficult) it is to run those tests.  High testability is a good thing since it means that the total cost of making changes to the software is correspondingly lower.  It typically also means that you can have more than one individual contributing to the software since it is easy for them to determine whether or not their contribution has been a net positive or net negative contribution, simply by running the tests again.  Hence, we like test automation.

It turns out that when we value testability, we tend to write software that enables us to automate more types of tests, and we set up this fantastic feedback loop where defining a test becomes a means of production - the tests specify the desired behaviour, guide the development, and then verify the as-built software for as long as that behaviour remains desired.  Pretty nice, when we get to that level.

Tools like xUnit were revolutionary in the sense that they gave developers the means for valuing testability and a mechanism for expressing the low-level behaviour in the same language that our target software was (or would be) written in.  Developers liked it, and now teams in general are much more inclined to automate unit tests than ever before.

But is it enough?  That's a question that Brian Marick has been pondering publicly in his blog (it's one of those blogs that actually expands your mind when you read it) and in an on-line forum discussing automated functional testing.  I was struck by the phrase "Perhaps it would be better to invest heavily in unprecedented amounts of built-in support for manual exploratory testing." the most.  An alternative, he indicates, to automated functional testing with tools like FIT/FITNesse (read the blog entry to find out why this isn't agile heresy).

You can see where I'm going. If you have a set of Powershell cmdlets that exposes an application's functionality - and if the cmdlets expose ALL of the functionality - then 'exploring' means working with those cmdlets (or an API) in a Powershell session and using whatever testing heuristics you might have in your tester's toolkit to test the application.  You may not be using the same UI as the end-users, however, there is great value to first confirming the functionality is good enough, and then second moving on to confirm that the UI is good enough.

The Powershell features that make exploratory testing an excellent experience are (but not limited to):

• Objects on the pipeline, and in fact, having objects to work with at all.
• The get-help cmdlet that can access custom help files for custom cmdlets.
• The get-member (gm) cmdlet.
• The get-history (h) cmdlet.
• The naming convention for cmdlets.

You don't even really have to know the cmdlets ahead of time because of those features since you can retrieve their calling syntax at run-time.  The get-history cmdlet can then be used to save useful cmdlets and sequences of calls, in order to support future semi-automated exploratory testing (where some scripts are run, but only for the sake of returning to a known state) or ultimately fully automated testing for something like detecting regression.  Seems that the ability of a script for passing on knowledge isn't limited to administrators ...

I can envision, however, a mismatch between the cmdlets that an application designer foresees might be a useful way of exposing the functionality and the cmdlets that a tester might find useful.  They will each bring, I believe, a different perspective.

So it might be wise to undertake this 'unprecedented support for exploratory testing' or hyper-testability using a test-driven approach where the functionality of those cmdlets is influenced, or possibly specified, by the examples that a tester might deem appropriate.  Supporting manual exploratory testing by writing automated test scripts ahead of time ... seems contradictory. But there is a distinction between 'understanding-clarifying tests' (Brian Marick again) and 'verification and validation' tests.  So using a test-driven approach for understanding and clarifying requirements, and ultimately the design of the cmdlets, isn't the same as writing tests for verifying and validating the resulting application.

The bottom line?  Exposing application functionality in Powershell makes your application hyper-testable.  Use test scripts that you/your testers write ahead of time as a means of designing the cmdlets and separate the concern of testing the functionality of the application from the concern of testing the user interface.

I know we're not supposed to come up with solutions before fully understanding the problem, but I've got just enough experience and wide-eyed insensibility to believe that I've seen something that just might work: testing mashups.

Microsoft Popfly is a set of tools for creating web pages and the like from pre-defined blocks.  The one aspect that I am particularly intrigued by is called a mashup.  You use a tool called the Mashup Creator to build a web page comprised of blocks that come from disparate sources.  For example, build a page that lists your Facebook friends' statuses and your Twitter friends' most recent updates.  I'm assuming it's called a 'mashup' after the term that's been coined for mixing songs into single tracks (some of my favourites have sampled from 5 or more songs).

Now imagine if the "blocks" used to build a mashup were Powershell cmdlets instead of web applications and that those cmdlets were designed to expose the functionality of an application under test.  Each "mashup" you create is therefore another application scenario.  Imagine that some of the blocks you use on a mashup are data sources so that you could make the scenario data-driven.

So, why do I think this might be a good idea?

First, it may help non-programmers to develop automated test scenarios that are executable.  Second, the economics of building cmdlets for the application just changed: not only might the cmdlets be useful for system administration, but now they will also be useful in writing automated tests for the application.

Why cmdlets? Well they exhibit all the behaviour that I would want from blocks for building tests from - they have a naming convention, there are behavioural standards that they adhere to, and there is a mechanism already in place for stringing them together and running them as one scenario, so they're not just commands that can be invoked.  If the 'noun' part of the cmdlet always refers to something from the business domain, then we're working at the same level of abstraction as (hopefully) the requirements are written - so as testers we're grounded, too.  There is a tie-in to the whole world of domain-specific language and domain-driven development that just can't be ignored.

Now these aren't the only tests that would need to be considered for any system under development.  A test strategy for any software product is going to include extensive developer (unit and low-level integration) testing and manual functional and acceptance testing. I get that.   At the same time, automating more test scenarios means they are shareable and immediately executable by anyone on the team, enhancing communications and reducing communication overhead.  That's a good thing, and worth striving for.

or, A Testament to the Flexibility of Powershell

The greatest impact that the Functional Testing Tools Workshop in Portland (October, 2007) had on me was to raise my awareness of behaviour-driven development, in particular, the 'given-when-then' pattern for structuring examples.

I will defer to other blogs on describing given-when-then in detail, you'll see what it means soon enough in the examples below.  For now, think of given-when-then as a means of translating the meaning of a user story into a related set of acceptance criteria using examples.  Something that has always been a bit awkward for business users and developers alike.

Given you adopt the story-telling convention popularized by Mike Cohn (I think),

As a <blank>
I want to <blank>
So that <blank>

when you want to translate the story into a set of related acceptance criteria using examples,
then use the following convention popularized by BDD proponents such as Dan North:

Given <a known set of conditions or state>
When <something happens>
Then <check for an expected set of conditions or state>

There are tools that help you do this and to make statements worded like this executable - in that case, they almost become executable requirements.  They are certainly executable examples.  The most-often-used example is for a simple account manager class (I've seen this example on a number of blogs but didn't pay attention to where it originated, but I think it was Dan North's again).  I'm not the only person looking at this - there is a NBehave project on CodePlex that provides a mechanism for writing tests in this manner.  And there are many ruby examples in both rspec and rbehave.

# Story: Transfer to cash account from savings account
# As a savings account holder
# I want to transfer money from my savings account
# So that I can get cash easily from an ATM

# Scenario 1: Savings account is in credit
given "My savings account balance is 100 and my cash account balance is 10"
when "I transfer 20 to my cash account from savings"
then "my cash account balance should be 30 and my savings account balance should be 80"

given "My savings account balance is 400 and my cash account balance is 100"
when "I transfer 100 to my cash account"
then "My cash account balance should be 200 and my savings account balance should be 300"

My opinion is that this is a clear way of expressing an example, albeit all that I have really seen are simplistic ones.  I'm going to enjoy working through more complex examples on an upcoming project.

Now on to Powershell.  It was always my intention to introduce some sort of behaviour-driven development helper functions in PSExpect, but I was never certain of what that could be.  Until I went to the Functional Testing Tools conference in Portland.  I love it when I can take some time off to think, and be inspired by a group of brilliant people.

Turns out we can make those examples above executable by writing a function and adding a set of aliases to our Powershell environment.  The function accepts a string parameter and a script block parameter.  All that it does is write out the string to a log file and then run the script block.  I've called it Invoke-Block for lack of imagination, and it uses the PSExpect logging mechanism so that the same log file shows the results of assertions.

function global:Invoke-Block([string]$Label, [ScriptBlock]$Script)
{

    # build the log entry and write it out
    $entry = BuildLogEntry $Intention.ShouldPass $ResultPrefix.Passed "PSpec" $Label
    $added = $Assertions.Add($entry)
    if ($LogFileName -ne $null) {
        $logEntry = BuildMessage $entry
        $logEntry | out-file $LogFileName -append -width $LogFileLineLength
    }
   
    # now run the script block
    if ($Script)
    {
        &$Script
    }
}

set-alias given invoke-block
set-alias when invoke-block
set-alias then invoke-block

The aliases enable using the given-when-then triad in a Powershell script without generating any errors.  This means that the executable version of the example is the same as above, with the script blocks added in.  The example relies on a class library from the system under test (SUT) that the script loads using reflection (not shown) but the calling syntax to that library is hidden using a set of Powershell functions that utilize a verb-noun naming convention: create-account, transfer-money, set-accountbalance, and test-accountbalance (these are elements in the grammar of the domain-specific language, or ubiquitous language for this particular system under development).

# Story: Transfer to cash account from savings account
# As a savings account holder
# I want to transfer money from my savings account
# So that I can get cash easily from an ATM

# Scenario 1: Savings account is in credit
given "My savings account balance is 100 and my cash account balance is 10" {
    set-variable -name SvAccount -value (create-account "Savings" 100) -scope script
    set-variable -name csAccount -value (create-account "Cash" 10) -scope script
}

when "I transfer 20 to my cash account from savings" {
    transfer-money -From $SvAccount -To $CsAccount -Amount 20
}

then "my cash account balance should be 30 and my savings account balance should be 80" {
    test-accountbalance -Account $SvAccount -Expected 80 -Label "Savings.Balance.After"
    test-accountbalance -Account $CsAccount -Expected 30 -Label "Cash.Balance.After"
}

given "My savings account balance is 400 and my cash account balance is 100" {
    set-accountbalance -Account $SvAccount -Amount 400
    set-accountbalance -Account $CsAccount -Amount 100
}

when "I transfer 100 to my cash account" {
    transfer-money -From $SvAccount -To $CsAccount -Amount 100
}

then "My cash account balance should be 200 and my savings account balance should be 300" {
    test-accountbalance -Account $SvAccount -Expected 300 -Label "Savings.Balance.After"
    test-accountbalance -Account $CsAccount -Expected 200 -Label "Cash.Balance.After"
}

RaiseAssertions

The Output

29/11/2007 1:56:13 PM,SHOULDPASS,PASSED,PSpec,My savings account balance is 100 and my cash account balance is 10
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,I transfer 20 to my cash account from savings
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,my cash account balance should be 30 and my savings account balance should be 80
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Savings.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Cash.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,My savings account balance is 400 and my cash account balance is 100
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,I transfer 100 to my cash account
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,PSpec,My cash account balance should be 200 and my savings account balance should be 300
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Savings.Balance.After
29/11/2007 1:56:14 PM,SHOULDPASS,PASSED,Cash.Balance.After

The output above is from the log file that PSExpect generates on every run.  That whole SHOULDPASS concept is there to support a programming style where you first create some tests that you expect to fail - you mark them SHOULDFAIL and then PSExpect prints those assertion failures in YELLOW instead of RED like it would if an assertion labelled with SHOULDPASS fails.  I've come to rely on log files to use as evidence of passing tests, and to support debugging (since I tend to avoid using a step-through debugging tool on my own code; it's an anti-pattern for my personal coding style since using a tool like that means I'm confused and I don't have enough tests to show me the way).

A Way of Explanation

Yes, this really executes and the output demonstrates that each of the checks pass - this example is available in the most recent upload of PSExpect on the CodePlex site in the 'samples' directory.  A couple of things worth pointing out ...

First, the starting curly brace of the script block must be on the same line as the call to invoke-block, otherwise Powershell parses the function call without the script block and errors abound.  A minor inconvenience if that isn't already your scripting convention.

Second, the unfortunate use of set-variable in the first 'given' script block.  That's really just in order to set the scope of variables (hence the use of the -scope parameter).  Alternatively, you could declare the variable and instantiate it before any of the 'given' clauses.  I don't think it matters, but this was the way that it was done in the NBehave example on the CodePlex site so I've copied that approach in my example.

Third, those wrapper functions.  I've mentioned in other blog entries that Powershell uses the verb-noun naming convention for all the built-in cmdlets and that you are forced into this same convention when writing your own custom cmdlets.  Well you should follow that form when writing Powershell functions too and write your functions like they are prototypes of cmdlets.  Much better all-around style, and you don't lose any ground making changes to scripts if you choose to promote a function to a cmdlet.  I prefer to use the wrapper functions in test scripts since they enhance the readability of the script for non-developers.  I've rarely specified the wrapper functions ahead of time.  Instead, I write the test scripts first and determine what wrapper functions I need based on the tests that I want to run.  Yup, test-driven test helper functions.

If you like behaviour-driven development, and you're working with a .NET class library, then using the aliases for given-when-then as specified above seems a lot easier than writing compiled code versions of the test scripts.  Further, and I shouldn't even say this, you don't even need PSExpect to do this - you can write you own version of invoke-block and set up your own aliases to match your own style.  Meta-programming is just one of the advantages of a dynamic language.

Next Steps: Hyper-Testability

But there's more that can be done if you are using Powershell.

Visualize if the test authoring helper functions such as test-accountbalance, set-accountbalance, transfer-money, create-account were Powershell cmdlets instead of Powershell functions.  It's nice to have those tester helper functions initially in Powershell because they can be quickly modified to match the needs of the test authors, however, they will eventually stabilize.  That's when it's time to make them cmdlets and release them along with the rest of the code.  Release the testing support tools as part of the working software you're delivering (at least when you're releasing into a test environment).

Having the cmdlets for supporting the automated testing (behaviour-driven or otherwise) makes any number of scenarios possible - and certainly enhances the testability of the system.  And I'm willing to bet that there is massive overlap between the cmdlets used for system administration and the cmdlets used for testing.  In other words, if you're planning to use cmdlets (and an MMC snap-in) to support the administration of your product, then taking it just-a-little further and creating cmdlets for testing purposes isn't all that much more work.  And the result is something that I'm starting to label "hyper-testability" - where an empowered, educated tester can write any number of scenarios using building blocks provided by the system under test.

Three cheers for empowered testers, testable systems, and oh yes - three cheers for enlightened discourse among a group of really smart people that were kind enough to let me listen in.

I'm at a workshop in Portland this week, participating in a workshop for envisioning the next-generation functional testing tools.  My interest is in how dynamic languages like Powershell fit with functional testing.

My experience on my last project gave me a tremendous amount of hope that Powershell is effective for executing tests and I can easily envision that a cmdlet-adorned application reduces the cost of testing, especially if _all_ the features of the application are exposed through cmdlets.  You just need the scripting skills and the domain knowledge to map the scripts to meaningful user scenarios.

The spreadsheet front-end no doubt that I slapped onto PSExpect helps the business users to get involved, but we need more than their involvement, we need them to be driving it.  We need to support the communication and the craft - two things that are hard to balance.

I'm hoping this week to see what other people are doing, what they are saying they need, and what they have done in their practice to make automated functional testing more mainstream and more accessible to the people with the domain knowledge ...

A.

I've reached a milestone on my current project today – a Developer-Tester double-triple. Triple-digit automated unit tests and triple-digit automated acceptance tests. Unit tests are automated with NUnit and run interactively using TestDriven.net and NCover/NCoverExplorer that ships with TestDriven.net. The automated acceptance tests are all written in PowerShell using PSExpect. The unit tests require about 5-7 seconds to run and the acceptance tests take about 1-2 minutes.

This won't be all of the testing though, the automated unit tests and the automated acceptance tests are only part of the testing strategy on the project. There are GUI tests to come, but we've not yet decided on a tool for automating them, if any. Since we have a focus group of business testers, it might make sense for us to create test checklists for them to use and run those tests manually. We'll see. We can afford to be adaptable since we're adapting an existing site and the number of affected pages is less than 5.

There are also additional unit tests to come as the final integration point gets wrapped in. We've isolated that point for now with a mock provider but we'll have to swap out that mock and use the real thing soon –after it passes all its unit tests.