Executable Test Plans for BDD

There are many good libraries for developing automated software tests, but we also want to allow non developers to specify test plans and acceptance tests early in the development life cycle.

• we want test plans and acceptance tests to help guide feature development
• we want to track test plans as artifacts in git
• we want to use Gherkin or some similar "living documentation" technology that allows us to link test automation with human readable test plans

Test automation frameworks like jasminejs can support manual tests via interactive helpers.

Test Plans and QA

A test plan is a living document that describes how a system should fulfill different use cases and how the system behaves under various scenarios. The system or feature design process should produce an initial test plan which guides the work of developers and testers, and helps ensure that the development process yields a system that matches its requirements. The test plan changes as the requirements for a system change.

Unit tests and integration tests are both important parts of the quality assurance process. During system development the developer team produce unit tests that run as part of the system's continuous integration process.

The QA team deploys each new build of the feature to a test environment, and caries out a test cycle that tests the behavior of the system against the system's test plan, and generates a report. The level of automation applied to a test cycle generally increases as a feature matures. A CICD system can run a fully automated test cycle that deploys each new build to a test environment, runs the test plan, and publish a report without human intervention.

Test Automation

A test plan should naturally evolve from manual checks carried out by the QA team to a fully automated test suite execute by a CICD pipeline. Test frameworks like jasminejs support automated tests, and can be leveraged to execute and generate reports for manual test cycles.

The testHelper.ts helpers allow us to write jasminejs test suites that present instructions to a human tester, then interactively collect the results of the test. This facility allows us to intermix manual and automated tests, and also support partially automated tests.

For example, one test in the testHelperSpec looks like this:

    it("allows interactive tests", ... ifInteractive(async () => {
        let result = await interactive("press enter at the prompt");
        expect(result.didPass).toBe(true);
        result = await interactive("enter \"y\" at the prompt");
        expect(result.didPass).toBe(true);
        result = await interactive("enter \"n\" at the prompt, then some explanation at the next prompt");
        expect(result.didPass).toBe(false);
        expect(result.details.length > 0).toBe(true);
    }, 3600000) as any,
    );

What's going on here?

• ifInteractive tests the LITTLE_INTERACTIVE environment variable. If it's not false, then ifInteractive just returns its arguments (a lambda and timeout for jasmine's it method); otherwise ifInteractive returns a do-nothing test lambda for non-interactive environments
• interactive prints the given instructions to the console for a human tester, and prompts the tester for whether the test passed or failed
• there's also an isInteractive helper that just returns true if the LITTLE_INTERACTIVE environment variable is not false, and allows an automated test case to include optional manual elements

That's it! A simple test driven development process has a project manager specify acceptance tests in the feature definition, then a developer or tester translates those tests directly to code that fits into existing tooling for tracking changes, running tests, and generating reports. If you use jasminejs, then you can get started with the following code, otherwise it's straight forward to adapt the code to other frameworks. Good luck!

npm install @littleware/little-elements
const { interactive, ifInteractive } = require('@littleware/little-elements/commonjs/bin/testHelper.js');