I2b2 Test Plan

Document Change History

** Note to Document Author – R ed and blue text (with the exception of the title and document name above) in this document is directed at the template user to describe processes, build standards and help build the document from the template. All such red and blue text should be removed before submitting any formal documentation, including both draft and/or final, deliverables. ****                            

Table of Contents

1 Introduction ..............................................................................................................

1.1 Scope .................................................................................................................

1.1.1 In Scope .......................................................................................................

1.1.2 Out of Scope ................................................................................................

1.2 Quality Objective ................................................................................................

1.2.1 Primary Objective ........................................................................................

1.2.2 Secondary Objective ....................................................................................

1.3 Roles and Responsibilities ..................................................................................

1.3.1 Developer ....................................................................................................

1.3.2 Tester ...........................................................................................................

1.3.3 Business Analyst ..........................................................................................

1.4 Assumptions for Test Execution .........................................................................

1.5 Constraints for Test Execution ...........................................................................

1.6 Definitions ..........................................................................................................

2 Test Methodology .....................................................................................................

2.1 Purpose ..............................................................................................................

2.1.1 Overview ......................................................................................................

2.1.2 Usability Testing ..........................................................................................

2.1.3 Unit Testing ..................................................................................................

2.1.4 Iteration/Regression Testing ........................................................................

2.1.5 Final Release Testing ...................................................................................

2.1.6 Testing Completeness Criteria .....................................................................

2.2 Test Levels .........................................................................................................

2.2.1 Build Tests ...................................................................................................

2.2.1.1 Level 1 - Build Acceptance Tests ...................................................................

2.2.1.2 Level 2 - Smoke Tests ..................................................................................

2.2.1.3 Level 2a - Bug Regression Testing .................................................................

2.2.2 Milestone Tests ............................................................................................

2.2.2.1 Level 3 - Critical Path Tests ...........................................................................

2.2.3 Release Tests ...............................................................................................

2.2.3.1 Level 4 - Standard Tests ...............................................................................

2.2.3.2 Level 5 - Suggested Test ..............................................................................

2.3 Bug Regression ..................................................................................................

2.4 Bug Triage ..........................................................................................................

2.5 Suspension Criteria and Resumption Requirements

2.6 Test Completeness .............................................................................................

2.6.1 Standard Conditions: ...................................................................................

2.6.2 Bug Reporting & Triage Conditions:

3 Test Deliverables ......................................................................................................

3.1 Deliverables Matrix ............................................................................................

3.2 Documents .........................................................................................................

3.2.1 Test Approach Document ............................................................................

3.2.2 Test Plan ......................................................................................................

3.2.3 Test Schedule

3.2.4 Test Specifications .......................................................................................

3.2.5 Requirements Traceability Matrix ................................................................

3.3 Defect Tracking & Debugging ............................................................................

3.3.1 Testing Workflow .........................................................................................

3.3.2 Defect reporting ...........................................................................................

3.4 Reports ...............................................................................................................

3.4.1 Testing status reports ..................................................................................

3.4.2 Phase Completion Reports ...........................................................................

3.4.3 Test Sign-Off ................................................................................................

3.5 Responsibility Matrix ..........................................................................................

4 Resource & Environment Needs ...............................................................................

4.1 Testing Tools ......................................................................................................

4.1.1 Tracking Tools ..............................................................................................

4.1.1.1 Configuration Management ...........................................................................

4.2 Test Environment ...............................................................................................

4.2.1 Hardware

4.2.2 Software

4.3 Bug Severity and Priority Definition

4.3.1 Severity List .................................................................................................

4.3.2 Priority List ...................................................................................................

4.4 Bug Reporting ....................................................................................................

5 Terms/Acronyms .......................................................................................................

1     Introduction

This test approach document describes the appropriate strategies, process, workflows and methodologies used to plan, organize, execute and manage testing of the i2b2 project. 

To assure the i2b2 data extraction process from original sources is accurate and complete.

1.1.1 In Scope

The i2b2 Test Plan defines the unit, integration, system, regression, and user acceptance testing approach.  The test scope includes the following:

•     Testing of all functional, application performance, security and use cases requirements listed in the Use Case document
•     Quality requirements
•     End-to-end testing and testing of interfaces of all systems that interact with i2b2

1.1.2 Out of Scope

The following are considered out of scope for i2b2 Test Plan and testing scope:

•     Functional requirements testing for systems outside i2b2
•     Testing of disaster recovery and business continuity plan

1.2.1 Primary Objective

A primary objective of testing i2b2 is to: assure that the system meets the full requirements, including quality requirements and fit metrics for each quality requirement and satisfies the use case scenarios and maintain the quality of i2b2.  At the end of i2b2 project development cycle, the end-user should find that the project has met or exceeded all of their expectations as detailed in the requirements. 

Any changes, additions, or deletions to the requirements document, functional specification, or design specification will be documented and tested at the highest level of quality allowed within the remaining time of the project and within the ability of the development and testing teams.

1.2.2 Secondary Objective

The secondary objective of testing i2b2 will be to: identify and expose all issues and associated risks, communicate all known issues to the project team, and ensure that all issues are addressed in an appropriate matter before release.  As an objective, this requires careful and methodical testing of the application to first ensure all areas of the system are scrutinized and, consequently, all issues (bugs) found are dealt with appropriately. 

I2b2 roles are defined as follows:

1.3.1 Developer

(a) Develop/maintain the system/application

(b) Develop use cases and requirements in collaboration with the Business Analyst

(c) Conduct alpha unit, system, regression, and integration testing

(d) Support user acceptance testing

   (e) Consists members if the Infrastructure Team

1.3.2 Tester

Manages the entire testing process, workflow and quality with activities and responsibilities to:

(a) Conduct beta unit, system, regression, and integration testing

   (b) Manage testing integrity

   (c) Support development activities

(d) Document testing activities

1.3.3 Business Analyst

Undertakes formal user acceptance and end-user communications in application of Projects or solutions adopted by Developers. Responsible to:
 

(a) Contribute to use case, requirement development through review

(b) Contribute to develop and execution of the development test scripts through review

(c) Conduct full user acceptance, regression, and end-to-end testing; this includes identifying testing scenarios and providing end-user feedback to Developers

•     For user acceptance testing, the Development/Testing teams have completed unit, system and integration testing and met all the Requirements based on the Requirement Traceability Matrix .
•     User Acceptance testing will be conducted by Business Analyst
•     Test results will be reported and maintained in i2 b 2 Testing   . Failed scripts and defects will be reported with evidence and will be sent to Development team directly
•     Use cases have been developed by Business Analyst for user acceptance testing and reviewed by Development team
•     Test scripts are developed and approved by Business Analyst
•     Test Team will support and provide appropriate guidance to Business Analyst and Developers to conduct testing
•     Major dependencies should be reported immediately after the testing kickoff meeting. 

Below are some minimum assumptions followed by example constraints

•     Developer will support ongoing testing activities based on priorities
•     Test scripts must be approved by Business Analyst prior test execution
•     Test scripts, test environment and any dependencies should be addressed during testing kickoff meeting in presence of Infrastructure Team and request list should be submitted within due time to prepare
•     The Developer cannot execute the user acceptance. After alpha testing, the developer will make known any development issues they may encounter to resolve, but reporting of the issues/incidents is not required.

Bugs - Any error or defect that cause the software/application or hardware to malfunction. That is also included in the requirements and does not meet the required workflow, process or function point.  An error or defect that causes the software/application or hardware to malfunction.

Enhancement - Any alteration or modification to the existing system for better workflow and process.  What is NOT included in the requirements can be categorized as an enhancement. 

Enhancements can be added as a new requirement after appropriate Change Management process.

2     Test Methodology

2.1.1 Overview

The below list is not intended to limit the extent of this test plan and can be modified as necessary.

The purpose of the test plan is to achieve the following:

• Define testing strategies for each area and sub-area to include all the functional and quality requirements
• Divide Design Spec into testable areas and sub-areas.  Be sure to also identify and include areas that are to be omitted (not tested) also
• Define bug-tracking procedures
• Identify testing risks
• Identify required resources and related information

2.1.2 Usability Testing

The purpose of usability testing is to ensure that the new components and features will function in a manner that is acceptable to the end-user. 

Development will create a non-functioning prototype of the UI components to evaluate the proposed design.  Usability testing can be coordinated by testing, but actual testing must be performed by non-tester/Business Analyst (in this case) .   Testing will review the findings and provide the development team with its evaluation of the impact these changes will have on the testing process and to the project as a whole.

2.1.3 Unit Testing

Unit Testing is conducted by the Developer during code development process to ensure that proper functionality and code coverage have been achieved by each developer both during coding and in preparation for acceptance into functional and quality testing. 

The following are the areas of the i2b2 that must be unit-tested and signed-off before being passed on to regression Testing:

• Databases
• Stored Procedures
• Triggers
• Tables
• Indexes
• Services
• Database conversion
• other binary formatted executables

2.1.4 Iteration/Regression Testing

During the repeated cycles of identifying bugs and taking receipt of new builds (containing bug fix code changes), there are several processes which are common to this phase across all projects.  These include the various types of tests: functionality, performance, stress, configuration, etc.  There is also the process of communicating results from testing and ensuring that new drops/iterations contain stable fixes (regression). 

At each iteration of testing, a debriefing will be held.  All identified bugs have been communicated and addressed.  At a minimum, all priority bugs should be resolved prior to entering the beta phase.

Important deliverables required for acceptance into Final Release testing include:

• Installation/ Configuration procedures
• Any User instructions
• All test scripts, requirements, or training docs, etc.

2.1.5 Final Release Testing

Testing team with Business Analyst participates in this milestone process as well by providing confirmation feedback on any new issues uncovered, and input based on identical or similar issues detected earlier.  The intention is to verify that the Project is ready for distribution, acceptable to the customer and iron out potential operational issues. 

Assuming critical bugs are resolved during previous iterations testing; testing will continue its process of verifying the stability of the application through regression testing (existing known bugs, as well as existing test cases). 

The milestone target of this phase is to establish that the application under test has reached a level of stability, appropriate for its usage (number users, etc.), that it can be released to the i2b2 community (end users).

2.1.6 Testing Completeness Criteria

Release for Projection can occur only after the successful completion of the application under test throughout all of the phases and milestones previously discussed above. 

The milestone target is to place the release/app (build) into Projection after it has been shown that the app has reached a level of stability that meets or exceeds the client expectations as defined in the Requirements, Functional Spec., and i2b2 Projection Standards.

Testing of an application can be broken down into three primary categories and several sub-levels.  The three primary categories include tests conducted every build (Build Tests), tests conducted every major milestone (Milestone Tests), and tests conducted at least once every project release cycle (Release Tests). The test categories and test levels are defined below:

2.2.1 Build Tests

2.2.1.1 Level 1 - Build Acceptance Tests

These test cases simply ensure that the application can be built and installed successfully.   Other related test cases ensure that adopters received the proper Development Release Document plus other build related information (drop point, etc.).  The objective is to determine if further testing is possible. If any Level 1 test case fails, the build is returned to developers un-tested.

2.2.1.2 Level 2 - Smoke Tests

These tests cases verify the major functionality a high level.  

The objective is to determine if further testing is possible.  These test cases should emphasize breadth more than depth.  All components should be touched, and every major feature should be tested briefly by the Smoke Test. If any Level 2 test case fails, the build is returned to developers un-tested.

2.2.1.3 Level 2a - Bug Regression Testing

Every bug that was “Open” during the previous build, but marked as “Fixed, Needs Re-Testing” for the current build under test, will need to be regressed, or re-tested.  Once the smoke test is completed, all resolved bugs need to be regressed. 

2.2.2 Milestone Tests

2.2.2.1 Level 3 - Critical Path Tests

Critical Path test cases are targeted on features and functionality that the user will see and use every day.

Critical Path test cases must pass by the end of every 2-3 Build Test Cycles.  They do not need to be tested every drop, but must be tested at least once per milestone.  Thus, the Critical Path test cases must all be executed at least once during the Iteration cycle, and once during the Final Release cycle.

2.2.3 Release Tests

2.2.3.1 Level 4 - Standard Tests

Test Cases that need to be run at least once during the entire test cycle for this release.  These cases are run once, not repeated as are the test cases in previous levels.  Functional Testing and Detailed Design Testing (Functional Spec and Design Spec Test Cases, respectively).  These can be tested multiple times for each Milestone Test Cycle (Iteration, Final Release, etc.). 

Standard test cases usually include Installation, Data, GUI, and other test areas.

2.2.3.2 Level 5 - Suggested Test

These are Test Cases that would be nice to execute, but may be omitted due to time constraints.

Most Performance and Stress Test Cases are classic examples of Suggested test cases (although some should be considered standard test cases).  Other examples of suggested test cases include WAN, LAN, Network, and Load testing.

Bug Regression will be a central tenant throughout all testing phases.  

All bugs that are resolved as “Fixed, Needs Re-Testing” will be regressed when the Tester is notified of the new drop containing the fixes.  When a bug passes regression it will be considered “Closed, Fixed”.  If a bug fails regression, the tester will notify development team immediately.  When a Severity 1 bug fails regression, Tester should also put out an immediate email to development.  The Tester will be responsible for tracking and reporting to development and Project management the status of regression testing.

Bug Triages will be held throughout all phases of the development cycle.  Bug triages will be the responsibility of the Tester.  Triages will be held on a regular basis with the time frame being determined by the bug find rate and project schedules.  

The Tester, Project Manager, and Development Lead should all be involved in these triage meetings.  The Tester will provide required documentation and reports on bugs for all attendees.  The purpose of the triage is to determine the type of resolution for each bug and to prioritize and determine a schedule for all “To Be Fixed Bugs’.  Development will then assign the bugs to the appropriate person for fixing and report the resolution of each bug back into the bug tracker system.  The Tester will be responsible for tracking and reporting on the status of all bug resolutions.

This section should be defined to list criteria’s and resumption requirements should certain degree and pre-defined levels of test objectives and goals are not met.

Please see example below:

- Testing will be suspended on the affected software module when Smoke Test (Level 1) or Critical Path (Level 2) test case bugs are discovered after the 3 rd iteration.

- Testing will be suspended if there is critical scope change that impacts the Critical Path

A bug report should be filed by Development team.  After fixing the bug, Development team will follow the drop criteria (described above) to provide its latest drop for additional Testing.   

Testing will be considered complete when the following conditions have been met:

2.6.1 Standard Conditions:

• When Adopters and Developers, agree that testing is complete, the app is stable, and agree that the application meets functional requirements
• Script execution of all test cases in all areas have passes
• Automated test cases have in all areas have passed
• All priority 1 and 2 bugs have been resolved and closed
• NCI approves the test completion
• Each test area has been signed off as completed by the Tester
• 50% of all resolved severity 1 and 2 bugs have been successfully re-regressed as final validation
• Ad hoc testing in all areas has been completed

2.6.2 Bug Reporting & Triage Conditions:

Please add Bug reporting and triage conditions that will be submitted and evaluated to measure the current status

• Bug find rate indicates a decreasing trend prior to Zero Bug Rate (no new Sev. 1/2/3 bugs found)
• Bug find rate remains at 0 new bugs found (Severity 1/2/3) despite a constant test effort across 3 or more days
• Bug severity distribution has changed to a steady decrease in Sev. 1 and 2 bugs discovered
• No ‘Must Fix’ bugs remaining prior despite sustained testing

3     Test Deliverables

Testing will provide specific deliverables during the project.  These deliverables fall into three basic categories: Documents, Test Cases / Bug Write-ups, and Reports.  Here is a diagram indicating the dependencies of the various deliverables:

As the diagram above shows, there is a progression from one deliverable to the next.  Each deliverable has its own dependencies, without which it is not possible to fully complete the deliverable.

The following page contains a matrix depicting all of the deliverables that Testing will use.

Below is the list of artifacts that are process driven and should be produced during the testing lifecycle. Certain deliverables should be delivered as part of test validation, you may add to the below list of deliverables that support the overall objectives and to maintain the quality.

This matrix should be updated routinely throughout the project development cycle in you project specific Test Plan.

3.2.1 Test Approach Document

The Test Approach document is derived from the Project Plan, Requirements and Functional Specification documents.  This document defines the overall test approach to be taken for the project.

When this document is completed, the Tester will distribute it to the Project Lead, Development Leads, Business Analyst, and others as needed for review and sign-off.

3.2.2 Test Plan

The Test Plan is derived from the Test Approach, Requirements, Functional Specs, and detailed Design Specs.  The Test Plan identifies the details of the test approach, identifying the associated test case areas within the specific Project for this release cycle. 

The purpose of the Test Plan document is to:

• Specify the approach that Testing will use to test the Project, and the deliverables (extract from the Test Approach).
• Break the Project down into distinct areas and identify features of the Project that are to be tested.
• Specify the procedures to be used for testing sign-off and Project release.
• Indicate the tools used to test the Project.
• Identify risks and contingency plans that may impact the testing of the Project.
• Specify bug management procedures for the project.
• Specify criteria for acceptance of development drops to testing (of builds).

3.2.3 Test Schedule

This section is not vital to the document as a whole and can be modified or deleted if needed by the author.

3.2.4 Test Specifications

A Test Specification document is derived from the Test Plan as well as the Requirements, Functional Spec., and Design Spec documents.  It provides specifications for the construction of Test Cases and includes list(s) of test case areas and test objectives for each of the components to be tested as identified in the project’s Test Plan.

3.2.5 Requirements Traceability Matrix

A Requirements Traceability Matrix (RTM) which is used to link the test scenarios to the requirements and use cases is a required part of the Test Plan documentation for all projects.  Requirements traceability is defined as the ability to describe and follow the life of a requirement, in both a forward and backward direction (i.e. from its origins, through its development and specification, to its subsequent deployment and use, and through periods of ongoing refinement and iteration in any of these phases) .

3.3.1 Testing Workflow

The below workflow illustrates the testing workflow process for Developers and Adopters for User Acceptance and End to End testing.

Note the yellow highlighted process where the Adopter is required to directly send defect list with evidence to the Developer. Similarly, Developer is required to confirm directly with the Adopter after bug fixes along with updating the selected bug tracking tool.

3.3.2 Defect reporting

ALL defects should be logged to address and debug defects. Adopters are also requested to send a report of bugs to the developer. Developers will update the defect list and notify the requestor after the defect has been resolved.

Debugging should be based on Priority – High > Medium > Low, these priorities are set by the Adopters and are based on how critical is the test script in terms of dependency and mainly based on use case scenario.

All High priority defects should be addressed within (1 day) of the request and resolved/closed within 2 days of the initial request

All Medium priority defects should be addressed within (2 days) of the request and resolved/closed within 4 days of the initial request

All Low priority defects should be resolved/closed no later than (5 days) of the initial request.

The Tester will be responsible for writing and disseminating the following reports to appropriate project personnel as required.

3.4.1 Testing status reports

A status report will be forwarded by the Tester to Development and filed in the Project Plan .   This report will summarize weekly testing activities, issues, risks, bug counts, test case coverage, and other relevant metrics.

3.4.2 Phase Completion Reports

When each phase of testing is completed, the Tester will file a Testing Report to the Project Manager and Development Team, for review and sign-off.

The document must contain the following metrics:

• Total Test Cases, Number Executed, Number Passes / Fails, Number Yet to Execute
• Number of Bugs Found to Date, Number Resolved, and Number still Open
• Breakdown of Bugs by Severity / Priority Matrix
• Discussion of Unresolved Risks
• Discussion of Schedule Progress (are we where we are supposed to be?)

3.4.3 Test Sign-Off

Certifies as to the extent to which testing has actually completed (test case coverage report suggested), and an assessment of the Project’s readiness for Release to Projection.

Add resource matrix – who is involved and what is the role.

4     Resource & Environment Needs

4.1.1 Tracking Tools

<test tool> bug tracker is used by i2b2 to enter and track all bugs and project issues.  The Tester is responsible for maintaining the <test tool> database. 

4.1.1.1 Configuration Management

Please include your configuration management plan (unique to each site)

4.2.1 Hardware

Include the minimum hardware requirements that will be used to test i2b2.

Testing will have access control to one or more application/database servers separate from any used by non-test members of the project team.  Testing will also have access control to an adequate number of variously configured PC workstations to assure testing a range from the minimum to the recommended client hardware configurations listed in the project’s Requirements, Functional Specification and Design Specification documents.

4.2.2 Software

In addition to the application and any other customer specified software, the following list of software should be considered a minimum:

• Mozilla Firefox
• Internet Explorer
• Safari
• Test Software
• Talend, etc (list)

Bug Severity and Priority fields are both very important for categorizing bugs and prioritizing if and when the bugs will be fixed.  The bug Severity and Priority levels will be defined as outlined in the following tables below.  Testing will assign a severity level to all bugs.  The Tester will be responsible to see that a correct severity level is assigned to each bug.

The Tester, Development Lead and Project Manager will participate in bug review meetings to assign the priority of all currently active bugs.  This meeting will be known as “Bug Triage Meetings”.  The Tester is responsible for setting up these meetings on a routine basis to address the current set of new and existing but unresolved bugs.

4.3.1 Severity List

The tester reporting a bug is also responsible for indicating the bug Severity. 

4.3.2 Priority List

Testing team recognizes that the bug reporting process is a critical communication tool within the testing process.  Without effective communication of bug information and other issues, the development and release process will be negatively impacted.

The Tester will be responsible for managing the bug reporting process.  Standard bug reporting tools and processes will be used. 

5     Terms/Acronyms

The below terms are used as examples, please add/remove any terms relevant to the document.