Define
System Test Entrance and Exit Criteria

 

Introduction

 

The
entrance and exit criteria for system testing define the necessary hardware,
software, and resources required to effectively test and release an application
to the user acceptance stage. The entrance and exit criteria for system testing
should be identified within the Master Test Strategy, address potential risks
and suspension/resumption strategy.  The
goal of having entrance/exit criteria is to provide a means to check the status
of the project before moving it into the next phase/stage. It also serves as an
easy means for communicating expectations within the project team of the
requirements necessary for system testing.

 

This
guideline will provide an overview of the seven steps involved in the entrance
and exit criteria process for system testing. It is not enough to state the
criteria within the Master Test Plan. Rather, you must actively participate in
defining the activities associated with meeting the criteria, drive towards its
successful completion and manage risks if/when they arise. Examples will also
be provided so that you can tailor them to your specific project.

 

Step 1: Define
System Test Entrance and Exit Criteria

 

This step consists of not only identifying
what must be present to effectively begin system testing (entrance criteria)
but also determining the condition the system will be in when system test
completes (exit criteria). At a minimum, the entrance criteria should include a
condition that the business requirements with any subsequent change requests
are finalized and unit testing has completed with all critical and high defects
resolved. Likewise, the exit criteria should include the resolution of all
critical and high defects uncovered during system testing, and the execution of
all planned test scripts.

 

Because the success of system testing is
contingent on the quality of the Requirements, code, and the test Environment,
you must clearly define the system entrance criteria for each. Consider
developing checklists and conducting walkthroughs to validate the entrance
criteria. For example, when defining system test entrance criteria you should
incorporate activities such as requirements reviews, code inspections and
environment walk-through as follows.

 

Entrance Criteria
Examples:

 

Note that these are
only examples and should be modified for your project. Checklists have been
provided in the latter parts of this document.

 

#	Criteria Description	Due Date	Actual Date	Yes	No	Comment(s)
1	Requirements Review checklist complete
2	Unit/Integration tests complete via weekly Unit Test Walk-through meetings
3	Environment support is procured to execute batch cycles in the Test environment
4	Environment checklist and walk-through complete					tables, forms, screens, programs,
5	Test scripts (including expected results), and input parameters reviewed and approved

 

You
should incorporate these activities and deliverables within the Master Test
Plan under the system test entrance criteria. Append any checklists utilized to
the Master Test Plan.

 

A.   
Develop a Requirements
Review Checklist – Ensure
the entrance criteria within the Master Test Plan provide that the Requirements
review checklist must successfully be completed with all items marked as “Yes” 3
weeks prior to system build.

 

SAMPLE: Requirements Walk-Through Checklist
Technique	Yes	No	N/A	Resolution
Completeness – Check for Omissions ·         Dangling Else – Application must do “X.” Ask or else what? ·         Cause without Effect – A confirmation message will appear after the user logs into the system. Ask will the user then get automatically logged into the system? Where will they be taken? ·         Effect without Cause – User will receive a $20 check in the mail, 2 days after data is entered. Ask what will generate this type of system response?
Ambiguity – ·         Uncertain Name Reference –Uncertain Scope Reference – When a requirement states “A field +B field = C field, the field must be positive.” Ask which field must be positive? ·         Missed Logical Operators – If any “And” or “Or” statements, ensure the requirement is clear on whether both or one item must occur. ·         Sentence Parts – For nouns, verify all references to fields are named explicitly.  For adjectives, ensure words such as “typically,” “usually” or “sometimes are not in the removed.
Accuracy – Determine to what level of accuracy the requirement must be tested. If a requirement states, “the forecast model will need to be accurate.” Ask to what level of accuracy (dollars, cents)? Also determine how the accuracy will be verified.
Relevancy – If a requirement states, data input areas must be secure and require user login. Ask does the requirement apply to all
Requirements and change requests approved one week prior to system build.

 

 

Step 1: Define
System Test Entrance and Exit Criteria

 

B.   
Develop a Unit Test
Checklist and Establish Walk-Through meetings– Expend the effort during the
entrance criteria definition step to clearly define the quality of code that
will be received. Consider the following activities:

·        
Meet with the project team during Solution
Scoping/Planning

·        
Develop an Unit Test checklist similar to the one
provided below

·        
Refer to the Unit Test checklist within the
Entrance criteria of the Master Test Plan and append it to the Master Test
Plan.

·        
Specify that all items in the Unit Test
Walk-Through checklist must be marked as “Yes” prior to system testing.

 

SAMPLE: UNIT Test Walk-Through
Technique	Yes	No	Comments
All Critical/High defects resolved and retested
All Medium/Low defects reviewed by project management, test lead, and development and determinations are made on which defects require resolution prior to System Testing
All Unit Tests Complete
Unit Test documentation, including results, provided one business day prior to Systems Testing
Development trained on application and available during system test
Acceptance of service level agreement for defect resolution during system test: Critical Defects – development response within 2 hours of defect reported High Defects – development response within 4 hours of reporting Medium/High Defects – development response within 8 hours of reporting

 

 

 

C.   
Develop the
Environment Checklist and Establish Environment Walk-Through – Similar to the Unit
Test walk-through consider the following activities in defining the entrance
criteria for Test Environment:

·        
Meet with the project team during Solution
Scoping/Planning;

·        
Develop an Environment checklist similar to the one
provided below; and,

·        
Refer to the Environment checklist within the
Entrance criteria of the Master Test Plan and append it to the Master Test
Plan.

·        
Specify that all items within the Environment
checklist must be marked as “Yes” 1 week prior to system testing.

 

SAMPLE: Environment Checklist
Technique	Yes	No	Comments
Code has been copied and smoke tested within the Test Environment 1 week prior to System Testing and after any code changes.
Test data has been made available within the Test Environment 1 week prior to system testing.
Back up copy of code and test data created after smoke tests.
Smoke Test documentation, including results, provided four business days prior to Systems Testing
Ensure written communication to Test Lead/Manager prior to emergency code movements with an assessment of any risks to the application tested.
Environment support is procured to execute batch cycles in the Test environment
Notification of any server maintenance activities to Test Lead and Test Manager minimum, 8 hours prior to implementation with an assessment of potential impact to application.
Server Maintenance activities to be conducted after 5pm PST with written notification to Test Lead and Manager when server available.
Acceptance of service level agreement for defect resolution during system test: Critical Defects –  response within 2 hours of defect reported High Defects –  response within 4 hours of reporting Medium/High Defects – response within 8 hours of reporting
System configuration and documentation maintained and made available in a central location accessible to Test Lead and Test Manager

 

The
exit criteria define the condition the application will be in after system
testing. The following example may be used. You should work amongst your project
team in assessing in defining the exit criteria and incorporate the definition
into the Master Test Plan.

 

Exit Criteria
Examples:

1. All test
   scripts with the appropriate input parameters executed
2. Test scripts
   are updated as deemed necessary
3. All critical/high
   severity defects are resolved and successfully retested
4. System test
   results and metrics have been documented
5. Test
   Lead/Coordinator with Project Management and Development will review open
   defects of medium and low severities and will determine which defects
   require resolution before exiting system test.

 

Step 2: Determine
if Entrance Criteria Met

         

The second step in the entrance and exit
criteria process requires a determination if the entrance criteria has been
met. This is a key step as it ensures the system is ready for testing. You
should review the entrance criteria as specified in Step 1, (Define System Test
Entrance and Exit Criteria), and determine whether the requirements are testable,
unit testing is complete, and test environment is configured and ready for
system test. The following sub tasks are suggested activities and should help
with determining whether the entrance criteria have been met.

 

A. Review the Requirements/Change Requests
for Testability - You must expend the
effort to review the Requirements and any change requests. Work to resolve any
“No” items prior to system testing and if necessary proceed to step 3 if any
items are not resolved at least 3 weeks prior to system test.  

 

B. Conduct
Weekly Unit Test Review Meetings - You should
conduct weekly checkpoints during the system build and review the status of
unit testing. Consider the following process for ensuring entrance criteria met
for unit testing:

 

·        
Review the Unit Test Checklist during system build
during weekly meetings and confirm criteria have been met.

·        
Go to Step 3 of the Entrance/Exit Criteria process,
if any items are assessed as “No” are not resolved prior to System testing.

 

C.   
Conduct Environment Review - You should also conduct an Environment review a week prior to system
testing to confirm entrance criteria met. Consider the following process for
reviewing the Environment:

 

·        
Review the Environment Checklist one week prior to
testing and confirm criteria have been met.

·        
Go to Step 3 of the Entrance/Exit Criteria process,
if any items are assessed as “No” are not resolved prior to System testing.

 

Step 3: Identify
Risk and Determine Mitigation Strategy

         

If the entrance criteria have not been met,
then you must collaborate within the project team and begin an assessment of
project risks including potential delays in scheduled testing timelines,
possible increase in test effort due to scope creep, resource availability, and
any impacts to software quality.

 

After you identify the risks, you should
also determine how they can be mitigated. For example, if the test environment
is unavailable, then you should assess whether decreasing the test effort,
adding more test resources, or extending timelines may mitigate the risk of not
having the test environment available at the start of system testing. If you
are able to successfully mitigate the risks identified, then you should go
directly to conducting the system test.

 

Step 4: Determine
System Suspension/Resumption Strategy

         

          If you are unable to mitigate the
risks identified in step 3, then you should make a determination regarding
whether system testing will need to be suspended until the issues identified
have been resolved. You may find this step necessary if the Environment has not
been setup, if required test hardware/software is unavailable, or critical/high
unit testing defects prevent system testing. You must ensure that if system
test has been suspended due to a failure of entrance criteria being met, then
you also communicate when system testing will be resumed. Any issues that
require escalation to project stakeholders should also be documented and
communicated.

 

Step 5: Conduct
System Test

         

          When system entrance criteria have
been satisfied, then the next step in the process includes conducting the
system test as identified within the Master Test plan.

 

Step 6: Confirm
System Test Exit Criteria Met

           

            Upon the conclusion of system
testing, review the exit criteria and ensure at a minimum planned test scripts
have been executed and critical/high defects have been resolved. Should the
exit criteria fail to meet what was specified in step 1, then you’ll need to
repeat the risks identification process and ensure any exit criteria risks have
been mitigated.

 

Step 7: Communicate
System Test Complete

 

          The final step in the entrance/exit
criteria process is communicating system test complete. Ensure once all of the
exit criteria for system test have been addressed that you communicate to your
project team that the application has completed system test.

 

Conclusion

 

          The system testing entrance and exit
criteria process provides a means of communicating what must be met prior to
testing and releasing a system to its next stage. In documenting these
criteria, you ensure expectations are proactively shared within the project
team. You also provide a clear basis for assessing risks and ultimately the
consequences of those risks to the quality of system testing.