|
Validation |
Verification |
|
Am I building the right product |
Am I building the product right |
|
Determining if the system complies with the requirements and performs functions for which it is intended and meets the organization’s goals and user needs. It is traditional and is performed at the end of the project. |
The review of interim work steps and interim deliverables during a project to ensure they are acceptable. To determine if the system is consistent, adheres to standards, uses reliable techniques and prudent practices, and performs the selected functions in the correct manner. |
|
Am I accessing the right data (in terms of the data required to satisfy the requirement) |
Am I accessing the data right (in the right place; in the right way). |
|
High level activity |
Low level activity |
|
Performed after a work product is produced against established criteria ensuring that the product integrates correctly into the environment |
Performed during development on key artifacts, like walkthroughs, reviews and inspections, mentor feedback, training, checklists and standards |
|
Determination of correctness of the final software product by a development project with respect to the user needs and requirements |
Demonstration of consistency, completeness, and correctness of the software at each stage and between each stage of the development life cycle. |
Metric is a mathematical number that shows a relationship between two variables. It is a quantitative measure of the degree to which a system, component or process possesses a given attribute. Software Metrics are measures that are used to quantify the software, software development resource and software development process.
Metric generally classified into:
· Process Metric
· Product Metric
Process Metric: Metric used to measure the characteristic of the methods, techniques and tools employed in developing, implementing and maintaining the software system.
Product Metric: Metric used to measure the characteristic of the documentation and code.
The metrics for the test process would include status of test activities against the plan, test coverage achieved so far, among others. An important metric is the number of defects found in internal testing compared to the defects found in customer tests, which indicate the effectiveness of the test process itself.
What is API Testing?
An API (Application Programming Interface) is a collection of software functions and procedures, called API calls, which can be executed by other software applications. API testing is mostly used for the system which has collection of API that needs to be tested. The system could be system software, application software or libraries.
API testing is different from Unit, white box and UI testing, UI is rarely involved in API Testing. Tester need to setup initial environment, invoke API with required set of parameters and then finally analyze the result.
Initial environment means test environment setup and application setup, database creation, server configuration, config and properties file setup and deployment of application or any coding (if it is required).
Usually people or company synonym as Unit or white box testing but there is huge difference between API, Unit and while box testing. Tester need to or may need to do coding during API testing.
Difference between Api Testing and Unit Testing
Unit testing is an activity that is owned by the development team; developers are expected to build unit tests for each of their code modules (these are typically code may or may not contains classes, functions, stored procedures, or some other ‘atomic’ unit of code), and to ensure that each module passes its unit tests before the code is included in a build.
Unit tests are typically designed by the developers to verify the functionality of each unit. The scope of unit testing often does not consider the system-level interactions of the various units; the developers simply verify that each unit in isolation performs as it should.
API testing is typically an activity owned by the QA team, API tests are often run after the build has been created, and it is common that the authors of the tests do not have access to the source code; they are essentially creating black box tests against an API rather than the traditional GUI.
In API testing, QA team must consider the ‘full’ functionality of the system, as it will be used by the end user. This means that API tests must be far more extensive than unit tests, and take into consideration the sorts of ’scenarios’ that the API will be used for, which typically involve interactions between several different modules within the application.
API is mostly black box testing where as unit testing is essentially kind of white box testing. Unit test cases are typically designed by the developers and there scope is limited to the unit under test. In API testing, test cases are designed by the QE team and there scope is not limited to any specific unit, but it normally covers complete system.
So a tester should able to identify before start API testing…
*Find the way to approach the task?
*Do boundary analysis.
*Create or focus on the most likely usage scenarios (Functional Scenario).
* Check Return Values
*Focus also on negative testing to get exception and error handling.
*Check Event triggers (Optional and depends on API Type)
*Modify resources (Optional and depends on API Type)
Type of API and how to approach them:
API can be called directly or it can be called because of some event or in response of some exception. Output of API could be some data or status or it can just wait for some other call to complete in a-synchronized environment.
A. IF API return value based on input condition:
- In this condition, test cases will be based on the input and relative output.
- This is relatively simple to test as input can be defined and results can be validated against expected return value.
- User can pass different combinations of values or parameters and validate these against known results.
B. If API does not return anything:
- In this situation tester should need to identify some mechanism to check behavior of API on the system.
- For example, if you need to write test cases for delete (List Element) function you will probably validate size of the list, absence of list element in the list.
C. If API Trigger some other API/event/interrupt:
- If API is triggering some event or raising some interrupt, then you need to listen for those events and interrupt listener.
- Test suite should call appropriate API and asserts should be on the interrupts and listener.
D. If API is used to Update data structure:
- Updating data structure will have some effect on the system and that should be validated.
- If you have other means of accessing the data structure, it should be used to validate that data structure is updated.
E. If API is Modifying certain resources:
- If API call is modifying some resources, for example updating some database, changing registry, killing some process etc, then it should be validated by accessing those resources.
Challenges of API Testing:
a. Parameter Selection:
Ensuring that the test harness varies parameters of the API calls in ways that verify functionality and expose failures. This includes assigning common parameter values as well as exploring boundary conditions.
b. Parameter combination:
Generating interesting parameter value combinations for calls with two or more parameters.
c. Setting environment:
Determining the content under which an API call is made. This might include setting external environment conditions (files, peripheral devices, and so forth) and also internal stored data that affect the API.
d. Call sequencing:
Sequencing API calls to vary the order in which the functionality is exercised and to make the API produce useful results from successive calls.
Most common seniors in API Testing:
a. Test Response: Each API method in isolation using only the mandatory elements and typical content so if system calls any API then it send any response so tester need to check valid response from API.
b. Test Limit: These tests exercise each API method using all optional elements and maximum allowable content lengths and/or instances of repeated elements.
c. Test Business Logic: This is where the business application logic is simulated in the test code. Each API method will have a defined set of test cases that explore its interaction and influence on other API methods and any underlying database.
d. Test Negative or Illegal: These tests contain a sampling of typical error scenarios, such as missing required elements, empty content, and content exceeding maximum limits, across a representative sampling of the API methods, so API should enough intelligent to handle exception, errors and missing parameters issues.
e. Test Load: Most of applications are web 2.0 and running on internet and access by number of users so API should able to take heavy load and should not break during peak time when number of people will use application, tester also measure response time, through put , latency, memory leak and any other factor according to requirement
If you were to ask testers how to test API, you would get several different perspectives. Every people have different thinking and approaches to solve problems and every approach have option for enhancements.
Test Plan reflects your entire project testing schedule and approach.
Test Plan is a management planning document that describes:
1. How the testing will be done?
2. Who will do it?
3. What will be tested?
4. How long it will take?
5. What the test coverage will be?
Test plan document formats can be as varied as the products and organizations to which they apply, but there are three major elements of a test strategy that should be described in the test plan:
Test Coverage - Test coverage in the test plan states what requirements will be verified during what stages of the product life
Test Methods - Test methods in the test plan state how test coverage will be implemented
Test Responsibilities - Test responsibilities include what organizations will perform the test methods and at each stage of the product life
IEEE Standard Test Plan Structure (IEEE 829-1998)
This standard specifies the following test plan outline:
- Test plan identifier
- Introduction
- Test items
- Features to be tested
- Features not to be tested
- Approach
- Item pass/fail criteria
- Suspension criteria and resumption requirements
- Test deliverables
- Testing tasks
- Environmental needs
- Responsibilities
- Staffing and training needs
- Schedule
- Risks and contingencies
- Approvals
Test Plan Identifier:
- A unique identifier
Introduction:
- Summary of the items and features to be tested
- Need for and history of each item (optional)
- References to related documents such as project authorization, project plan, QA plan, configuration management plan, relevant policies, relevant standards
- References to lower level test plans
Test Items:
- Test items and their version
- Characteristics of their transmittal media
- References to related documents such as requirements specification, design specification, users guide, operations guide, installation guide
- References to bug reports related to test items
- Items which are specifically not going to be tested (optional)
Features to be Tested:
- All software features and combinations of features to be tested
- References to test-design specifications associated with each feature and combination of features
Features Not to Be Tested:
- All features and significant combinations of features which will not be tested
- The reasons these features won’t be tested
Approach:
- Overall approach to testing
- For each major group of features of combinations of features, specify the approach
- Specify major activities, techniques, and tools which are to be used to test the groups
- Specify a minimum degree of comprehensiveness required
- Identify which techniques will be used to judge comprehensiveness
- Specify any additional completion criteria
- Specify techniques which are to be used to trace requirements
- Identify significant constraints on testing, such as test-item availability, testing-resource availability, and deadline
Item Pass/Fail Criteria:
- Specify the criteria to be used to determine whether each test item has passed or failed testing
Suspension Criteria and Resumption Requirements
- Specify criteria to be used to suspend the testing activity
- Specify testing activities which must be redone when testing is resumed
Test Deliverables:
- Identify the deliverable documents: test plan, test design specifications, test case specifications, test procedure specifications, test item transmittal reports, test logs, test incident reports, test summary reports
- Identify test input and output data
- Identify test tools (optional)
Testing Tasks:
- Identify tasks necessary to prepare for and perform testing
- Identify all task interdependencies
- Identify any special skills required
- Environmental Needs
- Specify necessary and desired properties of the test environment: physical characteristics of the facilities including hardware, communications and system software, the mode of usage (i.e., stand-alone), and any other software or supplies needed
- Specify the level of security required
- Identify special test tools needed
- Identify any other testing needs
- Identify the source for all needs which are not currently available
Responsibilities:
- Identify groups responsible for managing, designing, preparing, executing, witnessing, checking and resolving
- Identify groups responsible for providing the test items identified in the Test Items section
- Identify groups responsible for providing the environmental needs identified in the Environmental Needs section
Staffing and Training Needs:
- Specify staffing needs by skill level
- Identify training options for providing necessary skills
Schedule:
- Specify test milestones
- Specify all item transmittal events
- Estimate time required to do each testing task
- Schedule all testing tasks and test milestones
- For each testing resource, specify its periods of use
Risks and Contingencies:
- Identify the high-risk assumptions of the test plan
- Specify contingency plans for each
Approvals:
- Specify the names and titles of all persons who must approve the plan
- Provide space for signatures and dates