Think Publication

What is API Testing?
An API (Application Programming Interface) is a collection of software functions and procedures, called API calls, which can be executed by other software applications. API testing is mostly used for the system which has collection of API that needs to be tested. The system could be system software, application software or libraries.
API testing is different from Unit, white box and UI testing, UI is rarely involved in API Testing. Tester need to setup initial environment, invoke API with required set of parameters and then finally analyze the result.
Initial environment means test environment setup and application setup, database creation, server configuration, config and properties file setup and deployment of application or any coding (if it is required).

Usually people or company synonym as Unit or white box testing but there is huge difference between API, Unit and while box testing. Tester need to or may need to do coding during API testing.

Difference between Api Testing and Unit Testing
Unit testing is an activity that is owned by the development team; developers are expected to build unit tests for each of their code modules (these are typically code may or may not contains classes, functions, stored procedures, or some other ‘atomic’ unit of code), and to ensure that each module passes its unit tests before the code is included in a build.
Unit tests are typically designed by the developers to verify the functionality of each unit. The scope of unit testing often does not consider the system-level interactions of the various units; the developers simply verify that each unit in isolation performs as it should.

API testing is typically an activity owned by the QA team, API tests are often run after the build has been created, and it is common that the authors of the tests do not have access to the source code; they are essentially creating black box tests against an API rather than the traditional GUI.

In API testing, QA team must consider the ‘full’ functionality of the system, as it will be used by the end user. This means that API tests must be far more extensive than unit tests, and take into consideration the sorts of ’scenarios’ that the API will be used for, which typically involve interactions between several different modules within the application.

API is mostly black box testing where as unit testing is essentially kind of white box testing. Unit test cases are typically designed by the developers and there scope is limited to the unit under test. In API testing, test cases are designed by the QE team and there scope is not limited to any specific unit, but it normally covers complete system.

So a tester should able to identify before start API testing…

*Find the way to approach the task?
*Do boundary analysis.
*Create or focus on the most likely usage scenarios (Functional Scenario).
* Check Return Values
*Focus also on negative testing to get exception and error handling.
*Check Event triggers (Optional and depends on API Type)
*Modify resources (Optional and depends on API Type)

Type of API and how to approach them:
API can be called directly or it can be called because of some event or in response of some exception. Output of API could be some data or status or it can just wait for some other call to complete in a-synchronized environment.

A. IF API return value based on input condition:
- In this condition, test cases will be based on the input and relative output.
- This is relatively simple to test as input can be defined and results can be validated against expected return value.
- User can pass different combinations of values or parameters and validate these against known results.

B. If API does not return anything:
- In this situation tester should need to identify some mechanism to check behavior of API on the system.
- For example, if you need to write test cases for delete (List Element) function you will probably validate size of the list, absence of list element in the list.

C. If API Trigger some other API/event/interrupt:
- If API is triggering some event or raising some interrupt, then you need to listen for those events and interrupt listener.
- Test suite should call appropriate API and asserts should be on the interrupts and listener.

D. If API is used to Update data structure:
- Updating data structure will have some effect on the system and that should be validated.
- If you have other means of accessing the data structure, it should be used to validate that data structure is updated.

E. If API is Modifying certain resources:
- If API call is modifying some resources, for example updating some database, changing registry, killing some process etc, then it should be validated by accessing those resources.

Challenges of API Testing:

a.     Parameter Selection:
Ensuring that the test harness varies parameters of the API calls in ways that verify functionality and expose failures.         This includes assigning common parameter values as well as exploring boundary conditions.

b.     Parameter combination:
Generating interesting parameter value combinations for calls with two or more parameters.

c.     Setting environment:
Determining the content under which an API call is made. This might include setting external environment conditions         (files, peripheral devices, and so forth) and also internal stored data that affect the API.

d.     Call sequencing:
Sequencing API calls to vary the order in which the functionality is exercised and to make the API produce useful results     from successive calls.

Most common seniors in API Testing:

a.     Test Response: Each API method in isolation using only the mandatory elements and typical content so if system calls        any API then it send any response so tester need to check valid response from API.

b.     Test Limit: These tests exercise each API method using all optional elements and maximum allowable content lengths        and/or instances of repeated elements.

c.     Test Business Logic: This is where the business application logic is simulated in the test code. Each API method will         have a defined set of test cases that explore its interaction and influence on other API methods and any underlying         database.

d.     Test Negative or Illegal: These tests contain a sampling of typical error scenarios, such as missing required elements,     empty content, and content exceeding maximum limits, across a representative sampling of the API methods, so API         should enough intelligent to handle exception, errors and missing parameters issues.

e.     Test Load: Most of applications are web 2.0 and running on internet and access by number of users so API should able     to take heavy load and should not break during peak time when number of people will use application, tester also             measure response time, through put , latency, memory leak and any other factor according to requirement

If you were to ask testers how to test API, you would get several different perspectives. Every people have different thinking and approaches to solve problems and every approach have option for enhancements.

Test Plan reflects your entire project testing schedule and approach.

Test Plan is a management planning document that describes:

1. How the testing will be done?

2. Who will do it?

3. What will be tested?

4. How long it will take?

5. What the test coverage will be?

Test plan document formats can be as varied as the products and organizations to which they apply, but there are three major elements of a test strategy that should be described in the test plan:

Test Coverage - Test coverage in the test plan states what requirements will be verified during what stages of the product life

Test Methods - Test methods in the test plan state how test coverage will be implemented

Test Responsibilities - Test responsibilities include what organizations will perform the test methods and at each stage of the product life

IEEE Standard Test Plan Structure (IEEE 829-1998)

This standard specifies the following test plan outline:

- Test plan identifier

-  Introduction

-  Test items

-  Features to be tested

- Features not to be tested

- Approach

- Item pass/fail criteria

- Suspension criteria and resumption requirements

- Test deliverables

- Testing tasks

- Environmental needs

- Responsibilities

- Staffing and training needs

- Schedule

- Risks and contingencies

- Approvals

Test Plan Identifier:

- A unique identifier

Introduction:

- Summary of the items and features to be tested

- Need for and history of each item (optional)

- References to related documents such as project authorization, project plan, QA plan, configuration management plan, relevant policies, relevant standards

- References to lower level test plans

Test Items:

- Test items and their version

- Characteristics of their transmittal media

- References to related documents such as requirements specification, design specification, users guide, operations guide, installation guide

- References to bug reports related to test items

- Items which are specifically not going to be tested (optional)

Features to be Tested:

- All software features and combinations of features to be tested

- References to test-design specifications associated with each feature and combination of features

Features Not to Be Tested:

- All features and significant combinations of features which will not be tested

- The reasons these features won’t be tested

Approach:

- Overall approach to testing

- For each major group of features of combinations of features, specify the approach

- Specify major activities, techniques, and tools which are to be used to test the groups

- Specify a minimum degree of comprehensiveness required

- Identify which techniques will be used to judge comprehensiveness

- Specify any additional completion criteria

- Specify techniques which are to be used to trace requirements

- Identify significant constraints on testing, such as test-item availability, testing-resource availability, and deadline

Item Pass/Fail Criteria:

- Specify the criteria to be used to determine whether each test item has passed or failed testing

Suspension Criteria and Resumption Requirements

- Specify criteria to be used to suspend the testing activity

- Specify testing activities which must be redone when testing is resumed

Test Deliverables:

- Identify the deliverable documents: test plan, test design specifications, test case specifications, test procedure specifications, test item transmittal reports, test logs, test incident reports, test summary reports

- Identify test input and output data

- Identify test tools (optional)

Testing Tasks:

- Identify tasks necessary to prepare for and perform testing

- Identify all task interdependencies

- Identify any special skills required

- Environmental Needs

- Specify necessary and desired properties of the test environment: physical characteristics of the facilities including hardware, communications and system software, the mode of usage (i.e., stand-alone), and any other software or supplies needed

- Specify the level of security required

- Identify special test tools needed

- Identify any other testing needs

- Identify the source for all needs which are not currently available

Responsibilities:

- Identify groups responsible for managing, designing, preparing, executing, witnessing, checking and resolving

- Identify groups responsible for providing the test items identified in the Test Items section

- Identify groups responsible for providing the environmental needs identified in the Environmental Needs section

Staffing and Training Needs:

- Specify staffing needs by skill level

- Identify training options for providing necessary skills

Schedule:

- Specify test milestones

- Specify all item transmittal events

- Estimate time required to do each testing task

- Schedule all testing tasks and test milestones

- For each testing resource, specify its periods of use

Risks and Contingencies:

- Identify the high-risk assumptions of the test plan

- Specify contingency plans for each

Approvals:

- Specify the names and titles of all persons who must approve the plan

- Provide space for signatures and dates

What is Usability Testing?

Usability testing is a technique for ensuring that the intended users of a system can carry out the intended tasks efficiently, effectively and satisfactorily.

When is Usability testing appropriate?

Usability testing it is carried out pre-release so that any significant issues identified can be addressed.
Usability testing can be carried out at various stages of the design process. In the early stages, however, techniques such as walkthroughs are often more appropriate.
Usability testing is not a substitute for a human centered design process.

What are the things keep in mind while doing Usability Testing?

There is no such specific guideline to follow; I am mentioning some points which helpful to get overall perspective of the software while testing.
Tester should always keep in mind that System or application fulfill following criteria while developing:

Mirror of Real World
Try to get 100% fitness
Work for Real customer
Believe in KEIS (Keep It Simple)
Prevention is better than cure
Consistent in everywhere
Ready to help

Mirror of Real World:
systems should be mirror of real world means it should be user centric and speak world language for example while developing a banking application need to use word which is used worldwide like credit and debit rather than borrow or any some other phrase.
So before start testing get some domain knowledge and prepare glossary for the domain and check that application is it fit with real world or not?

Try to get 100% fitness:
Always keep in mind that there is no 100% error free screens.
Some time usability designer and product management team has lack of domain knowledge or they are not so close to end customers so sometime application screens are very far away from real problem and do not fulfill all expectations, so always think around the problem and end customer and try to achieve 100% fitness.

Work for Real customer:
while testing an application behave and think like end customer and try to identify all representative tasks. Ideally, these should derive from scenarios or use cases developed earlier in the design process. Have a clear definition of when a task is complete.

Believe in KEIS (Keep It Simple):
Things always keep simple in application. Flow of application, business logic and all UI (User Interface) Screens should be simple and understandable for layman. Screen should be clearly defined and mirror of deliverables, Don’t provide too many information in screen, Just deliver all necessary information which is required to perform particular task on the screen.

Prevention is better than cure:
Prevention of error/exception always better then cure so application should be designed as a smart systems where all errors and exceptions handled properly, Application should have sufficient error message and warning messages which describe exact input and desired output of the systems.

Consistent in everywhere:
application should follow platform conventions, all UI screens should be simple and consistent so user always be in same platform and able to understand use of application.
All Error messages and other messages, guidelines and business term should be consistent and reflect same meaning.

Ready to help:
Application provides help document to end user so they can able to understand application and make it more useful in their life. System should display help message or guidelines for valid input also show proper error message and wait messages if any process is running behind the application.

These are ten general principles for user interface design. They are rules of thumb in usability guidelines.

Visibility of system status

The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.

Match between system and the real world

The system should speak the users’ language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.

User control and freedom

Users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

Consistency and standards

Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.

Error prevention

Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.

Recognition rather than recall

Minimize the user’s memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

Flexibility and efficiency of use

Accelerators — unseen by the novice user — may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.

Aesthetic and minimalist design

Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

Help users recognize, diagnose, and recover from errors

Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

Help and documentation

Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.

Algorithm is set of rules to perform certain operation in certain conditions.
While creation of algorithms developers follow some rules and mathematic computations which calculate or predict the application behavior in certain condition.
Testing an algorithms are not so simple, testing of algorithms need same kind of attention and treatment as it gets while creation.

As I spoke in my previous article, Tester is responsible for deliverables, same rule apply here also because most of time after implementation of algorithms no body bother to find exact working of algorithms.
So success of algorithms not only depends on development team but also on testing team.

There is no exact rules to test any algorithms, its depend on case by case and how you approach to find solution.
I am trying to figure out some set of guidelines which can be helpful while testing an algorithm

Whenever you get some task then approach should be:

Read as much as you can
Start Discussion with Developer team
Analyze Conditions and scope of algorithms
Find Right Approach
Create your own data set
Start Testing
Interact with Dev team while testing

Read as much as you can:
The fundamental rules of testing says, whenever any testing comes to the door tester should read end to end specification or use case document but in algorithm testing tester need to pay more attention to understand exact rules and calculations, Don’t read formally, read 2-4 times, research on local resource or internet and read same kind of algorithms, as much you read as much you understand of algorithms.

(Note: Here calculations mean mathematical, logical and conditional rules and computations.)

Start Discussion with Developer team:
So you did you home work and now it is time to give some shape of your effort so start talking with dev team and try to convey what you understand by design doc. Start discussion and actively participle and voice your concern.
Some time dev team did not mentioned all conditions and limitations of algorithms so ask all conditions and limitations also ask boundary conditions and do some boundary value analysis of algorithms.
As more you ask as more you get from them and as more you get as more you understand.

Analyze Conditions and scope of algorithms:
Third step is to analyze scope of the algorithm and to evaluate all conditions, evaluate all inner conditions, outer conditions and boundary conditions. Most of algorithms fail or does not work properly on boundary conditions or on set of boundary data. So tester need to pay attention to find the area where algorithms can be leak.

Find Right Approach
Take some time and try to find right approach as solution, there are many way to solve any problem just try to figure out which is most suitable and simple method for you to cover all scenario because some time we know which is the more flexible and fast method but don’t approach that method if you’re not so comfortable with it.
I am saying again we are going to test an algorithm which is one time job so don’t try to find any shortcut. Adopt that approach where you are comfortable, no issues if take more time and effort but end of the work you will be assure about correctness and precision of the algorithms.

Thumb rule: don’t try to find any shortcuts.

Create your own data set:
Now be relax, you did 60% of your task. Now think and evaluate properly and create you data set where you will test algorithm. Success of testing within the scheduled time depends on how efficiently you create your data set. Cover all possible conditions, especially boundary conditions.
Prepare test result so you can compare algorithms output from known results.

Start Testing
Start testing on single true conditions and then start testing on multiple conditions and rules,
don’t forget to include negative test cases, negative or wrong testing approach give an idea that how algorithm is robust and can able to handle all exception properly or not?

Interact with Dev team while testing
Interact with dev team and describe your approach include following points
1. How you tested?
2. What are the test data and test results?
3. How you compare results?
4. What are conclusions?

So you will get the idea how things are going and you are in right direction or not?
Take feedback from them and ask to advice so you can make better test case and do more carful testing because developer is the guy who knows how algorithms should behave in all conditions and exceptions.

Please feel free to highlight any hide area where need to more focus, I will appreciate your feedback.