Question:
What are the differences between sanity testing, smoke testing,
regression testing, and monkey testing?
Posted by prasad
Answer:
First, quite often the terms are not used correctly and you should learn
the meaning of the terms where you work and try to help adjust them to
industry norms. However, these terms may be used differently in your company since there are no formal standards for them, and most people wouldn't bother to learn the actual definitions if they existed.
Smoke tests get their name from the electronics industry. The circuits are
laid out on a bread board and power is applied. If anything starts smoking,
there is a problem. In the software industry, smoke testing is a shallow and
wide approach to the application. You test all areas of the application
without getting too deep. This is also known as a Build Verification test or
BVT.
In comparison, sanity testing is usually narrow and deep. That is they look
at only a few areas but all aspects of that part of the application. A smoke
test is scripted--either using a written set of tests or an automated
test--whereas a sanity test is usually unscripted.
A monkey test is also unscripted, but this sort of test is like a room full
of monkeys with a typewriter (or computer) placed in front of each of them.
The theory is that, given enough time, you could get the works of
Shakespeare (or some other document) out of them. This is based on the idea that random activity can create order or cover all options. Jamie Andrews
has been doing some work on an organised version of this at the University
of Western Ontario, London, Ontario, Canada. This method is called
coverage-checked random unit testing (CRUT). You can read about it at
http://www.csd.uwo.ca/faculty/andrews/papers/index.html.
Gorilla testing however has nothing to do with this. It is an intense round
of testing--quite often redirecting all available resources to the activity.
The idea here is to test as much of the application in as short a period of
time as possible.
Just to clarify here: a Sanity test is not the same as a Smoke Test or a
Build Verification test. The former is to determine a small section of the
application is still working after a minor change (which is not a good
policy, btw--you should do a regression test instead) and a Smoke or Build
Verification Test (BVT) is designed to touch every part of the application
in a cursory way. It's is shallow and wide. A Sanity test tends to be
unscripted, but you could (or should!) use a sub-set of your existing test
cases to verify a small part of your application. This is not quite a
regression test, where all areas of the application are tested using a
subset of the test cases.
A regression test is a more complete Smoke Test or BVT. A sanity test is a narrow regression test that focuses on a one or a few areas of functionality.
Posted by Walter Görlitz
Showing posts with label software testing. Show all posts
Showing posts with label software testing. Show all posts
Tuesday, August 12, 2008
What is Gray Box Testing
Question:
What is Gray Box Testing? Where & how Gray Testing is used?
Posted by ashu_akl
Answer:
Let's begin with the most basic forms of testing: white box, or glass box
versus black box testing: I will quote from Testing Computer Software:
Second Edition (1993) Cem Kaner, Jack Falk, Hung Quoc Nguyen.
Glass box testing is distinguished from black box testing, in which the program is treated as a black box. You can't see into it. The tester (or programmer) feeds it input data, observes output data, but does not know, or pretends not to know, how the program works. The test designer looks for interesting input data and conditions that might lead to interesting outputs. Input data are "interesting" representatives of a class of possible inputs if they are the ones most likely to expose an error in the program.
In contrast, in glass box testing, the programmer uses her understanding and access to the source code to develop test cases.
At this point, on page 41of Testing Computer Software, they go into explaining the benefits of white box testing. They also spend the next few pages going over the concepts of white box testing.
In recent years I have also heard of gray box testing. In this form of testing, the tester has access to some of the inner workings of the system, usually the database, but not the code.
White box testers have access to the code, but even a black box tester canknow the branches of code--the rules within the code that cause operations to fork. A white box tester generally uses the code, and the ability to create drivers and stubs to test the code directly. They do not rely on the UI to do it.
The typical gray box tester is permitted to set up his testing environment,
like seeding a database, and can view the state of the product after their
actions, like performing a SQL query on the database to be certain of the
values of columns. It is used almost exclusively of client-server testers or
others who use a database as a repository of information, but can also apply to a tester who has to manipulate XML files (DTD or an actual XML file) orconfiguration files directly.
The true black box tester looks only at the GUI and can not touch intermediate files, registry entries, databases, etc., nor are they permitted to see the results their actions have wrought, other than through the UI. They are, therefore, only permitted to use the UI to do their testing.
Posted by Walter Görlitz
What is Gray Box Testing? Where & how Gray Testing is used?
Posted by ashu_akl
Answer:
Let's begin with the most basic forms of testing: white box, or glass box
versus black box testing: I will quote from Testing Computer Software:
Second Edition (1993) Cem Kaner, Jack Falk, Hung Quoc Nguyen.
Glass box testing is distinguished from black box testing, in which the program is treated as a black box. You can't see into it. The tester (or programmer) feeds it input data, observes output data, but does not know, or pretends not to know, how the program works. The test designer looks for interesting input data and conditions that might lead to interesting outputs. Input data are "interesting" representatives of a class of possible inputs if they are the ones most likely to expose an error in the program.
In contrast, in glass box testing, the programmer uses her understanding and access to the source code to develop test cases.
At this point, on page 41of Testing Computer Software, they go into explaining the benefits of white box testing. They also spend the next few pages going over the concepts of white box testing.
In recent years I have also heard of gray box testing. In this form of testing, the tester has access to some of the inner workings of the system, usually the database, but not the code.
White box testers have access to the code, but even a black box tester canknow the branches of code--the rules within the code that cause operations to fork. A white box tester generally uses the code, and the ability to create drivers and stubs to test the code directly. They do not rely on the UI to do it.
The typical gray box tester is permitted to set up his testing environment,
like seeding a database, and can view the state of the product after their
actions, like performing a SQL query on the database to be certain of the
values of columns. It is used almost exclusively of client-server testers or
others who use a database as a repository of information, but can also apply to a tester who has to manipulate XML files (DTD or an actual XML file) orconfiguration files directly.
The true black box tester looks only at the GUI and can not touch intermediate files, registry entries, databases, etc., nor are they permitted to see the results their actions have wrought, other than through the UI. They are, therefore, only permitted to use the UI to do their testing.
Posted by Walter Görlitz
Zero-Defect Software Development
by Steve Pavlina Dexterity Software
In my corner of the software industry, I will work from six months to a couple years to develop and release a new computer game. I can sell sequels or expansion packs, but I generally cannot sell upgrades as with other software. When I release a new game, I must make sure it is of very high quality because my users won't think to look for an upgrade. Because of the short lifespan of most computer games and their dependence on transient technology, my opportunities to improve a released product based on customer feedback are minimal. To deal with these issues, I gradually adopted a system of Quality Assurance (QA) practices that allowed me to significantly increase product quality while simultaneously reducing development time.
Not to be taken as meaning "bug-free," Zero-Defect Software Development (ZDSD) is a practice of developing software that is maintained in the highest quality state throughout the entire development process. "Defects" are aspects of the evolving software that would not be suitable for the final product as-is. This broad definition includes bugs as well as unwanted deviations from the desired final outcome. Defects in the development of a computer game would include unpolished artwork, an unacceptably low frame rate on the target system, levels that aren't fun enough, or any number of unfinished features.
The basic tenet of ZDSD is this: Maintain your product in what you believe to be a defect-free state throughout the development process. This sounds simple, but it is a rare practice. The most common approach is to delay major testing until the final QA phase of software development, where defects are often discovered for the first time. Most bugs are not detected or fixed until long after their introduction. The longer a defect remains, the harder it is to fix. On large software products, each stage of development that a defect survives will increase the cost of fixing the defect by ten to fifty times. A defect introduced in the design phase can cost hundreds of times more to fix in the testing phase than it would if fixed immediately after its introduction.
By focusing on product quality throughout the development lifecycle, you will actually complete products faster than if you didn't pay attention to quality until the end of the project. The general rule of software quality is counter-intuitive: Improving quality actually reduces development time. This is because you eliminate all the time spent fixing bugs and reworking code, which can account for as much as 50% of development costs on a large project. The typical programmer writes between eight and twenty lines of code a day; the rest of the day is usually spent on debugging. ZDSD shortens schedules by eliminating most debugging time. Extensive studies done at NASA, IBM, and elsewhere have shown that better QA leads to shorter schedules. An IBM study concluded that software projects that make quality a top priority typically have the shortest schedules, the highest productivity, and even the best sales.
Here are the ten basic rules of ZDSD:
1. Test your product every day as you develop it, and fix defects as soon as you find them. Apply the daily build and smoke test. At the end of every day you work on your project, build the current version of your software, and test it for basic functionality. Microsoft enforces this policy religiously, using large teams to build each project on a daily basis. A programmer whose code breaks the build may be called in the middle of the night and must go back to work to fix the problem immediately. For shareware authors working on small projects, this is far easier. At the end of each day, test your program for at least ten minutes. Make a list of anything you would consider a "defect," and resolve to fix all defects before implementing any new features. Once you find a defect, fixing it becomes your number one priority, and you avoid writing any new code until the defect is 100% eliminated.
2. Review your code regularly. When most people think of QA, they think of testing, but testing is actually one of the least cost-effective strategies for finding bugs. The most rigorous testing will typically find less than 60% of all bugs in a program, and there are certain types of bugs that testing will rarely find. Studies conducted at many large software organizations have concluded that code inspections are far more cost-effective than testing. A NASA study found that code reading detected almost twice as many defects per hour as testing. Whenever you've added a few hundred lines of new code to your project, set aside an hour or two to read over your work and look for mistakes. One hour of code review is equivalent to two or more hours of methodical testing. As you gain experience, keep a list of the types of defects you find, and run down your list whenever reviewing new code. To find even more defects, have someone else read your code as well.
3. Rewrite poor-quality modules. When you discover an obscure new bug, do you ever pray, "Oh no! Please don't let it be in that module!" We all have monster modules of legacy code that were written when we weren't such seasoned programmers as we are today. Don't fear them; rewrite them. Often a better approach will only become clear when an inferior solution has already been implemented. This is certainly true for John Carmack, who coded dozens of different approaches when writing the Quake engine before discovering one that met his criterion. Defects will not be distributed evenly across your code. You will typically find that 20% of your routines are responsible for 80% of your errors. In my programs it is normally the modules that interact with the hardware or with third-party drivers, especially DirectX, that are the most buggy. Raise your standards for those modules that seem to produce a never-ending supply of bugs, and take the time to rewrite them from scratch. You may find that other intermittent bugs disappear completely as a result.
4. Assume full responsibility for every bug. 95% of all software defects are caused by the programmer. Only 1% of defects are hardware errors, and the remaining 4% are caused by the compiler, the OS, or other software. Never dismiss a potential bug; find out the exact cause of any anomaly. When the Mars probe suffered serious software glitches during its mission, it was learned that the same glitch had occurred only once during testing on earth, but the engineers dismissed it as a temporary hardware hiccup. Unless your hardware drinks soda, it does not hiccup.
5. Handle change effectively. You will always think of great new features to add after you have started coding. Carefully consider how each change will impact your pre-existing code. Poor integration of unanticipated features is a major cause of defects.
6. Rewrite all prototyping code from scratch. Sometimes you may quickly prototype a new feature to see if it will be viable. Often this is done by sacrificing code quality in the name of rapid development. If you eventually decide to keep the feature, it is very tempting to simply tack on some basic error checking to the prototyping code. Don't fall into this trap. If you weren't writing the code originally with quality as a priority, scrap the prototyping code, and re-implement the feature from scratch. Rapidly prototyped features that slip into the final product are a major source of bugs because they are not subject to the same quality standards as the rest of the code.
7. Set QA objectives at the beginning of every project. Studies have shown that developers who set reasonable QA goals will usually achieve them. Decide in advance if your product must be fast, small, feature-rich, intuitive, scalable, etc. Then prioritize those objectives. When designing the interface code for an upcoming game, I decided that my top three priorities were to make it beginner-intuitive, fast, and fun, in that order. Consequently, my game's interface isn't as graphically rich as other games, but it is easier to use and faster than any other game of its type. Whenever you have to make a design decision, keep your objectives in mind. If you do not set clear QA goals, then you are doomed to accept the results of random chance.
8. Don't rush debugging work. Fully 50% of all bug fixes are done incorrectly the first time, often introducing new bugs in the process. Never experiment by simply changing "x-1" to "x+1" to see if that will do the trick. Take the time to understand the source of the bug. Years ago when I was a boy scout and had to put out a campfire, the Scoutmaster would sometimes test my thoroughness by asking me to put my hand in the ashes. I learned very quickly how to put out a fire so well that I had complete confidence it was 100% extinguished. When you find a defect, it means your code is on fire. As long as the defect remains, any new code you write will add fuel to that fire. Whenever you find a defect, drop everything to fix it, and don't move on until you are 100% confident that your fix is correct. If you don't take the time to do it right the first time, when will you find the time to do it over?
9. Treat the quality of your code at the same level of importance as the quality of your product. Rate your code on a scale of one to ten for overall quality. The first time I did this, I rated my 30,000-line project as a four. I rewrote the worst of the code until I reached an eight overall. It was one of the best investments of time I ever made because I was then able to add new features at double my previous rate. The quality of your code is highly indicative of the quality of your product. You may find as I have that your best selling products also receive your highest ratings for code quality.
10. Learn from every bug; each one represents a mistake that you made. Learn why you made each mistake, and see if you can change something about your development practices to eliminate it. Over the years I have adopted many simple coding practices that allow me to avoid common bugs that used to plague me. There are many types of bugs that I now never encounter because my coding style makes it physically impossible for me to introduce them.
Each of these rules represents a simple concept, but their combined benefits are significant. You will achieve higher progress visibility, avoiding the situation of being "99% done" for the last 80% of your development time. Higher quality will make your products easier to maintain and less expensive to support. You will spend less time debugging old code and more time writing new code. And most importantly, it actually takes less time to write high-quality code than it does to write low-quality code, so you will save a great deal of time on overall development. If you have never developed products with a zero-defect philosophy from day one, its adoption can reduce your development time for new products by 30% or more while simultaneously improving product quality.
Testing Reviews
| A | B - C | D - E | F - G | H - I | J - K - L - M - N - O |
A
B - C
D - E
F - G
H - I
J - k - l - m - n - O
P
Q
R
S
T - u - v - w - x - y - Z
| P | Q | R | S | T - U - V - W - X - Y - Z |
A
| Build Acceptance Test | The build acceptance test is a simplistic check of a product's functionality in order to determine if the product is capable of being tested to a greater extent. Every new build should undergo a build acceptance test to determine if further testing can be executed. Examples of a build acceptance: Product can be installed with no crashes or errors that terminate the installation. (Development needs to install the software from the same source accessed by QA (e.g. Drop Zone, CD-ROM, Electronic Software Distribution archives, etc.). Clients can connect to associated servers. Simple client server communication can be achieved |
| Bottom - up | Start testing with the bottom of the program. The bottom - up strategy does not exist until the last module is added. |
| CET (Customer Experience Test) | An in-house test is performed before the Alpha, Beta, and FCS milestones which is used to determine whether the product can be installed and used without any problems, assistance, or support from others. |
| Client-Server Test | Testing Systems that operate in client/server environments. |
| Compatibility Test | This test is used to test compatibility between different client/server version combinations as well as other supported products. |
| Confidence Test | The confidence test ensures a product functions as expected by ensuring platform specific bugs are not introduced and functionality has not regressed from drop to drop. A typical confidence test is designed to touch all major areas of a product's functionality. These tests are run regularly once the Functional Freeze milestone is reached throughout the remaining development cycle. |
| Configuration Tests | These tests are run for product testing across various system configuration combinations. Examples of configurations: Cross platform (e.g. Windows Clients against a UNIX server). Client/server network configurations. Operating systems and database combinations (also including version combinations). Web servers and web browsers (for web products). The system configurations to test are determined from the product's compatibility matrix. This test is sometimes called a 'Platform test'. CET (Customer Experience Test) An in-house test is performed before the Alpha, Beta, and FCS milestones which is used to determine whether the product can be installed and used without any problems, assistance, or support from others. |
Depth Test | The depth tests are designed to test all the product's functionality in-depth. |
| Error Test | The error test is designed to test the dialogs, alerts, and other feedback provided to the user when an error situation occurs. The difference between this test and a Negative Test is that an Error Test is simply verifying that the correct dialogs are seen. The Negative Test is primarily looking at the robustness and recovery facets. |
| Event-Driven | Testing event-driven processes, such as unpredictable sequences of interrupts from input devices or sensors, or sequences of mouse clicks in a GUI environment. |
| Final Installation Test | Verification that the final media, prior to hand off to Operations for duplication, contains the correct code which was previously tested and is installable on all the supported platforms and databases. The product demo is executed and product Release Notes verified. |
| Functionality Test | This is designed to test the full functionality, features, and user interfaces of software based upon the functional specifications. |
Full Test | A full test is Build Acceptance + Sanity + Confidence + Depth. This is designed to test the full functionality, features, and user interfaces of software based upon the functional specifications. |
| Graphical User Interface (GUI) | Testing the front-end user interfaces to applications which use GUI support systems and standard such as MS Windows or Motif. |
| GUI Roadmaps | Step by step walk through of a tool or application, exercising each screen or window's menus, toolbar and dialog boxes to verify the execution and correctness of the Graphical User Interface. Typically, this is handled by automated scripts and rarely is used as a manual tests due to the low numbers of bugs found from them. |
| Module testing | To test large program its necessary to use module testing. Module testing (or unit testing) is a process of testing individual subprograms (small blocks), rather than testing the program as a whole. Module testing eases the task of debugging. When error is found, it is known is which particular module it is. |
| Multi-user Test | Test maximum number of users specified in the design concurrently, to simulate the real user environment when they use the product. |
| Negative Test | Tests that deliberately introduce an error to check an application's behavior and robustness. For example, erroneous data may be entered, or attempts made to force the application to perform an operation that is should not be able to complete. Generally a message box is generated to inform the user of the problem. If the program terminates, the program should exit gracefully |
| Object-Oriented | Testing systems designed or coded using an object-oriented approach or development environment, such as C++ or Smalltalk. |
| Parallel Testing | Testing by processing the same (or at least closely comparable) test workload against both the existing and new versions of a product, then comparing results. |
| Performance | Measurement and prediction of performance (e.g. Response time and/or throughput) for a given benchmark workload. Phased Approach A testing strategy where test cases are developed in stages so a minimally acceptable level of testing can be completed at any time. As new features are coded and frozen, they receive priorities for a given amount of time-so that a concentrated effort is directed toward testing those new features before the effort returns to validate the preexisting functionality. When no new features are available, preexisting features will be targeted-with priorities set by Project Leads. 1st level - Minimal Acceptance Test 2nd level - Confidence Tests 3rd level - Full Functionality Test 4th level - Error, Negative, and other Tests 5th level - System level tests |
| Phased Approach | A testing strategy where testcases are developed in stages so a minimally acceptable level of testing can be completed at any time. As new features are coded and frozen, they receive priorities for a given amount of times-so that a concentrated effort is directed toward testing those new features before the effort returns to validate the preexisting functionality. When no new features are available, preexisting features will be targeted-with priorities set by Project Leads. 1st level - Build Acceptance Test 2nd level - Sanity Test 3rd level - Confidence Test 4th level - Depth Test 5th level - Error, Negative, and other Tests 6th level - System level tests |
| Sanity Test | Sanity tests are subsets of the confidence test and are used only to validate high-level functionality. |
| Security Testing | It is a test how easy to break program's security system. |
| Stress Test | These tests are used to validate software functionality at the limit (e.g. Maximum throughput) and then testing at and beyond these limits. |
| System Level Test | These tests check for factors such as Cross-Tool testing, memory management and other operating system factors. |
| Top - Down strategy | Start testing with the top of the program. |
| Volume Testing | Is the process of feeding a program with heavy volume of data. |
| Usability | The effectiveness, efficiency, and satisfaction with which specified users can achieve specified goals in a particular environment. Synonymous with "ease of use". |
| |
Test Methods
Black - Box Testing
In using this strategy, the tester views the program as a black - box, tester doesn't see the code of the program: Equivalence partitioning, Boundary - value analysis, Error guessing.
A collection of test cases or scripts.
In using this strategy, the tester views the program as a black - box, tester doesn't see the code of the program: Equivalence partitioning, Boundary - value analysis, Error guessing.
White - Box Testing
In using this strategy, the tester examine the internal structure of the program: Statement coverage, Decision coverage, condition coverage, Decision/Condition coverage, Multiple - condition coverage.
Gray - Box Testing
In using this strategy Black box testing can be combine with knowledge of database validation, such as SQL for database query and adding/loading data sets to confirm functions, as well as query the database to confirm expected result.
Test Script
Type of test file. It is a set of instructions run automatically by a software or hardware test tool.
A collection of test cases or scripts.
Testing Risk Assessment
INTRODUCTION
The purpose of business risk analysis during software testing is identifying high-risk applications, which must be tested more thoroughly, and to identify error-prone components within specific applications which must be tested more rigorously. The results of the analysis can be used to determine the testing objectives for the application under test during test planning.
The key to performing a successful risk analysis is to formalize the process. An informal approach to risk analysis methods leads an ineffective analysis process.
The first is Judgment And Instinct. This is the most common approach to performing risk assessment during testing. It is an informal approach using the tester's Knowledge, and experience with past projects, to estimate the amount of testing required for the current project. This can be an effective method, but its primary fault the fact that the tester's experience is not readily transferable to other testers. It is a repeatable approach, but is not formally written down for others to use.
The second method is Dollar Estimation. This approach quantifies business risk using dollars as its unit of measure. This approach requires a great deal of precision and is difficult to use because results are base on estimates of frequency of occurrence and the loss per occurrence. Thus, the precision is not always there.
The third approach is Identifying And Weighting Risk Attributes. This approach identifies the attributes that cause a risk to occur. The relationship of the attributes to the risk is determined by weighting. The tester uses weighted numerical scores to determine what areas are at the most risk. The scores can be used to determine what application components should be tester more thoroughly than others, and total weighted scores can be used to compare one application to another.
The fourth approach is The Software Risk Assessment Package. This is an automated approach, which involves purchasing an assessment package from a vendor. The advantage is the ease of use and the ability to do What-If-Analysis with risk characteristics. Automated risk assessment software packages exist which use the second and third approaches above, however, testers can create their own risk assessment questionnaires with MS Word and do the what-if-analysis with MS Excel. This is the approached used here.
There are three identified risk dimensions. They are Project Structure, Project Size, and Experience with Technology.
With respect to project structure, the more structured a project the less the risks. Thus, software development projects, which employ some type of project management/ development life cycle approach, should be at less risk.
Project size is directly proportional to risk. The larger the project in terms of cost, staff, time, number of functional areas involved, etc., the greater the risk.
Technology experience is inversely proportional to project risk. The more experience the development team has with the hardware, the operation system, the database, the Network, and the development language the less the risk.
THE METHOD
Suggested risk assessment approach is a five-step risk assessment methodology. The steps are:
1. Ascertain the risk score
2. Create the risk profile
3. Modify the contributing risk characteristics
4. Allocate test resources
5. Create a risk database
Ascertaining the risk score.
This involves conducting a risk assessment review with the development team. During the session a risk assessment questionnaire is used to structure the process. Each question is asked of the group and a consensus is reached as to the perceived level of risk. The questions are closed-end questions with the possible responses of Low (one point), Medium (2 points), High (3 points), and Not Applicable (0 points). The weighted scores can be used to identify error-prone areas of the application and to compare the application with other applications.
Creating the risk profile.
A weighting system is used to compute a score that reflects the importance of each area. Areas, which are twice as important, can be weighted with a value of two (e.g. an area with medium risk (2 points) could be considered three times as important as the other areas and have a final weighted score of 6 points (the weight times the risk points)). A total score is computed for the project, but the individual scores are used to develop the risk profile. Perry is not specific is his description of how to construct the profile. So, I suggest using the following approach.
Once the risk data have been collected, create a spreadsheet, which computes the weighted scores. Sort the tabulated scores from the highest to the lowest (a pseudo-frequency distribution of sorts) and perform a Pareto Analysis (the 80/20 rule) to determine what project areas fall into the upper 20% of the distribution. These are the areas, which are at risk the most and they are the areas, which will need to be, tested the most. The results are rather obvious when charted using Kiviat Charts (radar charts). The high risk areas standout visually. The charting should be done based on data sorted in ascending order by question number not on data sorted in descending order by highest to lowest.
Modify the risk characteristics.
Perry argues that once the areas at risk have been identified a proactive approach can reduce risk. He suggests that steps be taken to change the development approach or the project structure in order to reduce risk. When these alternatives are not feasible, the process of using the risk information to decide what areas to test becomes even more critical.
Allocate test resources.
Allocating the most test resources to the high-risk areas, allocating less testing resources to medium risk areas and minimal testing resource to low risk testing areas is suggested. A sound strategy is to assure that all of, or as must of as is possible, of the medium to high risk areas are tested with the scope of the allotted testing resources. A possible split could be to commit 80% of the testing resources to medium and high-risk areas, and commit the remaining 20% to low risk areas. This is again applying the 80/20 rules.
Compile a risk assessment database.
A risk assessment database has two important functions. First, it can be used to improve the risk assessment process itself. Second, the data can be used to help management plan and structure development projects.
THE RISK ASSESSMENT REVIEW SESSION
Perry says that the entire test team along with end user representatives should be included in the session. He recommends conducting the session early in the test process. It should last no more than two hours. It should be run formally by the test team manager who facilitates the session. The session should have two objectives. The first objective is to answer each question on the risk assessment questionnaire. The second is to brainstorm and let the participants voice their concerns about the system under development.
RISK ASSESSMENT
A risk Assessment should be completed for the Ask Jeeves International projects. It was conducted following the guidelines below. Risk assessment was used to conduct a one-hour assessment of the project's risk factors. The risk assessment session was conducted with one QA person and five XYZ project development team members present.
The raw data were placed in an Excel workbook, and weighted scores were calculated for each question in each of the Test Documents. The data were analyzed using Pareto Analysis to determine the number of questions to consider (i.e the top 20% in terms of risk). The data were displayed through Kiviat charts. The results revealed three major areas of risk. The first is the lack of user documentation about the process being automated. The second is number of interfaces to other systems. The third is the use of new technology to "pioneer" parts of the system.
Use risk analysis to determine where testing should be focused. Since it's rarely possible to test every possible aspect of an application, every possible combination of events, every dependency, or everything that could go wrong, risk analysis is appropriate to most software development projects. This requires judgment skills, common sense, and experience. Considerations can include:
- Which functionality is most important to the project's intended purpose?
- Which functionality is most visible to the user?
- Which functionality has the largest safety impact?
- Which functionality has the largest financial impact on users?
- Which aspects of the application are most important to the customer?
- Which aspects of the application can be tested early in the development cycle?
- Which parts of the code are most complex, and thus most subject to errors?
- Which parts of the application were developed in rush or panic mode?
- Which aspects of similar/related previous projects caused problems?
- Which aspects of similar/related previous projects had large maintenance expenses?
- Which parts of the requirements and design is unclear or poorly thought out?
- What do the developers think are the highest-risk aspects of the application?
- What kinds of problems would cause the worst publicity?
- What kinds of problems would cause the most customer service complaints?
- What kinds of tests could easily cover multiple functionalities?
- Which tests will have the best high-risk-coverage to time-required ratio?
GENERAL RISKS LIST
- Complex - anything disproportionately large, intricate or convoluted.
- New - anything that has no history in the product.
- Changed - anything that has been tampered with or "improved".
- Upstream Dependency - anything whose failure will cause cascading failure in the rest of the system.
- Downstream Dependency - anything that is especially sensitive to failures in the rest of the system.
- Critical - anything whose failure could cause substantial damage.
- Precise - anything that must meet its requirements exactly.
- Popular - anything that will be used a lot.
- Strategic - anything that has special importance to your business, such as a feature that sets you apart from the competition.
- Third-party - anything used in the product, but developed outside the project.
- Distributed - anything spread out in time or space, yet whose elements must work together.
- Buggy - anything known to have a lot of problems.
- Recent Failure - anything with a recent history of failure.
REFERENCES
Perry, William E. A Standard For Testing Application Software 1992. Auerbach Publishers, Boston, 1992.
Testing GUI
shared by Mikhail Rakhunov SQAtester.com contributor
To test any Windows based application following points are to be considered:
- It is essential to have GUI consistency within the Application.
- Should be alike in look and feel as per any other standard Window software.
- Should have standard set of keys implemented for the software.
- Should have clean and neat exit.
While testing any Windows based application the testing can be broadly categorized into following compartments. They are:
- Standardization Testing.
- GUI Testing.
- Validation Testing.
- Functionality Testing.
Standardization Testing
This compartment mainly emphasizes on the standardization part of the application. Standardization means that the application being developed should have standard look and feel like any other window application. The general guidelines are as follows:
1. The application should have first "About Application" screen displayed.
2. Most of the screens/ dialog box (as on context) should have Minimize, Restore and Close clicks.
3. Proper icon should be attributed to the application
4. All screens/ dialog box should have a proper caption as per the context used.
5. The application should be seen in the Windows Task Bar as well as status bar.
GUI Testing
This compartment mainly emphasizes on the GUI - Graphics User Interface aspect of the Application. It is not concrete that once GUI guidelines are set that can be followed blindly. GUI standards may vary from company to company and also from application to application. But still one can set general guidelines to have an overall idea on how to start GUI testing. These guidelines apply for every screen/ dialog box of the application. General guidelines are:
1. All the dialog box should have a consistent look through out the Application system. For e.g.- If the heading within a dialog box is blue then for each dialog box the heading should be of this color.
2. Every field on the screen should have an associated Label.
3. Every screen should have an equivalent OK and cancel button.
4. The color combination used should be appealing.
5. Every field in the dialog box should have a Short Cut Key support. For e.g.- User Name
6. Tab order should be normally set horizontally for the fields. In some case as per the case the Tab Order can be set vertically.
7. Mandatory fields should have * (RED ASTERIK) marked to indicate that they are mandatory fields.
8. Default key should be set as OK for the dialog box.
9. Default key should be set as Cancel for the dialog box.
Validation Testing
This compartment mainly emphasizes on the Validation aspect of the Application. Validation testing mainly depends on the fields set in the dialog box and the functions it has to perform. But still there are certain common rules that can be applied. General guidelines are:
1. For text box fields where value entered has to be numeric check following:
¨ It should accept numbers only and not alphabets.
¨ If field usage is such that for e.g., To accept
Total number of days
Telephone number
Zip code etc.
then it should not accept 0 and negative values.
2. For text box fields where value entered has to be alpha-numeric check following:
¨ It should accept alphabets and numbers only.
¨ If field usage is such that for e.g., accepting
First Name
Middle Name
Last Name
City
Country etc.
then field value should start with an alphabet only.
¨ Depending on the condition this fields may accept special characters like -, _, . etc.
3. If the field is a combo box then it has to be checked for following points:
¨ Check the combo box has drop down values in it, it is not empty.
¨ Drop down values should be alphabetically sorted. This might change as per requirement but as standard practices it should be alphabetically sorted. For e.g. to select data type from the list it will be as follows:
Date
Integer
String
Text, etc.
¨ Selection of any drop down value is displayed on closing and opening the same dialog box.
¨ By default some value like "Select Value" or "_______" string is displayed. This is because User comes to know that value is to be selected for this field. Avoid displaying the first default value in the list.
4. If the field is a list box then it has to be checked for following points:
¨ Check the list box has values in it, it is not empty.
¨ List box values should be alphabetically sorted and displayed. This might change as per requirement but as standard practices it should be alphabetically sorted.
¨ Selection of any list box value should put a check before the value and should display the correct value(s) selected on closing and opening of the same dialog box.
¨ If the list box supports multiple selection then check whether multiple values can be selected.
5. If the field is a list of radio button then it has to be checked for following points:
¨ Check whether as per requirements all the values are listed. For e.g. to select date format. Possible values displayed will be as follows:
mm/dd/yyyy
dd/mm/yyyy
mm/dd/yy
dd/mm/yy
yyyy/mm/dd etc.
¨ Same selected value should be displayed on closing and opening of the same dialog box.
6. Data Controls are to be tested as part of functionality testing.
Functionality Testing
This compartment mainly emphasizes on the Functionality aspect of the Application. The first step to test the Functionality aspect of the Application is to check whether all the requirements are covered in the software. The actual functionality testing totally depends from software to software. Still one can frame general guidelines. General guidelines are:
1. Check the functionality is covered as per Requirement specifications or Functional specifications developed for the software.
2. Within a dialog box identify the dependent fields. Depending on the dependency check the enabling and disabling of the fields. For e.g.: to create Contact addresses in any application. To create contact addresses user should be able to add, delete and modify the information. Contact Addresses will contain information like, First Name, Last Name, Address1, Address2, City, State, Country, Zip, Phone, etc., any other information may also be added.
This form will have the required fields and in addition to that will have Add, Delete and Update buttons. The functionality of the buttons is as follows:
¨ Initially only Add button will be enabled. Delete, Update buttons will be disabled. This is because initially there is no data available and unless one adds one cannot delete or update. In short, unless there is a single valid record available it is not possible to update or delete.
¨ Only on selecting a record from the list Delete and Update buttons are enabled and Add button is disabled. By default No records will be selected.
¨ Delete and Update should always give confirmation message before actually performing the operation.
¨ Delete operation should not show the deleted item in the list.
To test any Windows based application following points are to be considered:
- It is essential to have GUI consistency within the Application.
- Should be alike in look and feel as per any other standard Window software.
- Should have standard set of keys implemented for the software.
- Should have clean and neat exit.
While testing any Windows based application the testing can be broadly categorized into following compartments. They are:
- Standardization Testing.
- GUI Testing.
- Validation Testing.
- Functionality Testing.
Standardization Testing
This compartment mainly emphasizes on the standardization part of the application. Standardization means that the application being developed should have standard look and feel like any other window application. The general guidelines are as follows:
1. The application should have first "About Application" screen displayed.
2. Most of the screens/ dialog box (as on context) should have Minimize, Restore and Close clicks.
3. Proper icon should be attributed to the application
4. All screens/ dialog box should have a proper caption as per the context used.
5. The application should be seen in the Windows Task Bar as well as status bar.
GUI Testing
This compartment mainly emphasizes on the GUI - Graphics User Interface aspect of the Application. It is not concrete that once GUI guidelines are set that can be followed blindly. GUI standards may vary from company to company and also from application to application. But still one can set general guidelines to have an overall idea on how to start GUI testing. These guidelines apply for every screen/ dialog box of the application. General guidelines are:
1. All the dialog box should have a consistent look through out the Application system. For e.g.- If the heading within a dialog box is blue then for each dialog box the heading should be of this color.
2. Every field on the screen should have an associated Label.
3. Every screen should have an equivalent OK and cancel button.
4. The color combination used should be appealing.
5. Every field in the dialog box should have a Short Cut Key support. For e.g.- User Name
6. Tab order should be normally set horizontally for the fields. In some case as per the case the Tab Order can be set vertically.
7. Mandatory fields should have * (RED ASTERIK) marked to indicate that they are mandatory fields.
8. Default key
9. Default key
This compartment mainly emphasizes on the Validation aspect of the Application. Validation testing mainly depends on the fields set in the dialog box and the functions it has to perform. But still there are certain common rules that can be applied. General guidelines are:
1. For text box fields where value entered has to be numeric check following:
¨ It should accept numbers only and not alphabets.
¨ If field usage is such that for e.g., To accept
Total number of days
Telephone number
Zip code etc.
then it should not accept 0 and negative values.
2. For text box fields where value entered has to be alpha-numeric check following:
¨ It should accept alphabets and numbers only.
¨ If field usage is such that for e.g., accepting
First Name
Middle Name
Last Name
City
Country etc.
then field value should start with an alphabet only.
¨ Depending on the condition this fields may accept special characters like -, _, . etc.
3. If the field is a combo box then it has to be checked for following points:
¨ Check the combo box has drop down values in it, it is not empty.
¨ Drop down values should be alphabetically sorted. This might change as per requirement but as standard practices it should be alphabetically sorted. For e.g. to select data type from the list it will be as follows:
Date
Integer
String
Text, etc.
¨ Selection of any drop down value is displayed on closing and opening the same dialog box.
¨ By default some value like "Select Value" or "_______" string is displayed. This is because User comes to know that value is to be selected for this field. Avoid displaying the first default value in the list.
4. If the field is a list box then it has to be checked for following points:
¨ Check the list box has values in it, it is not empty.
¨ List box values should be alphabetically sorted and displayed. This might change as per requirement but as standard practices it should be alphabetically sorted.
¨ Selection of any list box value should put a check before the value and should display the correct value(s) selected on closing and opening of the same dialog box.
¨ If the list box supports multiple selection then check whether multiple values can be selected.
5. If the field is a list of radio button then it has to be checked for following points:
¨ Check whether as per requirements all the values are listed. For e.g. to select date format. Possible values displayed will be as follows:
mm/dd/yyyy
dd/mm/yyyy
mm/dd/yy
dd/mm/yy
yyyy/mm/dd etc.
¨ Same selected value should be displayed on closing and opening of the same dialog box.
6. Data Controls are to be tested as part of functionality testing.
Functionality Testing
This compartment mainly emphasizes on the Functionality aspect of the Application. The first step to test the Functionality aspect of the Application is to check whether all the requirements are covered in the software. The actual functionality testing totally depends from software to software. Still one can frame general guidelines. General guidelines are:
1. Check the functionality is covered as per Requirement specifications or Functional specifications developed for the software.
2. Within a dialog box identify the dependent fields. Depending on the dependency check the enabling and disabling of the fields. For e.g.: to create Contact addresses in any application. To create contact addresses user should be able to add, delete and modify the information. Contact Addresses will contain information like, First Name, Last Name, Address1, Address2, City, State, Country, Zip, Phone, etc., any other information may also be added.
This form will have the required fields and in addition to that will have Add, Delete and Update buttons. The functionality of the buttons is as follows:
¨ Initially only Add button will be enabled. Delete, Update buttons will be disabled. This is because initially there is no data available and unless one adds one cannot delete or update. In short, unless there is a single valid record available it is not possible to update or delete.
¨ Only on selecting a record from the list Delete and Update buttons are enabled and Add button is disabled. By default No records will be selected.
¨ Delete and Update should always give confirmation message before actually performing the operation.
¨ Delete operation should not show the deleted item in the list.
Beyond Broken Links
by Rick Hower. Internet Systems, July 1997
Although Web site testing has much in common with the testing of standard client/server applications, there are unique considerations that can affect the focus of your testing strategy. In this article I discuss some of the factors that determine what to test in a Web site, the special considerations of database-driven Web sites, and how new Web test tools can help.What to Test
Ideally, all components and functionality of a Web site on both the client and server sides should be completely tested. However, this is rarely possible in modern software projects, especially in fast-paced Web-related projects. The best approach is to examine the project's requirements, set priorities based on risk analysis, and then determine where to focus testing efforts within budget and schedule constraints.
Requirements
Web sites are typically more graphics-oriented than standard software applications, and this, along with other multimedia functionality, will need to be considered in terms of requirements specifications for page layouts, appearance, site-specific standards, video and audio file sizes, and plug-in files. Database-driven Web sites will merit particular focus on performance requirements such as response times at various load levels and rate of error-free interactions per unit time. As in any project, requirements should ideally be clear, complete, reasonably detailed, cohesive, attainable, and testable. When this isn't the case, or when the project's requirements aren't stable, a subjective "how good is it?" testing approach may be more sensible than an objective "is it good enough (to meet requirements)?" testing strategy. This "how good is it?" approach might focus more on load testing, stress testing, usability testing, and link testing, and less on other aspects of testing.
Risk Analysis
Database-driven Web sites can involve a complex interaction among Web browsers, operating systems, plug-in applications, communications protocols, Web servers, databases, CGI programs or use of ISAPI or NSAPI features, security enhancements, and firewalls. Such complexity makes it impossible to test every possible dependency and everything that could go wrong with a site. The typical Web site development project will also be on an aggressive schedule, so the best testing approach will employ risk analysis to determine where to focus testing efforts. Risk analysis should include consideration of how closely the test environment will match the real production environment. Will the test scenarios closely mimic real-life users, Internet connections, modems, communications, hardware, clients, loads, data, database table sizes, and so on? Will the differences be significant? Ideally, the test and production environments will be the same, but budget constraints often prevent this, and the risk is that the differences may distort test results and make certain types of testing irrelevant. Other typical considerations in risk analysis include:
Which functionality in the Web site is most critical to its purpose?
Which areas of the site require the heaviest database interaction?
Which aspects of the site's CGI, applets, ActiveX components, and so on are most complex? What types of problems would cause the most complaints or the worst publicity? What areas of the site will be the most popular? What aspects of the site have the highest security risks? Factors such as the project's requirements, risk analysis, budget, and schedule determine which of the following types of testing are appropriate for your particular Web project: Validation or functional testing. This
is typically a core aspect of testing to determine if the Web site functions correctly as per the requirements specifications. Sites utilizing CGI-based dynamic page generation or database-driven page generation will often require more extensive validation testing than static-page Web sites.
Load testing.
Will there be a large number of interactions per unit time on the Web site? If so, you may want to perform testing under a range of loads to determine at what point your system's response time degrades or fails. The Web server software and configuration settings, CGI scripts, database design, and other factors can all have an impact. You will probably want to test the entire system under various conditions to get realistic results, but you may also want to consider separate testing of database response, server response, applet responsiveness, and other areas if the application is especially complex.
Stress testing.
This refers to testing system functionality while the system is under unusually heavy or peak load; it's similar to the validation testing mentioned previously but is carried out in a "high-stress" environment. This requires that you make some predictions about expected load levels of your Web site.
Usability testing.
Is your intended audience the general public? In-house Intranet users? Computer experts? Schoolchildren? The intended audience will determine the "usability" testing needs of your Web site. Additionally, such testing should take into account the current state of the Web and Web culture, because these will influence user expectations (for example, Web site navigation is expected to be extremely intuitive -- Web users do not expect to read manuals or help files).
Security testing.
If your site requires firewalls, encryption, user authentication, financial transactions, or access to databases with sensitive data, you may need to test these and also test your site's overall protection against unauthorized internal or external access.
Unit and integration testing.
Unit testing of code modules, objects, or discreet application functions is a standard part of testing any client/server or distributed application; integration testing may be needed to determine if various modules, other applications, and other parts of your site work together properly.
Regression testing.
If the project is large and complex, you may need to continuously retest everything as the site is initially developed and code is reworked to accommodate changes and bug fixes. Smaller, less complex projects may have minimal regression testing needs. As the site is upgraded and modified over the years, will there be significant changes requiring continuous retesting?
Link testing.
This type of testing determines if your site's links to internal and external Web pages are working. A Web site with many links to outside sites will need regularly scheduled link testing, because Web sites come and go and URLs change. Sites with many internal links (such as an enterprisewide Intranet, which may have thousands of internal links) may also require frequent link testing.
HTML validation.
The need for this type of testing will be determined by the intended audience, the type of browser(s) expected to be used, whether your site delivers pages based on browser type or targets a common denominator, and how strictly you want to adhere to HTML standards and/or Netscape, Microsoft, or other extensions.
Reliability and recovery testing.
Do you require uninterrupted 24 3 7 availability? Redundant database servers? Scalable Web servers? Depending on how critical your Web site is to your business, you may want to simulate various "emergency" scenarios (such as failure of a hard drive on the Web or database server, or communication link failures) in a test system to be sure that your production system will handle them successfully.
Server log
testing/report testing. Web sites that use advertising and track site usage for marketing needs may need extensive testing to ensure the accuracy of logging and reporting capabilities. (For more information on log analysis, see Dan Rahmel's article on page S12.)
Performance
Most non-Web client/server systems involve a much more controlled, predictable environment than Web-based systems. In Web-based systems, the client side may be any one of dozens of Web browsers installed on various operating systems and hardware platforms. Each browser will conform in varying degrees to multiple standards, with numerous possible plug-ins,
user-controlled options, and settings (variations in browser cache settings and graphics on/off settings are especially influential on performance). Web site usage and loads can vary with such factors as time of day, publicity, and site changes. With so many variables, system performance can be extremely difficult to predict and plan for, and testing more than a limited number of variations can be difficult or impossible. The primary factors
in performance that are under your control are your Web server software and hardware, the database, and the Web server connection capacity. Use of scalable systems can mitigate performance problems that aren't initially apparent, but the true scalability of such systems should be tested if it's a critical factor. Also, Web servers, database servers, and other system servers may need to be tested separately to find any bottlenecks that would negate improvements in other system servers.
Communications
Internet communication pathways and quality can be extremely variable. Intranet-based applications may employ more consistent and tightly controlled communication links, but public Web sites will have little control other than the connections into the Web server and connections between the Web server, database server, and other system servers. For non-Intranet systems, it may be preferable to use end-to-end test environments that are not overly concerned with communication variations, because they'll be outside your control anyway; it will often be sufficient just to emulate realistic communications environments. The same factors apply to client-side communications, especially dial-up type connections.
Security
Because Web-based systems operate in a relatively "open" environment, security can be a much greater concern than in many non-Web client/server systems. In addition to client-side access control (such as passwords and encryption), there are server-side access control issues concerning who can
publish and modify HTML files, who gets final publication approval, and who approves graphics files, sound files, and page layouts. Although corporate Intranets may require sharing and wide dissemination of information, they also need some type of control on the publication and availability of that information. Publicly available Web sites may involve outside access to
an organization's internal computer systems, which can entail some level of risk no matter what safeguards are put in place. Use of multiple public Internet communications links to pass information between browsers and Web servers entails security risks as well. Another issue is whether or not to require each Web user to have a personal user ID for logging into the DBMS
or instead to funnel all browser users through a single surrogate user ID in the DBMS. However, in my experience Intranet or Internet access is not used for direct access to a DBMS; any database interaction is usually handled by the CGI code, and the HTML pages just show user-friendly-type forms and aren't used for SQL-level interaction.
The Tools
Test tool vendors have been developing new products to help in several areas of testing database-driven Web sites. These products include load-test tools for performance testing, regression-test tools for capture/playback and test scripting, tools geared specifically to testing of Java applets and applications, and other tools for Web-related testing such as link-checking.
Load Test Tools.
The use of new test tools can significantly improve the efficiency of Web site load testing and scalability testing. Many of the leading test-tools, including Mercury Interactive Corp.'s LoadRunner/WebTest, Rational Software Corp.'s recently purchased preVue-Web product (from Performance Awareness Corp.), Pure Atria's PurePerformix/Web (PureAtria is soon to be bought by Rational Software as well), and Platinum Technology Inc.'s WebLoad, have come out with these types of products during the past year. A load test tool can emulate hundreds or even thousands of users accessing a Web site during short periods of time. Typically, such software includes capture/playback capabilities, a programming or scripting
language, monitoring and control functions, and test logging and reporting tools. The capture/playback functions can be used to "record" representative Web-user activities during interaction with your Web test site, save them as scripts, and then play back multiple instances of these scripts to simulate real loads (in the form of HTTP requests) on your site. Because these tools are relatively new, they can vary considerably in price, reliability, and capabilities. You'll want to consider the product's limitations and
capabilities regarding OS platforms, compatible browsers, secure HTTP capabilities, and the quality of the scripting language. Cost is often based on the number of simulated users. Some tools have a demo version available.
Regression Test Tools.
These are most useful for the automation of tests that will be repeated at least several times. The effort that may go into
scripting and debugging these tests is only justified if the same test scripts can be reused without major modifications. For example, a set of tests might need to be carried out each time a new build of your site's CGI software is modified or each time there are modifications to your Web site database stored procedures. As with the load test tools, regression test
tools are designed to work efficiently with Web-based applications and have capture/playback, scripting, and logging/reporting capabilities. Load test tools create direct HTTP requests; however, regression test tools instead simulate a user interacting with a browser and let the browser create the HTTP requests. Such tools can also be used in tandem with load test tools for stress testing by repeating a suite of tests under varying load conditions. Segue Software Inc.'s SilkTest and Platinum's FinalExam InternetTest are two examples of such tools.
Java Test Tools.
Java applets on the client side and Java applications on the server side can be tested separately, if necessary, with new test tools specifically geared to Java, such as those from the SunTest division of Sun Microsystems Inc. These tools test APIs and code coverage, and they perform capture/playback.
Other Test Tools.
A wide variety of other tools are available, including link-checking tools such as the Electronic Software Publishing Corp.'s LinkScan and Tetranet Software Inc.'s Linkbot.
Database-driven
Web sites are taking an increasingly critical role in the day-to-day operations of many organizations. You can minimize your risk of site problems, errors, and failures through careful analysis of requirements and risks, well-planned site test strategies, and appropriate use of new Web test tools.
Rick Hower is an independent software QA and test consultant and maintains an extensive Web site covering software QA and testing at www.charm.net/~dmg/qatest/. You can contact him via his Web site or by email at rhower@netcom.com.
QA Practices and Procedures
Testing Methodology
The following is an overview of the quality practices of Software Quality Assurance team:
-The iterative approach to software development presents a significant challenge for SQA. The iterative, rapid deployment process is characterized by a lack of strict adherence to a traditional waterfall development methodology (marketing first specs the feature set, then engineering refines the marketing requests into more detailed specifications and a schedule, then engineering starts building to specification and SQA starts building tests, then a formal testing cycle, and finally product release). Here is a variant of development:
-As progress is made toward a release, the first priority features are done to a significant level of completion before much progress is made on the second priority features. A similar approach is taken for the hopefullys and third priority features. The first priority feature list is all that has to be completed before a product is feature complete, even though, there has been time built into the schedule to complete the second priority, as well.
-Other than the initial OK from the executive team that they want a particular product built, there is not a rigorous set of phases that each feature must pass.
-Developers (designers, coders, testers, writers, managers) are expected to interact aggressively and exchange ideas and status.
-By not going heavily into complete specifications, the impact of a better idea along the way need not invalidate a great deal of work.
-One prototype is worth a pound of specification. However, this does not mean that large scale changes should not be specified in writing. Often times, the effort to do paper based design is significantly cheaper than investing in a working prototype. The right balance is sought here.
Complementing the strategy of iterative software development, the SQA testing assessment is accomplished through personal interaction between SQA engineers and Development engineers. Lead SQA engineers meet with the development team to assess the scope of the project, whether new features for an existing product, or the development of a new product. Feature, function, GUI, and cross-tool interaction are defined to the level of known attributes. When development documentation is provided, the understanding of the SQA engineer is greatly enhanced. The lead SQA engineer then meets with the test team, to scope the level and complexity of testing required. An estimate of test cases and testing time is arrived at and published, based upon the previous discussions.
-Working with the development team, the SQA team takes the builds, from the first functioning integration, and works with the features as they mature, to determine their interaction and the level of testing required to validate the functionality throughout the product.
-The SQA engineers, working with existing test plans and development notations on new functionality, as well as their notes on how new features function, develop significant guidelines for actual test cases and strategies to be employed in the testing. The SQA engineers actively seek the input of the development engineers in definition and review of these tests.
-Testing is composed of intertwined layers of manual ad hoc and structured testing, supplemented by automated regression testing which is enhanced as the product matures.
Test Plan Components
Test requirements based on new features or functions.
Specific testing based on the features defined as Development Priority 1. There must be a plan in place for these features and they must be scheduled for testing. A product release date will be slipped in order to complete adequate testing of the Priority 1 features.
Specific testing based on new features or functions defined as Development Priority 2. There must be a plan in place for these features and they must be scheduled for testing. If testing of the Priority 1 features impacts adequate testing of these, they may be dropped from the product.
Specific testing based on new features or functions defined as Development Priority 3. Software Quality Assurance will not schedule or plan for these features. However, Priority 3 completed prior to Functional Freeze will be added to the SQA Priority 2 for testing and appropriate risk assessment will be taken with respect their inclusion in the released product.
SQA has its own set of Priority 1, Priority 2, Priority 3, which include not only the Development activities, but also testing required as due diligence for product verification prior to shipment.
-Priority 1, features include the testing of new features and functions, but also a defined set of base installations, program and data integrity checks, regression testing, documentation (printed, HTML and on-line Help) review and final "confidence" (high level manual or automated tests exercising the most frequently used features of the product) checks on all media to be released to the public. Products being distributed over the Web, also have their Web download and installation verified.
-Priority 2, include a greater spectrum of installation combinations, boundary checking, advanced test creation and more in-depth "creative" ad hoc testing.
-Priority 3, usually reflect attempts to bring greater organization to the SQA effort in documentation of test scripts, creation of Flashboards for metric tracking, or expanded load testing.
Testing
One of the test methods SQA team practice is "Black Box" testing. The SQA engineers, like the customers whom they attempt to emulate, are isolated from the source code and must rely upon their understanding of the application and its features and functions. SQA engineers work with Development engineers toward development of code which lends itself to the exercise of automated test tools, thus providing for stable, repeatable, reliable testing of base features and functions. The deductive, reasoning, creative skills of the SQA engineers are thus freed from the more repetitive tasks to focus on development of more user centric testing, which expands the scope coverage.
Manual Testing
-GUI - SQA team members upon receipt of the Development builds, walk through the GUI and either update existing hard copy of the product Roadmaps, or create new hard copy. This is then passed on to the Tools engineer to automate for new builds and regression testing. Defects are entered into the bugs tracking database, for investigation and resolution. Questions about GUI content are communicated to the Development team for clarification and resolution. The team works to arrive at a GUI appearance and function which is "customer oriented" and appropriate for the platform, Web, UNIX, Windows, Macintosh. Automated GUI regression tests are run against the product at Alpha and Beta "Hand off to QA" ,(HQA) to validate that the GUI remains consistent throughout the development process. During the Alpha and Beta periods, selected customers validate the customer orientation of the GUI.
-Features & Functions - SQA test engineers, relying on the team definition, exercise the product features and functions accordingly. Defects in feature/function capability are entered into the defect tracking system and are communicated to the team. Features are expected to perform as expected and their functionality should be oriented toward ease of use and clarity of objective. Tests are planned around new features and regression tests are exercised to validate existing features and functions are enabled and performing in a manner consistent with prior releases. SQA using the exploratory testing method, manually tests and then plans more exhaustive testing and automation. Regression tests are exercised which consist of using developed test cases against the product to validate field input, boundary conditions and so on... Automated tests developed for prior releases are also used for regression testing.
-Installation - Product is installed on each of the supported operating systems in either default, flat file configuration, or with one of the supported databases. Every operating system and database, supported by the product, are tested, though not in all possible combinations. SQA is committed to executing, during the development life cycle, the combinations most frequently used by the customers. Clean and upgrade installations are the minimum requirements.
-Documentation - All documentation, which is reviewed by Development prior to Alpha is reviewed by the SQA team prior to Beta. On-line help and context sensitive Help are considered documentation as well as manuals, HTML documentation and Release Notes. SQA not only verifies technical accuracy, clarity and completeness, they also provide editorial input on consistency, style and typographical errors.
Automated Testing
-GUI - Automated GUI tests are run against the product at Alpha and Beta "Hand off to QA" (HQA) to validate that the GUI has remained consistent within the product throughout the development process. The automated Roadmaps, walk through the client tool windows and functions, validating that each is there and that it functions.
-Data Driven - Data driven scripts developed using the automation tools and auto driver scripts are exercised for both UNIX and Windows platforms to provide repeatable, verifiable actions and results of core functions of the product. Currently these are a subset of all functionality. These are used to validate new builds prior to extensive manual testing, thus assuring both Development and SQA of the robustness of the code.
-Future - Utilization of automated tools will increase as our QA product groups become more proficient at the creation of automated tests. Complete functionality testing is a goal, which will be implemented feature by feature.
Reporting
The SQA team lead produces a formal test plan which is submitted to the project team for validation and information. Input is solicited for the understanding of requirements, so that the plan can accurately reflect the testing which will be required. Because of the variances between Development teams' style of disseminating feature, function scope and definition, the amount of information available is not always consistent. The SQA lead attempts to establish a good working relationship with the Development team, so that questions can be asked and comprehensive, identical understanding of the product will exist. The lead also pairs a QA engineer with a development counter part to facilitate day to day interaction.
SQA engineers produce status sheets for distribution to the members of the product teams. The data on the status sheets have included: features, functions, time estimated to test, time consumed and amount of testing yet to be done, the last has proved to be too subjective. Some test engineers include the staff assigned to specific testing and the percentage complete. The later is difficult to estimate due to the iterative process, as feature and function specifics change often and rapidly during the development cycle, as they are refined. The evolving report, which appears to give the most information, is one which lists the features, the build in which they were tested or re-tested and the pass or fail status of the test.
SQA provides monitoring of the defects which appear in the product, through the use of QA designed Flashboards (graphical representations of the aggregate numbers of defects) and reports. The defects found in the product are recorded in an Bug Tracking database, where the information is made available to the development group. Information stored in the database then provides statistical and trend analysis for defect find rates and product areas. This information is compared to that presented by the Development team and the Product team. Customer support is kept abreast of these defects and influences the priority assigned to the defect by the team.
Team leads have established directories for their products in which test plans and weekly status reports have been posted. These are updated weekly by the team lead and reviewed by the manager and linked or posted to the QA home page on the intranet.
QA managers work with their teams to assess better forms and methods of information dissemination. These reviewed with the larger engineering and project teams so that the teams feel they understand the scope of work to be done by SQA and the status of a project currently being tested.
Subscribe to:
Posts (Atom)
-
http://podian.blogspot.com/ One of the Good blog in Tamil so you can read current news also.
-
ஸ்ரீ இராம நாம மந்திர மகிமை 🌷 1. நமக்கு நன்மை வரவேண்டுமானால் 'ராம நாமத்தை இடைவிடாமல் கூறவேண்டும். நமது ஒவ்வொரு மூச்சும் ...
-
import os import openpyxl # Specify the target folder folder_path = "C:/Your/Target/Folder" # Replace with the actual path # Cre...
