Applications of Syntax Testing

Traditionally, the following are the applications of syntax testing:

1. Command-Driven Software: This is the most obvious application and probably the most common. If a system is mostly command driven, then much of its testing can be organized under syntax testing.
2. Menu-Driven Software: The popular alternative to command-driven software is menu-driven software, in which actions are initiated by selecting from choices presented in a menu. However, menu-driven or not, there are still data fields to enter and those fields have a syntax against which syntax testing is effective. It’s usually very easy, though, because the syntax tree tends to be shallow
3. Macro Languages: Many commercial software packages for PCs have a macro language, also called a scripting language. This is a programming language that can be used to automate repetitive operations. These languages are often implemented in commercial packages not just for the users’ convenience but because they are a neat way to implement a lot of complicated features, such as mail-merge in a word processor. In their best form, these are full-blown, albeit specialized, programming languages with formal specifications (often in BNF or the equivalent). Non-commercial software or software that serves a narrow segment of an industry^*also may have a macro language, but usually, it’s not as clearly defined or implemented as the commercial (horizontal) packages. Such languages, if they are part of an application, warrant syntax testing.
4. Communications: All communications systems have an embedded language. It is the language of the format of messages. Even telephone exchanges use language to communicate with each other. But proper telephone numbers, local, long distance and international, have a very formal syntax called a number plan. Every message used to communicate must have a format (i.e., syntax) that must be parsed.
5. Database Query Languages: Any database system has a command language used to specify what is to be searched and what is to be retrieved. The simplest ones allow only one key. The more mature systems allow boolean searches based on user-supplied parameters, which we recognize as being predicates. Such predicates obviously have a formal syntax and semantics, and should, therefore, be treated to syntax testing. 
6. Compilers and Generated Parsers: The one place syntax testing should not be used, especially dirty syntax testing, is to test a modern compiler. This might seem perverse and strange, but a tester should never repeat the tests previously done by another. Modern compilers have a parser, of course. But that parser is generated completely automatically by use of a parser generator given a formal definition in something like BNF. Lexical analyzers are also generated automatically from formal specifications. From a testing viewpoint, there’s not much that syntax testing, even when fully automated, can do to break such lexers and parsers. Before you spend a lot of effort on syntax testing (even if automated), look at the application and how it has been implemented. If a generated lexer and parser are used or planned, you’re unlikely to have great success in using syntax testing; those kinds of bugs have been totally prevented. The only thing left to test is semantics, which is often done by another technique such as domain testing.

Limitations

The biggest potential problem with syntax testing is psychological and mythological in nature. Because design automation is easy, once the syntax has been expressed in BNF, the number of automatically generated test cases measures in the hundreds of thousands. Yet, as in the case of generated parsers, such tests may be no more cost-effective than trying every possible iteration value for a loop. The mythological aspect is that there is great (undeserved) faith in the effectiveness of keyboard-scrabbling or monkey testing. Monkey Testing is just pounding away at the keyboard with presumably random input strings and checking the behaviour. Though amateurish software can still be broken by this kind of testing, it’s rare for professionally created software today. However, the myth of the effectiveness of the wily hacker doing dirty things at the keyboard persists in the public’s mind and in the minds of many who are uneducated in testing technology. Another caveat is that syntax testing may lead to false confidence, much akin to the way monkey testing does.

Conclusion

One major benefit of syntax testing comes from the assurance that there are no misunderstandings about what are legal data and what is not. When a formal syntax description is written out, such problems will surface even before the testing begins. This is another example in which the process of designing and creating test cases helps to prevent errors. Ideally, the formal syntax should be used to specify the system in the first place. The applications and limitations specified above may prove beneficial to adopt syntax testing.

Give us 30 minutes and we will show you how many millions you can save by outsourcing software testing. Make Your product quality top notch. Talk to us to see how

The post Part II- Beginner’s Guide to Syntax Testing: Applications and Limitations in Software Testing appeared first on KiwiQA.

The need for syntax testing arises since most systems have hidden languages (a programming language that has not been recognized as such). Syntax testing is used to validate and break the explicit or implicit parser of that language. A complicated application may consist of several hidden languages, an external language for user commands and an internal language (not apparent to the user) out of which applications are built. These internal languages could be subtle and difficult to recognize. In such cases, syntax testing could be extremely beneficial in identifying the bugs.

Syntax structures can be used for testing in several ways. We can use the syntax to generate artefacts that are valid (correct syntax), or artefacts that are invalid (incorrect syntax). Sometimes the structures we generate are test cases themselves, and sometimes they are used to help us design test cases. To use syntax testing we must first describe the valid or acceptable data in a formal notation such as the Backus Naur Form, or BNF for short. Indeed, an important feature of syntax testing is the use of a syntactic description such as BNF or a grammar. With syntax-based testing, however, the syntax of the software artefact is used as the model and tests are created from the syntax.

[podbean playlist=”http%3A%2F%2Fplaylist.podbean.com%2F3293175%2Fplaylist_multi.xml” type=”multi” height=”315″ kdsowie31j4k1jlf913=”9bf59f909e7edc039590e5084da2bebd1846ce87″ size=”315″ share=”1″ fonts=”Helvetica” auto=”0″ download=”1″ rtl=”0″ skin=”0″]

Syntax testing is a powerful, easily automated tool for testing the lexical analyzer and parser of the command processor of command-driven software.

Syntax Testing Techniques

Syntax testing is a shotgun method that depends on many test cases. What makes this method effective is that though any one case is unlikely to reveal a bug, many cases are used which are also very easy to design. It usually begins by defining the syntax using a formal metalanguage, of which BNF is the most popular. Once the BNF has been specified, generating a set of tests that cover the syntax graph is a straightforward matter.

Syntax testing has many applications beyond testing typed commands, but that’s a good application for which to illustrate this technique. The process is as follows-

1. Get as formal a specification as you can for all the commands/strings that you intend to test, in whatever form they are available. This information must exist, or else what did the programmers implement and how do the users know how to run the software? If it’s an existing system, look at the help files (such as the MS-DOS command HELP) or, at worst, find the commands’ syntax experimentally.
2. Search through the commands to find common parts that apply to many commands. For example, in MS-DOS, the following fields are used in many commands: <address>, <device>, <directory>, <drive_name>, <filename>, <integer>, <ON|OFF>, <path>, <time>. You do this in order to avoid redundant specifications for common fields. If you specify the same thing twice, there’s a possibility that your specification won’t be identical each time and, therefore, a possibility for creating a test design bug.
3. Search the commands to find keywords. In MS-DOS, every command has a keyword, but other keywords appear within commands, such as AUTO, AUX, COM1, COM2, COM3, COM4, CON, LPT1, LPT2, LPT3, ON, OFF, PATH, PRN. Again, this is done to avoid repetitions of specifications and test design bugs.
4. Start your definitions with the keywords because those are most likely to be modified through lexical changes.
5. Create BNF specifications for the common fields, such as <drive>.
6. List the commands in order of increasing complexity, where complexity is measured by the number of fields in the command and how many lower-level definitions to which you have to refer.
7. Group the commands: It is not convenient to order the commands by their operational meaning. That may be good for a sales demonstration, but it is not for testing. Group them by characteristics, such as: uses common keywords, uses common field definitions, follows a similar pattern and so on.

Well-chosen groups make designing and testing the tests easier and help you avoid test design bugs, and can reduce your design labour.

8. For every field that has variable content (e.g., numerical, integer, string, etc.), there is usually an associated semantic specification (e.g., min and max values). Define all such semantic characteristics and decide what test technique you will use, for example, domain testing.
9. Design the tests: Each command creates a separate set of tests (both clean and dirty). Each clean test will correspond to a path through the syntax graph of that command. As usual, you pick the path, sensitize it, predict the outcome, define validation criteria, confirm outcomes and so on.

You must do the first eight steps whether you use automatic test generators or do it by hand. The first eight items on this list are 50 to 75 per cent of the labour of syntax testing.

Summary

Syntax testing is primarily a testing process that is hard to stop once it is started. A little practice with this testing technique will help you perform the aforementioned tasks easily and efficiently.

Give us 30 minutes and we will show you how many millions you can save by outsourcing software testing. Make Your product quality top notch. Talk to us to see how

The post Part I- Beginner’s Guide to Syntax Testing: Understanding the Basics appeared first on KiwiQA.

Though machine learning is effective and efficient, as humans, we don’t have the same rate of effectiveness. Though the machine learning algorithms are designed to be capable of minimizing errors, it is possible that defects in coding algorithms may have an effect of not minimizing right errors or instead, errors in the code may lead to more defects than before. Consequently, tests are required to address human errors and also to document the progress. One of the most popular ways of writing these tests is referred to as test-driven development (TDD).

There are multiple benefits to use TDD approach to machine learning. Though TDD may usually take more time (up to 15-35% of normal time), it can also reduce the bugs by a whopping range of up to 90%. Further, TDD is used widely to document how the code is intended to work, which becomes helpful as the complexity of the code increases and enables making crucial decisions based on what comes out of the analysis.

Importance of using Test-Driven Approach in Machine Learning

1. TDD is akin to a scientific method

One of the major reasons why TDD is preferred in the machine learning process is because it can easily since with the people and their working style. Thus, the testing, theorizing and hypothesizing process is very similar to any other scientific method and enables better test processes through cleaner and more stable codes.

There are other reasons as well why TDD is considered a scientific method-

1. Helps to work in feedback loops
2. Sharing results through documentation is efficient
3. It makes a logical proposition of validity

It is also possible to experiment test-driven development, as you can utilize the tests to experiment with things that might never work, instead of following a traditional approach to write tests first and eventually fix the error created by the initial test. This methodology helps in solving many complex problems that are hitherto not easily solvable.

2. TDD and Scientific Method Work in Feedback Loops

It is important to also understand that both TDD and scientific method tends to work in feedback loops. This can be seen through the fact that when someone forms a hypothesis and tests the same, he could find more data on the problem he is investigating in a scientific environment. In the same way, when someone forms a test to arrive at a prospective conclusion and goes through the process of writing the code, he develops and identifies more information on how to proceed. This method of making a hypothesis, testing and revisiting them is, therefore, a scientific process that is used by many TDD practitioners.

3. TDD Involves Documentation

Academic institutions usually require professors to publish their research, without which there are high chances that the research would be worthless. In the same way, TDD can be peer-reviewed and serve as a documentation process. Since software is abstract and continuously changing, documentation helps to keep a track of such change and ensures the progress is tracked by new developers.

4. TDD Makes a Logical Proposition of Validity

Usually, a scientific method involves trying to find the solution to a problem and proving that the solution is valid, which may require creating guessing. In order to justify a belief we consider true, we construct a logical, stable proposition through the use of necessary and sufficient conditions.

• Necessary conditions are defined as those conditions without which the hypothesis would fail. For instance, this could be a preflight checklist or a majority vote. The necessity is that all conditions should be fulfilled in order to convince us that whatever is being tested is correct.
• Sufficient conditions, however, refers to those conditions that present just enough evidence for an argument.

TDD uses both sufficient and necessary conditions to make a set of propositions that come together in a cohesive way. However, it must be noted that unlike scientific method which uses axioms and hypothesis testing, TDD uses unit and integration tests.

Conclusion

As we’ve seen above, machine learning is nothing but a science requiring an objective approach to issues in the testing process. TDD can aid in solving issues arising in machine learning algorithms, because of four major reasons identified above (i.e. proposing logical and valid propositions, working in feedback loops and sharing results through documentation). However, there are risks associated with machine learning, such as:

• Underfitting
• Unpredictable future
• Unstable data
• Overfitting

The best part, however, about TDD based machine learning process is that you can think and write heuristics before writing actual codes to solve machine learning problems.

Give us 30 minutes and we will show you how many millions you can save by outsourcing software testing. Make Your product quality top notch. Talk to us to see how

The post Machine Learning: Importance of using a Test-Driven Approach appeared first on KiwiQA.