Snippets Groups Projects

blackbox_testing.md

A little more clarification based on past student misunderstandings

Christopher Bohn authored 5 years ago

f1bbc6e7

f1bbc6e7 5 years ago

blackbox_testing.md 16.38 KiB

Blackbox Testing

Due: June 18, 2020 at 11:00am (CDT, UTC-5)

Overview

In this assignment you will test a simple calculator program based on its functional specification. The functional specification follows the assignment instructions.

You will:

Demonstrate equivalence partitioning, aka category partitioning
Develop blackbox unit test cases for those partitions
Create JUnit tests for those test cases

Instructions

This assignment is to be completed individually; no collaboration is permitted.

Setup

Navigate to the starter code on GitLab and click on the "Fork" button to copy it to your account.
Set the copied project to Private and add the professor and TAs with "Maintainer" access.
- On your repository's side-menu, click on Settings->Members. On the resulting page, find "Visibility, project features, permissions" and click on its Expand button. Using the dropdown, set the project visibility to Private, and click on the Save changes button.
- Add the professor and TAs in the same manner you did for your homework repository.
Verify that GitLab shows a lock icon on the fork. Note that, per the course’s collaboration policy, you are responsible for safeguarding your own work, and it is an academic integrity violation to have your work posted publicly.
Clone your copied project: git clone *URL* where *URL* is your calculator repository's "Clone with SSH" URL. (The URL probably is git@git.unl.edu:your_username/calculator.git)
- Do NOT place your calculator repository inside your csce361-homework repository!
Import your cloned repository into your IDE as a Maven project.
- Instructions for importing Maven projects into some IDEs are available in the "Importing Maven Projects" page on Canvas.
  - For other IDEs, after you find instructions to import Maven projects, please share them on Piazza for the benefit of others
- If you are using an editor that does not integrate Maven, you can still use Maven from the command line
  - Check wither Maven is already installed:
```
mvn --version
```
    - If Maven is not installed, you should be able to install it if you have a package manger (such as apt or brew) installed. Otherwise, download and install it manually.
  - To build the project:
```
mvn compile
```
  - To run the JUnit tests:
```
mvn test
```
  - To run the program:
```
mvn exec:java -Dexec.mainClass=edu.unl.cse.csce361.calculator.Calculator
```

What is a Postfix Calculator?

If you already know what a postfix calculator is, you can safely move on to the Assignment section. This description is to help students understand the behavior of Calculator.

The calculators you are probably accustomed to using use infix (or algebraic) notation, in which a mathematical operator is located between its operands. In the early 20th Century, Polish logician Jan Łukasiewicz invented prefix notation (which came to be known as Polish Notation) in which a mathematical operator is located in front of both of its operands.

Postfix notation (also known as Reverse Polish Notation, or RPN) places the mathematical operator after both of its operands. Postfix notation is simpler to parse and cannot produce an ambiguous expression. Consider the infamously ambiguous viral expression 8÷2(2×2). In RPN, this would be expressed either as 8 2 2 × 2 × ÷, equalling 1, or 8 2 ÷ 2 2 × ×, equalling 16. The reason RPN expressions are so easy to parse is because each token is pushed onto a stack. If the token is an operator then the previous two tokens at the top are popped off the stack to be used as operands. The result of the computation is then pushed onto the stack.

The first handheld electronic calculator was an RPN calculator, and engineers continued to use RPN calculators for decades. The computation is faster than algebraic calculators (though on human scale, the difference is unnoticeable). Data entry, requiring fewer keystrokes, is faster. And studies show that engineers make fewer errors when using postfix notation than infix notation.

The postfix Calculator program differs from RPN calculators in one regard: real postfix calculators do not have an equals sign. The Calculator uses the equals sign to instruct the program to display the value at the top of the stack. For example:

8 2 2 * 2 * / =
	1.0
8 2 / 2 2 * * =
	16.0
+ =
	17.0

Assignment

Run Calculator (not CalculatorTest) a few times to get a feel for its behavior. The functional specification for the full program follows the assignment instructions.
Complete your test plan for the Calculator.calculate() method.
- You do not need to look at Calculator's source code to develop your tests. That's the point of blackbox testing. If you do look at Calculator's source code, be warned that it has some bad design in it; this is intentional for discussion purposes later. Bear in mind that these bad design aspects would make it more error-prone and harder to maintain.
1. Complete developing the testing requirements. The testing requirements should be traceable to Calculator.calculate()'s functional requirements (listed after the assignment instructions).
  - Do not use the "Postfix Calculator Functional Specification." This system-level specification is included for context. Your testing requirements need to trace to the "Calculator.calculate() Functional Specification."
2. Develop the rest of the test cases in your test suite. The test cases should be traceable to the testing requirements.
3. There is a file called TESTING.md in the top-level directory of your calculator repository. Record your test plan in TESTING.md.
  - The .md suffix indicates that this is a "markdown" file. You can use markdown formatting annotations (see the Markdown Cheatsheet on Canvas).
  - Clearly identify the equivalence partitions.
  - Specify your test cases. Each test case should have a short description (such as "Pop an Empty Stack") the number(s) of the functional requirement(s) the test case addresses, the inputs, and the expected output (the oracle).
  - Some of your test cases may require multiple calls to Calculator.calculate().
Edit CalculatorTest.java. Create a JUnit test for each test case in TESTING.md. The name of each JUnit test should be in camelCase, beginning with test and then the name of the test case (omitting definite & indefinite articles), such as testPopEmptyStack().
- Remember that you should use the @Test annotation and that the method must be public void and with no parameters.
Run CalculatorTest.
1. For any test cases that fail, look at the details and compare the expected output to the actual output.
2. Do not attempt to fix the faults revealed by your tests.
- There are bugs that are intentionally included in Calculator. If a test fails, do the due diligence to make sure that your test isn't faulty, but otherwise it's okay that you found a bug. On a real project, finding a bug is a good thing because it means that you can fix it and that bug won't get shipped. (Again, do not attempt to fix the bugs that you find in this assignment.)
Produce a test report. Normally you wouldn't commit an auto-generated test report to the repository; however, for grading purposes we will require that you do so.
- If your IDE will generate an html test report, then you may use that for your test report. Give it a meaningful name and store it in the top- level directory of your calculator repository.
  - Your IDE might try to place the html test report in the target/site directory. If it does, or if it places the test report in any location other than the top-level directory, move the test report from wherever the IDE stores it to the top-level directory.
- Alternatively, you can take a screenshot of your IDE's JUnit "stoplight" results and save it in the top-level directory of your calculator repository as a jpg, gif, png, or pdf file. Give it a meaningful name.
  - If you're running mvn test from the command line, copy the output and save it in the top-level directory of your calculator repository as a txt file. Give it a meaningful name.
Upload all of the deliverables to your copied project using the standard add/commit/push pattern:
```
git add .
git commit
git push
```
(or you can use your IDE to add/commit/push)

Go to your copy of the project on git.unl.edu to confirm that the required deliverables are in place.

Deliverables

For grading, we will clone your copy of the project after it is due, and we will look for:

Updated TESTING.md in the top-level directory, containing your test cases, traced to the functional requirements.
Your test report in the top-level directory, either an auto-generated html file, or a screenshot of the IDE's JUnit "stoplight" results.
Updated CalculatorTest.java file in the directory specified by the Maven convention (src/test/java/edu/unl/cse/csce361/calculator/).

It is your responsibility to ensure that your work is in the master branch of the correct repository and that we have the correct level of access to the repository at the time the assignment is due. We will grade what we can retrieve from the repository at the time it is due. Any work that is not in the correct repository, or that we cannot access, will not be graded.

History

You may notice there are some unusual design decisions in Calculator.java, such as using character arrays for Strings and an array for a Stack. This is intentional.

This program began as an example in the 1st edition of Kernighan & Ritchie's The C Programming Language published in 1978. Dr. Gregg Rothermel modified it for the first version of this assignment, which required a specialized command-line script to run automated tests. Dr. Chris Bohn transliterated the C code into Java code, taking pains to preserve a pernicious bug found in the C version and to place an upper bound on the stack's depth to benefit the equivalence partitioning aspect of the assignment. Dr. Bohn subsequently modified the code to make it more suitable for JUnit testing; however, he left the design substantially suboptimal for testing, to benefit a future lesson.

Postfix Calculator Functional Specification

This is the original specification for the full program, which may help you get the feel for the program's behavior in Assignment Step 1. I have removed the numbers so you are not tempted to use these for developing your test plan.

The calculator program is run from the command line and takes no parameters.
When invoked, the calculator displays no message; it simply waits for input.
The expected input to the calculator is a valid (well-formed) postfix expression containing integers, real numbers, and the mathematical operators +, -, *, /, =, separated by whitespace.
Integers and real numbers are limited to 20 characters; entry of numbers exceeding this size results in an error message.
When the user inputs the character =, the program responds by printing the value of the number that’s on the top of the stack. If the stack is empty then the program prints an error message.
The program detects attempts to divide by zero and prints an error message.
At any point in time, the user may delete whatever value is on the top of
the stack by typing c. Typing c when the stack is empty produces no results.
Tabs and newlines are treated like spaces; they’re all whitespace.
The program continues to maintain its stack and respond to the typing of
additional characters until the user ends its execution by typing q.
The size of the stack is limited to 100 elements. Attempts to exceed the maximum stack size results in the display of an error message.
Entry of any character other than 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, ., +, -, *, /, =, c, q results in an error message.

`Calculator.calculate()` Functional Specification

These are the requirements you should use to develop your test plan in Assignment Step 2.

The function takes two arguments, an integer, type (which may be interpreted as a character) and a character array, token, which must have a \0 character after the last printable character.
The function returns a string; however, it also has side-effects. It will update the stack backing the postfix calculator, and it may output to stdout.

If type is the declared constant NUMBER then token will be treated as the textual representation of an integer or fixed-point real number. token will be parsed to determine the number's value, and this number will be pushed onto the stack. The function will return null.
- n.b., Java (unlike C) requires all characters after the last printable character in token to be \0.
If type is the declared constant OPERAND_TOO_LONG then the function will return the first 20 characters of token followed by "... is too long".
If type is the character = then the function will return a tab character followed by a textual representation of the value at the top of the stack; the stack will remain unchanged.
If type is the character +, -, *, or / then the function will pop a value off the stack
1. If the value is 0.0 and type is / then the function will return "zero divisor popped".
2. Otherwise, the function will pop a second value off the stack and apply the appropriate arithmetic operation. The second value popped off the stack will be treated as the first operand, and the first value popped off the stack will be treated as the second operand. The result of the operation will pushed onto the stack. The function will return null.
  - e.g., if the stack is: top->2.0,8.0,... and type is / then the function will compute
    $8.0 \div 2.0 = 4.0$
    and push 4.0 onto the stack.
If type is the character c then the function will pop the top value off the stack and return null.
If type is any other value then the function will return "unknown command " followed by type (interpreted as a character).
If the function attempts to pop a value off the stack when there are no values on the stack, the function will output "error: stack empty", place the value 0 on the stack, and return the tab character followed by 0.0.
The stack is limited to 100 values. If the function attempts to push a value onto a full stack then the function will output "error: stack full" and return null.

Rubric

The assignment is worth 14 points:

2 points for tracing all test cases to functional requirements.
1 point for having at least one testing case for each functional requirement.
3 points for identifying equivalence partitions
3 points for having each equivalence partition covered by at least one test case.

3 points for JUnit tests matching the test cases.
1 points for providing the test report.
1 point for meaningful commit message.

If at any time your repository is public or has internal visibility then you will receive a 10% penalty. Further, if another student accesses your non-private repository and copies your solution then I will assume that you are complicit in their academic dishonesty.