Testing

Every decent software project contains tests—usually a lot of them. Here is the testing directory for IntelliJ, which powers Android Studio. And here’s the testing directory for Chromium, the open-source version of Chrome. And here are the tests for Discourse, the open source software that powers our CS 125 forum. Anyway—our point is that all sane software developers write tests. Frequently a lot of them. Sometimes you actually end up with more testing code than you have non-testing code—particularly if you are trying to test a particularly complex or important piece of software.

Companies also devote a huge amount of time to software testing. Somewhere in a room at Google or Microsoft or Facebook or Twitter or Instagram is a room full of computers (or phones) that are churning away night and day running tests to identify bugs in new versions of their products. When you begin writing software in industry, frequently the first step in changing something is making sure all existing tests have passed. Then you’ll be required to write your own tests for whatever new feature you are trying to add. Only once existing tests all pass and new ones are added will anyone even think about actually accepting your new change.

Why? Because almost every piece of useful software is complicated. And really useful ones can become incredibly complex. What seems like a small change to you can affect other parts of the application or program in ways that you didn’t intend and could never foresee. This problem is exacerbated by the fact that frequently no one person understands the entire program or system. So rather than try and use our limited human brains to try and determine how a change will affect the rest of the program, we just write a lot of tests and hope that they will identify anything that breaks. Computers helping us write more computer code 2.

Software testing employs many different strategies and methods—far, far too many to cover in one lab.

But one important thing to realize is that tests are typically done at different levels of granularity. From bottom to top:

Given the size of the problems you are able to solve in CS 125, almost all of our tests are unit tests.

However, we want to point out that the tests that you find on our CS 125 MPs are not good examples of how to write tests. Why not? Because they were written with access to a solution. Normally you are testing your code because you aren’t sure whether it solves a problem. In contrast, when grading your code we are sure that our solution is correct and we want to test whether or not your code behaves the same way. However, we also don’t want to give away our solution.

The solution we’ve settled on and used for MP0 and MP1 this semester is to use our hidden solution to generate pre-computed inputs and outputs. That’s why you found stuff like this in the MP1 test suite:

Even we aren’t crazy enough to write stuff like this by hand! We used a separate computer program to generate these test cases.

That said, MP2 does provide better examples of hand-written test cases. And, it’s worth pointing out that the test suite for MP2 contains 836 lines of code while the solution itself contains only 530 lines.

On today’s homework problems we give you practice writing test cases for a few small programming problems. These are unit tests, since each is testing a single function.

Writing good tests requires a certain adversarial mindset. You have to think—what kind of mistakes is someone implementing this likely to make? Or, put another way, you can ask the question: how would I mess this up? Then, for each potential error, you construct a test case to catch it.

How do you know the right answer? Well, you could write a function to compute it—but then you’d have something else to test! Instead, what we typically do is compute the result beforehand, offline.

As an example, say that I am testing a function that takes an array as input and returns an int. The documentation says that, if the array is null, the function should return -1. Knowing myself, I might miss that part. So I know that I want to test to make sure that null is handled properly, and I should check that it returns -1 in that case, since that’s what the documentation says will happen.

Coming up with good adversarial inputs also depends on what you are trying to test. For example, one of the homework problems today is testing a generalized version of String rotation. An input like aaa is an awful input for testing String rotation—while abc could be a good input. Similarly, tests that always use 0 as the amount to rotate will fail to uncover many bugs.

OK, now get started on our testing homework challenge. Please feel free to discuss the problems with other students in your lab, although everyone needs to submit their own work.