6. Overusing mock frameworks
Let me start off by saying, I'm not a big fan of mock frameworks because, they increase the test suite run-time and usually decrease re-usability by repeating the same mock set-up instead of creating one re-usable test double.
This repeated mock framework boilerplate, sooner or later, will break in a lot of places when something changes (e.g. its contract). This is the exact opposite of what we do in production code (keeping things DRY). However, having a re-usable test double follows that good practices, because we don't repeat ourselves.
These handwritten test doubles are written by us, and can be extended to suit our needs. For example, we could have additional functions which we can use for verification avoiding some boilerplate and protect ourselves from additional changes:
class FakeNavigator : Navigator {
var commands: List<NavigationCommand> = emptyList()
override fun navigate(command: NavigationCommand) {
commands.add(command)
}
fun assertLatestCommandEquals(item: Item) {
assertEquals(item, commands.last())
}
}Another thing with mocking frameworks is that they are also easy to misuse by testing implementation details of the SUT instead of its behavior / outcome. This in turn leads to brittle test suites that break frequently when changing some implementation details even though the behavior stays the same. I will elaborate on this more in the next point.
I do see the benefit of mocking frameworks and use them sometimes, however in my opinions they have more downsides than upsides. If you want to keep using mocking frameworks, go ahead. My only suggestion would be that if you have classes / interfaces which are mocked frequently, create a Mock Wrapper or a Test Double by hand, it will keep your test code DRY and encourage more re-usability.
7. Testing implementation details instead of the outcome
Before delving into the details, I want to point out that the popular "mock" represents all form of test doubles: Fakes, Stubs, Spies etc. Because of this, it's hard to differentiate between different types of test doubles. In most cases, a Stub or a Fake should cover most of the needs, however with mock frameworks, usually everything becomes a mock.
Mock verification can lead to brittle tests when it is not used correctly. Let me show you this through a SUT which retrieves items from a database:
@Test
fun databaseIsCalledWhenRetrievingItems() {
sut.getItems()
verify {
mockDatabase.getAll()
}
}The test passes, so this means that everything works, right? Well, not exactly, we just verify the implementation details that the database was called, not that its return value is used. The following implementation passes, but it doesn't exactly behave how we expect:
class SystemUnderTest(private val database: Database) {
fun getItems(): List<Items> {
database.getAll()
return emptyList()
}
}A better way to test this would be to use the "Stubbing" functionality that most mock frameworks provide:
@Test
fun databaseItemsAreRetrieved() {
val items = listOf(Item("1"), Item("2"))
every { mockDatabase.getAll() } returns items
val result = sut.getItems()
assertEquals(items, result)
}The test above doesn't verify that the database was called directly, it simply checks if the SUT returned value is the same as what we set up in the mock. This way we actually verify the behavior of the SUT which is "returning items from a database" not just the implementation details. I know that the above example is really trivial, but I just wanted to get the point across, and believe me, I sometimes see the test suites where most of the focus is on the implementation details, instead of the behavior of the SUT.
In most cases, it's better to set up return values and verify that the SUT uses them correctly, instead of just checking if a function was called. However, there are cases where we want to verify that some function was called a certain amount of times, for example when checking if a function correctly returns early in some conditions. In these situations verifying the number of calls is fine, because it is part of the SUT behavior, e.g. "It shouldn't call the API again if the previous request was a success".
In the above database example, instead of using a mocking framework to set up the return value, I would simply create a FakeDatababase, which will probably be an in-memory database that behaves like the real implementation:
interface Database {
fun getAll(): List<Item>
fun addItem(item: Item)
}
interface FakeDatabase : Database {
var items: List<Item> = emptyList()
override fun getAll(): List<Item> = items
override fun addItem(item: Item) {
items.add(item)
}
}
@Test
fun databaseItemsAreRetrieved() {
val items = listOf(Item("1"), Item("2"))
fakeDatabase.items = items
val result = sut.getItems()
assertEquals(items, result)
}This fake implementation can be easily re-used across multiple tests, and makes the test code work closer to how the production code does it. That is because we're still using a database, but it is just an in-memory one.
If you're interested about different types of Test Doubles and their purpose, checkout this write-up:
Martin Fowler
8. Comparing whole complex data structures instead of only the tested fields
A lot of tests verify correctness by comparing Data Structures, usually as a result of some conversion from another type (e.g. from API or Database). However, sometimes the data structure has fields that have some complex logic behind them.
Ideally, such complex logic should be tested in isolation. Otherwise, changes in unrelated parts could cause the complex logic tests to fail, even though they should pass.
For arguments’ sake, take a look at this example, which uses some hidden logic for determining a Person's Title
enum Title { Mr, Mrs, Ms, Miss }
data class Person(
val name: String,
val age: Int,
val title: Title,
)
fun createPerson(name: String, age: String): Person {
// name and age is just assigned as is
// title is calculated based on age and name
}
@Test
fun bobAge5IsCorrectlyCreated() {
val result = createPerson("Bob", 5)
val expectedResult = Person("Bob", 5, Title.Mr)
assertEquals(expectedResult, result)
}
@Test
fun bethanyAge30IsCorrectlyCreated() {
val result = createPerson("Bethany", 30)
val expectedResult = Person("Bethany", 30, Title.Mrs)
assertEquals(expectedResult, result)
}
...
As you can see, every test repeats the assertions for the name and age property. If we were to add a new property the Person structure, all tests would need to be updated to include that property, and if the new field is complex then every test would need to take care of using the correct value.
A better way to test this logic would be to have one test that verifies the simple fields, and the rest would focus on the complex title logic
@Test
fun simplePropertiesAreCorrectlyCreated() {
val result = createPerson("Bob", 5)
assertEquals("Bob", result.name)
assertEquals(5, result.age)
}
@Test
fun bobAge5HasCorrectTitle() {
val result = createPerson("Bob", 5)
assertEquals(Title.Mr, result.title)
}
@Test
fun bethanyAge30HasCorrectTitle() {
val result = createPerson("Bob", 5)
assertEquals(Title.Mrs, result.title)
}
...This way, the complex logic has granular tests which should be less brittle and not tied up with other irrelevant logic. However, this is a double-edged sword, because while the tests are more resilient, they won't detect if a new field is added. This is up to the developer to remember about it, and add tests for it, either by adding another assertion to the simple Properties test, or by adding a couple tests is the field is more complex.
9. Not using Soft assertions
Sometimes in tests we want to make assertions on a couple of fields, either because they are part of the same behavior, or they are so simple, that we don't want to bother with adding separate tests for all of them. Or sometimes the field is more complex, and we want to verify its type, nullability etc.
Usually assertions throw an error, meaning that if we have multiple assertions, when something fails the rest of the assertions won't execute:
data class Pet(
val name: String,
val age: Int,
)
@Test
fun multipleFieldsAreCorrect() {
...
assertEquals("Kitty", result.name)
assertEquals(3, result.age)
}In the above example, if the name is incorrect, the age won't even be verified, so we're not sure if the age was correct or not. To help with this issue, we can use soft assertions:
@Test
fun multipleFieldsAreCorrect() {
...
assertSoftly {
result.name shouldBe "Kitty"
result.age shouldBe 3
}
}With soft assertions, everything will be checked first, and then the test will fail. In the above example, if the name is invalid, the age is still verified, and we get a full report of what's incorrect:
io.kotest.assertions.MultiAssertionError:
The following 2 assertions failed:
1) expected:<"Kitty"> but was:<"Cat">
at com.akjaw.example.Test.multipleFieldsAreCorrect(Test.kt:125)
2) expected:<3> but was:<4>
at com.akjaw.example.Test.multipleFieldsAreCorrect(Test.kt:126)
Most assertions frameworks have soft assertions which you can use in your tests, the above example uses Kotest assertions.
Another example where I like to use soft assertions is for the type of the class (a Kotlin sealed class in this example)
sealed class Pet {
data class Cat(val name: String, val age: Int) : Pet()
data class Dog(val name: String) : Pet()
}
@Test
fun catIsCorrect() {
...
assertSoftly {
result.shouldBeInstanceOf<Pet.Cat>()
result.name shouldBe "Kitty"
result.age shouldBe 3
}
}shouldBeInstanceOf asserts the type along with smart casting it, meaning that the result variable will behave like it was the Pet.Cat type allowing access to all the Pet.Cat specific fields.
10. Repeating the SUT Algorithm in assertions instead of an explicit value
Sometimes when writing tests, we want to go the easy way, and make them as simple to write as possible. This laziness can lead to tests which have assertions that mirror the implementation details / algorithm of the class we want to test.
class Stringifier {
fun execute(list: List<Int>): String =
list.joinToString("|")
}
class StringifierTest {
@Test
fun stringifyReturnsTheValuesSeparatedByPipe() {
val input = listOf(1, 2, 3, 4, 5)
val systemUnderTest = Stringifier()
val result = systemUnderTest.execute(input)
result shouldBe input.joinToString("|")
}
}In the above example, the assertions are the same as the SUT algorithm, and it makes the test pass. However, for more complex cases these assertions are hard to understand and sometimes require a debugging session to just understand what is the expected outcome.
This confusion makes the tests harder to understand, and if we were to refactor the algorithm to use a different function. The result would be a hard-to-understand test which doesn't even match what the system under test is doing. A better way of doing this would be:
@Test
fun stringifyReturnsTheValuesSeparatedByPipe() {
val input = listOf(1, 2, 3, 4, 5)
val systemUnderTest = Stringifier()
val result = systemUnderTest.execute(input)
result shouldBe "1|2|3|4|5"
}
This explicit assertion value makes it easy to understand what the SUT does. Additionally, we don't have to wonder about some aspects of how the SUT works, e.g: will there be pipes at the beginning or end?
