Unit tests in your average web app
Some 10 years ago, when I was just starting my software developer career, I saw much less investment in automated software testing than we have today. Be it unit tests, integration tests, or any other kind of tests, whatever you call them. Nowadays, the biggest camp seems to think the more you have unit tests, the better (Uncle Bob camp). And there is Jim Coplien, who thinks we’ve gone too far with promoting unit tests, to a zone of negative ROI.
I want to share my journey through the topic of unit tests, as well as higher level tests. From what I was thinking as just initiated programmer, to what are my beliefs now.
When I first started coding and heard about unit testing, I thought it was crazy. To have a comprehensive unit test suite, you’d have to add smth like 3x LOC of your production code just for tests. That was my estimate, probably way off, but even today I believe it’s huge amounts of code, way more than system under test. And what you get in return? A guarantee that you’ve coded exactly what you thought you’ve coded. This obviously goes against developer’s self-esteem. Are you so incapable of writing correct code that you have to double check yourself? So I thought, there is no way I’m unit testing anything if it is all my personal project. I’m just too proud of myself. For collaborative projects, I thought it makes sense to have some unit tests in critical places, so that someone else who knows little about the code he’s working on doesn’t break it.
The reality is quite different though. Everyone makes mistakes, much more in fact than they recognize. I also made mistakes which made my code work not as expected. Then I thought I just need to get better at coding, and the mistakes will go away. But years passed, and I was still making errors in my code. And it is embarrassing ofc. So I started thinking, how can I fix this? Looking at those errors, they were mostly trivial stupid mistakes you laugh at when you spot them. You obviously didn’t mean to write that. You cannot combat absent-mindedness, no matter how focused you try to be when coding, eventually you’ll write something stupid. Other time, you’d modify some component to accommodate for someone that is using it, being not aware that there are others using it and not expecting that change in behavior. That’s when I turned to unit testing. It was on a hype at the time, and seemed to help with both those problems. The systems I was working on were not designed to be unit testable. But still, I started writing some unit tests in places where it was easy to add them, and thinking about how to make unit testable systems. I cannot say I noticed any immediate positive effects back then, with the exception of being more confident about the code that is covered by tests. The problem was, unit tests would still cover only very small amount of code, the “business logic” which was easily testable. And if you look at any function[ality] of the system, it’s never just business logic. E.g., typically there is interaction with database. Or anything else which is called side effects in FP.
Unit testing enthusiasts offered a solution to this problem — mocking. You can mock any outside dependency for your code to interact in a way that you define, and not in a way that real thing works. I played with mocking techniques for a bit and saw value in them, but I thought it to be too much of an effort to be universally used. Faking the real thing and expecting mock that you’ve programmed to behave exactly the same way means the interface to that dependency has to be very simple and well understood. Should work nicely with FTP client. Not the case with SQL database where you pass SQL query, and expect some rows to be returned. So, to me mocking was not an answer I was looking for. Just yet another contextual technique in tool belt.
Then I realized I was too focused on how to make unit tests cover whole system with reasonable amount of code (i.e., no mock proliferation). What about more end-to-end tests, such as integration? At some point earlier, I’ve dismissed them. I thought, the more code that you cover, the more there are possible execution paths, and so the amount of tests required grows exponentially. And they’re slow. If you have everything covered in unit tests, there is no need for integration tests. I regarded them as just compulsory method to test something, if the system was built non unit testable. Then I heard people saying — but that’s not the point of integration tests. They are not supposed to test logic. They just test that components are correctly wired up. Most trivial success path test is often what is needed. And you already do this kind of testing manually. Say you’re implementing some HTTP API endpoint. You will be hitting it with some HTTP client multiple times to make sure it works. You’ll have to setup some test data, and you’ll have to fill in the request data. Is that so much easier than doing this programmatically? If you know how to code, then no, it is comparable effort. And as a bonus, you now have a regression check. That sounded reasonable. And given how much of a coding is usually just moving data around from component to component through multiple layers until it reaches the database, and then all the way back to the UI, call it CRUD if you will, with little interesting logic involved, such test is what gives the most bang for the buck.
Fast forward to today
Currently, I do write some unit tests for parts with more involved business logic, and where it makes sense to have it expressed in code with no side effects. But these are few. Most of the code that I write is generating SQL, and passing it to database. And it can be any SQL, we extensively use MERGE statements, and in general try to offload as much as possible to single SQL statement. Sure it goes against testability — you cannot reasonably mock SQL queries. So the majority of tests that I write are kind of integration tests, with real database. They cover different scenarios, not just simplest success path. ORMs are too clumsy and not powerful enough to let you make the query exactly as you want it. I am very much for state based testing, as opposed to behavior testing that mockists advocate.
What about complete end-to-end tests (e.g., Selenium)? My experience is that most often they are not worth it. The UI is not natural interface to manipulate programmatically. It is for humans. If you run such tests on a system with persistent state, it is the data that someone changes that usually makes them fail. Also, very time consuming to debug. Better have a test suite for UI logic which mocks API calls, and doesn’t care about presentation. Contemporary UI frameworks such as Angular fully support that.
As a final note. Most errors in code are trivial mistakes which any kind of test exercising that code would catch. Just write any test that covers the code, unit or whatever. The less code that the test exercises, the easier it is to debug it later. But if I can write unit test which would require mocking, and I can write integration test which checks system state, I’m going with latter. It’s easier, more maintainable, less fragile. Also, e2e tests are usually not worth the effort. In theory they can verify that the frontend is communicating with backend as expected, that is following the same protocol (which is mocked in other frontend tests). The API interface however is the one thing that I find is worth mocking. It is hard but not impossible to mock it wrong, and I rarely see that happen to justify the costs of e2e tests. Usually what such tests catch is something that could have been more easily checked by lower level tests.
