TL;DR
This weekend I attended CITCON2024 held in Zagreb. One session caught my attention, To Mock or not to Mock? Session author used some mocks in her unit testes, and wanted to present her approach in order get feedback on it.
The Problem
On the image above, you can see the problem. A system is owned by the author of unit test, while B and C are external services. Module in service A has logic related to responses from B an C. Depending on response from B, module in service A could make a request towards system C, or stop interaction. A, B and C communicate via HTTPS protocol. Five mocks were created in unit tests that test this logic (language is Typescript and tool is Jest). Question was, was that too much of mocks?
When to Mock?
Unit Testing. You want to mock in your unit tests for five reasons:
- Isolation. Once you write your mock, you do not need to worry about external service. You just concentrate on logic in module that you are testing.
- Kills the flakiness. Network communication is very unreliable. Without mocks, it is for sure that your unit tests will fail from time to time with “Network error”.
- Performance. Communication over network is much slower that using local mock calls. Test run in seconds versus microseconds? What would you like to have?
- Cost. Interaction with external service costs money. One good example is ChatGpt API.
- Rate-limited external services. External services have limits on how many times you can call them per minute. As your unit test are fast, this rate-limit could be depleted per test run. Again, example is ChatGpt API.
When not to Mock?
Any other type of testing: integration testing, security testing, end to end testing. In those cases you want that your application talks to the real systems (staging or live). You need real data from those services, not mocked one.
Discussion
In our group, there were four developers and two QA. Developers arguments aligned with When to Mock?, and QA arguments aligned with When not to Mock? And that was expected. One problem was that QAs dismissed When to Mock? arguments. Reason is that they write end-to-end test automation, and this is the only test automation in their organizations. Unit testing is rare breed!
Time mocking
It popped up that time mocking in unit testing is an absolute must! Reasons are time zones. If you have logic in your application related to time zones, your tests that are using today() function could fail depending if test was run in the morning or in the afternoon. So for example if you have date range calculations that include time zones, morning test run will have expected date range, but afternoon test run could have different date range, depending on time zone that you are setting in your test.
Do not mock what you do not own
The quote “Do not mock what you do not own” is attributed to Tatham Oddie. Reasoning is that you have control and behavioural understanding of mocked component. I think that in the open source world where you have at disposal source of libraries that you are using, this quote does not holds the water.
Here is one example how I mock in Elixir ChatGpt AI library that is a wrapper around OpenApi library:
describe "get_type/1" do
test "returns the event type" do
with_mocks([
{AI, [], [chat: fn _prompt -> {:ok, "crime"} end]},
{AI, [], [sigil_l: fn _prompt, _ -> [] end]},
{AI, [], [sigil_LLM: fn _prompt, _ -> [] end]}
]) do
assert "crime" == ChatGpt.get_type("Some news")
end
end
end
def get_type(news) do
case chat(
~l"model:gpt-4 user: Please return event type, with no any additional text. Use only one word to describe type. If event is crime, return crime. If event is related to DUI, return dui. If you can not determine event type, return other. #{news}"
) do
{:ok, type} ->
String.slice(type, 0, 254)
{:error, error} ->
case wait_time(error) do
{:ok, wait_time} ->
:ok = :timer.sleep(wait_time)
get_type(news)
{:error, _} ->
Logger.error("#{__MODULE__}.get_type: #{error}")
"chatgpt error"
end
end
end
Here we test function get_type that calls external system. In mock for any input to get_type we always return {:ok, “crime”}. We also mock methods sigil_l and sigil_LLM, because this is how Mock library is implemented, you must always mock all public functions.
Conclusion
If you mock your calls with external systems, you can achieve 100% line code coverage. This is only important if you change implementation of your subsystem that communicates with external system. So with test you can check your design assumptions.
Mocking risk is that interface with external system changes, but your mock does not. Then your integration test will fail. This is why we do integration test. But when you mock library, and you update that library and interface changed, your mock could detect that change. For example method name changed or function attribute was added.
But that reveals bad design of library that you are using and it reveals that author of the library does not care about backward compatibility.
Tetris Design
One interesting design term popped out (participant on the left in photo), Tetris Design. It is in lower part of photo. When you try to mock subsystem (or module, or library) and you realize that it’s interface does not have clean edges as Tetris elements, than this is a code smell of bad design.
