Last week I struck a chord in the Elixir community when I tweeted about a trap I fell into while writing a seemingly simple test using Elixir’s with special form. Based on the reaction to that tweet, I thought it’d be a good idea to explore where I went wrong and how I could have prevented it.

The Test

The test in question was fairly simple. Let’s imagine it looked something like this:

test "foo equals bar" do
  with {:ok, foo} <- do_foo(),
       {:ok, bar} <- do_bar() do
    assert foo == bar
  end
end

We’re using with to destructure the results of our calls to do_foo/0 and do_bar/0 function calls. Next, we’re asserting that foo should equal bar.

If do_foo/0 or do_bar/0 return anything other than an :ok tuple, we’d expect our pattern match to fail, causing our test to fail. On running our test, we see that it passes. Our do_foo/0 and do_bar/0 functions must be working as expected!

The False Positive

Unfortunately, we’re operating under a faulty assumption. In reality, our do_foo/0 and do_bar/1 functions actually look like this:

def do_foo, do: {:ok, 1}
def do_bar, do: {:error, :asdf}

Our do_bar/0 is returning an :error tuple, not the :ok tuple our test is expecting, but our test is still passing. What’s going on here?

I find myself producing lots of ranging content from books and articles, like the one you're reading now, to open source software projects. If you like what I'm doing, nothing shows your support more than signing up for my mailing list.

It’s easy to forget (at least for me, apparently) that when a with expression fails a pattern match, it doesn’t throw an error. Instead, it immediately returns the unmatched value. So in our test, our with expression is returning the unmatched {:error, :asdf} tuple without ever executing its do block and skipping our assertion entirely.

Because our assertion is never given a chance to fail, our test passes!

The Fix

The fix for this broken test is simple once we recognize what the problem is. We’re expecting our assignments to throw errors if they fail to match. One surefire way to accomplish that is to use assignments rather than a with expression.

test "foo equals bar" do
  {:ok, foo} = do_foo()
  {:ok, bar} = do_bar()
  assert foo == bar
end

Now, the :error tuple returned by our do_bar/0 function will fail to match with our :ok tuple, and the test will fail. Not only that, but we’ve also managed to simplify our test in the process of fixing it.

Success!

The Better Fix

After posting the above fix in response to my original tweet, Michał Muskała replied with a fantastic tip to improve the error messaging of the failing test.

Michał's pro tip.

Currently, our test failure looks like this:

** (MatchError) no match of right hand side value: {:error, :asdf}
code: {:ok, bar} = do_bar()

If we add assertions to our pattern matching assignments, we set ourselves up to receive better error messages:

test "foo still equals bar" do
  assert {:ok, foo} = do_foo()
  assert {:ok, bar} = do_bar()
  assert foo == bar
end

Now our failing test reads like this:

match (=) failed
code:  assert {:ok, bar} = do_bar()
right: {:error, :asdf}

While we’re still given all of the same information about the failure, it’s presented in a way that’s easier to read and internalize, leading to a quicker understanding of how and why our test is failing.

I’ll be sure to incorporate that tip into my tests from now on. Thanks Michał!