So you want to contribute to EF? Part 3: Testing

This is the third part of a series providing some background to those who may want make contributes to the Entity Framework. This post will cover the types of tests that at exist and expectations around writing new tests.

If you make a contribution to EF it is expected and required that you will write tests for that contribution and that all existing tests will continue to pass. You can run the tests from the Visual Studio solution and running the build command also runs all tests. Build must pass before any change can be pushed to master.

If you expand the Tests folder in the VS solution you will see projects for three kinds of test: unit tests, functional tests, and VB tests. All the tests make use of the xUnit test framework for which there are several runners available—my personal favorite is TestDriven.NET.

Unit tests

Unit tests test code at the unit level—that is at the class and method level. Some things to keep in mind when writing unit tests:

  • Unit tests have access to internal classes and members of the product assemblies. It is an accepted practice that some code is made internal (as opposed to private) purely to facilitate testing.
  • Unit tests make use of Moq for mocking dependencies. The way you use Moq (if at all) is not strictly dictated.
  • Unit tests must run fast; it is almost always inappropriate to take slow actions such as accessing a database in a unit test.
  • Unit tests should be organized into namespaces and classes that match the namespaces and classes of the product. For example, tests for System.Data.Entity.Config.DbConfiguration should be placed in System.Data.Entity.Config.DbConfigurationTests. (Note that many existing tests don’t follow this convention but all new tests should.)
  • You may optionally group unit tests into a nested class named after the method being tested if this helps clarity. For example, see DbConfigurationTests.
  • It is expected that every change is unit tested. If you submit a contribution without full coverage in unit tests it will almost certainly not be accepted until the tests have been written. This is just the same as contribution from a member of the EF team which would also be rejected without tests.

Functional tests

Functional tests exercise many units of code to ensure that all the units all work together as expected. (The exact level varies such that some EF functional tests may be more correctly called integration tests.) Examples include writing some data to the database and then querying it back to ensure that it round-trips correctly, or building and validating an entire Code First model.

Some things to keep in mind when writing functional tests:

  • Functional tests should not access internal classes and members; they should make use of only the public API. (At the time of writing the functional tests do sometimes make use of internals for historical reasons.)
  • Functional tests should still run fast but are expected to do things like access the database and so can be slower than the unit tests.
  • Functional tests are usually organized into “scenario” classes that group tests by the type of scenario being tested.
  • Functional tests are not required for every change if it is sufficiently tested by unit tests and the existing functional tests. However, most changes will require some functional tests to ensure that the behavior is working end-to-end.

VB tests

The VB tests test Visual Basic-specific code, mostly involving Visual Basic code generated by the Database First T4 templates. There is no expectation that VB tests be added for most changes.

Writing and naming tests

In general, each test method should be a simple, self-contained test. It is fundamental that every test be easy to read and understand and that the expected outcome is readily apparent. If a test fails then it should not require debugging to figure out how the test code failed. (It may of course require debugging to discover the root cause of the failure in the product code.)

Test method names

When reading a test method name it should be clear:

  • What is being tested
  • What the expected outcome is

For example:

[Fact]
public void Add_starts_tracking_a_new_entity_and_puts_it_in_the_Added_state()
{
    // Test some stuff...
}

Keeping tests simple and clean means that good test code does not have the same requirements as good product code. In particular:

  • Duplication of test code in many tests is acceptable when it keeps each test easy to read and understand. In other words, don’t repeat yourself (DRY) does not apply to test code in the same way that it applies to product code.
  • Copy-and-paste followed by some small edits is an acceptable way to create a new test that covers a slightly different scenario than an existing test. You may choose to factor the common code out but don’t feel that you must always do this—use your judgment.
  • Avoid anything complicated, including elaborate testing frameworks and validation mechanisms. Simple asserts are good.
  • Avoid exhaustive testing by looping over all possible values and especially all possible combinations of several values. It is hard to ensure that these tests are really validating what they think they are, they are often slow to run, and can become hard to understand or debug, especially when they fail.
  • Don’t write anything to the console for manual verification or create text baselines. Tests should use assertions for verification.

Tests for core code

We faced a difficult situation when deciding to make EF open source—what do we do with all the tests written by both product developers and the QA team over the years? Making most of the DbContext tests and a significant proportion of the Code First test open source was not a problem and we have done that.

On the other hand, most of the test for the core code and almost all of the tests written by the QA team could not be made open source. I won’t go into the reasons for this, but the upshot is that if you run “build” on the open source code most of the 6000+ tests that run are for DbContext and Code First. This isn’t as bad as it might seem because the DbContext and Code First code is dependent on the core code and hence you get a lot of coverage of the core just through running these tests.

One thing this does mean is that if you make changes to the core code then you are likely to find that the code you are changing doesn’t have any existing public tests. (We can still run some tests internally but as much as possible we want tests to be public.) It is expected that you write tests for this code as part of your change so that we gradually build up direct test coverage for the core code. If doing this is such a burden that you feel you can’t make a contribution because of it then contact the EF team and we will try to figure out a solution.

Designing and refactoring for testability

A lot of the EF code, especially in the core, was not designed for testability. This means that you may need to do some refactoring in order to introduce seams, make classes mockable, and so on. This video has a good introduction to testable code.

All new code should be designed and implemented with testability in mind. Just because you see patterns or practices in the existing code that are not testable doesn’t mean that we want the code to stay that way or that new code should follow the same patterns. Design and refactor for testability so that we can keep improving the code base.

Summary

All contributions must be unit tested and most must also have functional tests. All tests use xUnit and should be as simple and clear as possible.

If there is anything here that you need more information about don’t hesitate to contact me or others on the EF team. In the next post I’ll look at the developer experience.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s