I do have some skin in the game here, I'm one of the maintainers of Mockery, the mocking framework Frank uses in his examples. I maintain it because I use it a lot, but it doesn't provide any income etc, so my defence is purely theoretical. I won't lose my livelyhood if people stop using Mockery.

NB: When I use the word "mock" in this post, I usually use it in the vague way of describing any kind of test double, rather than the by the book definition.

I'm going to focus on the examples Frank uses, how I would see it and were I would go from there.

/**
 * @test
 */
public function testing_something_with_mocks(): void  
{
    $mock = Mockery::mock(ExternalDependency::class);

    $mock->shouldReceive('oldMethodName')
        ->once()
        ->with('some_argument')
        ->andReturn('some_response');

    $dependingCode = new DependingCode($mock);
    $result = $dependingCode->performOperation();

    $this->assertEquals('some_response', $result);
}

Frank argues that performing a rename method refactoring with his IDE, this test would be broken. I have no argument against this, when it happens, I fix the tests manually.

In more recent versions of Mockery, we can remove some of the stringiness, but I still don't think it would make the IDE's any more capable of refactoring without some sort of dyanmic return types for the Mock::allows and Mock::expects methods.

$mock->allows()
    ->oldMethodName('some_argument')
    ->andReturns('some_response');

It won't help much with the refactoring, but it looks a lot better and would give us some extra context when searching the codebase in that we can see it is a method call, rather than just a string matching the old name.

Frank then goes on to show a more concrete example, with an InvoiceService. His first iteration of the test looks like this:

use League\Flysystem\Filesystem;  
use PHPUnit\Framework\TestCase;

class TestInvoiceServiceWithMocksTest extends TestCase  
{
    /**
     * @test
     */
    public function invoicing_a_client(): void
    {
        // Need to work around marking this test as risky
        $this->expectNotToPerformAssertions();

        $mock = Mockery::mock(Filesystem::class);
        $mock->shouldReceive('write')
            ->once()
            ->with('/invoices/abc/2020/3.txt', '{"client_id":"abc","amount":42,"invoice_period":{"year":2020,"month":3}}');

        $invoicePeriod = InvoicePeriod::fromDateTime(DateTimeImmutable::createFromFormat('!Y-m', '2020-03'));
        $invoiceService = new InvoiceService($mock);
        $invoiceService->invoiceClient('abc', $invoicePeriod);
    }

    protected function tearDown(): void
    {
        Mockery::close();
    }
}

First thing I notice is that Frank isn't using Mockery's PHPUnit Integration. We ship a base MockeryTestCase, or you can use a trait. I've just created a PR to add a note to the readme about this.

Pulling in the integration makes the tearDown method and telling PHPUnit to ignore the lack of expectations redundant, so we can remove them.

use MockeryPHPUnitIntegration;

/**
 * @test
 */
public function invoicing_a_client(): void
{
    $mock = Mockery::mock(Filesystem::class);
    $mock->shouldReceive('write')
        ->once()
        ->with('/invoices/abc/2020/3.txt', '{"client_id":"abc","amount":42,"invoice_period":{"year":2020,"month":3}}');

    $invoicePeriod = InvoicePeriod::fromDateTime(DateTimeImmutable::createFromFormat('!Y-m', '2020-03'));
    $invoiceService = new InvoiceService($mock);
    $invoiceService->invoiceClient('abc', $invoicePeriod);
}

Now the PHPUnit awkwardness is out of the way, Frank notes three things:

• Amount of mocking code vs other code.
• High coupling between test and implementation.
• The test does not describe desired behavior, it validates implementation.

Only the first of those items is specific to using mocking frameworks, the second two are possible regardless of mocking frameworks.

Frank's first step in improving this test is to replace the generated mock with a concrete one. This does give him some improved readability with regards to Arrange, Act, Assert, but that is achievable using a mocking framework too and is the first step I would have taken.

/**
 * @test
 */
public function invoicing_a_client(): void
{
    // Arrange
    $mock = spy(Filesystem::class);
    $invoiceService = new InvoiceService($mock);

    // Act
    $invoiceService->invoiceClient(
        'abc'
        InvoicePeriod::fromDateTime(DateTimeImmutable::createFromFormat('!Y-m', '2020-03'))
    );

    // Assert
    $mock->shouldHaveReceived()
        ->write('/invoices/abc/2020/3.txt', '{"client_id":"abc","amount":42,"invoice_period":{"year":2020,"month":3}}');
}

This looks very similar to Frank's first revision. I've replaced the mock with a spy, which allows me to verify that calls were made after the fact, rather than declaring them up front.

The big difference is Frank's test uses a concrete implementation of a FileSystem. Frank gains confidence from this, because he knows the InMemoryFilesystemAdapter runs against the same set of tests as the other more useful versions. I wouldn't have the same confidence as Frank, because I don't trust that library. I don't trust most of the libraries I use and I try not to mock types I don't trust. And by that I mean either mocks generated by a framework or concrete mocks created by hand.

Both my version and Frank's version still have that big horrible JSON string, the high coupling between the test and the implementation. Both versions still don't describe the desired behaviour, they describe what the SUT does and what the outcome is, with explicit detail. We both tackle this next, but in a different way.

The domain experts tell us the Invoice should be submitted to the InvoicePortal, so I'm going to update my test to describe that exactly. I'm also going to make sure the test name describes that.

/**
 * @test
 */
public function invoicing_a_client_submits_the_invoice_to_the_invoice_portal(): void
{
    // Arrange
    $invoicePortal = Mockery::mock(InvoicePortal::class);
    $invoiceService = new InvoiceService($invoicePortal);

    // Act
    $invoicePeriod = InvoicePeriod::fromDateTime(DateTimeImmutable::createFromFormat('!Y-m', '2020-03'))
    $invoiceService->invoiceClient('abc', $invoicePeriod);

    // Assert
    $expectedInvoice = new Invoice('abc', 42, $invoicePeriod);
    $invoicePortal->shouldHaveReceived()
        ->submitInvoice(equalTo($expectedInvoice));
}

At this stage, the InvoicePortal doesn't exist. The author's behind the GOOS book call this programming by wishful thinking. I know the InvoiceService needs to submit an Invoice to an InvoicePortal, so I'm going to describe that in my test and Mockery facilitates it for me. I then would update the SUT accordingly. The astute among you will notice that Mockery isn't really helping me test the SUT here and we will need more tests to actually ensure the system works. What Mockery is doing, is helping me design the SUT. It has perfectly described that I will need an InvoicePortal, and that thing will need a submitInvoice method that takes an Invoice.

I would now move in to the next layer of my application and start building the InvoicePortal, which will have it's own suite of tests, just like in Frank's example. One difference being, I have defined my meaningful boundaries already, by writing an executable example of the way the consumer, would like things to work. By writing the InvoicePortal first, Frank is defining the meaningful boundaries in the inner layer, and once he has built them, the outer layer gets to try them on for size.

In the outside-in methodology, the outer layers describe what is required from the inner layers, and then the programmer builds them. In the inside-out methodology, the programmer builds the inner layer which prescribes what is available and the outer layer works around it. With experienced programmers, the outcome will most often be the same, just a slightly different journey.

Another difference is that I wouldn't have bothered creating a FakeInvoicePortal. I already have a fake generated for me by Mockery and unless I'm going to get a lot of benefit and reuse, I don't take the time to write concrete fakes by default.

What did we achieve?

We still don't have the nice IDE refactoring as mentioned earlier, but I would argue that my example achieves the same other things Frank achieved with his refactoring, but with less effort and more preferable (to me) design guidance.

The tests no longer contain implementation details.

We have the same outcome here, there are no more or no less implementation details in either my test or Frank's. You could argue about the amount of coupling, Frank's are coupled to his manually crafted mock, mine are coupled to Mockery's generated mocks.

Clearer high-level code.

This is irrelevant with regards to the tests and therefore mocking frameworks or not, the code for the SUT is identical for both my way of work and Frank's.

Low-complexity fakes are easy to control.

I personally find Mockery's mocks very easy to control, but I have a bias there. Mockery ships with the ability to stage exceptions and fix responses and if you know how to use them, would save you time writing code. That said, it's far easier to understand and trust code you have written yourself, so I see where Frank is coming from. I personally trust Mockery enough to lean on it and save myself the time.

$invoicePortal = Mockery::mock(InvoicePortal::class);
$invoicePortal
    ->allows()
    ->submit(anyArgs())
    ->andThrows(new \Exception());

Fin.

⚠️ Update!

Troy Hunt recently introduced HIBP Passwords, a freely downloadable list of over 300 million passwords that have been pwned in the various breaches the site records. There is an API to access the list for auditing and checking passwords, but it's rate limited, and I thought it would be more friendly to import the passwords in to a database we control. It looks like HIBP uses Azure Table Storage to make the data quickly accessible, I do most of my work on AWS so I thought I'd take a look at importing the hashes in to DynamoDB. It's relatively cheap to run and easy to use. I thought it might be useful for others, so here's the rundown.

The first thing I tried was to follow a tutorial using AWS Data Pipeline, which seemed to be exactly what I needed. In the end though, I found the tutorial made a few assumptions that I either missed or they failed to mention, mostly around the expected data format for the CSV files. Thankfully though, this lead me towards using AWS Elastic Map Reduce directly and this turned out to be the winning formula.

To start with, I needed to get the data from HIBP and upload it to S3. I think this method would have worked if the source files had been gzipped, but I wasn't too sure about 7zipped, so I decompressed them before uploading them to S3.

mkdir hibp-passwords
cd hibp-passwords
wget https://downloads.pwnedpasswords.com/passwords/pwned-passwords-1.0.txt.7z
wget https://downloads.pwnedpasswords.com/passwords/pwned-passwords-update-1.txt.7z
wget https://downloads.pwnedpasswords.com/passwords/pwned-passwords-update-2.txt.7z
7zr e pwned-passwords-1.0.txt.7z
7zr e pwned-passwords-update-1.txt.7z
7zr e pwned-passwords-update-2.txt.7z
rm *.7z
aws s3 mb s3://hibp-passwords-123
aws s3 sync ./ s3://hibp-passwords-123

I created a DynamoDB table to hold the data. It only needs the one column, which will also be the partition key.

Now adjust the write capacity to allow our import to go reasonably quickly. I set the capacity to 10000 units, which I think was the maximum without having to asking AWS to lift the limits.

We then need to create our EMR cluster. I experimented with different sizes to begin with, but settled on a cluster of 16 c4.8xlarge instances. I left the rest of the settings as the defaults. Between this and the write capacity we set on the DynamoDB table, this meant the import took around 16 hours. There's probably a better combination of cluster size and write capacity, but it was good enough for me. I should point out here that this isn't a cheap way to do this, I think the work of this EMR cluster will come to around $400. I imagine someone who knows what they are doing with Hadoop/Hive/etc, could do this much more efficiently.

Now we're all good to go. Log in to the EMR master node, install tmux and fire up Hive.

ssh -i ~/.ssh/your-key.pem hadoop@your-master-node-public-url
sudo yum install tmux
tmux
hive

We need to tell Hive to use as much DynamoDB write capacity as it sees fit, no other resources are currently trying to access the table.

> SET dynamodb.throughput.write.percent=1.5;

We then create our first Hive table, by telling it where to find our data.

> CREATE EXTERNAL TABLE s3_hibp_passwords(a_col string)
    ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' 
    LOCATION 's3://hibp-passwords-123/';
OK
Time taken: 3.371 seconds

The first time I tried running the import, I ran in to trouble with it trying to enter duplicate items in to the DynamoDB table. A bit of searching lead me to a stackoverflow question and a means to create another table, but with only unique hashes.

> CREATE TABLE s3_hibp_passwords_dedup AS
    SELECT a_col
    FROM (SELECT a_col, rank() OVER
            (PARTITION BY a_col)
            AS col_rank FROM s3_hibp_passwords) t
    WHERE t.col_rank = 1
    GROUP BY a_col;
Query ID = hadoop_20170810144127_818de1ea-0c47-4a16-a5bf-5b77e3336d69
Total jobs = 1
Launching Job 1 out of 1
Tez session was closed. Reopening...
Session re-established.


Status: Running (Executing on YARN cluster with App id application_1502375367082_0202)

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container     SUCCEEDED    257        257        0        0       0       0
Reducer 2 ...... container     SUCCEEDED     53         53        0        0       0       0
Reducer 3 ...... container     SUCCEEDED     27         27        0        0       0       0
----------------------------------------------------------------------------------------------
VERTICES: 03/03  [==========================>>] 100%  ELAPSED TIME: 105.63 s
----------------------------------------------------------------------------------------------
Moving data to directory hdfs://ip-172-31-43-51.eu-west-1.compute.internal:8020/user/hive/warehouse/s3_hibp_passwords_dedup
OK
Time taken: 112.395 seconds

Now we need to create another hive table, this one will be backed by our DynamoDB table.

> CREATE EXTERNAL TABLE ddb_hibp_passwords (col1 string)
    STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler' 
    TBLPROPERTIES ("dynamodb.table.name" = "pwned_passwords", "dynamodb.column.mapping" = "col1:sha1");  
OK
Time taken: 0.925 seconds

That's it, we're all ready to go. This is going to take some time, so set it running and go to bed.

> INSERT OVERWRITE TABLE ddb_hibp_passwords 
    SELECT * FROM s3_hibp_passwords_dedup;
Query ID = hadoop_20170810144407_872afa4b-ca88-4b57-a284-5496fcb0d30f
Total jobs = 1
Launching Job 1 out of 1


Status: Running (Executing on YARN cluster with App id application_1502375367082_0002)

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container     SUCCEEDED    254        254        0        0      18       0
----------------------------------------------------------------------------------------------
VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 59056.94 s
----------------------------------------------------------------------------------------------
OK
Time taken: 59058.426 seconds
hive>

I don't know what happened with the failures, I didn't bother checking. I know I ended up with 319,169,449 hashes in the database, so good enough for me.

Back in the terminal, we can quickly check to see how this is going to work. Note that I shifted the hash to uppercase, the HIBP source data hashes are all uppercase.

davem@wes:~$ SHA1DAVE=$(echo -n dave | sha1sum | tr "[a-z]" "[A-Z]" | awk '{ print $1 }')
davem@wes:~$ echo $SHA1DAVE
BFCDF3E6CA6CEF45543BFBB57509C92AEC9A39FB
davem@wes:~$ aws dynamodb get-item --table-name hibp_passwords --key "{\"sha1\": {\"S\": \"$SHA1DAVE\"}}"
{
    "Item": {
        "sha1": {
            "S": "BFCDF3E6CA6CEF45543BFBB57509C92AEC9A39FB"
        }
    }
}
real 0.46
user 0.30
sys 0.04
davem@wes:~$

Have fun auditing your passwords.

Feature::isEnabled("something_awesome");

I needed to force a particular feature on in a PHPUnit integration test, but in order to tidy up after myself, I would need to ensure that the test reset the Feature system after it had finished. There are a few ways of doing this.

The first is to consider using PHPUnit's built in features for preserving global state . I've never really used them, but I have seen plenty of problems caused by them, so I didn't even bother looking.

Another option is to wrap the test execution in a try/catch/finally block and ensure the global state is reset regardless of what the test code does:

    /** @test */
    public function single_nasty_test_with_global_scope_changes()
    {
        GlobalScope::$thing = true;

        try {
            // exercise system that uses GlobalScope::$thing
            $this->assertTrue(GlobalScope::$thing);
        } finally{
            GlobalScope::$thing = false;
        }
    }

This isn't too bad, but I'd prefer to avoid the indentation and would look a lot messier if there were several lines of test code.

Another option, you could add the reset code to your tearDown method or add a totally new @after method.

    /** @after */
    public function reset_global_state()
    {
        GlobalScope::$thing = false;
    }

    /** @test */
    public function single_nasty_test_with_global_scope_changes()
    {
        GlobalScope::$thing = true;

        // exercise system that uses GlobalScope::$thing
        $this->assertTrue(GlobalScope::$thing);
    }

This isn't too bad again, but it gets a bit lost for me, it applies to every test method in the class (which would be a dozen or so in my instance), rather than the one test method that needs it. It's also separated from the test method, so it's not immediately clear if the global scope is being reset accordingly.

None of these really appealed to me, so I had a quick look inside PHPUnit to see if it had anything that would allow me to set it up contextually, right next to that particular test methods setup. There wasn't that I could see, but it didn't take two minutes to write this little trait:

<?php

trait AfterHooks
{
    private $afterHooks = [];

    public function after(callable $callback)
    {
        $this->afterHooks[] = $callback;
    }

    /**
     * @after
     */
    public function runAfterHooks()
    {
        $afterHooks = $this->afterHooks;
        $this->afterHooks = [];

        foreach ($afterHooks as $afterHook) {
            $afterHook();
        }
    }
}

And here is how you use it:

    use AfterHooks;

    /** @test */
    public function some_nasty_test_with_global_scope_changes()
    {
        $this->after(function () { GlobalScope::$thing = false; });

        GlobalScope::$thing = true;

        // exercise system that uses GlobalScope::$thing
        $this->assertTrue(GlobalScope::$thing);
    }

    /** @test */
    public function another_test()
    {
        // exercise system that uses GlobalScope::$thing
        $this->assertFalse(GlobalScope::$thing);
    }

Doesn't have to be global scope, you can use it to tear down anything that's particularly relevant to a specific test. You are welcome.

Running specific tests and test classes

As I am writing code, I'm usually practicing TDD and writing my tests before writing and refactoring code. This usually means I have a test class open and at least the corresponding code class open in splits on the screen. I move back and forth between the splits, writing test code, production code and refactoring. If I were to run the whole test suite every time I've finished writing something, it would make the feedback loop too slow for me on even a medium sized project. As I write a test, if the whole set of tests in the class run very quickly, I might run them all by asking PHPUnit to run that test file, allowing me to quickly check the correctness of the test I'm working on, that it's failing and that the production code I write then makes it pass.

phpunit tests/integration/App/Controller/AccountControllerTest.php

However, usually I prefer to run just that single test that I am working on right now. PHPUnit has a filter switch, which makes this possible, though it can be a little fiddly.

phpunit --filter test_it_should_deny_unauthenticated_requests tests/integration/App/Controller/AccountControllerTest.php

I'll then run the full test class after I've finished refactoring, ensuring that I haven't affected any related code.

Vim Bindings

In order to facilitate this, I use a set of vim key bindings, mostly inspired by watching Gary Bernhardt's Destroy all Software.

<leader>t If I am in a PHPUnit test file, save and run the file. If I am not in a test file, save the file I am in and run the last test file I ran.

<leader>s If I am in a PHPUnit test file, save the file and run the test case the cursor is in. If I'm not in a test file, save the current file and run the last test case I ran.

I love how quick and easy this makes running the tests and because I use terminal vim, I run them right there using bang(!) and my attention doesn't leave the current window. Hitting enter dismisses the results.

I also have a few other key bindings set up, though I don't use them all that often.

<leader>Tc Run PHPUnit with code coverage on the current test file, or the last test file if looking at production code.

<leader>Td Run PHPUnit with TestDox output on the current file or the last run test file if looking at production code.

<leader>T Run PHPUnit without any arguments, so the default test suite as per the phpunit.xml configuration.

There are a whole host of ways to set this kind of thing up with Vim. Some people like to have Vim send commands to a tmux pane and have the results show there.

Whatever your favourite editor is, it might be worth seeing what sort of support they have for running tests.

Running with filesystem watchers

It's always my preference to run the tests whenever I see fit, but if your editor doesn't make it easy, or if it doesn't suit your workflow, perhaps some of the file watcher setups could be right for you.

These tools watch the filesystem and run PHPUnit for you whenever a file is updated. I think most people have them running in a window on screen, but I know some of them show you pass/fail notifications via growl and such.

I think one of the most well known tools is Ruby's combination of guard and rspec, but Eric Hogue has a post on how he uses guard with his PHPUnit tests. There are loads of other tools, a quick search shows there are a bunch of packages available for grunt and gulp, I'm sure one of them works. If you use one and really like it, comment below or tweet at me and I'll update this post with details.

Groups/Suites

Moving on from the inner TDD loop, having your test suite organised into directories, suites or groups can be really beneficial for running on workstations. Sometimes we have tests that are slow to run or simply can't be run due to hardware, operating system, licencing or networking, so making sure developers can run a reasonable set of tests before commiting or merging code is essential. Giving the developers a choice of different runs can make the feedback loop more pleasant, meaning the tests get run more often, but not so often that you're wasting developer time and CPU cycles.

Continuous Integration

Here is the place to make sure all of the tests get run at some point. My preference would be for all tests to run before code is released. However, if you have some tests that are too hard or to slow to run before every release (assuming you've done everything you can to fix those tests), and the risk is acceptable, having those tests run in a different CI task is a good idea, allowing your release cycle to move quickly.

As mentioned in the organising your test suite post, being able to run your fastest tests first can help the test runs fail fast, meaning you get to work fixing the code faster.

Another quick tip, unrelated to testing itself, if you run any code coverage or static analysis on the code purely for metrics, have that run separately from the main release build, either automatically or on demand. These things take time and if a drop in code coverage or an increase in cyclomatic complexity wouldn't stop you from releasing code, make it part of a separate build/task.

Faster tests in PHP

This is the third in a series about keeping your test runs fast, check back for more or leave your email address below to get notified when a new post goes up.

1. Avoiding latency with Fakes
2. Organising Test Suites
3. Selectively Running Tests

PHPUnit offers a number of ways to organise your test suite, but the docs are a little light on commentary, these are my thoughts.

Using the filesystem

The first and easiest way to get started is to use the filesystem. If you give PHPUnit a directory as an argument, it will scan that directory (recursively) for *Test.php files and then execute them. This means we can start to split our test suite up, just by putting files in different places.

For the main Childcare.co.uk app's test suite, I have two top level PHPUnit directories.

The first is tests/unit and this is for isolated, fast, unit tests. These tests tend not to interact with any external systems, quite often not interacting with other classes, functions or components at all. They take just a few milliseconds each to run and there are quite a lot of them.

The second directory is called tests/integration and this directory gets everything else. I try not to get too caught up in naming the types of tests I write, but depending on your way of thinking, this directory includes unit, functional, integration, integrated, system and acceptance tests. The tests in here exercise larger parts of the code and usually interact with third party systems like databases or HTTP APIs. In a lot of cases, I use fakes to speed up these tests, but they're still slow, taking a couple of minutes to run the entire directory.

Under these directories, I tend to follow the file structure of the production code as close as possible, though that quite often doesn't apply to the acceptance tests that I run out of tests/integration, which tend to be something like tests/integration/src/Childcare/<module>/NameOfFeature.php. If I were starting green field today, I might have a separate top level directory for some of those, but I don't lose sleep over it. Either way, having the tests with a decent hierachy allows for flexibility in running just the tests for a section/module/component as needed.

Failing fast

Given this simple split of faster tests and slower tests, I am able to run the faster tests first, followed by the slow tests. In theory, this should mean I'm getting the feedback I need quicker. To faciliate this, I use a simple Makefile as a task runner.

.PHONY: check

check: 
    vendor/bin/phpunit tests/unit
    vendor/bin/phpunit tests/integration

Using groups

Even given this separation, I still find I need some further organisation. Some of those tests in tests/integration are really slow. I have a bunch of code that interacts with the Facebook API and to make sure my code keeps in tune with the API responses, I really want to have some tests that actually hit the API. It turns out that creating test user accounts is quite slow and this slows my tests down significantly, to the point where I start to get annoyed when I'm running tests before a merge. Rather than creating another top level directory to house the dozen or so tests that create Facebook users, I decided to use PHPUnit's group feature to allow me to exclude these tests as I please.

/**
 * @test
 * @group facebook
 */
public function some_expensive_test() {}

Again, using our Makefile as a task runner, I make a recipe that runs the tests without those facebook tests.

.PHONY: check check-quick

check-quick: 
    vendor/bin/phpunit tests/unit
    vendor/bin/phpunit --stop-on-failure --exclude-group=facebook tests/integration

I have a few other groups excluded and I use this run regularly throughout the day, only running the full suite occasionally. Our continuos integration server always runs the full suite when I push code, so I can carry on developing carefree with regards to our Facebook integration. Because I use this for rapid feedback, I also use the --stop-on-failure switch. This can sometimes be annoying if you've broken code in several places, but more times than not the first failing test allows me to identify a smaller set of tests to run, make fixes and run again.

Using @small, @medium and @large

One alternative to separating faster and slower tests in to directories is to use the special @small, @medium and @large annotations. These are aliases for @group small etc, but also have special meaning if you install the phpunit/php-invoker package. With this package installed, if you run PHPUnit with --enforce-time-limit, PHPUnit will mark these tests as risky if they do not execute within a set of time thresholds.

Using test suites

Another way to organise the PHPUnit tests is to use test suites. This isn't something I currently utilise, but they are quite flexible and can mimic the behaviour I achieve using the Makefiles.

<phpunit bootstrap="src/autoload.php">
  <testsuites>
    <testsuite name="all">
      <directory>tests/unit</directory>
      <directory>tests/integration</directory>
    </testsuite>
  </testsuites>
</phpunit>

The first thing to note is that when using a test suite, PHPUnit will run the tests in the order specified. This means with the example above, we get that failing fast scenario as our faster tests/unit tests get run before the slow tests/integration tests.

You can also get fairly specific about the files you want to include or even exclude, so depending on the way you set your files and directories up, you can achieve something like I did above with the facebook group.

    <testsuite name="quick">
      <directory>tests/unit</directory>
      <directory>tests/integration</directory>
      <exclude>tests/integration/facebook</exclude>
    </testsuite>

I tend not to use test suites as I find the combination of directories and groups to be enough.

I plan to follow up on this post with a more detailed look at which tests to run at what time, drop your email in the box below to be notified when that one lands.

Faster Tests in PHP

1. Avoiding latency with Fakes
2. Organising Test Suites

There are a handful of ways to do this, it's quite common for people to reach for their favourite test double tool, and create mocks and stubs for database connections or SDKs, but I'm not going to talk about that. I use a lot of mocks, but I don't use them to keep my tests fast. We can use another type of test double to try and get our tests running faster: Fakes. A Fake is a simpler or more lightweight version of the real system or component.

All of the methods I'm going to mention make compromises regarding the thouroughness of your tests. We're going to be changing the way the system operates to be different from how it will ultimately operate in production, to make our tests quicker to run. In doing so, we will be sacrificing the surety of testing the system end to end as it should be run.

There are three reasons why you might use a Fake to replace a component or system:

• When the system/component is not available
• When it makes your tests easier to write or run
• When the system/component is slowing you down

We're going to concentrate on the slowing down part. These days we all have super fast computers and networds at our disposal and most of the I/O we run for the basic web apps is pretty quick, but as your test suite grows, even milliseconds for a database query or HTTP request that gets run for every test soon adds up.

Fake Objects

One of the easiest way to avoid latency is to replace an object that uses disks or the network with an object that offers the same API, but does things a little differently, hopefully faster. If you are using any modern PHP framework that provides testing support, you are probably using a few fakes like these already. If your framework doesn't provide testing support, maybe you should choose another framework.

use Aws\S3\S3Client;

class S3Storage implements Storage
{
    private $bucket;
    private $client;

    public function __construct(S3Client $client, $bucket)
    {
        $this->client = $client;
        $this->bucket = $bucket;
    }

    public function put($targetPath, $contents)
    {
        $this->client->putObject(array(
            'Bucket' => $this->bucket,
            'Key'    => $targetPath,
            'Body'   => $contents,
        ));
    }

    public function get($targetPath)
    {
        $result = $this->client->getObject(array(
            'Bucket' => $this->bucket,
            'Key'    => $targetPath,
        ));

        return (string) $result['Body'];
    }
}

Assuming we're going to need to use this in our tests, we might go ahead and create a test bucket on S3 to run all our tests against and for the first few tests, this seems like it's working great.

As we write a few more tests, we start to realise things are running a little slowly. Even worse, your internet connection becomes intermittent or drops out entirely. Sure we could refactor some code, ditch those integrated tests and write more isolated unit tests, avoiding the problem, but that's not always ideal and definitely isn't the only way to approach the problem.

In Memory Fake Objects

Writing a simple implementation of Storage that stores the data in memory avoids our network problems (as well as a bunch of CPU cycles in the S3 SDK), getting our tests nice and snappy again.

class ArrayStorage implements Storage
{
    private $data;

    public function put($targetPath, $contents)
    {
        $this->data[$targetPath] = $contents;
    }

    public function get($targetPath)
    {
        return $this->data[$targetPath];
    }
}

More persistent Fakes Objects

One disadvantage to holding data in memory is that it goes away! If we wanted the data to stick around after a test run (this can be helpful for debugging) or if we need to have access to the same data across processes, (maybe you're shelling out or hitting a real webserver), we'll need something with more persistence.

In the previous example, the real system is somewhere on the internet, so a local disk based implementation will still incur some latency, but will operate much faster and more reliably than API calls over the internet.

class DiskStorage implements Storage
{
    private $dir;

    public function __construct($dir)
    {
        $this->dir = $dir;
    }

    /*
     * No error checking for brevity
     */

    public function put($targetPath, $contents)
    {
        file_put_contents($this->dir."/".$targetPath, $contents);
    }

    public function get($targetPath)
    {
        return file_get_contents($this->dir."/".$targetPath);
    }
}

You'll quickly find Fakes like the one above become useful outside of tests. You might find you use them in your QA environments, or for your local setup for development or for demonstrations. It's also quite possible that Fakes developed for testing end up being good enough to ship as production alternatives to the real systems.

Self Initialising Fake Objects

Self initialising fakes are kind of like a stub/fake hybrid. They act like a stub, in that they return canned results, but they're more like a fake because they actually proxy to a true implementation, caching the calls indefinitely. Ruby's vcr is a popular library that does this at the HTTP level, intercepting calls to Net::HTTP and a bunch of other HTTP clients, replaying the results on subsequent calls. At any given time, you can choose to forego the recordings and make the actual underlying HTTP calls. There's a PHP port that hooks in to curl, but I'm yet to try it out. For the purposes of a demonstration, we'll write our own naive implementation using the decorator pattern to cache calls to an underlying object.

class VCRStorage implements Storage
{
    private $storage;
    private $libraryDir;

    public function __construct(Storage $storage, $libraryDir)
    {
        $this->storage = $storage;
        $this->libraryDir = $libraryDir;
    }

    public function put($targetPath, $contents)
    {
        return $this->call("put", $targetPath, $contents);
    }

    public function get($targetPath)
    {
        return $this->call("get", $targetPath);
    }

    private function call($method, ...$args)
    {
        $file = rtrim($this->libraryDir, "/")."/".md5($method."|".implode("|", $args));

        if (file_exists($file)) {
            return file_get_contents($file);
        }

        $contents = $this->storage->{$method}(...$args);
        file_put_contents($file, $contents);

        return $contents;
    }
}

This could get quite complicated depending on how easily the arguments and return types serialise to disk, but you get the idea. For something as simple as the example above, I much prefer this solution to intercepting calls via PHP's autoloader à la php-vcr..

Verified Fake Objects

Once our Fakes get a little more complicated, we might want to start writing tests for them to make sure they're behaving like the real thing, particularly if the system/component is likely to change regularly.

abstract class StorageTest extends \PHPUnit_Framework_TestCase
{
    abstract protected function getStorage(): Storage;

    /**
     * @test
     */
    public function what_goes_in_must_come_out()
    {
        $targetPath = "/some/path";
        $contents = "the contents";
        $storage = $this->getStorage();

        $storage->put($targetPath, $contents);

        $this->assertEquals($contents, $storage->get($targetPath));
    }
}

Subclassing this class and providing S3Storage, ArrayStorage, DiskStorage and VCRStorage instances in the getStorage method enables us to run the same tests against the different implementations. Adam Wathan has a nice screencast on this if you fancy watching how easy he makes it look in just 10 minutes.

Fake Systems with In Memory Backends

Sometimes, if something is particularly cross cutting and difficult to isolate, it can be hard to replace the client code or object with a fake. In this instance, it is sometimes possible to replace or change the whole system in the backend to speed up our tests.

If you need to shift large amounts of data in your databases, it might be worth keeping the client code the same, but switching out to a memory based backend.

MySQL comes with a memory storage engine, but I prefer create a ramdisk and configure MySQL to keep it's data there.

davem@wes:~$ mount  | grep ramdisk
tmpfs on /tmp/ramdisk type tmpfs (rw,nosuid,nodev,relatime,size=1048576k)

Completely Fake systems

Another quick win can be to replicate the whole third party system with a local equivalent.

Can you run your tests against SQLite rather than your full blown RDBMS? SQLite can be given a URL to tell it to store everything in memory.

$conn = \Doctrine\DBAL\DriverManager::getConnection([
    'url' => 'sqlite:///:memory:',
] , new \Doctrine\DBAL\Configuration());

Amazon provides a downloadable version of DynamoDB, which you can run locally for your tests avoiding the latency of making calls across internet. There are also a bunch of compatible implementations of other AWS services to be found on github, though your mileage may vary.

Wrapping up

If something is slowing your test runs down, make it quicker. If you can't make it quicker, replace it.

This is the first in a series of posts describing how you can go about making test runs faster, I'll be back to update this post as more posts in the series get published. If you'd like to be notified, pop your email address in the box below.

Faster tests in PHP

1. Avoiding latency with Fakes
2. Organising Test Suites

Brandon's article carries an overarching message and he states it in his rule of thumb:

Service locators don’t belong inside controllers. Period.

I couldn't disagree more. Controllers are the number one place I use a service locator and it seems I'm not the only one, as hinted at by Konstantin, also tweeting today:

I used to think like Brandon. I used to try and keep my controllers clean too, but like Konstantin, at some point a few years ago I realised I didn't need to anymore. I was getting better at putting clean clode in the proper places and as Konstantin puts it, I could make my controllers as dirty as sin.

In a typical MVC framework, controllers are for converting a HTTP request in to something suitable to send to your actual application code. The better you get at extracting business logic or even just complicated HTTP layer logic to places outside your controllers, the thinner and dumber your controllers get. Once they get dumb and thin, it makes sense to leverage the conveniences that a decent MVC framework provides for you.

If I'm making changes to a controller and it starts to get painful, there's a good chance I would look to extract some logic out of the controller, rather than extracting the conveniences afforded me by the framework. This usually manifests in some sort of Service Layer for things coming in, and as Presenters for things going out.

Use and abuse your chosen framework, that's what it's there for.

interface UserRepository { }

class Foo 
{
    /* ... */

    function bar()
    {
        $this->userRepository->delete(123);
    }
}

/** @test */
function should_delete_user_123()
{
    $userRepository = Mockery::mock(UserRepository::class); 
    $userRepository->shouldReceive("delete")->with(123)->once();

    $foo = new Foo($userRepository);

    $foo->bar();
}

Despite the delete method not existing on the UserRepository interface, this test will pass. For Mockery, this is by design. When I'm in my TDD loop, , I'm designing the UserRepository interface as I develop the Foo service, programming by wishful thinking. In order for Foo to do bar, I'd like to assume I have a UserRepository::delete method, but I'll care about adding that later. The problem manifests when we don't necessarily remember to deal with it later. We should notice the problem when we run some higher level test, but they don't always exist and even if they do, we might make the mistake of adding the delete method to the concrete UserRepository used in that higher level test, rather than the abstract. All the tests will pass, but things still won't be quite right.

The ruby community came up with a solution to this, rspec-fire, which was subsequently made obselete in favour of verifying-doubles in rspec core. This works by allowing you to program by wishful thinking, until you actually create the class, at which point rspec will check to make sure the method you are stubbing or expecting actually exists. Rspec will also check the arity of the stubs and expectations against the real thing.

This sounds great, but kind of annoys me. Just because a class or type exists, doesn't mean it's API is finalised. I would prefer to continue programming by wishful thinking within my TDD loop.

So how are we going to deal with this problem in PHP? For methods that don't exist at all, Prophecy will throw exceptions if you try to set up stubs or expectations.

There was 1 error:

1) ProphecyTest::prophecy_test
Prophecy\Exception\Doubler\MethodNotFoundException: Method `Double\UserRepository\P1::update()` is not defined

I don't believe PHPUnit mocks has any such feature at present, but it looks like something is in the works.

Mockery comes with a global configuration option to prevent stubbing and expecting methods that don't exist yet, that is disabled by default. You can turn it on in your test bootstrap. I like to turn it on for test runs outside the TDD loop. These test runs are more looking for regressions like on a CI server, rather than helping me develop behaviour in the system, so it seems sensible to verify we aren't doing anything stupid. I get to ignore warnings during my TDD loop, but get the safety net of having them verified at a later date.

Mockery::getConfiguration()->allowMockingNonExistentMethods(false);

As for the arity of existing methods, I think that's a problem best solved with the proper use of PHP's type hints.

For a long time, I thought the only way to have database records for my tests, was to manage one large sql dump that contained lots of records, all of which were required for one or more tests within the test suite, or to use DBUnit with a bunch of XML files. This changed when I came across factory_girl in some ruby test suites.

There are a bunch of similar packages for php out there (factory-muffin springs to mind), but I've always tended to roll my own, mostly as I've worked in gnarly legacy code bases and to be honest, it's not really the most complicated thing to do. I also keep them as simple as possible and avoid the production code, so data gets inserted directly in to the database, rather than using an ORM to store the data. I've previously written about Object Mothers and Test Data Builders, which I would use alongside the ORM if I wanted to do things that way.

If you're generating a lot of fake data, you might want to look in to using something like Faker, but I've found a handful of simple functions cater for most of my needs.

As a starting point, given a users table with an email and password field, I'll add the schema to the fixture.sql file as mentioned in the setting up a database fixture article, then create a class like this:

<?php

namespace tests\support;

use Doctrine\DBAL\Connection;

class UserFixtureFactory
{
    private $conn;

    public function __construct(Connection $conn)
    {
        $this->conn = $conn;
    }

    public function create()
    {
        $data = [
            'email' => "user@example.org",
            "password" => password_hash("password", PASSWORD_DEFAULT),
        ];

        $this->conn->insert('users', $data);
    }
}

This seems simple enough and is easy enough to get working in one of our tests.

    /** @test */
    public function fixture_factory_works()
    {
        $userFixtureFactory = new UserFixtureFactory($this->conn());
        $userFixtureFactory->create();

        $this->assertEquals(1, $this->conn()->fetchColumn("SELECT COUNT(*) FROM users"));
    }

Like any good users table, our email field has a unique constraint on it, so we need to work around that:

    /** @test */
    public function fixture_factory_works_with_lots_of_users()
    {
        $userFixtureFactory = new UserFixtureFactory($this->conn());

        for($i = 0; $i < 10; $i++) {
            $userFixtureFactory->create();
        }

        $this->assertEquals(10, $this->conn()->fetchColumn("SELECT COUNT(*) FROM users"));
    }

Adding a simple counter to the method will keep our email addresses unique:

    public function create()
    {
        static $counter = 0;
        $counter++;

        $data = [
            'email' => "user{$counter}@example.org",
            "password" => password_hash("password", PASSWORD_DEFAULT),
        ];

Yay green test! It's kinda slow though, slower than I expected. Probably the password hashing, let's fix that value with a literal.

        $data = [
            'email' => "user{$counter}@example.org",
            "password" => '$2y$10$Fx9LBid2/HV24SseoTp/sulorRnkykwN7D8HbUvsIgPtrDsxBqnUq', # password_hash("password", PASSWORD_DEFAULT),
        ];

The next thing I want is to allow the caller to override the default data:

    /** @test */
    public function fixture_factory_allows_overriding_defaults()
    {
        $userFixtureFactory = new UserFixtureFactory($this->conn());

        $userFixtureFactory->create(['email' => 'dave@example.org']);

        $this->assertEquals('dave@example.org', $this->conn()->fetchColumn("SELECT email FROM users"));
    }

    public function create(array $data = [])
    {
        static $counter = 0;
        $counter++;

        $data = array_merge([
            'email' => "user{$counter}@example.org",
            "password" => '$2y$10$Fx9LBid2/HV24SseoTp/sulorRnkykwN7D8HbUvsIgPtrDsxBqnUq', # password_hash("password", PASSWORD_DEFAULT),
        ], $data);

        $this->conn->insert('users', $data);
    }

Finally, I want the factory to return the data it used, so that the test code can make use of it as necessary:

    /** @test */
    public function fixture_factory_returns_data()
    {
        $userFixtureFactory = new UserFixtureFactory($this->conn());

        $id = $userFixtureFactory->create()['id'];

        $this->assertEquals($id, $this->conn()->fetchColumn("SELECT id FROM users"));
    }

    public function create(array $data = [])
    {
        static $counter = 0;
        $counter++;

        $data = array_merge([
            'email' => "user{$counter}@example.org",
            "password" => '$2y$10$Fx9LBid2/HV24SseoTp/sulorRnkykwN7D8HbUvsIgPtrDsxBqnUq', # password_hash("password", PASSWORD_DEFAULT),
        ], $data);

        $this->conn->insert('users', $data);

        $data['id'] = $this->conn->lastInsertId();

        return $data;
    }

That's pretty much it.

All the test examples so far have created the fixture factory when required, I don't recommend doing this and would probably create a helper method as a trait or on a base class.

    /** @test */
    public function fixture_factory_works()
    {
        $this->hasAUser();

        $this->assertEquals(1, $this->conn()->fetchColumn("SELECT COUNT(*) FROM users"));
    }

    public function hasAUser(array $data = [])
    {
        $userFixtureFactory = new UserFixtureFactory($this->conn());

        return $userFixtureFactory->create($data);
    }

Things tend to get more complicated than this, particularly when your factories need to be aware of other factories, in order to create and maintain relationships. I'll cover how I tackle that in another article, but needless to say, it's not much different from managing dependencies in your production code.

Happy testing!

It's worth noting that I tend to work on products, rather than software products. My software doesn't get distributed, I don't have to support multiple database vendors etc.

The stages of the test

When we think about how a database is involved in a test, we can think of the usual Arrange, Act, Assert steps, but with set up and tear down either side of them.

1. Set the database up to a known state for all tests
2. Arrange the database records for this specific test
3. Act, exercising the system under test
4. Assert against the state of the database
5. Tear the database back down to it's known state for the next test

This article is purely going to focus on the first and last stages, setting the database up and tearing it down. I have a test support class that manages this for me, it takes a Doctrine DBAL connection, which is already configured with access to a test database server.

<?php

namespace tests\support\Database;

use Doctrine\DBAL\Connection;

class Fixture
{
    private $conn;

    public function __construct(Connection $conn)
    {
        $this->conn = $conn;
    }

    public function setup() { }
}

Set up

Given we have a database server available to us, the set up stage is going to include getting our schema loaded. While it's tempting to run your database migrations to set the schema up, I feel like this is a waste of time. I rarely write database migrations and once they've hit all development, staging and production environments, they're not really useful to me any more and don't need to be tested. I like to keep a copy of the current schema alongside the tests in version control, tests/support/Database/fixture.sql. I then import this fixture during the set up stage.

    public function setup()
    {
        $this->load(); 
    }

    private function load()
    {
        $file = $file ?: __DIR__ . '/fixture.sql';
        $params = $this->conn->getParams();
        system("MYSQL_PWD={$params['password']} mysql -h{$params['host']} -P {$params['port']} -u{$params['user']} {$params['dbname']} < $file");
    }

I shell out to the mysql command line client as I've found it to be a shade faster, but your mileage will vary and you could try going through the DBAL instance as well.

In order to keep the fixture up to date when creating migrations, I have a short script that will load the fixture, run the migrations and then dump the database back in to that fixture file, ready to be committed to version control alongside the migration.

    public function update()
    {
        $this->load();

        # It's not quite like this, but you get the idea
        system("APPLICATION_ENV=testing php atstisan migrate");

        $this->dump();
    }

    public function dump()
    {
        $file = $file ?: __DIR__ . '/fixture.sql';
        $params = $this->conn->getParams();

        system("MYSQL_PWD={$params['password']} mysqldump --set-gtid-purged=OFF -h{$params['host']} -P {$params['port']} -u{$params['user']} --opt {$params['dbname']} > $file");
    }

If I have the need for some standard data to be available to every test, I don't bother writing code to seed the database at the start of every test. I load the fixture, do whatever it takes to get the data in there, be it scripting or by hand, and then dump the fixture again, ready to be stored in version control. This keeps my start up time quick and deterministic, what's in the fixture.sql file is what every test starts with.

Tear down

You could tear the database down by deleting the entire fixture, allowing the setup method to reload the entire fixture again, but this can be very slow. To speed things up, we can load the fixture once at the start of the test suite, and then have every test tear the database back down to this initial state.

    public function setup()
    {
        if ($this->fixtureLoaded) {
            return;
        }

        $this->load(); 
        $this->fixtureLoaded = true;
    }

A popular way of tearing a database down is to run each test in a transaction and then roll back the transaction after the test has completed. This works quite well and is really fast, but has a couple of drawbacks.

The first is that all database operations need to run through that same connection that holds the transaction. This isn't so easy when you're wanting to do headless browser tests with tools like selenium, or if you need to do any out of bounds processing like queue workers. To get around this, you can use a truncation strategy. The tear down stage truncates all the necessary tables to return the database to known state (except for auto increment counts, but I can live with that).

    public function tearDown()
    {
        $this->truncate();
    }

    public function truncate()
    {
        foreach ($this->tablesToTruncate as $table) {
            $this->conn->delete($table, array(1 => 1));
        }
    }

Again, here I use the equivalent of DELETE FROM $table WHERE 1=1, which I've found to be a hair quicker than TRUNCATE $table, but you should benchmark for yourself. I manually keep a list of tables to truncate, but you could easily make a list of tables not to truncate, or if you don't have any data in the fixture.sql file, truncate all tables.

The second drawback to using transactions and the most important for me, is that the state of the database is always torn back down at the end of the test. If you're trying to debug a particular problem, it can be very useful to be able to examine the database after the test has finished. Surely this is the same using a truncation strategy, I hear you say. You would be right, but with truncation, we can move the truncation to the setup method, not do anything on tear down and don't call me Shirley.

    public function setup()
    {
        if ($this->fixtureLoaded) {
            $this->truncate();

            return;
        }

        $this->load(); 
        $this->fixtureLoaded = true;
    }

    public function tearDown() {}

This way, each class gets a clean state, but after a test has run, you get the chance to inspect the state of the database. It's pretty fast too, at least fast enough for me.

Usage

I use this code in a setup method of a base class in my day job, but you can also do the same thing with a trait. This example uses a singleton to get hold of a Fixture instance, but I'll write about other ways of doing that in another article.

<?php

namespace tests\support;

use tests\support\Database\Fixture;

trait UsesDatabase
{
    /**
     * @before
     */
    public function setupDatabase()
    {
        Fixture::instance()->setup();
    }
}

<?php

namespace tests\integration;

use tests\support\UsesDatabase;

class SomeTest extends \PHPUnit_Framework_TestCase
{
    use UsesDatabase;

    // ...
}

Happy testing!