Jobs that are cancelled while queued don't get cancelled

[Sarah Hodne]

I'm seeing a log like this:

time="2017-03-08T20:01:23Z" level=info msg="received amqp delivery" job=[redacted] pid=11807 processor=d96a4562-264c-47a8-90d9-9956279393e6 
time="2017-03-08T20:01:23Z" level=info msg="starting job" job=[redacted] pid=11807 processor=d96a4562-264c-47a8-90d9-9956279393e6 repository="[redacted]" 
time="2017-03-08T20:01:23Z" level=info msg="using build script generator to generate script" job=[redacted] pid=11807 processor=d96a4562-264c-47a8-90d9-9956279393e6 repository="[redacted]" 
time="2017-03-08T20:01:23Z" level=info msg="generated script" job=[redacted] pid=11807 processor=d96a4562-264c-47a8-90d9-9956279393e6 repository="[redacted]" 
time="2017-03-08T20:01:23Z" level=info msg="starting instance" job=[redacted] pid=11807 processor=d96a4562-264c-47a8-90d9-9956279393e6 repository="[redacted]" 
time="2017-03-08T20:01:23Z" level=info msg="cancelling job" command="cancel_job" component=canceller job=[redacted] pid=11807 
time="2017-03-08T20:01:23Z" level=warning msg="job already cancelled" command="cancel_job" component=canceller job=[redacted] pid=11807 
time="2017-03-08T20:01:24Z" level=info msg="selected image name" dist=precise group=stable image_name=travis-ci-macos10.12-xcode8.2-1481567376 job=[redacted] language=objective-c os=osx osx_image=xcode8.2 pid=11807 processor=d96a4562-264c-47a8-90d9-9956279393e6 repository="[redacted]" 
time="2017-03-08T20:02:23Z" level=warning msg="job already cancelled" command="cancel_job" component=canceller job=[redacted] pid=11807 
time="2017-03-08T20:02:23Z" level=warning msg="job already cancelled" command="cancel_job" component=canceller job=[redacted] pid=11807 
time="2017-03-08T20:02:24Z" level=warning msg="job already cancelled" command="cancel_job" component=canceller job=[redacted] pid=11807 

It looks like hub is trying to tell us that the job should be cancelled, but for some reason we think it's already cancelled and never actually cancel it.