Omnibus Cache Exploitation

I've been playing with Omnibus a bit lately, in hopes of making our product installable in locations where internet access is not an option. We have a decent list of dependencies that we install at deploy time, and we need some consistent way to get those items put in place when public repos aren't available.

If you've never seen Omnibus before, it's pretty great (though there is a bit of a learning curve).

Expectedly, our dependency list gets hairy when we start pulling in the dependencies of our dependencies. That's not so bad, by itself. What sucks is that any change to your project definition (omnibus-my-project/config/projects/my-project.rb) - even whitespace - causes Omnibus to invoke its HopelesslyPessimistic mechanism which in turn invalidates your software cache faster than you can say, "but it's just a pip install!".

This is done for perfectly rational reasons, sure, but it's annoying when you're working in a container whose only purpose in life is to run Omnibus builds, and your build times start to creep into the tens of minutes.

Fortunately you can route around this pessimism while you're getting your dependency list ironed out by adding your dependencies to a dependency. To elaborate...

You normally add your run-time dependencies to your project definition (my-project.rb), but you can also add those dependencies to another software definition that you depend on in omnibus-my-project/config/software.

So for example, instead of:

You'd have a more minimal definition:

Then your pycrypto definition might look something like this:

Now you can add dependencies to the bottom of pycrypto.rb and take full advantage of the Omnibus software cache! Saves a lifetime of waiting on builds to complete.

NOTE: You really shouldn't leave your project like this, it'd be terribly irresponsible of you, and then you'll say that I told you to do it, and I'll catch all kinds of grief for it. So don't do that.

When you're done adding all your dependencies, you can simply shovel all the dependency "<name>" lines out of (in this example) pycrypto.rb back into my-project.rb, and kick off one more (very long) build to get a sane final package.

Happy packaging!