Taming the autotest Beast with FSEvents
autotest is a great tool and all, but it is simply resource intensive. Due to autotest’s implementation, it eats up CPU resources, not because the tests are always running (only after you modify your file), but because autotest continually polls each file in your directory, and sub directories, and checks to see if it has been modified.
This continual polling isn’t good for CPU resources. Now I may have some spare cycles to let autotest do its thang, but it also doesn’t sound too healthy for my hard drive.
Note: This is Mac OS X 10.5 Leopard specific.
Along Comes a Leopard
Whilst reading the Arstechnica review on Leopard, I came upon the section on the File System Events (FSEvents) that was introduced in 10.4 actually (for Spotlight), but used once again for Time Machine. In Mac OS X 10.5 Leopard, the API was opened up for the public to consume.
File System Events (FSEvents)
In its simplest level, your application will notify FSEvents, that it wants to listen to a particular directory, and when that directory (or its sub directories) are modified, FSEvents basically triggers a callback in your application. This allows you to hook to, listen to file system changes, and react to accordingly.
This is exactly what I needed to calm the fury of the autotest beast.
Install RubyCocoa
Update: It looks like the code works on the stock Ruby and RubyCocoa out of the box actually. You don’t need to install the latest version.
First of all, you need to install RubyCocoa, as this provides us with the bindings required to communicate with FSEvents. As I installed Ruby via MacPorts, I opted to do a source install (the MacPorts at present, is one version behind). I did run into trouble, encountering this error “file is not of required architecture“.
If you ever needed a reason to actually use RubyCocoa, let this be your reason!
Taming the beast
Once installed, dump this little gem into your ~/.autotest file:
For an easy to copy dump, click “view plain“.
Now when you run autotest, you’ll run into something like this:
/Users/aizat/.autotest:4: warning: method redefined; discarding old run
Don’t worry about it, and feel free to ignore it.
With this code, you can also tame the beast. There you have it, a much saner autotest, only for Mac OS X 10.5 Leopard.
Laurent Sansonetti said,
November 28, 2007 @ 2:59 am
Nice job!
BTW, why did you install RubyCocoa in Leopard? It’s already pre-installed. Did you need something in the latest version?
Mark Pauley said,
November 28, 2007 @ 3:54 am
Woah! I was just about to ask about how to do callbacks from rubycocoa!
That proc do trick is SO SLICK.
Carl Porth said,
November 28, 2007 @ 4:17 am
Very nice.
Works just fine on 10.5 with stock ruby install and latest zentest gem.
aizatto said,
November 28, 2007 @ 6:57 am
@Laurent, ah good point…don’t know, just felt like installing the latest version
At least from the comments, it works on the stock ruby and rubycocoa.
Josh Nichols said,
November 29, 2007 @ 9:22 am
One thing I’m noticing is that the signal handler isn’t being set. This means if you hit control-c, it really gets killed, rather than re-running all tests, or really getting killed when hitting it a second time.
Carl Porth said,
November 29, 2007 @ 11:31 am
I’m having the same issue as Josh.
Ryan Davis said,
November 30, 2007 @ 5:38 am
Sorry… But I have to call BS on this one. I run autotest ALL DAY on my laptop on battery and it is NEVER resource intensive. Your OS provides disk cache for a reason.
Tammer Saleh said,
November 30, 2007 @ 7:31 am
Ryan - I love autotest, but before this post I’d noticed that it takes roughly 13% cpu on my dual core macbook (same on my iMac). Not a big deal, but not something to ignore. Anyways, it doesn’t really make sense to use polling if there’s already a notification system in place.
Ezwan : I do like this idea, but I found that the failures seem to get constantly re-run. If nobody else has seen that, then it must be an issue with my installation.
aizatto said,
December 1, 2007 @ 1:01 pm
I’ll have to look into trapping the interrupt, as I’m still new to the available Cocoa/CoreFoundation apis.
@Tammer - I haven’t encountered that actually, though I’ll try to reproduce it.
@Ryan - Now I may not be resource starved, and sure I may have some spare resources to allocate to autotest, but when cost of implementing an algorithm to efficiently use a computer’s resources is really low, why hesitate to implement it?
When running autotest, I find out it eats about 15-25% of cpu utilization, which is completely unnecessary now.
You also say you constantly run on battery, wouldn’t it also be better to use anything that could potential reduce the power drain?
Ryan Davis said,
December 1, 2007 @ 6:57 pm
On my old G4 12″ powerbook it used MAYBE 5%.
On my 13″ macbook (core duo) it uses MAYBE 2% (currently says 1.3%)
If it is using more on equivalent hardware and a modern OS, then you’re doing something wrong or something is broken on your machine.
@aizatto: why hesitate using this? because your “efficient algorithm” is 25% the size of all of autotest (ie, not the simplest thing that works), isn’t nearly as maintainable, and isn’t cross platform. There are other users out there that don’t use OSX. I use autotest on 3 different platforms semi-regularly.
reduce power drain? what power drain?
Ryan Davis said,
December 1, 2007 @ 6:59 pm
I should also add… the sleep time IS parameterized… a 1 line addition to your .autotest vs 97 lines of not-cross-platfrom ruby? You decide.
P.S. alias the method and you won’t get that warning.
aizatto said,
December 1, 2007 @ 9:47 pm
@Ryan actually I opted to re implement find_files_in_paths and run_tests_in_paths just to specifically look in the directories that have been updated, rather than scan the whole current directory again. Even so, most of the code from the two functions is relatively the same, with a one or two line change. I previously used the original code but thought I would optimize it a bit.
Is there anything wrong with targeting a certain platform? If you are working on OS X, then one should be happy that because you are on this platform you can use this feature, whilst working on the other ones, you don’t really have much choice. Just pros and cons of the platforms one uses. Does it take away from the functionality of autotest? No.
Yes, you can put it to sleep for a while, but I still find it unnecessary to traverse the directory tree again if you don’t really need to.
Brandon Keepers said,
December 6, 2007 @ 9:21 am
Sorry Ryan…but I have to call BS too. I use autotest all the time and notice a significant decrease (25%-50%) in battery life when it’s running. It’s a great tool and I don’t think I could code without it, but it does use a fair amount of resources.
aizatto, thanks for sharing the code. Now I can use autotest on flights.
Ryan Davis said,
December 16, 2007 @ 12:49 pm
I just had a thought…
You’re all doing rails development with a stuffed-to-the-gills vendor/ aren’t you? Ever think of excluding the vendor directory?
Next version of autotest will make doing that a lot cleaner… coming soon.
Yossef said,
December 18, 2007 @ 4:24 am
To possibly assuage any problems with this not being cross-platform, I personally put this in my .autotest wrapped in a begin block with
rescue LoadError => e
puts “Failed to use FSEvents instead of polling files”
end
Only recently and only on one project have I noticed the problem Tammer’s talking about. I’m not sure what it is and haven’t bothered to investigate yet.
Matt Schick said,
January 5, 2008 @ 11:50 am
Tammer and Yossef, I’ve been having the same problem of failed tests re-running over and over again. After digging around in the code for awhile I figured out what was causing the problem for me:
When a test fails it’s file and failure message are placed in @files_to_test. I believe this is how autotest remembers what tests failed before so it can just rerun that failed test if any changes occurred that are nonspecific to any tests. Now a few of my tests create files while running and then delete the files when the test is over. The deletion of these files triggers an FSEvent. run_tests_in_paths is called from the callback and no files are found. So ideally it should stop right here. But remember that our friend @files_to_test still has the last failed test in it so run_tests_in_paths keeps running and reruns our test again, and again, and again….
To fix this problem I made a few changes that allow FSEvent to ignore certain files based on a project by project specification. First I hooked into initialize and added a class level variable to stash our ignore pattern into (~/.autotest):
Autotest.add_hook :initialize do |i|
@@fsevent_ignore_pattern = nil
end
I modified the call back like so (~/.autotest):
callback = proc do |stream, ctx, numEvents, paths, marks, eventIDs|
paths.regard_as(’*')
rpaths = []
numEvents.times { |i| rpaths < 0
end
What this does is converts the file paths taken from the FSEvent, converts them to relative paths (i.e. ./spec/models/blog_spec.rb) and then filters them through @@fsevent_ignore_pattern. I setup the setting of @@fsevent_ignore_pattern as a hook so it can be specified on a project by project basis. So in my project directory autotest file (rails_app/.autotest):
Autotest.add_hook :set_fsevent_ignore_pattern do |i|
@@fsevent_ignore_pattern = /^\.\/(?:spec\/test_data|log\/)/
end
I should also mention one last thing that I did: Since the paths from FSEvents are now converted to relative @exceptions now works as it should (before the paths were coming in absolute and the regexp was failing all the time). In find_files_in_paths I swapped out the line:
filename = f[current_dir..-1]
for:
filename = f.sub(/^\.\//, ”)
So I realize that this is a huge amount of reading, but I hope someone finds it helpful. It’s a basically a hack on top of a hack. I’m going to try and write a plugin for autotest that handles this much cleaner in the future.
and thanks to Ryan for the tip on running autotest in verbose to get the file changes (apologies to Ryan: I said I had my .autotest disabled, but I guess I was wrong, however I also said it was most likely my fault
Scott Nedderman said,
January 6, 2008 @ 9:38 am
Ryan is correct that this issue is directly related to having a stuffed vendor directory (mine has 3500 files, not rails). Autotest was using 25-30% of my cpu when running. However, the issue was complicated by also using the rspec_on_rails plugin which prevents excluding other directories because of the way it overrides AutoTest:initialize (see my example).
Here’s the solution: http://blog.netphase.com/2008/01/05/autotest-cpu-fix/
Matt Schick said,
January 9, 2008 @ 12:58 am
Bah! Ignore what I wrote, autotest loads either the project .autotest or your home directory .autotest.
Woody Peterson said,
April 11, 2008 @ 7:36 am
With the newer versions of rspec/zentest (i’m using 1.1.3/3.9.2, respectively) it looks like you need to change hook :run to hook :initialize on line 3.
Luke Kanies said,
May 26, 2008 @ 6:29 am
Just thought I’d throw a belated non-rails perspective in here.
I’m the core developer for Puppet, and I’ve currently got 172 spec files, with a total of about 23k lines of code (I’ve still got a lot of legacy test/unit code).
According to two important but qualitative tests, autotest was taking a lot of my machine’s energy. First, when it was running, the fan *never* stopped, and second, the MenuMeters graph in my menubar was always at around 25% cpu.
I’d recently taken to hitting ^z when not actively programming, since it was making my computer so much hotter, louder, and generally more annoying.
I’ve switched in this code and now my machine once again is behaving like I’m not playing a video all the time.
So, Ryan: I’d say autotest in its current form works fine for smaller projects or maybe if you’re interested in waiting longer for tests to trigger, but it’s pretty clear that events are a better way to handle this than polling, in the general case but especially for larger projects. Most platforms support some kind of polling interface — Linux has had FAM-like daemons for years, and now OS X has a good one too. I think this bit of autotest should be abstracted out so it can be made to support whatever events system exists locally, with a fall-back to using polling when necessary.