updates @ m.blog

Ruby & Bundler: They Took Our Jobs!

Since version 1.4.0, Ruby’s Bundler has allowed you to specify a number of --jobs to run concurrently, speeding up gem dependency installation. But what’s the right number of jobs to set?

The prevailing wisdom comes from these benchmarks by Jeff Dickey, from a year ago when Bundler 1.4.0 was in prerelease. His conclusion was to use N-1 jobs where N is the number of cores on your development machine.

However, since version 1.5.3, Bundler now actually baked part of this logic in and automatically uses N-1 cores, when N is the number of jobs requested with the --jobs= argument.2 So presumably you would want to use your actual number of cores since that change…

But I was still curious – did this mean physical cores, or logical cores? My main dev machine is a quad-core Core i7 3.4GHz iMac, which has hyper threading enabled by default. So as far as most applications are concerned, it has 8 logical cores, not 4. The performance implications of hyperthreading are still a bit weird (plus you should never trust any benchmarks you haven’t run on your own environment) so I decided to do some benchmarking of my own.

Benchmarks

All tests done with ruby 2.1.3p242, Bundler version 1.7.3. For me I was testing installing the dev dependencies for the lolcommits gem. 35/35Mbps fiber internet connection.

Core i7 3.4GHz iMac (quad core, with hyperthreading), 12GB RAM, SSD hard drive. 35/35Mbps fiber internet connection.

| jobs | cpu  | time  |
|:-----|-----:|------:|
| 1    |  53% |  47.3 |
| 2    |  54% |  46.7 |
| 3    |  86% |  29.9 |
| 4    |  89% |  28.9 |
| 5    |  99% |  26.7 |
| 6    | 105% |  25.3 |
| 7    | 117% |  23.4 |
| 8    | 122% |  23.2 |
| 9    | 117% |  24.4 |
| 10   | 125% |  22.9 |
| 12   | 125% |  22.8 |
| 20   | 134% |  23.0 |
| 40   | 149% |  22.5 |

For comparison here is a much lower end system, a Core 2 Duo 2.1GHz Macbook Air (dual core), 4GB RAM, SSD hard drive.

| jobs | cpu  | time  |
|:-----|-----:|------:|
| 1    |  64% |  79.9 |
| 2    |  68% |  75.6 |
| 3    |  98% |  53.0 |
| 4    | 102% |  50.8 |
| 5    | 106% |  49.4 |
| 6    | 104% |  50.6 |
| 7    | 109% |  48.6 |
| 8    | 123% |  46.7 |
| 9    | 108% |  49.2 |
| 10   | 110% |  48.3 |
| 12   | 112% |  47.6 |
| 20   | 113% |  48.7 |
| 40   | 116% |  49.3 |

Discussion

It appears things have changed in the world of bundler 1.7.3. Some conclusions I draw from my results:

  1. Bundler jobs are IO bound, not CPU bound. Note that even when --jobs=40 on my iMac, CPU utilization never reached over 150% – if we were efficiently using all the CPUs then utilization would have approached 400%).
  2. There appears to be no real speed penalty to going too high with the number of jobs.
  3. N-1 cores does not appear to be the ideal way to set number of jobs in bundler. For my MacBook Air, that would result in N=1 jobs, when it gained proximate speed increases up to 4 jobs (despite only being dual-core).

Note that more benchmarks should be done to confirm these results. I invite others to please try to replicate on their hardware. In particular, I’d like to see a range of different gem dependencies tested, and for someone without a SSD drive to test so we can see the impact of lower IO speed.

In the meantime though, for my hardware at least, my recommendation is actually to set the number of jobs to N*2, where N is the number of physical CPUs available to the workstation. On my iMac, this would equal 8, whereas on my MacBook Air, it would equal 4. This formula seems to get all the benefits of the added concurrency without going too crazy.

TL;DR more jobs seem to be better. Don’t let them take your jobs.

Appendix

Determining number of CPU cores programatically

This is especially pertinent because the way most scripts that set this automatically work is to use the number of logical CPUs, rather than physical CPU cores. The common command on Darwin that scripts such as Boxen use is systl hw.ncpu. Thankfully, there actually many variables we can use to determine actual CPU count on a Mac:

$ sysctl hw | grep cpu | grep -v frequency | grep -v type | grep -v family
hw.ncpu: 8
hw.activecpu: 8
hw.physicalcpu: 4
hw.physicalcpu_max: 4
hw.logicalcpu: 8
hw.logicalcpu_max: 8
hw.cpu64bit_capable: 1
hw.ncpu = 8
hw.availcpu = 8

The _max variants (e.g. logicalcpu vs. logicalcpu_max) are there to support hardware that step down the number of active cores when in power save mode (e.g. a portable on battery life). A delta between those would be visible in that situation.

Run your own tests

For my purposes, I tested via the following command in my project directory:

for j ({1..8}); do rm -rf .bundle; time bundle install --quiet --jobs=$j; done

Note that I have bundler configured to install in the local directory, you will want to modify the remove command to reflect your own, which you can determine from bundle config path.

The Year in (Mostly Stupid) Side Projects

2012 was a very busy year for me, as things really ramped up at Bitly, getting settled in on a different coast of the country, etc. However, I managed to find some time in the margins for side projects, and I thought it would be useful to reflect on the things I built and what I learned.

Like many geeks, side projects are often more about having a task (it doesn’t really matter what) to explore learning a new technology, rather than building something out of a fervent desire it must exist in the world. Thus, my side projects tend to veer towards single-serving jokes, but hopefully there is something in the underlying tech that could be useful to others.

Note: While 2012 is the first time I feel like my code is approaching the point where it might actually be useful to other people to learn from, I’m by no means an actual software developer – so I’m probably a bad source for “the right way to do things” and possibly a better source for “what it looks like when a part-time hacker screws around making something.”

Anyhow, enjoy?! On to the projects (in chronological order)…

What Comes Next

IMG_9279

Well, looks like this just went public, so might as well announce it here as well.

After completing my tour of duty at Flickr, I’ll be joining Bitly as Head of Product this spring. Bitly has a fantastic small team of really smart folks, and I’m excited to go help build some new and exciting things there. Plus, I knew that for my next thing I needed to go somewhere with a wicked cool animal mascot (and the only thing that could beat a pufferfish in the department might be Octocat).

And yes, this means I’ll be moving back to NYC. On a personal note, seven years ago I swore I would soon return to New York when “temporarily” moving to the SF Bay Area. So it’s advantageous that my significant other decided that moving to the East Coast would be the best way for us to get a fresh start on a new life together. We’re currently alternating between looking forward to our coming adventure and being somewhat terrified! Thanks everyone for your kind words in the past 48 hours, it’s meant a lot to me. Onward!

On Leaving Flickr

So uh, yeah… I know these sorts of personal departure posts are a bit trite, and I certainly have mixed feelings about posting one. However, as Yahoo-bashing is apparently quite de rigueur in the tech press these days, I wanted to make sure my reasons for leaving Flickr were publicly posted in my own words, so that my departure would not continue to be used as ammunition for other people’s agendas, or incite any speculation about the well-being of Flickr.

The change

After nearly five years of awesome, I have decided to step away from my current role of Head of Product of Flickr, and resign from Yahoo! Inc.

I’ve had the privilege of developing some amazing things at Flickr. I’ve worked with many iterations of an amazing team. I managed to accidentally invent a new holiday celebrated in 28 countries. I’ve had so many unbelievable opportunities while working on Flickr, it’s impossible to even begin to enumerate them.

yah Flickr Mobile!

Very early in my career at Flickr, I worked on the first version of our mobile website. Shortly after launch, while I was travelling on a business trip in the UK, I was waiting for my delayed flight home in Heathrow Airport, bored and lonely. I pulled up m.flickr.com on my Treo 650 and within seconds was looking at a photo my dogsitter had taken of my pooch minutes before, and was able to engage in a conversation in the comments with her. Nowadays, these sort of interactions seem old hat, but at the time I distinctly remember thinking: I helped build something that will change people’s lives for the better. This sentiment was important for me, and for the remainder of my career at Flickr, one of my fundamental underlying motivations in product development has been how will what we’re building enrich the lives of our members and change the way they communicate for the better? I’m proud to say that not only did I have the opportunity to build many things at Flickr that embrace this desire, but also that this sentiment permeates the entire team and I believe it will inform everything that they will continue to build in the future.

Nonetheless, the time has come for me to move on.

The good news

To steal and paraphrase from J. Allard’s excellent goodbye letter — “my life has been 95% Flickr and 5% other for quite some time, and I know the first step is to flip that ratio around.”

Against all odds (and in the oddest of places), I have met someone who having a future with is more important to me than any job could ever be. Leaving Flickr behind will allow me to focus more passionately on investing in a future with her.

"Must be able to tolerate office tomfoolery"

The future of Flickr

Flickr now has a newly focused and developed product strategy to win in the photo sharing space. It’s there. It’s coherent. It’s consistent. I strongly believe it will work if executed on effectively. While I obviously can’t share metrics or specifics, in the things we’d begun rolling out in the past year, we’ve begun to see fantastic returns on our bets, in just the way we had been looking for.

Flickr is a small, scrappy team, working on challenges way larger than itself, and as a result it requires intense focus and effort. That sort of focus and effort is actually a good thing for any company, and the Flickr team has been able to step up to the challenge. As the photo sharing landscape evolves and becomes more complex, Flickr will require additional investment from Yahoo. I’ve done my best to advance this fight, and know those who come after me will continue to push that forward as well.

The short version? While I believe highly in the future of Flickr, my time to be leading the charge has passed. I need to pass on the torch to the next generation of Flickreenos, all of whom have a clear mission, sense of purpose, and the drive and talent to get it done. I know that they are up for the task.

Then We Attack Here

What I’m most proud of

Hiring and mentoring some of the most amazingly smart and driven up-and-coming talent in the industry — I hope Flickr was (and will continue to be) as rewarding an experience for them as it has been for me.

Working on a team so committed to “doing the right thing” for our members. Whatever the outside perspective has been at time, and while public detractors (often pushing their own agendas) are certainly easy to find, I’ve never in my life met a group of people so strongly committed to this belief.

Finally, I’m proud to finally have the courage to step away, which anyone who has worked for many years on something they love knows, is an exceptionally difficult thing to do.

Interwebs

So what next?

I’m not going to announce my next thing here just yet, as I don’t want to muddle a post that’s meant to be about Flickr. Suffice to say, it is decided upon and exists, and I’ll share it sometime soon. In the meantime, I know that I’m walking away from one of the most important chapters of life, and hopefully into another.

mroth, thursday

Farewell.

Palm Pre Mojo SDK Experiments

As a new Pre owner, I was curious to learn to code for the Palm WebOS SDK (Mojo). Whenever I try to learn a new technology, I try to build a functional test project. So… what to build for WebOS? Since the Pre’s web browser doesn’t currently support the W3C Geolocation spec used by Flickr Nearby, I figured a good “Hello World” might be an application to get the GPS coordinates from the Pre, and then load the appropriate Flickr page for that location.

location services call

CAVEAT: This is a trivial hack, which was never designed for a production release. I’m writing up this blog post on how it works because (much to my surprise!) some screenshots of my little hack generated a bit of interest. So, I’ll provide some background, which hopefully will be either helpful or interesting to someone. Also note, that while I’m a bit of a minor nerd, I’m not a software engineer by any stretch of the imagination (I primarily do business / strategy / management-y type stuff), AND this was pretty much my first foray in Javascript, so I’m sure there are horrible things in my code that will make real developers cringe. Sorry!

To begin, follow one of the existing tutorials out there for setting up your environment, learning the basic concepts, generating the project and first scene, etc. etc. I’m just going to show here what I did from when I actually started coding.

Tumbleweeds

So, I haven’t been actively blogging here in a long, long time…

However, I have been posting various things that interest me fairly regularly on my tumblr page, which you very well may want to read if you are already following this blog.

Nokia HSDPA Support in Mac OS X 10.5

My Nokia N95-3 has a deliciously fast HSDPA connection, and it is automatically recognized in Mac OS X 10.5 for syncing and tethering. Only one slight problem:

Using the developer tools and iSync Plugin Maker.app, I was able to update Barkman’s script to work in Leopard. My preliminary tests are getting ~750kbps in my apartment with so-so signal reception from AT&T. Woo! (Now if only Nokia would make a QWERTY version, so I didn’t have to carry around both this and a Blackberry…)

You can grab my updated modem script here: NokiaHSDPA.zip

Just dump it in /Library/Modem Scripts and then follow one of the many existing setup guides for either bluetooth or USB tethering, making sure to select the new HSDPA device instead of the 3G device from the Nokia category.

Why I Didn’t Respond to Your Instant Message

  • I didn’t actually get it (unlike email, most IM systems have no delivery confirmation, and client/server sync issues are unfortunately quite common).
  • You have a history of sending me and my team annoying queries that are a waste of our time, so I’m going to pretend the above happened and hope you figure out the “solution” on your own before messaging me a second time.
  • You sent the message hours ago, when I wasn’t even online, and it was just now delivered from offline mode. If I wasn’t online to begin with, I obviously could not respond to an “instant” message, so why didn’t you just send an email?
  • I’m at work, and you sent me a hyperlink that is obviously frivolous (anything with youtube.com in the URL, for example). I typically don’t have time to look at these, let alone reply with my “thoughts” on them.

Now that the cranky rant is over, I’ll try to actually be helpful. Here are some handy-dandy tips on Giving Good IM in a Workplace Environment:

  1. Be specific: Context is valuable. Bad: “hey did u see http://ambiguousurl.com/72d7a8f?” Good: “Hey, have you seen Bob Blowhard’s latest blog post on our product? (http://ambiguousurl.com/72d7a8f). If not, you should check it out, it has some good commentary on the XYZ feature!”
  2. Get to the point: If you are IM’ing because have something to ask, just ask. Don’t make “small talk” first, I’m going to be spending he entire time wondering what you’re working up to anyhow.
  3. Exercise good timing: If my status message says “In a meeting,” then a message like “What time are you getting out?” may be appropriate. “What are your detailed thoughts on the implications of yesterday’s reorg on our marketplace strategy for Q3?” is not.
  4. No reply necessary: Very uncommon, but people will love you if you do this. If you’re just passing on a piece of information and don’t need confirmation, let someone know they don’t need to reply, and save them a few seconds.
  5. Quick queries only: My general guideline is that anything that will require someone to think for more than two seconds in order to answer is probably not appropriate for IM.

People have strong existing workflows to handling incoming email. Filters, folders, flags, et al. allow the recipient to delegate the incoming flow of information and respond to it in a way that works best for them. Instant messages “jump the queue” people have set up, so while it can be a powerful medium for lightweight communication, be considerate of helping keep people from becoming overloaded.

Do you have any other tips for dealing with IMs? Post them in the comments, perhaps I’ll compile a list of reader contributions.