Screwed by Design

A little over a year ago, some of my fellow mozillians and I decided we wanted to have a place to host websites and e-mail, stick off-site backups, and perform other disk or CPU intensive things not typically permitted in a shared hosting environment. After  exploring a few options, we decided to split a dedicated server and eventually settled on Voxel as our host.

Of course, with just a few low-traffic websites, backups, and an SSL cert survey or two, we typically found ourselves far under our monthly included bandwidth.  Not wanting to leave our investment underutilized, we took advantage of our server package’s included access to the VoxCAST CDN and put it to work as a Mozilla mirror.

Mozilla’s Bouncer mirror management system distributes traffic proportionally, based on assigned weights. Since we only had a small amount of bandwidth to offer compared to most of the other mirrors in the system, we usually kept our mirror’s weight low to avoid eating up our entire bandwidth allotment. Firefox 3.0.11′s release happened to line up with the end of a billing period where we had some extra bandwidth leftover, so we decided to burn through it by increasing our weight dramatically. Our traffic quickly climbed from a few Mbps to over 3Gbps, which managed to grab the attention of the folks in the Voxel NOC. They pinged us to make sure we were aware of the situation, and after learning what the traffic was being used for and a few discussions with Reed², they generously offered to donate a substantial amount of bandwidth to be used by the Mozilla mirror network.

Everything got setup just in time for the Firefox 3.5 release, and in the four months since, we’ve served more than 1.3 petabytes of updates, installers, and add-ons. And all of that coming from a single origin server that, thanks to VoxCAST, is so lightly loaded it’s able to run a BOINC client most of the time!

As impressive as that is, it’s worth noting that this mirror only accounts for (as of this writing) just over 25% of the total available capacity. Mozilla wouldn’t be able execute releases and updates for all its products with the same level of quality and reliability we all currently enjoy thanks to the behind-the-scenes and often unsung heroes of our mirror network.

So to those who volunteer your time, machines, and bandwidth to make every release and the day-to-day a success, thank you. Mozilla simply wouldn’t be able to do it without you.

¹ Megabytes
² Reed’s back at school and doing some work for Voxel as well – if you were curious as to why he’s not around Mozilla stuff as much anymore

Progress

• Landed bug 499123 on the 1.9.2 branch.
• Bug 475289 landed on trunk for a ~20ms improvement with the currently shipped plugin set.
• Reduced the number of getService calls in the microsummary service.
• Wrote a patch to get the search service using the new update-timer category, removing one of the last few pieces of non-critical work the search service does on startup.

Next Steps

• Need bug 520334 to avoid losing the benefits of 475289 once we ship a plugin with an update URL.
• Work on delaying spell check dictionary initialization in bug 496217.

Progress

• Landed bug 514407 – should help with startups where we rebuild the search plugin cache (first start, after updates) and with UI responsiveness after managing or installing new plugins.
• Bug 475289 has been updated, hopefully getting review this week.
• Wrote tests for bug 507073 – looking to get it up for review over the weekend.

In other news, I’ll be landing a patch soon that adds a new ioService property to NetUtil. Once in, it will no longer be necessary to create your own reference to nsIIOService if you’re already using NetUtil for asyncCopy or newURI.

I’ll be taking some time off starting Tuesday until the 22nd, so the next update won’t be until the 25th. See you then!

Picking up from the last post, here’s what I’ve been up to for my part of the startup improvement project:

Progress

• Landed bug 499123 – the effects on Ts as seen by Talos were mostly inconclusive (though were trending in the right direction!) and didn’t match what I had seen while testing locally.
• Did some profiling for bugs 441355 and 496217. I posted some findings for 441355 and filed bug 512837 for a potential improvement discovered in the process. I haven’t been able to reproduce 496217 before the end of DelayedStartup() yet, though I suspect the default start page and its text field is probably what’s causing this to show up at times.

Next Steps

• Didn’t have much time  to finish up bug 507073 due to the Firefox work week, but hope to do so next week.

The Firefox team has recently started using small, well-scoped projects as a way to speed development and to better split up and coordinate tasks among many developers.  Since these projects are typically small and have well defined goals, we’re also using them train ourselves to communicate more effectively and more often (has it really been three years since I last blogged‽) about what our team is up to.

My project for the last couple weeks has been startup performance – mainly focusing on identifying and fixing areas where file IO is causing us pain.  Working with David, Dietrich, Drew, Vlad and Taras, we’ve found and have started working on some potential wins.

I didn’t have much time this week due to a cold and dealings with lawyers, but I started wrapping my head around the component loader in order to try having components packaged as a single JAR so that we can cut down on individual file reads during cold, non-fastload backed startups and stats in the warm case.

As far as what’s on tap for the next week I need to:

• Write tests and finalize the patch for bug 507073.
• Figure out the right direction for bug 507101.
• Push to get outstanding reviews looked at. This has mostly been an issue of vacations and bandwidth and they’ll hopefully see some action within the next week.
• File a bug with my findings for JAR’d components as I don’t really have the platform chops to properly tackle it

Read more