MirrorManager 1.4 now in production in Fedora Infrastructure

After nearly 3 years in on-again/off-again development, MirrorManager 1.4 is now live in the Fedora Infrastructure, happily serving mirrorlists to yum, and directing Fedora users to their favorite ISOs – just in time for the Fedora 19 freeze.

Kudos go out to Kevin Fenzi, Seth Vidal, Stephen Smoogen, Toshio Kuratomi, Pierre-Yves Chivon, Patrick Uiterwijk, Adrian Reber, and Johan Cwiklinski for their assistance in making this happen.  Special thanks to Seth for moving the mirrorlist-serving processes to their own servers where they can’t harm other FI applications, and to Smooge, Kevin and Patrick, who gave up a lot of their Father’s Day weekend (both days and nights) to help find and fix latent bugs uncovered in production.

What does this bring the average Fedora user?  Not a lot…  More stability – fewer failures with yum retrieving the mirror lists, not that there were many, but it was nonzero.  A list of public mirrors where the versions are sorted in numerical order.

What does this bring to a Fedora mirror administrator?  A few new tricks:

  • Mirror admins have been able to specify their own Autonomous System Number for several years.  Clients on the same AS get directed to that mirror.  MM 1.4 adds the ability for mirror admins to request additional “peer ASNs” – particularly helpful for mirrors located at a peering point (say, Hawaii), where listing lots of netblocks instead is unwieldy.  As this has the potential to be slightly dangerous (no, you can’t request ALL ASNs be sent your way), ask a Fedora sysadmin if you want to use this new feature – we can help you.
  • Multiple mirrors claiming the same netblock, or overlapping netblocks, were returned to clients in random order.  Now they will be returned in ascending netblock size order.  This lets an organization that has a private mirror, and their upstream ISP, both have a mirror, and most requests will be sent to the private mirror first, falling back to the ISP’s mirror.  This should save some bandwidth for the organization.
  • If you provide rsync URLs, You’ll see reduced load from the MM crawler as it will now use rsync to retrieve your content listing, rather than a ton of HTTP or FTP requests.

What does this bring Fedora Infrastructure (or anyone else running MirrorManager)?

  • reduced memory usage in the mirrorlist servers.  Especially with as bad as python is at memory management on x86_64 (e.g. reading in a 12MB pickle file blows out memory usage from 4MB to 120MB), this is critical.  This directly impacts the number of simultaneous users that can be served, the response latency, and the CPU overhead too – it’s a win-win-win-win.
  • An improved admin interface – getting rid of hand-coded pages that looked like they could have been served by BBS software on my Commodore 64 – for something modern, more usable, and less error prone.
  • Code specifically intended for use by Debian/Ubuntu and CentOS communities, should they decide to use MM in the future.
  • A new method to upgrade database schemas – saner than SQLObject’s method.  This should make me less scared to make schema changes in the future to support new features.  (yes, we’re still using SQLObject – if it’s not completely broken, don’t fix it…)
  • Map generation moved to a separate subpackage, to avoid the dependency on 165MB of  python-basemap and python-basemap-data packages on all servers.

MM 1.4 is a good step forward, and hopefully I’ve laid the groundwork to make it easier to improve in the future.  I’m excited that more of the Fedora Infrastructure team has learned (the hard way) the internals of MM, so I’ll have additional help going forward too.

s3cmd sync enhancements and call for help

Coming soon, Fedora and EPEL users with virtual machines in Amazon (US East for starters) will have super-fast updates.  I’ve been hacking away in Fedora Infrastructure and the Fedora Cloud SIG to place a mirror in Amazon S3.  A little more testing, and I’ll flip the switch in MirrorManager, and all Amazon EC2 US East users will be automatically directed to the S3 mirror first.  Yea!  Once that looks good, if there’s enough demand, we can put mirrors in other regions too.

I hadn’t done a lot of uploading into S3 before.  It seems the common tool people use is s3cmd.  I like to think of ‘s3cmd sync’ as a replacement for rsync.  It’s not – but with a few patches, and your help, I think it can be made more usable.  So far I’ve patched in –exclude-from so that it doesn’t walk the entire local file system only to later prune and exclude files – a speedup of over 20x in the Fedora case.  I added a –delete-after option, because there’s no reason to delete files early in the case of S3 – you’ve got virtually unlimited storage.  And I added a –delay-updates option, to minimize the amount of time the S3 mirror yum repositories are in an inconsistent state (now down to a few seconds, and could be even better).  I’m waiting on upstream to accept/reject/modify my patches, but Fedora Infrastructure is using my enhancements in the meantime.

One feature I’d really like to see added is to honor hardlinks.  Fedora extensively uses hardlinks to cut down on the number of files, amount of storage, and time needed to upload content.  Some files in the Fedora tree have 6 hardlinks, and over three quarters of the files have at least one hardlink sibling.  Unfortunately, S3 doesn’t natively understand anything about hardlinks.  Lacking that support, I expect that S3 COPY commands would be the best way to go about duplicating the effect of hardlinks (reduced file upload time), even if we don’t get all the benefits.  However, I don’t have a lot more time available in the next few weeks to create such a patch myself – hence my lazyweb plea for help.  If this sounds like something you’d like to take on, please do!

FUDCon Blacksburg videos

I shot videos of several of the presentations at the Fedora User and Developer Conference yesterday.  For your viewing pleasure:

  • “State of Fedora” from the Fedora Project Leader, Jared Smith [ogg]
  • Mike McGrath, team lead for OpenShift, demoing OpenShift [ogg]
  • Jon Masters and Chris Tyler, on the ARM architecture in Fedora [ogg]. ARM is a secondary architecture today.  By Fedora 18, with your help, it needs to become a primary architecture.
  • David Nalley presented on CloudStack, which is aiming for Fedora 17 inclusion. [ogg]
  • Dan Prince and Russell Bryant giving an introduction to OpenStack [ogg]
  • Mo Morsi presenting the Aeolus cloud management project [ogg]

[Update 1/18/2012] I was able to upload all the videos to YouTube.  http://www.youtube.com/playlist?list=PL2BAA7FF83E6482C2
is a playlist with all 6.

Consistent Network Device Naming updates

Today I released biosdevname v0.3.7, after listening to feedback from all across the web, including NetworkWorld, LWN, and Slashdot.  No, I’m not killing the feature, as some might hope, but some changes are in order.

First, it’s amazing how many people hated the ‘#’ character in device names.  Yes, that was bound to cause some problems, but nothing that couldn’t be fixed given enough time.  But since it’s early in the game, changing that character from ‘#’ to ‘p’ accomplishes the same goal, with less chance of breakage, so that’s done.  pci<slot>p<port>_<vf> it is….

Second, the various virtual machine BIOSes each do something slightly different for the network devices they expose.  VMware exposes the first NIC (traditionally eth0) as in PCI slot 3.  KVM exposes the first NIC as in PCI slot 2, but has no information about the second NIC.  Xen doesn’t expose anything, so those all kept the ethX naming convention.

To address these discrepancies, and because there is no physical representation of a (virtual) NIC in a virtual machine, biosdevname no longer suggests a new name for NICs if running in a VM guest.  This means all VM guests keep ethX as their naming convention. Thanks to colleague Narendra K for this fix.

Third, for everyone who still thinks renaming devices is a really bad idea, you get an out.  A new kernel command line option, honored by udev, lets you disable biosdevname.  biosdevname=0 will prevent biosdevname from being invoked, effectively disabling this feature, leaving you with the ethX names.

All this, and the usual assorted bug fixes as biosdevname gets more widespread exposure and testing.

Love it?  Hate it?  Let me know.  You can find me (mdomsch) on IRC on FreeNode in #biosdevname, #udev, or #fedora-devel, as well as the usual mailing lists.

Fedora Test Day today – please join us

Today is the official  Fedora Test Day for Consistent Network Device Naming.  Given all the coverage this week on NetworkWorld and Slashdot, I would like to see widespread testing of this feature, to assuage the concerns and misconceptions raised there.  Testing is simple – download and boot the LiveISO, and report success or failure on the wiki page.  You can even try it out on a running Fedora 14 instance if you like.

The Dell engineers who have been working on this for years will be online in #fedora-test-day on FreeNode IRC today if you have any questions.  Please join us.  Thanks for your time and participation.

Consistent Network Device Naming coming to Fedora 15

One of my long-standing pet projects – Consistent Network Device Naming, is finally coming to Fedora (emphasizing the 2 of the Fedora F’s: Features and First), and thereafter, all Linux distributions.  What is this, you ask?

Systems running Linux have long had ethernet network devices named ethX.  Your desktop likely has one ethernet port, named eth0.  This works fine if you have only one network port, but what if, like on Dell PowerEdge servers, you have four ethernet ports?  They are named eth0, eth1, eth2, eth3, corresponding to the labels on the back of the chassis, 1, 2, 3, 4, respectively.  Sometimes.  Aside from the obvious confusion of names starting at 0 verses starting at 1, other race conditions can happen such that each port may not get the same name on every boot, and they may get named in an arbitrary order.  If you add in a network card to a PCI slot, it gets even worse, as the ports on the motherboard and the ports on the add-in card may have their names intermixed.

While several solutions have  been proposed over time (detailed at Linux Plumbers Conference last year), none were deemed acceptable, until now.

Enter biosdevname, the tool Dell has developed to bring sanity (and consistency!) to network device names.  Biosdevname is a udev helper, which renames network interfaces based on information presented by system BIOS.

The new naming convention is as follows:

  • em[1-N] for on-board (embedded) NICs (# matches chassis labels)
  • pci<slot>#<port> for cards in PCI slots, port 1..N
  • NPAR & SR-IOV devices add a suffix of _<vf>, from 0..N depending on the number of Partitions or Virtual Functions exposed on each port.
  • Other Linux conventions, such as .<vlan> and :<alias> suffixes remain unchanged and are still applicable.

This provides a sane mapping of Linux network interface name to externally visible network port (RJ-45 jack).

Where do we get this information?  The algorithm is fairly simple:

  • If system BIOS exposes the new PCI Firmware Specification 3.1 ACPI _DSM method, we get the interface label and index from ACPI, and use those.
  • Else if system BIOS exposes an index and label in SMBIOS 2.6 types 9 and 41, use the index value.
  • Else if system BIOS exposes index via the HP proprietary SMBIOS extension, use that.
  • Else fall back to using the legacy PCI IRQ Routing Table to figure out which slots devices are in, sort the PCI device list in breadth-first order, and assign index values.

How will this affect you?

If you have scripts that have hard-coded eth0 or have assumptions that ethX is a particular port, your scripts are already broken (you may just not know it yet).  Begin planning on using the new interface names going forward, adjusting your scripts as necessary.

Fedora 15 will be the first distribution to use biosdevname by default.  There will be a Test Day on Thursday, January 27.  I encourage you to download the Live image, boot it on your system, and verify that your network interfaces are now named according to the above convention, and that all works as expected.  You may also take the opportunity to review your custom scripts, looking for hard-coded ethX values, and prepare for the coming name change.

Once we get sufficient exposure and verification using Fedora, I expect to see this change roll into other Linux distributions, and other operating systems, over time.  Consider yourself warned.

Fedora Elections – one day left

Quick reminder that Fedora elections for the Board, FESCo, and FAMSCo are open through Sunday, November 28.  Go Vote! I did.  While it’s a bit weird for me not to be running for re-election to the Board, I’m excited by the slate of candidates, and know it’ll be in good hands.

Fedora Elections – it starts now

You know it’s election season.  The commercials are unavoidable.  Candidates kissing babies.  “The other guys say…”.  Shameless promotion and pandering.  Division and false dichotomies rule the airwaves.

However, in Fedora-land, the tenor is quite calm, welcoming, and actively non-partisan, even though we have several elections going on.

First (timewise), the code name for Fedora 15.  This is the quirky name that we give each release, that both serves as inspiration for the artwork that our fantastic design team generates, and that ties our releases together, both backwards in history to our Red Hat Linux roots and continuing on with each subsequent Fedora release.  Each code name has a non-obvious link to the predecessor release, while leaving wide open the range of future release names.

After sifting through 50 suggestions, and trimming the list for possible trademark conflicts, we are left with 5 names that any Fedora member can vote for:

- Asturias
- Blarney
- Sturgis
- Lovelock
- Pushcart

Take a moment to reflect on the nature of each of the above, and then vote for your favorite.

Second, we have started the nomination process for the semi-annual election of seats on the Fedora Project Board and the Fedora Engineering Steering Committee (FESCo), and for the annual election of members of the Fedora Ambassadors Steering Committee (FAMSCo).  These are important, highly visible, volunteer positions which guide the Project as a whole, the technical details of the “product” of Fedora the distro, and coordinate the activities of our outreach and evangelism teams.  Please consider running for one of these offices by nominating yourself, or, you may nominate someone you would like to see be elected (with their consent of course).

For my part, I have had the pleasure of serving on the Board for the last 5 years since its inception.  We have come a long way during my tenure, from the elimination of the dreaded “Core vs Extras” distinction, to the creation of the Spins and Remixes processes that let anyone, anywhere, use Fedora for any purpose, to the creation of the Stable Updates policy, and finally a Vision Statement we can all get behind.  While there is plenty more to be done, it’s time for me to step aside from policy-making, and assist a new Board in delivering on these visions.  I thank the community for allowing me to serve on the Board thus far, and am counting on you to support the newly elected Board, FESCo, and FAMSco, in continuing to develop and deliver a fantastic distribution.

Interview with Jared Smith, new Fedora Project Leader

I thought this was a well-done interview by Henry Kingman of Linux.com, welcoming new Fedora Project Leader Jared Smith.

I’ve been fortunate to serve on the Fedora Project Board since 2006, and to have the opportunity to work with several FPLs (Max and Paul directly, and their predecessors Michael, Christian, and Greg in various capacities), and I look forward to working with Jared even more now.  He brings a wealth of experience, talent, and enthusiasm that’s contagious.

I’m also quite pleased with the way the transitions between FPLs have been handled.  Both Max and Paul knew for themselves when they were ready for new challenges – not that they were “burned out” (e.g. CATB lesson #5), or that they were no longer being effective, but realized that they could apply their talents towards Fedora in new ways, while opening new opportunities for another talented and respected contributor.  That’s a big part of building a healthy community.

Fedora Elections – Go Vote!

It’s that time of year again. Fedora is holding elections for 3 seats on the Board, and for 5 seats on the Engineering Steering Committee (FESCo). I am very exited by the caliber of contributors who have volunteered their time to serve on these committees, some looking for their first elected leadership role, some looking to continue or redouble their involvement.

I look forward to serving with the new Board members as we continue to map out the high-level goals for Fedora.

Please take a look at the slate of candidates, review the questionnaire answers and the Town Hall logs, make up your mind, and Go Vote!