GlusterFest Test Day and the Gluster Test Framework

The first beta of glusterfs 3.4 is scheduled for release tomorrow, and the project plans to greet this new beta with GlusterFest: a 24-hour test day, starting at 8pm PDT May 7/03:00 UTC May 8.

Since I plan on participating in the testing, I thought it’d be a good idea to study up on Gluster’s new test framework. You can learn all about the test framework in the video below, but I’ll also walk you through a brief testing experience of my own using the pre-beta code running on my Fedora 18 test machine.

The Gluster framework makes use of the app “prove” which is provided, for Fedora and related OSes, by the package “perl-Test-Harness.” (note: to figure out which package to install, I used the command “yum provides prove”)

sudo yum install -y perl-Test-Harness

To get the test framework itself, I headed over to the shiny new Gluster Forge to grab the source code, pull it down, to my test machine, and check out the 3.4 release branch:

git clone git://forge.gluster.org/glusterfs-core/glusterfs.git
git checkout origin/release-3.4

When Gluster Fest commences, there will be new glusterfs 3.4 beta 1 packages to install and test out, but for my pre ‘Fest test, I built new packages from the source I pulled down from git.

My colleague Justin Clift has written up a very nice howto for building Gluster RPMs from source, which you can also find at the Gluster Forge.

With my freshly-built glusterfs, glusterfs-server & glusterfs-fuse packages installed, I entered the glusterfs source directory and ran one of the “basic” tests. Note: before you run any of these tests, be aware that they delete “/var/lib/glusterd/*” in the course of creating test volumes and data and cleaning up after themselves. For more README-type information, see the README.

cd glusterfs
sudo prove ./tests/basic/bd.t

The test returned a passing result, which is great, but kind of boring. More interestingly, the second test I ran did not run successfully:

sudo prove ./tests/basic/mount.t

This test returned an error while attempting to mount a test volume over the NFS protocol, which is one of the ways you can consume Gluster storage.

mount.nfs: access denied by server while mounting MY_HOSTNAME:/patchy

Debugging time! Following guidance gleaned from the #gluster irc channel, I edited the mount.t test file to add a line containing “set -x” following the first instance of “cleanup;” in the file. I ran the test again, and it spit out a good deal more output, including the command being run to try and mount the volume:

mount -t nfs -o vers=3,nolock,soft,intr MY_HOSTNAME:/patchy /mnt/nfs/0

I could see that the test was, rightly, attempting to mount the volume with NFS version 3 (Gluster’s built-in NFS server doesn’t support version 4), so that wasn’t the problem. I also checked my SELinux log for denials (sudo tail -f /var/log/audit/audit.log | grep denied) and didn’t find any issues. I consulted my /var/log/messages as well, and found the following error message:

rpc.mountd[5465]: refused mount request from MY_IP_ADDRESS for /patchy (/patchy): unmatched host

After a little while Googling around, I remembered something I’d encountered while tangling with Gluster / oVirt integration last year: Gluster’s NFS server doesn’t play nice with another running NFS server. I checked to see if I had an NFS server running on my test machine (sudo service nfs-server status), and, sure enough, I did.

I shut the server down (sudo service nfs-server stop), re-ran my mount.t test, and — hooray — it completed successfully.

That’s perhaps not the most exciting example of Gluster’s test framework in action, but it should give you a sense of how to use it, and, when tests fail, how to go about figuring out why.

I hope to see you, virtually, at the GlusterFest this week. I’m jbrooks in the #gluster irc room — please drop in on Wednesday (or any time) and say hello.

OpenStack Summit highlights amazing open source outcomes

Last month, I attended my first OpenStack Summit as part of a team from Red Hat helping to launch a new community distribution of the popular open source infrastructure as a service (IaaS) project.

I came away from the Summit impressed with the size and velocity of OpenStack. The conference drew some 3000 users, developers, and members of the vendor community, roughly twice the draw from the previous Summit. What’s more, several of OpenStack’s component sub-projects reported a doubling in the number of active developers in their ranks over the past six months.

What impressed me more than the growth of the project, however, was the way that OpenStack embodies one of things I love most about open source, namely, its knack for helping people to peel back or eliminate barriers to innovation. The more freedom we afford ourselves to experiment with and improve the systems we care about, the more amazing results we can achieve.

For instance, the last sentiment I expected to leave the Summit with was a newfound excitement about my cable company, but after hearing what Comcast has been able to do with OpenStack, that’s just how I’m feeling. In a keynote presentation on Tuesday, Comcast SVP of Product Engineering Mark Muehl explained that the set top boxes that deliver television services to most of his company’s subscribers are comprised of closed hardware and software, produced by a separate vendor through a vertically-integrated process into which Comcast has limited visibility.

Given the fact that most of the intelligence that shapes one’s cable TV experience has been locked up in these rigid, limited devices, it’s no surprise that these services have seemed frozen in time.

Muehl reported that for its new platform, called X1, Comcast shifted most of the intelligence for its service from the set top box to an open source-based server and software infrastructure running on an OpenStack cloud. This shift is allowing Comcast to provide features that we take for granted on the Internet, like search suggestions that auto-complete as you key letters into your remote, alongside new applications that combine TV services with data from the Web. Muehl talked about the three-week development effort required to display Rotten Tomatoes movie ratings alongside program data, something that previously would not have been possible at all.

Another keynote presentation on Wednesday picked up on a similar theme, but from a very different domain. In the talk, NSA computer scientist Nathanael Burton told his own story about how rigid processes were holding back innovation at the NSA, where technologists across a wide range of disciplines work to analyze vast amounts of data.

Burton described the series of siloed computing teams and lengthy approval processes that stood between NSA technologists and the resources they required to pursue new ideas as their “carbon-based IaaS,” and noted that this highly-manual system could take weeks or months to navigate.

After learning about OpenStack around the time of its “Cactus” release, Burton brought up a below-the-radar pilot installation of OpenStack running on repurposed hardware in an NSA lab. Within two weeks, they were up and running with an initial group of users who, despite the limited capabilities of the pilot cloud, were amazed at how much simpler setting up application hosting could be.

The initial pilot cloud grew to occupy more hardware and gained access to data resources outside of the isolated lab network before eventually expanding to a hardened, production-ready deployment. Burton explained how his team further lowered the barriers to accessing the system by implementing automatic account creation, with access to a limited resource quota, based on the PKI certificates already issued to each of its users.

When their production OpenStack deployment was ready to go live, Burton’s team opted to launch quietly, without any broad announcement, but picked up hundreds of users within the first few weeks through word of mouth. Now, after a year of running their cloud in production, what started as a dramatically simpler way to allocate resources has led to more collaborative development patterns, and a model for transforming IT operations that Burton indicated the NSA’s sister agencies are preparing to adopt in the coming months.

Of course, success with OpenStack, or any other open source project or practice, can’t boil down just to smashing barriers and getting rid of pesky processes. For Comcast, Muehl acknowledged that the company’s embrace of OpenStack for deployment has come with a learning curve for developer teams unaccustomed to building for cloud environments. For the NSA, Burton explained how their introvert-friendly auto-account creating OpenStack deployment relied heavily on existing systems and had to comply with agency requirements.

The early success that can come with quick deployment and faster development cycles often leads to growing user communities that depend on these clouds to scale and to stay up and running. Nathanael Burton described the burst of popularity their cloud received at the NSA as a unicorn outbreak. Fortunately, with a healthy open source project like OpenStack, there’s no shortage of community members available to help.

Where to catch OpenStack at Red Hat Summit

If you’re planning to come to Red Hat Summit next month, you’ll have plenty of chances to learn more about OpenStack in 12 sessions on various aspects, including:

Introduction to Red Hat OpenStack

Red Hat has contributed to OpenStack for more than a year, and is committed to releasing a commercial product based on the project. In this session, Chuck Dubuque and Gerry Riveros will provide an overview of Red Hat OpenStack and detail:

  • How Red Hat OpenStack is different from community distributions
  • Red Hat’s OpenStack strategy
  • Where OpenStack fits in an open hybrid cloud infrastructure (spanning datacenter virtualization, private, and public cloud)

OpenStack: Core Components & Capabilities

OpenStack complements Red Hat’s cloud products by enabling enterprises and service providers to build an open source Infrastructure-as-a-Service (IaaS) platform. In this Taste of Training session, Bowe Strickland will introduce OpenStack’s core components and capabilities. Additionally, attendees will gain hands on experience performing basic tasks in a live OpenStack environment.

Integration of Storage, OpenStack & Virtualization

In this session, senior product and engineering leaders from Red Hat’s cloud, storage, and vitualization teams will outline the integration roadmap for Red Hat Storage, Red Hat Enterprise Virtualization, and Red Hat OpenStack.

Red Hat OpenStack Performance & Scale

In this session, Mark Wagner will review performance and scale testing of Red Hat OpenStack, including management platform performance and individual node performance. He will also discuss the tools, methodologies, and strategies used in the testing. And using test data from Red Hat’s Performance Lab, Mark will demonstrate tunings that improve performance and show where these tunings will be applied to improve out-of-the-box performance. He will also reveal tips and tricks for achieving higher density.

>> Get the complete details and find more OpenStack sessions.