The open source web of trust

by —

When do you trust open source? With so many recent attacks using insecure software as a vector–thousands upon thousands of attacks–the reliance on the vigilance of others seems almost foolhardy.

Open source has an answer for the lack of security, in that the code is presumably visible to all. However, few actually read the source code of the applications they rely on; there are simply too many of them, their codebases are too large, and chances are that most people reading the source code couldn’t do a proper security analysis anyway.

Doesn’t this mean that open source and closed source end up being on an even playing field with respect to security? If nobody reads the source code, open or closed, where’s the difference in security between the two licensed codebases?

With the aid of Josh Bressers, lead of Red Hat’s Product Security Team, let’s look at a process by which one can gain some reassurance concerning open source security, along with a pointed look at one more way in which Red Hat contributes to the entire Linux community.

The basic argument, in simplified straw man form, looks like this:

  • Open source is no more secure than closed source.
  • You don’t review every line of open source code; you trust the projects.
  • Since you don’t review every line of code, trusting your closed source vendor is the same as trusting an open source project.

This almost makes sense–but according to Josh, this is what actually happens:

You probably haven’t reviewed the source code to your web browser (how you’re reading this page). Even if you did review the code, you need to trust whomever built the source code. Up to this point, open source is on par with closed source.

If you have a closed source product, you have to trust the vendor. There’s zero choice. If they make a claim, you probably can’t prove them wrong. Many closed source vendors have been caught doing silly things, but security isn’t trivial, and people easily forget the sins of the past.

Open source and closed source diverge when you consider the people who do review your source code. Red Hat has a number of security teams who do actually conduct source code audits. Do they catch everything? Certainly not, but they are essentially an independent third party looking at public source code. Even if you don’t like what they find, you can go look and verify security for yourself. You don’t have to trust the third party; you can still conduct your own analysis.

There are also other groups doing third-party security verification. That’s the amplification power of open source.

All of this is what makes open source more secure. We can’t lie, we can’t cheat, we can’t steal. The emperor is naked and knows it. More than once Red Hat has hit security issues where we wish we could have weaselled out of them, but at the end of the day, open source kept us honest.

We think open source is more secure than closed source. Every time.

GlusterFest Test Day and the Gluster Test Framework

by —

The first beta of glusterfs 3.4 is scheduled for release tomorrow, and the project plans to greet this new beta with GlusterFest: a 24-hour test day, starting at 8pm PDT May 7/03:00 UTC May 8.

Since I plan on participating in the testing, I thought it’d be a good idea to study up on Gluster’s new test framework. You can learn all about the test framework in the video below, but I’ll also walk you through a brief testing experience of my own using the pre-beta code running on my Fedora 18 test machine.

The Gluster framework makes use of the app “prove” which is provided, for Fedora and related OSes, by the package “perl-Test-Harness.” (note: to figure out which package to install, I used the command “yum provides prove”)

sudo yum install -y perl-Test-Harness

To get the test framework itself, I headed over to the shiny new Gluster Forge to grab the source code, pull it down, to my test machine, and check out the 3.4 release branch:

git clone git://forge.gluster.org/glusterfs-core/glusterfs.git
git checkout origin/release-3.4

When Gluster Fest commences, there will be new glusterfs 3.4 beta 1 packages to install and test out, but for my pre ‘Fest test, I built new packages from the source I pulled down from git.

My colleague Justin Clift has written up a very nice howto for building Gluster RPMs from source, which you can also find at the Gluster Forge.

With my freshly-built glusterfs, glusterfs-server & glusterfs-fuse packages installed, I entered the glusterfs source directory and ran one of the “basic” tests. Note: before you run any of these tests, be aware that they delete “/var/lib/glusterd/*” in the course of creating test volumes and data and cleaning up after themselves. For more README-type information, see the README.

cd glusterfs
sudo prove ./tests/basic/bd.t

The test returned a passing result, which is great, but kind of boring. More interestingly, the second test I ran did not run successfully:

sudo prove ./tests/basic/mount.t

This test returned an error while attempting to mount a test volume over the NFS protocol, which is one of the ways you can consume Gluster storage.

mount.nfs: access denied by server while mounting MY_HOSTNAME:/patchy

Debugging time! Following guidance gleaned from the #gluster irc channel, I edited the mount.t test file to add a line containing “set -x” following the first instance of “cleanup;” in the file. I ran the test again, and it spit out a good deal more output, including the command being run to try and mount the volume:

mount -t nfs -o vers=3,nolock,soft,intr MY_HOSTNAME:/patchy /mnt/nfs/0

I could see that the test was, rightly, attempting to mount the volume with NFS version 3 (Gluster’s built-in NFS server doesn’t support version 4), so that wasn’t the problem. I also checked my SELinux log for denials (sudo tail -f /var/log/audit/audit.log | grep denied) and didn’t find any issues. I consulted my /var/log/messages as well, and found the following error message:

rpc.mountd[5465]: refused mount request from MY_IP_ADDRESS for /patchy (/patchy): unmatched host

After a little while Googling around, I remembered something I’d encountered while tangling with Gluster / oVirt integration last year: Gluster’s NFS server doesn’t play nice with another running NFS server. I checked to see if I had an NFS server running on my test machine (sudo service nfs-server status), and, sure enough, I did.

I shut the server down (sudo service nfs-server stop), re-ran my mount.t test, and — hooray — it completed successfully.

That’s perhaps not the most exciting example of Gluster’s test framework in action, but it should give you a sense of how to use it, and, when tests fail, how to go about figuring out why.

I hope to see you, virtually, at the GlusterFest this week. I’m jbrooks in the #gluster irc room — please drop in on Wednesday (or any time) and say hello.

OpenStack Summit highlights amazing open source outcomes

by —

Last month, I attended my first OpenStack Summit as part of a team from Red Hat helping to launch a new community distribution of the popular open source infrastructure as a service (IaaS) project.

I came away from the Summit impressed with the size and velocity of OpenStack. The conference drew some 3000 users, developers, and members of the vendor community, roughly twice the draw from the previous Summit. What’s more, several of OpenStack’s component sub-projects reported a doubling in the number of active developers in their ranks over the past six months.

What impressed me more than the growth of the project, however, was the way that OpenStack embodies one of things I love most about open source, namely, its knack for helping people to peel back or eliminate barriers to innovation. The more freedom we afford ourselves to experiment with and improve the systems we care about, the more amazing results we can achieve.

For instance, the last sentiment I expected to leave the Summit with was a newfound excitement about my cable company, but after hearing what Comcast has been able to do with OpenStack, that’s just how I’m feeling. In a keynote presentation on Tuesday, Comcast SVP of Product Engineering Mark Muehl explained that the set top boxes that deliver television services to most of his company’s subscribers are comprised of closed hardware and software, produced by a separate vendor through a vertically-integrated process into which Comcast has limited visibility.

Given the fact that most of the intelligence that shapes one’s cable TV experience has been locked up in these rigid, limited devices, it’s no surprise that these services have seemed frozen in time.

Muehl reported that for its new platform, called X1, Comcast shifted most of the intelligence for its service from the set top box to an open source-based server and software infrastructure running on an OpenStack cloud. This shift is allowing Comcast to provide features that we take for granted on the Internet, like search suggestions that auto-complete as you key letters into your remote, alongside new applications that combine TV services with data from the Web. Muehl talked about the three-week development effort required to display Rotten Tomatoes movie ratings alongside program data, something that previously would not have been possible at all.

Another keynote presentation on Wednesday picked up on a similar theme, but from a very different domain. In the talk, NSA computer scientist Nathanael Burton told his own story about how rigid processes were holding back innovation at the NSA, where technologists across a wide range of disciplines work to analyze vast amounts of data.

Burton described the series of siloed computing teams and lengthy approval processes that stood between NSA technologists and the resources they required to pursue new ideas as their “carbon-based IaaS,” and noted that this highly-manual system could take weeks or months to navigate.

After learning about OpenStack around the time of its “Cactus” release, Burton brought up a below-the-radar pilot installation of OpenStack running on repurposed hardware in an NSA lab. Within two weeks, they were up and running with an initial group of users who, despite the limited capabilities of the pilot cloud, were amazed at how much simpler setting up application hosting could be.

The initial pilot cloud grew to occupy more hardware and gained access to data resources outside of the isolated lab network before eventually expanding to a hardened, production-ready deployment. Burton explained how his team further lowered the barriers to accessing the system by implementing automatic account creation, with access to a limited resource quota, based on the PKI certificates already issued to each of its users.

When their production OpenStack deployment was ready to go live, Burton’s team opted to launch quietly, without any broad announcement, but picked up hundreds of users within the first few weeks through word of mouth. Now, after a year of running their cloud in production, what started as a dramatically simpler way to allocate resources has led to more collaborative development patterns, and a model for transforming IT operations that Burton indicated the NSA’s sister agencies are preparing to adopt in the coming months.

Of course, success with OpenStack, or any other open source project or practice, can’t boil down just to smashing barriers and getting rid of pesky processes. For Comcast, Muehl acknowledged that the company’s embrace of OpenStack for deployment has come with a learning curve for developer teams unaccustomed to building for cloud environments. For the NSA, Burton explained how their introvert-friendly auto-account creating OpenStack deployment relied heavily on existing systems and had to comply with agency requirements.

The early success that can come with quick deployment and faster development cycles often leads to growing user communities that depend on these clouds to scale and to stay up and running. Nathanael Burton described the burst of popularity their cloud received at the NSA as a unicorn outbreak. Fortunately, with a healthy open source project like OpenStack, there’s no shortage of community members available to help.