Since I plan on participating in the testing, I thought it’d be a good idea to study up on Gluster’s new test framework. You can learn all about the test framework in the video below, but I’ll also walk you through a brief testing experience of my own using the pre-beta code running on my Fedora 18 test machine.
The Gluster framework makes use of the app "prove" which is provided, for Fedora and related OSes, by the package "perl-Test-Harness." (note: to figure out which package to install, I used the command "yum provides prove")
sudo yum install -y perl-Test-Harness
To get the test framework itself, I headed over to the shiny new Gluster Forge to grab the source code, pull it down, to my test machine, and check out the 3.4 release branch:
git clone git://forge.gluster.org/glusterfs-core/glusterfs.git git checkout origin/release-3.4
When Gluster Fest commences, there will be new glusterfs 3.4 beta 1 packages to install and test out, but for my pre ‘Fest test, I built new packages from the source I pulled down from git.
With my freshly-built glusterfs, glusterfs-server & glusterfs-fuse packages installed, I entered the glusterfs source directory and ran one of the "basic" tests. Note: before you run any of these tests, be aware that they delete "/var/lib/glusterd/*" in the course of creating test volumes and data and cleaning up after themselves. For more README-type information, see the README.
cd glusterfs sudo prove ./tests/basic/bd.t
The test returned a passing result, which is great, but kind of boring. More interestingly, the second test I ran did not run successfully:
sudo prove ./tests/basic/mount.t
This test returned an error while attempting to mount a test volume over the NFS protocol, which is one of the ways you can consume Gluster storage.
mount.nfs: access denied by server while mounting MY_HOSTNAME:/patchy
Debugging time! Following guidance gleaned from the #gluster irc channel, I edited the mount.t test file to add a line containing "set -x" following the first instance of "cleanup;" in the file. I ran the test again, and it spit out a good deal more output, including the command being run to try and mount the volume:
mount -t nfs -o vers=3,nolock,soft,intr MY_HOSTNAME:/patchy /mnt/nfs/0
I could see that the test was, rightly, attempting to mount the volume with NFS version 3 (Gluster’s built-in NFS server doesn’t support version 4), so that wasn’t the problem. I also checked my SELinux log for denials (sudo tail -f /var/log/audit/audit.log | grep denied) and didn’t find any issues. I consulted my /var/log/messages as well, and found the following error message:
rpc.mountd: refused mount request from MY_IP_ADDRESS for /patchy (/patchy): unmatched host
After a little while Googling around, I remembered something I’d encountered while tangling with Gluster / oVirt integration last year: Gluster’s NFS server doesn’t play nice with another running NFS server. I checked to see if I had an NFS server running on my test machine (sudo service nfs-server status), and, sure enough, I did.
I shut the server down (sudo service nfs-server stop), re-ran my mount.t test, and — hooray — it completed successfully.
That’s perhaps not the most exciting example of Gluster’s test framework in action, but it should give you a sense of how to use it, and, when tests fail, how to go about figuring out why.
I hope to see you, virtually, at the GlusterFest this week. I’m jbrooks in the #gluster irc room — please drop in on Wednesday (or any time) and say hello.