Background On July 15th we ran into a number of issues with replication on mysql2 on a couple of session tables. This caused replication to be paused, and a large number of statements had to be skipped. Replication was restarted successfully. On July 16th some more issues with the same tables were encountered, but in far greater number. A ticket was created to track the issue. Replication was restarted several times, but on the week of the 20th a decision was made to entirely reload mysql2 and examine some alternative replication methods (primarily row-based replication).

The day is May 18. The location is the Portland’s Crystal Ballroom. The conference is Write the Docs (WtD). Excitement and anticipation fill the air as we collectively munch on breakfast foods and find a seat. The keynote begins and immediately sets the mood: docs are fun, docs are interesting, and here’s how you can make your docs awesome. WtD was quite the experience and it got me excited about documentation, something I admit I never expected to be all that excited about. At times it felt like a support group for non-technical individuals that work with engineers, other times it felt like a storyteller sharing with us their adventure in documenting some massive project, and most importantly it was always engaging and interesting. Some of my most memorable talks were of Twillio’s efforts to make their documentation better, GitHub’s workflow of writing docs for GitHub with GitHub, and Google’s new documentation tool and how it was developed and adopted in a grass roots effort as opposed to a top-down corporate approach. I even gave a Lightning Talk on “How to Write the Best Email You’ve Never Written… Until Now” which went over very well and seemed to speak to a lot of people.

Spring is almost here, and that means Beaver BarCamp 15 is quickly approaching. The OSL’s annual unconference is scheduled for Saturday, April 18th. Participants are beginning to gather ideas and information for their sessions, and the Open Source Lab is offering up some helpful suggestions on how to present a successful unconference session: Plan ahead. While the spirit of BarCamp is spontaneous and unstructured, it is a good idea to gather enough information ahead of time to present a well-informed BarCamp presentation. Planning for your session can help you budget your time and make you less nervous.

A few months ago, Emily Dunham and I were chatting about the talks we had recently presented at the Seattle GNU/Linux conference, and the topic of other conferences and events we were interested in speaking at came up. She pointed me to the Southern California Annual Linux Expo (SCALE) and encouraged me to submit one of my talks. Two hours later, I had successfully submitted a proposal to give a talk about the media suite Blender 3D and promptly forgot about the whole thing. A few weeks later I got an email congratulating me on being accepted as a speaker to Scale x13; a few weeks after that I was on a plane to LA getting ready for an eventful (and exhausting) weekend.

Over 100 students and community members attended the Open Source Lab’s DevOps DayCamp in the Kelley Engineering Center at Oregon State University on Saturday, Oct. 11. DevOps DayCamp, a new event this year, was the kick-off for DevOps BootCamp, an extracurricular training program in its second year that takes place throughout the school year for OSU students and community members interested in DevOps. DayCamp is a dual-track event designed to accommodate varying levels of DevOps experience. Tracks consisted of a beginner track and an advanced track, with one joint session for all experience levels.

At the Open Source Lab we host many of our sites as a Drupal multisite. This means that we have several instances of Drupal using the same theme, and then we can populate each site with different content as needed (for instance, cass.oregonstate.edu and osuosl.org would be different websites with different messages, but with Drupal they can look and act the same. Pretty neat!). Since not all of our sites are used by the OSL (i.e. the Center for Applied Systems and Software), I recently needed to make our logo and organization name into variables so that the user could just upload an image and fill in a text box to customize the site theme to their organization. Luckily, Drupal makes my job pretty easy.

The Open Source Lab will host the new DevOps DayCamp on Saturday, October 11, in the Kelley Engineering Center from 9 a.m. until 6 p.m. The event is free and open to the public. DevOps DayCamp will kick off DevOps BootCamp, allowing students to start their DevOps education early in the school year. In order to accommodate different experience levels, DayCamp is comprised of two tracks: a beginner track and an advanced track. The beginner track will help inexperienced attendees get started with DevOps through introductory sessions and workshops on the basics of DevOps. Additionally, the advanced track will be comprised of a hands-on hackathon with educational sessions throughout the day for the more experienced DevOps crowd. Advanced track sessions will be given by industry professionals and will include Ansible, Travis CI and Docker.

In the simplest of terms, OpenStack is a massive undertaking. The goal of OpenStack is to fit just about every use case imaginable. This goal brings with it a daunting list of configuration options and requires a larger understanding of networking and virtualization systems. Couple that with cryptic error messages, and this makes for a system that can easily crush a newbie’s confidence and cause them to scrap the system altogether. Luckily, there are some projects out there trying to lower the entry bar and get more people introduced to OpenStack. Two of the projects most referred to are DevStack (which the OpenStack developers actually use for development and testing) and RDO Packstack from RedHat. Last summer, I began teaching myself OpenStack using Packstack. This creates a Proof-Of-Concept (POC) deployment suitable to get comfortable with the architecture, the concepts and even some architectural design choices. Fast forward a few months and, having gathered a much larger (and yet still very small) understanding of how OpenStack works, the OSL deployed the POC Packstack setup and had our Systems Admins deploying VMs to develop and test our Chef cookbooks. Next comes the most daunting challenge: configuration management. Configuration management requires a much greater understanding of the underlying system because you have to know which options should be changed and which options should not. The problem with OpenStack in configuration management is that there are so many options to set/change in each OpenStack service. The other challenge is knowing how to choose the best option for your use case. You can create your own management scripts or you can look into the various other methods being developed (such as RedHat’s RDO Foreman using Puppet or Chef’s Stackforge). I first started with RDO Foreman mostly due to the RDO support community. This community is filled with helpful and knowledgeable people who are working on the configuration management scripts. Ultimately, however, we went with Stackforge over RDO Foreman, mainly because the latter lacked some flexibility we required. It also turned out that the Stackforge project would allow for us to run multiple environments as well as allowing us to run OpenStack on IBM Power CPUs. After a lot of testing, tweaking, frustration (head-desking, face palming, cursing), moments of inspiration, sudden realizations, and wonderful moments of clarity, we had a working test environment. Next, we had to reconcile the differences between the testing environment and the production environment (as well as throwing in things like changing from OpenStack’s Havana release to Icehouse for good measure). Currently, we are finishing up some testing of our x86 production OpenStack cluster and we even have a production OpenPower OpenStack cluster. Overall, this project has been an incredible opportunity that enabled me to experience different roles in project management, build connections in the various OpenStack community, learn more about the underlying systems that OpenStack uses (kvm, networking in too much detail, web servers, etc.), and, most of all, drastically increase my abilities to debug problems with little information while being persistent in learning how to properly fix those bugs. Ultimately the skills and concepts developed while working with OpenStack can be transferred to almost any setting, making it a valuable teaching tool.

At the OSL we have shared workstations, most of which are named after colors. In the NOC, I usually work at emerald.workstation.osuosl.bak (Figure 1). I use tmux (Figure *) to multiplex so I can have multiple terminals open in a single ssh connection and connect to my session from anywhere. When splitting the terminal ertically, the prompt can get so long that it’s hard to see the command that I’m entering (Figure 2). I’d like my prompt to automatically shorten itself in narrow windows. Fortunately, my terminal already knows how much space it has available: $COLUMNS is an environment variable which stores how wide your current terminal is, and the default unixism is 80. So in order to save space, I’d like to shorten my prompt to only a directory listing and a colored character replacing the normal $ or >. Behold! (Figure 3, 4) Using a case statement and filtering out the name of the workstation from the domain name, I can color code my prompt based on hostname. This very easily lets me know $HOSTNAME (again, this is an environment variable which contains /etc/hostname), and indirectly /usr/bin/whoami since almost every other user will preface their prompt with a $USER. This was a 10 minute exercise in learning how to write case statements in bash and provide some cute utility to an otherwise stale prompt. The other thing you might notice is that I directly add the unicode heart into the prompt. This causes difficulty on TTYs and some terminal emulators where it is replaced with a ♦ (which is an ASCII character). There should be a check to make sure it can be loaded and replaced with something else if it fails. All in all, this is just quick hack to make life prettier! Source!

The OSL made a strong showing at the O’Reilly Open Source Conference (OSCON) this year with the majority of the student employees attending along with all of the full time staff. The conference was held in Portland, OR, and fell on July 20-24. The Lab always places a special emphasis on education, and the conference was certainly an educational experience. The expo hall held dozens of booths filled with information and demonstrations about open source projects and the companies that support them.