gtuhl: startup IT and software development

I have really been enjoying the book Founders at Work. It is long and dense but broken down into lots of small interviews so I have been jumping back to it between reading other things.

If you have been a part of a startup the repeated themes will feel very familiar and real and the individual interviews are quick, informative reads. The fact that the interviewees represent successful startups also serves as a reinforcing support for work you may be doing now. I’d highly recommend checking it out.

A more in depth post about PostgreSQL or something similar should happen before too long, needed something brief to push the completely off topic previous post from the top.

This has no place on a technically-oriented blog, and perhaps I am just an unenlightened, uncultured grump - but despite what the media would lead people to believe other things happened in 1997 aside from a member of an irrelevant, though “royal” family passing away. A sampling from wikipedia of events happening around the same time.

• August 15 - India celebrates 50 years of independence from British rule.
• August 20 - Souhane massacre in Algeria; over 60 people killed, 15 kidnapped.
• August 25 - Egon Krenz, the former East German leader, is convicted of a shoot-to-kill Berlin Wall policy.
• August 26 - Beni-Ali massacre in Algeria; 60-100 people killed.
• August 26 - The Independent International Commission on Decommissioning is set up in Northern Ireland, as part of the peace process.
• August 29 - Rais massacre in Algeria; over 98 (and possibly up to 400) people killed.
• August 29 - Christopher Maier of Lexington, Kentucky is bludgeoned to death by serial killer Angel Maturino Resendiz. Angel also rapes and beats Christopher’s girlfriend, who survives. This is the first of a string of murders that Angel commits.
• September 3 - Arizona Governor Fife Symington is convicted for various crimes tied to his real estate business, effectively forcing him out of office.
• September 4 - In Lorain, Ohio, the last Ford Thunderbird for three years rolls off the assembly line.
• September 5 - Beni-Messous massacre in Algeria; over 87 killed.
• September 5 - The IOC picks Athens, Greece to be the host city for the 2004 Summer Olympics.
• September 5 - Mother Theresa of Calcutta dies of heart failure in Kolkata, India.
• September 6 - A Jean Michel Jarre Oxygene in Moscow concert, celebrating the city’s 850th anniversary, draws 3.5 million people.
• September 7 - First test flight of the F-22 Raptor.

My policy over the past week, and apparently today as well, is to remove any news source that runs more than 1 story about this event from RSS and/or iGoogle (which is a pretty lame name by the way).

I’ve become a pretty huge fan of openSUSE. The installer is excellent, it just works really well and I really like having the option of yast to manage most aspects of the system, even when working from a command line. In comparison to RHEL it has more file system options, newer/more rpms in the official repositories, and in my opinion yast is superior to RHEL’s up2date. If support is an issue you can get Suse Enterprise Linux preinstalled by hardware vendors (including Dell) as well as enterprise support from Novell.

Though the openSUSE installer is pretty solid manually booting, configuring, installing, and updating an OS can get old really fast especially if you are installing on machines meant to have the same or similar roles. As part of the effort to improve our ability to manage more machines at work I decided to explore two tools to make life easier:

• Setting up our own installation server
• Using autoyast to automate 95% of the install for new machines

I generally followed the guidance offered at this novell.com page but want to walk through the specific process I went through as well as some specific gotchas and details in the hopes of helping out anybody else trying to do the same with 64bit openSUSE 10.2 on servers. By “on servers” I mean “no x windows”.

Setting up the Installation Server

If you have an existing openSUSE box setting up the installation server is pretty easy. Here are the steps involved in setting the server up and linking it to the official Novell yast repostories so your new installations get updated packages.

• Run yast and goto Software -> Software Management
• Search for and install yast2-instserver
• Exit and restart yast and goto Miscellaneous -> Installation Server
• From here you will be walked through the process of copying the files from your installation media to the HD and exposing the sources with FTP, HTTP, or NFS
• For this particular example I went with FTP, openSUSE installed and attempted to configure vsftp
• I had to manually /sbin/service vsftpd start to make it work.
• By default vsftp was configured to allow only anonymous access with read-only permissions, and /srv/ftp was set as the root of what anonymous can see on the disk, so the config was perfect by default.
• The full path to the 64bit installation source CD contents was /srv/ftp/sources/suse-10.2-64bit/. It is a good idea to give the source directory a specific name as that allows you to add alternate sources (like 32bit) to the same installation server in the future.
• Go to /srv/ftp/sources/suse-10.2-64bit/CD1 and create a new file named add_on_products.
• Edit this new file and enter any number of source repositories that you want to be included in new installs - 1 on each line. In my case it looked like this:
  http://download.opensuse.org/distribution/10.2/repo/oss
http://download.opensuse.org/distribution/10.2/repo/non-oss
http://download.suse.com/update/10.2
• Sources entered here will also automatically be registered as installation sources for the new machines. If you aren’t using 10.2 your source repositories will be different. Check this page for all of them.
• That wraps up the installation server. Assuming the vsftp service started up you are good to go.

At this point, you can setup new openSUSE machines by installing against this server. You would need to boot the machine with some sort of openSUSE installation media (the DVD, CD1, a properly setup usb key, or the minimal install CD) to get to the installation menu. From there hit F4, enter your FTP installation server and the /sources/suse-10.2-64bit/CD1 directory, press enter, and then continue with the installation. Having the installation server is really nice because you can control and manage a single, consistent set of rpms.

Setting up autoyast

Just having a central installation server is great but with autoyast you can almost completely automate installation of new openSUSE servers. This works by creating an autoyast control file at which you point new installations. The control file can include instructions for disk partitioning, installed software, services, custom config files, and directions to run extra scripts at various stages of the installation. The link at the top of this post provides a pretty good overview and the documentation here is very helpful as well. That documentation provides almost all of the information you need so where details are excluded from the following look there.

In my specific case (an autoyast file for JBoss servers) the process went like this:

• Uploaded the latest versions of JBoss and Java (yast didn’t have 1.5), init.d scripts for JBoss, as well as our custom /etc/profile.d/environment.sh file to the installation server under a different directory accessible through FTP.
• Wrote a script meant to run after new installs to download and configure the above. Really just a bunch of wgets, copying, linking, chmod/chown changes. This was going to be downloaded and run in the init-scripts stage of the autoyast install.
• Setup a fresh install of openSUSE exactly as I wanted it for a JBoss server and ran yast2 autoyast from the command line.
• Selected Tools -> Create Reference Profile
• Selected the areas I cared about including. Note that selections here are in addition to a default set of information that includes partitioning and installed packages. In my case Firewall, Online Update Config (I enabled this on the reference server), Local Security, and User Management made sense.
• Next was to add a custom sshd_config file. With the reference profile loaded, went to Miscellaneous -> Complete Configuration Files and then alt-E for configure.
• Alt-w for new, file path of /etc/ssh/sshd_config for the new installs, and the loaded the contents of my existing sshd_config file for the contents.
• Lastly, I wanted to run the script I mentioned above as an init-script. These are scripts which run after installation is complete and networking is functional on a new server. init-scripts cannot be configured through the autoyast tool so I did File -> Save As and generated my baseline autoyast file.
• If you see warnings about the format of the generated xml file (the autoyast control file) ignore them. The Suse team has issues with their schema files.
• Finally, I edited the autoyast file and added my init-script to the end. It looked like this:
  
<scripts>
  <init-scripts config:type="list">
    <script>
      <location>ftp://myserver/myscript.sh</location>
      <interpreter>shell</interpreter>
    </script>
  </init-scripts>
</scripts>

  
• Then I just uploaded this file to the same FTP server so it was accessible during new installs.

Though the number of steps I just listed seems long, these autoyast files are really very quick to make. You could create any number of them for different machine roles and make them all available for new installs.

Setting up a New Server

Now that you have an installation server (FTP-based in this specific case) and all the autoyast files and other resources a new machine could need, you can setup a new machine from scratch by doing the following:

• Boot from the openSUSE DVD, CD1, or minimal installation CD. With some more work you can setup a bootable usb key or use the PXE boot capability of newer machines to boot from a network resource.
• Once you see the installation menu, Hit F4, enter your FTP installation server and the /sources/suse-10.2-64bit/CD1 directory, press enter.
• Move the cursor over the Installation option and type autoyast=ftp://[installserver]/[autoyast-file]. What you type appears in the command line options along the bottom of the screen.
• Press enter and walk away from the machine for awhile so the installation can complete.

Now, when I set this up, GRUB wouldn’t boot the newly installed machine. It turned out that the kernel version I was running on the reference server (and from which I generated the initial autoyast file) was different from the kernel provided by the installation server. This meant in my autoyast file the GRUB configuration portion was trying to reference a file (vmlinuz-2.6.18.2-34-default) that didn’t exist. So make sure your installation server is tied to the official repositories and make sure your reference machine is fully up to date before creating the baseline autoyast file.

I used this same approach to create configurations for JBoss, e-mail, and basic openSUSE-based servers.

At work we have been pulling various infrastructure tasks into development sprints to prep for some larger clients in the near future. I snagged the task of researching and setting up load balancing equipment and want to share the experience. Perhaps my google searching abilities are just not strong enough, but I had real difficulty finding current, meaningfully deep discussion or comparison about the hardware load balancing products available. I did find Load Balancing Digest to be pretty helpful for general information and introduction. This post is just a record of my relatively shallow and unqualified experience. I would really love to hear any comments, feedback, or opinions.

Options

In my searching I encountered a pretty clean separation of product categories. There are devices that cost less than $5k each and then devices that cost more than $10k. From my limited research it seems the 5-figure devices were presented as “appliances” and “platforms” that were full of features I really didn’t need. I just wanted a load balancer not a firewall + router + load balancer + ssl accelerator + whatever else all in one package. We also don’t anticipate needing the connection counts and throughput abilities of some of these more expensive products for a long while so the sub $5k market suited our situation just fine.

I spent some time looking at the following vendor’s spec sheets:

LoadBalancer.org
Coyote Point
Barracuda
CAI Networks

The fact that all of the various spec sheets offered different fields combined with my inability to find very much meaningful discussion/comparison online caused by decision to be weighted heavily by the small pieces of information I did find. The CAI Networks WebMux products were spoken highly of in several forums, their low end device had specs more than satisfying our requirements, they support replicated pairs, and they had more capable products should we need to upgrade in the future. So I contacted AVANU, one of the resellers listed on the CAI Networks website, and had an evaluation unit of the WebMux 481S shipped over free of charge. It arrived in 3 days.

Setup

Once we got it in the office I was able to set it up in 20 - 30 minutes. The documentation is reasonable and it is based on Linux Virtual Server (as are many of the load balancing products out there) so the documentation for that project can be consulted for details that the Webmux documentation leaves out about scheduling methods or terminology.

I went with the Out-of-Path Mode configuration described in the manual and we did layer 4 least connections persistent scheduling. Our servers kept their existing IPs and all clients are sent to the farm IP setup in the WebMux. The manual suggests adding a loopback to the machines involved in your cluster using iptables but I instead setup a loopback alias with # /sbin/ifconfig lo:1 [farm IP] netmask 255.255.255.255 up.

So literally the complete configuration involved only the following:

• Power up the WebMux and connect its server LAN port to our switch
• Follow the Common Configuration instructions in the manual for initial setup
• Use the web configuration panel to add a farm and assign it an IP
• Use the same panel to add the servers to the farm
• Login to each server and setup the loopback alias to the farm IP
• Done

We have not yet taken the step of using the JBoss Cluster capabilities so on the software end configuration was straight forward. The WebMux supports multiple farms as well so you could use the same device to cluster other services (SMTP, DB) behind the web servers.

After setup I fired up JBoss on all of the involved servers, used a load testing tool (WebLoad) to send a ton of fake users to the farm IP and watched the WebMux web panel to verify connections were appearing evenly across the machines. Everything worked perfectly.

Of course all of this was done in our development environment. Once I have actual experience with the machines in production maybe I’ll post again with more informative content.