From August 2011 until today, this site has been running under WordPress. During this time, we have done many experiments with several other blog platforms, but none of them made it to the “import all the old stuff from this one - this is what we're looking for!” stage. As you may have already guessed, though, this is no longer the case. WordPress does what it does very well. However, the last post before this one was August… of 2014! That means that, every page served from this site, a script was run on the server that accessed a database and dynamically assembled the page. This content did not need to be dynamic - even when there is a new post, there is very little in the way of dynamic content here.

Jekyll is a static site generator; these types of applications generate static sites based on a collection of templates and content files, that can then be served with no backing data store. Now, we can utilize the blazing fast nginx web server to push these files as quick as people can request them, where the request does not even have to escape the process.

There will be more to come on Jekyll; there are at least two posts to be written, one on automating the build process and another on the migration from WordPress. Until then, though, there are redirects that ensure the RSS feeds for both the main blog and the xine RPMs require no changes, and the category pages have redirects as well. If something does not look right, let us know via either of the social media accounts linked above.

After a little over a year running on Tech Blog 2.0, you are now viewing version 3.0. For this version, we've returned to WordPress from BlogEngine. There are several issues that colluded to drive this change, most of which surrounded PHP and its crazy behavior. (Geeky details follow - skip to the paragraph starting with “Bottom line:” if you don't want the geek stuff. I bolded it so it would be easy to spot.)

PHP's recommended configuration is to run under Apache using the pre-fork multi-processing module (MPM). The advantage to this is that Apache does not have to spin off another process to handle each request; it handles it in the same thread. However, this means that each instance of the server must have all enabled modules loaded. This means that each instance of the server (AKA “thread”) is very large, so the number of threads run is lower (typically 5-15 in a server the size we're on). Also, this means that each thread can only handle one request at a time; if you have 7 threads configured, each serving one of 7 requests, and an 8th request comes it, it has to wait for one to finish. If the requests are served quickly, this may not be a problem; however, the avalanche of request that follow the typical front-page mention on mega-blogs can easily overwhelm it.

To fix this problem, there is another MPM, this one called worker. In this scenario, there are spare thread waiting to fill requests, and these can spawn other threads to do further work if required. So, the Apache threads would realize that a request needs to be handled by PHP, and pass it off to that process to be completed. The Apache memory footprint is much smaller; it serves the images, scripts, and other static files, and passes off the requests that require heavy lifting. PHP, then, has a (FastCGI) process where it receives these requests, processes them, and returns the response to the caller. Because each of these threads only has to load the PHP requirements, they are smaller too, so you can have more threads processing at the same time; you just might survive that front-page mention! (This is the same technique applied by LightTPD and Nginx, two other servers I tried at various times.)

It is in this scenario where PHP fails to live up to its expectations. These PHP processes would simply stop responding, but the controller thinks they're still there. The end result to the user is a site that just sits and waits for output that will never come. Eventually, they may receive a Gateway Timeout or Bad Gateway error. The problem is worse on slower sites, but even popular sites seemed to fall victim to this from time to time. This was also a problem whether PHP controlled its threads, or Apache controlled them.

The one thing that really perturbs me is instability. If something is broken, I can fix it; if it works, I can fix it 'til it's broke. :) But something that works sometimes, and other times doesn't, simply won't fly. I was able to introduce some stability by restarting the server 4 times a day, but that's a band-aid, not a long term solution. I was tired of fighting.

Bottom line: the configuration required for a stable server is in opposition to a lean-and-mean configuration. So, I installed the required Apache modules, and will continue to run my PHP-serving server at a configuration twice as large as it needs to be. I'll eventually move the Mono (.NET) processes to another machine, where the fast configuration won't cause stability problems.

But, PHP isn't all bad. While I would still heartily recommend BlogEngine.NET to someone who was going to serve the blog from a Windows machine, but I had some issues getting upgrades to go smoothly under Mono. It also is optimized for fast serving, at the expense of RAM. At this point, that's not the tradeoff we need.

Finally, with this update, the blog has received its first new theme. It's a clean, clear theme that should serve the content well. Plus, the social media icons up in the corner are just too cool, IMO. I've also applied tags to all posts except the “My Linux Adventure” series, and this theme displays them. (Comments are not here now, but will be migrated shortly.)

So, there you have it. Enjoy!

We've begun running Mono on some Bit Badger Solutions servers to enable us to support the .NET environment, in addition to the PHP environment most of our other applications use. While Ubuntu has nice packages (and Badgerports even brings brought them up to the latest release), one thing that we were missing was a “conf.d”-type of configuration; my “/applications=” clause of the command was getting really, really long. We decided to see if we could create something similar to Apache / Nginx's sites-available/sites-enabled paradigm, and we have succeeded!

To begin, you'll need to create the directories /etc/mono/fcgi/apps-available and /etc/mono/fcgi/apps-enabled. These directories will hold files that will be used define applications. The intent of these directories is to put the actual files in apps-available, then symlink the ones that are enabled from apps-enabled. These files have no name restrictions, but do not put an extra newline character in them. The script will concatenate the contents of that file to create the MONO_FCGI_APPLICATIONS environment variable, which tells the server what applications exist. (The syntax is the same as that for the “/applications=” clause - [domain]:[URL path]:[filesystem path].) Here's how the site you're reading now is configured (from the file djs-consulting.com.techblog.conf)…

techblog.djs-consulting.com:/:/path/to/install/base/for/this/site

Finally, what brings it all together is a shell script. This should be named “monoserve” and placed in /etc/init.d. (This borrows heavily from this script a script we found online, which we used until we wrote this one.) Note the group of variables surrounded by the “make changes here” notes - these are the values that are used in starting the server. They are at the top so that you can easily modify this for your own needs.

#/bin/bash

### BEGIN INIT INFO
# Provides:          monoserve.sh
# Required-Start:    $local_fs $syslog $remote_fs
# Required-Stop:     $local_fs $syslog $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start FastCGI Mono server with hosts
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/mono
NAME=monoserver
DESC=monoserver

## Begin -- MAKE CHANGES HERE --
PROGRAM=fastcgi-mono-server2 # The program which will be started
ADDRESS=127.0.0.1            # The address on which the server will listen
PORT=9001                    # The port on which the server will listen
USER=www-data                # The user under which the process will run
GROUP=$USER                  # The group under which the process will run
## End   -- MAKE CHANGES HERE --

# Determine the environment
MONOSERVER=$(which $PROGRAM)
MONOSERVER_PID=""
FCGI_CONFIG_DIR=/etc/mono/fcgi/apps-enabled

# Start up the Mono server
start_up() {
    get_pid
    if [ -z "$MONOSERVER_PID" ]; then
        echo "Configured Applications"
        echo "-----------------------"
        # Construct the application list if the configuration directory exists
        if [ -d $FCGI_CONFIG_DIR ]; then
            MONO_FCGI_APPLICATIONS=""
            for file in $( ls $FCGI_CONFIG_DIR ); do
                if [ "$MONO_FCGI_APPLICATIONS" != "" ]; then
                    MONO_FCGI_APPLICATIONS=$MONO_FCGI_APPLICATIONS,
                fi
                MONO_FCGI_APPLICATIONS=$MONO_FCGI_APPLICATIONS`cat $FCGI_CONFIG_DIR/$file`
            done
            export MONO_FCGI_APPLICATIONS
            echo -e ${MONO_FCGI_APPLICATIONS//,/"\n"}
        else
            echo "None (config directory $FCGI_CONFIG_DIR not found)"
        fi
        echo

        # Start the server
        start-stop-daemon -S -c $USER:$GROUP -x $MONOSERVER -- /socket=tcp:$ADDRESS:$PORT &
        echo "Mono FastCGI Server $PROGRAM started as $USER on $ADDRESS:$PORT"
    else
        echo "Mono FastCGI Server is already running - PID $MONOSERVER_PID"
    fi
}

# Shut down the Mono server
shut_down() {
    get_pid
    if [ -n "$MONOSERVER_PID" ]; then
        kill $MONOSERVER_PID
        echo "Mono FastCGI Server stopped"
    else
        echo "Mono FastCGI Server is not running"
    fi
}

# Refresh the PID
get_pid() {
    MONOSERVER_PID=$(ps auxf | grep $PROGRAM.exe | grep -v grep | awk '{print $2}')
}

case "$1" in
    start)
        start_up
    ;;
    stop)
        shut_down
    ;;
    restart|force-reload)
        shut_down
        start_up
    ;;
    status)
        get_pid
        if [ -z "$MONOSERVER_PID" ]; then
            echo "Mono FastCGI Server is not running"
        else
            echo "Mono FastCGI Server is running - PID $MONOSERVER_PID"
        fi
    ;;
    *)
        echo "Usage: monoserve (start|stop|restart|force-reload|status)"
    ;;
esac

exit 0

This needs to be owned by root and be executable (chmod +x monoserve). You can use update-rc.d monoserve defaults to set this to start at boot.

I was finally able to resolve my problems with Apache and MySQL. When I decided to mount my FAT32 drive under /home/summersd, I inadvertently caused myself some problems. From talking to a Linux guy at work, I found that no processes that weren't running under my user ID could access those files. The reason is that Linux looks up the entire directory tree, back to /, to determine if you can access the file. So, although I had -rwxrwxrwx summersd summersd on every file, /home/summersd was drwx------ summersd summersd, and /home was drwxr-xr-x. The permissions on /home/summersd were keeping Apache from seeing /home/summersd/drive_d/wwwroot, and MySQL from seeing or writing to /home/summersd/drive_d/mysql/data. I moved the drive to /mnt/drive_d, with the mount point being owned by “root”, still mounting the drive with my user name, and everything worked.

In the process of reconfiguring Thunderbird, I believe I may have found out how to share the address book across operating systems. The file ~/.thunderbird/default.[something]/prefs.js has a listing of all the preferences and settings. I modified this file to change the location of my mail files, and there is a setting there for an address book (which isn't shown in the configuration dialog - after all, it is 0.7.3…) I'll play with that later - right now I'm just elated to have Apache and MySQL working again.

At work, we use an editor called Visual SlickEdit (VSlick). It's got a lot of features, and supports color-coding for many different languages. I decided that I'd give wine another shot, as we only have the Windows version of this program. I installed wine and winesetuptk, used winesetuptk to configure the installation, then ran the installation program. Everything installed, and the program ran up to a point, when it started complaining about a missing DLL. I booted to WXP, found the DLL, copied it to the FAT32 drive, rebooted to Linux, and copied the DLL into the “fake windows” system directory. Soon, it was working great! I can't believe it - success with wine!

I also have made little headway towards getting Apache and MySQL to working. I changed the process that Apache uses to run as “summersd”, and I was able to see pages (although any pages that relied on a database didn't work). I still haven't figured this one out yet…

I'm still getting kernel panics from time to time, and it seems to be whenever I access networking. A suggestion from one of the folks on the WBEL users list was to download the Ultimate Boot CD, filled with diagnostic programs. I downloaded it, burned it, and ran some memory checks. Those checked out, so I'm going to run a “CPU Burn-In” program to see if it can detect errors from the CPU. It runs for up to 7 days, but I think I'll just run it overnight - folding@home didn't take nearly that long to crash it before.

Well, the last month has been interesting. I was able to get my Windows and Linux installations synchronized by creating a mount point for my second drive under /mnt/drive_d. Under that, I created a directory called /thunderbird for my e-mail, and moved my e-mail and newsgroup folders over there. (The first time, I missed the “newsrc” file, which is important - it tells what newsgroups you've subscribed to and which messages you've read.) Under Windows, I pointed it to D:\thunderbird\pop3.knology.net, and under Linux, it was configured to /mnt/drive_d/thunderbird/pop3.knology.net. I then moved the wwwroot directory from C:\Inetpub to drive D:, and pointed IIS to the new location. Under Linux, I did something a little different. As “root”, I deleted the directory /var/www/html, and instead created /var/www/html as a symbolic link to /mnt/drive_d/wwwroot (the actual command is ln -s /mnt/drive_d/wwwroot /var/www/html). That worked great as well.

MySQL was more complicated, but I was eventually able to get it working as well. I created the directory D:\mysql\data for the data, then configured /etc/my.cnf under Linux to look at /mnt/drive_d/mysql/data. I kept getting “Could not connect to server using socket /var/lib/mysql/mysql.sock”. After some digging, it appeared to be a permissions problem. All the documentation said that the default socket was /tmp/mysql.sock, so I changed my.cnf to point there instead, restarted mysqld, and it worked! So, I have no idea what a Unix socket it, but I know that now I have one! :)

I was also able to get DVDs playing using xine, compiling it myself, and using libdvdcss, I can even watch commercial DVDs. I'm really impressed with xine - it handles all kinds of media out of the box, including DivX and up to version 8 of WMV files. You can add codecs to it as well, to support almost anything you want to do from an audio or video perspective. Compiling the player took around 20 minutes, and compiling the front end took another 5. And, it was simple - download the .tar.gz file, do tar xvfz [name].tar.gz, cd [name], ./configure, make install. The ./configure script is the key in the whole process - it looks at what you have installed, and creates make files that will work with your compiler.

Everything started going south, though, when I started having freezes. Eventually, I got to where I could not boot without a kernel panic, and then boot errors (which I detailed in this e-mail to the WBEL user's list. Encouraged by my success over the past month, I decided to return to WBEL - it's supposed to be more stable than FC2, and I bet that I can get ndiswrapper, the dual-booting web server, the common e-mail, and maybe even some other stuff working again.