What you can’t be afraid of to accomplish this

1. Forcefully breaking RPM dependencies (will be resolved upon completion)
2. Ignoring package checksum mismatches
3. Rebooting

New version of RPM
Seeing how its not possible to install EL6 RPM’s with EL5′s stock version of RPM, the first thing to do is to get that newer version of RPM installed on your system. Keith Chambers, a friend and colleague of mine, recompiled RPM 4.6 against glibc 2.5 so that it would work with the current EL5 system. RPM 4.8 may have worked as well, but 4.6 worked for me, and I only needed to use this version during the big upgrade. RPM will replace itself with the stock EL6 version later on. For dependencies, I needed to include xz-libs and lua. I force-upgraded these packages (–nodeps). What you want to end up with is:

# rpm --version
RPM version 4.6.0

Patch YUM
Since the new RPM’s coming from the repo will be verified with a newer hashing algorithm than what YUM/python2.4 will recognize, I needed to make a few quick patches to YUM itself to force it to ignore conflicts. These ghetto-hack patches will be automatically removed from the system when YUM upgrades itself, so really this is only needed one time.

diff -Nur /usr/lib/python2.4/site-packages/yum.orig/depsolve.py /usr/lib/python2.4/site-packages/yum/depsolve.py
--- /usr/lib/python2.4/site-packages/yum.orig/depsolve.py       2011-08-19 15:07:02.000000000 +0000
+++ /usr/lib/python2.4/site-packages/yum/depsolve.py    2012-01-07 20:43:21.634917883 +0000
@@ -142,6 +142,7 @@
                             'repackage': rpm.RPMTRANS_FLAG_REPACKAGE}

         self._ts.setFlags(0) # reset everything.
+        self._ts.addTsFlag(rpm.RPMTRANS_FLAG_NOMD5)

         for flag in self.conf.tsflags:
             if ts_flags_to_rpm.has_key(flag):
diff -Nur /usr/lib/python2.4/site-packages/yum.orig/__init__.py /usr/lib/python2.4/site-packages/yum/__init__.py
--- /usr/lib/python2.4/site-packages/yum.orig/__init__.py       2011-08-19 15:07:02.000000000 +0000
+++ /usr/lib/python2.4/site-packages/yum/__init__.py    2012-01-07 20:43:01.074133746 +0000
@@ -1217,6 +1217,7 @@
                 failed = True

+        failed = False
         if failed:
             # if the file is wrong AND it is >= what we expected then it
             # can't be redeemed. If we can, kill it and start over fresh
diff -Nur /usr/lib/python2.4/site-packages/yum.orig/yumRepo.py /usr/lib/python2.4/site-packages/yum/yumRepo.py
--- /usr/lib/python2.4/site-packages/yum.orig/yumRepo.py        2011-08-19 15:07:02.000000000 +0000
+++ /usr/lib/python2.4/site-packages/yum/yumRepo.py     2012-01-07 20:43:09.834466442 +0000
@@ -1467,6 +1467,7 @@
             file = fn.filename
         else:
             file = fn
+        return 1

         try:
             l_csum = self._checksum(r_ctype, file) # get the local checksum

I wrote a few quick lines in SED to handle this patching for me:

sed -i '/^        if failed:/i\        failed = False' /usr/lib/python2.4/site-packages/yum/__init__.py
sed -i '/^            file = fn$/a\        return 1' /usr/lib/python2.4/site-packages/yum/yumRepo.py
sed -i '/# reset everything.$/a\        self._ts.addTsFlag(rpm.RPMTRANS_FLAG_NOMD5)' /usr/lib/python2.4/site-packages/yum/depsolve.py

Package Conflicts
There are a number of packages while performing an upgrade from EL5 to EL6 that will cause you trouble, as some of the version numbers have actually decremented, and some cause dependency resolution issues that will be solved only after the upgrade is complete. The following is a small snippet that I used to force my way out of this situation:

declare -ra FORCEREMOVE=(
    m2crypto centos-release newt authconfig prelink tcp_wrappers sgpio
    iscsi-initiator-utils mkinitrd dmraid dmraid-events hmaccalc sysfsutils
    device-mapper device-mapper-multipath device-mapper-event
    vmware-open-vm-tools-common vmware-open-vm-tools-kmod less usermode
    libhugetlbfs lvm2 kpartx e4fsprogs-libs glib libsysfs
)
for PKG in ${FORCEREMOVE[@]}; do
    rpm -e --nodeps ${PKG}
done

PAM authentication work-around
By default, the RPM’s that install PAM do not clean out its include directory (/etc/pam.d). If the authentication modules you are using change at all (new ones added, old ones removed), you will need to manually clean out this directory. I found it easiest to just remove all of the configs and let them be re-populated by the new RPM’s. Without this step, after the upgrade completed, I was unable to log in to the console of the machine, because some authentication modules specified in the config files were now absent (no longer needed in the product I work on). If you have custom configs, you will need to account for them somehow, hopefully using Puppet or your configuration tool of choice.

rm -f /etc/pam.d/*

Re-install centos-release
You will notice in one of the steps above that I force-removed the centos-release package. The reason I needed to do this was because of the following:

package centos-release-5-7.el5.centos.x86_64 (which is newer than centos-release-6-0.el6.centos.5.x86_64) is already installed

By force-removing the centos-release package, and then performing a “yum install centos-release”, the centos-release package gets updated to the 6.x version.

Downgrade nss
This is a very important step. The version of “nss” actually decremented between EL5 and EL6. Since the new glibc 2.12 requires nss-softokn-freebl, which requires nss, we need to downgrade nss. This step should downgrade nss, install a few new packages, and update glibc. Once you do this, there is really no going back.

# yum downgrade nss
Loaded plugins: fastestmirror
Setting up Downgrade Process
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package nss.x86_64 0:3.12.7-2.el6 set to be updated
--> Processing Dependency: nss-softokn(x86-64) >= 3.12.7 for package: nss
--> Processing Dependency: nss-util >= 3.12.7 for package: nss
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.12.3)(64bit) for package: nss
--> Processing Dependency: nss-system-init for package: nss
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.12)(64bit) for package: nss
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.12.5)(64bit) for package: nss
--> Processing Dependency: libnssutil3.so()(64bit) for package: nss
---> Package nss.x86_64 0:3.12.8-4.el5_6 set to be erased
--> Running transaction check
---> Package nss-softokn.x86_64 0:3.12.8-1.el6_0 set to be updated
--> Processing Dependency: nss-softokn-freebl(x86-64) >= 3.12.8 for package: nss-softokn
---> Package nss-sysinit.x86_64 0:3.12.7-2.el6 set to be updated
---> Package nss-util.x86_64 0:3.12.8-1.el6_0 set to be updated
--> Running transaction check
---> Package nss-softokn-freebl.x86_64 0:3.12.8-1.el6_0 set to be updated
--> Processing Dependency: libc.so.6(GLIBC_2.7)(64bit) for package: nss-softokn-freebl
--> Running transaction check
---> Package glibc.x86_64 0:2.12-1.7.el6_0.5 set to be updated
--> Processing Dependency: glibc-common = 2.12-1.7.el6_0.5 for package: glibc
--> Running transaction check
---> Package glibc-common.x86_64 0:2.12-1.7.el6_0.5 set to be updated
--> Processing Conflict: glibc conflicts binutils < 2.19.51.0.10
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package binutils.x86_64 0:2.20.51.0.2-5.11.el6 set to be updated
--> Processing Conflict: glibc conflicts prelink < 0.4.2
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package prelink.x86_64 0:0.4.6-3.el6 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================
 Package                   Arch          Version                     Repository     Size
=========================================================================================
Updating:
 binutils                  x86_64        2.20.51.0.2-5.11.el6        base          2.8 M
 prelink                   x86_64        0.4.6-3.el6                 base          994 k
Downgrading:
 nss                       x86_64        3.12.7-2.el6                base          735 k
Installing for dependencies:
 nss-softokn               x86_64        3.12.8-1.el6_0              base          166 k
 nss-softokn-freebl        x86_64        3.12.8-1.el6_0              base          115 k
 nss-sysinit               x86_64        3.12.7-2.el6                base           26 k
 nss-util                  x86_64        3.12.8-1.el6_0              base           46 k
Updating for dependencies:
 glibc                     x86_64        2.12-1.7.el6_0.5            base          3.7 M
 glibc-common              x86_64        2.12-1.7.el6_0.5            base           14 M

Transaction Summary
=========================================================================================
Install       4 Package(s)
Upgrade       4 Package(s)
Remove        0 Package(s)
Reinstall     0 Package(s)
Downgrade     1 Package(s)

Once this has completed, you are clear to upgrade the rest of the system.

# yum -y upgrade

Re-install a few packages
A few of the packages that we force-removed for dependency resolution reasons will not get automatically installed. Therefore, we need to install them by hand:

declare -ra VMW_PKGS=(
    vmware-tools-core
    vmware-tools-foundation
    vmware-tools-guestlib
    vmware-tools-libraries-nox
    vmware-tools-plugins-guestInfo
    vmware-tools-plugins-vix
    vmware-tools-services
    vmware-tools-plugins-deployPkg
)
yum -y install ${VMW_TOOLS[@]} prelink lvm2 less

You can ignore the vmware packages if you are not using them.

Remove the old kernels
You should no longer need any of your EL5 kernels. You have hopped the fence and are in the land of EL6 now. You should be able to delete (rpm -e) any kernel-2.6.18* packages that are still installed.

Rebooting
Since I added all of the above logic in a script and executed it during a pre-install stage with Puppet, I had to have a way to automatically reboot the machines as well. In one of my previous posts, Puppet Self-Management, I detailed how to patiently wait for a puppet run to finish before executing some action from within a script. I applied this same technique to the reboot for my EL5 to EL6 upgrade script. Since I am executing my upgrade script during a puppet run, I do not want to bounce the machine before the run completes to avoid corrupting the state, and also to ensure that all of my other puppet-managed updates are applied before the machine boots in to EL6 for the first time. I accomplished this with the following code at the very end of my upgrade script:

/bin/sh -c "
    until [ ! -f /var/lib/puppet/state/puppetdlock ]
    do
        sleep 1
    done
    /sbin/shutdown -r now" &

Now then, first thing’s first: Connecting the thing to the PC. The Apple 27″ LED display that I purchased comes with *only* a mini displayport connection. No DVI, no HDMI. This actually didn’t bother me so much, since the mini displayport is present on a number of ATI cards. I set out to find a new graphics card. The first one I came across (and actually ended up purchasing alongside the monitor) was the Apple GeForce GT120. Yes, its an Apple product meant to be an upgrade to the Mac Pro’s from early 2009, but it was a PCI Express card, nothing out of the ordinary (probably just re-branded). Remembering back to my Hackintosh days, I figured that I probably had the best bet with this card. Apple makes it, so it should work right?

Wrong! While the DVI graphics worked just fine on an older monitor (Samsung P2350), the mini displayport would give me no video. I am guessing that Apple probes the card in some way that my PC does not. Also note, at this point I had still never seen the display even power on, since the new LED displays are controlled completely via the MDP, and thus there is not a single button on the entire monitor.

A little disgruntled, I removed the GT120 from my PC and moved on to the next solution: an ATI card with MDP. I found one at my local Fry’s Electronics, and ordered online for local pickup. It was the AMD HD 6870 XOC. All of the specifications looked correct on the site; 1x Dual-Link DVI, 1x Single-Link DVI, 1x HDMI, and 2x mini displayports. I picked the card up from the store and headed home for installation. To my surprise, when I pulled the card out of the box, it had 2x DVI ports and 1x displayport, not even a minidispalyport or any HDMI connections. It was identical to the item on the website in every other way, shape, and form, but the interface description on the left side of the box had a sticker over it with the downgraded specifications.

Needless to say, I had to return it. I didn’t want to run any displayport to mini-displayport adapters or any other nonsense like that.

I ended up finding the XFX version of the Radeon HD 6870 at MicroCenter for around the same price. This time I verified before walking out the door with the card.

While I was installing the card, I noticed the box sitting next to me mentioning the interface was PCI Express 2.1. I didn’t think anything of it, because I had done my homework beforehand and learned that any PCIE 2.1 card would work in a 2.0 slot.

To my surprise, after installing the card and connecting everything in my Gigabyte H55M-S2V, the card did not function at all! After further reading and research, I still did not find an answer as to why the card would not function in this particular motherboard. A BIOS update and still, nothing. Irritated and ready to just start using the darn monitor and stop fiddling with hardware, I went out and bought a new motherboard. My processor was LGA1156 based, so I had limited options. I ended up with an ASUS P7P55D-ELX. It had a few goodies that my old board didn’t, like 6GB/s SATA and USB 3.0. After further ripping apart/assembling my PC, I finally was able to boot up, and see the display work perfectly!

The BIOS POST showed up just fine, stretched to the size of the 2560×1440 native on the LED. Everything was all well and good. I did notice though, that the back light was looking pretty dim. I then realized that the factory default for the brightness setting is not set to the max, or even a reasonable brightness. It is in fact pretty dim, even with the ambient light detection/adjustment in the iSight. What I ended up doing to solve this was installing Windows 7, and on top of that, installing Boot Camp from a Snow Leopard DVD that I had. It is noteworthy though, that the version of Boot Camp that came on the Snow Leopard disk did not give me full access to the brightness control. Even with the slider all the way up to max, I still only had maybe 50% brightness. After running the Apple Software Update utility and getting the latest Boot Camp, and a subsequent reboot, I watched my Cinema Display come to life as I dragged the slider all the way up and the brightness began to burn my pupils!

After all of this fighting to get a Cinema Display to work sans the Mac, I would indeed say that it is worth it. After setting the brightness settings in Boot Camp, feel free to nuke your Windows installation and go back to using Ubuntu, Fedora, SuSE, or whatever it is you like using. The LED display will remember the settings internally and won’t need to set them again.

A note about the video card:
I found the XFX HD 6870 to be a bit noisy for my taste. Not overpowering noisy, but my tower sits right next to me on my desk and I don’t like the sound of fans for hours on end. Installing MSI Afterburner and setting an automatic fan profile like the following, the noise level is more than acceptable and the GPU remains relatively cool (~50-55C).

In conclusion: It is totally worth it. You should do it.

There are no dependencies on any external libraries. Pass in your storage, your string, and the number of columns to wrap to, and this will do the rest.

/*
 * This function will wrap large amounts of a text into a manageable and human-readable width
 * word by word. Just specify the number of columns you are working with and feed it a string,
 * and it will return a new string (including added line breaks) to accommodate the area you
 * are working with.
 *
 * {{{ proto( void ) wrap( char out, char str, int columns )
 */
void wrap( char *out, char *str, int columns )
{
    int len, n, w, wordlen=0, linepos=0, outlen=0;

    /*
     * Find length of string 'str' without using string.h
     */
    for( len=0; str[len]; ++len );

    /*
     * Allocate the full space of 'str' to 'word', so there is no possible way that the string
     * could contain a word that does not fit into the 'word' variable.
     */
    char word[len];

    /*
     * Loop through each individual character in the passed array (str) and detect white space
     * and word length to determine how to handle line wrapping.
     */
    for( n=0; n<=len; n++ )
    {
        /*
         * Detect spaces and newlines. We cannot gurantee that the passed string is null-
         * terminated, so we also need to handle cases where we reach the end of the string
         * without encountering wite space characters.
         */
        if( str[n] == ' ' || str[n] == '\n' || n == len )
        {
            /*
             * If the current word will not fit on the current line, add a newline.
             */
            if( linepos > columns )
            {
                out[outlen++] = '\n';
                linepos = wordlen;
            }

            /*
             * Append the found word to the output and reset the word character array in
             * preparation for accepting characters for the next word.
             */
            for( w=0; w<wordlen; w++ )
            {
                out[outlen++] = word[w];
                word[w] = '\0';
            }

            /*
             * If we reach the end of the string, add the null-terminator character.
             */
            if( n == len )
                out[outlen] = '\0';

            /*
             * If we encounter a newline character, append it to the string as usual, but
             * set the line position counter back to 0 (new line = start count from 0 again).
             */
            else if( str[n] == '\n' )
            {
                out[outlen] = str[n];
                linepos=0;
            }

            /*
             * If the word fits in the current line without trouble, just add the space.
             */
            else
            {
                out[outlen] = ' ';
                linepos++;
            }

            /*
             * Increment the final output length for the next loop, and set the word length
             * counter back to 0 (newline or space = new word).
             */
            outlen++;
            wordlen=0;
        }

        /*
         * If the current character is in the middle of a word somewhere, just append it and
         * move on, incrementing counters.
         */
        else
        {
            word[wordlen++] = str[n];
            linepos++;
        }
    }
}
/* }}} */

Examples

Wrap at 50 columns:

$ ./a.out
Lorem ipsum dolor sit amet, consectetur adipiscing
elit. Maecenas pulvinar blandit diam nec mattis.
Sed a ipsum nec ante porttitor feugiat. Morbi
ipsum lacus, dignissim at bibendum in, consectetur
eget ipsum. Donec dolor nibh, scelerisque ac
sodales et, posuere ac nulla. Aliquam eget
tincidunt ante. Vestibulum justo leo, congue ut
luctus ut, venenatis non dui. Cras sapien risus,
blandit at semper eu, cursus eu est. In hac
habitasse platea dictumst. Fusce libero dolor,
commodo nec rhoncus eu, rutrum vel arcu. Vivamus
ultrices faucibus tellus ac feugiat. Ut imperdiet
metus sed erat feugiat sed porta lorem tristique.
Mauris quis erat vel ante cursus pretium.
Curabitur arcu erat, consequat ac ornare et,
venenatis eu diam.

Curabitur posuere dui vitae tortor cursus cursus.
Pellentesque adipiscing lobortis sem at varius.
Mauris pellentesque sollicitudin ultricies.
Maecenas in tellus turpis. Integer convallis
mollis elit eu placerat. Nunc volutpat consectetur
facilisis. Curabitur eros arcu, dapibus ut
convallis non, rhoncus non magna.

Wrap at 20 columns:

$ ./a.out
Lorem ipsum dolor
sit amet,
consectetur
adipiscing elit.
Maecenas pulvinar
blandit diam nec
mattis. Sed a ipsum
nec ante porttitor
feugiat. Morbi ipsum
lacus, dignissim at
bibendum in,
consectetur eget
ipsum. Donec dolor
nibh, scelerisque ac
sodales et, posuere
ac nulla. Aliquam
eget tincidunt ante.
Vestibulum justo
leo, congue ut
luctus ut, venenatis
non dui. Cras sapien
risus, blandit at
semper eu, cursus eu
est. In hac
habitasse platea
dictumst. Fusce
libero dolor,
commodo nec rhoncus
eu, rutrum vel arcu.
Vivamus ultrices
faucibus tellus ac
feugiat. Ut
imperdiet metus sed
erat feugiat sed
porta lorem
tristique. Mauris
quis erat vel ante
cursus pretium.
Curabitur arcu erat,
consequat ac ornare
et, venenatis eu
diam.

Curabitur posuere
dui vitae tortor
cursus cursus.
Pellentesque
adipiscing lobortis
sem at varius.
Mauris pellentesque
sollicitudin
ultricies. Maecenas
in tellus turpis.
Integer convallis
mollis elit eu
placerat. Nunc
volutpat consectetur
facilisis. Curabitur
eros arcu, dapibus
ut convallis non,
rhoncus non magna.

*Update 11/24/2011: This solution is useful if you need to word wrap inside of a C program. If you are shell scripting, you can use the “fold” command, which comes standard with the “coreutils” package in EL-based distributions.

Usage: fold [OPTION]... [FILE]...
Wrap input lines in each FILE (standard input by default), writing to
standard output.

Mandatory arguments to long options are mandatory for short options too.
  -b, --bytes         count bytes rather than columns
  -c, --characters    count characters rather than columns
  -s, --spaces        break at spaces
  -w, --width=WIDTH   use WIDTH columns instead of 80
      --help     display this help and exit
      --version  output version information and exit

Report bugs to <bug-coreutils@gnu.org>.

The technique that I decided on was a dual-table model. The schema was laid out like so (for the purpose of this post, I’ve removed irrelevant tables of course):

properties
   |
   +-- key (varchar)
   |
   +-- value (varchar)

properties_c
   |
   +-- key (varchar)
   |
   +-- value (varchar)

You can see that they are both essentially the same exact table, just with a different name. The reason for this is because I am going to be using both tables as a single “view” of sorts.

The “properties” table will be the “default” values. This is the table we will modify during upgrades. Its contents are essentially unimportant, and can be emptied, re-populated, or otherwise modified with no concern of obstructing valuable user-entered data. My database initialization script simply drops this table if it exists, and re-creates it during every upgrade with fresh, new data.

The properties_c table is where all of the custom data goes. It is empty by default.

I have multiple tables applying this concept, so I keep their naming consistent with the suffix “_c” for custom data tables.

This is simply the layout. Things got interesting while writing the SELECT statements. I think that the concept of selecting one table “over” another table could be applied for other purposes as well, so I will share what I came up with for queries.

Let’s say that I have this example data:

properties

key                    value
--------------         ---------------------------
welcomemsg             Welcome to our website!
dateformat             mm dd, yyyy
enablelogin            false

properties_c

key                    value
--------------         ---------------------------
welcomemsg             Go Away
myproperty             test

I need to select all of this data, but always give priority to the “properties_c” table. If a key exists in properties_c, it should override the value set in “properties”. It should not matter if the key is not present in both tables, we want to get all of the properties every time. The result I am looking to get from my select would be:

key                    value
--------------         ---------------------------
welcomemsg             Go Away
dateformat             mm dd, yyyy
enablelogin            false
myproperty             test

The following query will give me those exact results:


SELECT key, value FROM properties_c UNION SELECT key, value FROM properties p WHERE NOT EXISTS (SELECT * FROM properties_c WHERE key = p.key ) ORDER BY key;


There are some interesting things going on in this query.

The part that took me a while to figure out was how to compare the “key” column inside of the sub-query. You will notice that I am defining an alias for the “properties” table, even though it is the only table I am selecting from. I am doing this because the sub-query will have access to to the key column by its alias, allowing me to compare the values between the two tables. The bigger take-away from this lesson is that sub-queries can access parent data via aliases, provided you do not re-define the same alias inside of the sub-query.

The second thing that is noteworthy in this query is the use of the “NOT EXISTS” statement. I am using this to check if properties_c contains a row with a key of the same name as the properties table. If it does exist, the row from properties is ignored, otherwise, it is selected. This is my “default value” logic, and so far it seems to be working quite well.

The Caveat
One thing I can think of that might be undesirable in implementations other than the one I have would be deleting records. By deleting a record from properties_c, you aren’t really deleting, you are “resetting to default” because the default value still exists in the “properties” table. The only way I can think of solving this would be by adding a third column to the “properties” table with a boolean value that would indicate whether or not the value should be displayed. A more hack-ish way, but also quicker, would be to define some constant that would indicate the value should not be selected from either table. The down side of doing that would be that you are now limiting the possible values the user can enter.

I’ll be adding more to this post when the project is completed.

The problem is this: Puppet begins to run, it updates its config file, and then notifies the service to restart due to the change. The service restart will not necessarily be the very last thing to happen during the puppet run, so if you are in the middle of a run and the service restarts, you are most likely going to be in a bit of trouble. Typically what happens is that the running process gets killed prematurely, causing puppet’s state file to become corrupted. This isn’t the end of the world. However, if the lock file did not get removed as part of the service shutdown, you might be in slightly more serious trouble. Normally at this point, you would probably need to log in to the machine via SSH to correct the issue. Not a huge deal, unless you are managing a large number of systems.

All of this trouble can be avoided. At some point, maybe puppetd will be able to catch the SIGHUP correctly and handle this whole thing gracefully on its own. Until that day comes, I have come up with a small script that will help avoid this.

Essentially, what this script does is fork an “until” loop. The loop will check if the “puppetdlock” file exists, which would indicate that a puppet run is in progress. If it does, the loop will go back to sleep for a second and then try again. This is repeated until the file goes away, indicating the whatever was running is now done. At this point it would be safe to do a puppet restart, so the loop ends and the restart is then carried out.

This of course will not give you an accurate “puppet restarted successfully” message, because the only thing really being tested here is the success of the fork operation. You must rely on centralized logging or similar to catch the unlikely puppet service failure.

#!/bin/bash
# File Name:     restart-puppetd.sh
# Author:        Ryan Uber <ryan@blankbmx.com>
#
# Description:   This script is a hack! However, it solves a very important
#                issue with puppet. Normally, if you subscribe the puppet
#                service to the puppet.conf file, the puppet service will
#                be restarted too soon, interrupting the current puppet
#                run. Various attempts at using configure_delayed_restart
#                among other things have not proven to be 100% effective.
#                This script will watch the puppetdlock file, which can
#                determine whether or not there is a run in progress. If
#                there is a run in progress, we sleep for a second and then
#                test again until the process is unlocked. Once unlocked, we
#                can safely call a puppet restart. The checker process
#                itself gets forked into the background. If it were not
#                forked into the background, the puppet run would sit and
#                wait for the process to return, or for the exec timeout,
#                whichever came first. This would cause serious trouble if
#                timeouts were disabled or very long periods of time.

# Begin waiting for the current puppet run to finish, then restart.
/bin/sh -c "
    until [ ! -f /var/lib/puppet/state/puppetdlock ]
    do
        sleep 1
    done
    /sbin/service puppet restart" &

# Always return true, since this script just forks another process.
exit 0

# EOF

I realized that not all of the files need the same permissions, and not all of them need the same owner or group. Most manifests, however, will contain 3 basic types: a package, a file, and a service, or “PFS“. These 3 types are related to the same thing, and most likely will have the same owner and the same group at the minimum. My thinking behind this is, you should only need to specify what user / group is going to own these files and folders one time, and have it apply to everything else.

To demonstrate, here is what a typical puppet manifest might look like, writing everything out, plain and simple:

file {

    "file_1":
        path    => "/path/to/file_1",
        owner   => "someuser",
        group   => "somegroup",
        mode    => 0644,
        content => "This is a test.\n";

    "file_2":
        path    => "/path/to/file_2",
        owner   => "someuser",
        group   => "somegroup",
        mode    => 0644,
        content => "This is another test.\n";
}

You can already see the patterns emerging even though we have only defined two files so far. So lets say we have 5 of these files. With the above method of defining the files / attributes, your manifests might become quite long, especially if you are defining more attributes that are common, like a require or maybe a notify.

The following example achieves the exact same effect as specifying the owner, group, and mode in each file definition, saving us 15 lines of duplicate definitions in just 5 file statements:

File {
    owner   => "someuser",
    group   => "somegroup",
    mode    => 0644
}

file {

    "file_1":
        path    => "/path/to/file_1",
        content => "This is a test.\n";

    "file_2":
        path    => "/path/to/file_2",
        content => "This is another test.\n";

    "file_3":
        path    => "/path/to/file_3",
        content => "This is test 3.\n";

    "file_4":
        path    => "/path/to/file_4",
        content => "This is test 4.\n";

    "file_5":
        path    => "/path/to/file_5",
        content => "This is test 5.\n";
}

Now things might get a little trickier. You obviously won’t have every file you manage owned by the same user, or the same group, or with the same permissions. Two things come into play here:

1. The fact that we are only defining the defaults, and
2. Scope

Since what we have specified for owner, group, and mode already are only the defaults, you can still define those attributes per-file. For instance:

File {
    owner   => "someuser",
    group   => "somegroup",
    mode    => 0644
}

file {

    "file_1":
        path    => "/path/to/file_1",
        content => "This is a test.\n";

    "file_2":
        path    => "/path/to/file_2",
        owner   => "someotheruser",
        content => "This is another test.\n";
}

In the above example, “file_1″ will get the defaults for all 3 attributes, and therefore be owned by “someuser:somegroup”. “file_2″, however, overrides the default owner, and will thus have ownership of “someotheruser:somegroup”.

Now for scope. Suppose I have an Apache class and a MySQL class. These should not have the same ownership. However, if I have defined the files, services, and other things related to each piece of software in separate classes, then I am in luck.

Global defaults can be defined per-class, and are inherited.

File {
    owner   => "root",
    group   => "root",
    mode    => 0700;
}

class "mysql"
{
    File {
        owner   => "mysql",
        group   => "mysql",
        mode    => 0644
    }

    file { "my.cnf":
        path    => "/etc/my.cnf",
        content => "This is a test.\n";
    }
}

class "httpd"
{
    File {
        owner   => "apache",
        group   => "apache",
        mode    => 0644
    }

    file { "httpd.conf":
        path    => "/etc/httpd/conf/httpd.conf",
        content => "This is another test.\n";
    }
}

You can see how the global defaults are defined at the top here, as “root:root” with “0700″ permissions. Then, within each class, new defaults are set for the files, and therefore, within the scope of that class only, all file statements get the class-specific permissions.

Use this technique throughout your manifests, and you will notice they will start to appear much more simplistic and organized while accomplishing the same result. Also keep in mind that this makes it infinitely easier to modify attributes at a wide scale without re-keying the modifications again and again. Do be warned, however, adding a new attribute to a defaults definition will affect many files, so be sure that it will not negatively impact any one particular item in your manifests.

Feedback would be greatly appreciated. I’m not really a kernel developer but would love to know if you have any suggestions.

The "consoleblank=" is a very handy little addition to the kernel boot arguments,
especially nowadays with virtual machines growing in number quite rapidly.

There is, generally speaking, no damageable screen attached to most Linux machines
these days. Since using "setterm" does not make customizations persistent, the only
way to disable screen blanking without this patch is by adding some script-fu
to /etc/rc.local or similar start script, with "setterm -blank N", for each TTY
you do not wish to activate a screensaver for.

Rather, let's add this functionality (which is already available in later versions
of the Linux kernel) as a configurable argument to the kernel boot line.

The "consoleblank" kernel argument will take an integer value, which will specify
the blanking timeout in seconds. Set it to "0" to disable screen blanking entirely.

Ryan R. Uber <ryan@blankbmx.com>

--- a/Documentation/kernel-parameters.txt	2011-02-08 23:30:42.758108332 +0000
+++ b/Documentation/kernel-parameters.txt	2011-02-08 23:31:14.309331657 +0000
@@ -425,6 +425,10 @@
 			switching to the matching ttyS device later.  The
 			options are the same as for ttyS, above.

+   consoleblank=   [KNL] The console blank (screen saver) timeout in
+           seconds. Defaults to 10*60 = 10mins. A value of 0
+           disables the blank timer.
+
 	cpcihp_generic=	[HW,PCI] Generic port I/O CompactPCI driver
 			Format:
 			<first_slot>,<last_slot>,<port>,<enum_bit>[,<debug>]
--- a/drivers/char/vt.c	2011-02-08 23:30:39.667988520 +0000
+++ b/drivers/char/vt.c	2011-02-09 00:16:56.715662063 +0000
@@ -171,8 +173,24 @@
 int console_blanked;

 static int vesa_blank_mode; /* 0:none 1:suspendV 2:suspendH 3:powerdown */
-static int blankinterval = 10*60*HZ;
 static int vesa_off_interval;
+static unsigned long blankinterval = 10*60;
+
+/*
+ *  Newer kernel releases use core_param here, as seen below:
+ *  core_param(consoleblank, blankinterval, int, 0444);
+ *  The EL5 kernel does not yet have the core_param() function,
+ *  so we backport by using module_param_named() instead, add
+ *  a call to __setup(), and provide a function to set the
+ *  proper value.
+ */
+static int __init consoleblank_config(char *val)
+{
+    blankinterval = simple_strtoul(val,NULL,0);
+    return 0;
+}
+__setup("consoleblank=", consoleblank_config);
+module_param_named(consoleblank, blankinterval, int, 0444);

 static DECLARE_WORK(console_work, console_callback, NULL);

@@ -1374,7 +1392,7 @@
 			update_attr(vc);
 			break;
 		case 9:	/* set blanking interval */
-			blankinterval = ((vc->vc_par[1] < 60) ? vc->vc_par[1] : 60) * 60 * HZ;
+			blankinterval = ((vc->vc_par[1] < 60) ? vc->vc_par[1] : 60) * 60;
 			poke_blanked_console();
 			break;
 		case 10: /* set bell frequency in Hz */
@@ -3374,7 +3392,7 @@
 		return; /* but leave console_blanked != 0 */

 	if (blankinterval) {
-		mod_timer(&console_timer, jiffies + blankinterval);
+		mod_timer(&console_timer, jiffies + (blankinterval * HZ));
 		blank_state = blank_normal_wait;
 	}

@@ -3408,7 +3426,7 @@
 static void blank_screen_t(unsigned long dummy)
 {
 	if (unlikely(!keventd_up())) {
-		mod_timer(&console_timer, jiffies + blankinterval);
+		mod_timer(&console_timer, jiffies + (blankinterval * HZ));
 		return;
 	}
 	blank_timer_expired = 1;
@@ -3438,7 +3456,7 @@
 	if (console_blanked)
 		unblank_screen();
 	else if (blankinterval) {
-		mod_timer(&console_timer, jiffies + blankinterval);
+		mod_timer(&console_timer, jiffies + (blankinterval * HZ));
 		blank_state = blank_normal_wait;
 	}
 }

• How are we going to keep track of our progress? For years, this has traditionally been done with an Excel sheet. However, this is not a good option because of sharing issues, particularly with file locking and corruption.
• How can every engineer see what is going on at all times? It can turn into a sad, sad day very quickly if two people log into the same system and start applying the same patches at the same time. The only way to do this using the old Excel sheets would be to do it through policy rather than through technology itself. For instance, you enforce a policy where every engineer _must_ open a fresh copy of the Excel document and check out “who’s working on what” before going to town on the server. He would then need to mark it as “in progress” or similar, save and exit so that the next person could open the Excel sheet from the share, and then go on his merry way. This is a huge waste of time, and far more clerical work than is necessary.

These concerns are solved quite easily with a little developer time, PHP, and jQuery. Using these technologies, you can slap an extremely useful web page together with minimal effort and amazing results in a justifiable amount of time.

Consider how much easier things would be if everyone had a dynamic, real-time view of everything going on in your project right in their browser. If employee “X” updates something, his update appears on employee “Y”‘s screen in seconds without him ever refreshing the page. He then knows what employee “X” is doing and can avoid stepping all over his toes and potentially breaking things.

Perhaps the most impressive thing to me about jQuery is how small and simple it actually is, but at the same time, what tremendous things it can achieve. Opening up the source to jQuery in your favorite editor will show you that it is actually just a bunch of plain old javascript. Easy to read? Maybe not, but very minimalistic at its core.

jQuery’s syntax is very intuitive as well. Let’s take a quick look at a simple little jQuery function:

function setUser ( server, user )
{
    $.ajax({
        url: 'update.php?type=user&srv='+server+'&user='+user,
        success: function(){
            document.getElementById('user'+server).innerHTML=user;
        },
        error: function(){alert('Failed to update user log');}
    });
}

In just those few lines of javascript code, I have made an AJAX call to the “update.php” script (which updated my database in the background) using GET, checked whether or not the operation succeeded, updated the page in real time if it did, and threw an error if it did not. Now anywhere in my application’s HTML, I can throw in something like:

<script type="text/javascript">setUser('serverIdHere','userNameHere');</script>

or like this:

<a href="javascript:void(0);" onclick="setUser('serverIdHere','userNameHere');">Click it!</a>

to update the last user who performed a certain action on the page. The “onclick” function in the hyperlink is more useful in this scenario, as it helps me instantly call the function each time I click it.

So there, in its simplest form, is an AJAX update to my back end database. Now the trickier part comes in: Although the information is already updated on my browser, how does that update get to everyone else’s browser?

I did this by means of an additional jQuery function. Depending on how your application is laid out, there are different ways of going about doing this. The easiest way, which is only effective for a very small number of dynamic fields, would be to use the “setInterval()” javascript function, making an AJAX call to another backend PHP script, maybe called “fetch.php” that would provide the data to your javascript, and then update the page in realtime using “document.getElementById(element).innerHTML”, or “.value”, or any number of other settable attributes in HTML.

The method mentioned above was my first approach at keeping my data up to date and real-time for all users. However, I ran into a problem with this. For each system we needed to update, I was running an AJAX call in the background every 20 seconds to get updated information. In my application, there were about 200 systems that were on any given page, so you can imagine that making 200 calls every 20 seconds to the web server in the background gets a little nutty. At times, the forms would stop allowing input until the javascript was finished executing, but sometimes that would not happen quickly enough for me to make my submission before the next event was queued, causing serious delays to the page updates.

While the page was technically working, the user experience was awful. I decided to try something a little more involved, and got outstanding results. Rather than querying individual items from my “fetch.php” script, I wanted to fetch everything in one run. To do that, I needed a way to get data from MySQL into an array that both PHP and jQuery could understand. XML seemed like the perfect fit.

Here is my PHP script to gather data and output XML:

<?php
header('Content-type: application/xml; charset=utf-8');
print "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n<DATA>\n";
require_once ( 'classes/DB.class.php' );
$DB = new DB;
$result = $DB->Query("SELECT * FROM servers;");
while ( $row = mysql_fetch_assoc ( $result ) )
{
    print <<<EOF
  <SERVER>
   <ID>{$row['serverId']}</ID>
   <STATUS>{$row['serverStatus']}</STATUS>
   <USER>{$row['user']}</USER>
   <NOTES>{$row['notes']}</NOTES>
  </SERVER>

EOF;
}
print "</DATA>";
?>

This would spit out everything I had in my database in XML format. Fortunately, jQuery had a very easy way of handling XML without needing any plugins or parsing anything manually. Here is what I ended up with to update the page in real time:

function refresh ( )
{
    $.ajax({
        url: 'fetch.php',
        dataType: 'xml',
        success: function(xml){
            $(xml).find('SERVER').each(function(){
                if ( document.getElementById($(this).find('ID').text()) )
                {
                    if ( $(this).find('STATUS').text() == 0 ) { var class = 'pending'; }
                    if ( $(this).find('STATUS').text() == 1 ) { var class = 'progress'; }
                    if ( $(this).find('STATUS').text() == 2 ) { var class = 'complete'; }
                    if ( $(this).find('STATUS').text() == 3 ) { var class = 'review'; }
                    document.getElementById($(this).find('ID').text()).setAttribute('class',class);
                    document.getElementById('user'+$(this).find('ID').text()).innerHTML=''+
                        $(this).find('USER').text();
                    document.getElementById('notes'+$(this).find('ID').text()).innerHTML=''+
                        $(this).find('NOTES').text();
                    document.getElementById('select'+$(this).find('ID').text()).selectedIndex=''+
                        $(this).find('STATUS').text();
                }
            });
        }
    });
}

The above code does a few things:

• Changes the background color of the affected row
• Updates the “User” column with the last username that updated the item
• Updates the select box that contains the status of the item (changes the item that is actually selected)
• Updates the “notes” column with fresh data

With all of these fields dynamic, what you end up with is essentially a web page that you and many others can all work on at the same time as if you were sharing a computer screen, but had your own mice and keyboards.

Since the above function cut down so much on the intensity of the web page (1 HTTP GET rather than 200), I was also able to set the update time to a lower value. At the end of my HTML document, I placed a line like this:

<script type="text/javascript">setInterval("refresh()", 10000 );</script>

This will call my refresh() javascript function every 10 seconds and update the page with new data.

I have left a lot out of this article for brevity’s sake. However, with a basic understanding of how Javascript works as well as PHP, you will likely be able to create such useful applications with little time investment.

Here is the description of the exercise, directly from the book:

Write a program entab that replaces strings of blanks by the minimum number of tabs and blanks to achieve the same spacing. Use the same tab stops as for detab*. When either a tab or a single blank would suffice to reach a tab stop, which should be given preference?

Some additional thoughts / potential TODO’s before you read the program:

• Should the tab-stop length (in spaces) be hard-set? Perhaps this would better be written as a command-line option for usability on programs and files where the programmer has set their editor to use, say, 4 spaces in place of a ‘\t’.
• Possibly add white space trimming to the end of the line, as in my previous getline.c program. This should likely happen before converting to tabbed output.
• Should the end-of-line white space trimming be in a function of its own? Perhaps there is already such a function available in string.h.

/*
 *  entab.c - Convert white space into tabbed output, being mindful that
 *            a tab stop is not simply a fixed number of consecutive
 *            white space.
 *
 *  Author: Ryan R. Uber <ryan@blankbmx.com>
 *  Date:   Fri Nov 19 04:39:29 CST 2010
 *
 */

#include <stdio.h>

#define MAXLINE 1000    /*  Maximum input per line */
#define TAB     8       /*  Tab stop interval */
#define TRUE    1       /*  Just symbolic names for readability. */
#define FALSE   0       /*  These could have just as effectively been
                            written as a 1 or 0 in the program */

void copy (char from, char to[]);
int getline (char line[], int lim);

/*  Replace white space with proper tabbing */
int main (void)
{
    int len, i, j, nspaces, stop;
    char line[MAXLINE], output[MAXLINE];

    nspaces = 0;

    while ((len = getline(line, MAXLINE)) > 0)
    {
        for (i = 0; i < len; ++i)
        {
            /*  This defines whether we are at a tab stop position. The
             *  division operation returns a zero if the quotient is an
             *  even number. As an example, try:
             *      ( 8 % 8 )
             *  in a separate C program. This would be the exact operation
             *  run if TAB is set to '8' and you are on the 8th character
             *  of input.
             */
            if ( i != 0 && i % TAB == 0 )
            {
                stop = TRUE;
            }

            else
            {
                stop = FALSE;
            }

            /*  We will not be adding any detected white space to our array
             *  if we are not currently at a tab stop. However, we need to
             *  keep track of the detected white space until to later determine
             *  if we need to replace it with the '\t' character.
             */
            if (line[i] == ' ' && stop == FALSE)
            {
                ++nspaces;
            }

            /*  The following tests that white space was detected all the way
             *  from the last non-white space up to the next tab stop. If this
             *  condition is true, we replace the white space with a tab stop.
             */
            else if (nspaces > 0 && stop == TRUE)
            {
                /*  Copy the '\t' character to the output array, and set our
                 *  space counter back to 0 in preparation for the next test.
                 */
                copy ('\t', output);
                nspaces = 0;

                /*  The current position we are at (i) has not yet had its
                 *  corresponding character added to the output array. We only
                 *  want to copy this character if it is not white space.
                 */
                if ( line[i] != ' ' )
                {
                    copy (line[i], output);
                }

                /*  As before, keep track of the white space if detected for
                 *  accurate white space replacement in the next tab stop.
                 */
                else
                {
                    ++nspaces;
                }
            }

            /*  If we have detected 1 or more white spaces that do not lead us
             *  all the way up until a tab stop, we need to add the actual white
             *  space to our output array so they are not lost.
             */
            else
            {
                /*  Add white space for each ' ' character counted */
                for (j = 0; j < nspaces; ++j)
                {
                    copy (' ', output);
                }

                /*  Set space count to 0 to count any remaining space before we
                 *  encounter the next tab stop. Copy non-white space characters.
                 */
                nspaces = 0;
                copy (line[i], output);
            }
        }
    }

    /*  Display the newly-formatted string and return */
    printf("%s", output);
    return 0;
}

/*  Append provided output to a designated character array */
void copy (char from, char to[])
{
    /*  Set character position to 0 */
    int pos;
    pos = 0;

    /*  Determine where we will append characters in the pre-existing array */
    while ( to[pos] != '\0' )
    {
        ++pos;
    }

    /*  Copy provided character to the last position in the array */
    to[pos] = from;
    to[pos+1] = '\0';
}

/*  Read input from stdin */
int getline (char s[], int lim)
{
    int c, i;

    /*  Read in characters until a newline '\n' is encountered */
    for (i = 0; i < lim-1 && (c=getchar()) != EOF && c != '\n'; ++i)
        s[i] = c;

    /*  If a newline was encountered, add to the array and increment counter */
    if (c == '\n')
    {
        s[i] = c;
        i++;
    }

    /*  Terminate input line, return number of characters found in the line */
    s[i] = '\0';
    return i;
}

/* EOF */