As pretty much every system administrator is now aware, a major security flaw was introduced to the Linux kernel in April of 2008, and just recently got some big exposure when a public exploit was published for it. Red Hat describes this vulnerability as:

an issue in the 32/64-bit compatibility layer implementation in the Linux kernel, versions 2.6.26-rc1 to 2.6.36-rc4. The compat_alloc_user_space() function is missing a sanity check on the length argument, and also a check to make sure the pointer to the block of memory in user-space that the process is attempting to write to is valid.

When panic struck, and systems began to be compromised, users of Red Hat Enterprise and similar rebuild-distributions such as CentOS remained vulnerable for some days. Red Hat is known for stability, and thus it makes perfect sense that they were not so quick to apply a patch, build an RPM, and push it to their mirrors. They tested their patch thoroughly.

However, some companies, like the one I work for, wanted a fix in place before an official release was available. Seeing how I build all of the RPM’s around here, I was tasked with patching the kernel into a distributable format for all of our managed customers.

I first went searching for any patches that already existed. I was not about to try writing my own and be held responsible for it. On September 19th, 2010, Roberto Yokota posted a patch on the Red Hat Bugzilla page. His patch does indeed work, as he demonstrates in his 3 posts from the 19th. However, his patch would not apply cleanly to the current RHEL kernel build, as it duplicated patches from previous releases. I consequently needed to modify his patch slightly to build cleanly into the kernel RPM’s that we released. Essentially all I needed to do was to remove all instances where “%eax” was replaced by “%rax”, as those bug fixes were included in other patches. I added my patch to the RPM (%patch25159):
(more…)

I had the sincere displeasure of troubleshooting an osCommerce site this morning. The user had lost their password, so naturally, I had to generate them a new one. I found their password hashing function burried in the “admin/includes/functions/password_funcs.php” script:

  function tep_encrypt_password($plain) {
    $password = '';

    for ($i=0; $i<10; $i++) {
      $password .= tep_rand();
    }

    $salt = substr(md5($password), 0, 2);

    $password = md5($salt . $plain) . ':' . $salt;

    return $password;
  }

Now then, I had a quick look in the database itself, at the “admin” table:

mysql> select * from administrators;
+----+-----------+-------------------------------------+
| id | user_name | user_password                       |
+----+-----------+-------------------------------------+
|  1 | admin     | 60a272d8be550ff714cf6bb385531d3a:6b |
+----+-----------+-------------------------------------+
1 row in set (0.00 sec)

Hmmm…. so its not a plain MD5sum. I was interested to see what they were doing beyond just MD5′ing the password so I actually went through what the code is doing.

Firstly, we call the function and pass it a plain-text password to encrypt:

tep_encrypt_password($plain)

Then, we are defining a variable called “$password” a random 10-character string:

    for ($i=0; $i<10; $i++) {
      $password .= tep_rand();
    }

Let’s just stop right there for a minute. We have two variables already: “$plain” and “$password”. The one called “plain” is actually the plain-text password, and the one called “password” is a random string. Excellent variable naming, osCommerce.

Moving on, we then use the random string, also known as “$password”, to get the “salt” for the password:

$salt = substr(md5($password), 0, 2);

Now let’s stop right there. A two-character salt? Essentially, this is just md5′ing the already-random “$password” variable, and snagging the first two characters of it.

Next, we do some flippity-floppying with variables:

    $password = md5($salt . $plain) . ':' . $salt;

The above is going to append the plain-text password to the two-character “salt” and MD5 it, the append a colon (:) character, then append the plain-text “salt” to the end of the string. The colon will likely be used for a preg_split or similar in a password checking function. What bugs me about the above code though, is that now we are reassigning the variable “$password” to the actual encrypted / salted version of the plain text password, rather than it just containing arbitrary characters as it did before.

Then we return the newly-encrypted password:

return $password;

That bugs me too, because we are not returning a password, we are returning an encrypted hash with a salt tailing it.

This is all a bunch of nit-picking, and the code actually does what it is supposed to do, but it is a big pet-peeve of mine because it makes the code a lot harder to follow, even due to something so small as a variable name.

Another thought, this actually does prove helpful to security of encrypted password storage. Of course anyone on the outside could still hit your login form 6 million times with their dictionary of commonly-used passwords, and if one matched they would be in. However, it does help in preventing someone who has obtained the hash from getting the actual password value from it. You wouldn’t be able to run osCommerce’s password hashes through an MD5 lookup and get passwords out of it.

So in conclusion, I might be too critical on the syntax, but this password encryption function actually does a decent job. I’d recommend doing something similar the next time you write a PHP application before just doing “INSERT INTO users ( $user_name, MD5($password);”.

One day a RAID issue came to my attention by means of a Nagios alert. I logged into the system to check out the 3ware array status, figuring a disk was likely going bad, needed to be replaced and the array rebuilt. However, to my horror and dismay, this is what I saw in tw_cli:

//server> show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c0    9650SE-4LPML 4         4        1       1       1       1      -        

//server> focus c0

//server/c0> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-10   DEGRADED       -       -       64K     279.377   Ri     ON     

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     DEVICE-ERROR     u0     139.73 GB   293046768     WD-WX20C7938667
p1     DEVICE-ERROR     u0     139.73 GB   293046768     WD-WX20C6901772
p2     DEVICE-ERROR     u0     139.73 GB   293046768     WD-WX20C7935022
p3     OK               u0     139.73 GB   293046768     WD-WXD0C7952295     

//server/c0>

A RAID 10 with 3 dying hard drives, and somehow, it was still online? This countered all of the logic I had come to know about RAID 10. How do you lose 3 disks and still be up and running? Two I can see happening if by coincidence two of the mirrored drives died, but this just doesn’t make any sense now, does it?

Notice the serial on each drive. All of them beginning with “WD-WX20C” died at the same time, while the one disk with “WD-WXD0C” is running just fine. Bad batch of drives? Possible, although the serials differ quite vastly among the “WX20C” drives.

This particular server was a miracle, no data was lost and we were able to get the array healthy again by taking the server offline, replacing one disk and rebuilding, 3 times.

The moral of the story? Monitor your RAID arrays if you like having usable data on your systems.