Tuesday, March 13, 2012

Incorrect string value: ‘\xCE\x94F508…’ for column ‘value’ at row 1

Problem

Last week I was working on importing data from a vendors database into our database and for, what seemed like, random records I’d receive the error message above. Upon further inspection it turned out that the records that weren’t able to be inserted had special characters, like 'Diagnostic, ΔF508 and R117H with 5T/9T'. I knew I had set our MySQL database to use UTF-8 for the collation of the server by default, I’d even posted about it here.

I was able to resolve this, but first just a little background.

Solution

After researching this and even exporting and importing my database, I found that my tables were being created specifically using the latin1 character set. I was able to determine this when I did MySQL dump of my database and examined the CREATE TABLE syntax, as shown below:

DROP TABLE IF EXISTS `addresstypes`; 
CREATE TABLE `addresstypes` 
( `id` int(11) NOT NULL AUTO_INCREMENT, 
`text` varchar(255) DEFAULT '', 
`createdat` datetime DEFAULT NULL, 
`updatedat` datetime DEFAULT NULL, 
`deletedat` datetime DEFAULT NULL, 
`createdby` varchar(25) DEFAULT NULL, 
`updatedby` varchar(25) DEFAULT NULL, 
`deletedby` varchar(25) DEFAULT NULL, 
`rixid` varchar(255) DEFAULT NULL, 
PRIMARY KEY (`id`) 
) ENGINE=MyISAM AUTO_INCREMENT=6 DEFAULT CHARSET=latin1;

I found three ways to correct this. Here are the first two ways as I found at this blog post.

Single Table

Use the following SQL code block and replace the “database_name” and “table_name” values with your actual database name and table name:

ALTER TABLE `database_name`.`table_name` 
CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;

Multiple Tables

Use the following SQL code block and replace the “database_name” with your actual database name:

SELECT CONCAT('ALTER TABLE `', t.`TABLE_SCHEMA`, '`.`', t.`TABLE_NAME`, '` 
CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;') as stmt 
FROM `information_schema`.`TABLES` t 
WHERE 1 AND t.`TABLE_SCHEMA` = 'database_name' ORDER BY 1

I also found that if I performed a MySQL dump of the database, then replaced the latin1 character set command with utf8, deleted the database and then re-imported it this resolved the issue as well.

Use the following code block and replace the “database_name” with your actual database name. You’ll also need to DROP or DELETE the database after you do the MySQL dump in the first step:

mysqldump -u root -p database_name > database_name.sql 
sed 's/utf8_bin/utf8_unicode_ci/' database_name.sql > database_name_fix.sql 
sed 's/CHARSET=latin1;/CHARSET=utf8;/' database_name_fix.sql > database_name.sql 
mysql -u root -p database_name < database_name.sql

Interesting Note

Interestingly if I copied and pasted the SQL INSERT syntax into the MySQL command-line, before I changed the tables to be UTF-8, the command was successful at inserting the data.

Update: MySQL 5.5.3+ Server Options

These options have changed in MySQL 5.5.3 and newer, in particular the "default-collation" and "default-character-set" server options have been deprecated. Instead, the following server options replace these "collation-server" and "character-set-server"

Wednesday, January 11, 2012

Move Mac OS X Users folder to secondary hard drive

I recently purchased a new 27" iMac and had it built with both an internal SSD disk and internal HD disk. I wanted to keep all of my applications and OS on the SSD while my user folders on the HD since it has so much more capacity. Here's how I was able to do this:

Restart Mac to Recovery Partition, use Terminal to copy Users folder to secondary drive and then add symlink on primary drive to secondary drive. In my directions below you need to aware of the following:

  • The primary disk drive was renamed to “Macintosh SSD”
  • The secondary disk drive was renamed to “Macintosh HD”
  • My user account short name is troym
  • My user account was the first one created on the system, so the system ID is 501

OK, so here’s the steps that I performed to complete this task:

  • Restart Mac, holding down Command + R on the keyboard (I had to do this four times before it worked)
  • Select your language
  • Click on the “Utilities” menu
  • Click on the “Terminal” menu option
ditto /Volumes/Macintosh\ SSD/Users /Volumes/Macintosh\ HD/Users 
rm -rf /Users
ln -s /Volumes/Macintosh\ HD/Users /Volumes/Macintosh\ SSD/Users
cd /Volumes/Macintosh\ HD/Users
chown -R 501:staff troym

Thursday, January 05, 2012

Updated Twitter Bootstrap OmniGraffle Stencil

I updated my Twitter Bootstrap CSS ToolKit OmniGraffle stencil with a number of changes.

  • Complete replaced the tables
  • Added helper text below form fields
  • Added a stacked-form section
  • Added a radio button control
  • Made sure all form labels are right-aligned
  • Set all stacked-form labels are left-aligned
  • Added a date range control
  • Increased the size of the textarea
  • Resized the navigation bar to be 940px wide
  • Grouped the pagination control

You can grab the stencil now from my GitHub repo, along with directions on how to update to the latest stencil if you're using the previous version.

Thursday, December 22, 2011

Migrating from Subversion (SVN) to Git

Requirements

You’ll need to have the following to perform this operation.

  • Access to the Subversion repository
  • Access to the git repository
  • The git command-line client installed
  • The subversion command-line client installed

Server Configuration

Now we need to create a directory on the server where we’ll host the git repository

sudo mkdir -p /var/www/project-name.git

Now let’s initialize the git repository in this directory

cd /var/www/project-name.git
sudo git --bare init

Now we need to tell this repository to point to itself. This is needed so when we clone it to our local system we’ll be able to push back to it.

sudo git remote add origin http://git.yourdomain.com/project-name.git/

Last we need to tell this repository that it’s a server, so it configures itself as such

sudo git update-server-info

Client Preparation

The first thing we need to do is create a list of the authors that are located in the Subversion repository. This means more then just the active users, but ALL authors that have ever committed to the repository.

The file format will be as follows:

subversion-author-name = first-name last-name <email>

We can use the subversion log command to create an XML file that will include a list of all of the subversion authors that have committed to your repository. You can do this using this command (on a machine that has grep, sort and perl):

svn log --xml | grep author | sort -u | perl -pe 's/.>(.?)<./$1 = /'

You’ll need to then add the corresponding information for git in the file format specified earlier.

Client Configuration

Now we’ll clone the Subversion repository to a local repository on the client. Replace the URL to your subversion repository in the command below:

git svn clone http://svn.yourdomain.com/path/to/svn-project project-name.git --authors-file=gitusers --no-metadata 

Now we need to remove the Subversion information from the configuration, since we won’t be using it anymore.

nano project-name.git/.git/config

In this file remove the section under [svn-remote “svn”]

Now let’s add our server repository to this local repository as a remote repository that it can work with:

cd cd project-name.git
git remote add origin http://git.yourdomain.com/projects/project-name.git/
git update-server-info

Last but not least we want to push our local repository to the server. We do this using the following command:

git push origin --all

Friday, December 16, 2011

Twitter Bootstrap Omnigraffle Stencil

I often use the Twitter Bootstrap CSS ToolKit for my web applications.  While this toolkit helps to make things quickly look great, I wanted to be able to use it in OmniGraffle.  I found a few Photoshop files that had the elements, but none for OmniGraffle, so I decided to try creating one.  This is my first OmniGraffle stencil, but I'm pretty proud of how well it came out.  Thanks to the OmniGroup for such a great product.

You can grab the stencil now from my GitHub repo.

I've submitted the stencil to Graffletopia, but still waiting for their review and approval.

Semantic Versioning

I really like this suggestion on software version formatting and incrementing taken from http://semver.org/
Consider a version format of X.Y.Z (Major.Minor.Patch). Bug fixes not affecting the API increment the patch version, backwards compatible API additions/changes increment the minor version, and backwards incompatible API changes increment the major version.

Thursday, December 15, 2011

Installing GitLabHQ on Ubuntu Server 11.10

Download the latest server updates

sudo apt-get update
sudo apt-get dist-upgrade -y

Dedicated gitlabhq user account

Next we need to create a dedicated gitlabhq user account to run the application, set a password for this account and add it to the admin group so it can perform root actions.
sudo useradd -s /bin/bash -m -G admin gitlabhq
sudo passwd gitlabhq
Now login as the gitlabhq user we just created. When prompted to accept the authenticity of the RSA key fingerprint type “yes”
ssh gitlabhq@localhost

Install Additional Packages

Now we’ll install the git version control system so we can clone repositories and setup the system. We’ll also install the postfix SMTP system so GitLabHQ can send emails to users.
sudo aptitude install git-core postfix -y
Now configure Git with some global variables that will be used when gitlabhq performs a git push operation. You can change the name and email address below if you wish:
git config --global user.email "admin@local.host"
git config --global user.name "GitLabHQ Admin User"

Generate SSH Keys

The GitLabHQ user will use SSH keys for login and authentication with the git user we’ll create later. So let’s generate our keys. Make sure to do the following: When prompted for the file in which to save the file, press Enter When prompted for a passphrase, press Enter When prompted to confirm the passphrase again, press Enter
ssh-keygen -t rsa

Prepare To Install GitLabHQ

First let’s clone the GitLabHQ installer scripts to help automate the installation
cd ~
git clone https://github.com/gitlabhq/gitlabhq_install.git
Now we’ll run the scripts to install any additional packages. Run the command below and select “Y” to confirm you want to install the packages.
cd ~ 
gitlabhq_install/ubuntu_packages.sh
Now we’ll run the script to install the ruby language.
cd ~ 
gitlabhq_install/ubuntu_ruby.sh
Now we’ll run the script to install the gitolite program. This creates a new user “git” on the system, and will store our repositories under this accounts home directory.
cd ~ 
gitlabhq_install/ubuntu_gitolite.sh
When you run this script it will stop at some point with a warning about the path, just press the “Enter” key to continue. On the next screen is the gitolite configuration screen. Here we need to make one change that’s very important. Find the line that reads:
REPO_UMASK = 0077;
If the install opened VIM, move over the first “7” character, press the “i” key on your keyboard to go into INSERT mode. Type a “0”, then remove the “7” so it now reads:
REPO_UMASK = 0007;
Press the Escape key once, then type the “:” to enter COMMAND mode. Now type “wq” which will Write the changes to the file and Quit.
You now need to change the directory privileges on the /repositories directory so GitLabHQ can use them:
sudo chmod -R g+rwX ~git/repositories/
sudo chown -R git:git ~git/repositories/
Next we need to logout of the system to allow environment settings to be set upon the next time we login.
logout

Install GitLabHQ

Log back into the system so the environment settings take place
ssh gitlabhq@localhost
Now we’ll install GitLabHQ, again using one of the install scripts. When prompted about installing additional packages, type “Y”
cd ~ 
gitlabhq_install/ubuntu_gitlab.sh

Configure GitLabHQ

You can configure GitLabHQ by editing the gitlab.yml file. One of the changes you’ll want to make is to set your computer name that GitLabHQ is running on, if not localhost, so the instructions to users for connecting to repositories is correct.
nano ~gitlabhq/gitlabhq/gitlab.yml
Change the host value to whatever your servers fully qualified domain name (FQDN) is. So for example if I’m running GitLabHQ on a server named “gitlabhq.corp.com” I’d change the value:
# Git Hosting congiguration
git_host:
  system: gitolite
  admin_uri: git@localhost:gitolite-admin
  base_path: /home/git/repositories/
  host: localhost
  git_user: git
  # port: 22
to
# Git Hosting congiguration
git_host:
  system: gitolite
  admin_uri: git@localhost:gitolite-admin
  base_path: /home/git/repositories/
  host: gitlabhq.corp.com
  git_user: git
  # port: 22

Running GitLabHQ

Now that we have GitLabHQ installed, let’s start the application using WEBrick (even if you’ll use something else later) so we can login and accept an RSA key, then confirm it works.
cd ~gitlabhq/gitlabhq
bundle exec rails s -e production
Now you can login to your server by pointing your web browser to http://:3000/ and login using the default credentials
  • Login Email: admin@local.host
  • Login Password: 5iveL!fe

Important!

You should now create a new PROJECT. It’s important to note that when you add this project the FIRST TIME you need to type “yes” on the console where you started the application running.

Installing nginx

Login as the gitlabhq user and then execute the following commands:
sudo gem install passenger
sudo passenger-install-nginx-module

Configure nginx

We need to edit the nginx configuration file so it points to the GitLabHQ public folder to run the application. Open the configuration file in the editor:
sudo nano /opt/nginx/conf/nginx.conf
Now locate the section for the server configuration and make the following changes:
  • Change the server_name key to your server’s fully qualified domain name (FQDN), so in this example the server is gitlabhq.corp.com
  • Change the root key to the location of the GitLabHQ public folder, this is important!
  • Add the key/value passenger_enabled on;
server {
        listen       80;
        server_name  gitlabhq.corp.com;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   /home/gitlabhq/gitlabhq/public;
            index  index.html index.htm;
            passenger_enabled on;
        }
Also on the very top of the file, add the first line that specifies we’ll run the server as the gitlabhq user account:
user gitlabhq staff;
Now we want to add a system user named nginx to run the server:
sudo adduser --system --no-create-home --disabled-login --disabled-password --group nginx
Next we want to setup the server to auto-start when the system starts. To do this we’ll:
  • Use an existing script to start nginx
  • Move the script to the system start directory
  • Set the correct permissions
  • Start the server.
sudo wget -O init-deb.sh http://library.linode.com/assets/660-init-deb.sh
sudo mv init-deb.sh /etc/init.d/nginx
sudo chmod +x /etc/init.d/nginx
sudo /usr/sbin/update-rc.d -f nginx defaults
sudo /etc/init.d/nginx start

nginx over SSL

So you want to run nginx over SSL huh? Good choice!

SSL Certificate

First you’ll need an SSL certificate, either self-signed or from a certificate authority like Verisign. You can find directions on using certificates here
However, to keep it simple and helpful we’ll use a self-signed certificate for our server gitlabhq.corp.com
Let’s create a 2048-bit certificate. When prompted for the passphrase, enter something at least four characters in length.
cd ~
mkdir ssl
cd ssl
openssl genrsa -des3 -out server.key 2048
Now let’s get that passphrase out of the key file just to keep it secret. You’ll be prompted for the passphrase you entered when creating the certificate.
openssl rsa -in server.key -out server.key.insecure
mv server.key server.key.secure
mv server.key.insecure server.key
openssl req -new -key server.key -out server.csr
Now let’s sign that shiny new certificate for 5 years
openssl x509 -req -days 1825 -in server.csr -signkey server.key -out server.crt
 
Finally we need to move the files to the correct locations on our Ubuntu server
sudo cp server.crt /etc/ssl/certs
sudo cp server.key /etc/ssl/private
 

Configure nginx

Open the nginx configuration file, scroll to the bottom and locate the commented out section for the HTTPS. You can uncomment this section and specify your certificate location and server name as well as the location.
# HTTPS server
    #
    server {
        listen       443;
        server_name  gitlabhq.corp.com;

        ssl                  on;
        ssl_certificate      /etc/ssl/certs/server.crt;
        ssl_certificate_key  /etc/ssl/private/server.key;

        ssl_session_timeout  5m;

        ssl_protocols  SSLv2 SSLv3 TLSv1;
        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers   on;

        location / {
            root   /home/gitlabhq/gitlabhq/public;
            index  index.html index.htm;
            passenger_enabled on;
        }
    }

}
Now we need to restart nginx for the configuration changes to take place
sudo /etc/init.d/nginx stop
sudo /etc/init.d/nginx start
Enjoy!

Sunday, December 11, 2011

Beating the Flying Spaghetti Code Monster with CFWheels - Part 2

Beating the Spaghetti Code Monster

Recap

As I mentioned in part 1 of this post, when I started writing web applications with ColdFusion I quickly amassed a large collection of autonomous files that would often break if I changed a variable or parameter that should be passed into the next. Adding or removing a column to a form / database table could take 25 minutes to update all of the SQL files. I had become enslaved to the “Flying Spaghetti Code Monster”, as I like to call it, and knew there was a better way and wanted to find it.

The Application

As I also mentioned in part 1, I had been re-hired to maintain and update a web application I wrote 7 years prior. This application had well over 350 individual files and the folder size was near 80MB. Although I had started to use the Fusebox framework, which helped start to organize my files, I didn’t follow-through on converting everything to Fusebox. This meant that my directory structure looked something like this
Spaghetti Code Directories

Conversion

As also mentioned in part 1 of this post that I decided to slowly convert the application to the Coldfusion on Wheels framework. I decided to do this cutting off one “tenticle” at a time, or one section/feature at a time.
First I identified a small section of the application that I could replace with the CFWheels framework. I selected one of the “system options” that allows users to add new values to a drop-down list of patient types. The code for this was located off the root of my application in a /system folder along 15 other similiar system options that were almost identifcal. I also needed a place to put my CFWheels core files.
I decided to put the CFWheels framework files in a directory named “hla”, which is the web application name as referred to by the staff, under the /system folder, so /system/hla/
CFWheels framework location
I setup my database connection info in the /conf/settings.cfm file so that my database connection would be the same as the spaghetti code portion. Next, I didn’t want to have two sets of Javascript, CSS or other asset files like images, so I had to modify my CFWheels layout.cfm file to reference the ones that the spaghetti code files were using.
layout.cfm
At this point I could browse to the CFWheels portion, by manually entering the URL, and receive the default CFWheels welcome page. I then added a new entry on my system options menu to point to the new CFWheels location where the system option I was going to convert would live.
At this point I created a model file, Patiettype.cfc, and in the init function I added code to specify my database table names that didn’t follow the Wheels convention, but were used by the Spaghetti Code application
<cfset table("tbl_authors")>
Next I followed with each database table column as a property, using the current database column name for the column attribute but a CFWheels convention name for the name attribute, as outlined in the CFWheels documentation.
<cfset property(name="firstName", column="tbl_auth_f_name")>
These system options required that I setup some associations, and this was a great place to learn about that as I’d need it for every other part of the application.
At this point I could follow the CFWheels conventions and develop the CRUD for the Patienttype just like I would a brand new CFWheels app. After I have the CRUD working for this system option, I’d then remove the link to the old files and leave just the new one. Then I’d have the users work with that section for about a week. Once we were sure everything worked correctly, I’d slowly remove the old files (that I now had in Git version control).
Using this same method I slowly worked through all of the 16 system options, as if cutting off one “tenticle” of the Spaghetti Monster at a time. After I finished with those small, simple CRUD actions, I moved onto more complicated sections. The whole Contacts portion, then Hospitals portion, etc.

Progress

I found the Coldfusion on Wheels framework in February 2011, then started implementing it in March 2011, working part-time and slowly through each of the tenticles I mentioned. If new features were requested, or something needed to be changed, I fought the urge to do a “quick and easy” one or two file solution and forced myself to do it the CFWheels way. At first this was challenging, but once I got use to it, the changes happened faster.
At this point, December 2011, I’ve removed at least 23 of these “tenticles” with two medium sized tenticles and one large tenticle left to remove from the monster. I’m excited to finish these sections off and sever the head of this monster once and for all. In hindsight I’m glad I started with smaller tenticles as it allowed me to learn CFWheels with smaller sections of code, but also to find “gotchas” early on.

Gotchas

One of the things I didn’t expect was the Spaghetti Monsters session scope being overwritten when I moved went to a CFWheels framework page. This made it a little tricky because I’d use the session scope to retain the currently logged in user ID and if they were a valid user or not (to prevent unauthorized access). After I’d leave a CFWheels page and go back to the Spaghetti Monster code I’d be kicked out of my application. My “fix” for this, at least for the time being, was to use client variables for the Spaghetti code portion of the application and session variables for the CFWheels portion of the application.
The next thing I ran into, since my session scope was being overwritten, was once I moved to a CFWheels portion of the application I’d didn’t have my session variable with the user ID from the Spaghetti Monster code. To fix this I added an obtuse URL variable on the menu links in the Spaghetti Monster code that CFWheels would then use to identify the user and create the session scope variables I wanted.

Summary

I feel converting this application to the CFWheels framework has helped with the following:
  1. Brought some sanity back to how this application is structured and maintained (huge benefit)
  2. Saved gobs of time not having to maintain query files with endless CFIF statements and cfqueryparam tags (CFWheels ORM does that all for me, huge benefit)
  3. Made the application run faster, I mean noticably faster to my users
  4. Decreased the code folder from 80MB to 33MB
  5. Allows me to add new features or make changes in less time then before because things break less when changes are made
  6. Allows anyone that knows about the MVC pattern to help or take over this application
  7. Made Coldfusion programming fun again (most important of course)
While converting this application from the Spaghetti code to the CFWheels framework from “the inside out” has been a slow process (because of my, not the framework), I don’t regret doing it at all for the reasons stated above. It’s been a great way to learn CFWheels and to work with the CFWheels community. At this point I can’t imagine developing Coldfusion applications without using the CFWheels framework.

Thursday, December 01, 2011

Beating the Flying Spaghetti Code Monster with CFWheels - Part 1

Flying Spaghetti Code Monster

Background

I started writing web application using Coldfusion in 2000 with version 4.5. Like probably most of us back in those early days, we had one hand on the keyboard and a copy of Ben Forta’s Web Application Contrution Kit (WACK) in the other. My development of applications consisted of creating a single autonomous file, with both database queries and conditional display code. This of course was tied to yet another autonomous which did the same, etc, etc, etc. Thus the “Flying Spaghetti Code Monster” Even after I started to use the Application.cfm “framework” within ColdFusion, I still had all of these autonomous files that would often break if I changed a variable or parameter that should be passed into the next.
After developing a few applications like this I realized there had to be a better way. Somehow I came across the Fusebox 2 framework and started to look into it. This framework encouraged you to break the application up into separate files, query file, action file, display file, and then “chain” these together like fuses in an electrical circuit.
Shortly after Fusebox version 3 was released and I begin using this framework to develop my applications. I especially loved the formurl2attributes function that put all of your url and form variables into a single scope. I started to convert small pieces of my existing applications from my spaghetti code to organized code following the Fusebox methodology, however I ended up sticking more with the file naming then actually using the framework after a while. When Fusebox version 4 was released I found it difficult to grasp, as well as the CFC concept that was brand new. My two and a half years of part-time web application programming was coming to a close though as I left to work full-time as a systems administrator.
Over the next 7 years the largest of the web applications I wrote, a Laboratory Information System (LIS), continued to work for the University lab I wrote it for with minimal adjustments to it.

Back to Developing

At the end of those 7 years I was hired back, part-time, for this same lab, and part-time for another lab (so it’s a full-time position) on campus. When I looked at this old source code I saw the “Flying Spaghetti Code Monster” glaring back at me. I had well over 350 individual files and the folder size was near 80MB, not to mention no source control software to help manage it. I knew there had to be a better way to handle this, I didn’t want to continue to develop this way.
The first thing I did was get the software into a source control system. Subversion was the choice as it was the easiest for me to grasp initially. I went back to look at the Fusebox framework and saw it was still around, but it didn’t seem like many were using it any longer. There was the Mach-II framework, but I couldn’t seem to understand that either. In the meantime I continued to serve the hard master known as the “Flying Spaghetti Code Monster” and maintain the code as it was, even adding new features, without much difference in my development pattern.
After five months of developing this way I came across the ColdFusion on Wheels framework website, not sure how (probably Twitter). After greedily consuming all of the available screencasts I knew this was the way I needed to develop the application. The question was, how do I learn the CFWheels framework and use it to power this application? I saw two choices:
  1. start from scratch in a parallel environment using CFWheels and recreate all of the functionality
  2. slowly, one “tenticle” at a time, battle the “Flying Spaghetti Code Monster” by converting a section to the CFWheels framework
After considering it I decided choice 2 was for me, to slowly convert the application. In the following post I’ll explain how I’m doing this.

Thursday, October 27, 2011

Astaro “ICMP Destination unreachable (Fragmentation needed)” Problems

Problem:

Twice now my Astaro Security Gateway VMWare Virtual Appliance performance has dramatically decreased any traffic through and reports “ICMP x.x.x.x unreachable - need to frag (mtu 1500)”

Solution:

Change your VMWare virtual appliance network adapters from the pre-configured “Flexible” to “E1000” by editing the .vmx file.

Description:

I have four virtual machines that run on VMWare ESX 4. One of these VM’s is the ASG virtual appliance connected to a public (Internet) interface and a private (virtual network) interface. Each of the remaining three systems are connected to the private virtual network as well. One is a web server, another a database server.

Using a DNAT rule I make my web server accessible to external (Internet) users. Using the SSL VPN I made my database server accessible to internal (company) users.

For some reason, and after performing very well for months, the ASG starts exhibitng awful performance is passing the traffic back and forth. Web pages that external users try to access rarely display and more often timeout. (Interesting note is that Windows clients seems to timeout 93% but Mac clients display the web page, after a few extra seconds, 99% of the time).

Users connected via SSL VPN and accessing the database server also see horrible performance as well. If I do a SCP of a 3MB file from the database server to my laptop over the SSL VPN it transfers in 30 seconds. However uploading this same 3MB file from my laptop to the database server requires a transfer time of 12 minutes!

The ASG stats show very little load on the system during these transfers:

CPU: 4%
RAM: 31%
Swap: 1%
Log disk: 2%
Data disk: 5%

Using SSH on the Astaro I run a tcpdump to see what’s going on. I’m able to narrow it down by using:

tcpdump -ni any -s0 -v -f 'icmp[icmptype] == 3 and icmp[icmpcode] == 4'

After shutting down the ASG virtual appliance, then changing the network adapter type to “E1000” and restarting the ASG virtual appliance the problem vanishes.

Astaro Support Rant

At this point I just want to rant a little about Astaro Support. I’ve been working with Astaro products since 2005 and really, really like them. They’ve always been, in my opinion, an EXCELLENT value for the feature set you receive and the price that you pay, just excellent.

In those 6 years I’ve worked with Astaro products (6 different Astaro product installs) I’ve had to contact their support about 10 times, which I don’t think is bad. In almost all cases the support follow-up, communication and technical ability of the engineers has really been excellent.

However, in the past four months I’ve had to contact Astaro three times. In each instance the response and communication was pathetic, meaning that I either didn’t hear anything from Astaro or I had to call them to get any response.

In another instance I opened a ticket on July 6th, received a response back on that same day with some comments on the case, exchanged a few more emails on the 6th and 7th, then I didn’t receive another response until July 15th!

I called Astaro support again and spoke to a gentlemen that said all of the engineers were in a meeting and weren’t available.  He said he would note the case and leave a note on the engineers desk to have him call me back after the meeting.  I didn’t receive a phone call that day before I left at 5:40pm nor later then evening although I had my work phone forwarded to my cell expecting the call.

The next day, Thursday, I called again.  I spoke to the same gentlemen that wasn’t sure why the engineer hadn’t contacted me.  He said the engineer wouldn’t arrive until later today, as he does west coast support, and would contact me by 1:00pm.  Again he said he would note the case, leave a note for the engineer on his desk and would ask him directly to contact me.  I never received another call from the engineer.

After sending off an email to the sales rep and the support manager I was finally able to get someone’s attention and later the following day a new engineer was assigned to my case.

This instance has damaged my faith in Astaro support. I’m not sure if this has anything to do with Sophos aquiring Astaro, but it sure seems like since then the support has plummeted.

I’ll continue to use Astaro for my existing projects, however the next new project that comes along will have me looking for a competing device to try.

Thursday, October 20, 2011

MySQL won’t start after Ubuntu 11.10 upgrade

I upgrade my Ubuntu database server from 10.10 to 11.04 to 11.10 and suddenly my MySQL Server stopped responding.

Turned out the problem was that I didn’t accept one of the new apparmor configurations since it looked minimal and I had customized mine. Because of this my /var/log/mysql/error.log was filled with the following:

111020 17:31:44 [Note] Plugin 'FEDERATED' is disabled.
111020 17:31:44  InnoDB: Initializing buffer pool, size = 896.0M
111020 17:31:44  InnoDB: Completed initialization of buffer pool
111020 17:31:44  InnoDB: Started; log sequence number 0 172373406
111020 17:31:44 [ERROR] Can't start server : Bind on unix socket: Permission denied
111020 17:31:44 [ERROR] Do you already have another mysqld server running on socket: /var/run/mysqld/mysqld.sock ?
111020 17:31:44 [ERROR] Aborting

I had to edit the app armor file for MySQL

sudo nano /etc/apparmor.d/usr.sbin.mysqld

changed

  /run/mysqld/mysqld.pid w,
  /run/mysqld/mysqld.sock w,

to

  /{,var/}run/mysqld/mysqld.pid w,
  /{,var/}run/mysqld/mysqld.sock w,

Wednesday, October 12, 2011

Changing Fonts in Sublime Text 2

I’ve been using the Sublime Text 2 app for this past week and really like it. One of the things I wanted to change though was the font used in the editor. A font I’ve liked in the past was Consolas but I couldn’t find that. I did find one named Inconsolata, which is almost identical from what I can tell, on Google Web Fonts. Here’s how I set everything up on my Mac

Get the font(s)

  • Head over to Google Web Fonts and search for “Inconsolata”, or any other font that you like (other suggestions are Ubuntu Mono or Varela Round)
  • Click the “Add to Collection” button to the far right of the font name
  • Click the “Download your Collection” link on the top-right.
  • Click the “Download the font families in your Collection as a zip-file” link on the pop-up dialog
  • Open the zip-file that’s downloaded
  • Double-click on each .ttf file
  • Select to “Install Font”

Configure Sublime Text 2

  • Open Sublime Text 2
  • Click on the “Sublime Text 2” application menu
  • Select “Preferences”
  • Select “File Settings - User”
  • In the file that opens, add the following line along with the name of the font you installed:
    "font_face": "Varela Round"
    Be sure to include a comma “,” if you’re adding this to a line already within the file, like a color scheme selection.

You can also change the font size using the following:

"font_size": 16
Notice that the number for the font size isn’t in quotes. Once you save the file you should see the results immediately as the configuration file you’re working in is redrawn using your selections.

Here’s my current complete configuration file

{
    "color_scheme": "Packages/Color Scheme - Default/Indy Merbivore Soft.tmTheme",
    "font_face": "Varela Round",
    "font_size": 16
}

Three New Sublime Text 2 Features

Here are three new Sublime Text 2 features that I learned today, thanks to this article at Net tuts+, that I feel really put it high on my list for a ColdFusion editor.

  • Function Crawling - while working within a CFC I press Command + R on my Mac and I now have a list all of functions within that component with my cursor focused in this box. I start typing the function and press enter to quickly jump to it. Awesome!
  • Command Palette - while working within a CFML page I press Shift + Command + P to have the entire list of snippets (i.e. cfml tags and functions) that I can start typing to find and press enter (sometimes I forget the exact name so this really helps).
  • Lightning-Fast File Switching - while working in a file I press Command + P and now have a list of the files with in the project I’m working in. I start typing the file to have it opened for me, I don’t even have to press Enter

Using Sublime Text 2 as ColdFusion Editor

I’ve tried a lot of different editors for writing my ColdFusion code, but never found one that really worked well for me until now. Now I think I’ve found the one that works best for me, Sublime Text 2.

In the past I’ve tried:

  • Coda by Panic = does a really good job with handling the auto-completion of quotes and parenthesis. Has tabs for multiple file editing and does CSS, has a web preview and a Terminal tab. It’s a Panic app, which has a tradition of working really well. However I started to run into these I’d really like, like code folding, easier code colouring and speed seemed to decrease with each release. Coda 2 is “in the works”, but no one really knows when it’ll show up.
  • CFEclipse = I like how extensible the Eclipse platform is, it’s much more of a Integrated Developer Environment (IDE) then Coda. With extensions like Subclipse (Subversion), eGit (GIT) and Mylyn, it can be a pretty sweet tool. However like any Eclipse based environment that I’ve used it suffers two serious problems that over-time frustrate me. It seems to get slower and slower, and the way that it auto-completes of quotes and parenthesis easliy gets confused.
  • ColdFusion Builder 2 = same as CFEclipse, except it costs money to run slow, and if I’m correct in what I’ve found, only runs in 32-bit mode and not 64-bit mode in Mac OS X.
  • TextMate = from what I’ve read this application is pretty “long in the tooth”, but it seems to have a good sized and very faithful following. There is a ColdFusion bundle available. It’s very fast, you can define projects to see all of the code in your directories and has tabs. It works very well with the auto-completes of quotes and parenthesis. TextMate charges for a license, which I paid, but I just had a hard time getting into how to use it and didn’t find much in the way of documentation or screencasts that explained it well to me.

Sublime Text 2

I’ve been using a beta copy of the Sublime Text 2 editor for the Mac and really like it so far. Some of the key things I like about it after a week are:

  • It’s fast
  • Auto-completes the use of quotes and parenthesis correctly.
  • Good code syntax colouring, that can be modified, although I’m using the “Merbivore Soft” theme by Indynagpal.
  • Can open a folder so I can see multiple files at once
  • For file contents that are taller then the screen, a “code overview” of sorts displays on the right to quickly jump to another locatio in the file.
  • Code-folding
  • Tabbed interface for files, and interestingly when you click on a file it will show the contents but not open it as a tab unless you start to edit the file or double-click on it, great for a “quick peek” of the contents!

As I work with Sublime Text 2 more, and as it evolves from it’s beta form, I’ll post more on how I’m using it and what I find benefical.

Tuesday, October 04, 2011

Force Internet Explorer 8 Not To use “Quirks” Mode

I’ve been using the excellent BluePrint CSS framework for a new web application I’m developing. It does a great job and making the site look very good, and consistent, on almost all browsers except Internet Explorer (I know, huge surprise).

In my particular case I have users that are forced to use Internet Explorer 8.0 as their browser. With the configuration that’s set by default for them, Compatibility View is enabled because the University Enterprise Business software doesn’t display correclty on modern browsers or recent versions, meaning anything newer then Firefox 3.6 or Internet Explorer 6.0.

To compound the annoyance, Internet Explorer Compatibility View can’t be set for one particular site only, it assumes that all pages under the domain also need it and forces this as well with no option to only use or exclude a subdomain.

When Internet Explorer 8.0 uses Compatiblity View, it renders the webpage as Internet Explorer 7.0 and earlier would, also called “quirks” mode, which doesn’t display correctly with BluePrint CSS or other web-standard layouts. I found that I can add HTML code to force Internet Explorer not to use Compatibility View on my web application. Not using Compatibility View uses the newer rendering engine that does a slightly better job with the newer HTML standards.

To do this, add the following to your HTML header:

<meta http-equiv="X-UA-Compatible" content="IE=EmulateIE8">

Installing VMware Tools for Ubuntu 10.04+

Update your system to the latest packages

This updates your system with the catalog of the latest packages available, then upgrades any packages on your system that may have updates available.

sudo apt-get update -y
sudo apt-get dist-upgrade -y

Install the dynamic kernel modules

These modules will allow you to dymanically load modules with the kernel. I’m of the understanding that this prevents you from having to re-install the VMware Tools everytime the Linux kernel is updated.

sudo apt-get install dkms -y

Install VMware Tools

First you’ll need to initiate the VMware Tools install from the VMware Host system. This can usually be found on a “Tools –> VMware Tools Install” like menu.

Next we install the Linux build commands that are needed to compile the kernel with the VMware Tools extensions. This also requires the correct Linux headers for our system, which are determined using a system variable. Then we remove any packages that are no longer used (generally none, but just to be sure).

Finally we install the VMware Tools by doing the following:

  1. Create a directory to mount the tools CD-ROM to
  2. Mount the CD-ROM to the directory
  3. Copy the tools archive to the temporary directory on the system
  4. Unmount the CD-ROM
  5. Change to the temporary directory
  6. Extract the archive of the VMware Tools
  7. Change to the VMware Tools extraction directory
  8. Execute the VMware Tools installer script and indicate to use the defaults for all choices during the installation
sudo apt-get install build-essential gcc -y
sudo apt-get install linux-headers-$(uname -r) -y
sudo apt-get autoremove -y
sudo mkdir -p /media/cdrom
sudo mount /dev/cdrom /media/cdrom
cp /media/cdrom/VM* /tmp
sudo umount /media/cdrom
cd /tmp
tar -xzvf VMware*.gz
cd vmware-tools-distrib/
sudo ./vmware-install.pl -d

Restart the system

Last we want to restart the system for all of the changes to take affect.

sudo shutdown -r now

Monday, October 03, 2011

Lion Mail Crashes When Searching

I’ve had a very frustrating problem with searching in Mac OS X 10.7 “Lion” Mail app since I upgraded. After installing Lion after release, searching in Mail using the keywords like “subject:” and “from:” weren’t being honored and the search results weren’t correct. I was able to resolve this by one of the Apple engineers pointing me towards a bad index and rebuilding my mailbox indexes. This resolved that issue.

After that I started to see a problem where randomly when I’d type anything into the Mail search field the app would freeze, then crash. I was able to do the searches through Spotlight with the keywords, but it was kind of annoying. This became a consistent, reproducable problem after Mac OS X 10.7.1 was installed.

Earlier today I was able to talk again with one of the Apple engineers and he pointed me towards a corruption problem with the “Previous Recipients” list in Mail. Sure enough when I’d try “Window -> Previous Recipients” the window wouldn’t show, and when it finally did after a number of times and restarts, it was only half drawn on the screen.

Since I wasn’t able to clear this list using the GUI, I dig around and found I could do it from Terminal. Since doing this I’m able to search in Mail now without crashing, and it’s so nice

To fix this we need to open the “Previous Recipients” database, which is a sqlite database. First, close your Mail app if it’s running. Next, open Terminal and type:

sqlite3 ~/Library/Application Support/AddressBook/MailRecents-v4.abcdmr

Next we need to clear out all of the “Previous Recipients” addresses in this database. To do that, type:

DELETE FROM ZABCDMAILRECENT

To exit sqlite press CONTROL + Z.

Open Mail and you should now be able to search!

Thursday, September 29, 2011

Adding Syntax Highlighting to Blogger

I’ve switched back to Blogger from Posterous after so many failed attempts in trying to get the latters markdown syntax to display correctly. I still write my posts using markdown syntax in Byword, but then I display it in a full-formatted HTML syntax (Alt + Command + P) and click the “Copy to HTML” which I paste into Blogger.

One of the things I wanted on my blog was improved visual display of code (i.e. CSS, Coldfusion) or commands (i.e. bash) and I found the SyntaxHighlighter project. This project can be added to any website, including a Blogger template (except for the dynamic ones right now).

I found a number of references on how to add this to Blogger, but many were from a year or more ago (which is decades in Internet time). So here’s a quick guide on how you can add the SyntaxHighlighter project to Blogger.
Note: I’m using the new Blogger interface

  1. Login to your Blogger account
  2. Click on Template
  3. Click on the “Edit HTML” button under the “Live on Blog” thumbnail
  4. Click the “Proceed” button to the message warning you about editing the code
  5. Search for the text </head>
  6. Paste the following code
    <script src='http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js' type='text/javascript'>
    <link href=’http://alexgorbatchev.com/pub/sh/current/styles/shCore.css’ rel=’stylesheet’ type=’text/css’/>
    <link href=’http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css’ rel=’stylesheet’ type=’text/css’/>
    <script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js’ type=’text/javascript’>
    <script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushColdFusion.js’ type=’text/javascript’>
    <script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js’ type=’text/javascript’>
    <script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js’ type=’text/javascript’>
    <script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js’ type=’text/javascript’>
    <script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js’ type=’text/javascript’>
    <script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPlain.js’ type=’text/javascript’>
    <script language=’javascript’> 
    SyntaxHighlighter.config.bloggerMode = true;
    SyntaxHighlighter.config.tagName = “pre”;
    SyntaxHighlighter.defaults.toolbar = false;
    SyntaxHighlighter.all();
      
  7. Click the “Save Template” button
  8. Click the “Close” button

Now let me explain a little of this to you.
The first line is loading the core Javascript file for the SyntaxHighlighter project, this is required.

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js' type='text/javascript'> 

The second line is loading the core CSS file for the SyntaxHighlighter project, this is also required.

<link href='http://alexgorbatchev.com/pub/sh/current/styles/shCore.css' rel='stylesheet' type='text/css'/>

The third line is loading the display theme to be used. If you don’t like this particular theme, there are others you can choose from, see the list here.

<link href='http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/>

Lines 4 through 10 are particular “brushes” that I want to load. A brush is specific for the language syntax you want to display. Here I’m loading brushes for Bash scripts, ColdFusion, CSS, JavaScript, SQL, XML (which includes HTML) and plain text. You must load at least one brush, but you can have as many as you want. You can find a list of the different language syntax brushes here.

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js' type='text/javascript'>
<script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushColdFusion.js’ type=’text/javascript’>
<script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js’ type=’text/javascript’>
<script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js’ type=’text/javascript’>
<script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js’ type=’text/javascript’>
<script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js’ type=’text/javascript’>
<script src=’http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPlain.js’ type=’text/javascript’>

Line 12 is required for the SyntaxHighlight project to work with Blogger.

SyntaxHighlighter.config.bloggerMode = true;

Line 13 is designating the <pre> tag as what will define the code block to have the brush applied to.

SyntaxHighlighter.config.tagName = "pre";

Line 14 disables the toolbar from displaying when the SyntaxHighlighter is applied. This seems to just be a “?” that’s displayed that the user can click on to see the name of the project. From what I can tell this is deprecated from previous versions so I disable it.

SyntaxHighlighter.defaults.toolbar = false;

Line 15 runs the SyntaxHighlighter code

SyntaxHighlighter.all();

User unable to login to Astaro User Portal

I recently setup some of the lab staff for access to their remote database through an SSL VPN using the awesome Astaro Security Gateway.

One of the users was having some problems to login to the user portal though to change their password. It turns out I gave them them wrong password initially, but even with this correct password they still weren’t able to login.

I entered the WebAdmin and checked the Logging & Reporting -> View Log Files -> User authentication daemon log. I could see the users attempts to login, shown as:

2011:09:29-13:19:48 secure aua[5336]: id="3005" severity="warn" sys="System" sub="auth" name="Authentication failed" srcip="x.x.x.x" user="smithjohn" caller="portal" reason="Too many failures from client IP, still blocked for 363 seconds"

I hadn’t seen this before. This is a really good thing to prevent brute force or automated attacks at guessing the users password. However in this case we didn’t want to wait the 6 1/2 minutes to try again.

Under the Management -> WebAdmin Settings -> Security I temporarily changed the values that block the authentication to:

 
After 30 attempts
block access for 3 seconds

The user was then able to login to the user portal, change their password and logout. I then changed the values back to the following:

 
After 3 attempts
block access for 300 seconds

Wednesday, September 28, 2011

CFWheels Controller Filters in Init May Need Super.init()

I'm writing a web application using the ColdFusion on Wheels framework and ran into something that I didn't expect. This is partly because of my "vague" understanding of object-oriented programming. In my Wheels application I have in my Controller.cfc an init function which looks like this:

<cfcomponent extends="Wheels">
   <cffunction name="init">
           <cfset filters(through="checkLogin", except="login,authenticate,logout,resetPassword,forgotPassword")>
   </cffunction>
   <cffunction name="checkLogin">
           <cfif StructKeyExists(session, "user")>
                   <cfset loggedInUser = model("user").findByKey(session.user.id)/>
           <cfelse>
                   <cfset redirectTo(controller="users", action="login")/>
           </cfif>
   </cffunction>

Pretty basic. So each time one of my controllers does something it inherits this init function and verifies the user has a session value, if not, send them back to the login screen. Worked great!

However in trying to DRY up my Controllers, and seeing an example that Tom has on his blog I wanted to create an init() within one of my controllers, say “Contacts.cfc”, that would populate a structure with a list of values I use over and over again (think yes/no or gender or US states). So in the Contacts.cfc controller I had this

<cfcomponent extends="Controller" output="false">
    <cffunction name = "init">
        <cfset filters(through="getYesNoValues", only="new,edit")/>
        <cfset filters(through="getStatesValues", only="new,edit")/>
    </cffunction>
</cfcomponent>

This worked great! However, I noticed that after I came back to my application after 30 minutes (when my session scope times out and my session.user.id would be destroyed) that I could still navigation through my CFWheels application within the Contacts controller, even though I was no longer logged into the application and shouldn’t be able to see this information!

I learned that if I create a “init()” function in my Contacts controller, then it doesn’t run my “init()” function in my Controller.cfc, so it’s not checking my session to make sure it’s valid.

I was able to solve this by changing my Contacts.cfc controller init() to call the Controller.cfc init() function by using the Super.init(), like so:

<cfcomponent extends="Controller" output="false">
 <cffunction name = "init">
   <cfset Super.init() />
   <cfset filters(through="getYesNoValues", only="new,edit")/>
   <cfset filters(through="getStatesValues", only="new,edit")/>
 </cffunction>
</cfcomponent>

This seems to work as I expect now. If my session times out, and I try to work with my Contacts.cfc controller, then I’m sent back to the user login. Turns out this was documented in the (excellent) CFWheels documentation on Filters