I first found out that the funny characters were due to the LANG environment variable being set to en_AU.UTF-8 (Australian locale with Unicode support). Doing a grep -R LANG /etc/* told me that the variable was being set to the value of GDM_LANG. But there was no trace of where GDM got its values from.

Whilst googling for answers, I kept running across the word locale, so I did an apropos locale on my RedHat box, followed by a few man pages, followed by some more googling, when I finally arrived at this

The crux of the issue was that I had to change the variable in /etc/sysconfig/i18n. Would have been nice if it had have been a bit easier to find.