Sporadic SMTP 5.7.1 "Relay access denied" error with Exchange

If users are receiving mail bounced back sporadically with the following message:

 You do not have permission to send to this recipient. For assistance, contact your system administrator. <[server name] #5.7.1 smtp;554 5.7.1 [recipient address]: Relay access denied>

and relaying is configured properly on your server, it could actually be a DNS issue. Login to the Exchange server and lookup the recipient domain:

C:\Documents and Settings\user>nslookup
Default Server: dns.yourdomain.local
Address:  192.168.1.1

> set type=mx
> recipient.com
Server:  dns.yourdomain.local
Address:  192.168.1.1

DNS request timed out.
    timeout was 2 seconds.
*** Request to 192.168.1.1 timed-out

The reason it was timing out is the DNS server included with Windows 2003/2008 enables the EDNS0 extension (RFC 2671) to DNS by default, which utilizes packet sizes greater than 512 specified in RFC 1035. Many firewalls block this by default. For more information:

http://support.microsoft.com/kb/832223 

 My solution was to modify the firewall configuration to increase the packet size. For a Cisco ASA:

fw# conf term
fw(config)# policy-map type inspect dns preset_dns_map
fw(config-pmap)# parameters
fw(config-pmap-p)# message-length maximum 1280
fw(config-pmap-p)# policy-map global_policy
fw(config-pmap)# class inspection_default
fw(config-pmap-c)# inspect dns preset_dns_map 
fw(config-pmap-c)# end
fw#

So Core, It's Hardcore.

My crew and I were up late last night upgrading Stevens Pass core network.  We replaced a Cisco 4510R (dual power supply, but single management) with a stack of Brocade FastIron FCX624S-F switches and moved edge ports previously consolidated in the core to a couple of FastIron Edge 4802 switches.   The server and other core switches are dual homed and use LACP to aggregate the uplink ports for fault tolerance in the core.

The second switch in the stack is otherwise bare because capital funding is needed to build redundant paths to the other lodges.

On top of the mitigated risk, it really cleaned things up and conserved rack space.

Old and busted:

New hotness:

Extreme closeup!

Resumes: The Darkside of the Force

Most posts I'm sharing what I've learned and find passion in doing, but today, I'm stepping out of my element to share something I struggle with.   My resume.

It seems everyone is looking for something different..  It's too long.  It's not specific enough.  It needs more fluff.   I get confused and frustrated.  If you look at my blog I have written detailed solutions I've developed to some fairly complex scenarios with XenServer.  I have also configured, tested, and use enterprise features like high availability and live motion.  So how much detail should I include when I mention "virtualization" in a resume?

I've never heard back on several jobs I found exciting and believed would be a great fit. On the other hand, I've never been unemployed and have always worked with and for great people, and people who have interviewed me seem like people I would enjoy.  So I go back and forth being confident my resume is effective.  Also, networking within the industry seems to be far more valuable. ("It's who you know")

Okay, no more ranting, I do have something to offer!  I have been using LaTeX (pronounced "lah-tech") for my resume since day one.  It is not a word processor, but a powerful typesetting language that provides greater control over layout, and offers a unique, professional look.   Each time I have gone to update mine, I'll search to see if someone has done something I like better, but I still come back to tweaking my own layout.

I'm making it available under a Creative Commons license, so feel free to use it.  The PDF version is also available to view document layout, and hopefully in return,  receive some constructive feedback on how to improve my own content.  I'd love to hear peoples thoughts on resumes and the job hiring process!

Download LaTeX Source
Download PDF

Mirrored / (root) btrfs on Ubuntu 11.04

btrfs (Pronounced "Butter FS") is a new, copy-on-write (COW) filesystem for Linux.  It includes advanced features such as subvolumes, writable snapshots, and integrated, multi-device support.  See the btrfs site for a complete feature list and documentation (which was used during this setup).

I do not recommend using btrfs on a production system, as it is under heavy development, does not have a utility to repair filesystem errors, and could use some performance improvement.   However, I wanted to test and follow the development progress, so I am using it on a system where data loss is not a concern.  I also wanted to use btrf's multi-device support on root.  My installation process:

• Boot from an Ubuntu 11.04 installation/Live CD. 
• Select Try Ubuntu
• Once the loaded, double-click the desktop icon Install Ubuntu 11.04
• Select appropriate settings language and packages
• For Allocate drive space, select Something else
• Create a new partition table on your first drive
• Click Add... to add partition
• Type for the new partition: Primary
• New partition size in megabytes: [preferably 8589 or more MB]
• Use as: btrfs journaling filesystem 
• Mount point: /
• Create additional partitions as desired, for swap space, data that you want to reside on a more filesystem, etc.
• Click Install Now

Once it has finished partitioning, open a terminal (Applications -> Accessories -> Terminal). Stop the installation, copy the partition table from the first drive to the second (my example: first = /dev/sda, second = /dev/sdb), and create the btrfs filesystem on / (my example: partition 1) with raid1 for both metadata and data:

$ sudo su
# pkill ubiquity
# sfdisk -d /dev/sda | sed -e s/sda/sdb/ | sfdisk --force /dev/sdb
# mkfs.btrfs -m raid1 -d raid1 /dev/sda1 /dev/sdb1
# exit

Note: Please be careful with the device names and partitions as to not destroy information on other disks and partitions you may have!

Restart the installation, and proceed using the first drive (be sure to not reformat!).  btrfs will mount and mirror regardless of the device selected as the target root.  Following the installation and initial reboot, install grub on the second harddrive so it will boot when the first drive fails:

$ sudo grub-install /dev/sdb
$ exit

Have fun testing snapshots, subvolumes, and more!

Deleting XenServer templates

In my quest to create Ubuntu templates, I ended up with a couple I wanted to delete, but couldn't using XenCenter, or the CLI for that matter.  This is an example of one of those little things that just eat at me, that "there just has to be a way" to do it.

So I figured out a way after several unsuccessful attempts.  My solution  to delete, destory, uninstall, or whatever you want to call it... convert it from a default template, then from a template to a VM, and finally destroy the VM:

# xe template-param-set other-config:default_template=false uuid=[uuid]
# xe template-param-set is-a-template=false uuid=[uuid]
# xe vm-destroy uuid=[uuid]

Where [uuid] is the uuid of the template.  Yessss!

Ubuntu 11.04 on Citrix XenServer 5.5, Part 5

This is part 5 (and final!) of a series on installing Ubuntu 11.04 (Natty Narwhal) on Citrix XenServer 5.5 in paravirtualization (PV) mode. [Part 1] [Part 2] [Part 3] [Part 4]

The last issue to resolve is detecting the hard drive at boot time. My recommendation is to install the kernel for virtual installations:

# apt-get install linux-image-virtual

IMPORTANT:  Because XenServer 5.5 is not compatible with grub 2, you will need to update the path to the kernel and initrd images of the VM (by modifying the PV-boot-loader-args parameter as shown in Part 4) for every kernel upgrade.  Verify the paths in /boot/grub/grub.cfg and update before rebooting the VM.

Another option is to add the xen block device driver to initrd:

# echo xen-blkfront >> /etc/initramfs-tools/modules
# update-initramfs -u

Your installation should be complete and booting without intervention.... now go have fun!

Ubuntu 11.04 on Citrix XenServer 5.5, Part 4

This is part 4 of a series on installing Ubuntu 11.04 (Natty Narwhal) on Citrix XenServer 5.5 in paravirtualization (PV) mode. [Part 1] [Part 2] [Part 3]

Sorry to have kept you waiting a few days because if you've finished your Ubuntu install and rebooted, the following error from pygrub occurred at VM startup:

RuntimeError: Unable to find partition containing kernel

This is due to pygrub in XenServer 5.5 not supporting grub 2, which is used by Ubuntu 11.04. To bypass pygrub trying to determine the boot configuration, the kernel location and arguments need to be supplied. Determine this by editing the bootloader configuration:

# xe-edit-bootloader -n [Disk Label] -p 1 -f /grub/grub.cfg

where [Disk Label] is the disk label, typically the vm name. It also assumes you took my advice in part 3 and made /boot the first and a separate partition, since "-p 1" specifies partition 1, and "-f" specifies the path to the grub.cfg relative to the partition. Otherwise you will need to determine the partition and path to /boot/grub/grub.cfg.

Find the first menuentry which should have entries similar to this:

linux /vmlinuz-2.6.38-8-generic root=UUID=5f05322a-a159-4604-9da9-905b0506d882 ro console=hvc0 quiet vt.handoff=7
initrd /initrd.img-2.6.38-8-generic

To modify boot parameters, obtain and verify the VM's uuid:

# VMUUID=`xe vm-list name-label="[VM name]" params=uuid --minimal`
# echo $VMUUID

All of these commands may not be needed, but it is good practice to verify PV is configured properly. Supply kernel arguments exactly as shown in your grub.cfg menuentry (Note: your root UUID will differ!) and the paths to the kernel/ramdisk files:

# xe vm-param-clear uuid=$VMUUID param-name=HVM-boot-policy
# xe vm-param-set uuid=$VMUUID PV-bootloader=pygrub
# xe vm-param-set uuid=$VMUUID PV-args="root=UUID=5f05322a-a159-4604-9da9-905b0506d882 ro console=hvc0  splash quiet vt.handoff=7"
# xe vm-param-set uuid=$VMUUID PV-bootloader-args="--kernel=/vmlinuz-2.6.38-8-generic --ramdisk=/initrd.img-2.6.38-8-generic"

Obtain and verify the VM's disk VBD uuid:

# VBDUUID=`xe vm-disk-list uuid=$VMUUID | grep -A1 VBD | tail -n 1 | cut -d: -f2 | awk '{print $1}'`
# echo $VBDUUID

Set the VM's disk to be bootable:

# xe vbd-param-set uuid=$VBDUUID bootable=true

Unfortunately, even after all of this the system will fail to boot giving the error:

ALERT! /dev/disk/by-uuid/... does not exist.

and will drop you to a "BusyBox" shell.   The hard disk device driver does not load by default.  Resolve this by loading it manually at the prompt:

(initramfs) modprobe xen-blkfront
(initramfs) exit

Booyah! The system is finally booted!  Another workaround to the boot issues would be to configure the virtual system to boot in HVM mode temporarily:

# xe vm-param-set uuid=$VMUUID HVM-boot-policy=BIOS\ order

You would just need to re-run all the xe commands to reconfigure PV mode after determining the kernel path/arguments, and configuring it to load modules at boot time.

With the system up and running, I'll save cleaning up the boot process for the final part of this series.