NTP synchronisation in RHEL the right way

I'm managing about 28 servers at the moment, a developer came up to me this morning telling me his machine was more then one minute late. Auch, embarrassing situation here I thought. Did I do something wrong? All servers are synced with ntpd on the same central timeserver. How can it be that one server is way off? Digging the internet on so called experts didn't help much, most use ntpd and are done. So why is the server off?

Well it seems when a server is off too much on boot time, ntpd will just not work anymore. That was a new for me. The catch is that servers don't sync on boot with ntp, they sync with the hwclock. If the clock is way off a certain threshold ntp will just silently fail. How can I fix the problem so the server will boot, set time correct and keep it correct?

Open your /etc/rc.local file and add ntpdate to it (use any time server you want), then sync the hwclock:

# sync time
ntpdate -sb pool.ntp.org 
hwclock --systohc

Now after boot, sysinit is run, then rc.local, the time is updated, the hwclock is synced, then the services in rc.d will start. Because we still use ntpd the server time will stay in sync till next reboot; where it is synced on boot et cetera.

Tnx to:
http://nuclearfusionreactor.blogspot.nl/
http://www.rgrjr.com/linux/ntp.html#ntpdate-boot