Discussion:
[lopsa-tech] Keeping time in sync in a VMware "cloud" environment
Mathew Snyder
2014-11-18 21:50:22 UTC
Permalink
We are currently in a situation where we are being pressured to re-engineer
our NTP service. We currently host it, along with other services, on
Windows DCs. Our initial plan is to move NTP off of those servers and host
it on dedicated Linux servers.

We likely won't get approval for hardware to host NTP and will thus have to
rely on VMs. This poses a few concerns such as: How will a Vmotion affect
the service? What happens if access to the sources that we use is lost (for
whatever reason)?

In the past, all of our VMs would run ntpd normally. That is, as a
constantly running daemon.. However, we found that time was drifting
significantly to the tune of several seconds a day on several servers. We
never figured out why it was happening. Instead, we found that using a cron
job which runs /usr/sbin/ntpd every five minutes kept time synced up
nicely. We haven't had any issues since.

However, now Red Hat is telling us we should (need) to be running ntpd as a
daemon because they are seeing timing issues. Interestingly, this was never
brought to the attention of us platform engineers so I don't know how bad
the problem is or how many servers are affected.

The problem could be VMware Tools conflicting with ntpd. But again, we
don't know what the problem is. Only that we have a workaround-type
solution that we're being told we have to replace.

This leads to my question to the list: those of you who have cloud
environments based on VMware solutions, how do you keep time in sync? What
issues have you encountered and how did you solve those problems? What can
you recommend for a virtualized NTP solution?


-Mathew

"When you do things right, people won't be sure you've done anything at
all." - God; Futurama

"We'll get along much better once you accept that you're wrong and neither
am I." - Me
Matt Simmons
2014-11-18 21:55:34 UTC
Permalink
As of a major version ago on ESXi, it was considered best practice to run
ntpdate in a cron rather than ntpd on VMs, particularly if you weren't
using the most up-to-date VMware tools (and I believe they recommended the
VMware version, rather than open-vm tools).

I've also seen that behavior when a VM Host had a screwed up NTP config.

I haven't noticed any significant clock skew during a vmotion, but I've
never checked it. I can give it a shot if you'd like.

--Matt
Post by Mathew Snyder
We are currently in a situation where we are being pressured to
re-engineer our NTP service. We currently host it, along with other
services, on Windows DCs. Our initial plan is to move NTP off of those
servers and host it on dedicated Linux servers.
We likely won't get approval for hardware to host NTP and will thus have
to rely on VMs. This poses a few concerns such as: How will a Vmotion
affect the service? What happens if access to the sources that we use is
lost (for whatever reason)?
In the past, all of our VMs would run ntpd normally. That is, as a
constantly running daemon.. However, we found that time was drifting
significantly to the tune of several seconds a day on several servers. We
never figured out why it was happening. Instead, we found that using a cron
job which runs /usr/sbin/ntpd every five minutes kept time synced up
nicely. We haven't had any issues since.
However, now Red Hat is telling us we should (need) to be running ntpd as
a daemon because they are seeing timing issues. Interestingly, this was
never brought to the attention of us platform engineers so I don't know how
bad the problem is or how many servers are affected.
The problem could be VMware Tools conflicting with ntpd. But again, we
don't know what the problem is. Only that we have a workaround-type
solution that we're being told we have to replace.
This leads to my question to the list: those of you who have cloud
environments based on VMware solutions, how do you keep time in sync? What
issues have you encountered and how did you solve those problems? What can
you recommend for a virtualized NTP solution?
-Mathew
"When you do things right, people won't be sure you've done anything at
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither am I." - Me
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Chris Lindbergh
2014-11-18 22:07:49 UTC
Permalink
Good question Matthew!

Here's how we tackled NTP & vCloud/ESXi at $job-1:

As our ESXi hosts were authenticating off an Active Directory domain, and
best practices for AD is to point NTP to your Domain Controllers (or
whatever the DC's are using if they don't run NTP themselves) to ensure
authentication doesn't break due to time skew, we pointed each host to the
DCs for NTP.

VMs in our public cloud environment required VMware Tools to be installed,
so we were able to force each VM to synchronize its time with the ESXi host
that it lived on, via Edit Settings.

We had hardware devices (Cisco routers) that we enabled NTP servers on, and
pointed our DCs to those. These routers themselves looked to the U.S. pool
of ntp.org for proper time.

This way NTP clients on the domain could get time from the DCs, and clients
not on the domain could talk to the routers.

If cloud customers ran ntpdate or ntpd or any other NTP client/server, we
figured it should stay very close to what we were setting it to, and this
was the case in our tests.
Post by Matt Simmons
As of a major version ago on ESXi, it was considered best practice to run
ntpdate in a cron rather than ntpd on VMs, particularly if you weren't
using the most up-to-date VMware tools (and I believe they recommended the
VMware version, rather than open-vm tools).
I've also seen that behavior when a VM Host had a screwed up NTP config.
I haven't noticed any significant clock skew during a vmotion, but I've
never checked it. I can give it a shot if you'd like.
--Matt
Post by Mathew Snyder
We are currently in a situation where we are being pressured to
re-engineer our NTP service. We currently host it, along with other
services, on Windows DCs. Our initial plan is to move NTP off of those
servers and host it on dedicated Linux servers.
We likely won't get approval for hardware to host NTP and will thus have
to rely on VMs. This poses a few concerns such as: How will a Vmotion
affect the service? What happens if access to the sources that we use is
lost (for whatever reason)?
In the past, all of our VMs would run ntpd normally. That is, as a
constantly running daemon.. However, we found that time was drifting
significantly to the tune of several seconds a day on several servers. We
never figured out why it was happening. Instead, we found that using a cron
job which runs /usr/sbin/ntpd every five minutes kept time synced up
nicely. We haven't had any issues since.
However, now Red Hat is telling us we should (need) to be running ntpd as
a daemon because they are seeing timing issues. Interestingly, this was
never brought to the attention of us platform engineers so I don't know how
bad the problem is or how many servers are affected.
The problem could be VMware Tools conflicting with ntpd. But again, we
don't know what the problem is. Only that we have a workaround-type
solution that we're being told we have to replace.
This leads to my question to the list: those of you who have cloud
environments based on VMware solutions, how do you keep time in sync? What
issues have you encountered and how did you solve those problems? What can
you recommend for a virtualized NTP solution?
-Mathew
"When you do things right, people won't be sure you've done anything at
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither am I." - Me
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
--
##### "The compassionate have no enemies, the wise have no worries."
##### - Jing-si Aphorism
##### http://kso.cc
Mathew Snyder
2014-11-18 22:15:58 UTC
Permalink
Post by Chris Lindbergh
VMs in our public cloud environment required VMware Tools to be
installed, so we were able to force each VM to synchronize its time with
the ESXi host that it lived on, via Edit Settings.

Has this ever been an issue? VMware recommends against using VMware Tools
to sync time:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427

"VMware recommends using NTP instead of VMware Tools periodic time
synchronization. NTP is an industry standard and ensures accurate
timekeeping in your guest."

I should point out that our environment is hosted. We don't have control
over how the ESXi hosts are configured, but we do believe they are syncing
to a source internal to the provider.


-Mathew

"When you do things right, people won't be sure you've done anything at
all." - God; Futurama

"We'll get along much better once you accept that you're wrong and neither
am I." - Me
Post by Chris Lindbergh
Good question Matthew!
As our ESXi hosts were authenticating off an Active Directory domain, and
best practices for AD is to point NTP to your Domain Controllers (or
whatever the DC's are using if they don't run NTP themselves) to ensure
authentication doesn't break due to time skew, we pointed each host to the
DCs for NTP.
VMs in our public cloud environment required VMware Tools to be installed,
so we were able to force each VM to synchronize its time with the ESXi host
that it lived on, via Edit Settings.
We had hardware devices (Cisco routers) that we enabled NTP servers on,
and pointed our DCs to those. These routers themselves looked to the U.S.
pool of ntp.org for proper time.
This way NTP clients on the domain could get time from the DCs, and
clients not on the domain could talk to the routers.
If cloud customers ran ntpdate or ntpd or any other NTP client/server, we
figured it should stay very close to what we were setting it to, and this
was the case in our tests.
Post by Matt Simmons
As of a major version ago on ESXi, it was considered best practice to run
ntpdate in a cron rather than ntpd on VMs, particularly if you weren't
using the most up-to-date VMware tools (and I believe they recommended the
VMware version, rather than open-vm tools).
I've also seen that behavior when a VM Host had a screwed up NTP config.
I haven't noticed any significant clock skew during a vmotion, but I've
never checked it. I can give it a shot if you'd like.
--Matt
Post by Mathew Snyder
We are currently in a situation where we are being pressured to
re-engineer our NTP service. We currently host it, along with other
services, on Windows DCs. Our initial plan is to move NTP off of those
servers and host it on dedicated Linux servers.
We likely won't get approval for hardware to host NTP and will thus have
to rely on VMs. This poses a few concerns such as: How will a Vmotion
affect the service? What happens if access to the sources that we use is
lost (for whatever reason)?
In the past, all of our VMs would run ntpd normally. That is, as a
constantly running daemon.. However, we found that time was drifting
significantly to the tune of several seconds a day on several servers. We
never figured out why it was happening. Instead, we found that using a cron
job which runs /usr/sbin/ntpd every five minutes kept time synced up
nicely. We haven't had any issues since.
However, now Red Hat is telling us we should (need) to be running ntpd
as a daemon because they are seeing timing issues. Interestingly, this was
never brought to the attention of us platform engineers so I don't know how
bad the problem is or how many servers are affected.
The problem could be VMware Tools conflicting with ntpd. But again, we
don't know what the problem is. Only that we have a workaround-type
solution that we're being told we have to replace.
This leads to my question to the list: those of you who have cloud
environments based on VMware solutions, how do you keep time in sync? What
issues have you encountered and how did you solve those problems? What can
you recommend for a virtualized NTP solution?
-Mathew
"When you do things right, people won't be sure you've done anything at
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither am I." - Me
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
--
##### "The compassionate have no enemies, the wise have no worries."
##### - Jing-si Aphorism
##### http://kso.cc
Mathew Snyder
2014-11-18 23:00:00 UTC
Permalink
I think I found the one you're referring to but I don't recall seeing
ntpdate mentioned.
VMWare has a big long KB article on all the tweaks you need to make to
your OS
for NTP depending on the OS and version. When you're in a shop like ours
that
runs 3 different RHEL versions with varying patch levels, it can be a bit
of a
nightmare to keep up with. I don't have the KB article handle, but I'm
sure you
can find it without too much trouble.
- Chris
Post by Mathew Snyder
Post by Chris Lindbergh
VMs in our public cloud environment required VMware Tools to be
installed, so we were able to force each VM to synchronize its time with
the ESXi host that it lived on, via Edit Settings.
Has this ever been an issue? VMware recommends against using VMware Tools
http://kb.vmware.com/selfservice/microsites/search.
do?language=en_US&cmd=displayKC&externalId=1006427
Post by Mathew Snyder
"VMware recommends using NTP instead of VMware Tools periodic time
synchronization. NTP is an industry standard and ensures accurate
timekeeping in your guest."
I should point out that our environment is hosted. We don't have control
over how the ESXi hosts are configured, but we do believe they are
syncing
Post by Mathew Snyder
to a source internal to the provider.
-Mathew
"When you do things right, people won't be sure you've done anything at
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither
Post by Mathew Snyder
am I." - Me
On Tue, Nov 18, 2014 at 12:07 PM, Chris Lindbergh <
Post by Chris Lindbergh
Good question Matthew!
As our ESXi hosts were authenticating off an Active Directory domain,
and
Post by Mathew Snyder
Post by Chris Lindbergh
best practices for AD is to point NTP to your Domain Controllers (or
whatever the DC's are using if they don't run NTP themselves) to ensure
authentication doesn't break due to time skew, we pointed each host to
the
Post by Mathew Snyder
Post by Chris Lindbergh
DCs for NTP.
VMs in our public cloud environment required VMware Tools to be
installed,
Post by Mathew Snyder
Post by Chris Lindbergh
so we were able to force each VM to synchronize its time with the ESXi
host
Post by Mathew Snyder
Post by Chris Lindbergh
that it lived on, via Edit Settings.
We had hardware devices (Cisco routers) that we enabled NTP servers on,
and pointed our DCs to those. These routers themselves looked to the
U.S.
Post by Mathew Snyder
Post by Chris Lindbergh
pool of ntp.org for proper time.
This way NTP clients on the domain could get time from the DCs, and
clients not on the domain could talk to the routers.
If cloud customers ran ntpdate or ntpd or any other NTP client/server,
we
Post by Mathew Snyder
Post by Chris Lindbergh
figured it should stay very close to what we were setting it to, and
this
Post by Mathew Snyder
Post by Chris Lindbergh
was the case in our tests.
Post by Matt Simmons
As of a major version ago on ESXi, it was considered best practice to
run
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
ntpdate in a cron rather than ntpd on VMs, particularly if you weren't
using the most up-to-date VMware tools (and I believe they recommended
the
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
VMware version, rather than open-vm tools).
I've also seen that behavior when a VM Host had a screwed up NTP
config.
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
I haven't noticed any significant clock skew during a vmotion, but I've
never checked it. I can give it a shot if you'd like.
--Matt
On Tue, Nov 18, 2014 at 4:50 PM, Mathew Snyder <
Post by Mathew Snyder
We are currently in a situation where we are being pressured to
re-engineer our NTP service. We currently host it, along with other
services, on Windows DCs. Our initial plan is to move NTP off of those
servers and host it on dedicated Linux servers.
We likely won't get approval for hardware to host NTP and will thus
have
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
Post by Mathew Snyder
to rely on VMs. This poses a few concerns such as: How will a Vmotion
affect the service? What happens if access to the sources that we use
is
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
Post by Mathew Snyder
lost (for whatever reason)?
In the past, all of our VMs would run ntpd normally. That is, as a
constantly running daemon.. However, we found that time was drifting
significantly to the tune of several seconds a day on several
servers. We
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
Post by Mathew Snyder
never figured out why it was happening. Instead, we found that using
a cron
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
Post by Mathew Snyder
job which runs /usr/sbin/ntpd every five minutes kept time synced up
nicely. We haven't had any issues since.
However, now Red Hat is telling us we should (need) to be running ntpd
as a daemon because they are seeing timing issues. Interestingly,
this was
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
Post by Mathew Snyder
never brought to the attention of us platform engineers so I don't
know how
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
Post by Mathew Snyder
bad the problem is or how many servers are affected.
The problem could be VMware Tools conflicting with ntpd. But again, we
don't know what the problem is. Only that we have a workaround-type
solution that we're being told we have to replace.
This leads to my question to the list: those of you who have cloud
environments based on VMware solutions, how do you keep time in sync?
What
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
Post by Mathew Snyder
issues have you encountered and how did you solve those problems?
What can
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
Post by Mathew Snyder
you recommend for a virtualized NTP solution?
-Mathew
"When you do things right, people won't be sure you've done anything
at
Post by Mathew Snyder
Post by Chris Lindbergh
Post by Matt Simmons
Post by Mathew Snyder
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither am I." - Me
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
--
##### "The compassionate have no enemies, the wise have no worries."
##### - Jing-si Aphorism
##### http://kso.cc
"The greatest pleasure in life is doing what people say you cannot do."
- Walter Bagehot
Chris Lindbergh
2014-11-18 23:53:43 UTC
Permalink
To the best of my knowledge, it has never been an issue.

Thanks for the link! I will update my mental records accordingly.
Post by Mathew Snyder
Post by Chris Lindbergh
VMs in our public cloud environment required VMware Tools to be
installed, so we were able to force each VM to synchronize its time with
the ESXi host that it lived on, via Edit Settings.
Has this ever been an issue? VMware recommends against using VMware Tools
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427
"VMware recommends using NTP instead of VMware Tools periodic time
synchronization. NTP is an industry standard and ensures accurate
timekeeping in your guest."
I should point out that our environment is hosted. We don't have control
over how the ESXi hosts are configured, but we do believe they are syncing
to a source internal to the provider.
-Mathew
"When you do things right, people won't be sure you've done anything at
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither am I." - Me
On Tue, Nov 18, 2014 at 12:07 PM, Chris Lindbergh <
Post by Chris Lindbergh
Good question Matthew!
As our ESXi hosts were authenticating off an Active Directory domain, and
best practices for AD is to point NTP to your Domain Controllers (or
whatever the DC's are using if they don't run NTP themselves) to ensure
authentication doesn't break due to time skew, we pointed each host to the
DCs for NTP.
VMs in our public cloud environment required VMware Tools to be
installed, so we were able to force each VM to synchronize its time with
the ESXi host that it lived on, via Edit Settings.
We had hardware devices (Cisco routers) that we enabled NTP servers on,
and pointed our DCs to those. These routers themselves looked to the U.S.
pool of ntp.org for proper time.
This way NTP clients on the domain could get time from the DCs, and
clients not on the domain could talk to the routers.
If cloud customers ran ntpdate or ntpd or any other NTP client/server, we
figured it should stay very close to what we were setting it to, and this
was the case in our tests.
Post by Matt Simmons
As of a major version ago on ESXi, it was considered best practice to
run ntpdate in a cron rather than ntpd on VMs, particularly if you weren't
using the most up-to-date VMware tools (and I believe they recommended the
VMware version, rather than open-vm tools).
I've also seen that behavior when a VM Host had a screwed up NTP config.
I haven't noticed any significant clock skew during a vmotion, but I've
never checked it. I can give it a shot if you'd like.
--Matt
Post by Mathew Snyder
We are currently in a situation where we are being pressured to
re-engineer our NTP service. We currently host it, along with other
services, on Windows DCs. Our initial plan is to move NTP off of those
servers and host it on dedicated Linux servers.
We likely won't get approval for hardware to host NTP and will thus
have to rely on VMs. This poses a few concerns such as: How will a Vmotion
affect the service? What happens if access to the sources that we use is
lost (for whatever reason)?
In the past, all of our VMs would run ntpd normally. That is, as a
constantly running daemon.. However, we found that time was drifting
significantly to the tune of several seconds a day on several servers. We
never figured out why it was happening. Instead, we found that using a cron
job which runs /usr/sbin/ntpd every five minutes kept time synced up
nicely. We haven't had any issues since.
However, now Red Hat is telling us we should (need) to be running ntpd
as a daemon because they are seeing timing issues. Interestingly, this was
never brought to the attention of us platform engineers so I don't know how
bad the problem is or how many servers are affected.
The problem could be VMware Tools conflicting with ntpd. But again, we
don't know what the problem is. Only that we have a workaround-type
solution that we're being told we have to replace.
This leads to my question to the list: those of you who have cloud
environments based on VMware solutions, how do you keep time in sync? What
issues have you encountered and how did you solve those problems? What can
you recommend for a virtualized NTP solution?
-Mathew
"When you do things right, people won't be sure you've done anything at
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither am I." - Me
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
--
##### "The compassionate have no enemies, the wise have no worries."
##### - Jing-si Aphorism
##### http://kso.cc
--
##### "The compassionate have no enemies, the wise have no worries."
##### - Jing-si Aphorism
##### http://kso.cc
Edward Ned Harvey (lopser)
2014-11-20 14:05:24 UTC
Permalink
On Behalf Of Chris Snell
VMWare has a big long KB article on all the tweaks you need to make to your
OS
for NTP depending on the OS and version. When you're in a shop like ours
that
runs 3 different RHEL versions with varying patch levels, it can be a bit of a
nightmare to keep up with. I don't have the KB article handle, but I'm sure
you
can find it without too much trouble.
I had a problem once with RHEL5 on virtualbox. But aside from that, have never had a single problem with time syncing via vmware-tools.

BTW, on debian/ubuntu systems, it's sufficient to "apt-get -y install open-vm-tools" instead of building vmware-tools the usual way.

Last I knew, redhat systems still need to build vmware-tools the usual way, but I haven't checked for the existence of open-vm-tools in rhel7. Maybe it exists now.
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
David Bronder
2014-11-20 16:37:18 UTC
Permalink
Post by Edward Ned Harvey (lopser)
BTW, on debian/ubuntu systems, it's sufficient to "apt-get -y install open-vm-tools" instead of building vmware-tools the usual way.
Last I knew, redhat systems still need to build vmware-tools the usual way, but I haven't checked for the existence of open-vm-tools in rhel7. Maybe it exists now.
As an alternative to installing/building VMware Tools through the image and
interface provided in ESXi and vCenter for various RHEL/CentOS, SUSE and
Ubuntu versions (those without open-vm-tools, which RHEL 7 now includes and
VMware supports the use of), you can look at the VMware Tools Operating
System Specific Packages (OSP).

https://www.vmware.com/support/packages

These are provided in the OSes' respective packaging formats (e.g. RPM). We
use these (sync'd into our RHN Satellite) for our RHEL environment so we can
manage the Tools through Satellite like most of our other patching. (Note
that the OSP Tools report in vCenter as "3rd party/unmanaged".)

(One caveat on the OSPs, though. VMware has had trouble over the years with
learning how to properly package and version their OSP RPMs, both within the
Tools for a particular ESXi or guest OS version and across ESXi or guest OS
versions, resulting in occasional conflicts between dissimilar packages with
identical RPM names and versions. They're getting better, but I think I'm
still seeing issues.)

As far as timekeeping, we've had no trouble with running ntpd normally in our
RHEL VMs lately. Some years back, we had issues (probably ESX 3.x but maybe
also 4.x, and RHEL 3 or 4, and maybe early 5, but details are fuzzy now),
which we worked around with tips from the (now old) VMware whitepaper on
timekeeping with Linux VMs (presumably since replaced by the
already-mentioned KB article). Definitely no problems with later RHEL 5 or
newer on ESXi 5.0/5.5, though.

=Dave
--
Hello World. David Bronder - Systems Architect
Segmentation Fault ITS-EI, Univ. of Iowa
Core dumped, disk trashed, quota filled, soda warm. david-***@uiowa.edu
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Mathew Snyder
2014-11-20 19:41:49 UTC
Permalink
It seems the consensus is to abide by the recommended approach (both by
VMware and Red Hat) and use ntpd as intended. That is to say, a constantly
running daemon. What we'll have to work out is the server-side of it.
Chances of getting hardware just to run NTP are slim and we don't have
control over how the VMware hosts are configured so we can only focus on
how we configure our VMs. One would think we'd be able to just point to the
NTP servers the provider uses, but for some reason or another that isn't an
option. I may have to revisit it, though.

-Mathew

"When you do things right, people won't be sure you've done anything at
all." - God; Futurama

"We'll get along much better once you accept that you're wrong and neither
am I." - Me
Post by Edward Ned Harvey (lopser)
Post by Edward Ned Harvey (lopser)
BTW, on debian/ubuntu systems, it's sufficient to "apt-get -y install
open-vm-tools" instead of building vmware-tools the usual way.
Post by Edward Ned Harvey (lopser)
Last I knew, redhat systems still need to build vmware-tools the usual
way, but I haven't checked for the existence of open-vm-tools in rhel7.
Maybe it exists now.
As an alternative to installing/building VMware Tools through the image and
interface provided in ESXi and vCenter for various RHEL/CentOS, SUSE and
Ubuntu versions (those without open-vm-tools, which RHEL 7 now includes and
VMware supports the use of), you can look at the VMware Tools Operating
System Specific Packages (OSP).
https://www.vmware.com/support/packages
These are provided in the OSes' respective packaging formats (e.g. RPM).
We
use these (sync'd into our RHN Satellite) for our RHEL environment so we can
manage the Tools through Satellite like most of our other patching. (Note
that the OSP Tools report in vCenter as "3rd party/unmanaged".)
(One caveat on the OSPs, though. VMware has had trouble over the years with
learning how to properly package and version their OSP RPMs, both within the
Tools for a particular ESXi or guest OS version and across ESXi or guest OS
versions, resulting in occasional conflicts between dissimilar packages with
identical RPM names and versions. They're getting better, but I think I'm
still seeing issues.)
As far as timekeeping, we've had no trouble with running ntpd normally in our
RHEL VMs lately. Some years back, we had issues (probably ESX 3.x but maybe
also 4.x, and RHEL 3 or 4, and maybe early 5, but details are fuzzy now),
which we worked around with tips from the (now old) VMware whitepaper on
timekeeping with Linux VMs (presumably since replaced by the
already-mentioned KB article). Definitely no problems with later RHEL 5 or
newer on ESXi 5.0/5.5, though.
=Dave
--
Hello World. David Bronder - Systems Architect
Segmentation Fault ITS-EI, Univ. of Iowa
Core dumped, disk trashed, quota filled, soda warm.
Edward Ned Harvey (lopser)
2014-11-21 19:00:07 UTC
Permalink
Post by Mathew Snyder
What we'll have to work out is the server-side of it.
Chances of getting hardware just to run NTP are slim
If you have only a handful of machines, you can just point them to the appropriate variant of pool.ntp.org.

If you have a whole mess of machines, you might not want to abuse the network that way - So you create a couple of VM's to act as time servers, which either use a hardware time source or pull from something upstream (such as pool.ntp.org). You should not have any trouble running the NTP server in a VM; as long as you acknowledge that it's yet another stratum separation so you might end up a little bit off.
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Mathew Snyder
2014-11-18 22:08:18 UTC
Permalink
That's good to hear. It at least validates what we are doing to some
extent. Do you happen to know which documentation makes that
recommendation? Everything we've found so far never mentions ntpdate.

Side note, I find it odd that VMware recommends against using their tools
for time sync. Why include it then?

-Mathew

"When you do things right, people won't be sure you've done anything at
all." - God; Futurama

"We'll get along much better once you accept that you're wrong and neither
am I." - Me
Post by Matt Simmons
As of a major version ago on ESXi, it was considered best practice to run
ntpdate in a cron rather than ntpd on VMs, particularly if you weren't
using the most up-to-date VMware tools (and I believe they recommended the
VMware version, rather than open-vm tools).
I've also seen that behavior when a VM Host had a screwed up NTP config.
I haven't noticed any significant clock skew during a vmotion, but I've
never checked it. I can give it a shot if you'd like.
--Matt
Post by Mathew Snyder
We are currently in a situation where we are being pressured to
re-engineer our NTP service. We currently host it, along with other
services, on Windows DCs. Our initial plan is to move NTP off of those
servers and host it on dedicated Linux servers.
We likely won't get approval for hardware to host NTP and will thus have
to rely on VMs. This poses a few concerns such as: How will a Vmotion
affect the service? What happens if access to the sources that we use is
lost (for whatever reason)?
In the past, all of our VMs would run ntpd normally. That is, as a
constantly running daemon.. However, we found that time was drifting
significantly to the tune of several seconds a day on several servers. We
never figured out why it was happening. Instead, we found that using a cron
job which runs /usr/sbin/ntpd every five minutes kept time synced up
nicely. We haven't had any issues since.
However, now Red Hat is telling us we should (need) to be running ntpd as
a daemon because they are seeing timing issues. Interestingly, this was
never brought to the attention of us platform engineers so I don't know how
bad the problem is or how many servers are affected.
The problem could be VMware Tools conflicting with ntpd. But again, we
don't know what the problem is. Only that we have a workaround-type
solution that we're being told we have to replace.
This leads to my question to the list: those of you who have cloud
environments based on VMware solutions, how do you keep time in sync? What
issues have you encountered and how did you solve those problems? What can
you recommend for a virtualized NTP solution?
-Mathew
"When you do things right, people won't be sure you've done anything at
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither am I." - Me
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Brian Mathis
2014-11-19 02:19:15 UTC
Permalink
I don't believe ntpdate has ever been a recommendation. They started out
telling people to use the time sync through vmware tools (which is similar
to ntpdate and makes sense in a VMware Workstation environment), then went
to using ntpd running as a daemon, potentially with the kernel tuning
parameters.

Both vmware tools and ntpdate have the huge problems of jolting the clock
to the correct time and the possibility of time going backwards. An ntp
daemon, on the other hand, slows down and speeds up the clock to keep it in
sync. Anything sensitive to time is going to have a problem with methods
that jolt the time around.

Using ntpdate in cron is a duct tape solution and doesn't actually fix the
problem, it just covers it up. The only real solution is to follow that
VMware KB article and get ntpd tuned correctly. If that doesn't work, open
a VMware support ticket?

However, all of this is really a discussion about clients. Running an ntp
server from a VM will be a lot more problematic. You're best bet there is
to get a dedicated hardware time source, like a GPS or something.
Otherwise, the global ntp pool (http://www.pool.ntp.org) is not a bad
solution. It's relatively secure since you're querying a giant pool of
servers and it would be pretty amazing if all of them were compromised at
once to serve bad time.


❧ Brian Mathis
@orev
Post by Matt Simmons
As of a major version ago on ESXi, it was considered best practice to run
ntpdate in a cron rather than ntpd on VMs, particularly if you weren't
using the most up-to-date VMware tools (and I believe they recommended the
VMware version, rather than open-vm tools).
I've also seen that behavior when a VM Host had a screwed up NTP config.
I haven't noticed any significant clock skew during a vmotion, but I've
never checked it. I can give it a shot if you'd like.
--Matt
Post by Mathew Snyder
We are currently in a situation where we are being pressured to
re-engineer our NTP service. We currently host it, along with other
services, on Windows DCs. Our initial plan is to move NTP off of those
servers and host it on dedicated Linux servers.
We likely won't get approval for hardware to host NTP and will thus have
to rely on VMs. This poses a few concerns such as: How will a Vmotion
affect the service? What happens if access to the sources that we use is
lost (for whatever reason)?
In the past, all of our VMs would run ntpd normally. That is, as a
constantly running daemon.. However, we found that time was drifting
significantly to the tune of several seconds a day on several servers. We
never figured out why it was happening. Instead, we found that using a cron
job which runs /usr/sbin/ntpd every five minutes kept time synced up
nicely. We haven't had any issues since.
However, now Red Hat is telling us we should (need) to be running ntpd as
a daemon because they are seeing timing issues. Interestingly, this was
never brought to the attention of us platform engineers so I don't know how
bad the problem is or how many servers are affected.
The problem could be VMware Tools conflicting with ntpd. But again, we
don't know what the problem is. Only that we have a workaround-type
solution that we're being told we have to replace.
This leads to my question to the list: those of you who have cloud
environments based on VMware solutions, how do you keep time in sync? What
issues have you encountered and how did you solve those problems? What can
you recommend for a virtualized NTP solution?
-Mathew
"When you do things right, people won't be sure you've done anything at
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither am I." - Me
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Mathew Snyder
2014-11-19 03:16:28 UTC
Permalink
With the exception of a handful of RHEL 5.8 VMs we run RHEL 6.4 and 6.5.
According to VMware no additional tuning parameters should be needed.

-Mathew

"When you do things right, people won't be sure you've done anything at
all." - God; Futurama

"We'll get along much better once you accept that you're wrong and neither
am I." - Me

On Tue, Nov 18, 2014 at 4:19 PM, Brian Mathis <
Post by Brian Mathis
I don't believe ntpdate has ever been a recommendation. They started out
telling people to use the time sync through vmware tools (which is similar
to ntpdate and makes sense in a VMware Workstation environment), then went
to using ntpd running as a daemon, potentially with the kernel tuning
parameters.
Both vmware tools and ntpdate have the huge problems of jolting the clock
to the correct time and the possibility of time going backwards. An ntp
daemon, on the other hand, slows down and speeds up the clock to keep it in
sync. Anything sensitive to time is going to have a problem with methods
that jolt the time around.
Using ntpdate in cron is a duct tape solution and doesn't actually fix the
problem, it just covers it up. The only real solution is to follow that
VMware KB article and get ntpd tuned correctly. If that doesn't work, open
a VMware support ticket?
However, all of this is really a discussion about clients. Running an ntp
server from a VM will be a lot more problematic. You're best bet there is
to get a dedicated hardware time source, like a GPS or something.
Otherwise, the global ntp pool (http://www.pool.ntp.org) is not a bad
solution. It's relatively secure since you're querying a giant pool of
servers and it would be pretty amazing if all of them were compromised at
once to serve bad time.
❧ Brian Mathis
@orev
Post by Matt Simmons
As of a major version ago on ESXi, it was considered best practice to run
ntpdate in a cron rather than ntpd on VMs, particularly if you weren't
using the most up-to-date VMware tools (and I believe they recommended the
VMware version, rather than open-vm tools).
I've also seen that behavior when a VM Host had a screwed up NTP config.
I haven't noticed any significant clock skew during a vmotion, but I've
never checked it. I can give it a shot if you'd like.
--Matt
Post by Mathew Snyder
We are currently in a situation where we are being pressured to
re-engineer our NTP service. We currently host it, along with other
services, on Windows DCs. Our initial plan is to move NTP off of those
servers and host it on dedicated Linux servers.
We likely won't get approval for hardware to host NTP and will thus have
to rely on VMs. This poses a few concerns such as: How will a Vmotion
affect the service? What happens if access to the sources that we use is
lost (for whatever reason)?
In the past, all of our VMs would run ntpd normally. That is, as a
constantly running daemon.. However, we found that time was drifting
significantly to the tune of several seconds a day on several servers. We
never figured out why it was happening. Instead, we found that using a cron
job which runs /usr/sbin/ntpd every five minutes kept time synced up
nicely. We haven't had any issues since.
However, now Red Hat is telling us we should (need) to be running ntpd
as a daemon because they are seeing timing issues. Interestingly, this was
never brought to the attention of us platform engineers so I don't know how
bad the problem is or how many servers are affected.
The problem could be VMware Tools conflicting with ntpd. But again, we
don't know what the problem is. Only that we have a workaround-type
solution that we're being told we have to replace.
This leads to my question to the list: those of you who have cloud
environments based on VMware solutions, how do you keep time in sync? What
issues have you encountered and how did you solve those problems? What can
you recommend for a virtualized NTP solution?
-Mathew
"When you do things right, people won't be sure you've done anything at
all." - God; Futurama
"We'll get along much better once you accept that you're wrong and
neither am I." - Me
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Frank Bulk
2014-11-20 00:05:09 UTC
Permalink
Matt,



If time is drifting on the VMs then you’re not pointing them to NTP servers that are staying in sync. Note that you should be using at least four NTP servers per NTP element to ensure good reliability.



Frank



From: tech-***@lists.lopsa.org [mailto:tech-***@lists.lopsa.org] On Behalf Of Mathew Snyder
Sent: Tuesday, November 18, 2014 3:50 PM
To: LOPSA Tech
Subject: [lopsa-tech] Keeping time in sync in a VMware "cloud" environment



We are currently in a situation where we are being pressured to re-engineer our NTP service. We currently host it, along with other services, on Windows DCs. Our initial plan is to move NTP off of those servers and host it on dedicated Linux servers.



We likely won't get approval for hardware to host NTP and will thus have to rely on VMs. This poses a few concerns such as: How will a Vmotion affect the service? What happens if access to the sources that we use is lost (for whatever reason)?



In the past, all of our VMs would run ntpd normally. That is, as a constantly running daemon.. However, we found that time was drifting significantly to the tune of several seconds a day on several servers. We never figured out why it was happening. Instead, we found that using a cron job which runs /usr/sbin/ntpd every five minutes kept time synced up nicely. We haven't had any issues since.



However, now Red Hat is telling us we should (need) to be running ntpd as a daemon because they are seeing timing issues. Interestingly, this was never brought to the attention of us platform engineers so I don't know how bad the problem is or how many servers are affected.



The problem could be VMware Tools conflicting with ntpd. But again, we don't know what the problem is. Only that we have a workaround-type solution that we're being told we have to replace.



This leads to my question to the list: those of you who have cloud environments based on VMware solutions, how do you keep time in sync? What issues have you encountered and how did you solve those problems? What can you recommend for a virtualized NTP solution?





-Mathew

"When you do things right, people won't be sure you've done anything at all." - God; Futurama



"We'll get along much better once you accept that you're wrong and neither am I." - Me
Robert Hajime Lanning
2014-11-20 00:54:03 UTC
Permalink
If time is drifting on the VMs then you’re not pointing them to NTP
servers that are staying in sync. Note that you should be using at
least four NTP servers per NTP element to ensure good reliability.
Sort of...

Time in a VM is a relative thing... The problem is that the timer
interrupt is virtual. It is not locked in step with an oscillator.

The kernel tries to figure out these missed/late interrupts and make
adjustments. The NTP daemon thinks the local clock is going haywire.

Add "tinker panic 0" to your VM ntpd instance.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427
--
Mr. Flibble
King of the Potato People
http://www.linkedin.com/in/RobertLanning
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Edward Ned Harvey (lopser)
2014-11-20 14:02:24 UTC
Permalink
Woah. That's a lot of wind. ;-) Here's what I've been doing for the last few years, that I'm completely happy with:

I configure the vmware hosts to get NTP from the internet (or whatever source you prefer.) (Configuration / Time Configuration / NTP Client).

Then, Edit Settings of each guest, and go to Options/VMWare Tools, and there's a checkbox for "[X] Synchronize guest time with host" which is not checked by default. Make sure vmware-tools is running in your guest. And disable any NTP client in the guest. Easy peasy.

The AD machines can be guests, and they can serve time to all the AD clients on the network.

By doing it this way, everybody will be in-sync +/- a few seconds, which is good enough for most purposes. But if you have more stringent requirements, I guess you wouldn't need to have this discussion at all. ;-)



From: tech-***@lists.lopsa.org [mailto:tech-***@lists.lopsa.org] On Behalf Of Mathew Snyder
Sent: Tuesday, November 18, 2014 4:50 PM
To: LOPSA Tech
Subject: [lopsa-tech] Keeping time in sync in a VMware "cloud" environment

We are currently in a situation where we are being pressured to re-engineer our NTP service. We currently host it, along with other services, on Windows DCs. Our initial plan is to move NTP off of those servers and host it on dedicated Linux servers.

We likely won't get approval for hardware to host NTP and will thus have to rely on VMs. This poses a few concerns such as: How will a Vmotion affect the service? What happens if access to the sources that we use is lost (for whatever reason)?

In the past, all of our VMs would run ntpd normally. That is, as a constantly running daemon.. However, we found that time was drifting significantly to the tune of several seconds a day on several servers. We never figured out why it was happening. Instead, we found that using a cron job which runs /usr/sbin/ntpd every five minutes kept time synced up nicely. We haven't had any issues since.

However, now Red Hat is telling us we should (need) to be running ntpd as a daemon because they are seeing timing issues. Interestingly, this was never brought to the attention of us platform engineers so I don't know how bad the problem is or how many servers are affected.

The problem could be VMware Tools conflicting with ntpd. But again, we don't know what the problem is. Only that we have a workaround-type solution that we're being told we have to replace.

This leads to my question to the list: those of you who have cloud environments based on VMware solutions, how do you keep time in sync? What issues have you encountered and how did you solve those problems? What can you recommend for a virtualized NTP solution?


-Mathew

"When you do things right, people won't be sure you've done anything at all." - God; Futurama

"We'll get along much better once you accept that you're wrong and neither am I." - Me
Brandon Allbery
2014-11-20 14:05:42 UTC
Permalink
On Thu, Nov 20, 2014 at 2:02 PM, Edward Ned Harvey (lopser) <
Post by Edward Ned Harvey (lopser)
Woah. That's a lot of wind. ;-) Here's what I've been doing for the
And that VMware itself tells you *not* to do.
--
brandon s allbery kf8nh sine nomine associates
***@gmail.com ***@sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net
Edward Ned Harvey (lopser)
2014-11-20 14:12:06 UTC
Permalink
Post by Brandon Allbery
On Thu, Nov 20, 2014 at 2:02 PM, Edward Ned Harvey (lopser)
Woah.  That's a lot of wind.   ;-)  Here's what I've been doing for the last few
And that VMware itself tells you *not* to do.
Have a link?
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http
Edward Ned Harvey (lopser)
2014-11-20 14:15:27 UTC
Permalink
From: Edward Ned Harvey (lopser)
Post by Brandon Allbery
On Thu, Nov 20, 2014 at 2:02 PM, Edward Ned Harvey (lopser)
Woah.  That's a lot of wind.   ;-)  Here's what I've been doing for the last few
And that VMware itself tells you *not* to do.
Have a link?
Sorry, literally seconds later I found it myself. In the aforementioned
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427

It says:
"VMware recommends using NTP instead of VMware Tools periodic time synchronization."

And Brian said:
"Both vmware tools and ntpdate have the huge problems of jolting the clock to the correct time and the possibility of time going backwards."

So... Ok, thanks guys. I'm convinced. Time to change.
Alexander Lobodzinski
2014-11-24 21:44:54 UTC
Permalink
[...] This poses a few concerns such as: How will a Vmotion affect
the service?
We sync the ESXi hosts themselves and all VMs (various CentOS 6.x,
CentOS 7, OpenBSD 5.x, some Windows) to a set of internal Stratum-2
servers that in turn sync to some Stratum-1 servers on the 'net.

ntpdate turned out to be unnecessary, synchronization on startup is very
quick if "iburst" is used in ntp.conf . So they run just ntpd and are
nicely synced with millisecond precision -- unless a VM gets VMotioned
which makes it lose a few clock ticks. Of course NTP soon gets a grip
on it but resynchronization takes some minutes.
We likely won't get approval for hardware to host NTP
From the above you see why I would recommend against running your
Stratum servers on VMs. Maybe you have the chance to use some Cisco or
Juniper? Re-commission an old one? If you can get approval for a
Raspberry Pi have a look here:
http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html

Ciao, Lobo
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Brandon Allbery
2014-11-24 21:59:32 UTC
Permalink
On Mon, Nov 24, 2014 at 4:44 PM, Alexander Lobodzinski <
Post by Alexander Lobodzinski
Of course NTP soon gets a grip
on it but resynchronization takes some minutes
Try "burst" instead of "iburst".
--
brandon s allbery kf8nh sine nomine associates
***@gmail.com ***@sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net
Tom Perrine
2014-11-25 17:23:02 UTC
Permalink
What I'm seeing is that for the purposes of NTP, you should pretty
much treat a virtual host the same as a physical host. This seems to
have changed from the olden days of VMware when you locked the
underlying physical server with NTP, then had the guests just trust
the presented clock as ground truth. This is new to me, and very
welcome. VMware vs. clock fights were always a royal pain. I'm glad
those are gone.

But in terms of time references, these days, I don't see much reason
to run my own primary clock, to be honest. The public pools are of
high enough quality that they should probably be considered a primary
reference for most peoples' purposes.

For 99% of the folks I've talked to over the years, the most important
thing about their time reference is that everything they control stay
sync'ed. Few of them have really truly cared if their entire
infrastructure was <1 millisecond off from the rest of the world.

The important thing is that their entire infrastructure slew together
if anything slews at all. As long as all your stuff stays synced, you
get the benefits that you're likely looking for: trustable timestamps
in logs, AD/Kerberos auth works, file timestamps are comparable across
all your systems, etc.

For that, building your own stratum of NTP servers that are based on
the public pools are fine, then sync everything off your own "top
level" stratum.

The farther you are from the "one true global top-level stratum", the
farther you might be from "atomic clock time", but everything that's
locked to your own internal top-level stratum should slew together, if
it slews at all.
Post by Brandon Allbery
On Mon, Nov 24, 2014 at 4:44 PM, Alexander Lobodzinski
Post by Alexander Lobodzinski
Of course NTP soon gets a grip
on it but resynchronization takes some minutes
Try "burst" instead of "iburst".
--
brandon s allbery kf8nh sine nomine associates
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Mathew Snyder
2014-11-25 23:49:04 UTC
Permalink
I agree for the most part. However, in the past we've seen time drift
significantly when running ntpd and syncing to VM-hosted NTP servers. This
is where my concerns stem from.

Any VM solution we implement will run RHEL6 and sync to the USNO.
Resiliency will be built in a far as we are able. After all, if external
network communication is lost we won't have a hardware clock to keep things
somewhat in line.
Post by Tom Perrine
What I'm seeing is that for the purposes of NTP, you should pretty
much treat a virtual host the same as a physical host. This seems to
have changed from the olden days of VMware when you locked the
underlying physical server with NTP, then had the guests just trust
the presented clock as ground truth. This is new to me, and very
welcome. VMware vs. clock fights were always a royal pain. I'm glad
those are gone.
But in terms of time references, these days, I don't see much reason
to run my own primary clock, to be honest. The public pools are of
high enough quality that they should probably be considered a primary
reference for most peoples' purposes.
For 99% of the folks I've talked to over the years, the most important
thing about their time reference is that everything they control stay
sync'ed. Few of them have really truly cared if their entire
infrastructure was <1 millisecond off from the rest of the world.
The important thing is that their entire infrastructure slew together
if anything slews at all. As long as all your stuff stays synced, you
get the benefits that you're likely looking for: trustable timestamps
in logs, AD/Kerberos auth works, file timestamps are comparable across
all your systems, etc.
For that, building your own stratum of NTP servers that are based on
the public pools are fine, then sync everything off your own "top
level" stratum.
The farther you are from the "one true global top-level stratum", the
farther you might be from "atomic clock time", but everything that's
locked to your own internal top-level stratum should slew together, if
it slews at all.
Post by Brandon Allbery
On Mon, Nov 24, 2014 at 4:44 PM, Alexander Lobodzinski
Post by Alexander Lobodzinski
Of course NTP soon gets a grip
on it but resynchronization takes some minutes
Try "burst" instead of "iburst".
--
brandon s allbery kf8nh sine nomine
associates
Post by Brandon Allbery
unix, openafs, kerberos, infrastructure, xmonad
http://sinenomine.net
Post by Brandon Allbery
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Brian Mathis
2014-11-29 06:35:19 UTC
Permalink
Its possible that your network routers or other hardware might provide NTP
service, so that could be an option, assuming they are actually synced to
something correctly.

Also, NTP can operate in peer mode where they all sync with each other, so
if any one or a few get wildly out of sync, they will be brought back in
line with the others. That doesn't mean they would all be correct, but
they would all be the same amount of wrong. That's probably a better
option in the event that all external sources are lost.


❧ Brian Mathis
@orev
Post by Mathew Snyder
I agree for the most part. However, in the past we've seen time drift
significantly when running ntpd and syncing to VM-hosted NTP servers. This
is where my concerns stem from.
Any VM solution we implement will run RHEL6 and sync to the USNO.
Resiliency will be built in a far as we are able. After all, if external
network communication is lost we won't have a hardware clock to keep things
somewhat in line.
Post by Tom Perrine
What I'm seeing is that for the purposes of NTP, you should pretty
much treat a virtual host the same as a physical host. This seems to
have changed from the olden days of VMware when you locked the
underlying physical server with NTP, then had the guests just trust
the presented clock as ground truth. This is new to me, and very
welcome. VMware vs. clock fights were always a royal pain. I'm glad
those are gone.
But in terms of time references, these days, I don't see much reason
to run my own primary clock, to be honest. The public pools are of
high enough quality that they should probably be considered a primary
reference for most peoples' purposes.
For 99% of the folks I've talked to over the years, the most important
thing about their time reference is that everything they control stay
sync'ed. Few of them have really truly cared if their entire
infrastructure was <1 millisecond off from the rest of the world.
The important thing is that their entire infrastructure slew together
if anything slews at all. As long as all your stuff stays synced, you
get the benefits that you're likely looking for: trustable timestamps
in logs, AD/Kerberos auth works, file timestamps are comparable across
all your systems, etc.
For that, building your own stratum of NTP servers that are based on
the public pools are fine, then sync everything off your own "top
level" stratum.
The farther you are from the "one true global top-level stratum", the
farther you might be from "atomic clock time", but everything that's
locked to your own internal top-level stratum should slew together, if
it slews at all.
Post by Brandon Allbery
On Mon, Nov 24, 2014 at 4:44 PM, Alexander Lobodzinski
Post by Alexander Lobodzinski
Of course NTP soon gets a grip
on it but resynchronization takes some minutes
Try "burst" instead of "iburst".
--
brandon s allbery kf8nh sine nomine
associates
Post by Brandon Allbery
unix, openafs, kerberos, infrastructure, xmonad
http://sinenomine.net
Post by Brandon Allbery
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Loading...