Once in a while I come across a production network where there are some unexplained issues in the environment. Some (or a lot depending on the environment) can be because of time synchronization woes. Mostly because of undocumented or partly unconfigured time services.
Why is this an issue?
Time is inherently important to the function of routers, networks, servers, clusters, applications, storage or name it. Without synchronized time, accurately information between hosts becomes difficult, if not impossible. When it comes to analyzing and security, if you cannot successfully compare logs between each of devices, you will find it very hard to develop a reliable picture of an incident. Some or part of application clustering will fail. Application services are waiting on data packets that will not be processed. Locks are not removed in time. If an authentication time stamp coming from the client differs with more than default 5 minutes from a Domain Controllers time, it will discard the packet as fake (or think what your two factor will do). Storage controllers providing CIFS access need the same access to the directory as the previous client example. Performance data cannot be accurately interpreted if the time stamps on the data are not synchronized between the managed and management components. When wanting to present logs as proof, even if you are able to put the pieces together, not synchronized times, may give an attacker with a good attorney enough wiggle room to escape prosecution. These are just some of the things that can be affected with a mis-configured or not in place time synchronization.
Often there isn’t anything in the standard installation which gives the engineer a chance to setup NTP. Proper NTP usage only occurs when the engineer is knowledgeable of NTP and its administration, and has as standard practice of configuring NTP as one of the post-installation tasks. This is often not the case or is partly done and documented.
And if you don’t have a blue box around, Wibbly Wobbly Timey Wimey stuff is not what you want!
But what is in a NTP architecture?
NTP is designed to synchronize the time on a network of hosts. NTP runs over the User Datagram Protocol (UDP), using port 123 as both the source and destination.
A NTP network usually gets its time from an authoritative time source, such as a radio clock or an atomic clock attached to a time server (a stratum 1) or one or more trusted time servers in the hierarchy. NTP then distributes this time across the network. Depending on your network and available time sources several options of your NTP infrastructure can be done. But often you will see a hierarchical structure with external time source on the Internet (pool.org for example). Core infrastructure components (core router, firewall or Active Directory Domain Controller) have a client-server relationship with external time sources (and are allowed through the firewall), the internal NTP services have a client-server relationship with the internal servers, devices et al, the internal workstations have there time services in a client-server relationship with the time synchronized server (via Windows services in this example), and so on down the tree. A hierarchical structure is the preferred technique because it provides consistency, stability, and scalability.
Relations and configurations
In the virtual world there will be varying resources. On busy systems, resources may be denied for short periods of time or during high workloads. Some may receive higher resources. This will result in something referred to as time drifting, the clock ticks will sometimes run in a faster or slower pace creating an offset. This really show to need for time synchronization.
NTP prefers to have access to several sources of time (preferable three) since it can then apply an agreement algorithm. Normally, when all servers are in agreement, NTP chooses the best server in terms of lowest stratum, closest (in terms of network delay), and claimed precision.
For external NTP sources a path must exist from the trusted device to the external sources, firewall rule must allow this traffic. When using fully qualified domain names in the configuration (not always possible but preferable) the NTP client relies on the DNS client to resolve. DNS must be set-up correctly.
With a Windows Domain your Windows Members are automatically configured to use Windows Time Service to their domain controllers (that is with the NT5DS setting). The domain controller (or the PDC emulator to be precise) needs to be manually changed to type NTP and setup with a peer list.
In other devices it is a configuration tab you can set from the web interface, other have CLI in a Linux or other OS. Linux can be setup to use the NTP daemon.
For VMware VM’s use the guest operating system time synchronization such as Windows w32tm or NTP, and not use VMware Tools time synchronization.
Be ware that all your devices, hosts, guests, database servers, application, services et al are working together and need to have the same reliable source of time. As a rule you will have to set up your time service, this is not done for you.
Set it up correctly and document for reference.
– Happy timey wimey!