Re: Bad clock setting hosed our entire domain

With this fix all computers will not accept any time that is greater than what you specify. So if one of your DC's has a bad CMOS battery it will not replicate to any other computer in your network. Same thing if the time server your PDCe is pointing to has the incorrect time, it will not update. So you will only have to worry about the DC that has the bad CMOS battery and fix that not all DC's.




Jeff wrote:
Ken wrote:
Check out this link on how to prevent this and it explains what happen. http://support.microsoft.com/kb/884776/en-us

Good enough. But what if our problem was caused by a bogus time jump in the CMOS clock? Then NTP would fail to bring it back to normal. If we were to use a hardware clock attached to a serial port, is there a way to make sure Windows syncs to that clock before it serves data to the domain?

Jeff wrote:
We have 5 W2K3 domain controllers. On the same night as the recent daylight saving change, our PDC apparently suffered a disk problem and rebooted. It was soon discovered that the date on the PDC was February 11, 2009 and not November 2, 2008.

After that was discovered, the clock was reset, but it was too late. The other DCs had stopped replicating, even with each other, and some even refused to serve as an authoritative source for logins. Even AD integrated DNS stopped working in some instances. EVERYTHING was screwed up.

Our event logs were overflowing with errors, most of them Kerberos related, which is no surprise with the time problem. But we were unable to resolve anything. The most frequent event was:

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 4
Date: 11/2/2008
Time: 10:18:49 AM
User: N/A
Computer: XXXXXXXX
Description:
The kerberos client received a KRB_AP_ERR_MODIFIED error from the server host/xxx.xxx.com. The target name used was . This indicates that the password used to encrypt the kerberos service ticket is different than that on the target server. Commonly, this is due to identically named machine accounts in the target realm (XXX.COM), and the client realm. Please contact your system administrator.

====

No useful information on this was found on the net. In fact searching for KRB_AP_ERR_MODIFIED resulted in NO results from microsoft.com.

Some web pages suggesting resetting machine accounts with "netdom resetpwd", but that procedure would not complete successfully. We tried demoting a DC, but it would not work because it could not communicate with the others.

Being Sunday afternoon and knowing that we could be there all night, we decided to go for the sure thing. We kept one DC online, seized the FSMO roles, then reinstalled Windows and AD on the other machines. What a pain. Good thing we don't have 100 domain controllers.

So, my question: What in the world could have happened and how could such a simple thing as a time skew trigger it? Was there an easy fix that we missed? Can we protect ourselves against this in the future?

Of course, you don't really have enough information to answer these questions, but at this point I will accept wild guesses.

Thanks,

Jeff

.

Relevant Pages

  • Re: Kernel clock issue "Clocksource tsc unstable"
    ... Their clocks stop updating. ... The only fix is a hard ... wheras hpet needs to access a chip on the mobo). ... cause your clock to stop updating however, is a mystery, unless the acpi ...
    (Ubuntu)
  • Re: OT: Re: Computer loosing time
    ... Time on the desktops are configured for ntp via pool.ntp.org servers, ... comments regarding the CMOS battery. ... dead CMOS battery _Ubuntu_ will run fine. ... My CMOS clock and battery are both perfectly fine. ...
    (Ubuntu)
  • ntpd and cmos clock update
    ... I think I saw more than once speculations that FreeBSD updates CMOS ... clock when time is set, so CMOS clock value should always be very close ... I believe that ntpd calls settime-family functions only if time offset ...
    (freebsd-hackers)
  • Re: Is the x86 clock stoppable?
    ... At the time I was told that CMOS ... to be used instead of the power or reset button as ... If you want a stop clock x86, ...
    (comp.arch)
  • Re: Problem with Invalid System Time dialog preventing autologon.
    ... The problem is that this is a system which we sell to customers, using of the shelf hardware. ... So I'd rather fix this in software instead of modifying every single board we deliver. ... CMOS clock during POST. ... If you compared the time/date seen in the computer's CMOS ram utility ...
    (microsoft.public.windowsxp.embedded)