DSM-G600, DNS-3xx and NSA-220 Hack Forum

I recently stumbled onto the DNS-323 "uptime" bug. See this thread for general discussion and some scripts that attempt to workaround this issue: http://dns323.kood.org/forum/t162-Wrong-uptime.html

I was curious about the issue and the root cause so I dug around a bit. I'm posting in this forum area since the details are gory and the real fix would be in a custom kernel.

Symptoms:

1) uptime is broken (reports ridiculously high values); this is due to bad data in /proc/uptime; 14313+ days on my machine
2) /proc/stat btime is similarly broken (reported as 0 - 2 on my machine with ntpd running)

Background:

The Linux 2.6 kernel (at least 2.6.12.6) internally keeps a monotonic clock as well as the current system time in nanoseconds. The monotonic clock is suppose to be initialized to zero at system boot while the system time is set based on the RTC. Later programs like ntpd may adjust the system time to match it with "true" time.

Two key kernel variables xtime and wall_to_monotonic are used for maintaining the system time and the monotonic clock. Basically xtime keeps track of the system time and wall_to_monotonic is an offset to convert system time to monotonic time. wall_to_monotonic should be negative as the offset is added roughly as follows (nanosecond precision details are ignored here):

MONOTONIC_TIME = xtime + wall_to_monotonic; // Zero at system boot

An examination of the procfs code shows that the  two problematic values noted in /proc are computed roughly as follows:

btime = -wall_to_monotonic;
uptime = xtime + wall_to_monotonic;

These details indicate that btime is being set very low which in turn results in uptime roughly equallying system time. So the DNS-323 thinks it has been up since 1970!

One interesting point is that system time can change while a monotonic clock cannot. Tracing kernel code reveals that flows that update the system clock make corresponding adjustments to wall_to_monotonic to ensure that it remains correct with regard to its original reference point. Monotonic time is also used to track the start time (as shown by ps or /proc/<pid>/stat field #22) of processes. So you'll notice very large values in /proc/<pid>/stat too. However general use of the monotonic clock is not broken; just cases that assume boot time = zero on the monotonic clock (there are a few places in the kernel that look at this stuff).

The Bug:

Given that xtime and wall_to_monotonic are kept in sync everything looks fine. It turns out that the problem is during kernel initialization. General references refer to utilities like hwclock running early in the kernel init process (I've seen some kernels like this where logging starts near 1970 and then suddenly jumps forward). However on the Marvell kernel for the DNS-323 it doesn't work like this. hwclock would certainly go through calls that would update both xtime and wall_to_monotonic. Both values initialize to zero (which is fine if your computer is really starting on Jan 1, 1970).

A quick grep of 'arch/arm/mach-mv88fxx81' for "xtime" turns up: /linux-2.6.12.6/arch/arm/mach-mv88fxx81/LSP/time.c

static int mv_rtc_init(void)
{
        MV_RTC_TIME time;
        mvRtcDS1339TimeGet(&time);

        /* same as in the U-Boot we use the year for century 20 only */
        xtime.tv_sec = mktime ( time.year + 2000, time.month,
                                time.date, time.hours,
                                time.minutes, time.seconds);

        to_tm(xtime.tv_sec, &time);
        set_rtc = mv_set_rtc;
        register_rtc(&rtc_ops);
        return 0;
}

This code appears broken because it will update the xtime value (presumably from near zero) to the current time read from the RTC but it does NOT adjust wall_to_monotonic.

This results in wall_to_monotonic staying at zero (until subsequent time updates by ntpd and such may move it a small amount depending on how far off the RTC is). Once this damage is done there is no clean way to fix it since all of the normal kernel functions for adjusting the time deliberately correct the delta between wall_to_monotonic and xtime.

It looks like this would be very easy to fix in a custom kernel. Just set wall_to_monotonic = -xtime above. Boot time would then be fixed at the time the RTC was read above; presumably very early in kernel initialization. I have not pursued this approach as I'm not set up for testing custom kernels (and have B1 hardware so I think there are issue with ffp-reload; I haven't looked much though).

Experimental Workaround:

I decided to experiment with a workaround just "for fun". I wrote the attached kernel module which adjusts the wall_to_monotonic value back to a reasonable boot time.

There are a few issues:

1) wall_to_monotonic is not exported; the workaround uses an nasty hack that assumes the address is fixed for the stock kernel (and relative to the exported xtime address)
2) The actual boot time is not known. The workaround adds up current CPU time (not sure if that code is right) to make a guess.
3) This hack makes the monotonic clock jump WAY back. Not so good for a monotonic clock.
3a) This makes process start times occur way in the future. The module corrects for this.
3b) The module does NOT handle existing kernel timers (like clock_was_set() does). This could be really bad. I don't know the Linux kernel enough to really know. It might be possible to replicate more code from clock_was_set; but likely more hacking around not-exported things.
3c) Any "other" userland, etc consumers of the monotonic clock may not like it jumping backward. Risk on DNS-323 unknown.

Anyway; coding the module was an interesting learning experience and it seems to work in practice. No crashes and no observable bad symptoms. Uptime works as expected, process start times look good, etc.

I wrote this up to share the info and also help anyone who might be patching the stock kernel; a patch for time.c above looks pretty simple. This should also dispel some of the confusion on exactly what is wrong with uptime.

I'm thinking of an alternative less intrusive fix but haven't investigated it yet. It probably wouldn't be clean either; but a kernel module that replaced the handling of /proc/uptime and /proc/stat could fix the user-mode cosmetic issues without causing the monotonic clock to jump forward. While I think kernel modules can add to /proc; I doubt it's clean to replace existing handlers.

-Jeff

---

Attachments:
fixdns323uptime.c, Size: 4,110 bytes, Downloads: 644