It turns out that the most common RTC is a tiny single-board, single chip solution that takes advantage of how the first four adjacent pins on the GPIO header are power, ground and i2c bus. Links:
The RTC board that I ordered. It seems that there are a couple of manufacturers doing this design and hard to tell who was first: https://www.sunfounder.com/ds3231-real-time-clock-module.html
One setup guide: http://www.raspberrypi-spy.co.uk/2015/05/adding-a-ds3231-real-time-clock-to-the-raspberry-pi/
Another setup guide: https://trick77.com/adding-ds3231-real-time-clock-raspberry-pi-3/
Here is a nice guide that shows configuration methods for both the old (modules) method and the new (Device Tree) method. It mentions steps for disabling the fake hwclock but it doesn't look like it shows the full procedure: http://www.bashpi.org/?page_id=500
Basically all the good guides are from this one google search: https://www.google.com/#q=sunfounder+ds3231+how+to+install
Having followed those installation guides, somehow it was coming up after rebooting with times that were only close to the current time although not reset to any particular default. It really seemed like the "fake hardware clock" was stepping on the rtc chip's time or something like that.
This site seemed to have the magic bullet for turning fake hwclock off, a beautiful single command which somehow alters all the rc.d files that had time setting stuff: http://forum.openmediavault.org/index.php/Thread/8770-RPi-2-RTC-module-from-unruly-Stepchild-to-Wunderkind/?pageNo=3
Specifically:
update-rc.d -f hwclock.sh remove
Unfortunately, after a few tests that seemed to show that the RTC was working, it stopped working. At first it looked like some tests that I had donewith an alternate power supply for the Pi might have damaged something, however a brand new sunfounder RTC continued to behave the same way. Upon reboot, the time would be set to 9:00 Dec 31, 1969 (upon reflection, this likely to be just midnight Jan 1 1970 GMT as viewed from EST although that doesn't quite add up right it should be 4 hours off not three), and attempting to read the hardware clock manually after a reboot resulted in the message "The hardware clock register contain values that are either invalid (e.g. 50th day of month) or beyond the range we can handle (e.g. Year 2095)."
A review of the messages file in /var/log seems to show that the chip is being registered as the hardware clock during bootup without any problems. /dev contains a device rtc0 with a simlink from /dev/rtc to /dev/rtc0.
A google search for the date 12/31/1969 shows a few people experiencing a similar problem on a variety of platforms with a variety of RTC chips.
A google search for the register message produces several other hits, one of which looks promising:
https://www.raspberrypi.org/forums/viewtopic.php?p=690492
Some suggestions that seem good that I haven't done yet are to turn off ntp:
update-rc.d ntp disable
fix /lib/udev/hwclock-set according to this link: https://www.raspberrypi.org/forums/viewtopic.php?f=44&t=16218&start=25
Also look for marcus15's post near the bottom of this page: https://www.raspberrypi.org/forums/viewtopic.php?p=690492
Furthermore, the same genius who began figuring things out in the above threads, marcus15, has worked through how to set things up for the Jessie distribution of Raspbian (which is what I have, so this applies in particular):
https://www.raspberrypi.org/forums/viewtopic.php?p=692662#p692662
The next to last entry on this page, by pemst, covers a lot of the same steps listed in marcus15's answer with some slight variations in technique: https://www.raspberrypi.org/forums/viewtopic.php?f=44&t=16218&start=100
Following the instructions in marcus15's and pemst's posts solved the problemThe problem seems to stem from a timing problem in the original configuration files in Raspbian, in that the usual setting in the hwclock.txt file to read the time from the hardware clock on boot is not being called before some code that decides that the time hasn't been read from the hardware clock and then writes crap to the RTC. The solution, aside from turning off fakehwclock and ntp, is to create a new service that reads the hwclock and then enable that service because apparently services are set up earlier in the boot sequence. Cutting and pasting from the forum threads above:
basic setup: Edit /boot/config.txt to include the following (notice unusual setup of i2c baud rate which I haven't seen anywhere else but which looks like a useful thing):
dtparam=i2c_arm=onNext create a service script named hwcock-start.service in /lib/systemd/system by copying hwclock-save.service and editing it to look like this. Note that the call to hwclock is more fancy that normal, with the -D option to print diagnostic to the log files in the event of a problem and the --hctosys option to use the local time zone.:
dtoverlay=i2c-rtc,ds3231
dtparam=i2c_baudrate=400000
[Unit]Then execute the commands to enable the new service. NTP off is shown below also for good measure.
Description=Synchronize Hardware Clock to System Clock
DefaultDependencies=no
After=sysinit.target
[Service]
Type=oneshot
ExecStart=/sbin/hwclock -D --hctosys
[Install]
WantedBy=graphical.target multi-user.target
sudo systemctl daemon-reloadThen set the system time and save it to rtc0 and on the next boot the time is read correctly from the RTC.
sudo systemctl enable hwclock-start
sudo systemctl disable ntp
sudo date -s "MM/DD/YYYY hh:mm"
sudo hwclock -w -f /dev/rtc0
