Least Squares Fit

I could have sworn that I'd created a post about the process of understanding how Tom used least squares fit to determine the slope of a set of data points. But I can't find it, so here is where I will go over the process in order.

First off, the concept of finding a least-squares fit can be understood simply as finding a linear function that most closely fits a set of points in (x,y) where for all (x,y), the sum of the squares of the differences of y=f(x) to the corresponding point (which is to say the errors) is a minimum.

This turns out to be something natural to solve using matrices, because the set of errors which is y - (mx + b) for all y's and x's turns out to be just a set of matrices:

[y0] - (m[x0] + b) = [e0]
[y1] - (m[x1] + b) = [e1]
. . . .
. . . .
[yn] - (m[xn] + b) = [en]

Some more links:

An example of a C program to compute standard deviation: http://alstatr.blogspot.com/2013/09/cc-variance-and-standard-deviation.html

An example of C code for computing the mean (as well as median and mode): http://stackoverflow.com/questions/20636914/c-mean-median-and-mode

A fairly obtuse presentation of least-squares-fitting basic concepts: http://mathworld.wolfram.com/LeastSquaresFitting.html

Matlab misc searches 2015

Various searches that I did while tweaking the Matlab demonstration algorithm from Tom:

A link about the matrix multiplication function in Matlab. The rule of thumb that I tried to remember about matrix multiplication is that the result is an array of the sums of multiplying each element of a row by the matching element of a column. This page also shows the syntax of how rows and columns are defined in Matlab, with columns being separated by semicolons and row elements being separated by spaceshttp://www.mathworks.com/help/matlab/ref/mtimes.html

A link about the matrix inverse function in Matlab. This page fails to reference the textbook definition of matrix inverse, but does have a couple of useful examples of using matrix inverse to solve linear equations. http://www.mathworks.com/help/matlab/ref/inv.html?refresh=true

The full tutorial on matrix indexing. The most important thing to remember is that the starting index for accessing matrixes in Matlab is 1, not 0. http://www.mathworks.com/help/matlab/math/matrix-indexing.html?refresh=true

A similar tutorial but this one is Array indexing: http://www.mathworks.com/help/matlab/learn_matlab/array-indexing.html

More on how to define matrixes: http://www.mathworks.com/help/stateflow/ug/how-to-define-vectors-and-matrices.html?refresh=true

How to do most common operations you'd want to do on matrixes, including defining, multiplying, inverting, concatinating, and printing: http://www.mathworks.com/help/matlab/learn_matlab/matrices-and-arrays.html

An answer to the question of how to format print output in Matlab, in this case how to format floating point number printing. I needed this while putting debug statements in the Matlab code, although it wasn't important for the final product: http://www.mathworks.com/matlabcentral/answers/99743-how-can-i-change-the-way-matlab-displays-the-significant-digits-of-floating-point-numbers

The matlab line continuation character (spoiler, it's weirdly an ellipsis, which is to say '...'): https://www.mathworks.com/matlabcentral/newsreader/view_thread/173340

The syntax for Matlab For Loops. TL;DR is if you have anything with a quantity, this allows you to perform an act on each item in the quantity: http://www.mathworks.com/matlabcentral/answers/31156-how-do-i-create-a-for-loop-in-matlab

The man page for one of MATLAB's most powerful function, one that gives you a matrix of all of the results of evaluating a polynomial on a matrix of inputs, POLYVAL. This is a very natural thing to use when computing least squares fits! http://www.mathworks.com/help/matlab/ref/polyval.html

Moving Thunderbird data to a new computer

As I've been chased from mail app to mail app and old computer to new computer at work, I've saved off my email into various formats. My current format is Thunderbird, which is actually a terrible choice because the mail is stored in non-human-readable "database" files that seem to make cabbage of the data inside them. Nonetheless, I have to accomplish the task of transplanting my Thunderbird files from my old computer to one of my new ones. So far, everything I've found seems to indicate that it's simply a matter of taking the database file from the old system and opening it on the new system. I wish I had more confidence in that assertion. Some links:

https://support.mozilla.org/en-US/kb/moving-thunderbird-data-to-a-new-computer

Ken Nordine Stare With Your Ears

Links for my search for more Stare With Your Ears tapes:

Ken Nordine on iTunes: https://itunes.apple.com/us/artist/ken-nordine/id5207291?trackPage=2#trackPage

A google group copy of alt.ken-nordine.word-jazz which contains an incredible 1996 listing of all the tapes, including the one that I have!!!!
https://groups.google.com/forum/#!topic/alt.ken-nordine.word-jazz/39wXAlB9HOE

Guy Carawan My Rincoeros

Here is where to order the CD copy of the tape that I got from Ruth Conley:

http://www.amazon.com/My-Rhinoceros-And-Other-Friends/dp/B0037TPI3K

http://www.gentlewind.com/gw1023.htm

pic microcontroller programming tutorial microchip

Some of my favorite tutorials for to program the PIC microcontroller. The TL;DR is figure out what microcontroller I have (it's a dsPIC30F6012A), download the IDE from Microchip, select the previous project, make changes and save it as a new project, hit compile until it doesn't fail compilation, plug in the IDE3 in circuit debugger, select it, find the refresh button to get the IDE to recognize the debugger cable, hit build for download which will do the download.

Some links:

An immense collection of tutorials for PIC16 microcontroller: http://pic-microcontroller.com/pic16-microcontrollers-video-tutorial-series/

An overview page from Microchip that doesn't really tell much that wasn't already obvious: https://www.microchip.com/pagehandler/en_us/designsupport/programming

A video for an earlier version of the Microchip IDE. This video helps penetrate the mysterious complexity of the IDE: https://www.youtube.com/watch?v=mUofSucHx_E

Another helpful video, again not for my version of the IDE but they make so many updates to the IDE that it's not a mystery. Fortunately, some things get carried through various versions: https://www.youtube.com/watch?v=dGc0BMCoeVg&spfreload=1

Somebody who had the same problem as me of needing to get started quickly without having to plow through long manuals. The second reply turned out to be a great list of tutorials: http://www.microchip.com/forums/m139145.aspx

A list of amazingly complete tutorials, alas for a different PIC than I have: http://www.amqrp.org/elmer160/lessons/index.html

A search that I did on Microchip's site when my compilation was having trouble finding a particular header. I think that the eventual solution was to correctly import the entire build directory for the previous project, which had everything properly set up. http://www.microchip.com/search/searchapp/searchhome.aspx?q=spi.h: No such file or directory&resperpage=10&id=2

Another search for how to solve an error message that I was getting. None of the answers in this tread were helpful to me, but they seemed to contain a lot of smart background information I might want to review later. My problem was solved by just reusing the setup from the previous project: http://www.microchip.com/forums/m594491.aspx

One of the problems that I had was that my project wasn't able to find the UART.h and other header files. I don't remember how I solved this; it involved adding a new path or something because the place that these headers were kept changed between the earlier version of the IDE that the program was originally compiled in and the new version that I used.http://www.microchip.com/forums/m766175.aspx
http://www.microchip.com/forums/m773469.aspx
http://www.microchip.com/forums/m528591.aspx

I'm not sure how I ended up looking into what seems to be something to do with graphics libraries, but here's the head of the search that I did: http://microchip.wikidot.com/hif2131:mla-installation

A step-by-step of how I used the MPLAB X IDE to build a project.

1. Start the IDE. For some reason two applications were installed, IDE and IPE. I have no idea what IPE is.

2. If another project is open, select File->Close Project

3. Copy the entire folder containing the old project to be modified to a new folder with the new version name

4. Under File->Open Project, select the new folder. The project will have the old name when it comes up in the "Projects" tab of the navigator pane of the IDE, but you can right click on it and from the context menu select "Rename..." which gives you the option to change the name. An option to "Also Rename Project Folder" is present, but it is not needed because the folder that has been copied already was given the new name in step 3.

5. If not already done, expand the project in the navigator pane to show all the files in the project. Double click on any of the files to edit them.

6. Compile using the hammer or hammer & dustbroom until a clean compile is achieved.

7. Plug the ICD3 adapter into both the programmer connector on the board and a USB port on the PC. *Power on the board with the dsPIC*. In the tiny "Dashboard" pane in the lower right of the MPLAB window, click the "recycle" icon to activate the ICD3 and make it check for a good connection to the dsPIC

8. Click the "Make and Program Device" icon to build the project gain and download it. After the usual build is done, a new tab with the label "ICD 3" will open in the status area and show the status of programming.

Debian On-Screen Keyboard

As part of my BeagleBone system, portability would improve if I didn't have to plug in a keyboard. As it turns out, there are several linux-compatible on-screen keyboard utilities that I can get for Debian and in particular in my case LDXE. I tried a few out; the one I like best is the boring-looking "Matchbox Keyboard". Interestingly, I found that it comes with the Debian distribution from BeagleBone. I tried "Florence"which looks lovely, except that when I installed it something went wrong with the colors or something and all the special function keys were blank even though their functions worked.

Here are some links:

Results from doing a package search right on debian.org: https://packages.debian.org/search?keywords=matchbox-keyboard

Triangular Number Series

It turns out that the minimum number of point to point resistance measurements that you have to make to test between all of the pins on any connector is given by something called the triangular number series. If you have only 2 pins to test, you just need to make one measurement. If you have three pins, you test from the first to the second, the first to the third, and the second to the third for three measurements. For four pins it's 6 measurements, for 5 it's 10 and so on. Each number of the series, 1, 3, 6, 10... is the previous number of the series plus the number of the next place!

Here is an online calculator for numbers in this series, although it's mostly of passing interest since it's really easy to code a triangular number calculator in any language:

http://www.mathgoodies.com/calculators/triangular-numbers.html

python tkinter label scrollbar 1

For my project I wanted to have an event log window showing to which I could write messages at the bottom and have them scroll up as more messages were added, with the top ones scrolling off the top of the window. I wanted to have a scrollbar on the side that would allow the user to access the earlier messages. In my searching, I found a few hints on how to implement a scrollbar, but initially couldn't make it work, so the first design that I got working was to instead have a fixed size Label widget to which new lines got added at the bottom, with the old lines simply scrolling out of view at the top.

There is some reason why a "Text" widget won't work for this application, I can't remember exactly why but I think that it might be an inability to dynamically update it. The "Label" widget allows you to use a StringVar to define its contents, and then update the StringVar any time you want to refresh the Label. As with most things in Python, there is no limit to the size of the StringVar, so I can be lazy and just keep adding and adding lines to the StringVar without having to worry about trimming text from the beginning; the fixed size of the label and anchoring the text to the bottom of the label then takes care of the scrolling for me.

Below is my actual design, in which I simply defined a fixed geometry for my Label widget, set wordwrap so that long horizontal lines get wrapped, then I defined the anchor for the text to be in the bottom left corner. Then I add lines to StringVar for the Label with '\n' after each new line I add. The Label redraws itself to show the new value of the StringVar, and any lines that don't fit within the fixed geometry are drawn but are off the top of the visible area so it gives the illusion of lines scrolling off the top. If the geometry of the widget weren't fixed, it would resize to show all of the lines. In this example code f is the main frame of my GUI; I pack the label widget into a new frame f5 which is in turn packed under some other objects that I'd defined earlier in the frame up to that point:

f5 = Tkinter.Frame(f)
self.stattext = Tkinter.StringVar()
self.stattext.set("")
self.status = Tkinter.Label(f5,textvar=self.stattext, justify="left",anchor="sw",width=80,height=10, wraplength=640)
self.status.pack(side="left", padx=10)
f5.pack(fill="x")

Here are some links with examples that look like what I did:

This is one where somebody is updating a StringVar to make a Label widget update. It doesn't have the fixed geometry like I used. The OP for this forum post has a minor problem urelated to the use of the Label. http://stackoverflow.com/questions/1918005/making-python-tkinter-label-widget-update

With regards to having a Label widget with a scrollbar, it turns out that scrollbars are not a native property of Labels, but since Labels by default resize themselves the answer is to put one in a "Canvas" which is a type of frame that you can define all kinds of parameters for such as that it doesn't resize but it can have scrollbars. Initially I was not able to get this to work, so I went with the other solution above. Here are some links about using a Canvas:

http://stackoverflow.com/questions/7113937/how-do-you-create-a-labelframe-with-a-scrollbar-in-tkinter

http://stackoverflow.com/questions/16188420/python-tkinter-scrollbar-for-frame

python tkinter how to pop up a window from inside a class

It took a little bit of searching and pondering of the info that I had found to accomplish one of the most important functions in my GUI after I recoded it to properly use the Tkinter mainloop, which is how to pop up custom dialog windows when something happened on the main GUI. My GUI is all under a single class, which is a good way of doing it but not much like the super simple examples that are often found online for creating dialog windows. I needed dialogs that were more customized than the built in Message or OK dialogs. The answer was for each dialog launched from the main GUI to create an instance of the Tkinter.Toplevel object and have that be the root of the new dialog window. This essentially creates a new window frame, it's like the Tkinter.Tk object that you use to create your GUI at the initialization of the class. You then pack data entry and/or display widgets into that Toplevel object. All of the objects you put in this frame are still accessible from the main class as long as you store the pointers to them in the class; essentially this new window is just an extension of the original window. Declare the callbacks for each of the widgets in the dialog as part of the class; you can allow the callbacks to destroy the Toplevel object by its pointer in order to perform the function of dismissing the dialog. There's a magic function of the Toplevel object (.transient()) which allows you to define that it appears on top of the original GUI window, and then you can place it using geometry commands like you would your main window. The only trick is to keep all the coding event-driven; for a complex program that means employing the state machine type of coding where you build the dialog in one state and then let user interaction with the dialog widgets cause advancement to a new state.

Here's my example of how I did it; not necessarily the best way but fairly functional:

import Tkinter,tkMessageBox,ttk
class mygui(Tkinter.Tk):

def __init__(self, *args, **kwargs):

Tkinter.Tk.__init__(self, *args, **kwargs)

self.title('My GUI Main Window')

# Making a fancy frame here with a blue border just for fun
# Bottom layer is a frame with blue fill
frm_0 = Tkinter.Frame(self, bg="blue")

# This is the main frame that will be on top of the blue frame
f = Tkinter.Frame(frm_0)

# Putting some text in the frame just to have a frame.
# Actual GUI would have more stuff
self.mylabel = Tkinter.Label(f,text="This is a label ",justify="left")
self.mylabel.pack(side="left",padx=10,pady=10)

# Place the main frame inside the blue frame with some padding
# so that the blue of the lower layer shows all around it
f.pack(fill="both",padx=5,pady=5)
frm_0.pack()

# Place the bottom frame where I want it (in the center)
frm_0.place(relx=0.5,rely=0.5,anchor="center")

# Make this example gui the size of the packed controls,
# in the middle of the screen
self.update_idletasks()
width = frm_0.winfo_width()
height = frm_0.winfo_height()
x = self.winfo_screenwidth() // 2 - width // 2
y = self.winfo_screenheight() // 2 - height // 2
self.geometry('{}x{}+{}+{}'.format(width, height, x, y))

# Initialize the state machine to the first state
self.state = 0

# Initialize other variables in this class
self.mytext = ''
self.text_accepted = False
self.abort_selected = False

# Define all the widget callback functions for the class
def accept(self, event=None):

self.mytext = self.e.get()
self.text_accepted = True
self.top.destroy()

def close_top(self):

self.mytext = ''
self.abort_selected = True
self.top.destroy()

# Main executive loop. This is where the part of the program that
# actually does things would go
def executive(self):

if (self.state == 0):

# First state, for doing stuff before popping up the dialog
print "Doing stuff in first state."

# Set next state, schedule the next call of exeutive() and exit
# In this example there is only one state before dialog; could
# be any number of states before the dialog
self.state = 1
self.executive()

elif (self.state == 1):

# Draw dialog popup
self.top = Tkinter.Toplevel()
msg = 'Enter text and press accept'
self.top.title(msg)

frm1 = Tkinter.Frame(self.top)
self.e = ttk.Entry(frm1)
self.e.config(width=(len(msg)+20))
self.e.insert(0,"default text")
self.e.pack(side="left",padx=5,pady=5)
b = Tkinter.Button(frm1, text="Accept", command=self.accept)
b.pack(side='left',padx=5,pady=5)
frm1.pack()

self.top.protocol("WM_DELETE_WINDOW", self.close_top)

# Make sure this dialog stays on top of the root console
self.top.transient(self)

# Make this dialog positioned in the center of the screen
self.update_idletasks()
win_width = self.top.winfo_width()
win_height = self.top.winfo_height()
x = self.winfo_screenwidth() // 2 - win_width // 2
y = self.winfo_screenheight() // 2 - win_height // 2
self.top.geometry('{}x{}+{}+{}'.format(win_width, win_height, x, y))

# set state to next state and exit
self.state = 2
self.executive()

elif (self.state == 2):

# waiting for the accept button to be pushed.
if self.text_accepted:

self.state = 3
self.executive()

elif self.abort_selected:

self.state = 4
self.executive()

else:

self.after(100,self.executive)

elif (self.state == 3):

# Do something with the input (boring example).
print self.mytext
self.state = 4
self.executive()

elif (self.state == 4):

# Exit state. Call the built-in .quit() function to exit the GUI
# Note that you can still call a built-in MessageBox widget any time
tkMessageBox.showinfo(title="Example complete",message="Example complete. Click OK to

exit.")
self.quit()

# Note that this "start" function is basically redundant, however having
# a function with this name helps readability.
def start(self):

self.state = 0;
self.executive()

if __name__=='__main__':

# Main code for example

app = mygui()
app.start()
app.mainloop()
app.destroy()

BeagleBone Debian Compile Driver Module

This is a page for documenting links and discoveries while I try to figure out different options for getting my WaveShare RTC cape working with my BeagleBone Debian distribution.

Steps tried so far: 1) Download the pre-compiled Debian distro from WaveShare site and updating it with Tkinter to match the Debian distro that I got from BeagleBone. 2) Trying to compile the drivers that WaveShare supplies into my distro from BeagleBone. 3) Trying to compile the driver source code for the PCF8563 chip that is available online. 4) Buy a BeagleBoneBlack from Adafruit which comes loaded with Debian already and hope whatever distro that Adafruit supplies has driver module for the PCF8563

Results for approach 1) Trying the pre-compiled distro from Waveshare: The Debian Distro from the WaveShare site interfaced smoothly with the RTC cape. It was a complete SD card image, which I downloaded and installed on an SD card. It booted up and found the LCD screen and had LXDE with about the same desktop configuration as the Debian distro from the BeagleBone site. I was able to execute all of the steps I had previously learned to install the PCF8563 chip as a new device on the i2c bus, it was recognized by the driver and a new rtc device was installed in /dev, I was able to write the date to the rtc and read it back, and the steps from several sites for setting up a service to set the system time from the rtc during bootup all worked. This distro, like the BeagleBone distro, did not have Tkinter. I executed the steps to use apt-get to install Tkinter and they worked fine. I was able to run my Tkinter code. But I found that there is a huge problem in that the file manager freezes and crashes during any "drag and drop" operations!!!

From here, it seems like my options branch into: a) try to figure out if the pcf8563 driver in this distro is somehow a module that I can copy and install into the BeagleBone distro, or b) try to figure out how to fix the drag and drop crashing problem in the WaveShare distro. While looking for module files in the WaveShare distro, the module directory name showed that it seems to be numbered "Bone40" while the distro that I got from BeagleBone is "Bone70" so perhaps there were issues that got fixed between when WaveShare compiled their drivers and when I downloaded Debian from the BeagleBone site.

Some links to sort through later:

A link which might help me with updating the WaveShare distro to fix the drag and drop issue: https://www.debian.org/doc/manuals/debian-faq/ch-uptodate.en.html

Results from approach 2) Try to compile the driver files supplied by Waveshare: I am making a stab at trying to understand the instruction on the Waveshare site for compiling their modules. The document XXX_CAPE_Porting_Drivers.pdf says to do *something* (it's not clear) with the .dts file provided by the site in a zip file of .dts files. It might be that the use of the word "Coding" in the instructions is an incorrect translation from another language, since the instructions seem to be saying just to find the file and place it in a particular location. Also, apparently the instructions cover doing the same driver porting for an i2c driver which makes a little bit of sense since the PCF8563 is on the i2c bus for the BeagleBone, but I'm not sure if there aren't already i2c drivers in my Debian distribution, and if there are I'm not sure if they'd be compatible with whatever RTC driver WaveShare is providing or if Waveshare's RTC driver requires Waveshare's i2c driver only. The results seem to be .dtbo files which have to be placed into certain directories and then a text string which is similar but not exactly the same as the name of each of the .dtbo files gets echoed to some kind of capemanager device similar to how the chip gets installed as a new i2c device. Among other questions this raises, I have to wonder if this is a one-time operation or does it have to be performed on every bootup by some kind of service? There's a hint in section 16 of the document that says that their Angstrom distribution pretty much does this, although the directories discussed don't look like those that I put service scripts into while playing around with Debian. The Section 14 described in the card has a lot of discussion about something called a TF card, and seems to be rather Angstrom-specific, giving me despair that this entire process could be inapplicable to Debian.

http://www.waveshare.com/w/upload/2/20/XXX_CAPE_Porting_Drivers.pdf

.dts apparently stands for device tree source file. There's something called a .dtsi which is the device tree source include file. This is explained in the following very dense tutorial, most of which I still don't get after the acronym expansions:

http://elinux.org/Device_Tree

This seems to be a tutorial specifically addressing what Device Tree is in the context of BeagleBone. It talks about kernel 3.8 which I've seen is what both my official BeagleBone Debian distro is as well as the Waveshare Debian distro. It gets confusing about 1/4 of the way in, but it seems that I need to fully understand what is being discussed at this link if I ever hope to recompile the Waveshare drivers for the Bone70 kernel:

http://elinux.org/BeagleBone_and_the_3.8_Kernel

Interestingly, now that I have begun reading the above links, the driver loading instructions for the original but discontinued cape that I had hoped to get are now making a lot more sense. This shows a step not in the Waveshare instructions which is how to check the event logs for successful load of the device tree overlay. Also the links at the bottom of the page, which previously seemed random, now look like additional tutorials about Device Tree that could help me:

http://elinux.org/CircuitCo:RTC_Cape

Update Jan 31 2016: So far I have been able to compile the .dts files from Waveshare but it hasn't resulted in the pcf8563 chip being recognized or a new rtc being added for it under /dev. Here is the latest new information that I've dug up.

First of all, I've found a gorgeous tutorial on how to compile dts files for BeagleBone, once again on Adafruit. The tutorial is found here:

https://learn.adafruit.com/introduction-to-the-beaglebone-black-device-tree/overview

A pdf of the same tutorial is here:

https://learn.adafruit.com/downloads/pdf/introduction-to-the-beaglebone-black-device-tree.pdf

The above tutorials seemed to help with compiling the dts files from Waveshare. This was despite concerns due to the dts directory from Waveshare zip file explicitly having "Angstrom" in its name. The dts file was compiled to a dtbo file which I copied to /lib/firmware as shown in the adafruit tutorial (but not shown in the Waveshare tutorial, for which none of the kernel directories listed match the ones in my Debian distro), and I was able to echo the module name to the "slots" file in /sys/devices/bone_capemgr.8 and then see that it had been added. However there was no new rtc in /dev. The messages from 'dmesg | tail' all looked good up to a point, it shows the overlay being found and applied, but it still doesn't show the RTC chip being recognized. I tried manually doing the steps to add the pcf8563 to i2c-1 and it instantiated the device the same as it has been but the device still seemed to be unrecognized by any driver.

In a simple world, it should be enough to try restarting the system, but I was pretty sure that would just roll back my overlay installation, and sure enough at the bottom of the Adafruit tutorial it says "We know that restarting the system will cause the overlays to unload" and gives instruction on adding something to a configuration file to cause them to be loaded on every bootup.

Looking at the results from dmesg after manually adding the pcf8563 made me wonder what the 'i2c i2c-1: new_device: Missing parameters' message could mean, particularly since it was followed by the cheerful looking 'i2c i2c-1: new_device: Instantiated device pcf8563 at 0x51' message. The google search for this came up with some good hints but nothing that has lead to a clear 'ah-ha' moment.

Here's a readme about i2c that seems to have four separate hints about what that message could mean, none of them clear. https://www.kernel.org/doc/Documentation/i2c/instantiating-devices

Here's a forum thread with a guy who was having trouble getting his ds1307 RTC to work. It seems like he was missing a module or something, but it's not entirely clear what his problem was or what the resolution was that he found. It's also a little sad that this issue was with a pi and not a BeagleBone. https://github.com/adafruit/adafruit-raspberrypi-linux/issues/2

Possibly a related forum thread on the problem. It seems that the problem was simply that the required modules were not loaded? http://forums.adafruit.com/viewtopic.php?f=50&t=31821

Since it seems like people most often solve problems when trying to interface the ds1307 RTC, I wanted to see what would happen if I did a google search for the ds1307 module. I got more hints at the links below:

Here's the Google search: https://www.google.com/search?q=ds1307+kernel+module

Here's a forum thread started by somebody that's following the Adafruit 1307 tutorial and having trouble getting the module to load. Somebody replies with a bunch of really specific instructions which are a lot like the instructions I saw on another thread, one of which is to get the package for i2c-tools. I'm wondering if this is something that would help me? Sadly, when I did 'apt-get update' and 'apt-get install i2c-tools', I got back 'i2c-tools is already the newest version' meaning I probably already have i2c-tools. https://www.raspberrypi.org/forums/viewtopic.php?p=273354

Here's another tutorial, completely unrelated to the Adafruit one, on adding the Adafruit ds1307 stamp to a BeagleBone. It's a very nicely done tutorial, and it also shows a step of downloading the i2c-tools package. It seems like the author is using a crazy version of linux ("Archlinux Allstar"?), and to do the download he uses a command I've never seen before which is "pacman". http://www.crompton.com/hamradio/BBB_realtime_clock/

So, here is the same Google Search but with the PCF8563. The returned links seem a lot less numerous and helpful. https://www.google.com/search?q=pcf8563+kernel+module

One of the links to the above search is one that I keep hitting which is a list of module files. It doesn't provide any hints about where to find them, but it shows an entire dependency tree that is intriguing. It shows a dependence on i2c-core, which modpobe seems to indicate that I have, but it also shows dependency on rtc-core which comes back from modeprobe as missing, and also rtc-lib which also seems to be missing. http://modules.libres.ch/browse/linux/v2.6.18/powerpc/rtc-pcf8563/

This forum thread has astonishingly complete step-by-step answers to problems interfacing a DS1307 chip to a Raspberry Pi, then for some reason the same treatment is given to somebody trying to interface a PCF8563. On the surface, none of this seems to help me find a kernel module for rtc-pcf8563, but there is so much good information in this link that maybe I'm just missing something in it. https://www.raspberrypi.org/forums/viewtopic.php?t=85683

This link filled me with hope, as it lists a module named "kernel-module-rtc-pcf8563". The fact that the page is all about Angstrom modules was ominous, and indeed on my Debian distro, when I did 'apt-get update' and 'apt-get install kernel-module-rtc-8563' it came back with 'E: Unable to locate package kernel-module-rtc-8563'
http://feeds.angstrom-distribution.org/feeds/v2012.05/ipk/eglibc/armv7a/machine/beaglebone/Packages

Having read the translation, it seems like there is no new information here, but for novelty's sake here is a page from China (also the source of the Waveshare Misc Cape) on interfacing a PCF8563 chip: https://translate.google.com/translate?hl=en&sl=ko&u=http://blog.daum.net/_blog/BlogTypeView.do%3Fblogid%3D0Ps5d%26articleno%3D49%26_bloghome_menu%3Drecenttext&prev=search

Results for approach 3) Trying to compile the driver source code into either the kernel or a kernel module: I have frequently encountered some source code for a driver module for the PCF8563 chip during my web searches. It seems to come from the manufacturer of the chip, which would be Phillips (which was subsequently bought by another company):

https://github.com/jeffegg/beaglebone/blob/master/drivers/rtc/rtc-pcf8563.c

also here: https://github.com/tekkamanninja/linux-beagleboard/blob/master/drivers/rtc/rtc-pcf8563.c

and also here: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/rtc/rtc-pcf8563.c?id=57d54889cd00db2752994b389ba714138652e60c

Interestingly, there seems to be a slightly different version (which still contains a line returning version 0.4.3). It can be found here:

https://stuff.mit.edu/afs/sipb/contrib/linux/drivers/rtc/rtc-pcf8563.c

and here: http://lxr.free-electrons.com/source/drivers/rtc/rtc-pcf8563.c

The reason why I think this code might be the thing that I want to try to compile is that it seems to have a version number (0.4.3) that is the same as the one shown in log messages from the rtc-pcf8563.ko module for people that already have it in their linux distribution, like as seen at these links:

http://www.boonsanti.com/setting-a-real-time-clock-to-beaglebone-black-with-rtc-pcf8563/

(Note: this is the one where the blogger talks about "Chris Boot's Forked Kernel" and he's working on a Pi) http://www.susa.net/wordpress/2012/06/raspberry-pi-pcf8563-real-time-clock-rtc/

https://groups.google.com/forum/#!topic/beagleboard/YytMAKTdYrE

https://www.raspberrypi.org/forums/viewtopic.php?f=44&t=6994

Aside from that, all I have so far are some links that seem to have helpful general advice on how to compile driver modules. I need to read all of these before I give this approach a try:

A very friendly and hopefully basic tutorial on compiling stuff into Debian: http://www.aboutdebian.com/compile.htm

A basic linux faq about kernel modules. It shows modules in the location where I find them in the BeagleBone distro, but not where the PCF8563 files seem to be in the WaveShare distro (which possibly indicates that WaveShare compiled their drivers into the kernel instead of creating a module): http://www.cyberciti.biz/faq/add-remove-list-linux-kernel-modules/

A very sparse page of information about modules, which includes at the bottom a link to the "all modules" page which shows a rtc-pcf8563 module (although it's not clear if it's the same one that I need): https://wiki.debian.org/Modules

A fairly sparse HowTo on how to compile modules. This plus some other links might give me a starting point to trying my own compilation: http://www.cyberciti.biz/tips/build-linux-kernel-module-against-installed-kernel-source-tree.html

A beginner's article for understanding Make, Configure, and other generic steps in most compilations in Linux. Probably doesn't contain anything about drivers, but could help me get started: http://www.codecoffee.com/tipsforlinux/articles/27.html

Results for approach 4) Buy a BBB preloaded with Debian: Sadly, this didn't pan out as hoped, although there might be some advantages to pursuing this further. First of all, the BBB is half the price of a regular BB, it's only $45 which is basically less than a tank of gasoline. Secondly, it's supposedly faster.

However, it didn't recognize the pcf8563 chip when I plugged it in place of the BeagleBone and didn't seem to have the driver module. It did recognize the LCD7 screen, and came up with an LDXE desktop very similar to my current one except that it had four instead of 2 desktops. However, less than 15 minutes after I started using the BBB, it crashed and wouldn't reboot; apparently the OS in flash memory got corrupted. In the short time that I used it, I noticed that it took three times longer to boot than the BB I've been using, and when I tried to run Python it didn't work despite being found in the executable path. The BBB froze while I was trying to figure out why I wasn't getting the python prompt; trying to run Python might have been the cause for the bricking. I didn't get a chance to try to figure out if it had Tkinter or even what distribution they had shipped.

My initial impression is that having the the OS in flash memory on the BBB is unreliable. I began looking into how to just run the BBB off of the SD drive like I've been doing for the BB. It turns out that this is not very easy, since the BBB will always boot from flash memory if there is something there. Also, the BBB requires you to depress a tiny switch on the board to get it to boot from SD, which is astonishing. Here are some links that I could try to sort out later if I really want to try this:

http://elinux.org/Beagleboard:Updating_The_Software#Image_for_booting_From_microSD

A commenter in this thread seems to think that you only need to push the button the first time you boot from SD, which would be helpful if true: https://groups.google.com/forum/#!category-topic/beagleboard/beaglebone-black/_mOlo6T-70E

Some of the info in this thread, however, seems to contradict the previous one. This link has a lot of info about the uboot utility which seems to be the heart of finding the answer to this question. https://www.element14.com/community/thread/28768/l/beagleboneblackunable-to-boot-u-boot-image-from-sd-card-how-to-do-this?displayFullThread=true

Probably I just need to go through the steps of downloading a new Debian image compatible with BBB and flashing it.

Update Feb 21, 2016: Since getting started on trying to compile the driver module from source code, I've gone down a few different paths. Specifically: 3a) Learning how to compile a simple driver module from the source headers. 3b) Trying to replicate the simple driver compilation with rtc-pcf8563.c, 3c) Learning that rtc-pcf8563.c is provided in the linux source tree, trying to download the source and compile it in place.

Step 3a) Learning how to compile a simple driver module using the source headers: There are several tutorials around the internet on how to compile a simple module called hello.ko which just prints some stuff to the event logs when started and stoppped. The following two links are occasionally unreachable for some reason but from Chrome sometimes a "cached version" is offered. These examples worked perfectly as written.

http://www.cyberciti.biz/tips/build-linux-kernel-module-against-installed-kernel-source-tree.html
http://www.cyberciti.biz/tips/compiling-linux-kernel-module.html
http://www.paulkiddie.com/2009/10/creating-a-hello-world-kernel-module-in-linux/
http://www.lai18.com/content/1641206.html
http://icarus-edward.blogspot.com/2006/09/howto-build-linux-kernel-module.html

The main problem with the above turtorials was that the kernel header package was not named kernel-headers-$(uname -r) but rather linux-headers-$(uname -r). I figured this out using apt-cache search with vaguer and vaguer search terms until I realized what the problem was. Then I used sudo apt-get install to download the source header package and it unzipped it and placed it under /usr/src. The makefile provided by the first link above just worked like a charm and produced a .ko file which could be installed and uninstalled.

Step 3b) Trying to replicate the simple driver compilation with rtc-pcf8563.c: So, I excitedly downloaded rtc-pcf8563.c, and put it in a directory with the makefile from the hello.ko example and changed the name of the target in the makefile. Unfortunately, the C code didn't compile; in particular it barfed on some kind of define named "THIS_MODULE".

At this point I embarked on a big link sausage trying to figure out if I could fix my problems by downloading the entire source. My problem was that when I was doing apt-cache searches, I could find the header package as linux-headers-$(uname -r) but there didn't seem to be a corresponding package for just the source code tree. So, here are some interesting links that probably would fix things for me except none of them were dumbed-down enough for me:

Build the entire kernel? That might work. OMG way too many steps. http://elinux.org/Building_BBB_Kernel

At about the sixth step I was unable to follow what was going on with this otherwise promising tutorial, but I did notice that it employed an apt-get source linux command to get the source. https://wiki.debian.org/HowToRebuildAnOfficialDebianKernelPackage

This link tantalizingly promises that you can build any driver with a simple make command if you install build-essentials. This doesn't seem like it could be true. http://askubuntu.com/questions/177329/how-do-i-install-a-driver-provided-as-source-code

Some more hints about how to use apt-get source. Apparently you can compile directly from apt-get? http://askubuntu.com/questions/28372/how-do-i-get-and-modify-the-source-code-of-packages-installed-through-apt-get

Step 3c) Learning that rtc-pcf8563.c is provided in the linux source tree, trying to download the source and compile this file in place. I mean, presumably the file can be compiled if it's part of the source. For some reason, the next night's searching turned up a magic link that contains step-by-step instructions that even a dummy like myself can follow:

Here's the magic tutorial! Sadly it's got some RPM stuff in it that doesn't apply to my Debian distro, and running make oldconfig led to an hour of banging through menus because I was too impatient to copy over my .config file and the default wasn't even close to right, and I shot myself in the foot in general by skipping a lot of steps and I wasn't even using the right source tree. But this is the closest I've gotten yet to getting a clean compile. It built the .o file for the module, but something called modpost didn't see fit to make it into a .ko file. More on this learning point in later links.
http://www.pixelbeat.org/docs/rebuild_kernel_module.html

Here's the forum thread that led me to the above link. After a lot of the OP getting basic questions from commenters, there are a couple of useful answers in this thread. One of them towards the bottom has a 'make M=' example that I see a lot but which failed to work for me, however maybe I typed something wrong when I tried it.
http://www.linuxquestions.org/questions/linux-kernel-70/rebuilding-a-single-kernel-module-595116/

A link that I've posted to myself before to study, here's somebody basically doing the exact same process of compiling a single driver module, except for a raspberry pi. He downloads build-essential with apt-get, then uses wget to get the source tree, then does a patch which is one of the opaque steps of kernel building for me still. He does the same setup step for .config that the guy from the first link does that I screwed up by skipping, then does a make oldconfig. He does another wget which seems to have to do with getting his module source, then does a make modules_prepare. Then he does a different make command that hasn't worked for me yet but mostly because I keep doing with wrong, which is make -C M=. So this is a good example to try to correlate with the other examples that I've tried now that I've tried them. Just to see if it would help, I tried doing a apt-get install build-essential but I think that it came back already installed or something.
http://blog.remibergsma.com/2013/05/08/adding-a-hardware-clock-rtc-to-the-raspberry-pi/

So, after finding the above information, I first tried to do a compilation while not being very patient. I tried to apt-get the linux source, and for some reason I got a tarball for kernel 3.2 (whereas my distro is 3.8.13-bone70, so kernel 3.8 which was a big change from 3.2). However, 3.2 had all of the same RTC driver directories that the bone70 linux header directories expected, so I thought that I'd give it a try. I didn't do the step of copying my .config file from my existing /boot directory, so make oldconfig was just a mess and I ended up just accepting all of the default choices that it kept offering when it turned out to be a lot. I'm not sure if make prepare and make modules_prepare worked, but I think that they might have. In the end I got it to build the rtc-pcf8563.o file but I couldn't make it do the .ko file. I did a pile of hunting for solutions to the error message that I was getting; now I have a bunch of interesting links that didn't help, and I later learned that my problem was something else altogether.

Next I tried downloading the Debian source directories from Waveshare. The tar file was named with kernel version 3.8.13, and the directory structure that it untar'd looked exactly like what I expected. The download came with an additional directory of config files, for either beagleboard, beaglebone, and something else. I checked the beaglebone config file that came in that directory with the one in my distro and it turned out to be wildly different, and also the config file that came with the Waveshare distro didn't have the pcf8563 enabled which was odd since this was supposed to be a kernel for use with the Waveshare board. The big disappointment is that make prepare bombed out completely. I did a pile more research on what was happening, which seemed to have something to do with the Waveshare distro being set up to use a different cross compiler than what was provided in my Debian distro. A couple of half-hearted attempts to switch over the cross compiler selection in the makefile quickly led to deciding to ditch the Waveshare source altogether.

Next I tried to get the 3.8.18-bone70 source tree. Most of the beaglebone Debian pages end up pointing to a github repository. The troubling thing about those links is that it seemed to be for a 4.1 kernel which I don't even know what that would be in relation to 3.8.18. I fiddled around in the github branch that the beaglebone/4.1 code was in, and eventually found a tag for 3.8.18-bone70, but I could not figure out how to download it with git. The site has a big "Get Zip File" button, so I downloaded that and used 7zip to just completely unzip and untar it (I can't remember why I couldn't untar it on my beaglebone) then transferred the entire 400MB directory to a local directory on my beaglebone. I copied the .config file from my boot directory and updated it to enable the pcf8563 driver. Amazingly, make oldconfig, make prepare, and make modules_prepare all went completely smoothly. The make for scripts/mod didn't seem to do anything, and I got the "MODPOST 0 modules" message but kept going.

make SUBDIRS=drivers/rtc ran to completion, alright! But it still didn't make a .ko file for the rtc-pcf8563 driver, boo! It did make two others, rtc-ds2404.ko and rtc-snvs.ko. I looked for these in the .config file, and found that instead of being set to '=y' in the config file, these two were set to '=m'! I set that for the rtc-pcf8563 driver, tried to do another make and didn't get anything, redid make oldconfig, make prepare, make modules_prepare, and make SUBDIRS=scripts/mod, then did make SUBDIRS=drivers/rtc and it made my rtc-pcf8563.ko file!!!!

Following a couple of the examples that I've been trying for ages to get working, I did a modprobe i2c-tools, but when I did modprobe rtc-pcf8563, it returned with rtc-pcf8563 not found. I tried copying the file to the same directory that the rtc-ds2404.ko and rtc-snvs.ko files were, but it still didn't work. I also tried insmod and I got back a message stating: "Error: could not insert module /rtc-pcf8563.lo: Invalid module format"!!!! So, I went back to google and learned a few more things. This seems to be a versioning mismatch, despite the care with which I tried to get the correct source tree. It also seemed to produce an error in dmesg that said "rtc_pcf8563: no symbol version for module_layout". I eventually learned that this has to do with a compilation error that I watched go by during the compilation, which was "Warning: Symbol version dump /usr/src//Modules.symvers is missing; modules will have no dependencies and modversions." Apparently this is because the source tree that I built against was previously used to build the whole kernel, which produces the Modules.symvers file.

Somebody online recommended using modprobe -f to just overcome this error; I eventually further learned that this requires either copying or linking my .ko file into the directory /lib/modules/, and also not stupidly using the .ko suffix when calling the file with modprobe. insmod doesn't seem to have a -f option although I didn't check the man page for some similar force parameter. Once I finally properly did a modprobe -f rtc-pcf8563, it returned with a new error, which was "ERROR: could not insert 'rtc_pcf8563': Exec format error." This error also produces more "rtc_pcf8563: no symbol version for module_layout" lines in dmesg.

It seems that usually people brute-force their way past this by recompiling the whole kernel, but a few links that I've found show how some steps for building Modules.symvers so I thought that I'd give that a try. However, a search of my directory tree revealed that the Debian distro image had come with a Modules.symvers file. I copied it to the top of the downloaded source tree, did the entire build process again, and did not get the warning messages. I copied the new .ko file to the proper location and repeated the steps of depmod -a and modprobe and it installed! Because I had earlier installed the pcf8563 chip onto the i2c bus, the chip was instantly recognized by the driver, /dev/rtc1 was created, and I was able to read the time from it! I then copied the .ko file to the other copy that I've been using of the Debian distro, and when I placed it in /lib/modules/ I was able to repeat the steps to install the chip onto the i2c bus, have it be recognized, and was able to read the time from the RTC.

So, in conclusion, the Debian image distributed by Waveshare is broken, I'm still too stupid to figure out how to use the Device Tree and the .dts files from Waveshare, the source file trees from Waveshare are completely broken, the driver for the PCF8563 that other people seemed to be able to use already comes with the source tree for my Debian distro but it hadn't been compiled and it is still barely clear to me how to download the source tree, once the source tree for my distro was downloaded and the .config file was properly modified, a specific set of make commands could be used to build the driver into a module which is, as hoped, portable to copies of the same distro.

BeagleBone Install Debian SD Image

Because I had to search for it again, here are the steps for downloading and installing a Debian image on an SD card for the BeagleBone.

Some links:

Here is the page from BeagleBone.org which explains all of the steps: http://beagleboard.org/getting-started

BeagleBone Debian Install Tkinter

This is the procedure that I've used twice now to update a fresh Debian distribution with Tkinter. The distro comes with Python 2.7 but not Tkinter.

Step 1: Put the BeagleBone on my home network with the CAT5 cable from the router. It gets a DHCP address on the network automatically from the router.

Step 2: Go to root user.

sudo su

Step 3: Update apt-get, which synchronizes what packages my distro has and doesn't have and establishes locations to download from.

apt-get update

Step 4: Download and install Tkinter. Despite what I read online about other users' issues with this step, it seemed to work fine for me.

apt-get install python-tk

Some related links:

The basic instructions, which include the above steps http://tkinter.unpythonic.net/wiki/How_to_install_Tkinter

About how to use apt-cache search which while not specifically needed to do this installation is generally helpful. Also, aptitude is mentioned: http://askubuntu.com/questions/160897/how-do-i-search-for-available-packages-from-the-command-line

More about apt-cache search, and also some more about how to use aptitude: http://www.cyberciti.biz/faq/searching-for-packages-in-debian-ubuntu-aptitude/

BeagleBone installing PCF8563 RTC

I ordered the Waveshare "Misc Cape" which uses the PCF8563 RTC chip with an I2C interface like all of them. Waveshare's online documentation is unhelpfully brief regarding how to install drivers for it. They offer a fully compiled Angstrom distribution with their drivers but since Debian is working out so well for me on my project I was hoping to try to find a way to install drivers for Debian. At the bottom of their wiki for this cape there are a bunch of links, some of which seem to be source code and other documentation, but it's not clear what it all is.

http://www.waveshare.com/wiki/MISC_CAPE

One of the links turns out to be to two SD card images, one of which is a fully compiled Debian image that has the drivers for the PCF8563 chip built in, however when I downloaded and tried it I found that although the RTC driver seemed to work well there were other issues with the distro that have to be overcome:

http://www.waveshare.com/wiki/MISC_CAPE_Image

There are other links on the same wiki page that I'm still trying to figure out. The "User Manual" link is just to a PDF of the wiki. The Schematic is helpful but really the cape is quite simple and the schematic is a single page. The Test Code link is to a zip file named XXX_CAPE_Angstrom_API.7Z which seems to contain a mix of C programs and compiled programs which don't have anything to do with the PCF8563 and furthermore seem to be for the Angstrom kernel. The section labelled "Kernel" seems to have several different source code trees for both Angstrom and Debian kernels, presumably with driver code but at the moment I still don't know how my way around kernel source code or how to compile kernels although it looks like I'm going to have to learn. The section labeled "Kernel_config" also seems to have to do with compilation so I will have to become smarter to even understand what it is. I have absolutely no idea what the link labelled "Cross-compilation toolchain" is even referring to. Under Source Code there are several links to zip files but some obviously have nothing to do with the RTC chip (like "wifi" and "mjpg-steamer") and the one link labelled "driver" is to a zip file that just has one c file with code that also doesn't seem to have anything to do with the RTC chip. Of the three links under "Documentations" the first one "Porting Drivers" actually mentions the RTC chip but the compilation instructions are like greek to me. The link for "Porting Kernel" mysteriously seems to be about building a Ubuntu kernel, and these instructions are also opaque to me at the moment.

Looking for a shortcut to figuring out the poorly documented materials on the Waveshare wiki, I started looking at the following tutorial, which showed how to interface a different chip that is on Adafruit's RTC stamp to BeagleBone.

https://learn.adafruit.com/downloads/pdf/adding-a-real-time-clock-to-beaglebone-black.pdf

Since I also bought one of those Adafruit RTC stamps, I may yet give following this tutorial as written a try, but I thought that I could adapt the steps shown to the PCF8563. I tried doing the i2cdetect command and it found the PCF8563 at its address of 0x51. I did the command to add it as a new device and that seemed to go okay. There was an rtc0 device in /dev that I was able to write a time to and read from. I went through all the rest of the steps to set up a service to read from the rtc at bootup but whenever I power cycled the BeagleBone it would come up with a default date which was March 1, 2013.

I realized that something was writing time to the rtc, and went through a bunch of research to discover that there is a hwclock.sh script that is being called by several different boot levels in Debian that seems to be setting this default time to rtc0. The hwclock.sh script is at /etc/init.d/; the directories /etc/rcS.d/, /etc/rc0.d/, and /etc/rc6.d/ all seem to have the same script also but these are really just sim links to the one in /etc/init.d/. The other directories are used to define init scripts called at different kernel boot levels. This information is all swirling around inside the following comment thread:

https://www.raspberrypi.org/forums/viewtopic.php?f=44&t=16218

I tried modifying this hwclock script as advised in the thread but didn't have any luck. I began to realize from reading the script and the messages in /var/log that the rtc0 device was some kind of default device and not the PCF8563 chip on my Waveshare cape after all. I tried doing modprobe commands as shown in the link above and in some other links I found and it seems like the rtc-pcf8563 "module" is something that I need and don't have and which should run when I add the chip as a new device on the I2C bus. I found some .ko files on my Debian system under /lib/modules/3.8.13-bone70/kernel/drivers/rtc but none of them are rtc-pcf8563.ko or rtc-ds1307.ko either for that matter.

A frequent hit that I keep getting while googling for the problems I am having installing this cape is the following site in what seems to be Hebrew or something. Without using Google Translate, I can tell that he's going through all the same steps that I did, using i2cdetect, adding it as a new device, and creating a service to load the system time from it a bootup. However, when he uses dmesg to check his log files after adding the device, he sees the "chip found" message that I'm not getting. I need to look at this site more carefully to understand what OS and distro he is using; if there's a chance he's using Debian it might give me a clue:

http://www.boonsanti.com/setting-a-real-time-clock-to-beaglebone-black-with-rtc-pcf8563/

Here's a listing for a Debian distribution that shows an rtc-pcf8563.ko file in the right place, but clearly this isn't my distribution. I need to find out the chances are that I can just find one of these .ko files and copy it onto my system (probably not great):

https://packages.debian.org/squeeze/armel/linux-image-2.6.32-5-versatile/filelist

Along the way as I did this research, I've learned that there is something called "fake-hwclock" which maintains current time in nonvolatile memory and reloads it during bootup, but I don't have that on my system. Here's a link with some info:

http://unix.stackexchange.com/questions/187261/automatically-update-hwclock-at-boot

I learned that Debian has a very nice editor called "nano" that has keyword color support and is easy to use (though not as easy as vi of course)

To edit system files, I had to change from the default login of user 'debian' to root, but when I tried "su root" it asked for a password. Various sites said that this distro had no password for root, but just hitting return on the password prompt wasn't working. I learned from the following page that I need to do "sudo su" to become root on this system:

http://www.element14.com/community/thread/32106/l/beagleboardorg-releases-debian-for-beaglebone-black?displayFullThread=true

I think that my system is using systemd instead of init. Here is what looks like nice info on systemd:

https://denibertovic.com/posts/setting-up-systemd-on-debian-in-10-minutes/

There's a book called "BeagleBone For Secret Agents" that had a google books excerpt that helped me better understand that rtc0 was not my cape, that it's an rtc inside the processor that doesn't retain time when powered off. It showed messages from a process named omap_rtc that was loading the default time and I was able to find those same messages on my system showing it setting the default clock to March 1, 2013. Here's the link to the google books excerpt:

https://books.google.com/books?id=i2OZBAAAQBAJ&pg=PT130&lpg=PT130&dq=omap_rtc+setting+system+time&source=bl&ots=9hLmjeWaPe&sig=j3ScuHRzYYN2Un2ljn_0gbsglgs&hl=en&sa=X&ved=0ahUKEwie9MXq5anKAhUIPT4KHZ6XB60Q6AEIJjAB#v=onepage&q=omap_rtc%20setting%20system%20time&f=false

Here's a blog by a guy who had a problem very similar to mine. He had an RTC chip that there was no module for in the distribution of Raspian that he had. He found drivers and compiled them and verified that the resulting module detected his chip. I need to carefully study this link to understand all of his steps and maybe try the same thing for my kernel and chip:

http://blog.remibergsma.com/2013/05/08/adding-a-hardware-clock-rtc-to-the-raspberry-pi/

When I google for drivers for the PCF8563, I usually get the following link, which interestingly is from a BeagleBone site so it seems that compiling this for BeagleBone is something that is done, although there don't seem to be any instructions or compiled module files at this link:

https://github.com/jeffegg/beaglebone/blob/master/drivers/rtc/rtc-pcf8563.c

Here's a guy interfacing a PCF8563 to a raspberry pi, and for some reason his kernel has the driver module already there. He does a modprobe command on rtc-pcf8563 and it works, so he goes merrily along with the rest of his installation steps.

http://www.susa.net/wordpress/2012/06/raspberry-pi-pcf8563-real-time-clock-rtc/

Some links to stuff that I found which didn't directly help but I'm saving just in case:

Here's a hackaday article on the PCF8563 which goes into accessing its registers and stuff without showing any kind of finished code. Interesting, but I hope I never have to use it: http://hackaday.com/2009/06/26/parts-i2c-real-time-clock-calendar-pcf8563/

Here's a raspberry pi forum where some people are digging into the source code to find out why the rtc interface for the PCF8563 doesn't work for the PI2. Not useful to me but crazy smart: https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=100132

Here's setup documentation for an expansion board for the pi that uses the PCF8563. It shows the "chip found" message that I should be seeing in my logs but am not. I wish my RTC cape had this kind of documentation: https://ludwig.im/en/projects/raspberry-pi/raspberry-pi-arpi600

It turns out that Waveshare makes another RTC board with a PCF8563 for a different kind of stamp computer development system. This information wasn't helpful however because the documentation for this other RTC board included no drivers compatible with the BeagleBone: http://www.waveshare.com/pcf8563-rtc-board.htm

Python printing percent signs

Since percent is used as a formatting character, obviously it needs to be escaped somehow in formatted print statements to appear as itself. It turns out that the way to do this is to use two percent signs to print one, rather than using the backslash for escaping. The following slightly snarky forum answer describes this:

https://bytes.com/topic/python/answers/540689-printing-percent-sign

Python Tkinter progress bar

Progress bars are super easy in Tkinter, and there are two flavors. The regular kind is a bar that expands from left to right based on setting the object's "value" with respect to its "maxvalue". That is called a "determinate" progress bar. There is also an "indeterminate" progress bar that has a section that bounces back and forth from end to end to show general aliveness of a task. With an indeterminate progress bar, you can just call it's start() and stop() functions, or use a more direct method of showing aliveness by inserting step() commands in the code (.step accepts a percent value from 0 to 100). Some links:

http://stackoverflow.com/questions/7310511/how-to-create-downloading-progress-bar-in-ttk

https://gist.github.com/livibetter/6850443

The default Windows progress bar was pretty nasty looking, especially the indeterminate one which had a very thin indicator. It turns out that the way to change this is to define a ttk progressbar "style," which allows the programmer to chose from a handful of named styles, and then set colors for components. There's a command that you can do in a python console to get a list of available styles on whatever OS you're working in. Some are available on most platforms, others might be specific to whatever platform you're on. Some commentators to some forum threads that I checked out seemed to love the "clam" style, I preferred "classic" because it was big and blocky and with that name I was sure to find it on both Windows and Linux platforms. Some links about styles:

This shows how to use the theme_names() function of a Style object to get a list of additional styles. I can't see how you'd want to use this more than once, unless you were writing some really fancy code. http://infohost.nmt.edu/tcc/help/pubs/tkinter/web/ttk-theme-layer.html

Here is the examples showing how to change the progressbar color, and the first example of using Style that I found. For some reason this guy thinks that "clam" is good looking style: http://stackoverflow.com/questions/13510882/how-to-change-ttk-progressbar-color-in-python

The man page for Style: https://docs.python.org/3/library/tkinter.ttk.html#tkinter.ttk.Style

Some solutions to trying to change the appearance of the progress bar. I solved my problem by just picking a different theme. This post proposes a torturous use of a canvas, while the second says to use a Style that allows for size adjustment. http://stackoverflow.com/questions/17912624/ttk-progressbar-how-to-change-thickness-of-a-horizontal-bar

Here's a man page for Progressbar that mentions that they have a Style option: https://www.tcl.tk/man/tcl/TkCmd/ttk_progressbar.htm

Here's a link that mentions somebody having a problem with their progressbar. The issue however was that the programmer wasn't properly making his code event-driven. http://stackoverflow.com/questions/16400533/why-ttk-progressbar-appears-after-process-in-tkinter

A nice man page for Progressbar that helped show the three methods that indeterminate progressbars have, start(), stop(), and step(). In the end, I found it helpful to just use the step() function for my indeterminate progressbar from my code rather than let it run on its own with start() and stop()
http://infohost.nmt.edu/tcc/help/pubs/tkinter/web/ttk-Progressbar.html

My init code for drawing the progress bars on my gui:

import Tkinter, ttk

class MyGui(self, *args, **kwargs):

def __init__(self, *args, **kwargs):

Tkinter.Tk.__init__(self, *args, **kwargs)

self.title('My GUI')

s = ttk.Style()
s.theme_use('classic')
s.configure("blue.Horizontal.TProgressbar", foreground='blue', background='blue')

f = Tkinter.Frame(self)

self.aliveness = ttk.Progressbar(f, style="blue.Horizontal.Tprogressbar", orient="horizontal", length=600, mode="indeterminate")
self.aliveness.pack(padx=10,pady=10)

self.progress = ttk.Progressbar(f, style="blue.Horizontal.Tprogressbar", orient="horizontal", length=600, mode="determinate")
self.progress["value"] = 0
self.progress["maximum"] = 100 ## Note, not strictly necessary. If not specified 100 is assumed
self.progress.pack(padx=10,pady=10)

f.pack()

Debian set system time

To set the system time in Linux is very easy; the date command allows setting of the system date and time. This does not affect any time kept by a hardware clock, the hwclock command is needed for that. The BeagleBone does not come with a hardware clock, however, so the hwclock command fails out gracefully.

sudo date --set 1998-11-02
sudo date --set 21:08:00

Reference link:

https://wiki.debian.org/DateTime

Python measuring and printing elapsed time

In order to benchmark the return times of some hardware interface code, I needed to print elapsed times from different parts of the code. Super easy to do in python, just use variable_x = datetime.datetime.now() to get datetime variables at different places in the code, and you can just directly subtract them from each other to get variables of type datetime.timedelta which have a seconds and microseconds part.

variable_1 = datetime.datetime.now()
.
.
variable_2 = datetime.datetime.now()
delta = variable_2 - variable_1
print "elapsed time is %d seconds %d microseconds" % (delta.seconds, delta.microseconds)
print "or %.6f seconds" %((delta.seconds * 1000000 + delta.microseconds)/1000000.0)

Reference links:

http://stackoverflow.com/questions/766335/python-speed-testing-time-difference-milliseconds

Corcom EMI power entry with fuses

The power entry that somebody else bought for my project has been nothing but trouble.

The first problem that I had was that its terminals are not clearly labelled. I needed to pick off the hot and neutral from the four terminals in the back, which were labeled A-D but that's it. No schematic on it, and also Corcom's documentation is completely demure, showing some schematics with no hints as to what the terminals on the schematic are (in some schematics they are labelled 1-4 which is suggestive of an ordering like A-D but not definitive.

My application was 120V, just the hot and neutral. I finally found that Corcom sells a connector for the unit, part number PA105. Finally, I found a photo of it off of the DigiKey catalog that showed it. The white wire is on one side, the black on the other, based on the keying tab I could tell which went to terminal A and which went to terminal D.

Here's the DigiKey page for the connector:

http://www.digikey.com/product-detail/en/PA105/PA105-ND/1288422

And the image:

http://media.digikey.com/Photos/Tyco%20Photos/PA105.jpg

The Mouser catalog was no help:

http://www.mouser.com/ProductDetail/TE-Connectivity-Corcom/PA105/?qs=%252bek46ohmru8%252bpeoK71MReg%3D%3D

Nor Newark:

http://www.newark.com/te-connectivity-corcom/pa105/pa105-j0-f6806/dp/23B4271

Nor these guys, though nice line drawing:

http://www.onlinecomponents.com/te-connectivity-corcom-brand-pa105.html?p=11171134

Python continuation character

A recent google search for a python line continuation character didn't turn up any hits, which is odd because I am sure there is one. However, I found this interesting style guide for how to continue lines without continuation characters. It's tricky because of python's already existing specific use of indentation.

https://www.python.org/dev/peps/pep-0008/

Google chrome exiting tabs when I click on themk

There are many, many google hits on this problem. Most of them agree that the issue is that somehow the middle mouse button (or mouse wheel button) has been stuck down, causing each left click to register as a middle click. Apparently Chrome has a bunch of middle click behaviors, one of which is closing a tab with one middle click. Since the mouse drivers on my laptop routinely go awry when I've overloaded it, this is fairly likely to be what happened. Also I'm getting other middle click behaivors, probably my only recovery option is to reboot the stupid thing. Some links:

https://productforums.google.com/forum/#!topic/chrome/Ok8jGiAXU00

BeagleBone RTC Cape

So, it turns out that the BeagleBone (or the BeagleBone Black for that matter) does not have a battery back-up for its clock, so it loses the time every time I turn it off. It is mentioned on some websites that you have to set up an NTP access to fire on bootup to keep the clock set to the current time, but of course my application is off the network. It turns out that a few people have made something called a Real Time Clock cape that has a button cell and a particular single-chip solution to act as the Beagle Board's hardware clock. Unfortunately, the nicest of these, originally sold by BeagleBoard Toys, is no longer being made.

Presently, a multipurpose board sold by Waveshare has an RTC chip and a battery. It can be bought from the company, or interestingly from Amazon.

It is also possible to either hook up the Adafruit RTC stamp to a cape protoboard as described in this article, or completely build the BeagleBoard Toys design on a protoboard.

This article shows what commands to use to set the time on the adafruit RTC stamp once the RTC cape using it is assembled.

This forum thread has more info about different RTC chips that can be used for this sort of project.

Python quit() function

For Python 2.7, there is no quit() function. However, here is an interesting discussion of what there is that does the same thing:

http://stackoverflow.com/questions/2823472/is-there-a-method-that-tells-my-program-to-quit

In retrospect, once I got better structure to my code, I didn't need this.

Python Tkinter File Dialog references

The built-in tkFileDialog classes are delightfully easy to use. Note, however, that they are not in the Tkinter library, you have to import the tkFileDialog library separately, which gets you the four different types of dialog objects. Some references:

http://effbot.org/tkinterbook/tkinter-file-dialogs.htm

http://tkinter.unpythonic.net/wiki/tkFileDialog

Python Tkinter Frame border color

It turns out that in Tkinter, you can set the width of a frame's border, but you can't set it's color. Since ttk objects differ from Tkinter objects, this may not necessarily be true for ttk frames (but I haven't checked yet). The only apparent way of creating color Tkinter frames around things is to layer frames within frames. First pack a frame with the desired color as it's bg color. Then pack another frame in that one, set fill="both" with some padx and pady to set the edges of the second frame in from the edges of the first frame, allowing the color of the first frame to show around the edges of the second frame. Then pack stuff into the second frame. Some links:

https://bytes.com/topic/python/answers/38448-changing-color-default-frame-border

http://stackoverflow.com/questions/4241036/how-do-i-center-a-frame-within-a-frame-in-tkinter

For possible future reference:

http://stackoverflow.com/questions/4320725/how-to-set-border-color-of-certain-tkinter-widgets

Linux how to make an icon that launches a python program

This turns out to be ridiculously easy. In most Linux distros, you can just put a text file where you want the launcher (like on the desktop), give it a .desktop extension, and add in text lines that define the location of the program to run and even the icon for the launcher file to show. Some links:

https://www.raspberrypi.org/forums/viewtopic.php?f=32&t=71468

This one is particularly nice because it's the actual standard, plus it's well-written to boot.

http://standards.freedesktop.org/desktop-entry-spec/latest/index.html

http://ubuntuforums.org/showthread.php?t=2221617

https://www.raspberrypi.org/forums/viewtopic.php?f=27&t=9817

Tkinter Frames are not the same as ttk Frames!

While doing a basic google search for command syntax, I stumbled across some forum threads which revealed that basic Tkinter objects such as Frame and Label have ttk equivalents with sometimes very different sets of features!

http://stackoverflow.com/questions/16639125/how-do-i-change-the-background-of-a-frame-in-tkinter

http://python.6.x6.nabble.com/LabelFrame-question-why-do-options-change-when-importing-from-tkFileDialog-td4527355.html

python tkinter tkFiledialog filetype order

For file dialogs, Tkinter very nicely allows control of the specification of file types to be shown in the dialog. This seems to actually be a native feature of the operating system that Tkinter simply has access to. To write a dialog that restricts the files that the user can chose from, you can set the file type to *.txt for instance, or a list such as [('text file','*.txt'),('shell scripts','*.csh')]. My problem is that I also want to allow the option to the user to select having access to all file types, like this: [('text file','*.txt'),('all files','*.*')], which works exactly as I want EXCEPT THAT which of these two options the dialog comes up restricting to seems completely random. It does not seem to be determinate by order of the list, or even by operating system (the problem occurs in both Windows and Linux at least in my experience). There's a "default file type" option that can be defined, but despite what seems like an obvious intention this also does not effect which of the file types out of my list is used as the default. So, I don't have an answer to this yet, but here are some results from google searches on the topic:

Here is somebody with my exact problem. Sadly nobody answered his question. I have to disagree that the issue is OS related, since the filetype dropbox default comes up differently at different times on my one single Windows system.

http://stackoverflow.com/questions/23392531/how-to-control-the-order-of-file-types-in-tkfiledialog-for-different-windows-pla

The basic man page and options for tkFileDialog:

http://tkinter.unpythonic.net/wiki/tkFileDialog

This answer didn't seem helpful to me since I've tried changing the order of my list also, but maybe it has something to do with how they're explicitly defining a dictionary for their file options? I don't know.

http://stackoverflow.com/questions/11352278/default-file-type-in-tkfiledialogs-askopenfilename-method

The original question in this thread was more simple than mine, but the discussion does show a little bit about how the definition of the file type options is a list of tuples:

https://www.reddit.com/r/learnpython/comments/3loul5/only_showing_certain_filetypes_in_filedialog/

Probably if I read up on the OS DLL for the file dialog itself, it might show how to define the order of the file type list in a way that the OS can understand, and then figure out how to do the same thing in Python. Maybe the key is to use an ordered dictionary instead of a regular dictionary?

How to write programs that run in Tkinter

There are thousands and thousands of tutorials and discussion threads about Tkinter (and by extension Tcl/Tk), but very few of them seem to address the proper approach to writing a useful program that uses Tkinter. It is easy to think first about the code you want to write to accomplish whatever task your program is supposed to do and work later on whatever GUI you'll build for your user interface. This approach however always results for me in hitting a big wall when I start coding up the GUI. That's because Tkinter (and by extension, etc) is not naitively multi threaded. It needs to either be the only thing running, or be the only thing running in a special sandbox that you make for it by implementing your own threading. It is often unhelpfully pointed out in forum threads that Tkinter is event driven. What this is meant to say is that you can't just write linear code and call routines that do Tkinter stuff; the code that you write can only consist of responses to events in your Tkinter GUI. Essentially, Tkinter is the language of whatever program you are writing; it just happens to have pythonic syntax. It's sort of like how C++ has C-like syntax. This seems terribly unfair at first, but after you decide to do it Tkinter's way it turns out to be not so bad. I've written GUIs using both methods, but these day's I'm finding it easiest to let my programs be Tkinter programs instead of Python programs that use Tkinter.

The breakthrough for me was an example that I found that was structured like an executive loop, although it wasn't presented as such by it's author. What was happening in this example was that the __init__ method was calling a start() method that then called a function that ran some code and then called itself again after a short wait using after(). mainloop() then serviced each of the calls of the executive function as well as the rest of the GUI. Here is that example:

So, the structure of the python __main__ routine for a useful Tkinter program needs to be:

1. define your top-level GUI object, which will call it's __init__. In that __init__ you build all the objects in your control screen and define its appearance.
2. If you don't do it in your __init__ method, you can call your GUI's start() function from your main python routine. But it won't run because mainloop() isn't running yet.
3. call your GUI's mainloop(). Now your start() function will run. It will need to do something that causes itself or another function to be scheduled to be run by mainloop(). The best thing to do here is to have an executive function that functions like a state machine; each time the executive is entered, it checks what the current state is and executes the code for that state, and decides whether to change the state variable before rescheduling itself.
4. Somewhere in your GUI's functions find a place to put a call to your GUI's quit() function, which will exit mainloop().
5. Now you're back in your python main function, where you can either do something else or exit.

Since most programs are basically linear, it is very easy to break them up into state machine pieces. One could even argue that all programs are state machines to begin with.

Here are some of the links that helped me figure this out:

Presently my go-to link that helped me get the idea:

http://stackoverflow.com/questions/29158220/tkinter-understanding-mainloop

From the above link, another example of the ball game with some better defined executive loop examples:

http://stackoverflow.com/questions/25430786/moving-balls-in-tkinter-canvas/25431690#25431690

http://stackoverflow.com/questions/8683217/when-do-i-need-to-call-mainloop-in-a-tkinter-application

https://www.reddit.com/r/learnpython/comments/2rpk0k/how_to_update_your_gui_in_tkinter_after_using/

This one has great discussions about both the threading and executive loop methods:

http://stupidpythonideas.blogspot.com/2013/10/why-your-gui-app-freezes.html

This question and answer sort of addressed the underlying Tkinter philosophy that is part of this topic:

http://programmers.stackexchange.com/questions/213935/why-use-classes-when-programming-a-tkinter-gui-in-python

A google search which brings more good answers of this sort:

https://www.google.com/#q=python+tkinter+define+code+to+be+run+with+mainloop()+is+called

One other tip that is helpful is that it's possible to call update() on your main screen from within the executive loop function to make things appear and move on the GUI while code is executing in the executive loop. In my example, one of my states has a loop of data processing that I want to show progress on; I can set the progress bar value and call update() for each iteration of the loop.