Setting up Windows XP to use a time server

I needed to set up my Windows and Linux systems to all use time servers so that the times in the error logs on all systems would match. This does not seem to be as easy as I remembered it being for XP systems, and super easy for the Linux systems except that we've lobotomized their ethernet interfaces so that the whole LAN can be taken off of the WAN whenever we need. Consequently, I never got to a working answer with either system. (Update: See final solution at the end of the post.) Here is what I've learned so far:

Network Time Protocol (NTP) works by having branch systems poll systems higher up the tree rather than going to a central hub. Presumably, the base of the tree is a reliable server. Reliability at the branch level is increased by having them poll more than one other system, in fact any number which are then voted against each other. The branching levels are called STRATUMs, and it's bad form to connect to a STRATUM1 system but STRATUM2 or lower is fine.

Astonishingly, in Windows it seems to be set up to use an NTP time server, but to configure it oddly requires nightmarish direct registry modification. Per the following links:

http://www.atomic-clock.galleon.eu.com/support/windows-xp-ntp-time-server.html
http://support.microsoft.com/kb/314054

This set of slides from NIST hints that is could be possible to set everything up by using net time, and then stopping and starting w32time, but that can't be true otherwise M$ and others would have that instead of the registry modifications in their hints.

http://tf.nist.gov/service/pdf/win2000xp.pdf


After reading these links, my thoughts were more along the lines of "maybe the laptops are already set up to time servers and I just need to prove it to myself.". So the question of how to check if a server has been set up led to further dead ends until I decided to give that up too. Apparently, net time and w32tm can be used to find this out, except that not so much on XP. Apparently the /querysntp option doesn't exist for net time on XP, and w32tm fails if the local domain, protocols, or firewall aren't set up right.

http://superuser.com/questions/425233/how-can-i-check-a-systems-current-ntp-configuration

When I tried net time on the laptops I got "System error 5 has occurred. Access is denied." (It worked fine on my own laptop.) This apparently means access to the remote system, apparently if any systems in the local domain are mismatched this will fail, or if there's a firewall issue, or oddly if the computer times are mismatched (hope this isn't it):

http://support.microsoft.com/kb/555644

From a rootkit vendor:

http://www.personalcomputerfixes.com/general-errors/how-to-fix-system-error-5-%E2%80%93-%E2%80%9Caccess-is-denied%E2%80%9D/

And w32tm was full of options, none of which included a simple query as to the setup, as far as I could tell.

So eventually, I decided that the PC times seem really close to my own which I know is on a server, so they must be getting time from a server. How about the linux box?

Finally, something easy:

https://help.ubuntu.com/8.04/serverguide/NTP.html

Getting less eay:

http://www.ntp.org/ntpfaq/NTP-a-faq.htm

Too bad we turned off both the nameservice and even the gateways for these machines because we couldn't be bothered with any sort of sophistication when setting them up to run off of the network. Ah well.

Update: From the first link just above, I learned that it is possible for a sysadmin to run ntpdate directly to do a one-shot clock setup from another server. Presumably I wasn't able to use the gateways on my linux box to get to any other servers, but here's where it all comes together: what about polling the other systems on my LAN that are in the /etc/hosts table? Yes, I was able to reset the time on the linux box by using the windows laptop as my NTP host. Using:

sudo ntpdate [ip of the windows XP system]

and boom the ubuntu time now matches the time on my Windows laptop. Crude but effective, and it wouldn't have worked if my XP firewall had been better set up).

Furthermore, I added an entry in /etc/cron.daily to run ntpdate every day since the unix boxes seem to have an impressive amount of clock drift (a couple of minutes over the course of a month). This required that I add an executable script (named ntpdate per the examples but I think that the name is unimportant) to this directory which contained my ntpdate command. This was covered in the first ubuntu link above, however I also had to do some reading up on cron and found some very nice links for that:

http://www.unixgeeks.org/security/newbie/unix/cron-1.html

http://www.unix.com/unix-dummies-questions-answers/5695-cron-how-tell-s-working.html

http://www.softpanorama.info/Utilities/cron.shtml

The last link above is the one that confirmed that these are indeed just shell scripts in this directory, and ignore lines that begin with the standard shell script comment character #. On a side note, apparently the version of cron that I'm running in ubuntu is called "Nixie cron".

Then I had to correct the timezone. I did this by doing:

sudo dpkg-reconfigure tzdata

Which launches a hilarious terminal server that offers bios-like choices for location that allowed me to pick "New York" and set the time to the correct time zone. I got this from the following link, which is very nice and also reiterates everything that my previous ntp references told me, but in a much prettier way:

https://help.ubuntu.com/community/UbuntuTime

Python one-line if statement

After days of fiddling, I finally broke the code for the one-line if syntax in Python. This is so that my gseos menu options can call a python routine, and have the result from that routine be the condition for calling other functions. I had previously already figured out that these menu options were essentially just python statements and that I could have multiple statements by linking them together with semicolons. This extends it even further, probably further than the gseos architects even intended.

My menu items now look like this:

if routine(): statement1;statement2;statement3

The key breakthrough was that after the semicolon had to be one space (a tab did not work) and to give up using curly braces around the three statements.

Windows 7 network usage

I was pretty sad about not getting that network usage icon in the bottom right corner of my task bar when my OS changed from XP to Windows 7. However, I just now found something even better, Resource Monitor. I'm still not clear on how I got it to start up, but it not only shows memory and network usage, but what task is accessing the network and where they are trying to access!

I began checking some of the addresses being accessed. One of them was a suspicious-looking 239.255.255.250. It turns out that this is a M$ kludge for discovering plug-and-play services on the network? Who would do such a terrible thing?!!

http://help.lockergnome.com/general/239-255-255-250-Port-1900--ftopict18953.html

Buried temple Turkey

Here are some pointers to info about that cool ancient temple that was found intentionally buried somewhere in Turkey. It's called Göbekli Tepe. It turns out that what was so amazing about this temple is that it predates the Sumerian civilization; the conclusion that has been reached is that it was built by hunter-gatherer civilizations although that barely makes sense. Another thing that I can't really wrap my brain around is that this grey, dusty, desert location is known to have been much more fertile in the past, and was the northern part of the "fertile crescent" which had the Euphrates Valley as the southern part. Apparently this is the result of human deforestation. Also interesting from the various narratives is some confusion over the temple burial timeline. Was is buried from the outside in, or were the whole temples buried with the side temples started after each burial? The pictures that I've found of this place simultaneously disappoint (there has been so little excavation to see the entire finding) and amaze (the reliefs and other details of pillars are remarkably clear).

Here of course is the Wikipedia article:

http://en.wikipedia.org/wiki/G%C3%B6bekli_Tepe

Here is a National Geographic article that has a humorously flip tone while still being fact-rich:

http://ngm.nationalgeographic.com/print/2011/06/gobekli-tepe/mann-text

Here is one guy's photostream of his personal visit there:

http://www.flickr.com/photos/rstiller/6372438647/in/photostream/

Replacement glass for Andersen SKS skylight

Here I go, having to get replacement glass for this skylight YET AGAIN. I installed the Andersens 5 years ago thinking that a respected brand name like that would be available for decades. Nope. They discontinued selling skylights the next year and I missed the memo from Home Depot. Fortunately they do have a parts department. Sadly, they want everything to be a difficult as possible and want measurements despite the fact that I have the part number of both the skylight and sash.

The skylights that I originally bought were SKS 2438 and 2446.

A pdf of the replacement parts catalog is here:

https://docs.google.com/viewer?a=v&q=cache:gT5lLBuraZIJ:www.andersenwindows.com/homeowner/pdfs/Skylight.pdf+&hl=en&gl=us&pid=bl&srcid=ADGEESiB0FRpA24VfZvvzk4U9n3D5iDzqVl3_uUeWQWjjY3G1nnlrfmWX57LxunbEMCcvyKp74XMB090-W25t8WiUq9t3G_JUirH0FFIP4arJlJ9PsbLt9LFfyleieh6gcmcbVibtf-w&sig=AHIEtbQHZddrMQM7Gum9j38W_wpK5QjhVw

That's right, I have to read the google doc because Andersen pulled it from here: www.andersenwindows.com/homeowner/pdfs/Skylight.pdf

It can also be found here:

http://www.windowparts.com/v/vspfiles/files/PDFs/Skylight.pdf

The part number for my replacement sash is 1880404

They don't have a way to buy it online, of course, but they apparently still do staff their parts phone center at 877-835-9090

Python how to get a list of files in a directory

Python has really nice utilities for getting a directory listing inside a program!

I used os.listdir, but there are many other methods. Some are not available in python 2.5.

Source:

http://stackoverflow.com/questions/2225564/python-get-a-filtered-list-of-files-in-directory

Python length of a "string"

len(object)

Python for-next loops

Here is an absolutely delightful summary of the python syntax for all the different kinds of loops:

http://wiki.python.org/moin/ForLoop

This is from wiki.python.org, which I should just go to from now on with all basic syntax questions.

Python open and read integers from binary file

How embarassing that I had to look this up. Here is the syntax in python for opening a binary file:

f = open("filename", "rb")

.
.
f.close()

From this example:

http://stackoverflow.com/questions/4971413/how-to-read-a-binary-file

This was part of a larger search where I was working on pulling values out of a (packet, obviously) binary file. The horrific answer was to read bytes from the file into a "string", then bust out each 8-bit (or other size) value as an integer. So what I ended up doing was reading in 6 "characters", yanking out the packet size from this primary header, and then using that value to read in another "string" of that many characters + 1.

Here is one example site, in which the read() command is used and the result is cast to different sizes of integers:

http://stackoverflow.com/questions/1163459/reading-integers-from-binary-file-in-python

Here is another example where bytes are read in one or chunksize at a time with a loop:

http://stackoverflow.com/questions/1035340/reading-binary-file-in-python

Python wait or pause

All examples that I've found for how to do this involve using the sleep() function from the time library. Here are two example pages:

http://www.pressthered.com/how_to_sleep_pause_wait_stop_your_python_code/

http://ubuntuforums.org/showthread.php?t=715907

sed insert lines before/after pattern match

Yet another wicked cool sed trick that I accomplished recently after a pile of googling. I needed to add a couple of lines of code into several files that all had great markers already in the code where I needed the new lines. I just had to do a pattern match on the marker phrases, and add the lines either before or after as needed. Looked like a job for sed (although some pointed out in the forum threads that I googled that awk would work probably easier). Anyhow, here's what I did to add a line after:

find . -name "prefix*.c" -exec sed -e '/pattern1/G;a\{new line1' -e'}' {} \;

I got the "G" part from examples and never did verify its offical capabilities, but that's what adds an extra blank line (from man, it seems to be appending the swap line to the current line?). The two -e arguments are the key to overcoming the difficulties with the "a" command.

In the spots where I wanted to add the line before a particular line, I didn't have time to figure out how to make a single command that would add the extra line plus a blank line, so I just wimped out and used two commands. The "x;p;x" bit is from another example in the single-liner file:

find . -name "prefix*.c" -exec sed -e '/pattern2/i\{new line2' -e'}' {} \;

find . -name "prefix*.c" -exec sed -e '/pattern2/{x;p;x;}' {} \;

Linotype Machine

After reading an effusive description of the technological achievement of the linotype machine, I started reading into it.

What I've learned is that it was a pretty cool machine, but mind-bogglingly so for when it was first made in the 1880's. It was also pretty cool that the technology was in use all the way up to the 1970's.

An image search shows that the machines changed very little over time, except that all the early images show it as belt driven:

https://www.google.com/search?q=linotype+machine&hl=en&tbo=u&tbm=isch&source=univ&sa=X&ei=GiGcUOmnEOWW2AWb54HABA&sqi=2&ved=0CDcQsAQ&biw=1117&bih=483

It was primarily a high speed sorter of tiny letter molds, coupled with a really ingenius high-speed caster. There was a fairly modern-looking keyboard that caused letter molds to be ejected from a big tray. They slid down a ramp into a line build-up bar. Spacers were wedges dropped in from a different source. Then the whole built-up line of molds would be mechanically moved to a different section of the machine, the spacing between words would be adjusted using the wedges to produce full allignment, then the face of the line would be pressed in a tight seal onto a channel into which hot metal would be flowed, the resulting slug would then undergo a surprising amount of further mechanical processing before being ejected, all at an incredible number of lines per minute. The built up line would then get all its pieces sorted back into the starting tray using a wonderful keying mechanism. Then folks had to set the slugs up on a printing press by hand. It's a pretty sobering example of a nearly impossibly complex job being enabled by automating selected parts of the process.

Some links, starting from the wikipedia article:

http://en.wikipedia.org/wiki/Linotype_machine

and

http://en.wikipedia.org/wiki/Ottmar_Mergenthaler

Which seems to have mostly been lifted from one Maryland history web page:

http://www.zionbaltimore.org/history_people_mergenthaler.htm

Here are a couple of perspectives on the genius of this machine. The 8th wonder of the world:

http://ntluckie.com/#The-Linotype-Machine

A museum piece with an interesting placard quantifying the complexity of this machine -- "it had more moving parts when it was constructed than anything else ever built by man" :

http://www.seobythesea.com/2012/08/indexing-recent-content-in-search-engines/

A brief summary of the mechanism:

http://www.woodsidepress.com/LINOTYPE.HTML

A modern person's attempt to understand the workings of the machine, particularly the sorting mechanism:

http://www.circuitousroot.com/artifice/letters/press/compline/studies/matrices/matrix-teeth/tooth-chart/index.html

Here is one of hundreds of linotype youTube videos. I love how this shows the mechanical action:

http://www.youtube.com/watch?v=PRYxOs1oCRY

Haunted Mansion "alligator tightrope girl"

This post on boingboing:

http://boingboing.net/2012/11/07/haunted-mansion-alligator-ti.html

Led down a rabbit hole of Haunted Mansion fandom for this one particular tiny part of the Disneyland attaction,:

http://longforgottenhauntedmansion.blogspot.com/2011/02/many-faces-of-tightrope-walker.html

http://micechat.com/forums/disneyland-resort/125031-long-forgotten-haunted-mansion-effect-part-6-prequel-time-unrelenting-126.html

and for the Haunted Mansion in general:

http://longforgottenhauntedmansion.blogspot.com/

history bang recall substitution

I had to look this up and try it a couple of times before I got the syntax right. The correct syntax for history bang recall substitution.

!1026:s/original/replacement/

Here is an absolutely incredible online tutorial about history bang recall:

http://www.softpanorama.org/Scripting/Shellorama/bash_command_history_reuse.shtml

Here is a much smaller page on the subject, it might have something extra to add:

http://pic.dhe.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%2Fcom.ibm.aix.baseadmn%2Fdoc%2Fbaseadmndita%2Fc_hist_sub.htm

Maryland laws regarding booster seat and rear facing

Apparently, the actual laws regarding when a kid can go from a car safety seat to a booster seat are pretty vague. It just says if the child is under 8, the child has to be in an "appopriatate" (read approved and manufacturer's instructions are being followed) car seat of some sort:

http://www.peoples-law.org/node/695

Recent changes to the law did not actually include the amount of time that a child has to be rear-facing, they just increased the time a car-seat must be used to 8 years:

http://articles.baltimoresun.com/2012-04-10/health/bal-child-safety-law_1_safety-seat-md-lawmakers-doctors-group

From the state itself:

http://fha.dhmh.maryland.gov/ohpetup/kiss/sitepages/mdlaw.aspx

In the FAQ, somebody points out what complete garbage "Booster" seats are (most do not attach to the car in any way):

http://fha.dhmh.maryland.gov/ohpetup/kiss/SitePages/faq.aspx

Also notable from the FAQ is apparently I've broken the law by installing the rear-facing seat using both LATCH and the seatbelt.

So, a precious few booster seats do have LATCH straps, like this one:

http://www.amazon.com/Britax-Parkway-Booster-Seat-Nutmeg/dp/B0047O3Y2G/ref=sr_1_1?ie=UTF8&qid=1352039944&sr=8-1&keywords=booster+car+seat+with+latch+system

But not this one:

http://www.amazon.com/Britax-Parkway-Booster-Seat-Onyx/dp/B002BH3NEU/ref=pd_sim_ba_2

Or this one:

http://www.amazon.com/Graco-Highback-TurboBooster-Seat-Spitfire/dp/B004GEC8LU/ref=pd_sim_ba_3

Here is a nice chart showing the transition points between each kind of car seat:

http://www.toysrus.com/info/index.jsp?categoryId=2279366&infoPath=2279362#E

sed awk join two lines and print nth field from line using awk

Just a series of old-unix-guy tricks that I half-remembered from the last time that I had to do it. The result is still bitchin though. My log files have the variable with the bad value on one line, and the value in question on the next line - d'oh! The variable names all have the same prefix, so I realized that I could grep or sed for that. Then all I had to do was find a way to grab the next line and do stuff with it to pull out the bad value in question. To make a single line printout with the time, date, variable name, and value, my command line turned out to be:

sed -n '/PREFIX/{N;s/\n//;p}' 20121022_123632.msg | awk '{print $2 "\t" $3 "\t" $8 "\t" $NF}' > output_file.txt

I remembered that N existed to read in the next line in sed. The key of removing the inner newline with s/\n// was from this page (I had to disregard the otherwise pretty cool-looking looping that the example used):

http://stackoverflow.com/questions/7852132/sed-join-lines-together

The nice thing about using awk is that it defaults to using spaces for field separators automatically. Here's a related example, which is also where I got that cool $NF trick for printing the last field:

http://www.delorie.com/gnu/docs/gawk/gawk_36.html

Printing tab characters in the output is helpful since the variable name widths vary. I was using spaces previously just by trying out using a space inside quote marks; what I learned was that to print the tab characters, they have to still have to be inside quotes. Here's an example (yes, this is essentially the usenet awk manual):

http://www.catonmat.net/blog/wp-content/uploads/2008/09/awk1line.txt

Here are a couple of examples from early in my search for answers to this problem, none of which I went with but which could be helpful to understand:

Has an example with brackets, carrets, and asterixes, no idea how this works: http://www.unix.com/shell-programming-scripting/105070-sed-command-extract-1st-word-every-line.html

Has more brackets, carrets, and astreixes, plus some numbers and nice example of regex construction: http://www.unix.com/shell-programming-scripting/175591-using-sed-extract-word-string.html

Update: I have since continued to work this problem, and have come up with another solution involving only awk. The reason was that I wanted to use my command line with a recursive find such that I could run this command line on each file found, depositing the results for each found file back into a new file in the same directory as each found file with the same name as the found file but with a new suffix. I could either use xargs or use the -exec feature of find to run the command line on the results. What several hours of trying taught me was that piping things between sed and awk wasn't possible to do under either find -exe or xargs. Eventually, I found a way to do all the line manipulation in awk with a single command line without piping and then use it with find -exec which has a much nicer argument expansion capability than xargs. Here's the final result:

find . -name "KEYWORD*.log" -execdir awk '/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"{}.ALARMS";getline;print $NF>>"{}.ALARMS"}' {} \;

The key features here are: 1) regex for matching just like sed, 2) the use of printf to print the result from the first line without a carriage return so that the next thing printed (from the next line) would be printed with the result from the first line, 3) the use of getline to just freaking get the next line, and do a print tailored just to that line, 4) the use of >> for each print statement for the filename (and the fliename *has* to be in quotes, but hahaha find -exec graciously expands the {} to form the filename with the new suffix right in the awk script, as well as sending the filename to awk as the final argument). 5) -execdir instead of -exec which causes the command to be run in the directory where the file was found (and thus the result to be deposited in the same directory)

Here's an example of methodology that I didn't end up going with but which is still cool: It shows saving the current line in awk in a variable, using $0 to get the entire line, and then something that I don't quite get going on with "next" that seems like it should be similar to getline but doesn't seem to be, all to join two lines together. Also hilariously makes fun of using cat when awk accepts an input filename as an argument (Useless Use Of Cat):

http://compgroups.net/comp.unix.shell/need-one-line-awk-or-sed-command-to-condition/1210370

Here's another page with more examples of using "next". Some of these examples are really deep and could use much more careful study:

http://www.theunixschool.com/2012/05/awk-join-or-merge-lines-on-finding.html

Here is the example of using getline. What's neat about this forum thread is somebody suggested the other side of the coin, how to do it in sed! I don't understand the sed syntax suggested, so there's probably something important to learn there:

http://www.linuxquestions.org/questions/programming-9/awk-help-print-matching-line-and-next-three-931966/

Here is the example site that I read for doing output to a file in awk. I'd seen a hint of this on another page where they used ">>":

http://www.unix.com/shell-programming-scripting/55172-awk-print-redirection-variable-file-name.html

Here is a nice sed tutorial that talks about those curly brackets (although it doesn't show them all one line like my command). This was a part of my search for trying to get sed to maybe send its output to a file that I could later maybe read in with awk (before I got smart and just decided to do everything in awk):

http://www.ibm.com/developerworks/linux/library/l-sed2/index.html

And, as part of that search, I ran across this deep page, which aside from all the many other wisardly tips clued me in to the secret that you can have multiple -e arguments to sed, and it just treats the result like one sed script:

http://stackoverflow.com/questions/3472404/how-to-go-from-a-multiple-line-sed-command-in-command-line-to-single-line-in-scr

So, the problem that I was having was having an output filename that I could make using filename expansion based on the original found file. I never got that working in sed (and it was essentially a dead end anyhow) because it wouldn't expand the filename. Here is a site about outputting to a file in sed; it's examples are pretty simple though:

http://www.thegeekstuff.com/2009/10/unix-sed-tutorial-how-to-write-to-a-file-using-sed/

I also spent a long time searching for a way to either call sed from awk or awk from sed. While it seems to be possible, all of the example sites that I found were actually pretty confused and incomplete.

And while searching for solutions to my problem of trying to get sed-piped-to-awk into my find command, I ran across the following two breathtakingly learned (but basically unhelpful) forum threads:

http://www.linuxquestions.org/questions/linux-software-2/using-a-pipe-inside-find-exec-how-does-it-work-and-why-is-it-so-slow-816525/

http://ubuntuforums.org/showthread.php?t=1436654

The Conclusion: After coming up with that bitchin find command and thoroughly testing it on my own system using Cygwin, it bombed when I ran it on the server. That one may be running BSD or something (although I'm not sure since a man for awk on that system returns the page for gawk). The failure that I got was a syntax error on the expanded result. For some reason, even though I fed the following into my -execdir in find,:

awk '/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"{}.ALARMS";getline;print $NF>>"{}.ALARMS"}' {}

The syntax error that I got back would show an extra ./ prepended to the expanded command line being fed to awk:

awk: syntax error in: './/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"input_file.ALARMS";getline;print $NF>>"input_file.ALARMS"}'

The expanded filenames were always correct, and I tried escaping a bunch of stuff to no avail. In the end, getting rid of the expansions for the output file and just using a generic hard-coded output file name mysteriously fixed the problem, and worked out better in the end. So, in the end, I had to go with this:

find . -name "KEYWORD*.log" -execdir awk '/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"ALARMS.txt";getline;print $NF>>"ALARMS.txt"}' {} \;

Oh, and while struggling to figure out this new problem, I came across this lovely site that compares and contrasts using find -exe and find | xargs. I can probably go back and read this top to bottom to better educate myself:

http://www.softpanorama.info/Tools/Find/using_exec_option_and_xargs_in_find.shtml

And here's an adorable little xargs page:

http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html