sed insert lines before/after pattern match

Yet another wicked cool sed trick that I accomplished recently after a pile of googling. I needed to add a couple of lines of code into several files that all had great markers already in the code where I needed the new lines. I just had to do a pattern match on the marker phrases, and add the lines either before or after as needed. Looked like a job for sed (although some pointed out in the forum threads that I googled that awk would work probably easier). Anyhow, here's what I did to add a line after:

find . -name "prefix*.c" -exec sed -e '/pattern1/G;a\{new line1' -e'}' {} \;

I got the "G" part from examples and never did verify its offical capabilities, but that's what adds an extra blank line (from man, it seems to be appending the swap line to the current line?). The two -e arguments are the key to overcoming the difficulties with the "a" command.

In the spots where I wanted to add the line before a particular line, I didn't have time to figure out how to make a single command that would add the extra line plus a blank line, so I just wimped out and used two commands. The "x;p;x" bit is from another example in the single-liner file:

find . -name "prefix*.c" -exec sed -e '/pattern2/i\{new line2' -e'}' {} \;

find . -name "prefix*.c" -exec sed -e '/pattern2/{x;p;x;}' {} \;

Linotype Machine

After reading an effusive description of the technological achievement of the linotype machine, I started reading into it.

What I've learned is that it was a pretty cool machine, but mind-bogglingly so for when it was first made in the 1880's. It was also pretty cool that the technology was in use all the way up to the 1970's.

An image search shows that the machines changed very little over time, except that all the early images show it as belt driven:

https://www.google.com/search?q=linotype+machine&hl=en&tbo=u&tbm=isch&source=univ&sa=X&ei=GiGcUOmnEOWW2AWb54HABA&sqi=2&ved=0CDcQsAQ&biw=1117&bih=483

It was primarily a high speed sorter of tiny letter molds, coupled with a really ingenius high-speed caster. There was a fairly modern-looking keyboard that caused letter molds to be ejected from a big tray. They slid down a ramp into a line build-up bar. Spacers were wedges dropped in from a different source. Then the whole built-up line of molds would be mechanically moved to a different section of the machine, the spacing between words would be adjusted using the wedges to produce full allignment, then the face of the line would be pressed in a tight seal onto a channel into which hot metal would be flowed, the resulting slug would then undergo a surprising amount of further mechanical processing before being ejected, all at an incredible number of lines per minute. The built up line would then get all its pieces sorted back into the starting tray using a wonderful keying mechanism. Then folks had to set the slugs up on a printing press by hand. It's a pretty sobering example of a nearly impossibly complex job being enabled by automating selected parts of the process.

Some links, starting from the wikipedia article:

http://en.wikipedia.org/wiki/Linotype_machine

and

http://en.wikipedia.org/wiki/Ottmar_Mergenthaler

Which seems to have mostly been lifted from one Maryland history web page:

http://www.zionbaltimore.org/history_people_mergenthaler.htm

Here are a couple of perspectives on the genius of this machine. The 8th wonder of the world:

http://ntluckie.com/#The-Linotype-Machine

A museum piece with an interesting placard quantifying the complexity of this machine -- "it had more moving parts when it was constructed than anything else ever built by man" :

http://www.seobythesea.com/2012/08/indexing-recent-content-in-search-engines/

A brief summary of the mechanism:

http://www.woodsidepress.com/LINOTYPE.HTML

A modern person's attempt to understand the workings of the machine, particularly the sorting mechanism:

http://www.circuitousroot.com/artifice/letters/press/compline/studies/matrices/matrix-teeth/tooth-chart/index.html

Here is one of hundreds of linotype youTube videos. I love how this shows the mechanical action:

http://www.youtube.com/watch?v=PRYxOs1oCRY

Haunted Mansion "alligator tightrope girl"

This post on boingboing:

http://boingboing.net/2012/11/07/haunted-mansion-alligator-ti.html

Led down a rabbit hole of Haunted Mansion fandom for this one particular tiny part of the Disneyland attaction,:

http://longforgottenhauntedmansion.blogspot.com/2011/02/many-faces-of-tightrope-walker.html

http://micechat.com/forums/disneyland-resort/125031-long-forgotten-haunted-mansion-effect-part-6-prequel-time-unrelenting-126.html

and for the Haunted Mansion in general:

http://longforgottenhauntedmansion.blogspot.com/

history bang recall substitution

I had to look this up and try it a couple of times before I got the syntax right. The correct syntax for history bang recall substitution.

!1026:s/original/replacement/

Here is an absolutely incredible online tutorial about history bang recall:

http://www.softpanorama.org/Scripting/Shellorama/bash_command_history_reuse.shtml

Here is a much smaller page on the subject, it might have something extra to add:

http://pic.dhe.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%2Fcom.ibm.aix.baseadmn%2Fdoc%2Fbaseadmndita%2Fc_hist_sub.htm

Maryland laws regarding booster seat and rear facing

Apparently, the actual laws regarding when a kid can go from a car safety seat to a booster seat are pretty vague. It just says if the child is under 8, the child has to be in an "appopriatate" (read approved and manufacturer's instructions are being followed) car seat of some sort:

http://www.peoples-law.org/node/695

Recent changes to the law did not actually include the amount of time that a child has to be rear-facing, they just increased the time a car-seat must be used to 8 years:

http://articles.baltimoresun.com/2012-04-10/health/bal-child-safety-law_1_safety-seat-md-lawmakers-doctors-group

From the state itself:

http://fha.dhmh.maryland.gov/ohpetup/kiss/sitepages/mdlaw.aspx

In the FAQ, somebody points out what complete garbage "Booster" seats are (most do not attach to the car in any way):

http://fha.dhmh.maryland.gov/ohpetup/kiss/SitePages/faq.aspx

Also notable from the FAQ is apparently I've broken the law by installing the rear-facing seat using both LATCH and the seatbelt.

So, a precious few booster seats do have LATCH straps, like this one:

http://www.amazon.com/Britax-Parkway-Booster-Seat-Nutmeg/dp/B0047O3Y2G/ref=sr_1_1?ie=UTF8&qid=1352039944&sr=8-1&keywords=booster+car+seat+with+latch+system

But not this one:

http://www.amazon.com/Britax-Parkway-Booster-Seat-Onyx/dp/B002BH3NEU/ref=pd_sim_ba_2

Or this one:

http://www.amazon.com/Graco-Highback-TurboBooster-Seat-Spitfire/dp/B004GEC8LU/ref=pd_sim_ba_3

Here is a nice chart showing the transition points between each kind of car seat:

http://www.toysrus.com/info/index.jsp?categoryId=2279366&infoPath=2279362#E

sed awk join two lines and print nth field from line using awk

Just a series of old-unix-guy tricks that I half-remembered from the last time that I had to do it. The result is still bitchin though. My log files have the variable with the bad value on one line, and the value in question on the next line - d'oh! The variable names all have the same prefix, so I realized that I could grep or sed for that. Then all I had to do was find a way to grab the next line and do stuff with it to pull out the bad value in question. To make a single line printout with the time, date, variable name, and value, my command line turned out to be:

sed -n '/PREFIX/{N;s/\n//;p}' 20121022_123632.msg | awk '{print $2 "\t" $3 "\t" $8 "\t" $NF}' > output_file.txt

I remembered that N existed to read in the next line in sed. The key of removing the inner newline with s/\n// was from this page (I had to disregard the otherwise pretty cool-looking looping that the example used):

http://stackoverflow.com/questions/7852132/sed-join-lines-together

The nice thing about using awk is that it defaults to using spaces for field separators automatically. Here's a related example, which is also where I got that cool $NF trick for printing the last field:

http://www.delorie.com/gnu/docs/gawk/gawk_36.html

Printing tab characters in the output is helpful since the variable name widths vary. I was using spaces previously just by trying out using a space inside quote marks; what I learned was that to print the tab characters, they have to still have to be inside quotes. Here's an example (yes, this is essentially the usenet awk manual):

http://www.catonmat.net/blog/wp-content/uploads/2008/09/awk1line.txt

Here are a couple of examples from early in my search for answers to this problem, none of which I went with but which could be helpful to understand:

Has an example with brackets, carrets, and asterixes, no idea how this works: http://www.unix.com/shell-programming-scripting/105070-sed-command-extract-1st-word-every-line.html

Has more brackets, carrets, and astreixes, plus some numbers and nice example of regex construction: http://www.unix.com/shell-programming-scripting/175591-using-sed-extract-word-string.html

Update: I have since continued to work this problem, and have come up with another solution involving only awk. The reason was that I wanted to use my command line with a recursive find such that I could run this command line on each file found, depositing the results for each found file back into a new file in the same directory as each found file with the same name as the found file but with a new suffix. I could either use xargs or use the -exec feature of find to run the command line on the results. What several hours of trying taught me was that piping things between sed and awk wasn't possible to do under either find -exe or xargs. Eventually, I found a way to do all the line manipulation in awk with a single command line without piping and then use it with find -exec which has a much nicer argument expansion capability than xargs. Here's the final result:

find . -name "KEYWORD*.log" -execdir awk '/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"{}.ALARMS";getline;print $NF>>"{}.ALARMS"}' {} \;

The key features here are: 1) regex for matching just like sed, 2) the use of printf to print the result from the first line without a carriage return so that the next thing printed (from the next line) would be printed with the result from the first line, 3) the use of getline to just freaking get the next line, and do a print tailored just to that line, 4) the use of >> for each print statement for the filename (and the fliename *has* to be in quotes, but hahaha find -exec graciously expands the {} to form the filename with the new suffix right in the awk script, as well as sending the filename to awk as the final argument). 5) -execdir instead of -exec which causes the command to be run in the directory where the file was found (and thus the result to be deposited in the same directory)

Here's an example of methodology that I didn't end up going with but which is still cool: It shows saving the current line in awk in a variable, using $0 to get the entire line, and then something that I don't quite get going on with "next" that seems like it should be similar to getline but doesn't seem to be, all to join two lines together. Also hilariously makes fun of using cat when awk accepts an input filename as an argument (Useless Use Of Cat):

http://compgroups.net/comp.unix.shell/need-one-line-awk-or-sed-command-to-condition/1210370

Here's another page with more examples of using "next". Some of these examples are really deep and could use much more careful study:

http://www.theunixschool.com/2012/05/awk-join-or-merge-lines-on-finding.html

Here is the example of using getline. What's neat about this forum thread is somebody suggested the other side of the coin, how to do it in sed! I don't understand the sed syntax suggested, so there's probably something important to learn there:

http://www.linuxquestions.org/questions/programming-9/awk-help-print-matching-line-and-next-three-931966/

Here is the example site that I read for doing output to a file in awk. I'd seen a hint of this on another page where they used ">>":

http://www.unix.com/shell-programming-scripting/55172-awk-print-redirection-variable-file-name.html

Here is a nice sed tutorial that talks about those curly brackets (although it doesn't show them all one line like my command). This was a part of my search for trying to get sed to maybe send its output to a file that I could later maybe read in with awk (before I got smart and just decided to do everything in awk):

http://www.ibm.com/developerworks/linux/library/l-sed2/index.html

And, as part of that search, I ran across this deep page, which aside from all the many other wisardly tips clued me in to the secret that you can have multiple -e arguments to sed, and it just treats the result like one sed script:

http://stackoverflow.com/questions/3472404/how-to-go-from-a-multiple-line-sed-command-in-command-line-to-single-line-in-scr

So, the problem that I was having was having an output filename that I could make using filename expansion based on the original found file. I never got that working in sed (and it was essentially a dead end anyhow) because it wouldn't expand the filename. Here is a site about outputting to a file in sed; it's examples are pretty simple though:

http://www.thegeekstuff.com/2009/10/unix-sed-tutorial-how-to-write-to-a-file-using-sed/

I also spent a long time searching for a way to either call sed from awk or awk from sed. While it seems to be possible, all of the example sites that I found were actually pretty confused and incomplete.

And while searching for solutions to my problem of trying to get sed-piped-to-awk into my find command, I ran across the following two breathtakingly learned (but basically unhelpful) forum threads:

http://www.linuxquestions.org/questions/linux-software-2/using-a-pipe-inside-find-exec-how-does-it-work-and-why-is-it-so-slow-816525/

http://ubuntuforums.org/showthread.php?t=1436654

The Conclusion: After coming up with that bitchin find command and thoroughly testing it on my own system using Cygwin, it bombed when I ran it on the server. That one may be running BSD or something (although I'm not sure since a man for awk on that system returns the page for gawk). The failure that I got was a syntax error on the expanded result. For some reason, even though I fed the following into my -execdir in find,:

awk '/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"{}.ALARMS";getline;print $NF>>"{}.ALARMS"}' {}

The syntax error that I got back would show an extra ./ prepended to the expanded command line being fed to awk:

awk: syntax error in: './/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"input_file.ALARMS";getline;print $NF>>"input_file.ALARMS"}'

The expanded filenames were always correct, and I tried escaping a bunch of stuff to no avail. In the end, getting rid of the expansions for the output file and just using a generic hard-coded output file name mysteriously fixed the problem, and worked out better in the end. So, in the end, I had to go with this:

find . -name "KEYWORD*.log" -execdir awk '/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"ALARMS.txt";getline;print $NF>>"ALARMS.txt"}' {} \;

Oh, and while struggling to figure out this new problem, I came across this lovely site that compares and contrasts using find -exe and find | xargs. I can probably go back and read this top to bottom to better educate myself:

http://www.softpanorama.info/Tools/Find/using_exec_option_and_xargs_in_find.shtml

And here's an adorable little xargs page:

http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html

The incredible batch call of wkhtmltopdf

Like extra credit, this command line combined nearly everything I've done recently all at once. Within cygwin, I'm using find on a directory with the -exec option, and calling wkhtmltopdf on every file found. The only clunky thing about this method is that it makes .htm.pdf files which I have to rename to .pdf.

find . -name "D*.htm" -exec cmd /C "C:\Program Files\wkhtmltopdf\wkhtmltopdf.exe" -s Letter {} {}.pdf \;

The awesome trick was doing this in cygwin. For some reason, wkhtmltopdf wasn't running in cygwin naitive mode (it is a GnuWin utility in the form that I have it in), so I found from the below blog how to use cmd /C and I stuck the whole thing into the -exec of find and it somehow worked.

Yes, well, the downside is that I have to trim an extra .htm out of the pdf filename, I suppose that a real unix master would run the result of find through something else to trim the orignal suffix off before passing it into cmd /C, I should figure that out some day.

http://siddesh-bg.blogspot.com/2012/07/how-to-run-dos-commands-from-cygwin.html