Linux print nth line of file

I have to google this all the time. I guess that the following article answered it for me back in Nov 2020. The answer includes piping the output of head into tail, using sed, or using awk.

https://linuxhandbook.com/display-specific-lines/

Honestly, the unrelated hits from searching for this are fascinating in their own right:

https://www.bing.com/search?q=linux+print+nth+line+of+file

Converting a directory listing into a shell script of copy commands

I needed to copy all the files of a certain name throughout a directory tree into a particular directory, renaming them to contain the name of the path where they had been found so that they could be restored from the copy if needed later.

Trying to be quick about it, I decided to just pipe the result of "find" into a file, and then use a stream editor to modify the file into a shell script that would do the copy.

find . -name "filename.txt" > ~/filelist.txt

After fiddling with sed as an option for changing each line of the file into a copy command, I eventually found awk to be easier to use for this purpose (gawk on all our Linux systems).

gawk '{{ORS=""};print "cp -v " $0;gsub(/\//,"+",$0);{ORS="\n"};print " ./files_backup/" $0}' ~/filelist.txt > ~/files_backup.sh

So this reads filelist.txt which is a list of all found instances of filename.txt with full paths, creates a cp -v command with the original path as the first argument and a second argument with a destination directory of files_backup and a destination filename consisting of the first argument with all "/" characters replaced with +; the result is output to files_backup.sh which is the shell script that will actually do the copying for me. I didn't want to waste more time getting fancy by trying to directly execute the command, plus this gives me a chance to check the result before running it. The "+" character was chosen for the destination filenames because some of the directories in the source paths already had underscores and dashes in their names, and it appears as though the + is an allowable filename character.

I was bedeviled by awk's default behavior of inserting \n to the results of each print statement; I knew that I'd overcome this in the past but for today my Google search yielded a workaround of changing the end of line character to nothing for the first print statement (I had to change it back for the second print statement in order to get the completed line with a \n at the end.

Some links:

A typical sed replacement example forum thread:

http://www.unix.com/shell-programming-and-scripting/12657-using-sed-replace-part-string.html

A fascinating diversion into bash for loops. At one point I tried using a for loop to generate copy commands "live" from the file list but it didn't work because the list that for uses seems to need to be an actual directory listing rather than lines from a file.

http://www.cyberciti.biz/faq/bash-for-loop/

Here's a bunch of different approaches to a similar (although simpler) task. This thread has examples of using sed, a for loop, and the "rename" command which I hadn't heard about before as possible solutions.

http://unix.stackexchange.com/questions/7161/copy-rename-multiple-files-using-regular-expression-shell-script

A link with basic awk syntax:

http://www.grymoire.com/Unix/Awk.html

Another page of basic awk syntax that shows the syntax for the gsub command and an example:

http://www.staff.science.uu.nl/~oostr102/docs/nawk/nawk_92.html

Another page of basic awk syntax, for print statments.

http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-awk-print-examples/

Here's a basic question that I keep forgetting the answer to with awk, how to control whether concatenated outputs are padded with white space or not; the answer is really quite simple!

http://stackoverflow.com/questions/9985330/how-can-i-get-awk-to-print-without-white-space

I was attempting to copy the input argument to a new variable and modify the new variable the first time I tried to do this, but for some reason the modifications that I was making to the new variable affected the input variable also. I didn't have time to figure out why this was happening, and thus had to reorder my awk statment such that I printed the input argument before modifying it, which is what led to having to fiddle with ORS. Perhaps something in this page of examples, which I was trying to follow but somehow failed, explains why I was having this problem:

http://www.thegeekstuff.com/2010/01/awk-tutorial-understand-awk-variables-with-3-practical-examples/

Here's where I got the solution to directly change ORS from:

http://stackoverflow.com/questions/2021982/awk-without-printing-newline

Here's the information on what are the allowed filename characters in *nix (and other OSs too):

http://en.wikipedia.org/wiki/Filename

Delete blank lines with sed

Use this:

sed '/^ *$/d'

Not sure that I understand why this works. ^ is beginning of line, $ is end of line. Why would " *" match only blank lines?

Found here: http://www.unix.com/shell-programming-scripting/22666-delete-blank-lines-sed.html

sed insert lines before/after pattern match

Yet another wicked cool sed trick that I accomplished recently after a pile of googling. I needed to add a couple of lines of code into several files that all had great markers already in the code where I needed the new lines. I just had to do a pattern match on the marker phrases, and add the lines either before or after as needed. Looked like a job for sed (although some pointed out in the forum threads that I googled that awk would work probably easier). Anyhow, here's what I did to add a line after:

find . -name "prefix*.c" -exec sed -e '/pattern1/G;a\{new line1' -e'}' {} \;

I got the "G" part from examples and never did verify its offical capabilities, but that's what adds an extra blank line (from man, it seems to be appending the swap line to the current line?). The two -e arguments are the key to overcoming the difficulties with the "a" command.

In the spots where I wanted to add the line before a particular line, I didn't have time to figure out how to make a single command that would add the extra line plus a blank line, so I just wimped out and used two commands. The "x;p;x" bit is from another example in the single-liner file:

find . -name "prefix*.c" -exec sed -e '/pattern2/i\{new line2' -e'}' {} \;

find . -name "prefix*.c" -exec sed -e '/pattern2/{x;p;x;}' {} \;

sed awk join two lines and print nth field from line using awk

Just a series of old-unix-guy tricks that I half-remembered from the last time that I had to do it. The result is still bitchin though. My log files have the variable with the bad value on one line, and the value in question on the next line - d'oh! The variable names all have the same prefix, so I realized that I could grep or sed for that. Then all I had to do was find a way to grab the next line and do stuff with it to pull out the bad value in question. To make a single line printout with the time, date, variable name, and value, my command line turned out to be:

sed -n '/PREFIX/{N;s/\n//;p}' 20121022_123632.msg | awk '{print $2 "\t" $3 "\t" $8 "\t" $NF}' > output_file.txt

I remembered that N existed to read in the next line in sed. The key of removing the inner newline with s/\n// was from this page (I had to disregard the otherwise pretty cool-looking looping that the example used):

http://stackoverflow.com/questions/7852132/sed-join-lines-together

The nice thing about using awk is that it defaults to using spaces for field separators automatically. Here's a related example, which is also where I got that cool $NF trick for printing the last field:

http://www.delorie.com/gnu/docs/gawk/gawk_36.html

Printing tab characters in the output is helpful since the variable name widths vary. I was using spaces previously just by trying out using a space inside quote marks; what I learned was that to print the tab characters, they have to still have to be inside quotes. Here's an example (yes, this is essentially the usenet awk manual):

http://www.catonmat.net/blog/wp-content/uploads/2008/09/awk1line.txt

Here are a couple of examples from early in my search for answers to this problem, none of which I went with but which could be helpful to understand:

Has an example with brackets, carrets, and asterixes, no idea how this works: http://www.unix.com/shell-programming-scripting/105070-sed-command-extract-1st-word-every-line.html

Has more brackets, carrets, and astreixes, plus some numbers and nice example of regex construction: http://www.unix.com/shell-programming-scripting/175591-using-sed-extract-word-string.html

Update: I have since continued to work this problem, and have come up with another solution involving only awk. The reason was that I wanted to use my command line with a recursive find such that I could run this command line on each file found, depositing the results for each found file back into a new file in the same directory as each found file with the same name as the found file but with a new suffix. I could either use xargs or use the -exec feature of find to run the command line on the results. What several hours of trying taught me was that piping things between sed and awk wasn't possible to do under either find -exe or xargs. Eventually, I found a way to do all the line manipulation in awk with a single command line without piping and then use it with find -exec which has a much nicer argument expansion capability than xargs. Here's the final result:

find . -name "KEYWORD*.log" -execdir awk '/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"{}.ALARMS";getline;print $NF>>"{}.ALARMS"}' {} \;

The key features here are: 1) regex for matching just like sed, 2) the use of printf to print the result from the first line without a carriage return so that the next thing printed (from the next line) would be printed with the result from the first line, 3) the use of getline to just freaking get the next line, and do a print tailored just to that line, 4) the use of >> for each print statement for the filename (and the fliename *has* to be in quotes, but hahaha find -exec graciously expands the {} to form the filename with the new suffix right in the awk script, as well as sending the filename to awk as the final argument). 5) -execdir instead of -exec which causes the command to be run in the directory where the file was found (and thus the result to be deposited in the same directory)

Here's an example of methodology that I didn't end up going with but which is still cool: It shows saving the current line in awk in a variable, using $0 to get the entire line, and then something that I don't quite get going on with "next" that seems like it should be similar to getline but doesn't seem to be, all to join two lines together. Also hilariously makes fun of using cat when awk accepts an input filename as an argument (Useless Use Of Cat):

http://compgroups.net/comp.unix.shell/need-one-line-awk-or-sed-command-to-condition/1210370

Here's another page with more examples of using "next". Some of these examples are really deep and could use much more careful study:

http://www.theunixschool.com/2012/05/awk-join-or-merge-lines-on-finding.html

Here is the example of using getline. What's neat about this forum thread is somebody suggested the other side of the coin, how to do it in sed! I don't understand the sed syntax suggested, so there's probably something important to learn there:

http://www.linuxquestions.org/questions/programming-9/awk-help-print-matching-line-and-next-three-931966/

Here is the example site that I read for doing output to a file in awk. I'd seen a hint of this on another page where they used ">>":

http://www.unix.com/shell-programming-scripting/55172-awk-print-redirection-variable-file-name.html

Here is a nice sed tutorial that talks about those curly brackets (although it doesn't show them all one line like my command). This was a part of my search for trying to get sed to maybe send its output to a file that I could later maybe read in with awk (before I got smart and just decided to do everything in awk):

http://www.ibm.com/developerworks/linux/library/l-sed2/index.html

And, as part of that search, I ran across this deep page, which aside from all the many other wisardly tips clued me in to the secret that you can have multiple -e arguments to sed, and it just treats the result like one sed script:

http://stackoverflow.com/questions/3472404/how-to-go-from-a-multiple-line-sed-command-in-command-line-to-single-line-in-scr

So, the problem that I was having was having an output filename that I could make using filename expansion based on the original found file. I never got that working in sed (and it was essentially a dead end anyhow) because it wouldn't expand the filename. Here is a site about outputting to a file in sed; it's examples are pretty simple though:

http://www.thegeekstuff.com/2009/10/unix-sed-tutorial-how-to-write-to-a-file-using-sed/

I also spent a long time searching for a way to either call sed from awk or awk from sed. While it seems to be possible, all of the example sites that I found were actually pretty confused and incomplete.

And while searching for solutions to my problem of trying to get sed-piped-to-awk into my find command, I ran across the following two breathtakingly learned (but basically unhelpful) forum threads:

http://www.linuxquestions.org/questions/linux-software-2/using-a-pipe-inside-find-exec-how-does-it-work-and-why-is-it-so-slow-816525/

http://ubuntuforums.org/showthread.php?t=1436654

The Conclusion: After coming up with that bitchin find command and thoroughly testing it on my own system using Cygwin, it bombed when I ran it on the server. That one may be running BSD or something (although I'm not sure since a man for awk on that system returns the page for gawk). The failure that I got was a syntax error on the expanded result. For some reason, even though I fed the following into my -execdir in find,:

awk '/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"{}.ALARMS";getline;print $NF>>"{}.ALARMS"}' {}

The syntax error that I got back would show an extra ./ prepended to the expanded command line being fed to awk:

awk: syntax error in: './/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"input_file.ALARMS";getline;print $NF>>"input_file.ALARMS"}'

The expanded filenames were always correct, and I tried escaping a bunch of stuff to no avail. In the end, getting rid of the expansions for the output file and just using a generic hard-coded output file name mysteriously fixed the problem, and worked out better in the end. So, in the end, I had to go with this:

find . -name "KEYWORD*.log" -execdir awk '/PREFIX/{printf $2"\t"$3"\t"$8"\t">>"ALARMS.txt";getline;print $NF>>"ALARMS.txt"}' {} \;

Oh, and while struggling to figure out this new problem, I came across this lovely site that compares and contrasts using find -exe and find | xargs. I can probably go back and read this top to bottom to better educate myself:

http://www.softpanorama.info/Tools/Find/using_exec_option_and_xargs_in_find.shtml

And here's an adorable little xargs page:

http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html

Recent nice sed commands

Here are two nice sed calls that I did today:

To delete 6 lines out of every file that has a particular line in it:

grep -n "The number of passes was" | xargs -L1 -t sed -i -e '/The number of passes was/,+6d'

To delete all lines containing a match out of every file:

grep -n "$start_" | xargs -L1 -t sed -i -e '/\$start_/,+1d'

the -t in xargs is just to echo which files were altered to the screen, the -i in sed is modify in place (very handy), the +1 before the d in the second command was probably unneccesary.

Stop Cygwin sed adding ^M

So, when I used sed on my files, it would convert windows ascii text to linux ascii text and changing the CRLF on the end of each line of every file it touched to just ^M, regardless of if it made the requested other substitution. This royally screwed up Tortise SVN's ability to tell if the file had been seriously changed. My search for the answer led me to a lot of interesting discoveries about how cygwin deals with file types, ways to finesse how it deals with file types (most of which didn't work), and then the answer itself so simple.

On the cygwin message boards, all the moderators could manage was snarky comments saying "read the FAQ about text mounts":

http://www.cygwin.com/ml/cygwin/2002-01/msg01432.html

Here's some not entirely clear additional info that this is happening during the i/o redirection process and not in sed:

http://www.cygwin.com/ml/cygwin/2002-01/msg01433.html

The basic concept from the above was that the problem I'm having is the standard behavior for cygwin sed; it's supposed to do this. So, what are text mounts? Never really got the feeling I learned for sure, but it seems to have something to do with how the file system was mounted by Cygwin initially? Here may be the faq on text modes:

http://www.cygwin.com/cygwin-ug-net/using-textbinary.html

Somewhere I got the idea that cygwin sed could be forced to output files as windows files if the input filename is given with a colon or backslash in the path name. However, after beating on xargs to make it do that, the output was still not a windows format (backing up ideas from the second link above).

During this process I got more familiar with using xargs with the -I option from this page:

http://www.mkssoftware.com/docs/man1/xargs.1.asp

From this next link I learned two things. First, that the -i argument to sed was the "edit in place" parameter, this was what made it possible to not need to manually command the redirect of sed's output. Secondly, sed has a -b argument to specifiy the file type as binary. The Cygwin man page didn't mention this, as I recall, but it was the answer. This has the effect of telling sed to not convert the line endings, and viola, now the lines in files which sed does not change do not show up as changed anymore:

http://www.gnu.org/software/sed/manual/sed.html

sed insert lines after a specific line

Ha, that wasn't the google search that I typed in, but better.

The approaches to this problem involve either insert or append:

http://www.thegeekstuff.com/2009/11/unix-sed-tutorial-append-insert-replace-and-count-file-lines/

or my personal favorites, "next" and "print". In the following example, the found line is deleted, and only specific lines after that are printed (I don't need to delete the found line):

http://prefetch.net/blog/index.php/2007/12/03/printing-a-set-of-lines-after-a-pattern-match/

A really impenetrable suggestion for the same thing: What is capital N?

http://codesnippets.joyent.com/posts/show/2043

So anyhow, what I am going to whip up will be to find the matching line, print the next couple of lines, then do some appends to write out the new code lines.

sed substitution case insensitive

This obviously gets googled often as it comes up as one of the autocompletes.

The answers that are returned are:
1) Use perl
2) Newer sed supports this option (i or I) but not older.

Links, which both have Use perl & update sed suggestions:

http://hintsforums.macworld.com/archive/index.php/t-19780.html

http://stackoverflow.com/questions/4412945/case-insensitive-search-replace-with-sed

How to stop cygwin sed from changing windows CRLF to CR

This question got asked a lot on the net, largely in cygwin forums, and the answers were usually dismissive and snarky.

This is essentially the defined behavior for sed.

There are "text mount" settings in cygwin to change the default definitions of various file types. This is apparently defined when mounting a disk, and is way overkill.

There is a trick with sed where if you feed it a windows-like file, it will not do the default behavior. It's supposed to detect either a colon or a backwards slash in the file name to do this. I found the effectiveness of this to be non-straightforward.

I experimented for a while with ways to get find or grep to return a windows-like filename. There are ways.

However, it turns out that all of that was a big detour; cygwin sed has a -b option in which the file is considered "binary" and it stops mashing the CRLF and just writes the line as is.

It was about at this point that I finally began to understand the -i option, in which files are modified "in place".

Using sed to delete lines

This question has already been blogged about plenty. Then answer is something like:

sed '/awk/d' filename.txt

It's just the d command. The blog that this came from is rich with good examples:

http://en.kioskea.net/faq/1451-sed-delete-one-or-more-lines-from-a-file

This other blog recommends using -e with sed in cases like this even when you don't think you need to, because there are many other cases where the quoted command contains "metacharacters". In any case, this blog is full of genius suggestions in general:

http://www.ceri.memphis.edu/computer/docs/unix/sed.htm

More about sed search and replace

Apparently with sed, it doesn't need the % in front of s as with vi to search and replace throughout the whole file.

Here is a very nice link with some hints, in particular:

$ find /home/bruno/old-friends -type f -exec sed -i 's/ugly/beautiful/g' {} \;

http://www.brunolinux.com/02-The_Terminal/Find_and%20Replace_with_Sed.html

Batch text replace (using grep and sed to replace all occurances of a string in all files recursively through directories)

Easy enough, just grep for the string, and pass the returned file list into sed using xargs. But what about the details?

This site was my best hit on google using the obvious search:

http://stackoverflow.com/questions/1169927/using-sed-and-grep-to-search-and-replace

I used a slighly different set of arguments when I did it, since egrep seems like a royal pain for my purposes. I added a -t to my xargs call so I could monitor what it did each time it did it:

grep -l oldstring *.* | xargs -l -t sed -i -e 's/oldstring/NEWSTRING/g'

Here's a version using find that worked for me:

$ find . -name "*.htm" | xargs -l -t sed -i -e 's/oldstring/NEWSTRING/g'

Here's a novel way to just search for the files that have the string. Might be interesting to find a way to link this up with xargs for the substitution. Not sure what the extra backslash is about:

find . -exec grep -l “string to find” {} \;