One of the first research programs I worked on as an undergraduate was taking large amounts of data from an HPGC running a fixed-bed reactor. There were many problems with the work: (1) since our filter materials were very good at filtering the chemicals used, the run-times to breakthrough were very long (which meant that I had to come into the lab every 4 hours and restart the machine), (2) even with the lowest data setting, we had gobs and gobs of data (gigabytes in a day we were used to kilobytes of data), and (3) the data was saved in the form of: A 1 B 2 C3 (return) D4 E5 F6 (return). To the rescue was a friend of mine, Nicolas, who was an excellent coder. He turned me onto AWK and it worked beautifully. The code he came up with is more complex than the following, and I’ve misplaced the old code, so for now, let’s deal with a simple case…

Let’s say you have too much data (you set the machine wrong, or you can’t set the machine properly), and you don’t care about throwing away the data, as the trends you want to see aren’t on the order of the data you wish to pitch. If you have thousands of data-points and you only need say every 5th line, or every other line- do you really put it in Excel and edit it manually? If so, you shouldn’t get a pay-raise next quarter- you can save crap tons of time by using the right tools.

Enter Awk – a great command-line program available on almost every linux/unix computer (maybe yours is called Gawk). [A huge book on how to use all the aspects of awk is available here: The Gnu Awk User's Guide.]

With a single one-liner of code in the terminal, you can accomplish your goal of reducing your data. My pal Brandon wanted to keep only every 5th line from his data:

cat "$@" | awk -F, '{if (count++%5==0) print $0;}' >~/Desktop/AwkOutput5thLines.txt

This is the code I used within Automator to accomplish his data-throw-away needs. (there was an error with line-endings I fix later on down in this post…) cat “$@” uses the output of the choose-file automator task (as argument) to feed the file to the awk command (in -F mode for piping). The count++ command is doing the dirty work. Hat-tip to Duane’s Brain blog for the awk portion of the code which does the dirty work here! Details of how to pass strings as arguments came from this great post on MacOsXHints.com.

Another frequent problem is throwing away every other line. Here’s the code to pitch every 2nd line:
cat "$@" | awk -F, 'NR % 2 == 1' > ~/Desktop/AwkOutput.txt

One of the errors my pal had with his data was the line-endings. So, note that awk requires linefeeds (unix format text) in order to accomplish it’s goals. A simple way to translate is to use tr (translate text) the line-endings into things unix understands.

So, I solved the text line-ending problems by adding the following code:
tr '\r' '\n'
making the final code appear like this (within automator) (all in one line):
cat "$@" | tr '\r' '\n' | awk -F, '{if (count++%5==0) print $0;}' >~/Desktop/AwkOutput5thLines.txt

Some more great awk links you may find useful:

• Awk Examples
• Famous Awk One Liners Explained

Finally, to give you a few presents for dropping in to read this meager blog, here’s a finished Automator script as well as an application form of the script in case you need exactly every 5th line like my pal, Brandon.

• Keep Every 5th Line Of My Data (ZIP file)
• Keep Every 5th Line Of My Data With Encoding (ZIP file)
• Throw Away Every Other Line (ZIP file)

Well, there’s an easy solution.  It’s not the perfect solution, as it’s a bit slow, I’ll get to that in a second, but it’s a great solution to the problem, and has worked for me a number of times now.  To top it off, it’s open-source, donation-ware, and cross-platform: Engauge Digitizer (see post at LifeHacker.com).  Don’t let the website and lack of recent updates deter you.  Tools that can do what Engauge does are few and far between.  So, it is definitely worth a try.  Here’s an example of how I’ve used it just the other day (prompting this post- I’ve used it for years now, but the recent use reminded me I should share it with others).  [click "More" to see an example use and learn more]

So, pretend you’re doing Cryogenic Cathodoluminescence measurements with a Gatan MonoCL system, and you want to know what the efficiency curves are, and in fact, you may want to use them to alter your data to correct for the grating response.  Only problem?  You can’t find the actual data for the grating response with wavelength anywhere… except for this one gif…

Great, so that’s the grating response (this is a Bentham 83, 1.2 micron blaze).  With a bit of work in Engauge Digitizer, and some simple plotting in MATLAB, you can turn that picture into this…

You say big woop- well, true, it’s not that big a deal until you need to use that curve to alter some of your data to correct for the 20% efficiency difference between say 1100 nm and 1600 nm.  [Digitizer can output your data as a comma delimited data file that you can import and use in say, MATLAB.]  Other uses I’ve come across is Data-Mining a micrograph for positional data of certain features/points of interest.  That positional data can then be compared to other micrograph data, or can be used to find nearest-neighbor distances etc.

So, give Engauge a try, and kudos goes to the author for a good program that he’s put out there for free!!  The only negative thing I have to say about it is that sometimes some functions are a bit counter intuitive, and the animating cursor etc., is a bit overboard- a color change for highlighting, or a thickness variation or something like that is a bit more intuitive.  In anycase, beggars can’t be choosers, and Engauge certainly delivers.