James Tanner

legacy-wiki

Text processing

January 24, 2011

Recovered from the older tannerjc.net wiki snapshot dated January 23, 2016.

Given vmstat in odd output, convert it to more normal (parsable by gnuplot) format

|** Sno **|** Run_Queue **|** Blocked_Procs **|** Swap(KB) **|** Free(KB) **|** Buff(KB) **|** Cache(KB) **|** Swap_In **|** Swap_Out **|** Blk_In **|** Blk_Out **|** Intr **|** Cont.Switch **|** Usr%_CPU **|** Sys%_CPU **|** WaitIO%_CPU **|** Idle%_CPU **|

|23:59:52 2010-08-15 (#289815)|0|0|6456|57256|147800|12337144|0|0|114|117|1128|3854|1|0|1|98|
|00:00:22 2010-08-16 (#289816)|0|0|6456|81652|150524|12301660|0|0|400|346|1185|5799|1|0|3|96|
|00:00:52 2010-08-16 (#289817)|0|0|6456|76900|150720|12305132|0|0|75|216|1190|4839|2|1|1|96|
|00:01:22 2010-08-16 (#289818)|2|0|6456|76440|150884|12308792|0|0|135|125|1129|4274|1|0|1|98|
|00:01:52 2010-08-16 (#289819)|0|0|6456|71512|151060|12312892|0|0|133|125|1136|4460|1|0|1|98|
|00:02:22 2010-08-16 (#289820)|0|0|6456|69692|151240|12318168|0|0|159|140|1172|4246|1|0|1|98|
|00:02:52 2010-08-16 (#289821)|0|0|6456|68816|151408|12320304|0|0|95|116|1115|3663|1|0|0|99|
|00:03:22 2010-08-16 (#289822)|0|0|6456|60616|151584|12327176|0|0|153|186|1257|5545|2|0|1|98|
|00:03:52 2010-08-16 (#289823)|0|0|6456|57160|151720|12329316|0|0|64|170|1240|4897|2|0|1|98|
|00:04:22 2010-08-16 (#289824)|2|0|6456|57424|151876|12325796|0|0|122|228|1286|5900|1|0|1|98|
|00:04:52 2010-08-16 (#289825)|0|0|6456|57484|152008|12327028|0|0|38|118|1127|3738|1|0|0|99|
|00:05:22 2010-08-16 (#289826)|0|0|6456|54976|152164|12330124|0|0|89|105|1112|4429|1|0|1|98|
|00:05:52 2010-08-16 (#289827)|0|0|6456|52720|152372|12326144|0|0|63|161|1163|5042|2|1|1|96|
|00:06:22 2010-08-16 (#289828)|1|0|6456|57120|152516|12331592|0|0|129|163|1195|4680|1|0|1|98|
snip more data/days.. through 2010-08-27

(before beginning manually remove the header line and empty lines)

Remove the beginning |

in vim execute:

:%s/^|//g
34531 substitutions on 34531 lines

file now looks like:

23:59:52 2010-08-15 (#289815)|0|0|6456|57256|147800|12337144|0|0|114|117|1128|3854|1|0|1|98|
00:00:22 2010-08-16 (#289816)|0|0|6456|81652|150524|12301660|0|0|400|346|1185|5799|1|0|3|96|
00:00:52 2010-08-16 (#289817)|0|0|6456|76900|150720|12305132|0|0|75|216|1190|4839|2|1|1|96|

Remove (#324370)

in vim execute:

:%s/(.*)//g
34531 substitutions on 34531 lines

file now looks like:

23:59:52 2010-08-15 |0|0|6456|57256|147800|12337144|0|0|114|117|1128|3854|1|0|1|98|
00:00:22 2010-08-16 |0|0|6456|81652|150524|12301660|0|0|400|346|1185|5799|1|0|3|96|
00:00:52 2010-08-16 |0|0|6456|76900|150720|12305132|0|0|75|216|1190|4839|2|1|1|96|

Replace | separator with space separator

in vim execute:

:%s/|/\ /g
587010 substitutions on 34530 lines

file now looks like:

23:59:52 2010-08-15  0 0 6456 57256 147800 12337144 0 0 114 117 1128 3854 1 0 1 98
00:00:22 2010-08-16  0 0 6456 81652 150524 12301660 0 0 400 346 1185 5799 1 0 3 96
00:00:52 2010-08-16  0 0 6456 76900 150720 12305132 0 0 75 216 1190 4839 2 1 1 96

break single file up into multiple files per day

for i in $(seq 16 27); do fgrep 2010-08-$i eprerb01-vmstat-Aug15-Aug28.txt  vmstat-2010-08-$i.txt; done;

[jbrier@jb August-29]$ head -n 3 vmstat-2010-08-16.txt
00:00:22 2010-08-16  0 0 6456 81652 150524 12301660 0 0 400 346 1185 5799 1 0 3 96
00:00:52 2010-08-16  0 0 6456 76900 150720 12305132 0 0 75 216 1190 4839 2 1 1 96
00:01:22 2010-08-16  2 0 6456 76440 150884 12308792 0 0 135 125 1129 4274 1 0 1 98

remove date from each individual file

for i in $(seq 16 27); do sed s/2010-08-$i//  vmstat-2010-08-$i.txt  vmstat-2010-08-$i-sed.txt; done;

[jbrier@jb August-29]$ head -n 3 vmstat-2010-08-16-sed.txt
00:00:22   0 0 6456 81652 150524 12301660 0 0 400 346 1185 5799 1 0 3 96
00:00:52   0 0 6456 76900 150720 12305132 0 0 75 216 1190 4839 2 1 1 96
00:01:22   2 0 6456 76440 150884 12308792 0 0 135 125 1129 4274 1 0 1 98

Now that we have cleaned up the data we can make some graphs with Gnuplot

remove HTML markup tags on either side of content you want

  • Issue**

How to convert this:

 tr class=oddtd class=active2008/10/28/europe-october-2008/tdtdnode/246/tdtda href=/admin/build/path/edit/117?destination=admin%2Fbuild%2Fpathedit/a/tdtda href=/admin/build/path/delete/117?destination=admin%2Fbuild%2Fpathdelete/a/td /tr

 tr class=eventd class=active2008/10/30/london-saturday-october-11th/tdtdnode/247/tdtda href=/admin/build/path/edit/118?destination=admin%2Fbuild%2Fpathedit/a/tdtda href=/admin/build/path/delete/118?destination=admin%2Fbuild%2Fpathdelete/a/td /tr

to this:

2008/10/28/europe-october-2008

2008/10/30/london-saturday-october-11th
  • Resolution**
[john@farina tmp]$ cat raw.txt | cut -d\ -f3 | cut -d\ -f1  redirect2.txt
  • Create Apache HTTPD conf file**
cat redirect2.txt | awk '{print     Redirect  index.php/ $1  / $i}'  redirect3.txt