Using tee and grep to save certain lines of output to a file

Question

I want to see the entire output of my script but only save lines matching "vlim" to a new file. I've nearly figured it out but can't quite get it to work the way I want it to. I'm currently stuck between using:

python gmakemovie.test movie.cfg.test --overwrite | tee >(grep vlim) /output.txt
grep vlim output.txt > vlim.txt

or

python gmakemovie.test movie.cfg.test --overwrite | grep vlim | tee  /output.txt

The top option shows my entire output but also copies all of the output to output.txt. The bottom option copies only "vlim" but doesn't show the rest of my output, so I can't tell where I am in my script.

For clarity, this is what my output looks like:

imported pygad v0.4.32+ga1eacb4.dirty
from parameter file (fragmentary):
  snaps:      200 - 200
  width:      1000.0 [kpc]
  pixel:      500 x 500
  x - y:      x - y
  softening:  [  0.2    0.45   2.52  20.     0.2    0.2 ] [ckpc h_0**-1]
====================================================
get orientation from trace file "filepath":
  L:       [ 241.01309204  544.96875    -366.44366455] [1e+10 ckpc h_0**-1      Msol h_0**-1 km s**-1]
get first non-zero center from trace files
  from snapshot 200:
  center:  [ 36821.75390625  35120.65625     33730.06640625] [ckpc h_0**-1]
====================================================
run in serial
====================================================
read snapshot "filepath"
  z = 2.84615386294
  t = 2.33634681236 [Gyr]
get center and orientation from trace file "filepath":
  center:  [ 36821.75390625  35120.65625     33730.06640625] [ckpc h_0**-1]
  R200:    47.4815177362 [kpc]
center snapshot
orientate snapshot
render...
  axis 1: stars...
  im_lum = np.log10(im_lum)
vlim: [-6.59883562 -4.09629962]
  np.ndarray.__idiv__(self, x)
  axis 2: gas...
vlim: [-0.46031536  0.83438775]

and I want the output file to look like:

vlim: [-6.59883562 -4.09629962]
vlim: [-0.46031536  0.83438775]

I'm working in the terminal on my mac

Answer 1

It woud be easiest to just

$ datagenerator | tee outfile-complete | grep 'pattern' >outfile-filtered &
$ less outfile-complete

If it's a long-running script, this will allow you to monitor the progress with less (use F in less to get it to act like tail -f) while the job is running as a background job.

EDIT: Actually, looking closer at your first command:

$ datagenerator | tee >( grep 'pattern' ) output

Only a tiny change is necessary to do what you intended:

$ datagenerator | tee >( grep 'pattern' >outfile-filtered ) output-complete

The grep in the sub-shell was writing to standard output. The change means that the filtered data goes to a file instead, and the complete output goes to the console and to output-complete

So now you have two solutions that does slightly different things.