Use multiple output stream in python?

Question

What I am going to do is to create multiple output steams in a python function, and refer them as 1, 2, 3.....: In test.py:

def main():
  ...
  print >>fd1, 'words1'
  print >>fd2, 'words2'
  print >>fd3, 'words3'
  ...

Redirect it while using:

python test.py 1>1.txt 2>2.txt 3>3.txt

The content of these files:

1.txt ->  words1
2.txt ->  words2
3.txt ->  words3

The question is, how to create those fd1, fd2, fd3?

---

Added:

I have used this:

outfiles = {}  
for _ in range(3): 
    fd = os.dup(1)
    outfiles[fd] = os.fdopen(fd, 'w')

def main():
  for no in outfiles: 
    print >>outfiles[no], "foo"
    print >>outfiles[no], outfiles[no].fileno()

But the results depends on how I execute this code:

eg1:

python test.py

foo
3
foo
4
foo
5

eg2:

python test.py 3>log.txt


foo
4
foo
5
foo
6

eg3:

python test.py 1>log.txt

Nothing printed

So I guess, the output is actually 1, and if a file descriptor have already been used in execute (eg:python test.py 3>log.txt), the os.dup(1) won't return it anymore.

Answer 1

On Linux, the file handles that you want exist in /proc/self/fd/. For example:

with open('/proc/self/fd/1', 'w') as fd1, open('/proc/self/fd/2', 'w') as fd2, open('/proc/self/fd/3', 'w') as fd3:
   print >>fd1, 'words1'
   print >>fd2, 'words2'
   print >>fd3, 'words3'

On some other unices, you may find similar file handles under /dev/fd.

Now, you can run your command and verify that the output files are as desired:

$ python test.py 1>1.txt 2>2.txt 3>3.txt
$ cat 1.txt
words1
$ cat 2.txt
words2
$ cat 3.txt
words3

Limitations on number of open file descriptors

The OS places limits on the maximum number of open file descriptors that a process may have. For a discussion of this see "Limits on the number of file descriptors".

When using bash's numbered file descriptors, the restrictions are much tighter. Under bash, only file descriptors up to 9 are reserved for the user. The use of higher numbers may cause conflict with bash's internal use. From man bash:

Redirections using file descriptors greater than 9 should be used with care, as they may conflict with file descriptors the shell uses internally.

If, as per the comments, you want to assign hundreds of file descriptors, then don't use shell redirection or the numbered descriptors in /proc/self/fd. Instead, use python's open command, e.g. open('255.txt', 'w') directly on each output file that you want.

Answer 2

You need module os for this. First, duplicate the standard output (as many times as you need) and build a high-level file object for it. Typically, the first copy of stdout is the file descriptor #3.

outfiles = {} # a future dictionary of output file objects
for _ in range(N): # How many streams do you want?
    new_stdout = os.dup(1)
    outfiles[new_stdout] = os.fdopen(new_stdout, mode='w')

Now, you can use the new file object for printing:

print("foo", file=outfiles[3]) # Sames as print >>outfiles[3], "foo" in 2.7

> python myfile.py 3>3.txt
# There is foo in 3.txt now

It is a very bad idea to redefine file descriptors 0, 1, and 2. Please don't.