The most efficient way to read a unformatted file

Question

Now I am data-processing 100,000 files by using Fortran. These data are generated by HPC using MPI I/O. Now I can just figure out the following ways to read the raw, which is not efficient. Is it possible that read every to read ut_yz(:,J,K), at one one time insteading of reading one by one? Thanks The old code is as follows and the efficiency is not so high.

  OPEN(10,FILE=trim(filename)//".dat",FORM='UNFORMATTED',&
           ACCESS='DIRECT', RECL=4, STATUS='OLD')
      !,CONVERT='big_endian'
      COUNT = 1
      DO K=1,nz
         DO J=1,ny
            DO I=1,nxt
              READ(10,REC=COUNT) ut_yz(I,J,K)
              COUNT = COUNT + 1
            ENDDO
         ENDDO
      ENDDO
    CLOSE(10)

The desired one is

 OPEN(10,FILE=trim(filename)//".dat",FORM='UNFORMATTED', RECL=4, STATUS='OLD')
      !,CONVERT='big_endian'
      COUNT = 1
      DO K=1,nz
         DO J=1,ny
              READ(10,REC=COUNT) TEMP(:)
          COUNT = COUNT + 153
          ut_yz(:,J,K)=TEMP(:)
         ENDDO
      ENDDO
    CLOSE(10)

However, it always fails. Can anyone make a comment on this? Thanks.

Answer 1

Direct IO read will read a single record, if I am not mistaken. Thus, in your new code version you need to increase the record length accordingly:

   inquire(iolength=rl) ut_yz(:,1,1)
   open(10, file=trim(filename)//'.dat', form='UNFORMATTED', recl=rl, status='OLD', action='READ')
   count = 1
   do k=1,nz
     do j=1,ny
       read(10, rec=count) ut_yz(:,j,k)
       count = count + 1
     end do
   end do
   close(10)

Of course, in this example you could also read the complete array at once, which should be the fastest option:

   inquire(iolength=rl) ut_yz
   open(10, file=trim(filename)//'.dat', form='UNFORMATTED', recl=rl, status='OLD', action='READ')
   read(10, rec=1) ut_yz
   close(10)