Drag and drop large virtual files from C# to Windows Explorer

Question

I have a C# WPF application with a section that serves as an FTP client, listing files on a remote server and allowing the user to download them. I want the user to be able to drag and drop files from the file listing onto their own machine (i.e. into a Windows Explorer shell).

To accomplish this, I used the VirtualFileDataObject code from Delay's blog, using the Action<Stream> overload of SetData. This works great on smaller files.

My problem is: some of the files I'm dealing with are very large (2+ GB), and the way the VirtualFileDataObject class handles the stream involves reading the entire thing into memory, which can end up throwing a "not enough storage" error for those very large files.

The relevant section of the VirtualFileDataObject code is below. How can I rewrite this code to not require the entire stream to be in memory?

    public void SetData(short dataFormat, int index, Action<Stream> streamData) {
        _dataObjects.Add(
            new DataObject {
                FORMATETC = new FORMATETC {
                    cfFormat = dataFormat,
                    ptd = IntPtr.Zero,
                    dwAspect = DVASPECT.DVASPECT_CONTENT,
                    lindex = index,
                    tymed = TYMED.TYMED_ISTREAM
                },
                GetData = () => {
                    // Create IStream for data
                    var ptr = IntPtr.Zero;
                    var iStream = NativeMethods.CreateStreamOnHGlobal(IntPtr.Zero, true);
                    if (streamData != null) {
                        // Wrap in a .NET-friendly Stream and call provided code to fill it
                        using (var stream = new IStreamWrapper(iStream)) {
                            streamData(stream);
                        }
                    }
                    // Return an IntPtr for the IStream
                    ptr = Marshal.GetComInterfaceForObject(iStream, typeof(IStream));
                    Marshal.ReleaseComObject(iStream);
                    return new Tuple<IntPtr, int>(ptr, NativeMethods.S_OK);
                },
            });
    }

In particular, this section of GetData is the culprit:

// Wrap in a .NET-friendly Stream and call provided code to fill it
using (var stream = new IStreamWrapper(iStream)) {
    streamData(stream);
}

streamData is the Action<stream> I provide which writes the actual file data to the stream. My delegate is just opening a file and reading the bytes into the provided stream.

Is there a way to avoid this last step, perhaps somehow passing the file stream directly to be read from by the Explorer shell? I'm thinking something like replacing iStream with a pointer to the .NET filestream I've got...but I don't know enough about COM interop to even know the syntax for doing that. Any tips/direction would be appreciated!

Answer 1

After more Googling and stumbling around and trying one thing and another, I've got something that works, but I'm still open to better solutions. For now, when the drop operation happens I'm retrieving the file to a temporary location then using SHCreateStreamOnFileEx to open an IStream to that location. The revised part, the GetData lambda, is as follows:

GetData = () => {
    var filename = getFilename();

    IStream stream = null;
    NativeMethods.SHCreateStreamOnFileEx(filename, NativeMethods.STGM_FAILIFTHERE, NativeMethods.FILE_ATTRIBUTE_NORMAL, false, null, ref stream);
    var ptr = Marshal.GetComInterfaceForObject(stream, typeof(IStream));
    Marshal.ReleaseComObject(stream);
    return new Tuple<IntPtr, int>(ptr, NativeMethods.S_OK);
}

As I say, I don't know whether this is the best way to do it or if I could manage it more cleanly, but this seems to work.

Answer 2

I got the same issue, easy to fix though ;)

The problem is that we are creating a new memory stream whereas it is not needed since we already have ours. You can create in c# an Stream wrapper that implements IStream:

    /// <summary>
/// Simple class that exposes a read-only Stream as a IStream.
/// </summary>
private class StreamWrapper : IStream
{

   private Stream _stream;

   public StreamWrapper(Stream stream)

   {
       _stream = stream;
   }

   public void Read(byte[] pv, int cb, System.IntPtr pcbRead)

   {
       Marshal.WriteInt32(pcbRead, _stream.Read(pv, 0, cb));
   }

   public void Seek(long dlibMove, int dwOrigin, System.IntPtr plibNewPosition)

   {
       Marshal.WriteInt32(plibNewPosition, (int)_stream.Seek(dlibMove, (SeekOrigin)dwOrigin));
   }

   public void Clone(out IStream ppstm)

   {
       throw new NotImplementedException();
   }

   public void Commit(int grfCommitFlags)

   {
       throw new NotImplementedException();
   }

   public void CopyTo(IStream pstm, long cb, IntPtr pcbRead, IntPtr pcbWritten)

   {
       throw new NotImplementedException();
   }

   public void LockRegion(long libOffset, long cb, int dwLockType)

   {
       throw new NotImplementedException();
   }

   public void Revert()

   {
       throw new NotImplementedException();
   }

   public void SetSize(long libNewSize)

   {
       throw new NotImplementedException();
   }

   public void Stat(out System.Runtime.InteropServices.ComTypes.STATSTG pstatstg, int grfStatFlag)

   {
       throw new NotImplementedException();
   }

   public void UnlockRegion(long libOffset, long cb, int dwLockType)

   {
       throw new NotImplementedException();
   }

   public void Write(byte[] pv, int cb, IntPtr pcbWritten)

   {
       throw new NotImplementedException();
   }
}

And then in the VirtualFileDataObject class, change the signature of the SetData method so that you pass now a Stream:

public void SetData(short dataFormat, int index, Stream stream)
{
  ...
  var iStream = new StreamWrapper(stream);
  ...
  // Ensure the following line is commented out:
  //Marshal.ReleaseComObject(iStream);
  return new Tuple<IntPtr, int>(ptr, NativeMethods.S_OK);
 ...
}

Now, no new memory stream will be created.

For further info, go to http://blogs.msdn.com/b/delay/archive/2009/11/04/creating-something-from-nothing-asynchronously-developer-friendly-virtual-file-implementation-for-net-improved.aspx#10496772 and read my comments