Use gdal.Open to open an in-memory GDAL Dataset

Question

I have a GDAL dataset that exists only in memory (inmem_ds):

In: gdal.Info(inmem_ds)
Out:
('Driver: NUMPY/Numeric Python Array\n'
 'Files: none associated\n'
 'Size is 500, 500\n'
 'Coordinate System is:\n'
 'PROJCS["WGS 84 / UTM zone 19N",\n'
 '    GEOGCS["WGS 84",\n'
 '        DATUM["WGS_1984",\n'
 '            SPHEROID["WGS 84",6378137,298.257223563,\n'
 '                AUTHORITY["EPSG","7030"]],\n'
 '            AUTHORITY["EPSG","6326"]],\n'
 '        PRIMEM["Greenwich",0,\n'
 '            AUTHORITY["EPSG","8901"]],\n'
 '        UNIT["degree",0.0174532925199433,\n'
 '            AUTHORITY["EPSG","9122"]],\n'
 '        AUTHORITY["EPSG","4326"]],\n'
 '    PROJECTION["Transverse_Mercator"],\n'
 '    PARAMETER["latitude_of_origin",0],\n'
 '    PARAMETER["central_meridian",-69],\n'
 '    PARAMETER["scale_factor",0.9996],\n'
 '    PARAMETER["false_easting",500000],\n'
 '    PARAMETER["false_northing",0],\n'
 '    UNIT["metre",1,\n'
 '        AUTHORITY["EPSG","9001"]],\n'
 '    AXIS["Easting",EAST],\n'
 '    AXIS["Northing",NORTH],\n'
 '    AUTHORITY["EPSG","32619"]]\n'
 'Origin = (617085.000000000000000,-1512585.000000000000000)\n'
 'Pixel Size = (30.000000000000000,-30.000000000000000)\n'
 'Corner Coordinates:\n'
 'Upper Left  (  617085.000,-1512585.000) ( 67d55\' 2.38"W, 13d40\'47.58"S)\n'
 'Lower Left  (  617085.000,-1527585.000) ( 67d55\' 0.13"W, 13d48\'55.77"S)\n'
 'Upper Right (  632085.000,-1512585.000) ( 67d46\'43.12"W, 13d40\'45.26"S)\n'
 'Lower Right (  632085.000,-1527585.000) ( 67d46\'40.59"W, 13d48\'53.42"S)\n'
 'Center      (  624585.000,-1520085.000) ( 67d50\'51.55"W, 13d44\'50.54"S)\n'
 'Band 1 Block=500x1 Type=Int16, ColorInterp=Undefined\n'
 '  NoData Value=-9999\n'
 'Band 2 Block=500x1 Type=Int16, ColorInterp=Undefined\n'
 '  NoData Value=-9999\n'
 'Band 3 Block=500x1 Type=Int16, ColorInterp=Undefined\n'
 '  NoData Value=-9999\n'
 'Band 4 Block=500x1 Type=Int16, ColorInterp=Undefined\n'
 '  NoData Value=-9999\n'
 'Band 5 Block=500x1 Type=Int16, ColorInterp=Undefined\n'
 '  NoData Value=-9999\n'
 'Band 6 Block=500x1 Type=Int16, ColorInterp=Undefined\n'
 '  NoData Value=-9999\n'
 'Band 7 Block=500x1 Type=Int16, ColorInterp=Undefined\n'
 '  NoData Value=-9999\n'
 'Band 8 Block=500x1 Type=Int16, ColorInterp=Undefined\n'
 '  NoData Value=-9999\n')

Now I want to pass this dataset directly to an external Python library that uses gdal.Open() to open its input datasets. So I need a physical file path that I can pass to gdal.Open(). But I don´t want to write the in-memory GDAL dataset to disk in order to save IO time.

Is there any way to pass an in-memory dataset to gdal.Open()?

I tried to use a VRT dataset but it does not work:

drv = gdal.GetDriverByName("VRT")
vrt = drv.CreateCopy("/mypath/test.vrt", inmem_ds)
type(vrt.ReadAsArray()) # numpy.ndarray -> works

gdal.Open('/mypath/test.vrt') # None, last GDAL error: Exception: Error reading file:  Missing <SourceFilename> element in VRTRasterBand.

The saved VRT file (mypath/test.vrt) looks like this:

<VRTDataset rasterXSize="500" rasterYSize="500">
  <SRS>PROJCS["WGS 84 / UTM zone 19N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",-69],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32619"]]</SRS>
  <GeoTransform>  6.1708500000000000e+05,  3.0000000000000000e+01,  0.0000000000000000e+00, -1.5125850000000000e+06,  0.0000000000000000e+00, -3.0000000000000000e+01</GeoTransform>
  <VRTRasterBand dataType="Int16" band="1">
    <NoDataValue>-9999</NoDataValue>
    <SimpleSource>
      <SourceFilename relativeToVRT="0"></SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="500" RasterYSize="500" DataType="Int16" BlockXSize="500" BlockYSize="1" />
      <SrcRect xOff="0" yOff="0" xSize="500" ySize="500" />
      <DstRect xOff="0" yOff="0" xSize="500" ySize="500" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Int16" band="2">
    <NoDataValue>-9999</NoDataValue>
    <SimpleSource>
      <SourceFilename relativeToVRT="0"></SourceFilename>
      <SourceBand>2</SourceBand>
      <SourceProperties RasterXSize="500" RasterYSize="500" DataType="Int16" BlockXSize="500" BlockYSize="1" />
      <SrcRect xOff="0" yOff="0" xSize="500" ySize="500" />
      <DstRect xOff="0" yOff="0" xSize="500" ySize="500" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Int16" band="3">
    <NoDataValue>-9999</NoDataValue>
    <SimpleSource>
      <SourceFilename relativeToVRT="0"></SourceFilename>
      <SourceBand>3</SourceBand>
      <SourceProperties RasterXSize="500" RasterYSize="500" DataType="Int16" BlockXSize="500" BlockYSize="1" />
      <SrcRect xOff="0" yOff="0" xSize="500" ySize="500" />
      <DstRect xOff="0" yOff="0" xSize="500" ySize="500" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Int16" band="4">
    <NoDataValue>-9999</NoDataValue>
    <SimpleSource>
      <SourceFilename relativeToVRT="0"></SourceFilename>
      <SourceBand>4</SourceBand>
      <SourceProperties RasterXSize="500" RasterYSize="500" DataType="Int16" BlockXSize="500" BlockYSize="1" />
      <SrcRect xOff="0" yOff="0" xSize="500" ySize="500" />
      <DstRect xOff="0" yOff="0" xSize="500" ySize="500" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Int16" band="5">
    <NoDataValue>-9999</NoDataValue>
    <SimpleSource>
      <SourceFilename relativeToVRT="0"></SourceFilename>
      <SourceBand>5</SourceBand>
      <SourceProperties RasterXSize="500" RasterYSize="500" DataType="Int16" BlockXSize="500" BlockYSize="1" />
      <SrcRect xOff="0" yOff="0" xSize="500" ySize="500" />
      <DstRect xOff="0" yOff="0" xSize="500" ySize="500" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Int16" band="6">
    <NoDataValue>-9999</NoDataValue>
    <SimpleSource>
      <SourceFilename relativeToVRT="0"></SourceFilename>
      <SourceBand>6</SourceBand>
      <SourceProperties RasterXSize="500" RasterYSize="500" DataType="Int16" BlockXSize="500" BlockYSize="1" />
      <SrcRect xOff="0" yOff="0" xSize="500" ySize="500" />
      <DstRect xOff="0" yOff="0" xSize="500" ySize="500" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Int16" band="7">
    <NoDataValue>-9999</NoDataValue>
    <SimpleSource>
      <SourceFilename relativeToVRT="0"></SourceFilename>
      <SourceBand>7</SourceBand>
      <SourceProperties RasterXSize="500" RasterYSize="500" DataType="Int16" BlockXSize="500" BlockYSize="1" />
      <SrcRect xOff="0" yOff="0" xSize="500" ySize="500" />
      <DstRect xOff="0" yOff="0" xSize="500" ySize="500" />
    </SimpleSource>
  </VRTRasterBand>
  <VRTRasterBand dataType="Int16" band="8">
    <NoDataValue>-9999</NoDataValue>
    <SimpleSource>
      <SourceFilename relativeToVRT="0"></SourceFilename>
      <SourceBand>8</SourceBand>
      <SourceProperties RasterXSize="500" RasterYSize="500" DataType="Int16" BlockXSize="500" BlockYSize="1" />
      <SrcRect xOff="0" yOff="0" xSize="500" ySize="500" />
      <DstRect xOff="0" yOff="0" xSize="500" ySize="500" />
    </SimpleSource>
  </VRTRasterBand>
</VRTDataset>

Answer 1

You can write the VRT to memory by using the vsimem prefix, beware that the data actually gets written to memory, so any format other then VRT can increase the memory significantly, depending on the actual filesize.

Also note that the in-memory file remains, even after closing the dataset handle. So you need to unlink it yourself when your done using it, or it will stay memory.

Therefore keep track of the in-mem locations you are using, so you can unlink them properly. Writing to an existing vsimem location, before its 'unlinked' wont work, and remain the original file. That can be a bit of a gotcha.

To output either a file on-disk, or an already open Dataset to an in-mem VRT:

ds = gdal.Translate('/vsimem/inmem.vrt', some_file_or_dataset, format='VRT')
print(ds.ReadAsArray().shape)
ds = None

Open it again:

ds2 = gdal.Open('/vsimem/inmem.vrt')
print(ds2.ReadAsArray().shape)
ds2 = None

Release the file from memory:

gdal.Unlink('/vsimem/inmem.vrt')

I'm not entirely sure about the scope of vsimem, so if you mean another Python interpreter by 'external', it might not work.

Answer 2

As you have seen, gdal.Open takes a filepath and in-memory datasets don't have filepaths. Data coercion techniques like VRT still require filepaths (<SourceFilename>). From my experience and the research for this question, I don't think there is a way to pass a dataset (in-memory or file-based) into gdal.Open. You might have to write to disk or try to modify the external library...