pytorch RuntimeError: Expected object of scalar type Double but got scalar type Float

Question

I am trying to implement a custom dataset for my neural network. But got this error when running the forward function. The code is as follows.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
import numpy as np

class ParamData(Dataset):
    def __init__(self,file_name):
        self.data = torch.Tensor(np.loadtxt(file_name,delimiter = ','))    #first place
    def __len__(self):
        return self.data.size()[0]
    def __getitem__(self,i):
        return self.data[i]

class Net(nn.Module):
    def __init__(self,in_size,out_size,layer_size=200):
        super(Net,self).__init__()
        self.layer = nn.Linear(in_size,layer_size)
        self.out_layer = nn.Linear(layer_size,out_size)

    def forward(self,x):
        x = F.relu(self.layer(x))
        x = self.out_layer(x)
        return x

datafile = 'data1.txt'

net = Net(100,1)
dataset = ParamData(datafile)
n_samples = len(dataset)

#dataset = torch.Tensor(dataset,dtype=torch.double)   #second place
#net.float()                                          #thrid place

net.forward(dataset[0])         #fourth place

In the file data1.txt is a csv formatted text file containing certain numbers, and each dataset[i] is a size 100 by 1 torch.Tensor object of dtype torch.float64. The error message is as follows:

Traceback (most recent call last):
  File "Z:\Wrong.py", line 33, in <module>
    net.forward(dataset[0])
  File "Z:\Wrong.py", line 23, in forward
    x = F.relu(self.layer(x))
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\Python38\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "E:\Python38\lib\site-packages\torch\nn\functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm

It seems that I should change the dtype of the numbers in dataset to torch.double. I tried things like

• changing the line at the first place to self.data = torch.tensor(np.loadtxt(file_name,delimiter = ','),dtype=torch.double)

• changing the line at the fourth place to net.forward(dataset[0].double())

• uncommenting one of the two lines at the second or the thrid place

I think these are the solutions I have seen from similar questions, but they either give new errors or don't do anything. What should I do?

---

Update: So I got it working by changing the first place to

self.data = torch.from_numpy(np.loadtxt(file_name,delimiter = ',')).float()

which is weird because it is exactly the opposite of the error message. Is this a bug? I'd still like some explaining.

Answer 1

In short: your data has type double but your model has type float, this is not allowed in pytorch because only data with the same dtype can be fed into the model.

In long: This issue is related to the default dtype of PyTorch and Numpy. I will first explain why this error happens and then suggest some solutions(but I think you will not need my solution once you understand the principle.)

• PyTorch has a couple of dtypes https://pytorch.org/docs/stable/tensors.html. Two of them are closely related to the issue you had:

1. torch.float32(aka torch.float)
2. torch.float64(aka torch.double)

• It's important to know the default dtype of PyTorch Tensors is torch.float32(aka torch.float). This means when you create a tensor, its default dtype is torch.float32.try: torch.ones(1).dtype . This will print torch.float32 in default case. And also the model's parameters are of this dtype by default.

• In your case, net = Net(100,1) will create a model whose dtype of parameters are torch.float32

Then we need to talk about Numpy:

• The default dtype of Numpy ndarray is numpy.float64. This means when you create a numpy array, its default dtype is numpy.float64.try: np.ones(1).dtype . This will print dtype('float64') in default case.

• In your case, your data come from a local file loaded by np.loadtxt, so the data is first loaded as dtype('float64')(as a numpy array) and then converted to a torch tensor of dtype torch.float64(aka torch.double). This is what happens when you convert a numpy array to torch tensor: they will have the corresponding dtype.

I think now the issue is pretty clear, you have a model whose parameters are of torch.float32(aka torch.float) but tries to run it on data of torch.float64(aka torch.double). This is also what the error message tries to say:Expected object of scalar type Double but got scalar type Float for argument

Solutions:

1. You have alreay found one: convert your data to torch.float32 by calling tensor.float()
2. You can also specify the dtype when load the data: np.loadtxt(file_name,delimiter = ',',dtype="float32")

Answer 2

Now that I have more experience with pytorch, I think I can explain the error message. It seems that the line

RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm

is actually refering to the weights of the linear layer when the matrix multiplication is called. Since the input is double while the weights are float, it makes sense for the line

output = input.matmul(weight.t())

to expect the weights to be double.