Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Parsing CSV into Pytorch tensors

I have a CSV files with all numeric values except the header row. When trying to build tensors, I get the following exception:

Traceback (most recent call last):
  File "pytorch.py", line 14, in <module>
    test_tensor = torch.tensor(test)
ValueError: could not determine the shape of object type 'DataFrame'

This is my code:

import torch
import dask.dataframe as dd

device = torch.device("cuda:0")

print("Loading CSV...")
test = dd.read_csv("test.csv", encoding = "UTF-8")
train = dd.read_csv("train.csv", encoding = "UTF-8")

print("Converting to Tensor...")
test_tensor = torch.tensor(test)
train_tensor = torch.tensor(train)

Using pandas instead of Dask for CSV parsing produced the same error. I also tried to specify dtype=torch.float64 inside the call to torch.tensor(data), but got the same error again.

like image 359
hildebro Avatar asked Aug 15 '18 11:08

hildebro


People also ask

How do you make a torch tensor?

To create a tensor with pre-existing data, use torch.tensor() . To create a tensor with specific size, use torch.* tensor creation ops (see Creation Ops). To create a tensor with the same size (and similar types) as another tensor, use torch.*_like tensor creation ops (see Creation Ops).


2 Answers

Try converting it to an array first:

test_tensor = torch.Tensor(test.values)
like image 89
karla Avatar answered Sep 30 '22 20:09

karla


I think you're just missing .values

import torch
import pandas as pd

train = pd.read_csv('train.csv')
train_tensor = torch.tensor(train.values)
like image 24
Arash Avatar answered Sep 30 '22 18:09

Arash