Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

SciPy NumPy and SciKit-learn , create a sparse matrix

I'm currently trying to classify text. My dataset is too big and as suggested here, I need to use a sparse matrix. My question is now, what is the right way to add an element to a sparse matrix? Let's say for example I have a matrix X which is my input .

X = np.random.randint(2, size=(6, 100))

Now this matrix X looks like an ndarray of an ndarray (or something like that).

If I do

X2 = csr_matrix(X)

I have the sparse matrix, but how can I add another element to the sparce matrix ? for example this dense element: [1,0,0,0,1,1,1,0,...,0,1,0] to a sparse vector, how do I add it to the sparse input matrix ?

(btw, I'm very new at python, scipy,numpy,scikit ... everything)

like image 816
Olivier_s_j Avatar asked Dec 06 '12 11:12

Olivier_s_j


People also ask

Does Sklearn work with sparse matrices?

Sklearn has many algorithms that accept sparse matrices. The way to know is by checking the fit attribute in the documentation. Look for this: X: {array-like, sparse matrix}.

How do you create an empty sparse matrix in python?

to construct an empty matrix with shape (M, N) dtype is optional, defaulting to dtype='d'. csr_matrix((data, (row_ind, col_ind)), [shape=(M, N)])


1 Answers

Scikit-learn has a great documentation, with great tutorials that you really should read before trying to invent it yourself. This one is the first one to read it explains how to classify text, step-by-step, and this one is a detailed example on text classification using sparse representation.

Pay extra attention to the parts where they talk about sparse representations, in this section. In general, if you want to use svm with linear kernel and you large amount of data, LinearSVC (which is based on Liblinear) is better.

Regarding your question - I'm sure there are many ways to concatenate two sparse matrices (btw this is what you should look for in google for other ways of doing it), here is one, but you'll have to convert from csr_matrix to coo_matrix which is anther type of sparse matrix: Is there an efficient way of concatenating scipy.sparse matrices?.

EDIT: When concatenating two matrices (or a matrix and an array which is a 1 dimenesional matrix) the general idea is to concatenate X1.data and X2.data and manipulate their indices and indptrs (or row and col in case of coo_matrix) to point to the correct places. Some sparse representations are better for specific operations and more complex for other operations, you should read about csr_matrix and see if this is the best representation. But I really urge you to start from those tutorials I posted above.

like image 143
zenpoy Avatar answered Nov 07 '22 12:11

zenpoy