Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

compute Sparce Matrix Standard Deviation python

I generated the following PCA plot with Python which I'm not really familiar with.PCA

However, there is one things I don't like that is I'm not plotting the explained variance by PC1 and PC2. Unfortunately, the tool I used to compute the eigenvectors does not output the explained variance — which I've seen could be obtained from the standard deviation of the PCs' sparse matrix.

In my case this is a 50 × 4009 matrix with 50 PCs and 4009 observations/samples. How can I feasibly calculate its standard deviation to then get the one of PC1 and PC2 to include in the plot?

I've noticed there is nothing straightforward and searching-up a bit I found this:

def vars(a, axis=None):

    # Variance of sparse matrix a var = mean(a**2) - mean(a)**2

    a_squared = a.copy()
    a_squared.data **= 2
    return a_squared.mean(axis) - np.square(a.mean(axis))
def stds(a, axis=None):

    # Standard deviation of sparse matrix a std = sqrt(var(a))

    return np.sqrt(vars(a, axis))

But upon running it on the PCs matrix I get the following:

AttributeError: 'DataFrame' object has no attribute 'data'

MWE — 3 rows, 50 columns

m3_dict = {'PC1': {0: 69.57156317774135, 1: -117.9620345635911, 2: -12.478042368060661},
 'PC2': {0: 51.770675957028, 1: 42.385600131572744, 2: -145.05657241585325},
 'PC3': {0: 25.83574477759894, 1: 7.4109238483809445, 2: 20.563605887821144},
 'PC4': {0: 30.2888722939247, 1: 5.921433408819105, 2: 37.71330715630336},
 'PC5': {0: -46.934377149619074, 1: -2.372550887758882, 2: 2.9528365827375747},
 'PC6': {0: -14.226122320296678, 1: -2.572791063933768, 2: 5.383480088589302},
 'PC7': {0: -23.592719796044523, 1: -4.679119788073276, 2: 3.613121927186616},
 'PC8': {0: -3.4115560980292123, 1: 7.851894441473657, 2: 3.0211583910082576},
 'PC9': {0: -0.5167582584204253,
  1: -0.17719782335462825,
  2: -2.7965021067570968},
 'PC10': {0: 1.784642272835071, 1: -7.908607615243156, 2: -2.0575907887586107},
 'PC11': {0: -6.195265958903007,
  1: -2.2779214009685624,
  2: -2.9061104691595507},
 'PC12': {0: 1.2915812247960767, 1: -3.0554999840992125, 2: 7.79690755716544},
 'PC13': {0: -14.049030598342528,
  1: -3.9364648876070834,
  2: 0.4581483904467603},
 'PC14': {0: 2.08172174779253, 1: 7.858907874573533, 2: -6.705202542227375},
 'PC15': {0: -1.4007445318504597,
  1: -5.620217127444764,
  2: -0.5155119885576762},
 'PC16': {0: 3.7675370863389985,
  1: -3.3936657924964884,
  2: -4.718401236458761},
 'PC17': {0: -10.439569016745324,
  1: -6.739904554408227,
  2: -2.645406111546428},
 'PC18': {0: 3.48519582357068, 1: 10.754678353468202, 2: -4.528377803421983},
 'PC19': {0: 2.944533085673919, 1: -9.555691050878757, 2: -1.8362832005641785},
 'PC20': {0: 3.475131075227604, 1: -1.2013520403535698, 2: -7.92325444096212},
 'PC21': {0: 9.690006608578697, 1: -2.431502561135152, 2: -12.061932253922391},
 'PC22': {0: -6.413371172743958, 1: 0.851350519924718, 2: -1.8812634509060506},
 'PC23': {0: 21.19950401155053, 1: 1.326797564405327, 2: -3.658291497189177},
 'PC24': {0: 0.6793372260006137,
  1: 5.5511973664233665,
  2: -2.7397445889697667},
 'PC25': {0: -2.8948615582157493,
  1: -9.99870477724031,
  2: -10.026813160980401},
 'PC26': {0: -0.3607062018069712, 1: 9.183918926332778, 2: 3.2458030227751506},
 'PC27': {0: -0.47912820411358764,
  1: -0.13851679158717511,
  2: 5.118330021092383},
 'PC28': {0: -11.956046826425984, 1: -4.28277646564643, 2: -4.764463878786569},
 'PC29': {0: -4.218840194190469,
  1: 3.0374770331315104,
  2: -12.458193640633908},
 'PC30': {0: 1.1532093756676263, 1: 4.574218588136119, 2: 1.26949939592581},
 'PC31': {0: -0.7027350025082383,
  1: -2.5659450958184955,
  2: -2.981499332412195},
 'PC32': {0: -0.28725191479703927,
  1: 3.63321466220521,
  2: -10.424739472432096},
 'PC33': {0: -21.239152963747348, 1: -7.615106122854909, 2: 1.299283675531835},
 'PC34': {0: -13.222805186162939,
  1: 3.2484647404134894,
  2: -4.282123595120055},
 'PC35': {0: -2.8877599724634893, 1: 4.824752782330242, 2: 1.5969273809011084},
 'PC36': {0: 1.1906043244856503, 1: 1.2377847827059554, 2: -3.947650095531824},
 'PC37': {0: -1.8604435092651217,
  1: 0.34449974700333447,
  2: 4.032402282264213},
 'PC38': {0: -6.953346294452693, 1: 6.981678283375523, 2: -21.498097850116384},
 'PC39': {0: -8.777362731018963, 1: 6.400346209659698, 2: 4.332411243989231},
 'PC40': {0: 2.243209286342812, 1: -6.573014321176919, 2: -8.687359832469264},
 'PC41': {0: 6.542106732655741, 1: 9.281573470008032, 2: 11.42872349904576},
 'PC42': {0: -14.389667757342263, 1: -3.777675481223019, 2: 9.994961417866548},
 'PC43': {0: 4.148901693395547, 1: 1.6984761532534385, 2: 7.28757198831338},
 'PC44': {0: 1.7371109830545364, 1: -5.482261774979829, 2: 1.166730791108879},
 'PC45': {0: -5.960845604665959, 1: 9.911187167916783, 2: 0.530708301173031},
 'PC46': {0: -7.860999829524104, 1: 2.7442446045444338, 2: 0.4130639756946138},
 'PC47': {0: -7.815214544978687, 1: 5.989300804353903, 2: -12.301411870970153},
 'PC48': {0: 1.9748933097639294, 1: 1.4476351142339074, 2: 6.199715285102192},
 'PC49': {0: -6.654670771972417,
  1: -2.699608345876845,
  2: -11.454061095672829},
 'PC50': {0: -0.7765893262384549,
  1: -0.2931403648275161,
  2: -4.269622622996996}}

EDIT

Workaround duplicating the former two functions

def vars_pc1(a, axis=None):
    """
    Variance of sparse matrix
    a var = mean(a**2) - mean(a)**2
    """
    a_squared = a.copy()
    a_squared.iloc[:, 0] **= 2
    return a_squared.mean(axis) - np.square(a.mean(axis))
def stds_pc1(a, axis=None):
    """
    Standard deviation of sparse matrix
    a std = sqrt(var(a))
    """
    return np.sqrt(vars_pc1(a, axis))
def vars_pc2(a, axis=None):
    """
    Variance of sparse matrix
    a var = mean(a**2) - mean(a)**2
    """
    a_squared = a.copy()
    a_squared.iloc[:, 1] **= 2
    return a_squared.mean(axis) - np.square(a.mean(axis))
def stds_pc2(a, axis=None):
    """
    Standard deviation of sparse matrix
    a std = sqrt(var(a))
    """
    return np.sqrt(vars_pc2(a, axis))
like image 963
Matteo Avatar asked Sep 26 '26 14:09

Matteo


1 Answers

This's a straightforward solution, where a is DataFrame:

stds_pc1 = np.std(list(a['PC1']))
stds_pc2 = np.std(list(a['PC2']))
like image 152
Stas Simonov Avatar answered Sep 29 '26 03:09

Stas Simonov



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!