Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Extremely large weighted average

I am using 64 bit matlab with 32g of RAM (just so you know).

I have a file (vector) of 1.3 million numbers (integers). I want to make another vector of the same length, where each point is a weighted average of the entire first vector, weighted by the inverse distance from that position (actually it's position ^-0.1, not ^-1, but for example purposes). I can't use matlab's 'filter' function, because it can only average things before the current point, right? To explain more clearly, here's an example of 3 elements

data = [ 2 6 9 ]
weights = [ 1 1/2 1/3; 1/2 1 1/2; 1/3 1/2 1 ]
results=data*weights= [ 8 11.5 12.666 ]
i.e.
8 = 2*1 + 6*1/2 + 9*1/3
11.5 = 2*1/2 + 6*1 + 9*1/2
12.666 = 2*1/3 + 6*1/2 + 9*1

So each point in the new vector is the weighted average of the entire first vector, weighting by 1/(distance from that position+1).

I could just remake the weight vector for each point, then calculate the results vector element by element, but this requires 1.3 million iterations of a for loop, each of which contains 1.3million multiplications. I would rather use straight matrix multiplication, multiplying a 1x1.3mil by a 1.3milx1.3mil, which works in theory, but I can't load a matrix that large.

I am then trying to make the matrix using a shell script and index it in matlab so only the relevant column of the matrix is called at a time, but that is also taking a very long time.

I don't have to do this in matlab, so any advice people have about utilizing such large numbers and getting averages would be appreciated. Since I am using a weight of ^-0.1, and not ^-1, it does not drop off that fast - the millionth point is still weighted at 0.25 compared to the original points weighting of 1, so I can't just cut it off as it gets big either.

Hope this was clear enough?

Here is the code for the answer below (so it can be formatted?):

data = load('/Users/mmanary/Documents/test/insertion.txt');
data=data.';
total=length(data);
x=1:total;
datapad=[zeros(1,total) data];
weights = ([(total+1):-1:2 1:total]).^(-.4);
weights = weights/sum(weights);
Fdata = fft(datapad);
Fweights = fft(weights);
Fresults = Fdata .* Fweights;
results = ifft(Fresults);
results = results(1:total);
plot(x,results)
like image 626
Micah Manary Avatar asked Oct 25 '11 01:10

Micah Manary


People also ask

What does a high WACC mean?

A high WACC typically signals higher risk associated with a firm's operations because the company is paying more for the capital that investors have put into the company. In general, as the risk of an investment increases, investors demand an additional return to neutralize the additional risk.

Is it better to have a higher or lower WACC?

It is essential to note that the lower the WACC, the higher the market value of the company – as you can see from the following simple example; when the WACC is 15%, the market value of the company is 667; and when the WACC falls to 10%, the market value of the company increases to 1,000.

How do you interpret the weighted mean in a thesis?

Weighted Mean is an average computed by giving different weights to some of the individual values. If all the weights are equal, then the weighted mean is the same as the arithmetic mean. It represents the average of a given data. The Weighted mean is similar to the arithmetic mean or sample mean.

What should my weighted average be?

To find a weighted average, multiply each number by its weight, then add the results. If the weights don't add up to one, find the sum of all the variables multiplied by their weight, then divide by the sum of the weights.


1 Answers

The only sensible way to do this is with FFT convolution, as underpins the filter function and similar. It is very easy to do manually:

% Simulate some data
n = 10^6;
x = randi(10,1,n);
xpad = [zeros(1,n) x];

% Setup smoothing kernel
k = 1 ./ [(n+1):-1:2 1:n];

% FFT convolution
Fx = fft(xpad);
Fk = fft(k);

Fxk = Fx .* Fk;

xk = ifft(Fxk);
xk = xk(1:n);

Takes less than half a second for n=10^6!

like image 166
John Colby Avatar answered Oct 11 '22 08:10

John Colby