how to proper read data vertically from horizontal array?

Question

here's the declaration of the infrastructure i have from a SDK:

struct alignas(32) Input {
    union {
        float values[16] = {};
        float value;
    };
    
    // other members variables
}

std::vector<Input> myInputs;

const int numInputsA = 4;
const int numInputsB = 4;
const int numInputsC = 4;
const int numInputsD = 4;
const int numInputsE = 4;
myInputs.resize(numInputsA + numInputsB + numInputsC + numInputsD + numInputsE);

what's the best way to load records faster with simd such as:

__m128 targetA0 = { myInputs[0].values[0], myInputs[1].values[0], myInputs[2].values[0], myInputs[3].values[0] }
__m128 targetB0 = { myInputs[4 + 0].values[0], myInputs[4 + 1].values[0], myInputs[4 + 2].values[0], myInputs[4 + 3].values[0] }
__m128 targetC0 = { myInputs[8 + 0].values[0], myInputs[8 + 1].values[0], myInputs[8 + 2].values[0], myInputs[8 + 3].values[0] }
...
__m128 targetA1 = { myInputs[0].values[1], myInputs[1].values[1], myInputs[2].values[1], myInputs[3].values[1] }
__m128 targetB1 = { myInputs[4 + 0].values[1], myInputs[4 + 1].values[1], myInputs[4 + 2].values[1], myInputs[4 + 3].values[1] }
__m128 targetC1 = { myInputs[8 + 0].values[1], myInputs[8 + 1].values[1], myInputs[8 + 2].values[1], myInputs[8 + 3].values[1] }
...
... and so on

as you can see, the struct i've inherit is not really oriented to catch data this way, but can't change it.

so the question, thanks to your experience: is it possible to load data to register with "offset" on each starting index? or the cacheline need anyway to load the whole block, involing lots of cache miss?

maybe there's some tricks to speed up the whole thing. as for my previous post, still on a windows/64 bit machine, using FLAGS += -O3 -march=nocona -funsafe-math-optimizations (imposed by the ecosystem where i'm developing into).

thanks for any helps/tips/suggestions you can give to me.

Answer 1

The only marginal improvement might be to change alignas to 64, since you have 64 bytes, to hopefully make it align into a single cache line.

64 bytes happens to be the size of a cache line these days. So, assuming you need to fetch the data from RAM, your simd setup will hardly matter. The expensive part will be getting the data to L1 cache, the rest are of the operations will be noise. Even, if you need two cache lines because of alignment, I expect the increase to be very small. Keep in mind that today's processors don't execute things sequentially. Likely all those assignments run somewhat in parallel, so the catual order doesn't matter that much.

I would suggest getting a fairly simple version of your code (two loops) and look at the assembly code generated. You're running with O3 so even naive code will likely be fairly well (if not better) optimized. If you are serious about optimizing this you should setup a benchmark to verify that what you're doing is actually speeding things up. I would expect the simple version will be fast enough (please post if you're getting better results).

Also you should profile the entire application. It's likely you will find other bits of code that are easier to optimize and give you more benefits.

Can you get faster? Probably, but you start adding significant complexity and limitations to your code. I can imagine a situation where your code works fast on your workstation but just average on other CPUs. Also you will be complicating some non-trivial code. Is that worth it to you?