The best performant way to push items into array?

Question

In my website i have many arrays with data. for example: vertices array, colors array, sizes array...

I'm working with big amounts of items. Up to tens of millions.

Before adding the data into the arrays I need to process it. Until now, I did it in the main thread and this made my website freeze for X seconds. It froze because of the processing and because of adding the processed data into the arrays.

Today I 'moved' (did a lot of work) the processing into web workers, but the processed data is being added in the main thread. I managed to save the freezing time of the processing but not of the adding.

The adding is simply done by array.push() or array.splice().

I've read some articles about how array works, and found out when we add item to an array, the array is being fully copied to a new place in the memory with array.length + 1 size and there adding the value. This makes my data pushing slow.

I also read that typed array are much faster. But for this I would need to know the size of the array, which I don't know, and for creating a big typed array with extra counter and managing adding items in the middle(and not the end of the array) would be a lot of code change, which i don't want to do at this time.

So, for my question, I have TypedArray that return from the web worker, and this i need to put into regular array. What is the best performant way to do it? (today i'm running in a loop and pushing one after the other)

EDIT

Example how the website work: the client add count of items, lets say 100000. The items raw data is being collected and send to the worker. The worker is processing all the information and sending back the processed data as typed-array (for using as transferable objects). In the main thread we are adding the processed data to arrays - to the end or in some specific index. 2nd round. the client add another 100000 items. sending to the worker and the result being added to the main thread arrays. 3nd round can be 10 items, 4nd round 10000, 5nd round can remove indices 10-2000, ...

Answer 1

Did some more research using the comments and thought about another direction.

I've tried using typedArray.set method and discovered that it is very very fast.

10 million of items using sets took 0.004 seconds, compares to array.push 0.866 seconds. I separated the the 10 millions into 10 arrays just to make sure the set method is not working faster when starting from index 0.

This way I think I would even implement my insertAtIndex of my own using the TypedArray, which pushing all the items forward and setting the new one\s in the right index.

In addition, I can use TypedArray.subArray to fetch my sub data according to the real amount of data in the array (which is not copying the data) - useful for uploading the data to the buffer (WebGL)

I said I want to work with regular arrays but this performance boost I don't think I would get other wise. And it is not so much work, when I'm wrapping MyNewTypedArray as TypedArray with all the push, splice, own implementation.

Hope this info helped anyone

var maxCount = 10000000;
var a = new Float32Array(maxCount);
var aSimple = [];

var arrays = [];
var div = 10;
var arrayLen = maxCount / div;
for (var arraysIdx = 0; arraysIdx < div; arraysIdx++) {
  var b = new Float32Array(arrayLen);
  for (var i = 0; i < b.length; i++) {
    b[i] = i * (arraysIdx + 1);
  }
  arrays.push(b);
}

var timeBefore = new Date().getTime();

for (var currArrayIdx = 0; currArrayIdx < arrays.length; currArrayIdx++) {
  a.set(arrays[currArrayIdx], currArrayIdx * arrayLen);
}
var timeAfter = new Date().getTime();
good.innerHTML = (timeAfter - timeBefore) / 1000 + " sec.\n";
timeBefore = new Date().getTime();
for (var currArrayIdx = 0; currArrayIdx < arrays.length; currArrayIdx++) {
  for (var i = 0; i < arrayLen; i++) {
    aSimple.push(arrays[currArrayIdx][i]);
  }
}

timeAfter = new Date().getTime();
bad.innerHTML = (timeAfter - timeBefore) / 1000 + " sec.\n";

Using set of TypedArray:
<div id='good' style='background-color:lightGreen'>working...</div>
Using push of Array:
<div id='bad' style='background-color:red'>working...</div>