Passing parameters from shell to awk as an array

Question

I am using a shell script and within that I am using an awk script. I am passing parameters to awk from shell script by using -v option. At some point of time, when the argument size exceeds a certain limit, I was getting the 'Argument list too long error'. This was my previous question but I have found out the root cause for the same. Now my question is:

Variable to be passed from shell to awk using -v option = too large ⟶ Hence getting argument list too long error

My idea is to break the large variable into small chunks and store it in an array and then pass the array into awk instead of passing the single variable into awk.

My question is:

• Is it possible to break the large variable into small array and then pass it back to awk. I know how to modify a variable of shell inside an awk script. But how can I modify the array of shell inside an awk script?

I read that -v option is not advisable and they suggested to pipe the variable values. So if that the case

echo variable | awk '{}' 

So variables would be piped. But I have to pipe an array along with some other variables. Could you please help me?

 CODE DESCRIPTION
 addvariable=""
 export variable
 loop begins
 eval $(awk -v tempvariable="$addvariable" '{tempvariable=tempvariable+"long string"  variable=tempvariable(Here is where the shell variable(variable) is being modified )}')

  In shell
  addvariable=$variable (Taking the new value of shell variable and feeding back to awk in the next iteration)
   loop ends

So the problem is now as the addvariable and variable keeps on increasing, I am get argument too long error .. So what I have to do is to split the tempvariable into small chunks and then store it in variable[1] variable[2] etc and then assign that to addvariable[1], addvariable[2] and the feed addvariable[1],[2] instead of feeding the entire addvariable as a whole.So my question is how to feed that as an array. and how to store the big data inside the awk into variable[1] variable[2]

CODE addshellvariable=""

for i in {0..10}
{
zcat normalfile{i} > FILE A
zcat hugefile{i} > FILE

export shellvariable=""
getdate=grep "XXX" FILE B|sort|Uniq (getdate contains a list of id's)
eval $(awk -v getdata="$getdata" -v addshellvariable="$addshellvariable" BEGIN {tempvariable="";split(addshellvariable,tempshellvariableArray,"*");while(t <= length(tempshellvariable)) {awkarray[tempshellvariableArray[t]];} {for(id in ids) {awkarray[id];} END {for(id in awkarray) {tempvariable=tempvariable"*"id"*"awkarray[id]}  **print "shellvariable"=tempvariable;**}} FILE A)

  addshellvariable=$shellvariable;
}

So as you can see awk is being embedded inside the shell . everytime I need the awkarray content to be feedback into the awk again .. So that I will be able to get the updated ones and that is the reason I am getting the awk array content in the shell variable by printing that, again the shell variable is stored in an another shell variable "addshellvariable" and that is being given to the awk in the next iteration. But the problem is when the shellvariable size increases a certain point then I am getting an Argument too long error . Thus I wanted a solution in such a way that , instead of doing
print "shellvariable"=tempvariable; I can make it as print "shellvariable[1]"=A part of tempvariable; and so on ...

Answer 1

Your shell appears to have limited you. I suspect that your guess is correct, and this isn't an awk problem, it's the scripting language from which you're calling awk.

You can pre-load awk with variables loaded from a file. Check this out:

$ printf 'foo=2\nbar=3\nbaz=4\n' > vars
$ printf 'snarf\nblarg\nbaz\nsnurry\n' > text
$ awk 'NR==FNR{split($0,a,"=");vars[a[1]]=a[2];next} $1 in vars {print vars[$1]}' vars text
4
$

How does this work?

The first two printf lines give us our raw data. Run them without the redirect (or cat the resultant files) if they're not completely clear to you.

The awk script has two main sections. Awk scripts consist of repetitions of condition { commands }. In this case, we've got two of these sets.

The first set has a condition of NR==FNR. This evaluates as "true" if the current record number that awk is processing (NR) is the same as the current record number in the current file. Obviously, this only works for the FIRST file, because as of the first line in the second file, NR is 1 plus the line count of the first file.

Within this section, we split() the line according to its equals sign, and put the data into an array called vars.

The second set has a condition of $1 in vars, which evaluates to true if the first word of the current line exists as a subscript of the vars array. I include this only as an example of what you can do with vars, since I don't know what you're trying to achieve with these variables.

Does this address your problem? If not, we'll need to see some of your code to get an idea of how to fix it.

---

UPDATE per suggestion in comments, here's proof that it works for large variables:

First, we prepare our input data:

$ dd if=/dev/random of=out.rand count=128k bs=1k
131072+0 records in
131072+0 records out
134217728 bytes transferred in 3.265765 secs (41098404 bytes/sec)
$ b64encode -o out.b64 out.rand out.rand
$ ls -lh out.b64
-rw-r--r--  1 ghoti  wheel   172M Jul 17 01:08 out.b64
$ awk 'BEGIN{printf("foo=")} NR>1{printf("%s",$0)} END{print ""}' out.b64 > vars
$ ls -lh vars
-rw-r--r--  1 ghoti  wheel   170M Jul 17 01:10 vars
$ wc -l vars
       1 vars
$ cut -c1-30 vars
foo=orq0UgQJyUAcwJV0SenJrSHu3j

Okay, we've got a ~170MB variable on a single line. Let's suck it into awk.

$ awk 'NR==FNR{split($0,a,"=");vars[a[1]]=a[2];next} END{print length(vars["foo"]);print "foo=" substr(vars["foo"],0,26);}' out.var bar
178956971
foo=orq0UgQJyUAcwJV0SenJrSHu3j

We can see the size of the variable, and the first 26 characters match what we saw from shell. Yup, it works.