Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Extracting certain rows from data using hash object in SAS

Tags:

hash

sas

I have two SAS data tables. The first has many millions of records, and each record is identified with a sequential record ID, like this:

Table A

Rec  Var1 Var2 ... VarX
1    ...
2
3

The second table specifies which rows from Table A should be assigned a coding variable:

Table B

Code  BegRec    EndRec
AA      1200      4370
AX      7241      9488
BY     12119     14763

So the first row of Table B means any data in Table A that has rec between 1200 and 4370 should be assigned code AA.

I know how to accomplish this with proc sql, but I want to see how this is done with a hash object.

In SQL, it's just:

proc sql;
 select b.code, a.*
 from tableA a, tableB b
 where b.begrec<=a.rec<=b.endrec;
quit;

My actual data contains hundreds of gigabytes of data, so I want to do the processing as efficiently as possible. My understanding is that using a hash object may help here, but I haven't been able to figure out how to map what I'm doing to use that way.

like image 418
itzy Avatar asked Aug 02 '26 01:08

itzy


1 Answers

A hash object solution (data input code borrowed from @Rob_Penridge).

    data big;
      do rec = 1 to 20000;
       output;
      end;
    run;

    data lookup;      
      input Code $ BegRec EndRec;
      datalines;
      AA      1200      4370
      AX      7241      9488
      BY     12119     14763
      ;
    run;


    data created;
      format code $4.;
      format begrec endrec best8.;
      if _n_=1 then do;
        declare hash h(dataset:'lookup');
        h.definekey('Code');
        h.definedata('code','begrec','endrec');
        h.definedone();
        call missing(code,begrec,endrec);
        declare hiter iter('h');
      end;

    set big;
    iter.first();
      do until (rc^=0);
       if begrec <= rec <= endrec then do;
       code_dup=code;
      end;
      rc=iter.next();
     end;
    keep rec code_dup;
    run;
like image 93
Robbie Liu Avatar answered Aug 05 '26 15:08

Robbie Liu



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!