Large Data Sets - It is worth the effort to be efficient?
How large is the data set created by this data step?
data bigone; drop i j k; array XX {*} X1-X100; do i=1 to 100; do j=1 to 100; do k=1 to dim(XX); XX(k)=k + 1000*j; end; output; end; end; run;
The SAS log says 10,000 observations and 100 variables. But how much space is needed to store the data set?
Let's see. Each numeric variable needs 8 bytes for each observation.
So, 8*100*10,000 = 8,000,000 bytes, or 7.63 megabytes. (1 meg = 1,048,576 bytes)
Some additional space is needed for the data descriptors, etc.
In all, this data set takes 8,209,408 bytes (7.83 meg).