yori aggregation
Yori is attempting to run on Paul's files, the files have been compressed. Anyways, its trying to run on 6 hours of data right now, so not even a full day, this is what it look like in top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10528 flo 20 0 69.1g 54.4g 4144 D 22.9 86.6 636:35.33 yori-aggr
I can't imagine that it is going to finish and even if it does its using 86.6% of the memory to run on 6 hours of data so I don't see how we can make it work on a whole day. Can you review your code to see if there are any memory optimization. I.e. not holding on to everything in memory, possibly writing out a variable as its produced and so on.
You can find sample data for testing here:
[steved@sipsdev ~]$ du -sh /mnt/dawg/dev/viirs/snpp/VNPCLDPROP_G3/0.1dev4/2014/* 3.9G /mnt/dawg/dev/viirs/snpp/VNPCLDPROP_G3/0.1dev4/2014/032 31G /mnt/dawg/dev/viirs/snpp/VNPCLDPROP_G3/0.1dev4/2014/033 4.0G /mnt/dawg/dev/viirs/snpp/VNPCLDPROP_G3/0.1dev4/2014/034
That is one full day plus 3 hours before and 3 hours after