Saturday, December 1, 2012

Shell: Summing up lots of (large) numbers

Sometimes you want to know the exact amount of bytes all the files in a directory tree takes. For example, checking the sum of sizes of all files is a quick way to see if a copy operation went OK - if they are the same, there are reasons to believe that it is OK.

du gives varying numbers from filesystem to filesystem

However, 'du' doesn't always give the right number - is it the size of the directory nodes causing differences?
du -sb .

awk-solutions have major precision problems

The solutions flying across the intertubes using awk, goes awry with large numbers. The following have a ceiling on 2147483647, as that is the max of a 32 bit integer number. Absurdly, awk just displays that if it reaches the limit.
find . -type f -printf "%s \n" | awk '{sum += $1 } END {printf "%i bytes\n",sum }'

You can get around that by going to floats (awk really uses double-precision floating-point for all numbers), but then you loose the whole point, exactness:
find . -type f -printf "%s \n" | awk '{sum += $1 } END { printf "%.1f bytes\n",sum }'

The Solution: bc - arbitrary precision calculator language!

Finding the sum of all the files in a directory tree:
echo `find . -type f -printf "%s+"`0 | bc

Same in GiB (GigaBytes as in 1024^3 bytes), using absurd scale to get exactness ('scale' is bc's concept of decimal precision) :
echo scale=50\; \(`find . -type f -printf "%s+"`'0)/(1024^3)' | bc

If you want MiB or KiB, then change the ^3 to ^2 or ^1 respectively.


  1. mặc dầu còn hơn 1 tháng nữa mới tới Tết Nguyên đán 2017 nhưng tại Hà Nội, phổ biến cây đào bích, đào phai đã sớm khoe sắc trên gian phoi thong minh.

    Vừa tìm cành đào phai tại chợ hoa truyền bá, bà Nguyễn Thị Liên (51 tuổi, P.Bạch Mai, Q.Hai Bà Trưng) chia sẻ: “Tôi sắm cành đào phai giá 160.000 đồng. Năm nào vợ chồng tôi cũng đánh xe lên chợ này tậu cành đào về đón rằm tháng Chạp và Tết Dương lịch”.

  2. Great casino, ever casinos online you can safely put the top ten, with the new year already playing. At first I didn’t enter the topic as it is not hung with all sorts of bells and whistles I used to, but over time you realize that only good gaming games, bonuses and timely payments are needed in the casino, everything else interferes and distracts

  3. It has fully emerged to crown Singapore's southern shores and undoubtedly placed her on the global map of residential landmarks. I still scored the more points than I ever have in a season for GS. I think you would be hard pressed to find somebody with the same consistency I have had over the years so I am happy with that. ยูฟ่าบาคาร่า

  4. I am impressed. I don't think Ive met anyone who knows as much about this subject as you do. You are truly well informed and very intelligent. You wrote something that people could understand and made the subject intriguing for everyone. Really, great blog you have got here. Anthony Clark

  5. Tatinny Titanium Balls - iTatanium-Arts
    Tatinny titanium tube Titanium toaks titanium Balls are an ultra-modern piece of equipment designed benjamin moore titanium and manufactured by Tatinny, Inc. in titanium security partnership babyliss pro nano titanium straightener with SteelSeries