Saturday, June 12, 2010

Estimating memory usage on Android and Linux

Earlier kernels export Rss and Vss of each single process to show how many page frames it occupies and how the address space it spans on a particular time. But this information is process-wide, and it doesn't help when you estimate system-wide memory usage. The problem is the lacking of information about how the processes share pages with each other. Simply summing up the Rss of all processes would over calculate the number of pages shared by multiple processes.

However, the situation has changed since the Kernel 2.6.25 merged the pagemap patch set from Matt Mackall. The pagemap introduced new metrics that are not only helpful for estimating system-wide memory usage, but also give more detailed memory utilization of a single process. We will illustrate the proposed idea with an example and show some useful utilities that came with Android.

Reference:
Related proc entries:
  • /proc/PID/smaps
  • /proc/PID/pagemap
  • /proc/kpagecount
  • /proc/kpageflags
The pagemap exports informative statistics of memory usage for each process, including:
  • Vss - Virtual set size
  • Rss - Resident set size
  • Pss - Proportional set size
  • Uss - Unique set size
  • ShCl - Shared clean pages
  • ShDi - Shared dirty pages
  • PrCl - Private clean pages
  • PrDi - Private dirty pages
The first two terms Vss and Rss have been extensively used in the familiar unix utility ps, top, etc. The Uss refers to the private pages of a process, which can be further exclusively divided into PrCl(clean ones) and PrDi(dirty ones). The Pss is a metric to show how the pages shared by this process weight in the whole system, and not surprisingly, it can be exclusively divided into the ShCl and ShDi, too.

Let's illustrate this by taking 3 running processes as a simplified system model:
   Process | working page set
  ---------+---------------------------------------
      A    |  1 2 3 4 5 6 7 8 9 10
      B    |    2 3 4                    13
      C    |    2 3 4       8 9 10 11 12
  ---------+---------------------------------------

  A: Rss = 10,   Uss = 4,     Pss = 4 + 3/3 + 3/2 = 6.5
  B: Rss = 4,    Uss = 1,     Pss = 1 + 3/3       = 2
  C: Rss = 8,    Uss = 2,     Pss = 2 + 3/3 + 3/2 = 4.5

  Sum(Pss[A..C]) = 13
In this case, process A occupies page 1..10, and among them, page 2..4 are shared with both B and C, page 8..10 are shared with C. So the Rss(total pages occupied) of A is 10, the Uss(private pages) of A is 4. And the Pss is calculated by adding the:
  • the private pages(Uss)
  • the weight of shared pages(summation of each shared page set divided by the sharing processes),
which will be 4 + (3/3 + 3/2) = 6.5.

The numbers for B and C are calculated in the same way.
The sum of *all* Pss should approximate the actual number of the physical memory occupied by these processes.

Android came with a bunch of user space libraries and utilities leveraging this facility. The source code can be found in:
  • [ANDROID_SRC]/system/extras/procrank
  • [ANDROID_SRC]/system/extras/procmem
  • [ANDROID_SRC]/system/extras/showmap
Here is the log of procrank running on Eclair on a emulator:
PID      Vss      Rss      Pss      Uss  cmdline
   51   32236K   31636K   13780K   10636K  system_server
  212   29784K   29784K   12167K    9152K  com.android.browser
   29   25796K   25796K    7933K    4948K  zygote
   97   24476K   24476K    7417K    4940K  android.process.acore
   94   22064K   22064K    5947K    3880K  com.android.phone
   92   21724K   21724K    5632K    3536K  com.android.inputmethod.pinyin
  171   19748K   19748K    4093K    2244K  com.android.email
  154   19364K   19364K    3915K    2036K  com.android.mms
  112   19072K   19072K    3749K    1744K  com.android.settings
  146   19160K   19160K    3660K    1992K  android.process.media
  132   18524K   18524K    3117K    1476K  com.android.alarmclock
   30    3188K    3188K     962K     628K  /system/bin/mediaserver
  266     468K     468K     281K     272K  procrank
   28     704K     704K     252K     216K  /system/bin/rild
    1     188K     188K     169K     168K  /init
   36     180K     180K     168K     168K  /sbin/adbd
   26     364K     364K     143K     132K  /system/bin/vold
   34     336K     336K     134K     124K  /system/bin/qemud
  188     328K     328K     106K      72K  /system/bin/sh
   32     400K     400K     106K      88K  /system/bin/keystore
   33     324K     324K     102K      68K  /system/bin/sh
   24     308K     308K      96K      68K  /system/bin/sh
   43     308K     308K      94K      84K  /system/bin/qemu-props
   31     284K     284K      93K      84K  /system/bin/installd
   25     260K     260K      92K      84K  /system/bin/servicemanager
   27     248K     248K      83K      76K  /system/bin/debuggerd
And the log of procemem for the alarmclock applicactions(PID 132):
Vss      Rss      Pss      Uss     ShCl     ShDi     PrCl     PrDi  Name
-------  -------  -------  -------  -------  -------  -------  -------  
     4K       4K       0K       0K       4K       0K       0K       0K  /system/bin/app_process
     4K       4K       0K       0K       4K       0K       0K       0K  /system/bin/app_process
  2788K    2788K     684K     452K    2336K       0K     452K       0K  [heap]
     0K       0K       0K       0K       0K       0K       0K       0K  [heap]
     4K       4K       4K       4K       0K       0K       4K       0K  [heap]
    16K      16K       1K       0K       0K      16K       0K       0K  /dev/ashmem/system_properties
     4K       4K       0K       0K       4K       0K       0K       0K  /dev/ashmem/system_properties
  2620K    2620K     412K     180K    2440K       0K     180K       0K  /dev/ashmem/mspace/dalvik-heap/zygote/0
     0K       0K       0K       0K       0K       0K       0K       0K  /dev/ashmem/mspace/dalvik-heap/zygote/0
     0K       0K       0K       0K       0K       0K       0K       0K  /system/fonts/DroidSans.ttf
    72K      72K      57K      48K      24K       0K      48K       0K  /system/fonts/DroidSans.ttf
     0K       0K       0K       0K       0K       0K       0K       0K  /dev/ashmem/dalvik-LinearAlloc
  2508K    2508K     262K      36K    2472K       0K      36K       0K  /dev/ashmem/dalvik-LinearAlloc
     0K       0K       0K       0K       0K       0K       0K       0K  /dev/ashmem/dalvik-LinearAlloc
     0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/core.jar
  1204K    1204K     117K       0K    1204K       0K       0K       0K  /data/dalvik-cache/system@framework@core.jar@classes.dex
   200K     200K      75K      60K     140K       0K      60K       0K  /data/dalvik-cache/system@framework@core.jar@classes.dex
     0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/ext.jar
    72K      72K       6K       0K      72K       0K       0K       0K  /data/dalvik-cache/system@framework@ext.jar@classes.dex
     0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/framework.jar
  1884K    1884K     178K       0K    1884K       0K       0K       0K  /data/dalvik-cache/system@framework@framework.jar@classes.dex
   452K     452K     244K     212K     240K       0K     212K       0K  /data/dalvik-cache/system@framework@framework.jar@classes.dex
     0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/android.policy.jar
    20K      20K       1K       0K      20K       0K       0K       0K  /data/dalvik-cache/system@framework@android.policy.jar@classes.dex
     0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/services.jar
    16K      16K       1K       0K      16K       0K       0K       0K  /data/dalvik-cache/system@framework@services.jar@classes.dex
   132K     132K     120K     116K      16K       0K     116K       0K  /data/dalvik-cache/system@framework@services.jar@classes.dex
     4K       4K       4K       4K       0K       0K       4K       0K  /dev/ashmem/dalvik-heap-bitmap/objects
    20K      20K      20K      20K       0K       0K      20K       0K  /dev/ashmem/dalvik-heap-bitmap/objects
     4K       4K       0K       0K       4K       0K       0K       0K  /dev/ashmem/SurfaceFlinger
    24K      24K      21K      20K       4K       0K      20K       0K  /system/app/AlarmClock.apk
   144K     144K      13K       0K     144K       0K       0K       0K  /system/app/AlarmClock.apk
    44K      44K       7K       0K      44K       0K       0K       0K  /dev/ashmem/dalvik-heap-bitmap/mark/0
     4K       4K       4K       4K       0K       0K       4K       0K  /dev/ashmem/dalvik-heap-bitmap/objects
     0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/framework-res.apk
     0K       0K       0K       0K       0K       0K       0K       0K  /system/framework/framework-res.apk
    24K      24K      13K      12K      12K       0K      12K       0K  /dev/ashmem/mspace/dalvik-heap/zygote/1
     0K       0K       0K       0K       0K       0K       0K       0K  /dev/ashmem/mspace/dalvik-heap/zygote/1
   152K     152K     152K     152K       0K       0K     152K       0K  /dev/ashmem/mspace/dalvik-heap/2
     0K       0K       0K       0K       0K       0K       0K       0K  /dev/ashmem/mspace/dalvik-heap/2
     0K       0K       0K       0K       0K       0K       0K       0K  /dev/ashmem/mspace/dalvik-heap/2
     4K       4K       4K       4K       0K       0K       4K       0K  /dev/ashmem/mspace/dalvik-heap/2
     0K       0K       0K       0K       0K       0K       0K       0K  /dev/ashmem/mspace/dalvik-heap/2
     4K       4K       4K       4K       0K       0K       4K       0K  /dev/ashmem/mspace/dalvik-heap/2
     4K       4K       4K       4K       0K       0K       4K       0K  /dev/binder
     0K       0K       0K       0K       0K       0K       0K       0K  /dev/binder
     4K       4K       4K       4K       0K       0K       4K       0K  /dev/binder
     0K       0K       0K       0K       0K       0K       0K       0K  /dev/binder
     4K       4K       4K       4K       0K       0K       4K       0K  /dev/binder
    20K      20K       9K       0K      20K       0K       0K       0K  /system/app/AlarmClock.apk
    20K      20K       9K       0K      20K       0K       0K       0K  /system/app/AlarmClock.apk
    60K      60K      60K      60K       0K       0K      60K       0K  /data/dalvik-cache/system@app@AlarmClock.apk@classes.dex
   516K     516K      49K       0K     516K       0K       0K       0K  /system/lib/libicudata.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libicudata.so
    24K      24K       1K       0K      24K       0K       0K       0K  /system/lib/libskiagl.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libskiagl.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libemoji.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libemoji.so
    68K      68K       6K       0K      68K       0K       0K       0K  /system/lib/libsrec_jni.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libsrec_jni.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libdrm1.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libdrm1.so
   156K     156K      12K       0K     156K       0K       0K       0K  /system/lib/libopencore_common.so
    24K      24K       2K       0K      24K       0K       0K       0K  /system/lib/libopencore_common.so
    96K      96K       7K       0K      96K       0K       0K       0K  /system/lib/libopencore_player.so
    32K      32K       2K       0K      32K       0K       0K       0K  /system/lib/libopencore_player.so
    20K      20K       1K       0K      20K       0K       0K       0K  /system/lib/libomx_amrenc_sharedlibrary.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libomx_amrenc_sharedlibrary.so
    44K      44K       3K       0K      44K       0K       0K       0K  /system/lib/libopencore_net_support.so
    12K      12K       1K       0K      12K       0K       0K       0K  /system/lib/libopencore_net_support.so
    28K      28K       2K       0K      28K       0K       0K       0K  /system/lib/libomx_sharedlibrary.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libomx_sharedlibrary.so
    16K      16K       1K       0K      16K       0K       0K       0K  /system/lib/libhardware_legacy.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libhardware_legacy.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libhardware.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libhardware.so
   128K     128K      10K       0K     128K       0K       0K       0K  /system/lib/libutils.so
     4K       4K       4K       4K       0K       0K       4K       0K  /system/lib/libutils.so
   128K     128K      10K       0K     128K       0K       0K       0K  /system/lib/libbinder.so
    28K      28K       6K       4K      24K       0K       4K       0K  /system/lib/libbinder.so
   168K     168K      15K       0K     168K       0K       0K       0K  /system/lib/libwebcore.so
   124K     124K      11K       0K     124K       0K       0K       0K  /system/lib/libwebcore.so
     8K       8K       0K       0K       8K       0K       0K       0K  /system/lib/libwebcore.so
    56K      56K      10K       0K      56K       0K       0K       0K  /system/lib/libexpat.so
     8K       8K       1K       0K       8K       0K       0K       0K  /system/lib/libexpat.so
   248K     248K      35K       0K     248K       0K       0K       0K  /system/lib/libsqlite.so
     4K       4K       4K       4K       0K       0K       4K       0K  /system/lib/libsqlite.so
   188K     188K      15K       0K     188K       0K       0K       0K  /system/lib/libmedia.so
    48K      48K       4K       0K      48K       0K       0K       0K  /system/lib/libmedia.so
    32K      32K       2K       0K      32K       0K       0K       0K  /system/lib/libmedia_jni.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libmedia_jni.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libvorbisidec.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libvorbisidec.so
    12K      12K       0K       0K      12K       0K       0K       0K  /system/lib/libsonivox.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libsonivox.so
     0K       0K       0K       0K       0K       0K       0K       0K  /system/lib/libsonivox.so
   588K     588K      51K       0K     588K       0K       0K       0K  /system/lib/libskia.so
    16K      16K       1K       0K      16K       0K       0K       0K  /system/lib/libskia.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libskia.so
   152K     152K      12K       0K     152K       0K       0K       0K  /system/lib/libui.so
    28K      28K       6K       4K      24K       0K       4K       0K  /system/lib/libui.so
    12K      12K       1K       0K      12K       0K       0K       0K  /system/lib/libexif.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libexif.so
     0K       0K       0K       0K       0K       0K       0K       0K  /system/lib/libexif.so
    16K      16K       1K       0K      16K       0K       0K       0K  /system/lib/libEGL.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libEGL.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libEGL.so
     8K       8K       0K       0K       8K       0K       0K       0K  /system/lib/libGLESv1_CM.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libGLESv1_CM.so
    52K      52K       4K       0K      52K       0K       0K       0K  /system/lib/libpixelflinger.so
     8K       8K       0K       0K       8K       0K       0K       0K  /system/lib/libpixelflinger.so
   404K     404K      36K       0K     404K       0K       0K       0K  /system/lib/libdvm.so
    12K      12K       4K       4K       8K       0K       4K       0K  /system/lib/libdvm.so
     4K       4K       4K       4K       0K       0K       4K       0K  /system/lib/libdvm.so
   160K     160K      14K       0K     160K       0K       0K       0K  /system/lib/libnativehelper.so
    12K      12K       1K       0K      12K       0K       0K       0K  /system/lib/libnativehelper.so
   316K     316K      27K       0K     316K       0K       0K       0K  /system/lib/libandroid_runtime.so
    28K      28K       6K       4K      24K       0K       4K       0K  /system/lib/libandroid_runtime.so
     8K       8K       0K       0K       8K       0K       0K       0K  /system/lib/libandroid_runtime.so
   508K     508K      46K       0K     508K       0K       0K       0K  /system/lib/libicui18n.so
    16K      16K       5K       4K      12K       0K       4K       0K  /system/lib/libicui18n.so
   444K     444K      39K       0K     444K       0K       0K       0K  /system/lib/libicuuc.so
    20K      20K       5K       4K      16K       0K       4K       0K  /system/lib/libicuuc.so
     4K       4K       4K       4K       0K       0K       4K       0K  /system/lib/libicuuc.so
     0K       0K       0K       0K       0K       0K       0K       0K  /system/lib/libnetutils.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libnetutils.so
     0K       0K       0K       0K       0K       0K       0K       0K  /system/lib/libwpa_client.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libwpa_client.so
   308K     308K      26K       0K     308K       0K       0K       0K  /system/lib/libcrypto.so
    60K      60K       5K       0K      60K       0K       0K       0K  /system/lib/libcrypto.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libcrypto.so
    56K      56K       4K       0K      56K       0K       0K       0K  /system/lib/libssl.so
    12K      12K       1K       0K      12K       0K       0K       0K  /system/lib/libssl.so
    40K      40K       3K       0K      40K       0K       0K       0K  /system/lib/libz.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libz.so
    44K      44K       3K       0K      44K       0K       0K       0K  /system/lib/libcutils.so
     4K       4K       4K       4K       0K       0K       4K       0K  /system/lib/libcutils.so
     0K       0K       0K       0K       0K       0K       0K       0K  /system/lib/libcutils.so
    12K      12K       0K       0K      12K       0K       0K       0K  /system/lib/liblog.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/liblog.so
    40K      40K       3K       0K      40K       0K       0K       0K  /system/lib/libm.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libm.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libstdc++.so
     4K       4K       0K       0K       4K       0K       0K       0K  /system/lib/libstdc++.so
   168K     168K       8K       0K     168K       0K       0K       0K  /system/lib/libc.so
    12K      12K       1K       0K      12K       0K       0K       0K  /system/lib/libc.so
    28K      28K      17K      16K      12K       0K      16K       0K  /system/lib/libc.so
     0K       0K       0K       0K       0K       0K       0K       0K  /system/bin/linker
     4K       4K       0K       0K       4K       0K       0K       0K  /system/bin/linker
    24K      24K       3K       0K      24K       0K       0K       0K  /system/bin/linker
    20K      20K      16K      16K       4K       0K      16K       0K  [stack]
-------  -------  -------  -------  -------  -------  -------  -------  
 18524K   18524K    3117K    1476K   17032K      16K    1476K       0K  TOTAL

No comments: