Re: [Caml-list] Memory fragmentation

Article View: fa.caml

Article #125

#125

From: ygrek
Date: Fri, 20 Apr 2012 21:35

36 lines
4928 bytes

On Fri, 20 Apr 2012 19:12:12 +0400
SerP <serp256@gmail.com> wrote:

> Hi!
> We have developped a daemon on ocaml using the lwt lib(libev). It processes
> about 800 requests per second, but it increases to 2 Gb in memory for a
> hour of work. We have studied the problem for a long time, using mtrace,
> mallinfo and other tools, and we tried to change GC params. We found out
> that setting up GC.minor_heap_size=10Mb а major_heap_increment=2Мb will
> make the daemon grow slower. mallinfо shows that the memory (1,5G) is
> allocated using mmap (blkhd field in mallinfo struct) and arena size -
> about 20M In this case, first calls of GC.compact compress the daemon to
> 200 Mb (live -110Mb) , and mallinfo show decreasing of total size of memory
> allocated by mmap. The daemon keeps growing, but it allocates the memory to
> arena (using brk), and calls of GC.compact leads to decrease of heap_bytes
> to 200M, but arena size (1.5 Gb) does not decrease, just doing less
> uordblks and more fordblks (fileds I.e., after first calls of GC.compact,
> the daemon starts using brk much more than mmap, and cant memory given back
> to the system.

Alternative malloc implementation may well be worth a try. I've seen the cases
when glibc was constantly growing RSS while tcmalloc was on stable level (and 10x less), 
guessing due to fragmentation.
Maybe related, see PR#5389 for ocaml compaction related issues.
Also have a look at http://ygrek.org.ua/p/code/mlvalues.py for the way to peep
into ocaml heap at runtime.

-- 
 ygrek
 http://ygrek.org.ua

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa-roc.inria.fr/wws/info/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Technical Details

Message-ID: <fa.dje/0VYwIcw/uOwSfQpKUQRzetY@ifi.uio.no>

Path:

arc1-ca.pugleaf.net!r9ni79397pbh.0!nntp.google.com!news1.google.com!goblin3!goblin1!goblin.stu.neva.ru!uio.no!nntp.uio.no!.POSTED!not-for-mail

References: <fa.BQPDkp4GTICgezqMipjyzfGHFCk@ifi.uio.no> <fa./QhU1susg5I2sxeCxgPZFPt/8n0@ifi.uio.no>

Show Full Headers

Complete Article Headers

X-Google-Thread: ff861,73392fc8a24639f4
X-Google-NewGroupId: yes
X-Google-Attributes: gidc0fe4b917e,domainid0,public,usenet
X-Google-Language: ENGLISH,UTF8
Received: by 10.68.202.37 with SMTP id kf5mr6890454pbc.7.1334957724616;
        Fri, 20 Apr 2012 14:35:24 -0700 (PDT)
Path: arc1-ca.pugleaf.net!r9ni79397pbh.0!nntp.google.com!news1.google.com!goblin3!goblin1!goblin.stu.neva.ru!uio.no!nntp.uio.no!.POSTED!not-for-mail
From: ygrek <ygrekheretix@gmail.com>
Newsgroups: fa.caml
Subject: Re: [Caml-list] Memory fragmentation
Date: Fri, 20 Apr 2012 21:35:23 UTC
Organization: Internet mailing list
Lines: 37
Message-ID: <fa.dje/0VYwIcw/uOwSfQpKUQRzetY@ifi.uio.no>
References: <fa.BQPDkp4GTICgezqMipjyzfGHFCk@ifi.uio.no> <fa./QhU1susg5I2sxeCxgPZFPt/8n0@ifi.uio.no>
NNTP-Posting-Host: mail-jess.uio.no
Mime-Version: 1.0
X-Trace: readme.uio.no 1334957723 9690 129.240.7.9 (20 Apr 2012 21:35:23 GMT)
X-Complaints-To: abuse@uio.no
NNTP-Posting-Date: Fri, 20 Apr 2012 21:35:23 +0000 (UTC)
To: caml-list@inria.fr
X-IronPort-AV: E=Sophos;i="4.75,455,1330902000"; 
   d="scan'208";a="154939346"
X-Authentication-Warning: walapai.inria.fr: sympa set sender to caml-list-owner@inria.fr using -f
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AvcDAOvVkU9KfVK2kGdsb2JhbABEhWiqQwSBBQgiAQEBAQkJDQcUBCOCCQEBAQMBEgIPDwENARseAwELBgULDwIYDgICIQIRAQUBHBkih14BAwYFBAecMwqLUlCCc4UkChknDVeIdgEFC4EkgmeFX4ESCoJ3gg2BGASIXYUyh2kBgRGKK4MhPYQN
X-IronPort-AV: E=Sophos;i="4.75,455,1330902000"; 
   d="scan'208";a="140990527"
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=date:from:to:subject:message-id:in-reply-to:references:x-mailer
         :mime-version:content-type:content-transfer-encoding;
        bh=nQS0swK30LdtY34lKnRQGrJp5cyd/zG9Z+5Zpbn7PlM=;
        b=zRN2S+98uofKY6U8dxrpC2n3vW663NyU2trGX8Q+7fzpugtRnzsdmHY7hA/IyKlA95
         3iMlROE/3ixlbe47q9vkhG1IQlf0ei2BhJ49SwZXDNtwA6XhQDM1qMZnzE+CQ5Yx0AWx
         7KAfyKNrf7Q+L6vG6lk3i3LAYFIYZc/g9wt4snLrCDz2K9B2wPJuw1y6gZ6aWfSzDAR9
         v6uca7csBbkRlmNfHIj+37KCalEpNWNNDWZyJ45KFY1OmOal7DKUaYv4TQl+aqDtGPY5
         nkkBbWzzBrqTMU7ePDuGJVKAKDfhUdQ6069i3nF0+L7K7BD6w2RWy8GEvE8SMGNKJLPX
         gvFw==
In-Reply-To: <CAHgca+_OD0twkZ6WPc2bkQtEAS6Qnj1ogTDLE-S4esw6GKAV9A@mail.gmail.com>
X-Mailer: Sylpheed 3.0.2 (GTK+ 2.20.1; x86_64-pc-linux-gnu)
X-Sequence: 3951
List-Id: <caml-list.inria.fr>
List-Archive: <https://sympa-roc.inria.fr/wws/arc/caml-list>
List-Help: <mailto:sympa@inria.fr?subject=help>
List-Owner: <mailto:caml-list-request@inria.fr>
List-Post: <mailto:caml-list@inria.fr>
List-Subscribe: <mailto:sympa@inria.fr?subject=subscribe%20caml-list>
List-Unsubscribe: <mailto:sympa@inria.fr?subject=unsubscribe%20caml-list>
Original-Date: Sat, 21 Apr 2012 00:32:19 +0300
Original-Message-Id: <20120421003219.ce3d8b2c.ygrekheretix@gmail.com>
Original-References: <CAHgca+9-K71NynNVzS0Stnb2v5Tc6n5ecCnJHZorhuB8y8EdTw@mail.gmail.com>
 <CAHgca+_OD0twkZ6WPc2bkQtEAS6Qnj1ogTDLE-S4esw6GKAV9A@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Reply to Parent Message