Article View: fa.caml
Article #122Re: [Caml-list] Memory fragmentation
From: Török Edwin
Date: Fri, 20 Apr 2012 16:42
Date: Fri, 20 Apr 2012 16:42
34 lines
5594 bytes
5594 bytes
On 04/20/2012 06:12 PM, SerP wrote: > Hi! > We have developped a daemon on ocaml using the lwt lib(libev). It processes about 800 requests per second, but it increases to 2 Gb in memory for a hour of work. We have studied the problem for a long > time, using mtrace, mallinfo and other tools, > and we tried to change GC params. We found out that setting up GC.minor_heap_size=10Mb а major_heap_increment=2Мb will make the daemon grow slower. > mallinfо shows that the memory (1,5G) is allocated using mmap (blkhd field in mallinfo struct) and arena size - about 20M In this case, first calls of GC.compact compress the daemon to 200 Mb (live > -110Mb) , and mallinfo show decreasing of total size of memory allocated by mmap. The daemon keeps growing, but it allocates the memory to arena (using brk), and calls of GC.compact leads to decrease > of heap_bytes to 200M, but arena size (1.5 Gb) does not decrease, just doing less uordblks and more fordblks (fileds I.e., after first calls of GC.compact, the daemon starts using brk much more than > mmap, and cant memory given back to the system. Sounds like a fragmentation problem in glibc's heap that causes increased VIRT usage. IIRC glibc has a default threshold of 128k for deciding to use brk() vs mmap(), and OCaml's default major_heap_increment (124k) is just below that, which would explain the behaviour you see, and also explain why using a larger major_heap_increment (2M) improves the situation: brk() cannot free "holes", while mmap arenas can. You can use malloc_stats() to check whether OCaml gives back all memory to glibc or not, and you can try to use an alternative malloc implementation (like jemalloc) that might cope better with fragmentation. However this doesn't explain why memory keeps growing after a Gc.compact, unless calls to malloc are mixed between the OCaml runtime, Lwt and libev. If that is the case then maybe it would help to try and allocate memory in the OCaml's runtime by using mmap() directly. Is your application completely single-threaded (since you're using lwt), or do you also use multiple threads? I think that glibc caches mmap-ed areas even if you free them (i.e. they're still mapped into your process but they might be changed via mprotect() to PROT_NONE), which is especially a problem if you use multiple threads. Best regards, --Edwin -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Technical Details
Message-ID:
<fa.WQddSqm8+UY6SC1medqyBYdIiMs@ifi.uio.no>Path:
arc1-ca.pugleaf.net!r9ni78631pbh.0!nntp.google.com!news1.google.com!news3.google.com!feeder3.cambriumusenet.nl!feed.tweaknews.nl!195.208.113.67.MISMATCH!goblin1!goblin.stu.neva.ru!uio.no!nntp.uio.no!.POSTED!not-for-mailReferences:
<fa.BQPDkp4GTICgezqMipjyzfGHFCk@ifi.uio.no> <fa./QhU1susg5I2sxeCxgPZFPt/8n0@ifi.uio.no>Show Full Headers
Complete Article Headers
X-Google-Thread: ff861,73392fc8a24639f4
X-Google-NewGroupId: yes
X-Google-Attributes: gidc0fe4b917e,domainid0,public,usenet
X-Google-Language: ENGLISH,UTF8
Received: by 10.68.202.37 with SMTP id kf5mr6137408pbc.7.1334940150358;
Fri, 20 Apr 2012 09:42:30 -0700 (PDT)
Path: arc1-ca.pugleaf.net!r9ni78631pbh.0!nntp.google.com!news1.google.com!news3.google.com!feeder3.cambriumusenet.nl!feed.tweaknews.nl!195.208.113.67.MISMATCH!goblin1!goblin.stu.neva.ru!uio.no!nntp.uio.no!.POSTED!not-for-mail
From: Török Edwin <edwin+ml-ocaml@etorok.net>
Newsgroups: fa.caml
Subject: Re: [Caml-list] Memory fragmentation
Date: Fri, 20 Apr 2012 16:42:29 UTC
Organization: Internet mailing list
Lines: 35
Message-ID: <fa.WQddSqm8+UY6SC1medqyBYdIiMs@ifi.uio.no>
References: <fa.BQPDkp4GTICgezqMipjyzfGHFCk@ifi.uio.no> <fa./QhU1susg5I2sxeCxgPZFPt/8n0@ifi.uio.no>
NNTP-Posting-Host: mail-jess.uio.no
Mime-Version: 1.0
X-Trace: readme.uio.no 1334940149 22900 129.240.7.9 (20 Apr 2012 16:42:29 GMT)
X-Complaints-To: abuse@uio.no
NNTP-Posting-Date: Fri, 20 Apr 2012 16:42:29 +0000 (UTC)
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.3) Gecko/20120329 Icedove/10.0.3
To: caml-list@inria.fr
X-IronPort-AV: E=Sophos;i="4.75,454,1330902000";
d="scan'208";a="154919085"
X-Authentication-Warning: walapai.inria.fr: sympa set sender to caml-list-owner@inria.fr using -f
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AuYFAD+RkU+wCYo3/2dsb2JhbABEgxyCS6tRgQeCCQEBBSMPAQ0BATYCDwsYAgIFFgsCAgkDAgECAUUTCAKID6Z6boNDAY9bBoEvjBOCDYEYiGGNHIV0ikmCag
X-IronPort-AV: E=Sophos;i="4.75,454,1330902000";
d="scan'208";a="154918995"
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=etorok.net;
s=MAILOUT; t=1334940118;
bh=C5i+0U6fJZSUsrqPypoi//aR4MjfW5BT9zk6e3RIpH8=;
h=Message-ID:Date:From:MIME-Version:To:Subject:References:
In-Reply-To:Content-Type:Content-Transfer-Encoding;
b=MVOx6KOzW52ZPT+syBGJVSs1VUdsHOO9CvBl3AP8gFfLCs2acRvgRnAZHta7f4iOC
64utvVA3NEvMTvMlANRtCEqv9jp1jzzfp4RKJM3tENNfxGwPpbdeW4iD5X8cwwGzNg
kOtgRBSSPlb2BdqHclzQFxPMZQMUdsz0y6EAVruA=
In-Reply-To: <CAHgca+_OD0twkZ6WPc2bkQtEAS6Qnj1ogTDLE-S4esw6GKAV9A@mail.gmail.com>
X-Virus-Scanned: clamav-milter 0.97.3 at mail
X-Virus-Status: Clean
X-Sequence: 3948
List-Id: <caml-list.inria.fr>
List-Archive: <https://sympa-roc.inria.fr/wws/arc/caml-list>
List-Help: <mailto:sympa@inria.fr?subject=help>
List-Owner: <mailto:caml-list-request@inria.fr>
List-Post: <mailto:caml-list@inria.fr>
List-Subscribe: <mailto:sympa@inria.fr?subject=subscribe%20caml-list>
List-Unsubscribe: <mailto:sympa@inria.fr?subject=unsubscribe%20caml-list>
Original-Date: Fri, 20 Apr 2012 19:41:57 +0300
Original-Message-Id: <4F9191D5.6060809@etorok.net>
Original-References: <CAHgca+9-K71NynNVzS0Stnb2v5Tc6n5ecCnJHZorhuB8y8EdTw@mail.gmail.com> <CAHgca+_OD0twkZ6WPc2bkQtEAS6Qnj1ogTDLE-S4esw6GKAV9A@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit