[Caml-list] Memory fragmentation

Article View: fa.caml

Article #120

#120

From: SerP
Date: Fri, 20 Apr 2012 15:12

27 lines
4366 bytes

Hi!
We have developped a daemon on ocaml using the lwt lib(libev). It processes
about 800 requests per second, but it increases to 2 Gb in memory for a
hour of work. We have studied the problem for a long time, using mtrace,
mallinfo and other tools, and we tried to change GC params. We found out
that setting up GC.minor_heap_size=10Mb � major_heap_increment=2�b will
make the daemon grow slower. mallinf� shows that the memory (1,5G) is
allocated using mmap (blkhd field in mallinfo struct) and arena size -
about 20M In this case, first calls of GC.compact compress the daemon to
200 Mb (live -110Mb) , and mallinfo show decreasing of total size of memory
allocated by mmap. The daemon keeps growing, but it allocates the memory to
arena (using brk), and calls of GC.compact leads to decrease of heap_bytes
to 200M, but arena size (1.5 Gb) does not decrease, just doing less
uordblks and more fordblks (fileds I.e., after first calls of GC.compact,
the daemon starts using brk much more than mmap, and cant memory given back
to the system.

If Gc.allocation_policy is first fit, memory usage is stable, and it's grow
very little, but worked very slowly.

Any suggestions are very welcome. Thanks

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa-roc.inria.fr/wws/info/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Technical Details

Message-ID: <fa./QhU1susg5I2sxeCxgPZFPt/8n0@ifi.uio.no>

Path:

arc1-ca.pugleaf.net!r9ni78409pbh.0!nntp.google.com!news1.google.com!news.glorb.com!uio.no!nntp.uio.no!.POSTED!not-for-mail

References: <fa.BQPDkp4GTICgezqMipjyzfGHFCk@ifi.uio.no>

Show Full Headers

Complete Article Headers

X-Google-Thread: ff861,73392fc8a24639f4,start
X-Google-NewGroupId: yes
X-Google-Attributes: gidc0fe4b917e,domainid0,public,usenet
X-Google-Language: ENGLISH,ASCII
Received: by 10.68.227.230 with SMTP id sd6mr5911308pbc.8.1334934758149;
        Fri, 20 Apr 2012 08:12:38 -0700 (PDT)
Path: arc1-ca.pugleaf.net!r9ni78409pbh.0!nntp.google.com!news1.google.com!news.glorb.com!uio.no!nntp.uio.no!.POSTED!not-for-mail
From: SerP <serp256@gmail.com>
Newsgroups: fa.caml
Subject: [Caml-list] Memory fragmentation
Date: Fri, 20 Apr 2012 15:12:37 UTC
Organization: Internet mailing list
Lines: 35
Message-ID: <fa./QhU1susg5I2sxeCxgPZFPt/8n0@ifi.uio.no>
References: <fa.BQPDkp4GTICgezqMipjyzfGHFCk@ifi.uio.no>
NNTP-Posting-Host: mail-jess.uio.no
Mime-Version: 1.0
X-Trace: readme.uio.no 1334934757 17097 129.240.7.9 (20 Apr 2012 15:12:37 GMT)
X-Complaints-To: abuse@uio.no
NNTP-Posting-Date: Fri, 20 Apr 2012 15:12:37 +0000 (UTC)
To: caml-list@inria.fr
X-IronPort-AV: E=Sophos;i="4.75,453,1330902000"; 
   d="scan'208";a="154908506"
X-Authentication-Warning: walapai.inria.fr: sympa set sender to caml-list-owner@inria.fr using -f
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AtkBAGJ7kU/RVaC2kGdsb2JhbABDhWerSQgiAQEBAQkJDQcUBCOCCgEBBBICDCABGx4DDAYJAQY7IgERAQUBHDuHXgEDC5wdCotJBlOCc4U4ChknDVeIdgEFC403gguBGgSIXYUyh2qOXT2EDw
X-IronPort-AV: E=Sophos;i="4.75,453,1330902000"; 
   d="scan'208";a="140960648"
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        bh=Jzp3i9vaooTJyYs7s16WRRmSF7ApsAcmQIubNOr/u84=;
        b=m5uB0iYr37mra5jSNAvI9RZjfvRUsqdJXpTXgpcalrFCYpAyLD+hd1Cz2Bfnsbts0g
         Fj/UdhKN2c5kA0URq9MAD4xwKtYneuf6tugMMLuV45I5oQLGl6/YRbbzA8M3h2ZJ30kW
         EdjWkmO2B4fUBZ31GVk+9OeTUrvNelOjKffSn2NzmuvrEwOLF5Y46ZDC38HOGprbJv1w
         zlEblRyi/e/ttoEKOhnoEk8QWWC2KBgAYSG74BskmJKBUN9JjL+FV/LZ+s+AsZHpIMSz
         O72DvkSu9bl4Q+YGXhkimUoq+tO9jP69x6c6vIh916tkIA1fw+zLpiXEBb9+uACDyCsS
         7lcg==
In-Reply-To: <CAHgca+9-K71NynNVzS0Stnb2v5Tc6n5ecCnJHZorhuB8y8EdTw@mail.gmail.com>
X-Sequence: 3946
List-Id: <caml-list.inria.fr>
List-Archive: <https://sympa-roc.inria.fr/wws/arc/caml-list>
List-Help: <mailto:sympa@inria.fr?subject=help>
List-Owner: <mailto:caml-list-request@inria.fr>
List-Post: <mailto:caml-list@inria.fr>
List-Subscribe: <mailto:sympa@inria.fr?subject=subscribe%20caml-list>
List-Unsubscribe: <mailto:sympa@inria.fr?subject=unsubscribe%20caml-list>
X-FA-Processed: Removed HTML alternative
Original-Date: Fri, 20 Apr 2012 19:12:12 +0400
Original-Message-Id: <CAHgca+_OD0twkZ6WPc2bkQtEAS6Qnj1ogTDLE-S4esw6GKAV9A@mail.gmail.com>
Original-References: <CAHgca+9-K71NynNVzS0Stnb2v5Tc6n5ecCnJHZorhuB8y8EdTw@mail.gmail.com>
Content-Type: text/plain; charset=KOI8-R
Content-Transfer-Encoding: 8bit

Reply to Parent Message