Re: pbulk using tmpfs

Article View: fa.netbsd.tech.kern

Article #67

Back to fa.netbsd.tech.kern Previous Next

#67

From: Jan Danielsson
Date: Sat, 20 Apr 2013 22:13

49 lines
4949 bytes

On 04/20/13 23:47, iMil wrote:
[---]
> In order to speed up builds, I would like to know if I
> could benefit from tmpfs by mounting the WRKOBJDIR as a tmpfs
> directory.
> 
> In particular, while most packages will fit into a couple of
> megabytes, some of them (firefox, libreoffice, openoffice,
> koffice and such) take a couple of gigabytes to get built.
> 
> My question is: how will tmpfs behave? If I mount the WRKOBJDIR
> without any argument, it will "use", if I understand correctly
> the manpage, all of the available memory. Does this mean I will
> end up in a situation were the system memory will be filled by
> the build or is there some kind of smart mechanism within tmpfs
> which will "overflow" somewhere?

   I use pkg_comp (a locally modded version of it, which configures the
work directory onto a tmpfs directory), and I don't know what pbulk is
(the name implies the basic idea though).

   I have been using tmpfs-mounted directories for work directories on
two of my systems for well over a year. I know one shouldn't trust top
at all, etc, but watching the build hog more and more memory, it's "all"
returned as the build completes and I run "make clean-depends" and/or
"make clean".

   One of the systems has 16GB ram, and on that I build both pkgsrc
packages and NetBSD itself on the tmpfs (read: object work dir only).
The other system has 2GB RAM (and 0.5G swap), and on that I only build
pkgsrc on tmpfs.

   With regards to packages with gigantic dependencies: I don't know how
pbulk works, but I have a script which goes through a file from top to
bottom and builds the packages listed in it, and for each
completed/failed build cleans up the workdir. I've strategically placed
common big near-leaf node dependencies at the top of the file. I doubt I
could build all packages on the system with 2GB ram if I wouldn't have
sorted the file that way. The swap is used, albeit little of it, on this
particular system, even when I build firefox. But it should be noted
that I have built all dependencies separately before building it.

   I've seen a few scary reports about problems with tmpfs, but I
haven't encountered any problems despite having rebuilt my packages
and/or NetBSD probably - on average - once a week.

-- 
Kind regards,
Jan Danielsson

Technical Details

Message-ID: <fa./m6DKxI71Cy6h4EiQTf2YA26U3s@ifi.uio.no>

Path: arc1-ca.pugleaf.net!hg5ni4810wib.1!nntp.google.com!goblin1!goblin.stu.neva.ru!uio.no!nntp.uio.no!.POSTED!not-for-mail

References: <fa.e83HgwaM2kaCuuUpS6tVoQ1+1Kk@ifi.uio.no>

Show Full Headers

Complete Article Headers

X-Google-Thread: 10c6f4,786e9905c90c1180
X-Google-NewGroupId: yes
X-Google-Attributes: gidb7e13b6095,domainid0,public,usenet
X-Google-Language: ENGLISH,ASCII-7-bit
X-Received: by 10.180.75.82 with SMTP id a18mr9516420wiw.0.1366495984532;
        Sat, 20 Apr 2013 15:13:04 -0700 (PDT)
Path: arc1-ca.pugleaf.net!hg5ni4810wib.1!nntp.google.com!goblin1!goblin.stu.neva.ru!uio.no!nntp.uio.no!.POSTED!not-for-mail
From: Jan Danielsson <jan.m.danielsson@gmail.com>
Newsgroups: fa.netbsd.tech.kern
Subject: Re: pbulk using tmpfs
Date: Sat, 20 Apr 2013 22:13:04 UTC
Organization: La Cosa Nostra
Lines: 50
Sender: tech-kern-owner@NetBSD.org
Message-ID: <fa./m6DKxI71Cy6h4EiQTf2YA26U3s@ifi.uio.no>
References: <fa.e83HgwaM2kaCuuUpS6tVoQ1+1Kk@ifi.uio.no>
NNTP-Posting-Host: mail-jess.uio.no
Mime-Version: 1.0
X-Trace: readme.uio.no 1366495984 16437 129.240.7.9 (20 Apr 2013 22:13:04 GMT)
X-Complaints-To: abuse@uio.no
NNTP-Posting-Date: Sat, 20 Apr 2013 22:13:04 +0000 (UTC)
User-Agent: Mozilla/5.0 (X11; NetBSD amd64; rv:17.0) Gecko/20130413 Thunderbird/17.0.5
Cc: tech-kern@NetBSD.org, tech-pkg@NetBSD.org
To: iMil <imil@home.imil.net>
Delivered-To: tech-kern@netbsd.org
X-Virus-Scanned: amavisd-new at NetBSD.org
Authentication-Results: mail.NetBSD.org (amavisd-new);
 dkim=pass (2048-bit key) header.d=gmail.com
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=x-received:message-id:date:from:organization:user-agent
         :mime-version:to:cc:subject:references:in-reply-to
         :x-enigmail-version:openpgp:content-type:content-transfer-encoding;
        bh=nP3dNBYqbjUcCyusuaCcO/yORZHFNLynX2dpit+IwFw=;
        b=fAcF41Nd1cw4cHq1fEb7SR/galB0/L/oV2frhtORhIX8JIlW+91TzudWSCUor1bkq5
         PMIaw/ePNFepUcLzVjLqM1EyAi9HZ8uumz013Ze9o1ZbhH0NwEdYzk2hztBg19Fs/iZt
         DhquFgdXak5Ts49bP/gfveJa5pS3PDQIs1Uun+5WdWa9cHGcDceYE9Y4zjpJYez4UHxs
         xucaAHLVY0/BHTBTS2guky+z/G3CV5aNJkJwmNhDXEgYn6oSZer3qn8yWsjxDP0ohPsi
         iK0lCr9YAYT6FB9nh6sowBUkZ5SDoVIt+IGbnlkDLVxvDqIFf2mMNPGmsYOUTvp1uoPQ
         IhaQ==
X-Received: by 10.112.129.170 with SMTP id nx10mr10509911lbb.81.1366495958624;
        Sat, 20 Apr 2013 15:12:38 -0700 (PDT)
In-Reply-To: <alpine.NEB.2.00.1304202332220.19430@korriban.imil.net>
X-Enigmail-Version: 1.5a1pre
OpenPGP: id=13D7A8ED
List-Id: tech-kern.NetBSD.org
Original-Date: Sun, 21 Apr 2013 00:12:35 +0200
Original-Message-Id: <517312D3.8070001@gmail.com>
Original-References: <alpine.NEB.2.00.1304202332220.19430@korriban.imil.net>
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: 7bit

Reply to Parent Message