🚀 go-pugleaf

Hi,

I am writing an application reading big text files (hundreds of 
megabytes) line by line. I was reading the files using std::getline and 
I noticed that it is *incredibly* slow. So I wrote three test programs, 
one using fgets(), one using std::getline() and one using 
std::ifstream::getline(). I have read the same file (200000 lines, 39MB) 
in all cases; here are the results:

Using fgets():

aisotton@zarathustra:~/test/iotest$ time ./stdio

real    0m0.222s
user    0m0.109s
sys     0m0.073s

------------

Using std::getline():

aisotton@zarathustra:~/test/iotest$ time ./streams

real    0m6.945s
user    0m5.316s
sys     0m0.074s

------------

Using std::ifstream::getline():

aisotton@zarathustra:~/test/iotest$ time ./streams2

real    0m1.002s
user    0m0.890s
sys     0m0.074s

------------

As you can see, fgets() is more than 30 times faster than 
std::getline(), and more than 4 times faster than std::ifstream::getline().

I am not 100% sure, but I more and more believe that it must be an 
inefficient stream implementation in g++; I could at least rule out a 
few points.

- It is not a buffering problem; using strace I found out that fgets() 
uses a 4096 byte buffer, and the C++ streams a 8192 byte buffer.

- It has nothing to do with optimization. The values are roughly the 
same for all optimization levels.

- g++ 2.95 performs better than 3.2, and 3.2 better than 3.3, but they 
are still slow. Here the exact versions:

aisotton@zarathustra:~/test/iotest$ g++-2.95 -v
Reading specs from /usr/lib/gcc-lib/i386-linux/2.95.4/specs
gcc version 2.95.4 20011002 (Debian prerelease)

aisotton@zarathustra:~/test/iotest$ g++-3.2 -v
Reading specs from /usr/lib/gcc-lib/i386-linux/3.2.3/specs
Configured with: ../src/configure -v 
--enable-languages=c,c++,f77,objc,ada --prefix=/usr 
--mandir=/usr/share/man --infodir=/usr/share/info 
--with-gxx-include-dir=/usr/include/c++/3.2 --enable-shared 
--with-system-zlib --enable-nls --without-included-gettext 
--enable-__cxa_atexit --enable-clocale=gnu --enable-objc-gc i386-linux
Thread model: posix
gcc version 3.2.3 (Debian)

aisotton@zarathustra:~/test/iotest$ g++-3.3 -v
Reading specs from /usr/lib/gcc-lib/i486-linux/3.3.3/specs
Configured with: ../src/configure -v 
--enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr 
--mandir=/usr/share/man --infodir=/usr/share/info 
--with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared 
--with-system-zlib --enable-nls --without-included-gettext 
--enable-__cxa_atexit --enable-clocale=gnu --enable-debug 
--enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc i486-linux
Thread model: posix
gcc version 3.3.3 (Debian 20040401)

And here the times for the second test:
-- 2.95 --
real    0m4.345s
user    0m4.058s
sys     0m0.067s

-- 3.2 --
real    0m5.728s
user    0m5.311s
sys     0m0.082s

-- 3.3 --
real    0m6.586s
user    0m5.298s
sys     0m0.091s

- It has nothing to do with stdio synchronisation, since we're reading 
from files. To be really sure, I tried using

std::ios_base::sync_with_stdio(false);

(yes, before the first call!) but that didn't make any difference.

- Increasing the buffer size of the stream didn't make any real 
difference either.

Almost two years ago there was a long discussion about this on

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=gidnla.uh5.ln%40ns1.irule.be&rnum=1&prev=/groups%3Fq%3Dg%252B%252B%2Bslow%2Bstd::getline%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3Dgidnla.uh5.ln%2540ns1.irule.be%26rnum%3D1

but they didn't really come to a solution.

So, my question: am I doing something wrong or is this a bug in g++?

I've uploaded a very small "test suite" demonstrating the strange 
behaviour to http://www.isotton.com/sandbox/iotest.tar.gz ; it would 
really help me if you could try it with other compilers, too. To use it 
just unpack it and type "make test". Notice that it will create a file 
of about 40MB.

Thanks for your time,
Aaron