Article View: borland.public.cppbuilder.database.ado
Article #359fastest way to add many records in batch
From: crhras
Date: Mon, 04 Sep 2006 15:38
Date: Mon, 04 Sep 2006 15:38
29 lines
1901 bytes
1901 bytes
I am adding 100,000 rows to an initialized MySQL database using the code
below. The application gets much slower as the number of records grows.
This is an obvious result of the database itself having more work to do with
indexes, etc. but is there also something that I can do on the application
side to speed things up ? I am wondering if the ClientDataSet, Provider,
etc. are also getting big and have more work to do. If so, what is the best
way to avoid this ?
Thank you.
Curt
I am using the following code to add the records:
ADOConnection1->Open();
DM->ClientDataSet1->Open();
for (int i = 0; i < equities; i++) {
for (int j = 0; j < days; j++) {
DM->ClientDataSet1->Insert();
... // table fields are loaded here
DM->ClientDataSet1->Post();
}
}
if (i%10 == 0) { // apply updates after 2000 rows are cached
DM->ClientDataSet1->ApplyUpdates(0);
}
}
DM->ClientDataSet1->ApplyUpdates(0);
DM->ADOConnection1->Close();
Technical Details
Message-ID:
<44fc486d$1@newsgroups.borland.com>Path:
arc1-ca.pugleaf.net!g2news2.google.com!news3.google.com!news4.google.com!border1.nntp.dca.giganews.com!border2.nntp.dca.giganews.com!nntp.giganews.com!newspeer1.nwr.nac.net!newsfeeds.ihug.co.nz!ihug.co.nz!newsgroups.borland.com!not-for-mailShow Full Headers
Complete Article Headers
X-Google-Thread: 10698b,d6a5998fcdb3776e,start X-Google-Attributes: gid10698b,public X-Google-Language: ENGLISH,ASCII-7-bit From: "crhras" <crhras@sbcglobal.net> Newsgroups: borland.public.cppbuilder.database.ado Subject: fastest way to add many records in batch Date: Mon, 4 Sep 2006 10:38:20 -0500 Lines: 31 X-Priority: 3 X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2900.2869 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962 X-RFC2646: Format=Flowed; Original NNTP-Posting-Host: 75.34.172.168 Message-ID: <44fc486d$1@newsgroups.borland.com> X-Trace: newsgroups.borland.com 1157384301 75.34.172.168 (4 Sep 2006 08:38:21 -0700) Path: arc1-ca.pugleaf.net!g2news2.google.com!news3.google.com!news4.google.com!border1.nntp.dca.giganews.com!border2.nntp.dca.giganews.com!nntp.giganews.com!newspeer1.nwr.nac.net!newsfeeds.ihug.co.nz!ihug.co.nz!newsgroups.borland.com!not-for-mail