🚀 go-pugleaf

RetroBBS NetNews Server

Inspired by RockSolid Light RIP Retro Guy

Article View: borland.public.cppbuilder.database.ado
Article #359

fastest way to add many records in batch

#359
From: crhras
Date: Mon, 04 Sep 2006 15:38
29 lines
1901 bytes
I am adding 100,000 rows to an initialized MySQL database using the code 
below.  The application gets much slower as the number of records grows. 
This is an obvious result of the database itself having more work to do with 
indexes, etc. but is there also something that I can do on the application 
side to speed things up ?   I am wondering if the ClientDataSet, Provider, 
etc. are also getting big and have more work to do.  If so, what is the best 
way to avoid this ?

Thank you.
Curt

I am using the following code to add the records:

ADOConnection1->Open();
DM->ClientDataSet1->Open();

for (int i = 0; i < equities; i++) {
  for (int j = 0; j < days; j++) {
    DM->ClientDataSet1->Insert();
    ...   // table fields are loaded here
    DM->ClientDataSet1->Post();
   }
  }
  if (i%10 == 0) {  // apply updates after 2000 rows are cached
   DM->ClientDataSet1->ApplyUpdates(0);
  }
}
DM->ClientDataSet1->ApplyUpdates(0);
DM->ADOConnection1->Close();
Technical Details
Message-ID: <44fc486d$1@newsgroups.borland.com>
Path: arc1-ca.pugleaf.net!g2news2.google.com!news3.google.com!news4.google.com!border1.nntp.dca.giganews.com!border2.nntp.dca.giganews.com!nntp.giganews.com!newspeer1.nwr.nac.net!newsfeeds.ihug.co.nz!ihug.co.nz!newsgroups.borland.com!not-for-mail

Show Full Headers
Complete Article Headers
X-Google-Thread: 10698b,d6a5998fcdb3776e,start
X-Google-Attributes: gid10698b,public
X-Google-Language: ENGLISH,ASCII-7-bit
From: "crhras" <crhras@sbcglobal.net>
Newsgroups: borland.public.cppbuilder.database.ado
Subject: fastest way to add many records in batch
Date: Mon, 4 Sep 2006 10:38:20 -0500
Lines: 31
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2900.2869
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962
X-RFC2646: Format=Flowed; Original
NNTP-Posting-Host: 75.34.172.168
Message-ID: <44fc486d$1@newsgroups.borland.com>
X-Trace: newsgroups.borland.com 1157384301 75.34.172.168 (4 Sep 2006 08:38:21 -0700)
Path: arc1-ca.pugleaf.net!g2news2.google.com!news3.google.com!news4.google.com!border1.nntp.dca.giganews.com!border2.nntp.dca.giganews.com!nntp.giganews.com!newspeer1.nwr.nac.net!newsfeeds.ihug.co.nz!ihug.co.nz!newsgroups.borland.com!not-for-mail