🚀 go-pugleaf

RetroBBS NetNews Server

Inspired by RockSolid Light RIP Retro Guy

Article View: borland.public.cppbuilder.internet.web
Article #374

How to get the "real live" HTML source?

#374
From: Enrique
Date: Fri, 15 Dec 2006 16:07
64 lines
3308 bytes
Hello !
I'm having problems to obtain the HTML source from webpages that refresh its 
content dynamically by javascript functions.
in example:
http://articulo.mercadolibre.com.ar/MLA-25012974-notebook-hp-nx-6325-con-micro-sempron-3400-y-gtia-real-_JM

If I use this code (which works very well most of the times):
//----------------------------------
AnsiString TInternetExplorer::SourceCodeHTML(TCppWebBrowser* CppWebBrowser1)
{
  AnsiString Source = "";

  IHTMLDocument2 *htm = NULL;     // #include <mshtml.h>


if(CppWebBrowser1->Document&&SUCCEEDED(CppWebBrowser1->Document->QueryInterface(IID_IHTMLDocument2,
(LPVOID*)&htm)))
    {
    IPersistStreamInit *spPsi = NULL;    // ocidl.h


if(SUCCEEDED(htm->QueryInterface(IID_IPersistStreamInit,(LPVOID*)&spPsi)) && 
spPsi)
       {
            IStream *spStream = NULL;            // objidl.h
            OleCheck(CreateStreamOnHGlobal(NULL, true, &spStream));
            if(spStream)
            {
                __int64 nSize = 0;
                STATSTG ss;
                LARGE_INTEGER nMove;
                nMove.QuadPart = 0;
                OleCheck(spPsi->Save(spStream, true));
                OleCheck(spStream->Seek(nMove,STREAM_SEEK_SET,(ULARGE_INTEGER 
*)&nSize));
                OleCheck(spStream->Stat(&ss, STATFLAG_NONAME));
                nSize = ss.cbSize.QuadPart;

                Source.SetLength(nSize);
                OleCheck(spStream->Read((void *)Source.data(), 
nSize,(ULONG*)&nSize));
                OleCheck(spStream->Release());
            }

            spPsi->Release();
        }
    htm->Release();
    }
   return Source;
}
//------------------------------------------------------------------------

I can't find the data that I'm seeing in my screen.
My solution (very inefficient) was to wait for the page loads completly, 
then select all, copy to clipboard and then "paste" it to an AnsiString... 
it works, but it has many problems, for example, I loss the content of the 
clipboard, it's very imprecise, requieres a lot of extra code and resources, 
etc.
Can somebody post a better solution? (if it's possible... I think yes, that 
is, if the TCppWebBrowser can show the content then, the content is in 
somewhere...)

Thanks !
enrique. 
Technical Details
Message-ID: <4582e657@newsgroups.borland.com>
Path: arc1-ca.pugleaf.net!g2news2.google.com!news4.google.com!news2.volia.net!hwmnpeer01.ams!news.highwinds-media.com!feed10.multikabel.net!multikabel.net!feed20.multikabel.net!newsfeed.freenet.de!newsfeed00.sul.t-online.de!newsfeed01.sul.t-online.de!t-online.de!newsgroups.borland.com!not-for-mail

Show Full Headers
Complete Article Headers
X-Google-Thread: feeec,b54437b3b496861e,start
X-Google-Attributes: gidfeeec,public
X-Google-Language: ENGLISH,ASCII-7-bit
Reply-To: "Enrique" <enridp@yahoo.com.ar>
From: "Enrique" <enridp@yahoo.com.ar>
Newsgroups: borland.public.cppbuilder.internet.web
Subject: How to get the "real live" HTML source?
Date: Fri, 15 Dec 2006 13:07:20 -0300
Lines: 66
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2900.2869
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962
X-RFC2646: Format=Flowed; Original
NNTP-Posting-Host: 200.45.197.46
Message-ID: <4582e657@newsgroups.borland.com>
X-Trace: newsgroups.borland.com 1166206551 200.45.197.46 (15 Dec 2006 10:15:51 -0800)
Path: arc1-ca.pugleaf.net!g2news2.google.com!news4.google.com!news2.volia.net!hwmnpeer01.ams!news.highwinds-media.com!feed10.multikabel.net!multikabel.net!feed20.multikabel.net!newsfeed.freenet.de!newsfeed00.sul.t-online.de!newsfeed01.sul.t-online.de!t-online.de!newsgroups.borland.com!not-for-mail