🚀 go-pugleaf

Hello Hans, Remy and Michael !
I will try to merge the answers:

[HANS]
"How does what Notepad shows differ from what SourceCodeHTML() gives?"
Nothing at really !
But that's the problem... the HTML source shows the javascript functions 
that loads dynamically some data in the webpage, but I need to parse the 
data, not the code that generates it.
i.e:
if you view this page:
http://articulo.mercadolibre.com.ar/MLA-25012974-notebook-hp-nx-6325-con-micro-sempron-3400-y-gtia-real-_JM
you will see something like:
"Finaliza en: 42d 3h  (26/01/2007 21:03)"
The date refresh loads dynamically, therefore is not in the sourcecode:
//-----------------
<td valign=top width=106 class=bl2n12><img src=/org-img/t.gif width=5 
height=1>Finaliza en:</td>
<td valign=top width=264 class=bl2n12><span id="TimeLeft" 
style="position:relative;width=170;height=16">
</span>
<ILAYER id=ns4TimeLeft height=15 width=180>
</ILAYER>
<nolayer>
<iframe id=iframeTimeLeft scrolling=no width=0 height=0 framespacing=0 
frameborder=0>
</iframe>
</nolayer></td>
//-----------

Internet Explorer shows the same HTML source that my code, and it is not 
usefull to me, I need to see in the code the same I see in my screen...
But in Firefox...
If you select the price and right click to "view source of selection", in 
Firefox, the resulting HTML code is revealed, so clearly once the
javascript has been executed, the browser has the HTML source of the page 
somewhere so it can render the page in the browser
//---------------
<tr><td class="bl2n12" valign="top" width="106"><img src="/org-img/t.gif" 
height="1" width="5">Finaliza en:</td>
<td class="bl2n12" valign="top" width="264"><span id="TimeLeft" 
style="position: relative;">42d 3h&nbsp;&nbsp;(26/01/2007 21:03)</span>
<ilayer id="ns4TimeLeft" height="15" width="180">
</ilayer>
<nolayer>
<iframe 
src="http://articulo.mercadolibre.com.ar/jm/item?act=timeleft&amp;site=MLA&amp;id=25012974" 
id="iframeTimeLeft" framespacing="0" frameborder="0" height="0" 
scrolling="no" width="0">
</iframe>
</nolayer></td></tr>
//---------------


[REMY]
"You don't need to copy to the clipboard to get the selected text.  You can
query it from the DOM directly."

Can you please send me an example code to do that? I can't see it...
Iam doing something like this:
(commented to help you to understand it)
//------------------------------------------
      MyCppWebBrowser->NavegateSync(Links[i], 10, /*images*/false, 
/*javascript*/true, /*cookies*/false, /*cache*/false);  //this method waits 
to download all the page

     MyCppWebBrowser->SelectAll();  //this method Select all the content
     AnsiString Texto = MyCppWebBrowser->CopiarSeleccion();  //copy the 
selection and assign it to an Ansistring
     MyCppWebBrowser->DesSeleccionar();  //Unselect all...

     Datos = Parse(CodigoHTML, Texto, Links[i]);  //parse the data...

      int cont = 0;
      while(!ChequearDatosCompletos(Datos) && cont < 3)  //checks if the 
dynamic data is loaded, if not... 3 tries maximun
      {
        EsperarXTiempo(2);  //wait 2 seconds...
        CodigoHTML = MyCppWebBrowser->CodigoFuenteHTML();  //Get the HTML 
source again
        MyCppWebBrowser->SeleccionarTodo(); //select all again
        Texto = MyCppWebBrowser->CopiarSeleccion(); //copy selection again
        MyCppWebBrowser->DesSeleccionar(); //unselect all again...
        Datos = Parse(CodigoHTML, Texto, Links[i]); //parse the data 
again...
        cont++;
      }
//------------------------------------------

And the above methods are defined:
SELECT ALL:
//--------------------------------------------------------------------------
void TInternetExplorer::SeleccionarTodo(TCppWebBrowser* pCppWebBrowser)
{
 pCppWebBrowser->ExecWB(Shdocvw_tlb::OLECMDID_SELECTALL, 
Shdocvw_tlb::OLECMDEXECOPT_DODEFAULT, NULL, NULL);
}
//--------------------------------------------------------------------------

UNSELECT ALL:
//--------------------------------------------------------------------------
void TInternetExplorer::DesSeleccionar(TCppWebBrowser* pCppWebBrowser)
{

 //pCppWebBrowser->ExecWB(/*Shdocvw_tlb::*/OLECMDID_UNSELECT, 
Shdocvw_tlb::OLECMDEXECOPT_DODEFAULT, NULL, NULL);

 IHTMLDocument2 *HTMLDoc = NULL;
 if(SUCCEEDED(pCppWebBrowser->Document->QueryInterface(IID_IHTMLDocument2, 
(LPVOID*)&HTMLDoc)))
 {
    VARIANT_BOOL ret;
    TOLEBOOL showUI = false;
    //HTMLDoc->execCommand(WideString("SelectAll"), showUI, 0, &ret);
    VARIANT Flag;
    Flag.vt = VT_I4;
    Flag.lVal = 0;    // noNavHistory
    HTMLDoc->execCommand(WideString("Unselect"), showUI, Flag, &ret);
    HTMLDoc->Release();
 }

}
//--------------------------------------------------------------------------

COPY SELECTION
//--------------------------------------------------------------------------
AnsiString TInternetExplorer::CopiarSeleccion(TCppWebBrowser* 
pCppWebBrowser)
{
 AnsiString Antes = Clipboard()->AsText;
 pCppWebBrowser->ExecWB(Shdocvw_tlb::OLECMDID_COPY, 
Shdocvw_tlb::OLECMDEXECOPT_DODEFAULT, NULL, NULL);
 AnsiString Ahora = Clipboard()->AsText;
 Clipboard()->AsText = Antes;
 return Ahora;
}
//--------------------------------------------------------------------------

NAVIGATESYNC  (commented to help you understand it)
//---------------------------------------------------------------------------
bool TMyCppWebBrowser::NavegateSync(AnsiString URL, int Tiempo/*Time 
inactive, cancel the Navigate*/, bool Imagenes/*Download Images?*/, bool 
JavaScript/*JavaScriptActive?*/, bool BorrarCookies/*Delete Cookies?*/, bool 
BorrarCache/*Delete Cache?*/)
{
  ModificarMaxTiempoInactivo(Tiempo);     //modify the max time inactive 
(int TIEMPOINACTIVOMAXIMO;)
  AsginarEventosVisitas();                            //Assign my methods to 
the events...
  if(Imagenes) {    IE.HabilitarImagenes();  } //Enable Images?
    else  {    IE.DeshabilitarImagenes();  }
  if(JavaScript)  {    IE.HabilitarJavascript();  } //Enable Javascript?
    else  {    IE.DeshabilitarJavascript();  }
  if(BorrarCookies)  { IE.DeleteNewCookies(); }  //delete cookies?
  if(BorrarCache)  {    IE.DeleteCache();  }  //delete cache?

  Timer->Enabled = false;
  Timer->Interval = 1000;
  Timer->OnTimer = OnTimer;
  TiempoInactivo = 0;           //InactiveTime
  HTMLCompleto = false;     //HTMLCompleted (checks that the webpage is 
completely loaded)
  TMyCppWebBrowser::STOP_ASYNC = false;     //STOP_ASYNC is an static 
member -> static bool STOP_ASYNC; to let me Stop at any time...
  CppWebBrowser->Navigate(WideString(URL));
  Timer->Enabled  = true;
  while(!HTMLCompleto && TiempoInactivo <= TIEMPOINACTIVOMAXIMO && 
!TMyCppWebBrowser::STOP_ASYNC)
  {
    Application->ProcessMessages();
    Sleep(10);
  }
  if(TMyCppWebBrowser::STOP_ASYNC)  //if the while finishes because 
STOP_ASYNC is true.
  {
   CppWebBrowser->Stop();
  }
  Timer->Enabled = false;
  return !TMyCppWebBrowser::STOP_ASYNC;
}
//---------------------------------------------------------------------------


MY EVENTS:
// if some of these events is called, then the navigation it's not inactive:
//---------------------------------------------------------------------------
void __fastcall TMyCppWebBrowser::OnDownloadBegin(TObject *Sender)
{
 TiempoInactivo = 0;
}
//---------------------------------------------------------------------------
void __fastcall TMyCppWebBrowser::OnDownloadComplete(TObject *Sender)
{
 TiempoInactivo = 0;
}
//---------------------------------------------------------------------------
void __fastcall TMyCppWebBrowser::OnNavigateComplete2(TObject *Sender, 
LPDISPATCH pDisp, TVariant *URL)
{
 TiempoInactivo = 0;
}
//---------------------------------------------------------------------------
void __fastcall TMyCppWebBrowser::OnProgressChange(TObject *Sender,long 
Progress, long ProgressMax)
{
 TiempoInactivo = 0;
}
//---------------------------------------------------------------------------
void __fastcall TMyCppWebBrowser::OnTitleChange(TObject *Sender,BSTR URL)
{
 TiempoInactivo = 0;
}
//---------------------------------------------------------------------------

//Counts the time that TCppWebBrowser has been without download anything...
//---------------------------------------------------------------------------
void __fastcall TMyCppWebBrowser::OnTimer(TObject *Sender)
{
 TiempoInactivo++;
}
//---------------------------------------------------------------------------

Sometimes OnDocumentComplete executes many times, but the main code it's not 
downloaded yet. We check that...
//---------------------------------------------------------------------------
void __fastcall TMyCppWebBrowser::OnDocumentComplete(TObject *Sender, 
LPDISPATCH pDisp, TVariant *URL)
{
 TiempoInactivo = 0;
 AnsiString HTML = CodigoFuenteHTML();  //SourceHTML()
 HTML = HTML.LowerCase();
 if(HTML.Pos("</html>"))
 {
   HTMLCompleto = true;   //HTML completed
 }
}
//---------------------------------------------------------------------------

I've asked this in other group, but it was a Delphi group and although they 
posted a solution, I can't translate it to Cpp...
Part of tthat conversation:
//---------------------------------- 
According to MS-SDK:
Code within the SCRIPT block that is not contained within a
function is executed immediately as the page is loaded.

Of course, such code can call other functions in the page
resulting in dynamic DOM modifications. Unless there is
a script that constantly changes the DOM, at some point
the document will be ented a READY state and the document
complete event will fire.

So, if you try to parse after the documentComplete event you
should have no problem. Although I show the behaviour you described
(I am on a slow dial-up and changes occur slowly ;-) at the end
I am getting the complete DOM tree.
(...)
But if I view the source
I get what Aaron said. The source is what was downloaded. The DOM is
what produced the page you see in the browser.

are you sure you start parsing after documentComplete? Are you
sure you are using the webBrowser.Document ?

fotis
//----------------------------------
more:
//-----------------------------
sorry, I am not quite happy with C++ :-(
The general guidelines are:

1. _use_ navigate method of webBrowser to get the page you want.
2. wait until the document is completly downloaded. That means
    that the only safe point to put your hands on the DOM is in the
    event handler of the DocumentComplete event of the WebBrowser.
   At this point the document in completly downloaded, the DOM
   tree has been created and the page has been rendered in the
   web browser window.
   Use WebBrowser.Document to get a ref to the document
   (is it WebBrowser->Document in C++ ??)

3. start parsing the dom tree starting from it's root node with is
   the node that coresponds to the <HTML> tag. use

IHTMLDocument3::documentElement Property

to retrieve a reference to the root node of the document.

then it's a typical tree parsing (a loop to parse the children
and a recursive call to go deeper).

Man! that C++ thing is a nightmare.....  ;-)

good luck
//--------------------------------------------

But I'm not very lucky... ^_^
I know is possible, but I really don't know how to do it, if somebody can 
translate the above steps to C++Builder code I think we are ready...

greets !
enrique.