Article View: borland.public.cppbuilder.internet.web
Article #167Re: Extracting all the HTMLElements from IHTMLDocument2
From: Van
Date: Mon, 24 Sep 2007 16:39
Date: Mon, 24 Sep 2007 16:39
132 lines
5085 bytes
5085 bytes
Dear gurus,
My name is Van and I have just been assigned an abnormal task. That is, to fix the bugs
in a project written by another former programmer. I discovered the reason
of the bugs as following.
The software specifications require us to find all links on a webpage.
The old guy extracts all frames inside this webpage and then find all
links in each frame. However, his component fails to deal with the webpages
using IFRAME tags. The code even returns NULL for those frames before we can
looks inside the HTML documents generated by those IFRAMES and grab
any link if there.
I have to plans:
1> Short-term solution: I will extract all frames rather than IFRAMEs
from the webpage. For each frame, i re-use the existing code to get the
links.
My question is how can i filter out all the IFRAMEs in the webpage?
2> Long-term solution: Once we get the prototype running by the mentioned
solution, we have to complete the software. That is to retrieve all
links in the webpage, even if they are generated dynamically.
Please find bellows the code to extract frame and links
inside an webpage:
***************************************************************************
// This code is supposed to return a frame given its index in a
webpage //
It fails when we see the IFRAME tag in for example www.cnn.com
// Question: How can we filter out the result so we will get the
frame //
that is not an IFRAME?
IHTMLDocument2* GetFrame(TCppWebBrowser *bp, int idx) {
IHTMLDocument2 *HTMLDoc = NULL, *res=NULL;
long collSize=0;
// van's note
if(!bp->Busy && bp->Document &&
AnsiString(bp->LocationURL) != "about:blank" &&
SUCCEEDED(bp->Document->QueryInterface(
IID_IHTMLDocument2, (LPVOID*)&HTMLDoc)))
{
IHTMLFramesCollection2 *pColl = NULL;
if(SUCCEEDED(HTMLDoc->get_frames(&pColl))) {
pColl->get_length(&collSize);
// van's note
assert(collSize>0); // passed with CNN.com
if (collSize>idx) {
VARIANT one;
one.vt = VT_UINT;
one.lVal = idx;
VARIANT out;
pColl->item(&one,&out);
//
IHTMLWindow2 *win=NULL;
if (SUCCEEDED(out.pdispVal->QueryInterface( IID_IHTMLWindow2,
(LPVOID*)&win ))) {
// van's note
assert(win); // passed with CNN.com
if (!SUCCEEDED(win->get_document(&res))) {
res=NULL;
}
win->Release();
} else {
res=NULL;
}
} // end if (collSize>idx)
pColl->Release();
}
}
if (HTMLDoc) HTMLDoc->Release();
return res;
}
***************************************************************************
***************************************************************************
// Comment: This code to return a link given its index inside a
frame //
So the caller will get a frame using the above code and get the //
link
inside this frame, given its index
// Question: If we get an IFRAME, how can we get the links generated
by //
the IFRAME script?
IHTMLAnchorElement* GetLink(IHTMLDocument2 *HTMLDoc, int idx) {
IHTMLAnchorElement* res=NULL;
IHTMLElementCollection *pColl = NULL;
if(SUCCEEDED(HTMLDoc->get_links(&pColl))) {
long collSize;
pColl->get_length(&collSize);
IDispatch* id;
VARIANT one;
if (idx<collSize) {
one.vt = VT_UINT;
one.lVal = idx;
VARIANT two = {0};
pColl->item(one,two,&id);
IHTMLAnchorElement* pElem = NULL;
if (SUCCEEDED(id->QueryInterface( IID_IHTMLAnchorElement,
(LPVOID*)&pElem ))) {
res=pElem;
}
}
pColl->Release();
}
return res;
}
***************************************************************************
From my POV, this code
will always assign res = NULL when try to get an HTMLDocument2 from an
IFRAME:
// From the code GetFrame
if (SUCCEEDED(out.pdispVal->QueryInterface( IID_IHTMLWindow2,
(LPVOID*)&win ))) {
.............................................................
// Comment: Failed if an out is an IFRAME
if (!SUCCEEDED(win->get_document(&res))) {
res=NULL;
}
}
So sorry for this lengthy mail and thank you very much for any advice
Technical Details
Message-ID:
<46f7e838$1@newsgroups.borland.com>Path:
arc1-ca.pugleaf.net!g2news2.google.com!news1.google.com!news.glorb.com!newsfeeds.ihug.co.nz!ihug.co.nz!newsgroups.borland.com!not-for-mailReferences:
<46c061ea$1@newsgroups.borland.com> <46c09536@newsgroups.borland.com>Show Full Headers
Complete Article Headers
X-Google-Thread: feeec,26c174002c4041d7 X-Google-Attributes: gidfeeec,public,usenet X-Google-Language: ENGLISH,ASCII-7-bit From: "Van" <suskovivor@yahoo.com> Sender: "Van" <suskovivor@yahoo.com> Reply-To: "Van" <suskovivor@yahoo.com> Subject: Re: Extracting all the HTMLElements from IHTMLDocument2 Newsgroups: borland.public.cppbuilder.internet.web X-User-Info: 192.38.86.70 References: <46c061ea$1@newsgroups.borland.com> <46c09536@newsgroups.borland.com> NNTP-Posting-Host: 207.105.83.66 Message-ID: <46f7e838$1@newsgroups.borland.com> Date: 24 Sep 2007 09:39:20 -0700 X-Trace: newsgroups.borland.com 1190651960 207.105.83.66 (24 Sep 2007 09:39:20 -0700) Lines: 132 Path: arc1-ca.pugleaf.net!g2news2.google.com!news1.google.com!news.glorb.com!newsfeeds.ihug.co.nz!ihug.co.nz!newsgroups.borland.com!not-for-mail