🚀 go-pugleaf

RetroBBS NetNews Server

Inspired by RockSolid Light RIP Retro Guy

Article View: borland.public.cppbuilder.internet.web
Article #164

How to filter out IFRAME in a webpage/extract a links generated by this iframe

#164
From: suskovivor
Date: Fri, 21 Sep 2007 16:04
137 lines
5421 bytes
Dear gurus,
My name is Van and i am a programmer in the Denmark.
I have just been assigned an abnormal task. That is, to fix the bugs
in a
project written by another former programmer. I discovered the reason
of the bugs as
following.

The software specifications require us to find all links on a webpage.
The
old guy extracts all frames inside this webpage and then find all
links in
each frame. However, his component fails to deal with the webpages
using
IFRAME tags. The code even returns NULL for those frames before we can
looks inside the HTML documents generated by those IFRAMES and grab
any
link if there.

I have to plans:
1> Short-term solution: I will extract all frames rather than IFRAMEs
from
the webpage. For each frame, i re-use the existing code to get the
links.
My question is how can i filter out all the IFRAMEs in the webpage? 2>
Long-term solution: Once we get the prototype running by the mentioned
solution, we have to complete the software. That is to retrieve all
links
in the webpage, even if they are generated dynamically.
Please find bellows the code to extract frame and links
inside an webpage:

***************************************************************************
// This code is supposed to return a frame given its index in a
webpage //
It fails when we see the IFRAME tag in for example www.cnn.com
// Question: How can we filter out the result so we will get the
frame //
that is not an IFRAME?

IHTMLDocument2* GetFrame(TCppWebBrowser *bp, int idx) {
  IHTMLDocument2 *HTMLDoc = NULL, *res=NULL;

  long collSize=0;

  // van's note
  if(!bp->Busy && bp->Document &&
        AnsiString(bp->LocationURL) != "about:blank" &&
        SUCCEEDED(bp->Document->QueryInterface(
        IID_IHTMLDocument2, (LPVOID*)&HTMLDoc)))
{
    IHTMLFramesCollection2 *pColl = NULL;

    if(SUCCEEDED(HTMLDoc->get_frames(&pColl))) {
      pColl->get_length(&collSize);
      // van's note
      assert(collSize>0); // passed with CNN.com

      if (collSize>idx) {
        VARIANT one;
        one.vt = VT_UINT;
        one.lVal = idx;
        VARIANT out;
        pColl->item(&one,&out);
        //
        IHTMLWindow2 *win=NULL;
        if (SUCCEEDED(out.pdispVal->QueryInterface( IID_IHTMLWindow2,
(LPVOID*)&win ))) {
          // van's note
          assert(win);   // passed with CNN.com

          if (!SUCCEEDED(win->get_document(&res))) {
            res=NULL;
          }
          win->Release();
        } else {
          res=NULL;
        }
      } // end if (collSize>idx)

      pColl->Release();
    }
  }
  if (HTMLDoc) HTMLDoc->Release();
  return res;
}
***************************************************************************

***************************************************************************
// Comment: This code to return a link given its index inside a
frame //
So the caller will get a frame using the above code and get the //
link
inside this frame, given its index
// Question: If we get an IFRAME, how can we get the links generated
by //
the IFRAME script?

IHTMLAnchorElement* GetLink(IHTMLDocument2 *HTMLDoc, int idx) {
  IHTMLAnchorElement* res=NULL;
  IHTMLElementCollection *pColl = NULL;

  if(SUCCEEDED(HTMLDoc->get_links(&pColl))) {
    long collSize;
    pColl->get_length(&collSize);
    IDispatch* id;
    VARIANT one;
    if (idx<collSize) {
      one.vt = VT_UINT;
      one.lVal = idx;
      VARIANT two = {0};
      pColl->item(one,two,&id);
      IHTMLAnchorElement* pElem = NULL;
      if (SUCCEEDED(id->QueryInterface( IID_IHTMLAnchorElement,
(LPVOID*)&pElem ))) {
        res=pElem;
      }
    }
    pColl->Release();
  }
   return res;
}
***************************************************************************
 From my POV, this code
will always assign res = NULL when try to get an HTMLDocument2 from an
IFRAME:
// From the code GetFrame
if (SUCCEEDED(out.pdispVal->QueryInterface( IID_IHTMLWindow2,
(LPVOID*)&win ))) {
.............................................................
// Comment: Failed if an out is an IFRAME
if (!SUCCEEDED(win->get_document(&res))) {
            res=NULL;
}
}

So sorry for this lengthy mail and thank you very much for any advice
Technical Details
Message-ID: <1190390698.606532.44040@19g2000hsx.googlegroups.com>
Path: arc1-ca.pugleaf.net!g2news2.google.com!postnews.google.com!19g2000hsx.googlegroups.com!not-for-mail

Show Full Headers
Complete Article Headers
X-Google-Thread: feeec,6680eff7f31fcbe5
X-Google-Attributes: gidfeeec,public,usenet
X-Google-Language: ENGLISH,ASCII-7-bit
Path: arc1-ca.pugleaf.net!g2news2.google.com!postnews.google.com!19g2000hsx.googlegroups.com!not-for-mail
From:  suskovivor@yahoo.com
Newsgroups: borland.public.cppbuilder.internet.web
Subject: How to filter out IFRAME in a webpage/extract a links generated by this iframe
Date: Fri, 21 Sep 2007 09:04:58 -0700
Organization: http://groups.google.com
Lines: 138
Message-ID: <1190390698.606532.44040@19g2000hsx.googlegroups.com>
NNTP-Posting-Host: 130.226.188.2
Mime-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
X-Trace: posting.google.com 1190390698 7714 127.0.0.1 (21 Sep 2007 16:04:58 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Fri, 21 Sep 2007 16:04:58 +0000 (UTC)
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1),gzip(gfe),gzip(gfe)
Complaints-To: groups-abuse@google.com
Injection-Info: 19g2000hsx.googlegroups.com; posting-host=130.226.188.2;
   posting-account=ps2QrAMAAAA6_jCuRt2JEIpn5Otqf_w0