Thread View: aus.computers.java
2 messages
2 total messages
Started by mario
Sun, 15 Aug 2004 10:50
Concurrent modification
Author: mario
Date: Sun, 15 Aug 2004 10:50
Date: Sun, 15 Aug 2004 10:50
70 lines
3400 bytes
3400 bytes
Hi guys,
I'm trying to solve a problem it's giving me an headhache. I'm a
newbie in java lang. I'm developing a web spider using jericho
libraries. My classes take a url, connect to it, put all the unique
links present in the html page in a Hashtable as key, then it takes
all the words of the page and put it as value of each key. What I need
is a recursive system that cycle through the hashtable and put new
found links in the Hashtable so the cycle would notice it and continue
until all the links have been parsed.
This is an example of what I would need
public class Parse {
public static Hashtable linksList = new Hashtable();
public static String url = "http://myaddress.com"
public static doParse() {
linksList.put(url,new Hashtable());
//generic cycle to fill the hashtable
for (Enumeration en = linksList.keys(); en.hasMoreElements(); )
{
String urlToParse = en.nextElement();
parse(htmlText);
}
}
public static parse(String url) {
String htmlText = Connection.getString(url); //get the html of
a given url
Hashtable words = getWords(htmlText); //get all the words of a
given html text
linksList.put(url, words);
for (... all the links in a fiven htmlText) {
String newUrl; //each new links found in the given html
text
linksList.put(newUrl, new Hashtable());
}
}
}
As you can see the parse method add recursively new hashtable element
for each new url it found.
The generic cycle should be aware of the change that has been made and
go on until the parse method
won't add any new link. This would let me to fill the hashtable with
an url and the words contained in that
url for all the pages in a particoular site. Then, when my ht would be
completely filled I would add
it to a db, having only one connection to it.
How would I do that? I see that HT doesn't see any changes made to
itself inside a cycle. What other
object I should use? Any other idea on how to perform it? I looked
through the internet but I only found
solution on how to avoid the concurrentModificationException on
ArrayList, not code samples on how to
do it.
I really hope you can help me as I'm stuck with this problem and have
no idea on how to solve it
Thanks a lot in advance!!!
Bye for now!
mario
Re: Concurrent modification
Author: windytummy
Date: Tue, 17 Aug 2004 16:07
Date: Tue, 17 Aug 2004 16:07
53 lines
2534 bytes
2534 bytes
> The generic cycle should be aware of the change that has been made and
> go on until the parse method
> How would I do that? I see that HT doesn't see any changes made to
> itself inside a cycle.
Haven't touch Java for a while but I hope I can help. I believe you're
on the right track. To make your HT "see" changes, store its key-value
pairs to a text file. If your HT compares itself with URLs in the text
file, then it can update itself properly.
I personally prefer HashMap as it's faster compared to HashTable like
what I used here (assuming you have the text file):
class Sample
{
//Instance variables
private HashMap map;
private BufferedReader reader;
private String line;
//Constructor
public Sample(String theFile) throws IOException
{
map = new HashMap();
reader = new BufferedReader(new FileReader(theFile));
line = reader.readLine();
while (line != null)
{
map.put(line.toUpperCase(), line.toUpperCase());
line = reader.readLine();
}
}
/**
* Checks if a particular word is in the file
* @param word
* @return true if word exists in dictionary
*/
public boolean wordExist(String word)
{
if (map.containsKey(word))
{
return true;
}
else
{
return false;
}
}
}
cheers,
wt.
Thread Navigation
This is a paginated view of messages in the thread with full content displayed inline.
Messages are displayed in chronological order, with the original post highlighted in green.
Use pagination controls to navigate through all messages in large threads.
Back to All Threads