Hi, I have a question for those of you who have experience with java.net, I would really appreciate your help.

I'm trying to extract data from web pages using Java, I understand that if a web page is static then we can use HttpURLConnection to get inputstream and therefore read its contents.

However the page I need to extract is dynamicly generated by a HTTP request query (using either GET or POST method), let's say the query is a HTTP GET request: www.myhost.com/search.jsp?param1=this¶m2=that , if I paste this query into a brower it would work just fine and return the results I want, but when I call this query through a URLConnection object it fails to retrieve the data because it complains the session is invalid, this doesn't seem to work even when I set the cookie into my request... :-(

So has anyone done anything similar to this before? Can you give me any leads on this? Or do you know how to monitor the HTTP traffic goes in and out of your local machine so I can somehow send the request and intercept the response?

Many thanks for your help!