Hi I am trying to automate a few searches on http://www.192.com/businesses/advanced/ using the WebClient. I can get it to work using a WebBrowser which I would prefer to avoid for loading times. I got it to work using a WebRequest but would like to see if theres any reason why the WebClient code below keeps giving me the error: "The remote server returned an error: (500) Internal Server Error".
This is the code using the WebClient:
I have also tried using a NameValueCollection to compile the postdata which also fails.
I can get it to work using a WebRequest as below:
Can anyone see if I have missed something in the WebClient approach or even let me know if it works for you so I know whether it is actually code related.
Thanks
Jay
This is the code using the WebClient:
VB.NET Code:
Dim WClient As New WebClient ' // I got the headers from IE Developer Tools WClient.Headers.Add(HttpRequestHeader.UserAgent, "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)") WClient.Headers.Add(HttpRequestHeader.ContentType, "application/x-www-form-urlencoded") ' // This header is used when trying to compress the postdata using the commented code below ' WClient.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate") ' // This is the postdata that is sent to the page as detected by IE Deleveloper Tools. I just add in the relevant search text i.e. company=company+name - with spaces replaced by +'s Dim Post As String = "company=&description=&subbuild=&buildno=&buildname=&street=&locality=&town=&county=&postcode=&btnSearch=Submit" Dim PostBytes As Byte() = Encoding.UTF8.GetBytes(Post) ' // This part is commented out but just to show that I have tried using the ICSharpCode.SharpZipLib dll to compress the PostData before sending to the page ' Dim PostCompressedBytes As Byte() = Nothing ' Using PostDataStream As New MemoryStream ' Using PostCompress As New GZipOutputStream(PostDataStream) ' 'PostCompress.IsStreamOwner = False ' PostCompress.Write(PostBytes, 0, PostBytes.Length) ' End Using ' PostCompressedBytes = PostDataStream.ToArray ' End Using 'Now when I try to write the postdata to the page and read the response I get the error message Dim DocumentBytes As Byte() = WClient.UploadData("http://www.192.com/businesses/advanced/", PostBytes)
I have also tried using a NameValueCollection to compile the postdata which also fails.
I can get it to work using a WebRequest as below:
VB.NET Code:
Dim postData As String = "company=&description=&subbuild=&buildno=&buildname=&street=&locality=&town=&county=&postcode=&btnSearch=Submit" Dim postBytes As Byte() = Encoding.UTF8.GetBytes(postData) Dim request = CType(WebRequest.Create("http://www.192.com/businesses/search/advanced/"), HttpWebRequest) request.UserAgent = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate") request.Method = "POST" request.KeepAlive = True request.ContentType = "application/x-www-form-urlencoded" request.Referer = "http://www.192.com/businesses/search/advanced/" request.ContentLength = postBytes.Length request.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate Using postStream As Stream = request.GetRequestStream postStream.Write(postBytes, 0, postBytes.Length) End Using Using response = request.GetResponse() Using responseStream = response.GetResponseStream() Using sr = New StreamReader(responseStream) MsgBox(sr.ReadToEnd()) End Using End Using End Using
Can anyone see if I have missed something in the WebClient approach or even let me know if it works for you so I know whether it is actually code related.
Thanks
Jay