How do I do a simple HTTP GET?

Submitted by vbp on January 29, 2008 - 00:53. Forums :

Project: WSO2 Mashup Server

I started playing with the Mashup Server tonight and I am about to go to bed without having been able to do a simple HTTP get. With some luck, a kind soul will be able to help. Here is what I tried:

function getPage() {

    var request = new WSRequest();
    var options = new Array();
    // options.useBinding = "HTTP";
    options.useSOAP = false;
    options.HTTPMethod = "GET";

    try {
        version.open(options, "http://wso2.com/", false);
        version.send();
        result = version.responseText;
    } catch (e) {
        print(e);
    }
    return this.result;
}

I consistently get a (null) result. I tried both useBinding and useSOAP to no avail. I tried to put a TCPMon tunnel in the middle to see the outgoing HTTP request, but it never even make it to my TCPMon listener (BTW, I noticed that you ship the TCPMon jar, so at that point you might as well provide a tcpmon.bat or tcpmon.sh in /bin to make it easy for people).

Thanks.

Bookmark/Search this post with:

Login or register to post comments
Printer friendly version
639 reads

The Scraper Host Object

Submitted by channa on January 29, 2008 - 09:27.

Hi,

The easiest way to get the content of a web page in a usable format is to scrape it using the host object we provide, which will return XML that you can then process further.

Just use the function below and you've got the page content. Some of our samples will demonstrate how you can get specific parts of a given page.


function getString()
{
var config =
        <config>
		<var-def name="response">
			<html-to-xml>
			   <http method='get' url='http://wso2.org'/>
			</html-to-xml>
		</var-def>
        </config>;
    var scraper = new Scraper(config);
    var result = scraper.response;
    return result;
}

Problems with feed objects

Submitted by vbp on February 7, 2008 - 01:39.

Thanks for the response Channa. In fact the pages I am trying to consume are blog feeds (even though the http://wso2.com/ URL I put in my code snippet obvisouly wasn't one). So I followed your recommendation but rather than a Scraper object I went with a Feed object, which I populated from a URL via a FeedReader object. The problem is that the Entry object that I get from the Feed object doesn't give me access to the "gui" RSS 2.0 element (more or less equiv to the "id" element of Atom) of a feed item. So I still need to work at the XML level to get that. But when I do "entry.XML" on an Entry instance that I got from a Feed object, the E4X object I get back is not a representation of the actual XML that was retrieved from the server. It seems to be an XML representation of the format-neutral (i.e. bridging between RSS, Atom...) Feed object. As far as I can tell, I can't use that E4X object to get my RSS 2.0 "guid" element. Can I get the original XML from a Feed object?

If not, I am back to my initial question: how can I get an arbitrary piece of XML from a URL as an E4X object?

re: Problems with feed objects

Submitted by Jonathan Marsh on February 7, 2008 - 10:24.

Channa's code should work pretty well, though there are a few tricks to watch out for:

1) the response from the Scraper object is a string containing the XML textual representation of the document, so conversion to an E4X xml object is required (we're considering a feature request to return an XML object instead.)
2) the response may contain an XML decl, which cannot be parsed by E4X (it stupidly thinks its an illegal processing instruction.)
3) there may be other processing instructions, if so you will get an XML List instead of the document element. Your logic has to accomodate this. In the example below, I strip off all leading processing instructions so I'm sure to have just a single xml element. Replace the while with the commented-out if if you only want to strip xml decls.

getFeedFirstItem.inputTypes = {"url" : "xs:anyURI?"};
getFeedFirstItem.outputType = "xml";
function getFeedFirstItem(url) {
    if (url==null) url = "http://auburnmarshes.spaces.live.com/feed.rss";
    var config =
            <config>
                <var-def name="response">
                    <http method='get' url={url}/>
                </var-def>
            </config>;
    var scraper = new Scraper(config);
    var result = scraper.response;
    while (result.indexOf("<?") == 0)  // if (result.indexOf("<?xml ") == 0)
        result = result.substring(result.indexOf("?>")+2);
    return new XML(result).channel[0].item[0];
}

It is on the top of our list to make this simple case simpler - the above is far too tortuous to abide (as you have noticed!) My current proposal is to add a one-line way to fetch xml, parse it, and return it as an XML or XML List object: If so, the whole function body above could be replaced by the following simple statement. I am hoping this will be available in a dot release within a few months.

return system.getXML(url);

Thanks Jonathan. +1 to

Submitted by vbp on February 7, 2008 - 11:15.

Thanks Jonathan. +1 to system.getXML(url) !!!

system.getXML(url) is implemented

Submitted by keith on February 7, 2008 - 15:42.

Hi,

I implemented the system.getXML(url) thing. This return an e4X XML object. You can use it if you pickup the nightly build from here

Its simple to get an XML now. The following is an example,

function getSomeXML()
{
return system.getXML("https://mooshup.com/search.jsp?query=mashups&scope=system&format=rss");
}

Thanks,

Keith.

How to do a simple GET with WSRequest

Submitted by tyrell on February 7, 2008 - 11:43.

Hi,

Please refer to the sample Mashup at mooshup.com

Basically the code is as follows. You were almost there :)

toString.documentation = "Does a simple GET operation to retrieve an RSS feed and returns the contents." ;
toString.inputTypes = { /* TODO: Add input types of this operation */ };
toString.outputType = "String"; /* TODO: Add output type here */
function toString()
{
var request = new WSRequest();

    var options = new Array();
    options.useSOAP = false;
    options.useWSA = false;
    options.HTTPMethod = "GET";

    try {
            request.open(options,"http://rss.news.yahoo.com/rss/topstories", false);
            request.send(null);
            result = request.responseE4X;
    } catch (e) {
                print(e);
        }

return result;
}