## org.das2.util.filesystem.HtmlUtil

HTML utilities, such as getting a directory listing, where a "file" is a link below the directory we are listing, and read a URL into a String.

## HtmlUtil( )

## checkRedirect

checkRedirect( java.net.URLConnection urlConnection ) → java.net.URLConnection

check for 301, 302 or 303 redirects, and return a new connection in this case. This should be called immediately before the urlConnection.connect call, as this must connect to get the response code.

### Parameters

urlConnection - if an HttpUrlConnection, check for 301 or 302; return connection otherwise.

### Returns:

a connection, typically the same one as passed in.

### See Also:

HttpUtil#checkRedirect(java.net.URLConnection)
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## consumeStream

consumeStream( java.io.InputStream err ) → void

nice clients consume both the stderr and stdout coming from websites. This reads everything off of the stream and closes it. http://docs.oracle.com/javase/1.5.0/docs/guide/net/http-keepalive.html suggests that you "do not abandon connection"

### Parameters

err - the input stream

### Returns:

void (returns nothing)

### See Also:

HttpUtil#consumeStream(java.io.InputStream)
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## getDirectoryListing

getDirectoryListing( java.net.URL url, java.io.InputStream urlStream ) → java.net.URL[]

Get the listing of the web directory, returning links that are "under" the given URL. Note this does not handle off-line modes where we need to log into a website first, as is often the case for a hotel. This was refactored to support caching of listings by simply writing the content to disk.

### Parameters

url - the address.
urlStream - stream containing the URL content, which must be UTF-8 (or US-ASCII)

### Returns:

list of URIs referred to in the page.
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## getDirectoryListing

getDirectoryListing( java.net.URL url, java.io.InputStream urlStream, boolean childCheck ) → java.net.URL[]

Get the listing of the web directory, returning links that are "under" the given URL. Note this does not handle off-line modes where we need to log into a website first, as is often the case for a hotel. This was refactored to support caching of listings by simply writing the content to disk.

### Parameters

url - the address.
urlStream - stream containing the URL content, which must be UTF-8 (or US-ASCII)
childCheck - only return links to URLs "under" the url.

### Returns:

list of URIs referred to in the page.
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## getDirectoryListing

getDirectoryListing( java.net.URL url ) → java.net.URL[]

Get the listing of the web directory, returning links that are "under" the given URL. Note this does not handle off-line modes where we need to log into a website first, as is often the case for a hotel.

### Parameters

url - an URL

### Returns:

list of URIs referred to in the page.
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## getInputStream

getInputStream( java.net.URL url ) → java.io.InputStream

get the inputStream, following redirects if a 301 or 302 is encountered. The scientist may be prompted for a password, but only if "user@" is in the URL. Note this does not explicitly close the connections to the server, and Java may not know to release the resources. TODO: fix this by wrapping the input stream and closing the connection when the stream is closed. This was done in Autoplot's DataSetURI.downloadResourceAsTempFile

### Parameters

url - an URL

### Returns:

input stream

### See Also:

org.autoplot.datasource.DataSetURI#downloadResourceAsTempFile
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## getInputStream

getInputStream( java.net.URL url, java.util.Map requestProperties ) → java.io.InputStream

get the inputStream, following redirects if a 301 or 302 is encountered. The scientist may be prompted for a password, but only if "user@" is in the URL. Note this does not explicitly close the connections to the server, and Java may not know to release the resources. TODO: fix this by wrapping the input stream and closing the connection when the stream is closed. This was done in Autoplot's DataSetURI.downloadResourceAsTempFile

### Parameters

url - the URL to retrieve
requestProperties - key-value pairs like "Cookie" or "PRIVATE-TOKEN"

### Returns:

input stream

### See Also:

org.autoplot.datasource.DataSetURI#downloadResourceAsTempFile
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## getLinks

getLinks( java.net.URL url, String content ) → java.util.List

return the links found in the content, using url as the context.

### Parameters

url - null or the url for the context.
content - the html content.

### Returns:

a list of URLs.
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## getMetadata

getMetadata( java.net.URL url, java.util.Map props ) → java.util.Map

return the metadata about a URL. This will support http, https, and ftp, and will check for redirects. This will allow caching of head requests.

### Parameters

url - ftp,https, or http URL
props - a java.util.Map

### Returns:

the metadata

### See Also:

HttpUtil#getMetadata(java.net.URL, java.util.Map)
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## isDirectory

isDirectory( java.net.URL url ) → boolean

### Parameters

url - an URL

### Returns:

boolean
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## readToString

readToString( java.net.URL url ) → String

read the contents of the URL into a string, assuming UTF-8 encoding.

### Parameters

url - an URL

### Returns:

a String
[search for examples] [view on GitHub] [view on old javadoc] [view source]

## readToString

readToString( java.net.URL url, java.util.Map requestProperties ) → String

read the contents of the URL into a string, assuming UTF-8 encoding.

### Parameters

url - an URL
requestProperties - a java.util.Map

### Returns:

a String
[search for examples] [view on GitHub] [view on old javadoc] [view source]