# Reading an HTML table into a Data Frame using XPath

Duncan Temple Lang
UC Davis

This document describes some approaches to scraping data from an HTML table into an R data frame. Let's take the example at http://www.bom.gov.au/climate/dwo/IDCJDW4036.latest.shtml url = "http://www.bom.gov.au/climate/dwo/IDCJDW4036.latest.shtml" Thanks to Bob Green for showing it to me. And, yes we know in this case that we can get the data as a .csv file directly! We are using this so that we can compare the two more readily.

We start by reading the HTML file into R. We use htmlTreeParse as this is a more tolerant XML parser that knows about "sloppy"/malformed HTML. doc = htmlTreeParse(url, useInternalNodes = TRUE) Note the use of useInternalNodes as we want to be able to do XPath queries on this node.

Next, we might find all the 'table' nodes with tables = doc["//table"] This gives us 4 different tables.

We can explore each of these and find out which we want in an interactive programmatic manner, but generally we can see from the HTML display precisely what we want. In our case, we want the table which has a column name of 'Date'. We can use another XPath expression to find this. We might match any node whose text matches 'Date' with n = doc["//*[. = 'Date']"] or alternatively, find any table which has a node th or td node whose content is 'Date' with n = doc[["//table//th[. = 'Date']|//table//td[. = 'Date']"]]

You need a version in excess of 1.9-1 to get the [[ operator on the node set. Otherwise, use [ and then access the first element.

Both of these return the actual th node. However, we want the table node. So we can walk back up the tree hierarchy one parent at a time until we find the enclosing parent. n = n[[1]] while(xmlName(n) != "table") n = xmlParent(n) And now we are ready to get the cell contents.

We can quickly get the contents of all the cells with the XPath expression cells = xpathApply(n, "//th|//td", xmlValue) The first row is quite complex. It has names that span columns and then other sub-names for each column and generally confuses matters. We could look for each row instead. rows = getNodeSet(n, "//tr") This too is complicated by the fact that the "second" of summary statistics is in fact part of this table (for purposes of alignment and presentation). We know this because a) there are 25 rows (length(rows)) and b) the contents of the, e.g., 23rd row are given by xmlValue(rows[[23]]) [1] "Lowest5.517.70\302\240\302\240\302\240\302\240\302\24013.043\302\240#41007.717.333\302\240#71005.2" which we can see corresponds to the second row of the "second" table. The string is the simple concatenation (without spaces) of the contents of all the sub-nodes.

But if we look at the second row: xmlValue(rows[[2]]) [1] "MinMaxDirSpdTimeTempRHCldDirSpdMSLPTempRHCldDirSpdMSLP" This is the second row that is part of the header. So we look at the 4th row, since there are 3 rows in the header. And sure enough this contains our data. So it would appear that rows 4 through 20 will contain days 1 through 17 inclusive.

Now, for each row node, we want to loop over its children and extract its value. And we want to do this for each of the 17 rows. Again, this is relatively easy in R. data = sapply(rows[4:20], function(x) xmlSApply(x, xmlValue)) This gives us a character matrix with 17 columns and 22 rows. So we need to transpose this. data = t(data) We are almost done. One thing we need to do is handle the "empty" cells. These come across as strings of the form \302\240 on my terminal. So we replace these with NA before we do anything else. data[data == "\302\240"] = NA And then we convert the entire matrix to a data frame data = as.data.frame(data) We want to clean this up a lot to convert columns to numbers rather than factors and handle those columns that are legitimate factors, e.g. the Dir columns and also handle the Time fields. We can do this with contextual knowledge (i.e. the column numbers) or some more automated mechanism, e.g. attempting coercion and handling "errors". But this is not an HTML/XPath issue and so we leave it as an excercise for the reader to think about. But here is one way dd = as.data.frame(lapply(data, function(x) { y = as.numeric(as.character(x)) if(sum(is.na(x)) != sum(is.na(y))) x else y }))