Import data from URL

Question

The St. Louis Federal Reserve Bank has a great set of data available on a variety of their web pages, such as:

http://research.stlouisfed.org/fred2/series/OILPRICE/downloaddata?cid=32217 http://www.federalreserve.gov/releases/h10/summary/default.htm http://research.stlouisfed.org/fred2/series/DGS20

The data sets get updated, some as often as daily. I tend to have an interest in the daily data (see the above settings on the URLS)

I'd like to import these kinds of price or rate data streams (accessible as CSV or Excel files at the above URLs) directly into Mathematica.

I've looked at the documentation on Importing[] but I find scant documentation (actually none) on how to go about something like this.

It looks like I need to navigate to the pages, send some data to select specific files and formats, trigger the download, then access the downloaded data from my own machine. Even better if I could access the data directly from the sites.

I had hoped Wolfram Alpha might make this sort thing easy, but I haven't had any success.

FinancialData[] would seem natural for this sort of thing, but I don't see anyway to do it. Financial data has lots of features, but I don't see a way yo get this sort of thing.

Does anyone have any experience with this or can someone point me in the right direction?

Answer 1

You can Import directly from a URL. For example, the data from federalreserve.gov can be obtained and visualized as follows.

url = "http://www.federalreserve.gov/datadownload/Output.aspx?";
url = url<>"rel=H10&series=a660e724c705cea4b7bd1d1b85789862&lastObs=&";
url = url<>"from=&to=&filetype=csv&label=include&layout=seriescolumn";
data = Import[url, "CSV"];
DateListPlot[data[[7 ;;]], Joined -> True]

I broke up url for convenience, since it's so long. I had to examine the contents of data before I knew exactly how to plot it - a step that is typically necessary. I'm sure that the data from stlouisfed.org can be obtained in a similar way, but it requires the use of an API with key to access it.

Answer 2

As Mark said, you can get the data directly from a URL. Your oil data can be imported from a different URL than you had:

http://research.stlouisfed.org/fred2/data/OILPRICE.txt

With that URL, you can do this:

oil = Import["http://research.stlouisfed.org/fred2/data/OILPRICE.txt",
"Table", "HeaderLines" -> 12, "DateStringFormat" -> {"Year", "Month", "Day"}];
DateListPlot[oil, Joined -> True, PlotRange -> All]

Note that "HeaderLines"->12 option strips off the header text in the first 12 lines (you have to count the header lines to know how many to remove). I've also specified the date format.

To find that URL, do as you did before, but click on a data series and then choose View Data from the menu on the left when you see the chart.

Answer 3

The documentation has a short example on extracting data out of a webpage:

http://reference.wolfram.com/mathematica/howto/CleanUpDataImportedFromAWebsite.html

Of course, what actually needs to be done will vary significantly from page to page.