Current Observation feed from weather.gov forbidden (403)

Question

Weather.gov Current Observation feeds have suddenly begun to fail for all requests from an HTTPClient, and likewise I've observed that many websites across the internet that use AJAX to make calls to weather.gov are also failing.

The result of all calls to weather.gov current observation feeds, e.g. http://w1.weather.gov/xml/current_obs/TAPA.xml, return a 403. Said URL resolves properly in a browser.

Answer 1

Contacting weather.gov resulted in a really fast response, which was:

Applications accessing resources on weather.gov now need to provide a User-Agent header in any HTTP request. Requests without a user agent are automatically blocked. We have implemented this usage policy due to a small number of clients utilizing resources far in excess of what most would consider reasonable.

We recommend providing a user agent string in the following format:

ApplicationName/vX.Y (http://your.app.url/; [email protected])

This will both uniquely identify your application and allow us to contact you and work with you if we observe abnormal application behavior that may result in a block.

Please feel free to email us back if you continue to have issues after verifying that your application is sending the proper headers.

Thanks for using weather.gov.

=======

Here's a snip of C# code. The key thing is that you need to create the request object then append a custom User-Agent string to it before making the call.

...
var request = new HttpRequestMessage(HttpMethod.Post, httpClient.BaseAddress.AbsoluteUri);
request.Headers.Add("User-Agent", "MyApplication/v1.0 (http://foo.bar.baz; [email protected])");
var httpResponse = httpClient.SendAsync(request).Result;
...

Hope this helps folks. Cheers

Answer 2

That is great but you cant set "User-Agent" when you are using an XMLDocument and calling Load() Like this (this used to work):

        XmlDocument doc = new XmlDocument();
        string stationName = @"http://w1.weather.gov/xml/current_obs/PASC.xml";
        doc.Load(stationName);  // exception 403 occurs here now

Instead now you need to perform a GET and then set the User-Agent to your company or email and then use the XmlDocument like:

       XmlDocument doc = new XmlDocument();

       string stationName = @"http://w1.weather.gov/xml/current_obs/PASC.xml";

        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(stationName);

        request.Method = "GET";
        request.ContentType = "application/x-www-form-urlencoded";
        request.UserAgent = "MyApplication/v1.0 (http://foo.bar.baz; [email protected])";
        request.Accept = "Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";

        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        Stream resStream = response.GetResponseStream();

        doc.Load(resStream);
        try
        {

            XmlNodeList list = doc.GetElementsByTagName("temp_f");
            if (list.Count > 0)
            {
                float fTemperature = float.Parse(list.Item(0).InnerText);

            }
        }