Is Scala/Java not respecting w3 "excess dtd traffic" specs?

Question

I'm new to Scala, so I may be off base on this, I want to know if the problem is my code. Given the Scala file httpparse, simplified to:

object Http {
   import java.io.InputStream;
   import java.net.URL;

   def request(urlString:String): (Boolean, InputStream) =
      try {
         val url = new URL(urlString)
         val body = url.openStream
         (true, body)
      }
      catch {
         case ex:Exception => (false, null)
      }
}

object HTTPParse extends Application {
   import scala.xml._;
   import java.net._;

   def fetchAndParseURL(URL:String) = {
      val (true, body) = Http request(URL)
      val xml = XML.load(body) // <-- Error happens here in .load() method
      "True"
   }
}

Which is run with (URL doesn't matter, this is a joke example):

scala> HTTPParse.fetchAndParseURL("http://stackoverflow.com")

The result invariably:

   java.io.IOException: Server returned HTTP response code: 503 for URL: http://www.w3.org/TR/html4/strict.dtd
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1187)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:973)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEnti...

I've seen the Stack Overflow thread on this with respect to Java, as well as the W3C's System Team Blog entry about not trying to access this DTD via the web. I've also isolated the error to the XML.load() method, which is a Scala library method as far as I can tell.

My Question: How can I fix this? Is this something that is a by product of my code (cribbed from Raphael Ferreira's post), a by product of something Java specific that I need to address as in the previous thread, or something that is Scala specific? Where is this call happening, and is it a bug or a feature? ("Is it me? It's her, right?")

Answer 1

I've bumped into the SAME issue, and I haven't found an elegant solution (I'm thinking into posting the question to the Scala mailing list) Meanwhile, I found a workaround: implement your own SAXParserFactoryImpl so you can set the f.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true); property. The good thing is it doesn't require any code change to the Scala code base (I agree that it should be fixed, though). First I'm extending the default parser factory:

package mypackage;

public class MyXMLParserFactory extends SAXParserFactoryImpl {
      public MyXMLParserFactory() throws SAXNotRecognizedException, SAXNotSupportedException, ParserConfigurationException {
        super();
        super.setFeature("http://xml.org/sax/features/validation", false);
        super.setFeature("http://apache.org/xml/features/disallow-doctype-decl", false); 
        super.setFeature("http://apache.org/xml/features/nonvalidating/load-dtd-grammar", false); 
        super.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false); 
      } 
    }

Nothing special, I just want the chance to set the property.

(Note: that this is plain Java code, most probably you can write the same in Scala too)

And in your Scala code, you need to configure the JVM to use your new factory:

System.setProperty("javax.xml.parsers.SAXParserFactory", "mypackage.MyXMLParserFactory");

Then you can call XML.load without validation

Answer 2

Without addressing, for now, the problem, what do you expect to happen if the function request return false below?

def fetchAndParseURL(URL:String) = {      
  val (true, body) = Http request(URL)

What will happen is that an exception will be thrown. You could rewrite it this way, though:

def fetchAndParseURL(URL:String) = (Http request(URL)) match {      
  case (true, body) =>      
    val xml = XML.load(body)
    "True"
  case _ => "False"
}

Now, to fix the XML parsing problem, we'll disable DTD loading in the parser, as suggested by others:

def fetchAndParseURL(URL:String) = (Http request(URL)) match {      
  case (true, body) =>
    val f = javax.xml.parsers.SAXParserFactory.newInstance()
    f.setNamespaceAware(false)
    f.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
    val MyXML = XML.withSAXParser(f.newSAXParser())
    val xml = MyXML.load(body)
    "True"
  case _ => "False"
}

Now, I put that MyXML stuff inside fetchAndParseURL just to keep the structure of the example as unchanged as possible. For actual use, I'd separate it in a top-level object, and make "parser" into a def instead of val, to avoid problems with mutable parsers:

import scala.xml.Elem
import scala.xml.factory.XMLLoader
import javax.xml.parsers.SAXParser
object MyXML extends XMLLoader[Elem] {
  override def parser: SAXParser = {
    val f = javax.xml.parsers.SAXParserFactory.newInstance()
    f.setNamespaceAware(false)
    f.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
    f.newSAXParser()
  }
}

Import the package it is defined in, and you are good to go.