Stream parse 4 GB XML file in PHP

Question

I'm trying and need some help doing the following:

I want to stream parse a large XML file ( 4 GB ) with PHP. I can't use simple XML or DOM because they load the entire file into memory, so I need something that can stream the file.

How can I do this in PHP?

What I am trying to do is to navigate through a series of <doc> elements. And write some of their children to a new xml file.

The XML file I am trying to parse looks like this:

<feed>
    <doc>
        <title>Title of first doc is here</title>
        <url>URL is here</url>
        <abstract>Abstract is here...</abstract>
        <links>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
       </link>
    </doc>
    <doc>
        <title>Title of second doc is here</title>
        <url>URL is here</url>
        <abstract>Abstract is here...</abstract>
        <links>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
       </link>
    </doc>
</feed>

I'm trying to get / copy all the children of each <doc> element into a new XML file except the <links> element and its children.

So I want the new XML file to look like:

<doc>
    <title>Title of first doc is here</title>
    <url>URL is here</url>
    <abstract>Abstract is here...</abstract>
</doc>
<doc>
    <title>Title of second doc is here</title>
    <url>URL is here</url>
    <abstract>Abstract is here...</abstract>
</doc>

I would greatly appreciate any and all help in streaming / stream parsing / stream reading the original XML file and then writing some of its contents to a new XML file in PHP.

Answer 1

Here's a college try. This assumes a file is being used, and that you want to write to a file:

<?php

$interestingNodes = array('title','url','abstract');
$xmlObject = new XMLReader();
$xmlObject->open('bigolfile.xml');

$xmlOutput = new XMLWriter();
$xmlOutput->openURI('destfile.xml');
$xmlOutput->setIndent(true);
$xmlOutput->setIndentString("   ");
$xmlOutput->startDocument('1.0', 'UTF-8');

while($xmlObject->read()){
    if($xmlObject->name == 'doc'){
        $xmlOutput->startElement('doc');
        $xmlObject->readInnerXML();
        if(array_search($xmlObject->name, $interestingNodes)){
             $xmlOutput->startElement($xmlObject->name);
             $xmlOutput->text($xmlObject->value);
             $xmlOutput->endElement(); //close the current node
        }
        $xmlOutput->endElement(); //close the doc node
    }
}

$xmlObject->close();
$xmlOutput->endDocument();
$xmlOutput->flush();

?>

Answer 2

For this scenario you can't afford to use a DOM parser, as you stated, it will not fit in memory due to the file size, and even if you could, it'll be slow as it first load the entire file and after that you have to iterate through it, so, for this case you should try a SAX parser (event/stream oriented), add a handler for those tag you're insterested in (doc, title, url, abstract) and for every event append the node found in the new XML file.

Here you have more information:

What is the fastest XML parser in PHP?

Here is a (not tested) sample of what the code would be:

<?php
    $file = "bigfile.xml";
    $fh = fopen("out.xml", 'a') or die("can't open file");
    $currentNodeTag = "";    
    $tags = array("doc", "title", "url", "abstract");

    function startElement($parser, $name, $attrs) {
        global $tags;

        if (isset($tags[strtolower($name)])) {
            $currentNodeTag = strtolower($name);
            fwrite($fh, sprintf("<%s>\n"));
        }
    }

    function endElement($parser, $name) {
        global $tags;

        if (isset($tags[strtolower($name)])) {
            fwrite($fh, sprintf("</%s>\n"));
            $currentNodeTag = "";
        }
    }

    function characterData($parser, $data) {
        if (!empty($currentNodeTag)) {
            fwrite($fh, $data);
        }
    }    

    $xmlParser = xml_parser_create();
    xml_set_element_handler($xmlParser, "startElement", "endElement");
    xml_set_character_data_handler ($xmlParser, "characterData");

    if (!($fp = fopen($file, "r"))) {
        die("could not open XML input");
    }

    while ($data = fread($fp, 4096)) {
        if (!xml_parse($xmlParser, $data, feof($fp))) {
            die(sprintf("XML error: %s at line %d",
                        xml_error_string(xml_get_error_code($xmlParser)),
                        xml_get_current_line_number($xmlParser)));
        }
    }

    xml_parser_free($xmlParser);
    fclose($fh);
?>