Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how converte large XML file to stream-like-java8 of object

Hy, I have a (very)large XML file (100GB) with a list of foo, I want to convert-it into a stream like they where introduce into java 8 of object:

Any idea of lib or code sample?

at the beginning:

<foos> 
  <foo>...</foo>
  <foo>...</foo>
</foos>

at the end:

Stream<Foo>  foosStream = ????("foo.xml")
streamFoos.forEach(foo->foo.doFooStuffs());

Edit: @Pierre Thank-you, here is the implementation of your solution:

  try {
            XMLEventReader reader = XMLInputFactory.newInstance().
                    createXMLEventReader(stream);
            final Unmarshaller unmarshaller = JAXBContext.newInstance(XXXXX.class).createUnmarshaller();

            Iterator<XXXXX> it = new XmlIterator<>(reader, unmarshaller, "xxxxxx");
            return StreamSupport.stream(Spliterators.spliteratorUnknownSize(it, Spliterator.ORDERED), false);
        } catch (XMLStreamException e1) {
            logger.error("XMLStreamException", e1);
        } catch (JAXBException e) {
            logger.error("JAXBException", e);
        }

and

public class XmlIterator<T> implements Iterator<T> {

    private final Logger logger = LoggerFactory.getLogger(this.getClass());

    XMLEventReader reader;

    XMLEvent event;

    Unmarshaller unmarshaller;
    String name;

    public XmlIterator(XMLEventReader reader, Unmarshaller unmarshaller, String name) {
        this.reader = reader;
        this.unmarshaller = unmarshaller;
        this.name = name;
        try {
            reader.next();
            this.event = reader.peek();
        } catch (XMLStreamException e) {
            logger.error("", e);
            event = null;
        }
    }

    @Override
    public boolean hasNext() {
        try {
            while (event != null && !(event.isStartElement() && name.equals(event.asStartElement().getName().getLocalPart()))) {
                Object a = reader.next();
                event = reader.peek();
            }
            return event != null;

        } catch (XMLStreamException e) {
            logger.error("", e);
            event = null;
        }
        return event != null;
    }

    @Override
    public T next() {
        try {
            T next = ((JAXBElement<T>) unmarshaller.unmarshal(reader)).getValue();
            event = reader.peek();
            return next;
        } catch (JAXBException e) {
            logger.error("error during unmarshalling ", e);
            return null;
        } catch (XMLStreamException e) {
            logger.error("error during stream ", e);
            return null;
        }
    }
}
like image 425
sab Avatar asked Mar 11 '23 02:03

sab


1 Answers

  • Create a Stax Event reader https://docs.oracle.com/javase/tutorial/jaxp/stax/example.html reading your xml file
  • Each time you see a tag 'foo' (using https://docs.oracle.com/javase/7/docs/api/javax/xml/stream/XMLEventReader.html#peek() ) , use the Stax reader to parse and build your object Foo

    Foo readFoo(XMLEventReader xmlIn) throws XMLStreamException {
       (...)
       return foo; 
    }
    
  • implements a java.util.Iterator that will use the previous function to return the 'next()' Foo

  • convert this iterator to a stream How to create a Java 8 Stream from an iterator?
like image 187
Pierre Avatar answered Mar 13 '23 14:03

Pierre