Wednesday, June 25, 2008

Escape from XML Parsing Hell

I just spent 3 days on the most futile effort of my life, or so it seems at the moment. I'm posting this in the hope that it might help someone who Google's "slow XML java parser" in the future. I've been parsing a lot of XHTML files using Java for a project. However, it was taking about 1s per file, and the time was the same for files of 15k or 150k. Clearly something was wrong. It was also the same for the Xerces SAX parser, the DOM parser, dom4j parser, the Piccolo SAX parser, ... nothing was fast enough. All the benchmarks I could find said it should be on the order of 20ms / file.

Anyway, I brought home my laptop tonight because I was still so frustrated with this stupid slow XML parsing problem and I couldn't put it away. I started running my test program when not connected to the internet, and it generated an exception saying it couldn't connect to www.w3.org. So, I was thinking, why the heck is it doing that? I'd already set the parser to be non-validating.

It turns out the parsers all fetch any external DTD that is referenced, even if the parser is non validating! So for every file, the header line was referencing the XHTML DTD and it was downloading it from an external site. So, if you have an abnormally slow XML parser, maybe this is why!

This feature disables that, in case you are ever parsing xhtml in the future and don't care about validation:

xmlReader.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd" , false);

Now it parses 100 trials in 250ms total (down from about 55 seconds). And, I guess my program has stopped retrieving the same XHTML DTD from www.w3.org thousands of times an hour (sorry w3.org, it wasn't a DoS attack, I swear).

Phew! What a waste of time. I would never have thought to check for unexpected network connections. Glad my internet wasn't working at home.

3 days of work, help from colleagues, testing various parsers, running on various machines, profiling my code, ... for a 1 line fix that I found by accident. Sometimes I hate computers.

Now, time to move on with my super-cool-top-secret project. :)

29 comments:

Shahan Khatchadourian said...

I use Woodstox (http://woodstox.codehaus.org/) which is a great XML parser, it's supposed to one of the fastest due to the STAX-based processing (as opposed to SAX). I never ran into the same problem you did but it's good to know that feature may sometimes be enabled by default.

Retreat Searcer said...

Not every SAX parser supports the http://apache.org/xml/features/nonvalidating/load-external-dtd feature. Piccolo for one, does not.

However, you can do the same thing by setting an EntityResolver which doesn't do anything. :-)

Specifics on what to do available here

jhoney said...

You can't really say what is beautiful about a place, but the image of the place will remain vividly with you.
http://empoweringhomebusiness.com|http://educationindexing.com|http://thehealthprovider.com|http://cotton-house-hotel.com|http://legalprecedence.com

jhoney said...

When virtue and modesty enlighten her charms, the lustre of a beautiful woman is brighter than the stars of heaven, and the influence of her power it is in vain to resist.
Double Glazing Preston

jhoney said...

There is no definition of beauty, but when you can see someone's spirit coming through, something unexplainable, that's beautiful to me.
Roller Shutters

jhoney said...

Cipro Online
The real sin against life is to abuse and destroy beauty, even one's own even more, one's own, for that has been put in our care and we are responsible for its well-being.

jhoney said...

wow, awesome post, I was wondering if there is a way to enlarge my penis quickly. and found your site by google, learned a lot, now i’m a bit clear. bookmarked and also signed up your rss. keep us updated.
Online Pharmacy

jhoney said...

wow, awesome post, I was wondering if there is a way to enlarge my penis quickly. and found your site by google, learned a lot, now i’m a bit clear. bookmarked and also signed up your rss. keep us updated.
Cerrajeros en Barcelona

jhoney said...

Door Lining
Thank you for this post. Thats all I are able to say. You most absolutely have built this blog website into something speciel. You clearly know what you are working on, youve insured so many corners.thanks

jhoney said...

We want to reinvent the phone. What's the killer app? The killer app is making calls! It's amazing how hard it is to make calls on most phones. We want to let you use contacts like never before - sync your iPhone with your PC or mac.
Buy Sleeping Tablets Online

jhoney said...

The connections that you make early in your career will take you far. After 20 years it is still true for me. This has been true regarding companies I worked for and those I worked with. All you have written is excellent advice.
Cialis Online

jhoney said...

There is no question that instantaneous water heaters are good options to trust in terms of their water heating services. However, you need to know that not all kiArt Classes Manchester
nds of this water heater will work best for you.

jhoney said...

You must know by now, your article goes to the nitty-gritty of the subject. Your clarity leaves me wanting to know more. Just so you know, i will immediately grab your feed to keep up to date with your online blog. Sounding Out thanks is simply my litZopiclone Online
tle

habshi75 said...

Flaming enthusiasm, backed up by horse sense and persistence, is the quality that most frequently makes for success.
ket testai​

GOD is BIG Power said...

The Mba insistences article is your chance to sparkle. Study how to use each piece to highlight specific characteristics of your demand and put your best foot forward.
Valium Online

habshi75 said...

Take up one idea. Make that one idea your life - think of it, dream of it, live on that idea. Let the brain, muscles, nerves, every part of your body, be full of that idea, and just leave every other idea alone. This is the way to success.
villa toscane

jhoney said...

I gathered useful information on this point as I am working on a business project. Thank you posting relative information and its now becoming easier to complete this assignmenPersonal injury lawyer Oshawa

jhoney said...

Oh, this is a magical post. It’s life the most majestic post there is on this planet right now. This is really good work from your side. I strongly recommend you keep doing this kind of work.
Accident Lawyer Calgary

jhoney said...

Good information here. I really enjoy reading them every day. I’ve learned a lot from them.Thanks so much for sharing this information. Greatly help me being a newbie.
http://www.costapharmacy.com/codeine.htm

Ethan Harry said...

fantastic work has done thanks for sharing and keep enjoy
High School Diploma

jhoney said...

Very Useful information, this is both good reading for, have quite a few good key points and I learn some new stuff from it too.
Tabletten Online Slapen

shams jee said...

Great info! I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have.Bootsvermietung am Gardasee

jhoney said...

Regalos Originales
This is a nice post in an interesting line of content.Thanks for sharing this article, great way of bring this topic to discussion.

mical3211 said...

The article is good and fit everyone expertise here and can be helpful to others also. This is a good post. This post give truly quality information.I’m definitely going to look into it.
Buy Facebook Likes

kiranjee17 said...

stilnox where to buy
This is a nice post in an interesting line of content.Thanks for sharing this article, great way of bring this topic to discussion.

shams jee said...

You have a real ability for writing unique content. I like how you think and the way you represent your views in this article. I agree with your way of thinking.
salt lake city 2nd hand sporting goods

Sultan Ali said...

This can deemed a fatal mistake that can cost time, money, and sometimes the power to sell the business at all.

Sultan Ali said...

To boost money to start business is not as difficult as most people appear to think. This is especially true when you have an idea that may make you and your backers rich.

hhaniya said...

ferienwohnungen toskana meer
After read a couple of the articles on your website these few days, and I truly like your style of blogging.