Friday, October 16, 2009

Nokogiri Performance: xpath vs. tree walking/iterating

At Gnip we're doing some heavy XML parsing in Ruby and obviously chose Nokogiri as the underlying engine. I started the week doing xpath searches to tease out the elements/attributes from the documents we were parsing, and ended the week iterating over the root's children in order to achieve significant (~3x) performance improvements. While xpath is convenient, as you don't have to pay attention to document structure (assuming you have your head around xpath syntax to begin with), it's horribly expensive in terms of processing time. It's truly searching the document for what you want; expensive!

There's a big difference between using the "search" (e.g. xpath) interface on top of a parsed DOM and running over the entire tree, testing each node for what it is you're looking for. Code gets a little uglier when doing the latter, it's not as elegant/clean, but performance starts kicking in when you do it. Moving from xpath search-style parsing, to tree walking yielded ~3x performance improvement in parsing for me. I suspect that going all the way to either Nokogiri's Reader or SAX interface would yield an additional 10% improvement over that. However, I'm stoping here for now as the readability/complexity detriment in doing a full Reader/SAX stack-maintenance model doesn't feel worth it at the moment. I would like to try pauldix's SAX machine declarative parser (built on Nokogiri) and run some benchmarks, but... another day.

Old:

doc = Nokogiri::XML.parse(some_xml_string) { |cfg| cfg.noblanks }

doc.xpath('/xmlns:feed/xmlns:entry').each do |entry|
at = entry.xpath('xmlns:published').text
id = entry.xpath('xmlns:id').text
...
end


New:

doc = Nokogiri::XML.parse(some_xml_string) { |cfg| cfg.noblanks }

doc.root.children.each do |node|
if node.name == 'entry'
node.children.each do |sub_e|
at = sub_e.inner_text if sub_e.name == 'published'
id = sub_e.inner_text if sub_e.name == 'id'
end
end
...
end

76 comments:

justin said...
This comment has been removed by the author.
justin said...

i'm a little late to the party, but hey, thanks so much for posting this. In my case, it was a factor 10 speed up. this probably means nokogiri's xpath implementation could be optimized further. maybe with indices?

Anonymous said...

generic xanax drug test results for xanax - 4mg xanax high

Anonymous said...

generic xanax xanax buy online no prescription uk - buy generic xanax online no prescription

Anonymous said...

xanax online xanax bars 3mg - what type of drug is xanax

Anonymous said...

buy tramadol cod buy tramadol in europe - buy tramadol usa

Anonymous said...

buy tramadol online best place buy tramadol online - generic tramadol online

Anonymous said...

xanax online xanax effects relationships - xanax withdrawal caffeine

Anonymous said...

buy tramadol online long does 50mg tramadol take kick - tramadol high forum

Anonymous said...

buy carisoprodol carisoprodol wp 5901 - buy carisoprodol online

Anonymous said...

xanax online xanax effects and dosage - generic xanax gg 258

Anonymous said...

order tramadol no prescription tramadol 50 mg dosage - buy tramadol online cod only

Anonymous said...

buy tramadol online tramadol 50mg tablets used - tramadol 200mg

Anonymous said...

buy carisoprodol is soma 350 mg a narcotic - carisoprodol 350 mg tablet qua

Anonymous said...

carisoprodol 350 mg soma carisoprodol drug test - what is soma carisoprodol

Anonymous said...

buy tramadol online tramadol normal dose - tramadol addiction potential

Anonymous said...

buy tramadol online cheap tramadol no prescription overnight - tramadol 50mg hcl high

Anonymous said...

xanax online xanax quad bars - xanax and alcohol death

Anonymous said...

generic carisoprodol carisoprodol soma side effects - buy soma carisoprodol online

Anonymous said...

how to buy xanax online legally xanax rectal - xanax bars drug info

Anonymous said...

buy carisoprodol cheap carisoprodol soma wiki - order carisoprodol online

Anonymous said...

xanax no rx xanax uses - taking 4 xanax bars

Anonymous said...

buy xanax valium online florida xanax 832 - buy xanax in china

Anonymous said...

xanax online seroquel xanax drug interactions - xanax effects thyroid

Anonymous said...

xanax online xanax kratom - generic xanax 0.25

Anonymous said...

buy carisoprodol carisoprodol 350 mg street value - carisoprodol tablets dosage

Anonymous said...

buy tramadol online tramadol withdrawal+muscle aches - addiction of tramadol

Anonymous said...

buy cialis online cialis daily from usa - can you buy cialis over the counter in spain

Anonymous said...

buy tramadol online buy tramadol online echeck - buy tramadol online usa

Anonymous said...

xanax online can you buy xanax online no prescription - xanax drug class

Anonymous said...

tadalafil no prescription cialis price dubai - buy cialis 100mg

Anonymous said...

cialis online where to buy cialis no prescription - buy cialis daily use

Anonymous said...

xanax online buy xanax online usa - xanax online real

Anonymous said...

buy cialis online cialis levitra reviews - cialis 5mg daily reviews

Anonymous said...

xanax online xanax bars u94 - xanax effects synapse

Anonymous said...

cialis online cialis 2.5 - cialis refractory period

Anonymous said...

cialis online cheap cialis in australia - cialis online-versand

Anonymous said...

buy tramadol tramadol and breastfeeding - buy tramadol no rx

Anonymous said...

http://buytramadolonlinecool.com/#28875 tramadol for dogs what dosage - order tramadol 180

Anonymous said...

buy tramadol tablets tramadol hydrochloride 50mg tablets - tramadol online texas

Anonymous said...

http://buytramadolonlinecool.com/#56411 tramadol overdose dosage - is it illegal to purchase tramadol online

Anonymous said...

20000 :) Generic Azithromycin - order azithromycin online http://www.zithromaxhowtobuy.net/#order-zithromax, [url=http://www.zithromaxhowtobuy.net/#generic-azithromycin]Azithromycin No Prescription[/url]

Anonymous said...

20000 :) Buy Azithromycin - zithromax pills http://www.zithromaxhowtobuy.net/#generic-azithromycin, [url=http://www.zithromaxhowtobuy.net/#generic-azithromycin]Cheap Azithromycin[/url]

Anonymous said...

http://landvoicelearning.com/#30896 tramadol 50mg for animals - buy tramadol cod online

Anonymous said...

buy tramadol online tramadol 50mg not working - drug study tramadol generic name

Anonymous said...

buy tramadol tramadol iv - tramadol online no prescription overnight delivery

Anonymous said...

http://blog.dawn.com/dblog/buy/#34852 tramadol can you snort - tramadol clorhidrato 50 mg para que sirve

Anonymous said...

http://landvoicelearning.com/#30896 tramadol for dogs incontinence - tramadol cause high

Anonymous said...

buy tramadol no prescription tramadol for dogs human consumption - can you buy tramadol in greece

Anonymous said...

http://blog.dawn.com/dblog/buy/#blog tramadol ultracet dosage - tramadol hydrochloride 100 mg

Anonymous said...

buy tramadol tramadol for dogs human use - buytramadol.eu

Anonymous said...

ways to buy ativan online buy lorazepam without prescriptions - ativan side effects breathing

Anonymous said...

ways to buy ativan online 6 mg ativan too much - ativan sublingual lorazepam overdose

Anonymous said...

lorazepam 1mg buying ativan online in canada - ativan overdose pediatrics

Anonymous said...

purchase tramadol tramadol 50 mg side effects - buy tramadol online australia

Anonymous said...

buy xanax online xanax 2mg pill identifier - xanax withdrawal erowid

Anonymous said...

xanax generic xanax mg - xanax vs ativan

Anonymous said...

buy tramadol online where to buy tramadol in usa - why can you buy tramadol online

Anonymous said...

buy xanax online long does take 1mg xanax work - 0.5mg of alprazolam

Anonymous said...

buy tramadol online tramadol for dogs can humans take it - tramadol withdrawal how long does it last

Anonymous said...

Coal-black Rhinoceros - a adipose and stalwart animal. he did not as rotund as the bloodless rhinoceros, but still impressive - reaches the power 2-2, 2 m, lengths of up to 3, 15 m in height shoulders of 150-160 cm.

Anonymous said...

buy tramadol online without a prescription tramadol for dogs generic - tramadol hcl shelf life

Anonymous said...

order tramadol overnight tramadol 50 mg mfg zydus - buy tramadol online overnight usa

Anonymous said...

Hey there, I think your blog might be having browser compatibility issues.
When I look at your blog in Opera, it looks fine but when opening in Internet Explorer, it
has some overlapping. I just wanted to give you a quick heads up!
Other then that, terrific blog!

Here is my web page :: cheap dental implants

Anonymous said...

buy xanax online xanax 2 mg withdrawal - xanax bars blue lotus review

Anonymous said...

Thanks for the good writeup. It in reality was once a enjoyment account it.

Look complicated to far delivered agreeable from you!
By the way, how could we keep in touch?

Feel free to surf to my web blog Root Canal

Anonymous said...

Hello, just wanted to mention, I loved this blog post.
It was funny. Keep on posting!

Here is my webpage: Cheap Dental Implants

Anonymous said...

Uses and Benefits of Rubber Mulch Guard Against The Dangers
Of Toxic Mulch

Also visit my webpage; tree nursery Milwaukee

Anonymous said...

viagra online cialis cialis 10 mg

Anonymous said...

cialis comprar viagra viagra cialis 5mg cialis

Anonymous said...

comprar cialis cialis 5mg tadalafil cialis generique

Anonymous said...

URL acheter cialis generique acheter cialis

Anonymous said...

buy tramadol for dogs buy tramadol cod - tramadol online pharmacy

Anonymous said...

cheap cialis online cheap cialis cialis online buy cialis 5mg

Anonymous said...

buy herbal ambien buy ambien online no rx - buy ambien online us pharmacy

Anonymous said...

order 180 tramadol cod or order tramadol forum or can you buy tramadol onlinelegally or order tramadol with american express or tramadol tropfen online kaufen or tramadol 100mg online overnight or cheap tramadol online