REXML parsing error

[Jim Ruther Nill]

Using REXML as parser, < inside CDATA is converted to <. Nokogiri does not have this issue.

irb(main):030:0> Nori.new(parser: :rexml).parse("<lixi_payload><![CDATA[<RealEstate Zoning=\"Residential (Development/Existing) &lt;=6 units/dwellings\"></RealEstate>]]></lixi_payload>")
=> {"lixi_payload"=>"<RealEstate Zoning=\"Residential (Development/Existing) <=6 units/dwellings\"></RealEstate>"}

irb(main):031:0> Nori.new(parser: :nokogiri).parse("<lixi_payload><![CDATA[<RealEstate Zoning=\"Residential (Development/Existing) &lt;=6 units/dwellings\"></RealEstate>]]></lixi_payload>")
=> {"lixi_payload"=>"<RealEstate Zoning=\"Residential (Development/Existing) &lt;=6 units/dwellings\"></RealEstate>"}

[Olle Jonsson]

@jvnill Thanks for pointing out this difference.

This StackOverflow thread has good discussion of the differences.

[Jim Ruther Nill]

Thanks for the prompt response @olleolleolle. It's not actually the REXML parser that is the issue. It's the Nori::Parser::REXML wrapper

require "rexml/document"

class Nori
  module Parser

    # = Nori::Parser::REXML
    #
    # REXML pull parser.
    module REXML

      def self.parse(xml, options)
        stack = []
        parser = ::REXML::Parsers::BaseParser.new(xml)

        while true
          event = unnormalize(parser.pull)
          case event[0]
          when :end_document
            break
          when :end_doctype, :start_doctype
            # do nothing
          when :start_element
            stack.push Nori::XMLUtilityNode.new(options, event[1], event[2])
          when :end_element
            if stack.size > 1
              temp = stack.pop
              stack.last.add_node(temp)
            end
          when :text, :cdata
            stack.last.add_node(event[1]) unless event[1].strip.length == 0 || stack.empty?
          end
        end
        stack.length > 0 ? stack.pop.to_hash : {}
      end

      def self.unnormalize(event)
        event.map! do |el|
          if el.is_a?(String)
            ::REXML::Text.unnormalize(el)
          elsif el.is_a?(Hash)
            el.each {|k,v| el[k] = ::REXML::Text.unnormalize(v)}
          else
            el
          end
        end
      end

unnormalize performs the conversion to unescaped characters but this shouldn't be the case for text inside CDATA.

[Jim Ruther Nill]

submitted #94