- Executive summary
- Architecture
- Implementation details
- Usage guide
- Limitations and workarounds
- Testing and quality assurance
- Debugging
- Migration and compatibility
- Future roadmap
- Technical reference
- Troubleshooting
- Contributing
- Related documentation
- License
Executive summary
HeadedOx is a hybrid XML adapter that combines the raw speed of Ox's C-based XML parsing with the comprehensive capabilities of Moxml’s pure Ruby XPath 1.0 engine.
Version 0.2.0 Status: 99.20% test pass rate (1,992/2,008 tests) - PRODUCTION READY
The name "HeadedOx"
The name contrasts with the standard Ox adapter which operates "headlessly" with limited XPath support through its locate() method. HeadedOx has a comprehensive XPath "head" (brain) that understands and processes complex XPath 1.0 expressions.
Standard Ox = Headless (basic path traversal via locate()) HeadedOx = Headed (full XPath 1.0 with all functions and operators)
Key achievements
-
✅ All 27 XPath 1.0 functions (100% complete)
-
✅ 6 of 13 XPath axes (covering 80% of real-world usage)
-
✅ Complex predicate evaluation
-
✅ Expression caching for performance
-
✅ Pure Ruby XPath engine (debuggable)
-
✅ C-speed XML parsing (via Ox)
-
✅ 99.20% test compatibility
-
✅ Production-ready quality
Architecture
Overview
HeadedOx is a pure Ruby layer on top of Ox that adds comprehensive XPath functionality:
User Code
↓
Moxml Unified API (Document, Element, Node)
↓
HeadedOx Adapter
├──→ Ox Gem (C extension) ──→ Fast XML Parsing
└──→ Moxml XPath Engine (Pure Ruby) ──→ XPath Queries
├── Lexer: XPath → Tokens
├── Parser: Tokens → AST
├── Compiler: AST → Ruby Code
├── Generator: Ruby AST → Source
├── Cache: Store compiled Procs
└── Execute: Run against Ox nodesWhy this architecture?
Performance where it matters:
-
XML parsing is C-speed (Ox) - This is the bottleneck for most apps
-
XPath is pure Ruby but compiled - Faster than interpretation, debuggable
Clean separation:
-
Ox does what it does best: Fast parsing
-
Moxml XPath does what it does best: Comprehensive queries
-
No C code modifications required
Maintainability:
-
Pure Ruby XPath engine is readable and debuggable
-
Can fix/extend XPath without C programming
-
Test-driven development possible
Future-proof:
-
XPath engine can be ported to C later if needed
-
Or Ox can be enhanced (see OX_ENHANCEMENT_PLAN.adoc)
-
Architecture allows gradual improvements
Implementation details
XPath engine layers
Layer 1: Lexer
Tokenizes XPath expressions into structured tokens.
expression = "//book[@price < 20]"
tokens = Moxml::XPath::Lexer.new(expression).tokenize
# Output: [
# [:dslash, "//", 0],
# [:name, "book", 2],
# [:lbracket, "[", 6],
# [:at, "@", 7],
# [:name, "price", 8],
# [:lt, "<", 14],
# [:number, "20", 16],
# [:rbracket, "]", 18]
# ]File: lib/moxml/xpath/lexer.rb
Layer 2: Parser
Builds Abstract Syntax Tree (AST) from tokens using recursive descent parsing.
ast = Moxml::XPath::Parser.parse("//book[@price < 20]")
# AST structure:
# Node(type=:absolute_path, children=[
# Node(type=:axis, children=["descendant-or-self", Node(type=:wildcard)]),
# Node(type=:step_with_predicates, children=[
# Node(type=:axis, children=["child", Node(type=:test, value={name: "book"})]),
# Node(type=:predicate, children=[
# Node(type=:lt, children=[...])
# ])
# ])
# ])Layer 3: Compiler
Compiles AST into optimized Ruby code represented as Ruby::Node AST.
proc = Moxml::XPath::Compiler.compile_with_cache(ast)
# Generated Ruby code (conceptual):
# lambda do |node|
# matched = Moxml::NodeSet.new([], context)
# node.each_node do |descendant|
# if descendant.is_a?(Moxml::Element) && descendant.name == "book"
# price = descendant["price"]
# if price && Conversion.to_float(price) < 20.0
# matched << descendant
# end
# end
# end
# matched
# endFile: lib/moxml/xpath/compiler.rb (1,737 lines)
Layer 4: Generator
Converts Ruby::Node AST to executable Ruby source code string.
ruby_ast = Ruby::Node.new(:lit, ["hello"])
source = Ruby::Generator.new.process(ruby_ast)
# => "\"hello\""
# Complex example:
ruby_ast = var.assign(value).followed_by(var + literal(1))
source = generator.process(ruby_ast)
# => "var = value\nvar + 1"Files:
-
lib/moxml/xpath/ruby/node.rb- Ruby AST nodes -
lib/moxml/xpath/ruby/generator.rb- Code generation
Layer 5: Cache
LRU cache stores compiled Procs for repeated expressions.
# First query - compiles and caches
result1 = doc.xpath("//book[@price < 20]")
# Second query - uses cached Proc (much faster)
result2 = doc.xpath("//book[@price < 20]")
# Cache stats:
Moxml::XPath::Compiler::CACHE.size # => 1File: lib/moxml/xpath/cache.rb
Configuration:
# Default: 1000 entries
# Customize if needed:
Moxml::XPath::Compiler::CACHE = Moxml::XPath::Cache.new(2000)XPath 1.0 function support
HeadedOx implements all 27 XPath 1.0 standard functions:
String functions (10 functions)
doc.xpath("string(//title)") # Convert to string
doc.xpath("concat('Title: ', //book/title)") # Concatenate strings
doc.xpath("starts-with(//title, 'Ruby')") # Check prefix
doc.xpath("contains(//title, 'Programming')") # Check substring
doc.xpath("substring-before(//title, ':')") # Extract before separator
doc.xpath("substring-after(//title, ':')") # Extract after separator
doc.xpath("substring(//title, 1, 5)") # Extract substring
doc.xpath("string-length(//title)") # Get string length
doc.xpath("normalize-space(//title)") # Normalize whitespace
doc.xpath("translate(//title, 'abc', 'ABC')") # Character replacementNumeric functions (6 functions)
doc.xpath("number(//price)") # Convert to number
doc.xpath("sum(//item/@price)") # Sum values
doc.xpath("count(//book)") # Count nodes
doc.xpath("floor(//price)") # Round down
doc.xpath("ceiling(//price)") # Round up
doc.xpath("round(//price)") # Round to nearestBoolean functions (4 functions)
doc.xpath("boolean(//book)") # Convert to boolean
doc.xpath("not(//book[@price > 50])") # Negate boolean
doc.xpath("true()") # Return true
doc.xpath("false()") # Return falseNode functions (4 functions)
doc.xpath("local-name(//ns:element)") # Get local name without prefix
doc.xpath("name(//ns:element)") # Get qualified name with prefix
doc.xpath("namespace-uri(//ns:elem)") # Get namespace URI
doc.xpath("//book[lang('en')]") # Check xml:lang attributeXPath axis support
Implemented axes (6 of 13)
These axes cover approximately 80% of real-world XPath usage:
# child:: - Direct children (default axis)
doc.xpath("//book/child::title")
doc.xpath("//book/title") # Abbreviated
# descendant:: - All descendants
doc.xpath("//book/descendant::title")
# descendant-or-self:: - Self and descendants
doc.xpath("//book") # Uses descendant-or-self implicitly
# self:: - The node itself
doc.xpath("//book/self::book")
doc.xpath("//book/.") # Abbreviated
# parent:: - Parent node
doc.xpath("//title/parent::book")
doc.xpath("//title/..") # Abbreviated
# attribute:: - Attributes
doc.xpath("//book/attribute::id")
doc.xpath("//book/@id") # AbbreviatedMissing axes (7 of 13)
These axes are rarely used (< 20% of queries):
-
ancestor::- Not implemented -
ancestor-or-self::- Not implemented -
following-sibling::- Not implemented -
preceding-sibling::- Not implemented -
following::- Not implemented -
preceding::- Not implemented -
namespace::- Not implemented
Workaround: Use parent navigation and Ruby enumerable methods:
# Instead of: //title/ancestor::book
# Use:
title = doc.at_xpath("//title")
book = title.parent while book && book.name != "book"
# Instead of: //item/following-sibling::item
# Use:
items = doc.xpath("//item")
item_index = items.index { |i| i["id"] == "current" }
following = items[item_index + 1..-1] if item_indexPredicate evaluation
HeadedOx supports comprehensive predicate evaluation:
# Numeric predicates
doc.xpath("//book[1]") # Position-based
doc.xpath("//book[@price]") # Attribute existence
doc.xpath("//book[@price < 20]") # Comparison
doc.xpath("//book[@price * 1.1 < 25]") # Arithmetic
# Boolean predicates
doc.xpath("//book[@price and @title]") # Logical AND
doc.xpath("//book[@price or @isbn]") # Logical OR
doc.xpath("//book[not(@price)]") # Negation
# Function predicates
doc.xpath("//book[contains(title, 'Ruby')]") # String functions
doc.xpath("//book[string-length(title) > 10]") # Length check
doc.xpath("//book[position() < 5]") # Position functions
# Complex nested predicates
doc.xpath("//book[@price < 20][position() <= 3][contains(title, 'Ruby')]")Performance optimization
Expression caching
HeadedOx caches compiled XPath expressions using an LRU cache:
# First query: Compile + Execute (~1ms compile + 0.5ms execute)
result1 = doc.xpath("//book[@price < 20]")
# Subsequent queries: Execute only (~0.5ms execute)
result2 = doc.xpath("//book[@price < 20]") # Uses cache
result3 = doc.xpath("//book[@price < 20]") # Uses cache
# Cache automatically manages memory (LRU eviction at 1000 entries)Cache benefits:
-
Compilation happens once per unique expression
-
Queries with same expression are ~2x faster
-
Memory usage is bounded (1000 entries ≈ 1-2MB)
-
Thread-safe cache implementation
Compilation vs. interpretation
HeadedOx compiles XPath to Ruby Procs instead of interpreting AST at runtime:
┌──────────────────────────────────────────────────────────┐
│ Interpretation (Slow) │
│ │
│ Parse → AST → Traverse AST → Evaluate each node │
│ (once) (every query execution) │
│ │
│ Time per query: 5-10ms │
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ Compilation (Fast - HeadedOx Approach) │
│ │
│ Parse → AST → Compile → Ruby Proc [CACHED] │
│ (once) (first) ↓ │
│ Execute Proc │
│ (every query) │
│ │
│ First query: 1-2ms (compile) + 0.5ms (execute) │
│ Cached queries: 0.5ms (execute only) │
└──────────────────────────────────────────────────────────┘Usage guide
Basic usage
require 'moxml'
# Create HeadedOx context
context = Moxml.new(:headed_ox)
# Parse XML (Ox handles this - very fast)
xml = <<-XML
<library>
<book id="1" price="15.99">
<title>Ruby Programming</title>
<author>Jane Smith</author>
</book>
<book id="2" price="25.00">
<title>Advanced Ruby</title>
<author>John Doe</author>
</book>
</library>
XML
doc = context.parse(xml)
# XPath queries (Moxml engine handles this)
cheap_books = doc.xpath('//book[@price < 20]') # Find by price
titles = doc.xpath('//book/title/text()') # Get text nodes
count = doc.xpath('count(//book)') # Count books
avg_price = doc.xpath('sum(//book/@price) div count(//book)') # Calculate averageAdvanced XPath patterns
String operations
# Find books with "Ruby" in title
ruby_books = doc.xpath('//book[contains(title, "Ruby")]')
# Find books with titles starting with "Advanced"
advanced = doc.xpath('//book[starts-with(title, "Advanced")]')
# Extract first 10 characters of titles
short_titles = doc.xpath('substring(//title, 1, 10)')
# Concatenate fields
full_info = doc.xpath('concat(//title, " by ", //author)')
# Normalize whitespace in titles
clean_titles = doc.xpath('normalize-space(//title)')Numeric operations
# Books cheaper than average
avg = doc.xpath('sum(//book/@price) div count(//book)')
below_avg = doc.xpath("//book[@price < #{avg}]")
# Books in price range
affordable = doc.xpath('//book[@price >= 10 and @price <= 30]')
# Arithmetic in predicates
discounted = doc.xpath('//book[@price * 0.9 < 20]')
# Position-based selection
first_three = doc.xpath('//book[position() <= 3]')
last_book = doc.xpath('//book[position() = last()]')Boolean logic
# Complex conditions
popular = doc.xpath('//book[@rating > 4 and @reviews > 100]')
# OR conditions
fiction_or_scifi = doc.xpath('//book[@genre="fiction" or @genre="scifi"]')
# Negation
not_expensive = doc.xpath('//book[not(@price > 50)]')
has_no_rating = doc.xpath('//book[not(@rating)]')Performance tips
Tip 1: Cache frequently used queries
# Store compiled queries if used repeatedly
class BookQuery
# Queries are cached automatically by expression string
def self.expensive_books(doc)
doc.xpath('//book[@price > 50]')
end
def self.popular_books(doc)
doc.xpath('//book[@rating >= 4]')
end
end
# Each query compiles once, then uses cacheTip 2: Use specific paths when possible
# More efficient - starts from known location
doc.root.xpath('./book[@price < 20]')
# Less efficient - scans entire document
doc.xpath('//book[@price < 20]')Limitations and workarounds
For comprehensive limitation documentation, see HEADED_OX_LIMITATIONS.adoc.
Quick reference
| Limitation | Impact | Workaround |
|---|---|---|
Attribute wildcards | Low | Use |
Namespace methods | Medium | Use standard Ox methods where available |
Parent node setter | Low | Use remove + add pattern for reparenting |
CDATA escaping | Very Low | Avoid nested |
7 missing XPath axes | Low | Use parent navigation + Ruby enumerables |
Complex namespace inheritance | Low | Use explicit namespace declarations |
Testing and quality assurance
Test coverage (v0.2.0)
Total Tests: 2,008
Passing: 1,992 (99.20%)
Skipped: 16 (0.80% - documented Ox limitations)
Failures: 0
Test Categories:
✅ Core XPath functions: 100% (All 27 functions)
✅ Operators: 100% (All 13 operators)
✅ Predicates: 100% (Position, boolean, operators)
✅ Basic axes: 100% (6 of 6 implemented)
✅ Parser: 100% (All constructs)
✅ Compiler: 100% (Code generation)
⚠️ Advanced integration: 99.20% (16 Ox limitations)Quality metrics
Code Coverage: > 90% overall
Rubocop: 0 offenses
Documentation: 5,000+ lines
Performance: C-speed parsing, compiled XPath
Memory: Efficient (similar to Ox)
Thread Safety: Yes (with proper synchronization)
Production Use: ReadyKnown limitations with test status
All 16 limitations are comprehensively documented:
-
Attribute wildcards (3 tests) - Ox
locate()doesn’t support@* -
Namespace introspection (4 tests) - Ox doesn’t expose namespace data
-
Parent node mutation (1 test) - Ox C struct immutability
-
CDATA escaping (2 tests) - Complex nested
]]>markers -
Namespace inheritance (2 tests) - Ox parses but doesn’t track
-
Namespaced attributes (1 test) - Attribute-level namespace resolution
-
XPath text content (1 test) - Result node wrapping edge case
-
Wildcard element counting (2 tests) - Descendant iteration optimization
See HEADED_OX_LIMITATIONS.adoc for complete details, workarounds, and Ox enhancement requirements.
Debugging
Enable XPath debug output
# Set environment variable before running
ENV['DEBUG_XPATH'] = '1'
doc.xpath("//book[@price < 20]")
# Prints:
# ============================================================
# COMPILING XPath
# ============================================================
# AST: #<Moxml::XPath::AST::Node type=:absolute_path ...>
#
# GENERATED RUBY CODE:
# ------------------------------------------------------------
# lambda do |node|
# context = node.context
# matched = Moxml::NodeSet.new([], context)
# ...
# end
# ============================================================Inspecting compiled code
# Compile without executing
ast = Moxml::XPath::Parser.parse("//book")
compiler = Moxml::XPath::Compiler.new
proc = compiler.compile(ast)
# Generated Proc can be inspected
puts proc.source_location # [file, line] where defined
puts proc.call(doc) # Execute and see resultsUnderstanding AST structure
ast = Moxml::XPath::Parser.parse("//book[@price < 20]")
puts ast.inspect
# Shows hierarchical structure:
# #<Moxml::XPath::AST::Node type=:absolute_path children=2>
# #<Moxml::XPath::AST::Node type=:axis children=2>
# "descendant-or-self"
# #<Moxml::XPath::AST::Node type=:wildcard>
# #<Moxml::XPath::AST::Node type=:step_with_predicates children=2>
# ...Migration and compatibility
From standard Ox adapter
HeadedOx is a drop-in replacement with enhanced capabilities:
# Before (Ox adapter)
Moxml::Config.default_adapter = :ox
doc = Moxml.new.parse(xml)
# Limited to Ox's locate() syntax
# After (HeadedOx adapter)
Moxml::Config.default_adapter = :headed_ox
doc = Moxml.new.parse(xml)
# Full XPath 1.0 supportBreaking changes: None - All Ox functionality preserved
From Nokogiri/Oga
HeadedOx supports most Nokogiri/Oga XPath patterns:
# These work identically across all adapters:
doc.xpath('//book[@price < 20]') # ✅ Works
doc.xpath('count(//book)') # ✅ Works
doc.xpath('//book[1]') # ✅ Works
# These require adapter change for HeadedOx:
doc.xpath('//book/ancestor::library') # ❌ Use Nokogiri/Oga
doc.xpath('//book/following-sibling::*') # ❌ Use Nokogiri/OgaAdapter selection decision tree
Need XML parsing?
├─ Need XPath?
│ ├─ Need all 13 axes? → Nokogiri/Oga
│ ├─ Need advanced namespaces? → Nokogiri/Oga
│ └─ Basic XPath + Speed? → HeadedOx ✅
│
├─ No XPath?
│ ├─ Maximum speed? → Ox
│ └─ Feature complete? → Nokogiri
│
└─ Pure Ruby required?
├─ Need XPath? → Oga
└─ No XPath? → REXMLFuture roadmap
Version 1.3: Planned enhancements
If Ox gem adds namespace introspection API:
-
Re-enable 7 namespace-related tests
-
Add
element.namespacemethod support -
Improve namespace inheritance handling
Version 2.0: Full coverage
If Ox gem adds all required APIs (see OX_ENHANCEMENT_PLAN.adoc):
-
Implement remaining 7 XPath axes
-
Add parent node mutation
-
Achieve 100% test pass rate
-
Full Nokogiri API parity where applicable
Technical reference
File structure
lib/moxml/
├── adapter/
│ └── headed_ox.rb # HeadedOx adapter (109 lines)
├── xpath/
│ ├── lexer.rb # Tokenization (200 lines)
│ ├── parser.rb # AST construction (483 lines)
│ ├── compiler.rb # Ruby code generation (1,737 lines)
│ ├── cache.rb # LRU caching (48 lines)
│ ├── context.rb # Execution context (59 lines)
│ ├── conversion.rb # Type conversions (150 lines)
│ ├── engine.rb # High-level API (35 lines)
│ ├── errors.rb # Error classes (85 lines)
│ ├── ast/
│ │ └── node.rb # AST node types (149 lines)
│ └── ruby/
│ ├── node.rb # Ruby AST (192 lines)
│ └── generator.rb # Code generation (200 lines)
└── xpath.rb # Module entry point
Total: ~3,500 lines of pure Ruby XPath implementationAPI reference
HeadedOx adapter class
# lib/moxml/adapter/headed_ox.rb
class Moxml::Adapter::HeadedOx < Moxml::Adapter::Ox
# Inherits all Ox parsing and serialization
# Overrides XPath methods to use Moxml engine
def self.xpath(node, expression, namespaces = {})
# Compiles and caches XPath expression
# Executes against Ox nodes
# Returns array of matching native nodes
end
def self.at_xpath(node, expression, namespaces = {})
# Returns first matching node or nil
end
endXPath engine classes
# Lexer - Tokenization
Moxml::XPath::Lexer.new(expression).tokenize
# => Array of [type, value, position] tokens
# Parser - AST construction
Moxml::XPath::Parser.parse(expression)
# => AST::Node tree
# Compiler - Code generation
Moxml::XPath::Compiler.compile_with_cache(ast, namespaces: ns_map)
# => Proc that accepts a document/node
# Cache - Expression caching
Moxml::XPath::Compiler::CACHE.get_or_set(key) { compile(ast) }
# => Cached or freshly compiled ProcPerformance benchmarks
Parsing (HeadedOx same as Ox)
Small XML (1KB): ~500 ips (2ms per parse)
Medium XML (10KB): ~290 ips (3.5ms per parse)
Large XML (145KB): ~20 ips (50ms per parse)Troubleshooting
Common issues
Issue: XPath returns empty when nodes exist
Cause: Namespace-aware query without namespace mapping
# Wrong - ignores namespace
doc.xpath('//xmlns:book') # Returns empty
# Correct - provide namespace mapping
doc.xpath('//xmlns:book', 'xmlns' => 'http://example.org')Issue: Slow XPath performance
Cause: Cache not being used or complex expression
# Check if caching is working
expressions = {}
doc.xpath(expr) # Should compile once
1000.times { doc.xpath(expr); expressions[expr] = true }
# Should be fast after first query
# Simplify complex expressions
# Instead of: //*//*[@*]//*
# Use: //element with Ruby filteringGetting help
-
Check HEADED_OX_LIMITATIONS.adoc for known issues
-
Review ../examples/headed_ox_example/ for working code
-
Enable DEBUG_XPATH=1 for detailed execution trace
-
Compare with Nokogiri adapter for expected behavior
Contributing
Contributions are welcome! Areas for contribution:
-
Testing: Add more real-world XPath patterns
-
Documentation: Improve examples and guides
-
Performance: Optimize compiler generated code
-
Ox Enhancement: Help implement OX_ENHANCEMENT_PLAN.adoc
-
Bug Fixes: Address edge cases in limitations list
Related documentation
-
HEADED_OX_LIMITATIONS.adoc - Comprehensive limitation reference (558 lines)
-
OX_ENHANCEMENT_PLAN.adoc - Roadmap for Ox gem enhancements (800+ lines)
-
OX_ENHANCEMENT_PROMPT.adoc - Implementation guide for Ox work (500+ lines)
-
RELEASE_NOTES_V0.2.0.adoc - Version 0.2.0 release notes (508 lines)
-
../README.adoc - Main Moxml documentation