- Overview
- Architecture
- KeyValueElement Class
- KeyValue::Transformation
- Integration with Adapters
- Serialization Flow
- KeyValueElement vs XmlElement
- Transformation Compilation
- Adapter Integration Details
- Extension Points
- Testing
- Migration from Hash-Based Serialization
- Common Patterns
- Debugging
- References
- History
Overview
KeyValueDataModel provides an object-oriented intermediate representation for JSON, YAML, and TOML serialization formats in Lutaml::Model.
This architecture achieves symmetric OOP design across all formats, eliminating primitive Hash usage during model transformation and providing clear separation between content definition and presentation rendering.
Purpose
The KeyValueDataModel layer serves three critical purposes:
-
Content/Presentation Separation: Defines WHAT to serialize independent of HOW
-
Type Safety: Eliminates primitive Hash objects during transformation
-
Extensibility: Simplifies adding new formats or customizing behavior
Key Benefits
-
MECE Design: Clear separation of concerns (Model → Transformation → DataModel → Adapter)
-
DRY Principle: Single transformation logic for all key-value formats
-
OOP Throughout: Proper encapsulation with well-defined interfaces
-
Testability: Each layer can be tested independently
-
Backward Compatible: Works alongside legacy Hash-based serialization
Architecture
Three-Layer Design
╔═══════════════════════════════════════════════════════════════════╗
║ UNIFIED KEY-VALUE SERIALIZATION SYSTEM ║
╚═══════════════════════════════════════════════════════════════════╝
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 1: MODEL → KeyValueElement (Content Generation) │
│ │
│ Lutaml::Model::Serializable │
│ │ │
│ ▼ │
│ KeyValue::Transformation.transform() │
│ │ │
│ ▼ │
│ KeyValueElement tree │
│ ├─ name: "person" │
│ ├─ value: nil (for objects) │
│ ├─ children: [KeyValueElement(...), ...] │
│ └─ (or) value: "John Doe" (for leaf values) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 2: KeyValueElement → Adapter Rendering │
│ │
│ Adapter.to_format(kv_element, options) │
│ ├─ Detects KeyValueElement vs Hash │
│ ├─ Builds format-specific structure │
│ └─ Produces final format string │
└─────────────────────────────────────────────────────────────────┘
│
▼
Final Format String
(JSON/YAML/TOML)Symmetric Design with XML
KeyValueDataModel mirrors the proven XmlDataModel architecture:
| Layer | XML Path | Key-Value Path |
|---|---|---|
Model |
|
|
Transformation |
|
|
DataModel |
|
|
Adapters | Nokogiri, Ox, Oga | StandardJson, MultiJson, Oj, StandardYaml, TomlRb, Tomlib |
Output | XML String | JSON/YAML/TOML String |
KeyValueElement Class
Purpose
KeyValueElement is the core data structure representing a single key-value pair or nested object in the intermediate representation.
Class Definition
module Lutaml
module Model
module KeyValueDataModel
class KeyValueElement
attr_accessor :name, :value, :children
def initialize(name, value = nil, children = [])
@name = name
@value = value
@children = children || []
end
def leaf?
children.empty?
end
def object?
!children.empty?
end
end
end
end
endAPI Methods
initialize(name, value = nil, children = [])
Creates a new KeyValueElement instance.
Parameters:
-
name(String): The key name -
value(String|Array|nil): The value for leaf nodes -
children(Array<KeyValueElement>): Child elements for object nodes
Examples:
# Leaf element (primitive value)
leaf = KeyValueElement.new("age", "30")
leaf.leaf? # => true
# Object element (nested structure)
object = KeyValueElement.new(
"person",
nil,
[
KeyValueElement.new("name", "John"),
KeyValueElement.new("age", "30")
]
)
object.object? # => trueUsage Patterns
Simple Key-Value Pair
element = KeyValueElement.new("color", "blue")
# Renders to JSON: {"color": "blue"}
# Renders to YAML: color: blue
# Renders to TOML: color = "blue"Nested Object
address = KeyValueElement.new(
"address",
nil,
[
KeyValueElement.new("street", "123 Main St"),
KeyValueElement.new("city", "Springfield")
]
)
# Renders to JSON:
# {
# "address": {
# "street": "123 Main St",
# "city": "Springfield"
# }
# }Collection (Array)
colors = KeyValueElement.new("colors", ["red", "green", "blue"])
# Renders to JSON: {"colors": ["red", "green", "blue"]}
# Renders to YAML:
# colors:
# - red
# - green
# - blueCollection of Objects
people = KeyValueElement.new(
"people",
nil,
[
KeyValueElement.new(
nil, # Array elements have no name
nil,
[
KeyValueElement.new("name", "John"),
KeyValueElement.new("age", "30")
]
),
KeyValueElement.new(
nil,
nil,
[
KeyValueElement.new("name", "Jane"),
KeyValueElement.new("age", "25")
]
)
]
)
# Renders to JSON:
# {
# "people": [
# {"name": "John", "age": "30"},
# {"name": "Jane", "age": "25"}
# ]
# }KeyValue::Transformation
Purpose
KeyValue::Transformation converts Lutaml::Model::Serializable instances into KeyValueElement trees through compiled transformation rules.
Compilation Process
Transformations are compiled once per model class and cached:
# Compilation happens at class definition or first use
transformation = KeyValue::Transformation.for(ModelClass)
# Compiled rules include:
# - Attribute extraction logic
# - Type transformation
# - Collection handling
# - Nested model recursionTransformation Flow
Model Instance (Ceramic)
├─ type: "Porcelain"
├─ glaze: Glaze instance
└─ colors: ["blue", "white"]
│
▼ KeyValue::Transformation.transform(model, options)
│
▼
KeyValueElement Tree
├─ KeyValueElement("type", "Porcelain")
├─ KeyValueElement("glaze", nil, [
│ ├─ KeyValueElement("color", "Clear")
│ └─ KeyValueElement("temp", "1200")
│ ])
└─ KeyValueElement("colors", ["blue", "white"])Key Methods
self.for(model_class)
Returns (or compiles) the transformation for a model class.
transformation = KeyValue::Transformation.for(Person)Compilation Example
class Person < Lutaml::Model::Serializable
attribute :name, :string
attribute :age, :integer
attribute :address, Address
json do
map "name", to: :name
map "age", to: :age
map "address", to: :address
end
end
# Compilation creates rules:
# 1. Extract :name → apply string type cast → KeyValueElement("name", value)
# 2. Extract :age → apply integer serialization → KeyValueElement("age", value)
# 3. Extract :address → recursively transform Address → KeyValueElement("address", nil, children)Integration with Adapters
Adapter Detection
All adapters support dual-mode operation:
def to_format(root, options = {})
if root.is_a?(KeyValueDataModel::KeyValueElement)
# NEW PATH: KeyValueElement-based serialization
build_from_key_value_element(root)
else
# LEGACY PATH: Hash-based serialization (backward compatible)
build_from_hash(root)
end
endJSON Adapters
StandardJson Adapter
File: lib/lutaml/model/json/standard_adapter.rb
Converts KeyValueElement to JSON using Ruby’s standard JSON library.
adapter = Json::StandardAdapter.new(kv_element)
json_string = adapter.to_json(options)TOML Adapters
Backward Compatibility
Adapters maintain full backward compatibility with Hash-based serialization:
# Both work identically:
# NEW: KeyValueElement-based
Model.to_json # Uses KeyValue::Transformation → KeyValueElement → Adapter
# LEGACY: Hash-based (if transformation not available)
Model.to_json # Uses Hash transformation → AdapterSerialization Flow
Complete Data Flow
┌──────────────────┐
│ Model Instance │
│ Person.new( │
│ name: "John",│
│ age: 30 │
│ ) │
└────────┬─────────┘
│ to_json()
▼
┌──────────────────────────────────────────────────┐
│ Serialize Module │
│ │
│ 1. Check for compiled transformation │
│ 2. Get KeyValue::Transformation.for(Person) │
│ 3. Call transformation.transform(model) │
└────────┬─────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ KeyValue::Transformation Layer │
│ │
│ 1. Extract attribute values from model │
│ 2. Apply transformations (export procs) │
│ 3. Recursively build KeyValueElement tree: │
│ - Person → KeyValueElement("person", nil, [ │
│ KeyValueElement("name", "John"), │
│ KeyValueElement("age", "30") │
│ ]) │
└────────┬─────────────────────────────────────────┘
│ KeyValueElement tree
▼
┌──────────────────────────────────────────────────┐
│ Adapter Layer (StandardJson/Oj/MultiJson/etc) │
│ │
│ 1. Detect KeyValueElement vs Hash │
│ 2. Build format-specific structure: │
│ a. Traverse KeyValueElement tree │
│ b. Convert leaf values to format primitives │
│ c. Build nested objects/arrays │
│ 3. Serialize to format string │
└────────┬─────────────────────────────────────────┘
│ Format String
▼
┌──────────────────┐
│ Final JSON/YAML/ │
│ TOML Output │
└──────────────────┘Entry Points
From Serialize Module
File: lib/lutaml/model/serialize.rb
def to_json(options = {})
# Get or compile transformation
transformation = self.class.transformation_for(:json)
if transformation.is_a?(KeyValue::Transformation)
# NEW PATH: Use KeyValueElement
kv_element = transformation.transform(self, options)
adapter.to_json(kv_element, options)
else
# LEGACY PATH: Use Hash
# ...
end
endFrom Transform Layer
File: lib/lutaml/model/transform/key_value_transform.rb
The KeyValueTransform module provides smart fallback:
module KeyValueTransform
def self.to_hash(model, options = {})
transformation = KeyValue::Transformation.for(model.class)
if transformation
# Use new Transformation
kv_element = transformation.transform(model, options)
# Convert to Hash if needed
else
# Fallback to legacy Hash building
end
end
endKeyValueElement vs XmlElement
Structural Comparison
| Feature | XmlElement | KeyValueElement |
|---|---|---|
Primary Use | XML serialization | JSON/YAML/TOML serialization |
Namespace Support | Yes ( | No (not applicable) |
Attributes | Separate | Part of children (flat structure) |
Content Types | Element, Attribute, Text, CDATA | Key-value pairs only |
Tree Structure | Elements + Attributes + Text nodes | Elements with name/value/children |
Special Features | Mixed content, ordered content | Arrays as first-class values |
Conceptual Mapping
XML Structure: Key-Value Structure:
<person id="123"> {
<name>John</name> "person": {
<age>30</age> "id": "123",
</person> "name": "John",
"age": "30"
}
}
XmlElement("person") KeyValueElement("person", nil, [
├─ XmlAttribute("id") KeyValueElement("id", "123"),
├─ XmlElement("name") KeyValueElement("name", "John"),
└─ XmlElement("age") KeyValueElement("age", "30")
])Design Rationale
Why separate from Hash:
-
Type Safety: KeyValueElement is a proper class with defined interface
-
Testability: Can mock/stub KeyValueElement, not raw Hash
-
Extensibility: Easy to add methods like
leaf?,object? -
Debugging: Clear object type in stack traces
-
Consistency: Mirrors XmlElement architecture
Why NOT reuse XmlElement:
-
Namespace Complexity: XmlElement tied to XML namespace system
-
Attribute Semantics: XmlAttribute not applicable to key-value
-
Format Differences: XML has attributes, key-value doesn’t
-
Simpler Structure: Key-value is flatter than XML tree
Transformation Compilation
Compilation Triggers
Transformations are compiled:
-
On first serialization if not already compiled
-
At class definition via TracePoint hook (eager resolution)
-
Manually via
Model.transformation_for(:json)
Compiled Rules Structure
class CompiledRule
attr_reader :name, :serialized_name, :attribute_type,
:collection?, :transform_export, :transform_import
def apply(model, options)
# 1. Extract attribute value from model
value = model.send(name)
# 2. Apply export transformation if defined
value = transform_export.call(value) if transform_export
# 3. Serialize based on type
if collection?
serialize_collection(value, options)
elsif attribute_type < Lutaml::Model::Serializable
# Recursive transformation
KeyValue::Transformation.for(attribute_type).transform(value, options)
else
# Leaf value serialization
attribute_type.serialize(value)
end
end
endExample Compilation
class Address < Lutaml::Model::Serializable
attribute :street, :string
attribute :city, :string
json do
map "street", to: :street
map "city", to: :city
end
end
class Person < Lutaml::Model::Serializable
attribute :name, :string
attribute :age, :integer
attribute :address, Address
attribute :hobbies, :string, collection: true
json do
map "name", to: :name
map "age", to: :age
map "address", to: :address
map "hobbies", to: :hobbies
end
end
# Compilation creates:
# Rule 1: name (String) → KeyValueElement("name", value)
# Rule 2: age (Integer) → KeyValueElement("age", value.to_s)
# Rule 3: address (Address) → KeyValueElement("address", nil, [children])
# (recursively compiles Address transformation)
# Rule 4: hobbies (Array<String>) → KeyValueElement("hobbies", array)Adapter Integration Details
Detection Logic
All adapters follow this pattern:
def to_json(root, options = {})
if root.is_a?(Lutaml::KeyValue::DataModel::Element)
build_from_key_value_element(root)
elsif root.is_a?(Hash)
build_from_hash(root) # Legacy path
else
raise TypeError, "Expected KeyValueElement or Hash, got #{root.class}"
end
endBuilding from KeyValueElement
Format-Specific Handling
JSON: StandardJson vs Oj
StandardJson: Uses JSON.generate(hash) Oj: Uses Oj.dump(hash, mode: :compat)
Both receive same Hash structure from build_from_key_value_element.
Extension Points
Custom KeyValueElement Subclasses
Create specialized element types:
class AnnotatedKeyValueElement < KeyValueElement
attr_accessor :metadata
def initialize(name, value = nil, children = [], metadata: {})
super(name, value, children)
@metadata = metadata
end
end
# Use in custom transformations:
class CustomTransformation < KeyValue::Transformation
def create_element(name, value, children)
AnnotatedKeyValueElement.new(
name, value, children,
metadata: { source_line: caller_locations.first }
)
end
endCustom Adapters
Implement new format adapters:
class CustomFormatAdapter < Lutaml::Model::SerializationAdapter
def to_custom_format(root, options = {})
case root
when KeyValueDataModel::KeyValueElement
build_from_key_value_element(root)
when Hash
build_from_hash(root)
else
raise TypeError, "Unsupported root type"
end
end
private
def build_from_key_value_element(element)
# Your custom logic here
end
end
# Register adapter:
Lutaml::Model::Config.configure do |config|
config.custom_format_adapter = CustomFormatAdapter
endCustom Transformations
Override transformation behavior:
class CustomKeyValueTransformation < KeyValue::Transformation
def apply_rule(model, value, rule, options)
# Add custom logic before/after standard transformation
result = super
# Post-process if needed
annotate_element(result, model, rule)
result
end
private
def annotate_element(element, model, rule)
# Custom annotation logic
end
endTesting
Unit Tests
KeyValueElement Tests
File: spec/lutaml/model/key_value_data_model/key_value_element_spec.rb
Tests object construction, leaf detection, and tree manipulation.
RSpec.describe KeyValueElement do
it "creates leaf elements" do
leaf = KeyValueElement.new("name", "John")
expect(leaf.leaf?).to be true
expect(leaf.object?).to be false
end
it "creates object elements" do
obj = KeyValueElement.new("person", nil, [
KeyValueElement.new("name", "John")
])
expect(obj.object?).to be true
expect(obj.children.size).to eq 1
end
endTransformation Tests
File: spec/lutaml/model/key_value/transformation_spec.rb
Tests transformation compilation and model-to-KeyValueElement conversion.
RSpec.describe KeyValue::Transformation do
it "transforms simple models" do
model = SimpleModel.new(name: "Test", value: 42)
transformation = described_class.for(SimpleModel)
result = transformation.transform(model, {})
expect(result).to be_a(KeyValueElement)
expect(result.children.size).to eq 2
end
it "handles nested models" do
# Test recursive transformation
end
it "handles collections" do
# Test array attribute transformation
end
endIntegration Tests
Test end-to-end serialization:
RSpec.describe "KeyValueDataModel Integration" do
it "round-trips through JSON" do
original = Person.new(name: "John", age: 30)
json = original.to_json
parsed = Person.from_json(json)
expect(parsed).to eq original
end
it "works with all adapters" do
model = ComplexModel.new(...)
# Test each adapter
expect { model.to_json }.not_to raise_error
expect { model.to_yaml }.not_to raise_error
expect { model.to_toml }.not_to raise_error
end
endMigration from Hash-Based Serialization
Backward Compatibility Strategy
Dual-Mode Operation: All adapters support both KeyValueElement and Hash inputs.
Fallback Mechanism: If KeyValue::Transformation not available, use legacy Hash transformation.
No Breaking Changes: Existing code continues to work without modification.
Performance Considerations
KeyValueElement Benefits:
-
Type Safety: Catch errors at compile-time, not runtime
-
Memory: Single object graph vs nested Hash structures
-
Debugging: Clear stack traces with object types
Minimal Overhead:
-
Transformation compiled once per class (cached)
-
KeyValueElement allocation similar to Hash
-
No performance degradation vs legacy path
Common Patterns
Pattern 1: Simple Key-Value Model
class Config < Lutaml::Model::Serializable
attribute :host, :string
attribute :port, :integer
json do
map "host", to: :host
map "port", to: :port
end
end
# Transformation produces:
# KeyValueElement("config", nil, [
# KeyValueElement("host", "localhost"),
# KeyValueElement("port", "8080")
# ])
# JSON output: {"host":"localhost","port":"8080"}Pattern 2: Nested Objects
class Database < Lutaml::Model::Serializable
attribute :host, :string
attribute :credentials, Credentials
json do
map "host", to: :host
map "credentials", to: :credentials
end
end
class Credentials < Lutaml::Model::Serializable
attribute :username, :string
attribute :password, :string
json do
map "username", to: :username
map "password", to: :password
end
end
# Transformation produces:
# KeyValueElement("database", nil, [
# KeyValueElement("host", "localhost"),
# KeyValueElement("credentials", nil, [
# KeyValueElement("username", "admin"),
# KeyValueElement("password", "secret")
# ])
# ])
# JSON output:
# {
# "host": "localhost",
# "credentials": {
# "username": "admin",
# "password": "secret"
# }
# }Pattern 3: Collections
class Person < Lutaml::Model::Serializable
attribute :name, :string
attribute :emails, :string, collection: true
json do
map "name", to: :name
map "emails", to: :emails
end
end
# Transformation produces:
# KeyValueElement("person", nil, [
# KeyValueElement("name", "John"),
# KeyValueElement("emails", ["john@example.com", "j.doe@example.com"])
# ])
# JSON output:
# {
# "name": "John",
# "emails": ["john@example.com", "j.doe@example.com"]
# }Pattern 4: Polymorphic Collections
class Reference < Lutaml::Model::Serializable
attribute :_class, :string, polymorphic_class: true
attribute :name, :string
json do
map "_class", to: :_class, polymorphic_map: {
"Document" => "DocumentReference",
"Anchor" => "AnchorReference"
}
map "name", to: :name
end
end
class DocumentReference < Reference
attribute :doc_id, :string
json do
map "doc_id", to: :doc_id
end
end
# Transformation handles polymorphic types:
# KeyValueElement("reference", nil, [
# KeyValueElement("_class", "Document"),
# KeyValueElement("name", "Doc 1"),
# KeyValueElement("doc_id", "DOC-001")
# ])
# JSON output:
# {
# "_class": "Document",
# "name": "Doc 1",
# "doc_id": "DOC-001"
# }Debugging
Inspecting KeyValueElement Tree
def print_tree(element, indent = 0)
prefix = " " * indent
if element.leaf?
puts "#{prefix}#{element.name}: #{element.value.inspect}"
else
puts "#{prefix}#{element.name}:"
element.children.each do |child|
print_tree(child, indent + 1)
end
end
end
# Usage:
model = Person.new(name: "John", age: 30)
transformation = KeyValue::Transformation.for(Person)
kv_element = transformation.transform(model, {})
print_tree(kv_element)
# Output:
# person:
# name: "John"
# age: "30"Common Issues
Issue 1: Transformation Not Compiled
Symptom: Adapter receives Hash instead of KeyValueElement
Cause: Transformation not registered for format
Solution: Define mapping block (json/yaml/toml do) in model
References
Core Files
-
lib/lutaml/model/key_value_data_model.rb- Module definition -
lib/lutaml/model/key_value_data_model/key_value_element.rb- Element class -
lib/lutaml/model/key_value/transformation.rb- Transformation engine -
lib/lutaml/model/serialize.rb- Integration point -
lib/lutaml/model/transform/key_value_transform.rb- Legacy bridge
History
Session 217: Initial Implementation
-
Created
KeyValueElementclass -
Created
KeyValue::Transformationengine -
Established compilation mechanism
-
45/45 tests passing