Class: Ibex::Runtime::Parser

Inherits:
Object
  • Object
show all
Includes:
Observation, ParserSyncRecovery
Defined in:
lib/ibex/runtime/parser.rb

Overview

Drives a table-defined LR parser without native extensions.

Subclasses provide .parser_tables, returning :tokens, :token_names, :actions, :gotos, and :productions, with optional :default_actions, :error_messages, and :recovery_sync_tokens. Actions are represented by [:shift, state], [:reduce, production], [:accept], or [:error]. Format-v2 and newer generated production entries mark their five-argument semantic methods with location_action: true. Format-v3 and newer composed actions additionally use composition_action: true for the six-argument contract carrying the lookahead location. Format-v4 location-free generated methods use values_action: true for a one-argument values contract. Format-v5 additionally marks proven-safe zero-to-four-value methods with positional_action: true. V1 and unmarked application actions retain the historical two-argument contract. Markers are honored only for the generated _ibex_action_N Symbol shape, never for callables.

Direct Known Subclasses

Frontend::GeneratedParserBase

Constant Summary collapse

ParseError =

: singleton(Ibex::Runtime::ParseError)

Ibex::Runtime::ParseError
EOF_TOKEN =

: Integer

0
ERROR_TOKEN =

: Integer

1
GENERATED_ACTION_NAME =

: Regexp

/\A_ibex_action_\d+\z/
NO_LOOKAHEAD =

: Object

Object.new.freeze
RECOVERY_SHIFTS =

: Integer

3
EMPTY_ROW =

: Hash[Integer, runtime_value]

empty_row.freeze
EMPTY_LOCATION_NAMES =

: Hash[Symbol, Integer]

empty_location_names.freeze
EMPTY_LOCATIONS =

: Array

empty_locations.freeze
EMPTY_GREEN_TRIVIA =

: Array

empty_green_trivia.freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Observation

#observe, #unobserve

Constructor Details

#initialize(resource_limits: ResourceLimits.new) ⇒ Parser

Returns a new instance of Parser.



404
405
406
407
 
# File 'lib/ibex/runtime/parser.rb', line 404

def initialize(resource_limits: ResourceLimits.new)
  validate_resource_limits!(resource_limits)
  initialize_runtime_state(resource_limits, preserve_existing: false)
end

Instance Attribute Details

#incremental_reused_descendants ⇒ Object (readonly)

: Integer



357
358
359
 
# File 'lib/ibex/runtime/parser.rb', line 357

def incremental_reused_descendants
  @incremental_reused_descendants
end

#syntax_parse_memo ⇒ Object (readonly) 



356
357
358
 
# File 'lib/ibex/runtime/parser.rb', line 356

def syntax_parse_memo
  @syntax_parse_memo
end

Class Method Details

.incremental_session(source_text, resource_limits: nil, blender: true) ⇒ Object

Start a syntax-only incremental session backed by a generated lexer. Parser production actions are suppressed, but generated lexer actions still execute. The generated class is trusted application code, not a sandbox.



396
397
398
399
400
401
 
# File 'lib/ibex/runtime/parser.rb', line 396

def self.incremental_session(source_text, resource_limits: nil, blender: true)
  require_relative "cst" unless defined?(Ibex::Runtime::CST::Kind)
  CST::IncrementalParseSession.new(
    self, source_text, resource_limits: resource_limits, blender: blender
  )
end

.syntax_execution_profile ⇒ Object

Report the trust boundary of syntax-only operations for this loaded parser class. Current generated artifacts can contain user sections and Ruby lexer actions, so they are always trusted application code.



363
364
365
 
# File 'lib/ibex/runtime/parser.rb', line 363

def self.syntax_execution_profile
  :trusted_application_code
end

.syntax_session(source, execution_profile: nil, resource_limits: nil, limits: nil, cancellation: nil, blender: true) ⇒ Object

Open a generated-language syntax session backed by the existing incremental Red/Green CST engine. The execution profile must be passed explicitly because generated lexer actions execute in this path.



371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
 
# File 'lib/ibex/runtime/parser.rb', line 371

def self.syntax_session(
  source,
  execution_profile: nil,
  resource_limits: nil,
  limits: nil,
  cancellation: nil,
  blender: true
)
  SyntaxSession.new(
    self,
    source,
    execution_profile: execution_profile,
    resource_limits: resource_limits,
    limits: limits,
    cancellation: cancellation,
    blender: blender
  )
end

Instance Method Details

#do_parse ⇒ Object

Pull tokens from next_token and execute parser production actions. A generated-lexer next_token executes lexer actions; a handwritten implementation does not invoke the generated lexer.



479
480
481
 
# File 'lib/ibex/runtime/parser.rb', line 479

def do_parse
  drive_parser(nil)
end

#expected_tokens ⇒ Object

Return token names accepted in the current parser state.



698
699
700
701
702
703
704
705
706
707
708
 
# File 'lib/ibex/runtime/parser.rb', line 698

def expected_tokens
  ensure_runtime_initialized!
  return expected_tokens_exact if parser_tables[:exact_expected_tokens]
  return [] if @state_stack.empty?

  state = @state_stack.last
  parser_tables.fetch(:token_names).keys.filter_map do |token_id|
    action = table_lookup(parser_tables.fetch(:actions), state, token_id) || default_action(state) || ERROR_ACTION
    token_to_str(token_id) unless error_action?(action) || token_id == ERROR_TOKEN
  end
end

#expected_tokens_exact ⇒ Object

Return token names that survive all required default reductions. Semantic actions are not evaluated during this lookahead correction.



713
714
715
716
717
718
719
720
721
722
 
# File 'lib/ibex/runtime/parser.rb', line 713

def expected_tokens_exact
  ensure_runtime_initialized!
  return [] if @state_stack.empty?

  parser_tables.fetch(:token_names).keys.filter_map do |token_id|
    next if token_id == ERROR_TOKEN

    token_to_str(token_id) if exact_lookahead_accepted?(token_id)
  end
end

#finish(location: nil) ⇒ Object

Supply EOF to a caller-driven parser session and return its result. Committed reductions execute parser production actions; this token-fed path does not invoke generated lexer actions.



538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
 
# File 'lib/ibex/runtime/parser.rb', line 538

def finish(location: nil)
  run_push_driver do
    start_push_session
    refresh_runtime_fast_path_after_user_code!
    if @repair_policy
      enqueue_or_assign_repair_input(
        RepairInput.new(token_id: EOF_TOKEN, token_name: token_to_str(EOF_TOKEN), value: nil, location: location)
      )
    else
      @lookahead = EOF_TOKEN
      @lookahead_value = nil
      @lookahead_location = location
      @runtime_fast_path = false unless nil.equal?(location)
      materialize_compatible_lookahead
    end
    outcome = run_push_lookahead
    return outcome.fetch(1) if outcome.is_a?(Array)

    raise ParseError, "(input):1:1: parser requested input after EOF"
  end
end

#loc(reference) ⇒ Object

Return the location of a one-based RHS position or named reference while a semantic action is running.

Raises:



727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
 
# File 'lib/ibex/runtime/parser.rb', line 727

def loc(reference)
  locations = @semantic_locations
  raise ParseError, "(runtime):1:1: loc is only available inside a semantic action" unless locations

  index = if reference.is_a?(Integer)
            raise ArgumentError, "location index must be positive" unless reference.positive?

            reference - 1
          else
            names = @semantic_location_names || EMPTY_LOCATION_NAMES
            names.fetch(reference.to_sym) do
              raise ArgumentError, "unknown named location #{reference.inspect}"
            end
          end
  locations.fetch(index) { raise ArgumentError, "location index #{reference.inspect} is outside the RHS" }
end

#next_token ⇒ Object

Override in pull parsers. Return [token, value], [token, value, location], false, or nil.

Raises:

  • (NotImplementedError) 


586
587
588
 
# File 'lib/ibex/runtime/parser.rb', line 586

def next_token
  raise NotImplementedError, "(input):1:1: next_token must be implemented"
end

#on_discard(_token_id, _value, _location, _reason) ⇒ Object

Called when yacc recovery discards an application token.



647
 
# File 'lib/ibex/runtime/parser.rb', line 647

def on_discard(_token_id, _value, _location, _reason); end

#on_error(token_id, value, _value_stack) ⇒ Object

Override to recover from syntax errors. The default raises unless a bounded automatic repair has already been selected.

Raises:



593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
 
# File 'lib/ibex/runtime/parser.rb', line 593

def on_error(token_id, value, _value_stack)
  return if @repair_selected
  return if cst_enabled?

  expected = expected_tokens
  token_name = token_to_str(token_id)
  state = @state_stack.last
  configured = parser_tables.fetch(:error_messages, EMPTY_ROW)[state]
  error_id, detail = configured_error_message(configured)
  raise ParseError.new(
    token_id: token_id,
    token_name: token_name,
    token_value: value,
    expected_tokens: expected,
    location: @lookahead_location,
    state: state,
    suggestions: token_suggestions(token_name, expected),
    error_id: error_id,
    detail: detail
  )
end

#on_error_recover(_token_id, _value, _value_stack) ⇒ Object

Called after the synthetic error token enters a recovery state. The payload describes the original error before recovery popped stacks.



638
 
# File 'lib/ibex/runtime/parser.rb', line 638

def on_error_recover(_token_id, _value, _value_stack); end

#on_error_recover_location(_token_id, _value, _value_stack, _location, _state) ⇒ Object

Location-aware recovery observer.



643
 
# File 'lib/ibex/runtime/parser.rb', line 643

def on_error_recover_location(_token_id, _value, _value_stack, _location, _state); end

#on_reduce(_production_id, _values, _result) ⇒ Object Also known as: __ibex_fast_path_on_reduce

Called after a production's semantic action and goto are committed. Override to observe its id, RHS values, and semantic result.



628
 
# File 'lib/ibex/runtime/parser.rb', line 628

def on_reduce(_production_id, _values, _result); end

#on_reduce_location(_production_id, _values, _result, _locations, _result_location) ⇒ Object Also known as: __ibex_fast_path_on_reduce_location

Location-aware reduction observer.



633
 
# File 'lib/ibex/runtime/parser.rb', line 633

def on_reduce_location(_production_id, _values, _result, _locations, _result_location); end

#on_repair(_plan) ⇒ Object

Called once after a repair is selected and before its edited token prefix is replayed through normal parser actions.



663
 
# File 'lib/ibex/runtime/parser.rb', line 663

def on_repair(_plan); end

#on_shift(_token_id, _value, _state) ⇒ Object Also known as: __ibex_fast_path_on_shift

Called after an ordinary input token is shifted. Override to observe the internal token id, semantic value, and destination state.



618
 
# File 'lib/ibex/runtime/parser.rb', line 618

def on_shift(_token_id, _value, _state); end

#on_shift_location(_token_id, _value, _state, _location) ⇒ Object Also known as: __ibex_fast_path_on_shift_location

Location-aware shift observer. The compatible hook above retains its original signature and runs first.



623
 
# File 'lib/ibex/runtime/parser.rb', line 623

def on_shift_location(_token_id, _value, _state, _location); end

#parse_with_syntax ⇒ Object

Parse through next_token and return both the semantic value and Red root. Parser production actions execute. Generated lexer actions execute only when next_token is supplied by the generated lexer.



487
488
489
 
# File 'lib/ibex/runtime/parser.rb', line 487

def parse_with_syntax
  syntax_parse_result(do_parse)
end

#push(token, value = nil, location = nil) ⇒ Object

Supply one token to a caller-driven parser session. Committed reductions execute parser production actions; this token-fed path does not invoke generated lexer actions. Returns :need_more after consuming it, [:accepted, result] after acceptance, or [:rejected, result] after recovery terminates.

Raises:



515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
 
# File 'lib/ibex/runtime/parser.rb', line 515

def push(token, value = nil, location = nil)
  raise ParseError, "(input):1:1: push requires a token; call finish for EOF" if token.nil? || token == false

  run_push_driver do
    start_push_session
    refresh_runtime_fast_path_after_user_code!
    if @repair_policy
      enqueue_or_assign_repair_input(repair_input(token, value, location))
    else
      @lookahead = internal_token_id(token)
      @lookahead_value = value
      @lookahead_location = location
      @runtime_fast_path = false unless nil.equal?(location)
      materialize_compatible_lookahead
    end
    run_push_lookahead
  end
end

#repair_policy ⇒ Object 



429
430
431
432
 
# File 'lib/ibex/runtime/parser.rb', line 429

def repair_policy
  ensure_runtime_initialized!
  @repair_policy
end

#repair_policy=(policy) ⇒ Object

Enable bounded automatic repair for the next parser session. Assign nil to restore the compatible yacc-only behavior.



443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
 
# File 'lib/ibex/runtime/parser.rb', line 443

def repair_policy=(policy)
  ensure_runtime_initialized!
  unless policy.nil? || policy.is_a?(RepairPolicy)
    raise ArgumentError, "repair_policy must be an Ibex::Runtime::RepairPolicy or nil"
  end

  @runtime_observation_mutex.synchronize do
    ensure_driver_available_without_lock!
    if @push_status == :active
      raise ParseError, "(repair):1:1: repair_policy cannot change during an active push session"
    end

    @repair_policy = policy
  end
end

#reset_push ⇒ Object

Discard a caller-driven session so this parser can accept a new one.



562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
 
# File 'lib/ibex/runtime/parser.rb', line 562

def reset_push
  ensure_runtime_initialized!
  @runtime_observation_mutex.synchronize do
    ensure_driver_available_without_lock!
    @push_status = :idle
    @source = nil
    # steep:ignore:start
    reset_parse_session_state!
    # steep:ignore:end
    @lookahead = NO_LOOKAHEAD
    @lookahead_value = nil
    @lookahead_location = nil
    @runtime_lookahead_token_display = nil
    @repair_input_buffer = nil
    @repair_selected = false
    @runtime_parser_tables = nil
    @runtime_fast_path = false
  end
  nil
end

#resource_limits ⇒ Object 



435
436
437
438
 
# File 'lib/ibex/runtime/parser.rb', line 435

def resource_limits
  ensure_runtime_initialized!
  @resource_limits
end

#resource_limits=(limits) ⇒ Object

Replace the limits used by future sessions.



461
462
463
464
465
466
467
468
469
470
471
472
473
 
# File 'lib/ibex/runtime/parser.rb', line 461

def resource_limits=(limits)
  ensure_runtime_initialized!
  validate_resource_limits!(limits)

  @runtime_observation_mutex.synchronize do
    ensure_driver_available_without_lock!
    if @push_status == :active
      raise ParseError, "(resource):1:1: resource_limits cannot change during an active push session"
    end

    @resource_limits = limits
  end
end

#result_loc ⇒ Object

Return the synthesized span of the reduction being evaluated.



746
747
748
749
750
751
752
 
# File 'lib/ibex/runtime/parser.rb', line 746

def result_loc
  unless @semantic_locations
    raise ParseError, "(runtime):1:1: result_loc is only available inside a semantic action"
  end

  @semantic_result_location
end

#syntax_root ⇒ Object

Return the Red source-file root built by the most recent CST parse.



493
494
495
496
 
# File 'lib/ibex/runtime/parser.rb', line 493

def syntax_root
  ensure_runtime_initialized!
  @syntax_root
end

#token_to_str(token_id) ⇒ Object Also known as: __ibex_fast_path_token_to_str

Return a human-readable name for an internal token id.



667
668
669
670
671
 
# File 'lib/ibex/runtime/parser.rb', line 667

def token_to_str(token_id)
  return @unknown_token_name || token_id.to_s if token_id == @unknown_token_id

  parser_tables.fetch(:token_names).fetch(token_id, token_id.to_s)
end

#trace_value_printer=(printer) ⇒ Object

Install an opt-in value formatter for human-readable yydebug traces.



651
652
653
654
655
656
657
658
 
# File 'lib/ibex/runtime/parser.rb', line 651

def trace_value_printer=(printer)
  ensure_runtime_initialized!
  unless printer.nil? || printer.respond_to?(:call)
    raise ArgumentError, "trace value printer must respond to call or be nil"
  end

  @trace_value_printer = printer
end

#yyaccept ⇒ Object

Accept immediately after the current semantic action completes.



690
691
692
693
694
 
# File 'lib/ibex/runtime/parser.rb', line 690

def yyaccept
  @runtime_fast_path = false
  @accept_requested = true
  nil
end

#yydebug ⇒ Object 



410
411
412
413
 
# File 'lib/ibex/runtime/parser.rb', line 410

def yydebug
  ensure_runtime_initialized!
  @yydebug
end

#yydebug=(enabled) ⇒ Object 



416
417
418
419
420
 
# File 'lib/ibex/runtime/parser.rb', line 416

def yydebug=(enabled)
  ensure_runtime_initialized!
  @runtime_fast_path = false
  @yydebug = enabled
end

#yydebug_output=(output) ⇒ Object 



423
424
425
426
 
# File 'lib/ibex/runtime/parser.rb', line 423

def yydebug_output=(output)
  ensure_runtime_initialized!
  @yydebug_output = output
end

#yyerrok ⇒ Object

Leave error recovery immediately.



683
684
685
686
 
# File 'lib/ibex/runtime/parser.rb', line 683

def yyerrok
  @recovery_shifts = 0
  nil
end

#yyerror ⇒ Object

Enter error recovery from a semantic action without calling on_error.



675
676
677
678
679
 
# File 'lib/ibex/runtime/parser.rb', line 675

def yyerror
  @runtime_fast_path = false
  @semantic_error = true
  nil
end

#yyparse(receiver, method_id) ⇒ Object

Parse tokens yielded by receiver.method_id and execute parser production actions. This caller-fed path does not invoke generated lexer actions.



502
503
504
505
506
507
 
# File 'lib/ibex/runtime/parser.rb', line 502

def yyparse(receiver, method_id)
  stream = Enumerator.new do |tokens|
    receiver.__send__(method_id) { |token| tokens << token }
  end
  drive_parser(-> { stream.next })
end