Module: Ibex::Runtime::GeneratedLexer

Defined in:
lib/ibex/runtime/generated_lexer.rb

Overview

Runtime contract mixed into parser classes that declare a lexer. rubocop:disable Metrics/ModuleLength -- matching, actions, state, and positions share one session invariant.

Constant Summary collapse

NO_EMISSION =

: Object

Object.new.freeze
EMPTY_GREEN_TRIVIA =

: Array

empty_green_trivia.freeze

Instance Method Summary collapse

Instance Method Details

#lex(source, file: "(input)") ⇒ Object

Reset the generated lexer to the beginning of an input.



34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
 
# File 'lib/ibex/runtime/generated_lexer.rb', line 34

def lex(source, file: "(input)")
  @lexer_input = LexerInput.new(source)
  @lexer_file = file
  @lexer_states = ["INITIAL"]
  @lexer_line = 1
  @lexer_column = 1
  @lexer_byte_column = 1
  @lexer_byte_offset = 0
  @lexer_lexeme = nil
  @lexer_emission = NO_EMISSION
  @lexer_skip_requested = false
  @lexer_pending_green_trivia = []
  configure_lexer_cst
  @lexer_has_token = false
  self
end

#lexer_state ⇒ Object

Return the current named lexer state.



95
96
97
 
# File 'lib/ibex/runtime/generated_lexer.rb', line 95

def lexer_state
  (@lexer_states || ["INITIAL"]).last.to_sym
end

#lexer_state=(state) ⇒ Object

Replace the current named lexer state.



101
102
103
104
105
106
 
# File 'lib/ibex/runtime/generated_lexer.rb', line 101

def lexer_state=(state)
  name = validate_lexer_state(state)
  states = @lexer_states ||= ["INITIAL"]
  states[-1] = name
  name.to_sym
end

#next_token ⇒ Object

Pull one token using longest match and declaration-order tie breaking.

Raises:



110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
 
# File 'lib/ibex/runtime/generated_lexer.rb', line 110

def next_token
  input = @lexer_input
  raise ParseError, "(lexer):1:1: call parse or lex before next_token" unless input

  loop do
    unless ensure_lexer_data?(input)
      location = lexer_zero_width_location
      location[:ibex_lexer_start_state] = lexer_state
      location = attach_cst_trivia(location) if cst_enabled?
      return [nil, nil, location.freeze]
    end

    rule, lexeme = select_lexer_match(input)
    raise_lexer_no_match(input) unless rule && lexeme

    location = consume_lexer_match(input, lexeme)
    location[:ibex_lexer_start_state] = lexer_state
    emitted = apply_lexer_rule(rule, lexeme, location)
    return emitted if emitted
  end
end

#parse(source, file: "(input)") ⇒ Object

Lex and semantically parse one String, IO, or Fiber source. Parser and generated lexer actions execute. Loading the generated file may also execute user sections; this trusted application path is not a sandbox.



56
57
58
59
60
61
62
63
64
65
 
# File 'lib/ibex/runtime/generated_lexer.rb', line 56

def parse(source, file: "(input)")
  lex(source, file: file)
  parser = self #: Parser
  parser.do_parse
rescue ParseError => e
  raise unless parser_tables[:cst].is_a?(Hash)

  parser = self #: Parser
  parser.__send__(:cst_lexical_failure, e)
end

#parse_syntax(source, file: "(input)") ⇒ Object

Parse without executing parser production actions. Generated lexer actions still run because they define tokenization and lexer state. Loading user sections and running lexer actions are trusted application boundaries, so this method is not a no-user-code sandbox.



89
90
91
 
# File 'lib/ibex/runtime/generated_lexer.rb', line 89

def parse_syntax(source, file: "(input)")
  parse_syntax_with_cache(CST::SourceText.new(source, file: file), CST::NodeCache.new)
end

#parse_with_syntax(source, file: "(input)") ⇒ Object

Parse one generated-lexer source and return its semantic and syntax results. Parser and generated lexer actions execute. This trusted application path is not a sandbox.



71
72
73
74
75
76
77
78
79
80
81
82
 
# File 'lib/ibex/runtime/generated_lexer.rb', line 71

def parse_with_syntax(source, file: "(input)")
  lex(source, file: file)
  parser = self #: Parser
  value = parser.do_parse
  parser.__send__(:syntax_parse_result, value)
rescue ParseError => e
  raise unless parser_tables[:cst].is_a?(Hash)

  parser = self #: Parser
  value = parser.__send__(:cst_lexical_failure, e)
  parser.__send__(:syntax_parse_result, value)
end