Tokenizer

Author(s): The Ciao Development Team.

This module defines the tokenizer for Ciao. In addition to optional flags, the main differences w.r.t. the ISO-Prolog standard are:

  • ` is a graphic char, there are no back_quoted_strings.
  • \ followed by any layout char (not only new_line) in a string is a continuation_escape_sequence.
  • \^ starts a control_escape_char in a string.
  • \c skips layout in a string.
  • \e = ESC, \d = DEL, \s = SPACE.
  • 13'23 is 23 in base 13 (same for other bases).
  • 0'' is accepted as 0''' (if not followed by ').

Usage and interface

Documentation on exports

PREDICATEread_tokens/2

Usage:read_tokens(TokenList,Dictionary)

REGTYPEtoken/1
A regular type, defined as follows:
token(atom(A)) :-
        atm(A).
token(badatom(S)) :-
        string(S).
token(number(N)) :-
        num(N).
token(string(S)) :-
        string(S).
token(var(T,S)) :-
        term(T),
        string(S).
token('/* ...').
token(',').
token('(').
token(' (').
token(')').
token('[').
token(']').
token('|').
token('{').
token('}').
token('.').

Documentation on multifiles

PREDICATEdefine_flag/3

Usage:define_flag(Flag,FlagValues,Default)

The predicate is multifile.

Documentation on imports

This module has the following direct dependencies: